this page collects the closed-form solutions to regularised regression (where they exist) and the iterative approximations we fall back on (where they don’t). 𐃏 along the way we will see that regularisation is not an ad-hoc hack but a perfectly sensible artefact of estimation: it drops straight out of MAP (maximum a posteriori) inference once you put a prior on the coefficients.
Regularisation
This page includes 𐃏 my Chapter notes for the book by Michael Nielsen.
- given the findings of the previous chapter (universality), why would we concern ourselves with learning deep neural nets?
- especially given that we are guaranteed to be able to approximate any function with just a single layer of hidden neurons?
well, just because something is possible, it doesn’t mean it’s a good idea!
considering that we are using computers, it’s usually a good idea to break the problem down into smaller sub-problems, solve those, and then come back to solve the main problem.
notes
topics: convolutions, pooling, GPUs (to do more training), algorithmic expansion of data (reduce overfitting), dropout (also reduce overfitting), ensembles of networks
Backlinks (2)
1. Wiki /wiki/
Knowledge is a paradox. The more one understand, the more one realises the vastness of his ignorance.