kernel methods are the great arbitrage of classical machine learning: keep the algorithm linear — with all its convexity and closed forms — but run it in a feature space so large it can bend around anything, and never pay for that space explicitly. 𐃏 one identity powers everything: if your algorithm touches the data only through inner products, you may replace every \(\langle x, x’\rangle\) with a kernel \(k(x, x’)\) and thereby work in the implicit feature space of \(k\) — possibly infinite-dimensional — at the cost of an \(n \times n\) matrix.
Machine Learning
2026-06-27
Type 1 error
there is exactly one theorem in machine learning that every practitioner rederives on a whiteboard at least once a year, and this is it. 𐃏 the squared-error risk of any learned predictor splits into three non-negative pieces — irreducible noise, squared bias, and variance — and every design decision you make (model class, regularisation strength, \(k\), ensemble size, early stopping) is secretly a transaction between the last two.
a loss function is not a detail of training — it is the definition of the problem. choose squared error and you have asked for the conditional mean; choose absolute error and you have asked for the median; choose hinge and you have asked only for the decision boundary; choose cross-entropy and you have asked for the whole probability. 𐃏 this page catalogues the standard losses, proves what each one’s minimiser actually is, and draws the classic picture that unifies the classification zoo: every one of them is a bribe paid to make the 0–1 loss differentiable.
geometric intuition is trained in \(p \le 3\) and it does not survive the trip upstairs. 𐃏 in high dimensions the volume of a cube hides in its corners, every point is near the boundary, all pairwise distances look alike, and “local” neighbourhoods must stretch almost the full width of the space before they contain any data. every method that reasons from closeness — knn, kernel smoothers, rbf kernels — inherits these pathologies at once.
averaged over all possible problems, every learning algorithm is exactly as good as random guessing — and every optimiser is exactly as good as blind enumeration. 𐃏 this sounds like nihilism but is actually the sharpest possible argument for inductive bias: an algorithm can only beat chance on some problems by losing to chance on others, so the whole game of machine learning is choosing whose lunch to eat.
a model is only as good as the number you judge it by, and most of the classic modelling disasters are really metric disasters — a fraud detector with \(99.9\%\) accuracy that never flags anything, a medical test tuned to a roc curve nobody deployed at the published threshold. 𐃏 this page is the field guide: what each metric measures, what it silently assumes, and which one to reach for when the classes are lopsided, the probabilities matter, or the target is continuous.
value-based methods (q-learning and family) learn how good actions are and act by argmax. policy-gradient methods skip the middleman: parameterise the policy itself, \(\pi_\theta(a \mid s)\), and do gradient ascent on expected return. 𐃏 the entire family — reinforce, actor-critic, trpo, ppo, and by extension rlhf — rests on one identity, the policy gradient theorem, whose derivation is three lines of calculus and one very good idea. the standard reference is sutton & barto, free at http://incompleteideas.net/book/the-book-2nd.html.
q-learning is the algorithm that made reinforcement learning feel inevitable: interact with an unknown world, nudge a table of numbers after every step, and the table converges to the value of optimal behaviour — even while you behave suboptimally the entire time. 𐃏 everything runs on one line of arithmetic, and the rest of this page is the machinery needed to say precisely why that line works. the canonical reference for all of it is sutton & barto’s reinforcement learning: an introduction, free at http://incompleteideas.net/book/the-book-2nd.html.
We also have models that walk around with the dataset in their carry-on. These are models such as:
- Decision Trees
- SVM
- Nonparametric Regressions: K-nearest neighbours, Locally Weighted
- Random Forests
deep learning pipeline
Recall that a Neural Network follows the following construction:
- Pass data (forward) through model to get predicted values
- Calculate loss with predicted values against labels
- Perform backpropagation w.r.t each weight / bias to get the direction in which to move that weight such that it moves closer to the global minima
- Update parameters with gradients using an optimiser.
momentum
ball’s pace slows down this makes total fkn sense! if the gradient signs are the same, increasing your confidence in that direction and move further. you want to take less steps over all
Backlinks (3)
1. My Undergraduate Computer Science Degree /blog/ugrad-unsw/
Summary
It took me 1,577 days to complete my undergraduate degree in Computer Science.
/
Overall, I think the degree was worthwhile and satiated my desire to obtain an understanding of Computers from the bit-level.
2. Literate Programming and Donald Knuth /blog/literate-programming/
I was first introduced to this concept by Distrotube (Derek Taylor’s) “literate config” files. At the time I was not using emacs and thus all the code I was writing was sparingly commented.
Since then, I have entered a world of Machine Learning and Deep Learning, where suddenly in 4 lines, I can sit atop my high-horse and perform sentiment analysis with tensorflow and keras!
from transformers import pipeline
classifier = pipeline('sentiment-analysis')
prediction = classifier("Donald Knuth was the greatest computer scientist.")[0]
print(prediction)
In such an age of abstraction complexity, it becomes paramount to distill what is happening at the last few \((n-k)\) layers.
3. Wiki /wiki/
Knowledge is a paradox. The more one understand, the more one realises the vastness of his ignorance.