Interactive demos

Thu 9/17 — Approximation theory

01 — Approximation

Fit a 1D target with a two-layer ReLU MLP. Slide H to see how more hidden units improve the piecewise-linear approximation.
02 — Effect of ReLU on Kinks

Shift different functions with bias and see how ReLU creates or removes kinks, including on the Telgarsky sawtooth.
03 — Width vs Depth

Fix the total neuron budget N, then vary the aspect ratio. Compare wide-shallow and narrow-deep networks on the same task.
04 — Sinusoidal Positional Encoding

Plain MLP versus a positional-encoding MLP on a noisy scatter plot. Drag H and L to see how the encoding unlocks high-frequency fitting.

Thu 10/1 — Generalization

Double descent

Test error falls as the model grows, spikes when it can fit every training point, then falls again. Ridge removes the spike.