(made with Claude Code and ChatGPT)
Overview
01 Approximation
02 Effect of ReLU on Kinks
03 Width vs Depth
04 Sinusoidal Positional Encoding
MIT 6.7960 — Approximation Theory
Overview
From one ReLU to learned functions, network shape, and richer inputs.
01 — Approximation
Fit a 1D target with a two-layer ReLU MLP. Slide H to see how more hidden units improve the piecewise-linear approximation.
02 — Effect of ReLU on Kinks
Shift different functions with bias and see how ReLU creates or removes kinks, including on the Telgarsky sawtooth.
03 — Width vs Depth
Fix the total neuron budget N, then vary the aspect ratio (width/depth). Compare wide-shallow vs narrow-deep networks on the same task.
04 — Sinusoidal Positional Encoding
Side-by-side: plain MLP vs pos-enc MLP fitting a noisy scatter plot. Drag H and L to see how encoding unlocks high-frequency fitting.