(made with Claude Code and ChatGPT)
MIT 6.7960 — Approximation Theory

Effect of ReLU on Kinks

Shift a function. Clip it at zero. Watch its linear pieces change.

0.000
ReLU(f(x) + b)
Large dot: current bias · small dots: exact thresholds
Why does the count change?

ReLU(z) = max(0, z). A linear segment crosses zero at most once. Starting with N maximal linear pieces, clipping produces at most 2N pieces; neighboring flat pieces merge. A continuous function with P maximal pieces has P − 1 interior kinks.

Try the linear ramp: at b = 0 it has two output pieces; at b = 1.25 it has one. More bias does not always mean more kinks.