AI

Ridge Regression

$$ J(\beta) = \frac{1}{2n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \;+\; \lambda \sum_{j=1}^{p} \beta_j^2 $$

$$ \hat{y}_i = \beta_0 + \beta_1 x_{i1} + \cdots + \beta_p x_{ip} $$

$$ J(\beta) = \frac{1}{2n} \| y - X\beta \|_2^2 + \lambda \|\beta\|_2^2 $$

$$ \beta = (X^T X + \lambda I)^{-1} X^T y $$

$$ \nabla_{\beta} J(\beta) = -\frac{1}{n} X^T (y - X\beta) + 2\lambda \beta $$

$$ \beta := \beta - \alpha \left( -\frac{1}{n} X^T (y - X\beta) + 2\lambda \beta \right) $$

$$ \beta := \beta - \alpha \left( \frac{1}{n} X^T (X\beta - y) + 2\lambda \beta \right) $$