Explain L1 and L2 Regularization

Theoretical Background: Regularization techniques are used to prevent overfitting by adding a penalty to the loss function. For linear regression, the loss function is typically the mean squared error (MSE): $J(\theta) = \frac{1}{2m} \sum (h_\theta(x^i) - y^i)^2$ where $h_\theta(x^i)$ is the hypothesis function, $y^i$ are the true values, and $m$ is the number of training examples.

L1 Regularization (Lasso): Adds the sum of the absolute values of the coefficients to the loss function: $J(\theta) = \frac{1}{2m} \sum (h_\theta(x^i) - y^i)^2 + \lambda \sum |\theta_j|$ This can result in sparse models with some coefficients being exactly zero, effectively selecting a subset of features.

L2 Regularization (Ridge): Adds the sum of the squares of the coefficients to the loss function: $J(\theta) = \frac{1}{2m} \sum (h_\theta(x^i) - y^i)^2 + \lambda \sum \theta_j^2$ This typically results in a model where all coefficients are small but not zero, which helps in managing multicollinearity and stabilizing the solution.

Practical Applications:

• L1 Regularization is useful in high-dimensional datasets where feature selection is beneficial. It's commonly used in scenarios like text classification with many features (e.g., word counts).
• L2 Regularization is preferred in situations where you have many features that may be correlated, such as in ridge regression for finance data modeling.

Python Code Example:

from sklearn.linear_model import Lasso, Ridge

# Lasso example
lasso = Lasso(alpha=0.1)
lasso.fit(X_train, y_train)

# Ridge example
ridge = Ridge(alpha=0.1)
ridge.fit(X_train, y_train)

External References:

• Scikit-learn documentation on Lasso and Ridge
• Regularization in Machine Learning

Mermaid Diagram:

graph LR
A[Linear Model] --> B[L1 Regularization]
A --> C[L2 Regularization]
B --> D{Effects}
C --> E{Effects}
D --> |Feature Selection| F(Sparse Model)
E --> |Stability| G(Reduced Overfitting)