Regression & Classification

Linear Regression

Models a linear relationship y ≈ wᵀx + b by minimizing Mean Squared Error (MSE):

MSE(w,b) = (1/n) Σ (yᵢ - (wᵀxᵢ + b))²
Closed form (Normal Eq.): θ = (XᵀX)⁻¹ Xᵀ y (if invertible)
        

• Training via Gradient Descent or Normal Equation
• Assumptions: linearity, independence, homoscedasticity, normal errors
• Metrics: MAE, MSE, RMSE, R²

Gradient Descent

Iteratively update parameters to minimize loss.

Initialize w,b
Repeat until convergence:
  w := w - α (2/n) Σ xᵢ (wᵀxᵢ + b - yᵢ)
  b := b - α (2/n) Σ (wᵀxᵢ + b - yᵢ)
        

• Variants: Batch, Stochastic (SGD), Mini-batch
• Learning rate α: too high diverges; too low is slow

Logistic Regression

Binary classification with sigmoid σ(z)=1/(1+e^{-z}). Optimizes log loss:

L(w,b) = -(1/n) Σ [ yᵢ log(σ(wᵀxᵢ+b)) + (1-yᵢ) log(1-σ(wᵀxᵢ+b)) ]
        

• Decision boundary from p(y=1|x) ≥ threshold
• Regularization to prevent overfitting
• Metrics: Accuracy, Precision-Recall, F1, ROC-AUC

Regularization

• L2 (Ridge): Adds λ||w||² — shrinks weights uniformly; closed form becomes (XᵀX + λI)⁻¹ Xᵀy
• L1 (Lasso): Adds λ||w||₁ — promotes sparsity/feature selection; solved by coordinate descent
• Elastic Net: Combination of L1 and L2

Classification Metrics

	Predicted +	Predicted −
Actual +	TP	FN
Actual −	FP	TN

• Precision = TP/(TP+FP); Recall = TP/(TP+FN)
• F1 = 2PR/(P+R); ROC-AUC measures rank quality

Model Validation & Cross-Validation

• Hold-out, K-Fold, Stratified K-Fold
• Bias–variance trade-off
• Hyperparameter tuning with Grid/Random Search

// Pseudocode for K-Fold
for k in 1..K:
  train = data - fold[k]
  valid = fold[k]
  fit model on train; score on valid
avg score over K folds