SVMs & Neural Networks

Support Vector Machines

Finds the separating hyperplane with maximum margin. For non-separable data, uses soft margin with penalty C.

• Margin = 2/||w||; maximize margin ⇒ minimize ||w|| subject to yᵢ(wᵀxᵢ + b) ≥ 1.
• Dual form uses Lagrange multipliers αᵢ; support vectors where αᵢ > 0.

Kernel Trick

Implicitly map inputs to higher-dimensional feature spaces using kernels: Linear, Polynomial, RBF.

Decision function: f(x) = sign( Σ αᵢ yᵢ K(xᵢ,x) + b )
Common K: linear xᵀx', poly (γ xᵀx' + r)^d, RBF exp(-γ ||x-x'||²)
        

Perceptron

Linear classifier trained with a simple update rule; basis for neural networks.

Initialize w,b=0
For each (x,y) with y∈{-1,+1}:
  if y(wᵀx + b) ≤ 0: w := w + η y x; b := b + η y
        

Multi-layer Perceptron

Feedforward neural network with hidden layers and nonlinear activations (ReLU, Sigmoid, Tanh).

• Architecture: input → [Dense + Activation] × L → output
• Loss: MSE (regression), Cross-Entropy (classification)
• Optimizers: SGD, Momentum, Adam

Backpropagation

Computes gradients layer-by-layer using chain rule; optimized with SGD/Adam. Use dropout/weight decay to regularize.

Given loss L and layer zᶫ = Wᶫ aᶫ⁻¹ + bᶫ, aᶫ = σ(zᶫ):
δᶫ = (Wᶫ⁺¹)ᵀ δᶫ⁺¹ ⊙ σ'(zᶫ)
∂L/∂Wᶫ = δᶫ (aᶫ⁻¹)ᵀ ; ∂L/∂bᶫ = δᶫ
        

Training Tips

• Normalize inputs; use learning-rate schedules and batch norm.
• Monitor validation loss; apply early stopping and checkpoints.
• Initialize weights properly (Xavier/He); set proper weight decay.
• Check data leakage and class imbalance; use stratification.