Statistical Machine Learning

• Fundamentals of Learning
  • Entropy, Cross-Entropy, NLL, KL
  • Defining the Objective
  • Optimisation for Optimality
• Statistical Decision Theory
  • Statistical Inference
    • Analytic Solutions
    • Approximating The Analytic Solutions
    • Bias-Variance Tradeoff
    • Model Selection and Model Assessment
  • Notation
  • Curse of Dimensionality
• Linear Methods for Regression
  • Objective Functions from MLE
  • Linear Regression
    • Optimisation: Least Squares
    • Orthogonalisation for Mutltiple Regression
  • Subset Selection Methods
  • Shrinkage Methods
    • Ridge Regression
    • LASSO
• Linear Methods for Classification
  • Discriminant Classifiers
    • Probabilistic Classifiers
    • Discriminative Models
    • Generative Models
    • Comparison Between LDA and Logistic Regression
  • Hyperplane Classifiers
    • Perceptron
    • Max-Margin Classifier
• Basis Expansion
  • Finite Dimensional Expansion
    • Polynomial
    • Piece-wise Polynomials
    • Polynomial Spline
    • Natural Spline
    • Smoothing Spline
    • Non-linear Classification
    • Moving Beyond 1 Dimension
  • Infinite Dimensional Expansion
    • A Point Mapping to a Function
    • Kernel Ridge Regression
    • Kernel Support Vector Machine
• Decision Trees
  • Classification and Regression Trees (CART)
    • Motivation: Approximating Bayes Estimators
    • Inhomogeneity (Impurity) Measure
    • Categorical Predictors
  • Bagging
    • Bootstrap Samples
  • Random Forest
    • Variance Reduction in Averaging Correlated Models
    • Algorithm Outline
  • Boosting
    • AdaBoost.M1: Algorithm Outline
    • Forward Stage-wise Additive Modeling
    • AdaBoost as a Additive Model
    • Gradient Boosting: Algorithm Outline
  • Resources
• Unsupervised Learning
  • Dimensionality Reduction
    • Geometric
    • Variational
    • Evaluation Criteria
  • Clustering
    • Metric Based
    • Density Based
    • Evaluation Criteria
  • Anomaly Detection
    • Classifier Based
    • Density Based
    • Evaluation Criteria
• Causal Inference
  • Explanatory Models
  • Causal Inference
    • Resources
    • Course Plan
  • Notes
    • Average Treatment Effect (ATE)
    • Individual Treatment Effect (ITE)
    • Summary of Techniques
    • References
• Distribution Shift
  • Definitions