The Big Picture

Probabilistic Framework

Note

  • For a given task, we can collect the stuff that we care about in a set \(\Omega\).

  • The goal of the probabilistic machine learning framework is to be able to define a probability measure, \(\mathbb{P}(\omega\in\Omega)\)

    • Equivalently, we can define a (or a set of) random variable(s) \(S(\omega):2^{\Omega}\mapsto\mathbb{R}\) and define a

      • distribution \(F_S(s)=\mathbb{P}(S\leq s)\) or a

      • density \(f_S(s)\) such that

        \[F_S(s)=\int\limits_{-\infty}^s f_S(t)\mathop{dt}\]

  • The types of items that can be in the set \(\Omega\) can be quite diverse, and therefore the associated rv can have the range which can confront to different types of mathematical structures.

    • Discrete variables:

      • A single binary variable, \(S\in\{0,1\}\).

      • A categorical variable, \(S\in\{1,\cdots,K\}\).

    • Continuous variables:

      • Real number \(S\in\mathbb{R}\)

    • Variables of higher dimensions

      • Finite dimensional Euclidean vectors \(\mathbf{S}\in\mathbb{R}^d\) with a common practice of associating each dimension with its own separate rv such that \(S_i\in\mathbb{R}\).

      • Infinite sequences \((S)_{i=1}^\infty\) where each \(S_i\in\mathbb{R}\).

Defining the Probabilities

Discrete variables

Note

  • For discrete variables, we can enumerate the class-marginal probabilities in using a finite set of parameters

  • Bernoulli, \(S\sim\mathrm{Ber}(p)\)

    \[f_S(s=\{0,1\})=p^s(1-p)^{1-s}\]

  • Multinoulli, \(S\sim\mathrm{Mult}(\pi_1\cdots\pi_K)\)

    \[f_S(s=(s_1,\cdots,s_K))=\prod_{k=1}^K\pi_k^{s_k}\]

Continuous variables

Note

  • For continuous case, explicit definition of the infinite dimensional density function is impossible.

  • We often therefore resort to some known parametric family of distributions

    • Example: Exponential family

Variables of higher dimensions

Discrete case

Note

  • TODO - splitting the joint distribution by product of conditionals

  • Mention that enumerating the table is exponential in the number of variables

  • Mention conditional indedepence

Continuous case

Note

  • Mention conditional independence and using it to define parametric family (GMM)

  • Mention Markov property for sequence models

  • Mention the lower manifold in the data space

Usage of the Framework

Note

  • We can use this framework to estimate some quantities of interest from the distribution e.g.

    • predict the mean \(\mathbb{E}_{S\sim F_S}[S]\) [regression framework]

    • predict the mode \(s^*=\underset{s}{\arg\max} f_S(s)\) [classification framework]

    • generate samples \(s^*=s\sim F_S\) [generative framework]

Utilising the Observations

Note

  • Data can be provided to help learn the parameters of the probability models

  • [Important] For regression and classification \(S=Y\) and for generative models \(S=X\).

The Inference (Learning) Problem

Note

  • Importance of the MLE framework to be able to learn the distributions through means of statistical inference.