(aside image)

Encyclopædia BayesICA

Bayesian statistics
  • "The physics of learning systems"
  • Probability is a measure of degree of belief
  • Bayes rule tells how beliefs are updated when new observations are made
  • Marginalization can be used for prediction and inference about unknown quantities
  • Decisions are based on expected utilities (where expectation is taken over the current beliefs/probabilities)
  • In practice, the integrals are approximated in some way or another
Ensemble learning
  • Bayesian ensemble learning is one type of variational learning
  • Misfit between the true posterior and its approximation is measured by Kullback-Leibler information
  • Yields bounds for probability of the data given the model structure
  • Close connection to minimum description length principle
Variational learning
  • Calculus of variations refers to optimization problems where the optimand is a function (as opposed to real number or a vector of real numbers)
  • Methods where a simpler function is fitted to the posterior
Latent variable models
  • Models which postulate unobserved variables to explain regularities in the observations
  • Gaussian linear factor analysis model is one of the simples examples: unknown factors are assumed to be responsible for the observations
  • Pretty much the same as generative models
Generative models
  • Models which explicitly state how the observations have been generated by partly or fully unknown factors
  • Pretty much the same as latent variable models
Unsupervised learning
  • The goal of unsupervised learning is to extract an efficient representation of the statistical structure implicit in the observations
  • Generative or latent variable models are common tools
Minimum description length (MDL) principle
  • Principle which states that the most compact explanation for the observations should be favoured over other explanations
  • Description length L(x) of x is measured in bits and is (more or less) L(x) = -log P(x), where P(x) is the probability of x
  • As a learning theory not as well founded theoretically as Bayesian statistics but can allow intuitive interpretations to various phenomena in learning
  • Bayesian ensemble learning bridges Bayesian statistics and MDL

Figure caption

Pine trees in winter in Espoo.