Encyclopædia BayesICA
 Bayesian statistics
 "The physics of learning systems"
 Probability is a measure of degree of belief
 Bayes rule tells how beliefs are updated when new observations are made
 Marginalization can be used for prediction and inference about
unknown quantities
 Decisions are based on expected utilities (where expectation is taken
over the current beliefs/probabilities)
 In practice, the integrals are approximated in some way or another
 Ensemble learning
 Bayesian ensemble learning is one type of variational learning
 Misfit between the true posterior and its approximation is measured
by KullbackLeibler information
 Yields bounds for probability of the data given the model structure
 Close connection to minimum description length principle
 Variational learning
 Calculus of variations refers to optimization problems where the
optimand is a function (as opposed to real number or a vector of
real numbers)
 Methods where a simpler function is fitted to the posterior
 Latent variable models
 Models which postulate unobserved variables to explain
regularities in the observations
 Gaussian linear factor analysis model is one of the simples examples:
unknown factors are assumed to be responsible for the observations
 Pretty much the same as generative models
 Generative models

 Models which explicitly state how the observations have been generated
by partly or fully unknown factors
 Pretty much the same as latent variable models
 Unsupervised learning
 The goal of unsupervised learning is to extract an efficient
representation of the statistical structure implicit in the
observations
 Generative or latent variable models are common tools
 Minimum description length (MDL) principle
 Principle which states that the most compact explanation for the
observations should be favoured over other explanations
 Description length L(x) of x is measured in bits and is (more or
less) L(x) = log P(x), where P(x) is the probability of x
 As a learning theory not as well founded theoretically as Bayesian
statistics but can allow intuitive interpretations to various
phenomena in learning
 Bayesian ensemble learning bridges Bayesian statistics and
MDL
Figure caption
Pine trees in winter in Espoo.