References

David Blei has many good write-ups / talks on this topic

• https://youtu.be/Dv86zdWjJKQ
  • Slides
  • "Review" Paper
• https://www.cs.princeton.edu/courses/archive/fall11/cos597C/lectures/variational-inference-i.pdf

Others

• https://ermongroup.github.io/cs228-notes/inference/variational/
• https://turing.ml/dev/docs/for-developers/variational_inference
• TTIC-31230 - Fundamentals of Deep Learning (ELBO)
• CS 285 (Sergey Levine)

Intro

Assumption:

• $x = x_{1:n}$: observations
• $z = z_{1:m}$: hidden variables

We want to model the posterior distribution (notice that this is a form of inference: estimating a hidden variable from observations):

The posterior links the data and a model. In most of the interesting problems, calculating denominator is not tractable. $x$ is the evidence about $z$.

The main idea: pick a family of distributions over the latent variables parameterized with its own variation parameters:

then find a setting of $\nu$ which makes it closest to the posterior of interest.


The closeness can be measured by Kullback-Leibler (KL) divergence:

We use the arguments ($q$ and $p$) in this order specifically to take expectation over $q$. If you flip the order (i.e., $\textrm{KL}(p\|q)$), that is called expectation propagation. It's a different kind of variational inference and it's more computationally expensive in general.

We cannot minimize this KL divergence directly (why??). But we can minimize a function that is equal to it up to a constant (ELBO).

The Evidence Lower Bound (ELBO)

Notes:

• The last term $\log p(x)$ is independent of $q$, thus:
• Minimizing KL divergence is equiv to Maximizing ELBO

:::message ELBO derivation Using Jensen's inequality:

Another derivation: forcibly extract KL divergence

The left hand side is called evidence probability. Hence ELBO. :::

The difference between the ELBO and the KL divergence is the log normalizer --- which is what the ELBO bounds (???).

Variational Auto Encoder (Pretty much the same thing)

Latent variable models:

We have data population, so we want to estimate $\Phi$ and $\Theta$ based on it:

The problem is that we can't typically compute $P_{\Phi, \Theta}(x)$.

• $P_{\Phi, \Theta}(x) = \int_z P_\Phi (z) P_\Theta (x|z) dz$ doesn't work as the sum is too large
• The same sum but with importance sampling with $P_{\Phi, \Theta}(z|x)$ is a better idea but doesn't work: (why???)

Variational Bayes sidesteps this with a model $P_\Psi(z|x)$ that approximate $P_{\Phi, \Theta}(z|x)$.

The ELBO:

Thus,

Minor but important: we can't do gradient descent w.r.t. $\Psi$ as there's sampling procedure. We use re-parameterization trick to circumvent this.

• $P_\Phi(z)$: the prior
• $P_\Psi(z|x)$: the encoder
• $P_\Theta(x|z)$: the decoder
• $E[\log P_\Psi(z|x)/P_\Phi(z)]$: rate term, KL-divergence
• $E[- \log P_\Theta (x | z)]$: distortion, Conditional entropy

:::message Something more that are covered in TTIC31230

• EM (Expectation-Maximization) algorithm is indeed a specific instantiation of VAE! EM corresponds to minimiing the VAE objective:
  • First w.r.t. encoder ($\Psi$): Inference step -- E step
  • And then w.r.t. $\Phi$ and $\Theta$, while fixing $\Psi$: Update step -- M step
• VAE is exactly the same as Rate Distortion Autoencoder (RDA) :::

Rate-Distortion Autoencoders (mathematically the same as VAE)

Setting: Image compression where an image $x$ is compressed to $z$.

We assume a stochastic compression algorithm (encoder): $P_\text{enc}(z|x)$

• $H(z)$: The number of bits needed for the compressed file. This is the rate (bits / image) for transmitting compressed images
  • This is modeled with a prior model $P_\text{pri}(z)$
• $H(x|z)$: The number of additional bits needed to exactly recover $x$. This is a measure of the distortion of $x$
  • This is modeled with a decoder model $P_\text{dec}(x|z)$