Comments on: When is first-order better than second-order?

By: Chang et al. (2009) Reading Tea Leaves: How Humans Interpret Topic Models « LingPipe Blog

Wed, 18 Nov 2009 21:22:24 +0000

[...] They do the usual sample cross-entropy rate evaluations (aka [pseudo expected] predictive log likelihoods). Reporting these to four decimal places is a mistake, because the different estimation methods for the various models have more variance than the differences shown here. Also, there’s a huge effect from the priors. For both points, check out Asuncion et al.’s analysis of LDA estimation, which the first author, Jonathan Chang, blogged about. [...]

By: slycoder

slycoder — Thu, 01 Oct 2009 17:31:01 +0000

So the second order approximation has two parts, one is expanding the conditional probabilities to second order, the other is evaluating its expectation. The first part is easy, it amounts to a covariance matrix and some weighting.

In the CVB paper, one computes the covariance matrix with respect to the variational distribution (because it’s not easy to do so with respect to the true distribution). Since there’s no need to bring in a variational distribution for the definition of CVB0, it’s unclear how you would extend it to second order tractably.

By: Max Welling

Max Welling — Thu, 01 Oct 2009 15:45:44 +0000

Hi Jonathan,

How are you?
This derivation is related to what Yee Whye, Dave and I called the “Callen equations” in our CVB paper.

One thing perhaps to mention is that this is only an approximation of the implicit equation that still needs to be iterated to convergence. Hence, errors can build up and the final answer can be very wrong. Instead, a variational approximation has an approximation to the objective function which may therefore have more value.

Another question: what is the second order approximation of what you call “the true expectations”? Isn’t that exactly equal to the CVB equations as well? I guess I am not quite clear what the difference is between true and variational expectations or how they are defined.