When training generative models, we usually optimize parameters to optimize joint likelihood. However, this is in no way (or is it? let me know if you know better) a guarantee that you’ll do better on many real-world benchmarks such as precision/recall. So I wondered, what would happen if you optimized for these quantities? Let’s take [...]
Entries from December 2008
December 6, 2008
Even more predictive models
In the last post, I presented a comparison of different ways of doing prediction. A natural follow-up question is whether or not there are even better functions? I observed in the last post that a straight line performs better than the logistic function which has a downward hump. A natural set of functions to explore [...]
December 3, 2008
Deriving and evaluating the second order approximation for links
In the previous post I argued that the second order approximation is useful for prediction. Let’s apply that to a model with links and see what happens. The random variable over which we take the expectation is now and the second order term is then where is the Hessian matrix. We address each of [...]
December 1, 2008
Further approximation improvements
I was a little bit sneaky in my previous post and the reason I say this will become apparent when I change the function being approximated. Let’s break it down like this: there are three things we want to compare and And just to add a little more notation, let and In the previous post, [...]
December 1, 2008
Answers to two questions
In the previous post, I posed two questions. I’ll answer the second first. This question considers what would happen if the response function (any response function) were to depend only on a single latent variable. To use the notation of the previous post, I’d write Here I will be a little more general and [...]
