Regularized Logistic Regression Gradients

In the previous post, I showed how to compute the approximate objective function for regularization.  I implemented such a harness in R with some real data from one of my runs to prototype it.  Directly using nlm seems to work in fits and starts, perhaps because the optimization routine does not use gradients.  The next step then is to compute the gradients.   The gradient of the zero’th order term is \sigma(\mu) (\vec{p}^+ \circ \vec{p}^+), where \mu = \eta^T \vec{p} \circ \vec{p} + \nu and \vec{p}^+ = \langle \vec{p}, 1\rangle.  

The second order term is where it gets hard.  Taking the derivative by parts, we get \frac{1}{2} A''(\mu) \nabla \mathrm{Var}(x) + \frac{1}{2}\mathrm{Var}(x)A'''(\mu)(\vec{p}^+ \circ \vec{p}^+). To compute \nabla \mathrm{Var}(x), we use the derivation in the previous post to show that this is equal to 2 \eta \circ (V - \mathrm{Diag}(C)) + C \eta + \eta \circ \mathrm{Diag}(C).

I implement all this in R and it seems to converge erratically.  Further more, the “converged” values perform worse than the not fitting at all.   So to summarize results:

  • Both \psi_e and \psi_\sigma perform best without any M-Step fitting.
  • \psi_\sigma > \psi_e when the first is unfit but the second is fit.
  • \psi_e > \psi_\sigma when both are fit or both are unfit.

This is all very peculiar.  In order to ameliorate the fitting process, I collapsed the parameter space so that \vec{\eta} = \langle \eta, \ldots, \eta\rangle so that there are two parameters.   This seems to converge ok, and give results comparable to without fitting for \psi_\sigma.  When I do the similar parameter collapsing in \psi_e it performs worse.

In summary, it seems like across the board not fitting is still the best.


Leave a comment

Filed under Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s