Custom Optimizer in OpenMx
I'm currently working on a custom optimizer for lasso regularized OpenMx models. The fitting function in these models is:
$$f(\theta) = L(\theta)+\sum_j |\theta_j|$$
where $L$ is the unregularized Likelihood. The problem is that the second part of the fitting function is non-differentiable, so I can't use the built in optimizers (if I understood it correctly). However, I want to use the gradients of the Likelihood with respect to the parameters computed by OpenMx for the first part of the fitting function in my custom optimizer. To get these gradients, I am using a custom compute plan:
gradientModel = mxModel(mxObject,
mxComputeSequence(steps=list(
mxComputeNumericDeriv(checkGradient = FALSE, hessian = FALSE))
)
)
mxRun(gradientModel)
gradients = gradientModel$compute$steps[[1]]$output[["gradient"]]
I then compute a step direction and a step size and update the parameters using omxSetParameters(gradientModel, ...). I repeat this procedure until a convergence criterion is reached. However, building and running the gradientModel in each iteration makes everything quite slow. Is there a faster way to get the gradients of the model?
Thanks,
Jannik
Wow, neat
Log in or register to post comments
abs() probably no big deal; derivative-free optimization
Well, it is true that the 3 main optimizers (CSOLNP, SLSQP, and NPSOL) are derivative-based quasi-Newton optimizers. Nonetheless, they can still get good results if the fitfunction involves the absolute-value function, or (NPSOL and SLSQP only) if the constraint functions involve `abs()`. Be advised, though, that OpenMx may give spurious status Red if the use of `abs()` in the fitfunction makes it impossible to zero the gradient at the solution.
However, OpenMx has built-in support for derivative-free optimizers as well. They're not settable as "Default optimizer", so you need to make a custom compute plan to use them (which shouldn't be too difficult for you, since you've already figured out how to make custom compute plans). Specifically, OpenMx has an original Nelder-Mead implementation, as well as two implementations of simulated annealing.
Log in or register to post comments
Thanks!
I have not yet tried out any of the derivative free optimizers of OpenMx, but this seems to be a very promising approach, too.
Log in or register to post comments
@jannik I'm curious if you
Best,
Caspar
Log in or register to post comments
mxregsem
Log in or register to post comments