  Nonlinear Mixed effects models are hidden variables models that are widely
used in many field such as pharmacometrics. In such models, the distribution
characteristics of hidden variables can be specified by including several
parameters such as covariates or correlations which must be selected. Recent
development of pharmacogenomics has brought averaged/high dimensional problems
to the field of nonlinear mixed effects modeling for which standard covariates
selection techniques like stepwise methods are not well suited. This work
proposes to select covariates and correlation parameters using a penalized
likelihood approach. The penalized likelihood problem is solved using a
stochastic proximal gradient algorithm to avoid inner-outer iterations. Speed
of convergence of the proximal gradient algorithm is improved by the use of
component-wise adaptive gradient step sizes. The practical implementation and
tuning of the proximal gradient algorithm is explored using simulations.
Calibration of regularization parameters is performed by minimizing the
Bayesian Information Criterion using particle swarm optimization, a zero order
optimization procedure. The use of warm restart and parallelization allows to
reduce significantly computing time. The performance of the proposed method
compared to the traditional grid search strategy is explored using simulated
data. Finally, an application to real data from two pharmacokinetics studies is
provided, one studying an antifibrinolitic and the other studying an
antibiotic.
