Optimizing Large-Scale Hyperparameters via Automated Learning Algorithm
Authors: Bin Gu, Guodong Liu, Yanfu Zhang, Xiang Geng, Heng Huang
Abstract: Modern machine learning algorithms usually involve tuning multiple (from one to  thousands) hyperparameters which play a pivotal role  in terms of model  generalizability. Globally choosing appropriate values of hyperparameters is extremely  computationally challenging. Black-box optimization and gradient-based algorithms are  two dominant approaches to hyperparameter optimization while  they have totally  distinct advantages. How to design a new hyperparameter optimization technique inheriting all benefits  from both approaches is still an open problem. To address this challenging problem, in this paper, we  propose a new  hyperparameter optimization  method with zeroth-order hyper-gradients (HOZOG). Specifically, we first exactly formulate  hyperparameter optimization  as  an $\mathcal{A}$-based constrained optimization problem, where $\mathcal{A}$ is  a black-box optimization algorithm (such as deep neural network). Then, we use the average zeroth-order hyper-gradients  to update  hyperparameters. We provide the feasibility analysis of using  HOZOG to achieve hyperparameter optimization.  The experimental  results on three representative  hyperparameter (the size is from 1 to 1250) optimization tasks  demonstrate the benefits of HOZOG  in terms of  \textit{simplicity, scalability, flexibility, effectiveness and efficiency} compared with the  state-of-the-art hyperparameter optimization methods.