  Although modern machine learning, in particular deep learning, has achieved
outstanding success in scientific and engineering research, most of the neural
networks (NNs) learned via these state-of-the-art techniques are black-box
models. For a widespread success of machine learning in science and
engineering, it is important to develop new NN architectures to effectively
extract high-level mathematical knowledge from complex dataset. To meet this
research demand, this paper focuses on the symbolic regression problem and
develops a new NN architecture called the Gumbel-Max Equation Learner (GMEQL)
network. Different from previously proposed Equation Learner (EQL) networks,
GMEQL applies continuous relaxation to the network structure via the Gumbel-Max
trick and introduces two types of trainable parameters: structure parameters
and regression parameters. This paper also proposes a new two-stage training
process and new techniques to train structure parameters in both the online and
offline settings based on an elite repository. On 8 benchmark symbolic
regression problems, GMEQL is experimentally shown to outperform several
cutting-edge techniques for symbolic regression.
