Constructive training of recurrent neural networks using hybrid optimization
Training of recurrent neural networks (RNNs) is known to be a very difficult task. This work proposes a novel constructive method for simultaneous structure and parameter training of Elman-type RNNs using a combination of particle swarm optimization (PSO) and covariance matrix adaptation based evolutionary strategy (CMA-ES). The proposed method allows the imposition of certain stability conditions, which can be maintained throughout the constructive process. The examples reported show a monotonic decrease in training error throughout the constructive process and also demonstrate the efficiency of the proposed method for structure and parameter training of RNNs.
recurrent neural networks hybrid optimization constructive method structure and parameter learning