How Well Can a Genetic Algorithm Fine-tune Transformer Encoders? A First Approach (2024)

Abstract: Genetic Algorithms (GAs) have been studied across different fields such as engineering or medicine to optimize diverse problems such as network routing, or medical image segmentation. Moreover, they have been used to automatically find optimal architectures for deep neural networks. However, to our knowledge, they have not been applied as a weight optimizer for the Transformer model. While gradient descent has been the main paradigm for this task, we believe that GAs have advantages to bring to the table. In this paper, we will show that even though GAs are capable of fine-tuning Transformer encoders, their generalization ability is considerably poorer than that from Adam; however, on a closer look, GAs ability to exploit knowledge from 2 different pretraining datasets surpasses Adam’s ability to do so.

## introduction genetic algorithms (gas), a set of optimization methods, although widely studied in other fields such as electric engineering @xcite or medicine @xcite , have not played a big role in the field of nlp as gradient descent algorithms. indeed, a disadvantage of gas is high running times when the search space is big. however, gas possess advantages that make us reconsider their usefulness in nlp such as 1) algorithmic simplicity, 2) no vanishingor exploding-gradient problems since no gradient signal is necessary, and 3) any mathematical expression can be optimized, such as accuracy. on the other hand, gradient descent approaches such as adam @xcite are widely used not only due to the high fine-tuning scores they achieve for nlp models, but also due to a common -and barely challenged-assumption that prevails in the nlp field: fine-tuning a transformer encoder that has been pretrained on two datasets will lead to considerably better scores than fine-tuning an encoder that was pretrained on either of the two pretraining datasets since, according to scaling laws @xcite , it is assumed that the former encoder has learned linguistic and (or) world knowledge from the two datasets, as opposed to the latter encoder which has only acquired knowledge from one dataset. however, we pose some skepticism on adam's ability to efficiently exploit hidden knowledge from the 2 pretraining datasets encoded in such encoders. in this paper, we propose a two-sided study of the ability of a ga to fine-tune pretrained transformer encoders. firstly, we study how well a ga can fine-tune pretrained encoders for the task of sentiment analysis across three datasets. and secondly, we put adam to the test by comparing its ability to leverage pretrained knowledge from 2 pretraining datasets, at fine-tuning time, with respect to the ability of a ga to do so. our main hypothesis is that the ga's crossover operator is the key factor to both fine-tune pretrained encoders and efficiently exploiting the knowledge from two pretraining datasets. to our knowledge, this is the first study of fine-tuning transformer encoders via gas. interestingly, our results are divided: we encountered both a negative and a positive result. on the one hand, although we confirm our hypothesis and show the ability of a ga to fine-tune transformer encoders, we find two big deficiencies when compared against adam: considerably higher training times (up to 46x) and a high drop in accuracy scores (up to 28 points) -a negative result. on the other hand, our results show that the ga outperforms adam's ability to leverage knowledge from two pretraining datasets at fine-tuning time: fine-tuning encoders, pretrained on 2 datasets, via adam leads to an average gain of 0.55 accuracy points with respect to fine-tuning encoders pretrained on only one pretraining dataset; but the ga's mean gain in performance under the same scenario is 1.65 points, a relative increase of 200% (and up to 1540% for a particular case) -a positive result. overall, we believe gas hold as an efficient mechanism for knowledge recombination of transformer encoders. we hope the community will follow our work to carry out a deeper exploration of gas on more challenging tasks. ## discussion and conclusions how well can ega fine-tune transformer encoders? we observed in