Automated Concatenation of Embeddings for Structured Prediction (2021)

Abstract: Pretrained contextualized embeddings are powerful word representations for structured prediction tasks. Recent work found that better word representations can be obtained by concatenating different types of embeddings. However, the selection of embeddings to form the best concatenated representation usually varies depending on the task and the collection of candidate embeddings, and the ever-increasing number of embedding types makes it a more difficult problem. In this paper, we propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks, based on a formulation inspired by recent progress on neural architecture search. Specifically, a controller alternately samples a concatenation of embeddings, according to its current belief of the effectiveness of individual embedding types in consideration for a task, and updates the belief based on a reward. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of a task model, which is fed with the sampled concatenation as input and trained on a task dataset. Empirical results on 6 tasks and 21 datasets show that our approach outperforms strong baselines and achieves state-of-the-art performance with fine-tuned embeddings in all the evaluations.

## introduction recent developments on pretrained contextualized embeddings have significantly improved the performance of structured prediction tasks in natural language processing. approaches based on contextualized embeddings, such as elmo @xcite , flair @xcite , bert @xcite , and xlm-r @xcite , have been consistently raising the state-of-the-art for various structured prediction tasks. concurrently, research has also showed that word representations based on the concatenation of multiple pretrained contextualized embeddings and traditional non-contextualized embeddings (such as word2vec @xcite and character embeddings (santos and zadrozny, 2014)) can further improve performance @xcite @xcite . given the ever-increasing number of embedding learning methods that operate on different granularities (e.g., word, subword, or character level) and with different model architectures, choosing the best embeddings to concatenate for a specific task becomes non-trivial, and exploring all possible concatenations can be prohibitively demanding in computing resources. neural architecture search (nas) is an active area of research in deep learning to automatically search for better model architectures, and has achieved state-of-the-art performance on various tasks in computer vision, such as image classification @xcite , semantic segmentation @xcite , and object detection @xcite . in natural language processing, nas has been successfully applied to find better rnn structures @xcite and recently better transformer structures @xcite . in this paper, we propose automated concatenation of embeddings (ace) to automate the process of finding better concatenations of embeddings for structured prediction tasks. ace is formulated as an nas problem. in this approach, an iterative search process is guided by a controller based on its belief that models the ef-fectiveness of individual embedding candidates in consideration for a specific task. at each step, the controller samples a concatenation of embeddings according to the belief model and then feeds the concatenated word representations as inputs to a task model, which in turn is trained on the task dataset and returns the model accuracy as a reward signal to update the belief model. we use the policy gradient algorithm @xcite in reinforcement learning @xcite to solve the optimization problem. in order to improve the efficiency of the search process, we also design a special reward function by accumulating all the rewards based on the transformation between the current concatenation and all previously sampled concatenations. our approach is different from previous work on nas in the following aspects: empirical results show that ace outperforms strong baselines. furthermore, when ace is applied to concatenate pretrained contextualized embeddings fine-tuned on specific tasks, we can achieve state-of-the-art accuracy on 6 structured prediction tasks including named entity recognition @xcite , part-of-speech tagging @xcite , chunking @xcite , aspect extraction @xcite , syntactic dependency parsing @xcite and semantic dependency parsing @xcite over 21 datasets. besides, we also analyze the advantage of ace and reward function design over the baselines and show the advantage of ace over ensemble models. ## automated concatenation of embeddings in ace, a task model and a controller interact with each other repeatedly. the task model predicts the task output, while the controller searches for better embedding concatenation as the word representation for the task model to achieve higher accuracy. given an embedding concatenation generated from the controller, the task model is trained over the task data and returns a reward to the controller. the controller receives the reward to update its parameter and samples a new embedding concatenation for the task model. figure 1 shows the general architecture of our approach. ## experiments we use iso 639-1 language codes to represent languages in the table foot_1 . ## discussion: practical usability of ace concatenating multiple embeddings is a commonly used approach to improve accuracy of structured prediction. however, such approaches can be computationally costly as multiple language models are used as input. ace is more practical than concatenating all embeddings as it can remove those embeddings that are not very useful in the concatenation. moreover, ace models can be used to guide the training of weaker models through techniques such as knowledge distillation in structured prediction @xcite @xcite , leading to models that are both stronger and faster. ## conclusion in this paper, we propose automated concatenation of embeddings, which automatically searches for better embedding concatenation for structured prediction tasks. we design a simple search space and use the reinforcement learning with a novel reward function to efficiently guide the controller to search for better embedding concatenations. we take the change of embedding concatenations into the reward function design and show that our new reward function is stronger than the simpler ones. results show that ace outperforms strong baselines. together with fine-tuned embeddings, ace achieves state-of-the-art performance in 6 tasks over 21 datasets. junru zhou and hai zhao. 2019. head-driven phrase structure grammar parsing on penn treebank. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 2396-2408, florence, italy. association for computational linguistics. wei zhu, xiaoling wang, xipeng qiu, yuan ni, and guotong xie. 2020. autotrans: automating transformer design via reinforced architecture search. arxiv preprint arxiv:2009.02070. barret zoph and quoc v le. 2017. neural architecture search with reinforcement learning. in international conference on learning representations. barret zoph, vijay vasudevan, jonathon shlens, and quoc v le. 2018. learning transferable architectures for scalable image recognition. in proceedings of the ieee conference on computer vision and pattern recognition, pages 8697-8710. a detailed configurations evaluation to evaluate our models, we use f1 score to evaluate ner, chunking and ae, use accuracy to evaluate pos tagging, use unlabeled attachment score (uas) and labeled attachment score (las) to evaluate dp, and use labeled f1 score to evaluate sdp. task models and controller for sequencestructured tasks (i.e., ner, pos tagging, chunking, aspect extraction), we use a batch size of 32 sentences and an sgd optimizer with a learning rate of 0.1. we anneal the learning rate by 0.5 when there is no accuracy improvement on the development set for 5 epochs. we set the maximum training epoch to 150. for graph-structured tasks (i.e., dp and sdp), we use adam @xcite to optimize the model with a learning rate of 0.002. we anneal the learning rate by 0.75 for every 5000 iterations following @xcite . we set the maximum training epoch to 300. for dp, we run the maximum spanning tree @xcite algorithm to output valid trees in testing. we fix the hyper-parameters of the task models. we tune the learning rate for the controller among {0.1, 0.2, 0.3, 0.4, 0.5} and the discount factor among {0.1, 0.3, 0.5, 0.7, 0.9} on the same dataset in section 5.2. we search for the hyperparameter through grid search and find a learning rate of 0.1 and a discount factor of 0.5 performs the best on the development set. the controller's parameters are initialized to all 0 so that each candidate is selected evenly in the first two time steps. we use stochastic gradient descent (sgd) to optimize the controller. the training time depends on the task and dataset size. take the conll english ner dataset as an example. it takes 45 gpu hours to train the controller for 30 steps on a single tesla p100 gpu, which is an acceptable training time in practice.