FAME : F eature-Based Adversarial Meta-Embeddings for Robust Input Representations (2021)

Abstract: Combining several embeddings typically improves performance in downstream tasks as different embeddings encode different information. It has been shown that even models using embeddings from transformers still benefit from the inclusion of standard word embeddings. However, the combination of embeddings of different types and dimensions is challenging. As an alternative to attention-based meta-embeddings, we propose feature-based adversarial meta-embeddings (FAME) with an attention function that is guided by features reflecting word-specific properties, such as shape and frequency, and show that this is beneficial to handle subword-based embeddings. In addition, FAME uses adversarial training to optimize the mappings of differently-sized embeddings to the same space. We demonstrate that FAME works effectively across languages and domains for sequence labeling and sentence classification, in particular in low-resource settings. FAME sets the new state of the art for POS tagging in 27 languages, various NER settings and question classification in different domains.

## introduction recent work on word embeddings and pre-trained language models has shown the large impact of language representations on natural language processing (nlp) models across tasks and domains @xcite @xcite . nowadays, a large number of different embedding models are available with different characteristics, such as different input granularities @xcite vs. subword-based (e.g., @xcite vs. characterbased (e.g., @xcite @xcite ), or different data used for pre-training (general-world vs. specific domain). since those characteristics directly influence when embeddings are most effective, combinations of different embedding models are likely to be beneficial @xcite @xcite , even when using already powerful largescale pre-trained language models @xcite . word-based embeddings, for instance, are strong in modeling frequent words while character-based embeddings can model outof-vocabulary words. similarly, domain-specific embeddings can capture in-domain words that do not appear in general domains like news text. different word representations can be combined using so-called meta-embeddings. there are several methods available, ranging from concatenation (e.g., @xcite , over averaging (e.g., @xcite to attentionbased meta-embeddings @xcite . however, they all come with shortcomings: concatenation leads to high-dimensional input vectors and, as a result, requires additional parameters in the first layer of the neural network. averaging simply merges all information into one vector, not allowing the network to focus on specific embedding types which might be more effective than others to represent the current word. attention-based embeddings address this problem by allowing dynamic combinations of embeddings depending on the current input token. however, the calculation of attention weights requires the model to assess the quality of embeddings for a specific word. this is arguably very challenging when embeddings of different input granularities are combined, e.g., subwords and words. infrequent in-domain tokens, for instance, are hard to detect when using subword-based embeddings as they can model any token. moreover, both average and attention-based meta-embeddings require a mapping of all embeddings into the same space which can be challenging for a set of embeddings with different dimensions. in this paper, we propose feature-based adversarial meta-embeddings (fame) that (1) align the embedding spaces with adversarial training, and (2) use attention for combining embeddings with a layer that is guided by features reflecting wordspecific properties, such as the shape or frequency of the word and, thus, can help the model to assess the quality of the different embeddings. by using attention, we avoid the shortcomings of concatenation (high-dimensional input vectors) and averaging (merging information without focus). further, our contributions mitigate the challenges of previous attention-based meta-embeddings: in our analysis, we show that the first contribution is especially beneficial when embeddings of different dimensions are combined. the second helps, in particular, when combining word-based with subwordbased embeddings. we conduct experiments across a variety of tasks, languages and domains, including sequencelabeling tasks (named entity recognition (ner) for four languages, concept extraction for two special domains (clinical and materials science), and partof-speech tagging (pos) for 27 languages) and sentence classification tasks (question classification in different domains). our results and analyses show that fame outperforms existing meta-embedding methods and that even powerful fine-tuned transformer models can benefit from additional embeddings using our method. in particular, fame sets the new state of the art for pos tagging in all 27 languages, for ner in two languages, as well as on all tested concept extraction and two question classification datasets. in summary, our contributions are metaembeddings with (i) adversarial training and (ii) a feature-based attention function. (iii) we perform broad experiments, ablation studies and analyses which demonstrate that our method is highly effective across tasks, domains and languages, including low-resource settings. (iv) moreover, we show that even representations from large-scale pretrained transformer models can benefit from our meta-embeddings approach. the code for fame is publicly available foot_0 and compatible with the flair framework @xcite . ## related work this section surveys related work on metaembeddings, attention and adversarial training. ## neural architectures in this section, we present the architectures we use for text classification and sequence tagging. note that our contribution concerns the input representation layer, which can be used with any nlp model, e.g., also sequence-to-sequence models. ## experiments and results we now describe the tasks and datasets we use in our experiments as well as our results.