Enhancing Out-of-Vocabulary Estimation with Subword Attention (2023)

Abstract: Word embedding methods like word2vec and GloVe have been shown to learn strong representations of words. However, these methods only learn representations for words in the training corpus and therefore struggle to handle unknown and new words, known as out-of-vocabulary (OOV) words. As a result, there have been multiple attempts to learn OOV word representations in a similar fashion to how humans learn new words, using word roots/subwords and/or surrounding words. However, while most of these approaches use advanced architectures like attention on the context of the OOV word, they tend to use simple structures like ngram addition or character based convolutional neural networks (CNN) to handle processing subword information. In response to this, we propose SubAtt, a transformer based OOV estimation model that uses attention mechanisms on both the context and the subwords. In addition to attention, we also show that pretraining subword representations also leads to improvement in OOV estimation. We show SubAtt outperforms current state-of-the-art OOV estimation models.

## introduction word embeddings are very useful in natural language processing tasks. methods like word2vec @xcite and glove @xcite train strong semantic representations of words using co-occurrence statistics on a large text corpus, and have been shown to be effective at semantically representing text data. however, one weakness of these methods is that they only learn representations for words that exist in the training corpus, and therefore have no representations on unknown terms, known as out-of-vocabulary (oov) words. contextualized embeddings like bert @xcite also suffer from weak performance on rare and unknown words, despite being able to build a contextualized representation of them @xcite . therefore learning representations for oov words is an important endeavour. in this work, we focus on static embeddings, where a large amount of oov work is focused on, and leave contextualized embedding to future work. current approaches combine subword and context information to estimate oov words. while these approaches apply attention mechanisms to aggregate context representations, they tend to do very little with subword representations. as a result, this paper proposes subatt, a deep neural network attention model that estimates oov word representations using attention layers @xcite on the subwords in addition to the contexts. subatt also pretrains subword representations, allowing it to learn quality representations before combining it with context. we show that both pretraining and applying attention on subwords improves oov estimates, and show that subatt generally outperforms state-of-the-art oov estimation models in both intrinsic and extrinsic tasks. ## related work there are multiple strategies to estimate oov embeddings. some oov strategies use word roots of the oov word to estimate oov embeddings @xcite @xcite while other methods use the oov word's context @xcite @xcite @xcite . however, more recent attempts combine both subwords and context approaches. schick and schütze propose the form-context model (schick and schütze, 2019c), which estimates oov embeddings by combining the sum of ngram embeddings (learned by the model) with the sum of word embeddings in the contexts multiplied by a weight matrix (also learned by the model). this model has been extended to the attentive mimicking model (schick and schütze, 2019a) which adds an attention mechanism to the context calculations. a second combined approach is the attention based hierarchical context encoder, known as hice @xcite . hice is a transformer based model that leverages the hierarchical structure of contexts, using a transformer encoder to encode each context sentence into a sentence embedding, and then using another transformer encoder to combine each sentence embedding into a full context embedding. it estimates subword information using a character based convolutional neural network (cnn), and then combines each piece into a final oov embedding. hice also adapts its model to the oov word's corpus using model-agnostic meta-learning (maml) @xcite . another approach, estimator vectors @xcite , trains its own word embeddings, along with subword and context embeddings for oov estimation. bertram (schick and schütze, 2019b) applies an approach similar to the above models, but for contextualized embedding models like bert @xcite . while these approaches create strong estimates for oov words, they have some weaknesses. first, although some use attention mechanisms with the context of an oov word, none of the aforementioned combined approaches use attention for processing the oov's subwords foot_0 . secondly, none of the static embedding approaches pretrain their subwords; they learn these representations at the same time as the whole model 2 . therefore, we propose subatt, a model that uses attention and pretraining on subwords, leading to stronger oov estimates. ## subatt we now present subatt, a transformer @xcite based model for oov estimation. first, we describe pretraining the subword representations in section 3.1, then how the model encodes each context sentence in section 3.2, and finally how subatt combines subword and context information in section 3.3. ## conclusion we propose subatt, an attention based model that estimates oov words by using pretrained subword embeddings and subword attention. we show through various experiments that this model estimates more accurate representations of oov words.