Robust B acked-off E stimation of O ut-of- V ocabulary E mbeddings (2020)

Abstract: Out-of-vocabulary (oov) words cause serious troubles in solving natural language tasks with a neural network. Existing approaches to this problem resort to using subwords, which are shorter and more ambiguous units than words, in order to represent oov words with a bag of subwords. In this study, inspired by the processes for creating words from known words, we propose a robust method of estimating oov word embeddings by referring to pre-trained word embeddings for known words with similar surfaces to target oov words. We collect known words by segmenting oov words and by approximate string matching, and we then aggregate their pre-trained embeddings. Experimental results show that the obtained oov word embeddings improve not only word similarity tasks but also downstream tasks in Twitter and biomedical domains where oov words often appear, even when the computed oov embeddings are integrated into a bert-based strong baseline.

## introduction the dynamic nature of language and the limited size of training data requires neural network models to handle out-of-vocabulary (oov) words that are absent from the training data. we thus use an unk embedding shared among diverse oov words or break those oov words into semanticallyambiguous subwords (even characters), leading to poor task performance @xcite . to solve this problem, several approaches @xcite @xcite learn subword embeddings from pre-trained embeddings and then use these subword embeddings for computing oov word embeddings ( § 2). however, the embeddings computed by these ap- * currently, he works for ntt laboratories. ## related work existing approaches for leveraging surface information in computing oov word embeddings basically learn the embeddings of characters or subwords to reconstruct pre-trained word embeddings from them and then use the obtained embeddings to compute embeddings for oov words @xcite @xcite . @xcite proposed bag-of-subwords (bos) to reconstruct pre-trained word embeddings from bag-ofcharacter n-grams in the same way as fasttext @xcite . @xcite extended bos to reduce the number of embedding vectors and introduce a self-attention mechanism into the aggregation of subword embeddings. however, these methods compute embeddings via ambiguous character or subword embeddings. this will degrade the quality of embeddings for target oov words as we will confirm later in § 4. other approaches utilize the embeddings of the known words around a target oov word as its contextual information @xcite @xcite . @xcite reported that they can improve bert @xcite for understanding rare words. notably, in these approaches for utilizing both surface and context information, the surface-based embeddings are the same as @xcite . these approaches can have difficulties in representing misspelled words or spelling variations when a small number of contexts are available in a text corpus. several approaches utilize external data such as a knowledge base @xcite @xcite . existing approaches successfully impute oov word embeddings by convolutional graph neural network @xcite or by spectral embeddings derived from an affinity matrix of entities @xcite . these approaches can have difficulties in representing oov words that do not exist in the external data and have little versatile applicability to misspelled words. recently, contextualized word embeddings such as bert @xcite mitigate the problem of subword ambiguities by dynamically inferring meanings of oov words from their contexts. however, several researchers reported that bert remains brittle to misspellings @xcite , rare words (schick and schütze, 2020), and out-of-domain samples @xcite . pre-trained word embeddings are reported to be more effective for these cases and morphological tasks such as entity typing and ner @xcite . we therefore improve not only models based on pre-trained word embeddings but also the brittle subword-based bert. our approach will broaden the application range of neural-network models. in this section, we describe our method of computing embeddings for target oov words by using a weighted sum of pre-trained word embeddings. specifically, we calculate the weights over known words from similarity scores derived on the basis of their surface information (figure 1 ). in what follows, we first describe how to retrieve known words that have surfaces similar to a target oov word ( § 3.1), and we next describe how to aggregate their pre-trained embeddings on the basis of the similarity scores calculated through a neural network surface encoder ( § 3.2). we then discuss how to integrate our method into the bert-based fine tuning model @xcite ) ( § 3.3). ## experiments we evaluated the performance of oov word embeddings calculated by the proposed method. we performed intrinsic evaluations on benchmark datasets ( § 4.1) and extrinsic evaluations on two downstream tasks: named entity recognition (ner) and part-of-speech (pos) tagging ( § 4.2). we then conducted experiments on the extension to a bertbased fine-tuning model ( § 4.3 and § 4.4). we used glove.840b embeddings 1 (2.2m vocabulary size) as the pre-trained embeddings (known words) following @xcite . in all the experiments, we used pytorch (v1.0.1) foot_2 as the core architecture and regarded words that were absent from glove.840b as oov words. for the extrinsic evaluations ( § 4.2, § 4.3 and § 4.4), the reported numbers are the medians of five trials. ## conclusion in this paper, inspired by two major processes for creating words, we proposed a method for computing oov word embeddings by learning the similarities between a target oov word and known words and integrated the method into bert. we conducted intrinsic and extrinsic evaluations, and we confirmed that the proposed method more successfully mimics the pre-trained word embeddings for oov words than existing subword-based methods that suffer from the noisiness and ambiguity of intermediate subwords. the proposed method boosted the performance of bert and equipped bert with robustness to adversarial typos. we will release all code to promote the reproducibility of our results. 9 in future work, we will investigate better integrating our method into bert.