PB o S : Probabilistic Bag-of-Subwords for Generalizing Word Embedding (2020)

Abstract: We look into the task of generalizing word embeddings: given a set of pre-trained word vectors over a finite vocabulary, the goal is to predict embedding vectors for out-of-vocabulary words, without extra contextual information. We rely solely on the spellings of words and propose a model, along with an efficient algorithm, that simultaneously models subword segmentation and computes subword-based compositional word embedding. We call the model probabilistic bag-of-subwords (PBoS), as it applies bag-of-subwords for all possible segmentations based on their likelihood. Inspections and affix prediction experiment show that PBoS is able to produce meaningful subword segmentations and subword rankings without any source of explicit morphological knowledge. Word similarity and POS tagging experiments show clear advantages of PBoS over previous subword-level models in the quality of generated word embeddings across languages.

## introduction word embeddings pre-trained over large texts have demonstrated benefits for many nlp tasks, especially when the task is label-deprived. however, many popular pre-trained sets of word embeddings assume fixed finite-size vocabularies 1, 2 , which hinders their ability to provide useful word representations for out-of-vocabulary (oov) words. we look into the task of generalizing word embeddings: extrapolating a set of pre-trained word embeddings to words out of its fixed vocabulary, without extra access to contextual information (e.g. example sentences or text corpus). in contrast, the more common task of learning word embeddings, or often just word embedding, is to obtain distributed representations of words directly from large unlabeled text. the motivation here is to extend the usefulness of pre-trained embeddings without expensive retraining over large text. there have been works showing that contextual information can also help generalize word embeddings (for example, @xcite . we here, however, focus more on the research question of how much one can achieve from just word compositions. in addition, our proposed way of utilizing word composition information can be combined with the contextual embedding algorithms to further improve the performance of generalized embeddings. the hidden assumption here is that words are made of meaningful parts (cf. morphemes) and that the meaning of a word is related to the meaning of their parts. this way, humans are often able to guess the meaning of a word or term they have never seen before. for example, "postemnlp" probably means "after emnlp". different models have been proposed for that task of generalizing word embeddings using word compositions, usually under the name of subword(level) models. @xcite ; @xcite ; @xcite model words at the character level. however, they have been surpassed by later subword-level models, probably because of putting too much burden on the models to form and discover meaningful subwords from characters. bag-of-subwords (bos) is a simple yet effective model for learning @xcite and generalizing @xcite word embeddings. bos composes a word embedding vector by taking the sum or average of the vectors of the subwords (character n-grams) that appear in the given word. however, it ignores the importance of different subwords since all of them are given the same weight. intuitively, "farm" and "land" should be more relevant in composing representation for word "farmland" than some random subwords like "armla". even more favorable would be a model's ability to discover meaningful subword segmentations on its own. @xcite bases their model over morphemes but needs help from an external morphological analyzer such as morfessor @xcite . @xcite use trainable self-attention to combine subword vectors. while the attention implicitly facilitates interactions among subwords, there has been no explicit enforcement of mutual exclusiveness from subword segmentation, making it sometimes difficult to rule out less relevant subwords. for example, "her" is itself a likely subword, but is unlikely to be relevant for "higher" as the remaining "hig" is unlikely. we propose the probabilistic bag-of-subwords (pbos) model for generalizing word embedding. pbos simultaneously models subword segmentation and composition of word representations out of subword representations. the subword segmentation part is a probabilistic model capable of handling ambiguity of subword boundaries and ranking possible segmentations based on their overall likelihood. for each segmentation, we compose a word vector as the sum of all subwords that appear in the segmentation. the final embedding vector is the expectation of the word vectors from all possible segmentations. an alternative view is that the model assigns word-specific weights to subwords based on how likely they appear as meaningful segments for the given word. coupled with an efficient algorithm, our model is able to compose better word embedding vectors with little computational overhead compared to bos. manual inspections show that pbos is able to produce subword segmentations and subword weights that align with human intuition. affix prediction experiment quantitatively shows that the subword weights given by pbos are able to recover most eminent affixes of words with good accuracy. to assess the quality of generated word embeddings, we evaluate with the intrinsic task of word similarity which relates to the semantics; as well as the extrinsic task of part-of-speech (pos) tagging which requires rich information to determine each word's role in a sentence. english word similarity experiment shows that pbos improves the correlation scores over previous best models under vari-ous settings and is the only model that consistently improves over the target pre-trained embeddings. pos tagging experiment over 23 languages shows that pbos improves accuracy compared in all but one language to the previous best models, often by a big margin. we summarize our contributions as follows: ## pbos model following the above intuition, in this section we describe the pbos model in detail. we first develop a model that segments a word into subword and associates each subword segmentation with a likelihood based on the meaningfulness of each subword segment. we then apply bos over each segmentation to compose a "segmentation vector". the final word embedding vector is then the probabilistic expectation of all the segmentation vectors. the subword segmentation and likelihood association part require no explicit source of morphological knowledge and are tightly integrated with the word vector composition part, which in turn gives rise to an efficient algorithm that considers all possible segmentations simultaneously (section 3). the model can be trained by fitting a set of pre-trained word embeddings. ## experiments we design experiments to answer two questions: do the segmentation likelihood and subword weights computed by pbos align with their meaningfulness? are the word embedding vectors generated by pbos of good quality? for the former, we inspect segmentation results and subword weights (section 4.1), and see how good they are at predicting word affixes (section 4.2). for the latter, we evaluate the word embeddings composed by pbos at word similarity task (section 4.3) and part-of-speech (pos) tagging task (section 4.4). due to the page limit, we only report the most relevant settings and results in this section. other details, including hardware, running time and detailed list of hyperparameters, can be found in appendix a. ## related work popular word embedding methods, such as word2vec @xcite , glove @xcite , often assume finite-size vocabularies, giving rise to the problem of oov words. fasttext @xcite attempted to alleviate the problem using subword-level model, and was followed by interests of using subword information to improve word embedding @xcite @xcite @xcite @xcite . among them are charagram by @xcite which, albeit trained on specific downstream tasks, is similar to bos followed by a non-linear activation, and the systematic evaluation by @xcite over various choices of word composition functions and subword segmentation methods. however, all works above either pay little attention to the interaction among subwords inside a given word, or treat subword segmentation and composing word representation as separate problems. another interesting thread of works @xcite @xcite attempted to model language solely at the subword level and learn subword embeddings directly from text, providing evidence to the power of subword-level models, especially as the notion of word is thought doubtful by some linguistics @xcite . besides the recent interest in subwords, there have been long efforts of using morphology to improve word embedding @xcite @xcite @xcite @xcite @xcite @xcite . however, most of them require an external oracle, such as morfessor @xcite , for the morphological segmentations of input words, limiting their power to the quality and availability of such segmenters. the only exception is the character lstm model by @xcite , which has shown some ability to recover the morphological boundary as a byproduct of learning word embedding. the most related works in generalizing pretrained word embeddings have been discussed in section 1 and compared throughout the paper. ## conclusion and future work we propose pbos model for generalizing pretrained word embeddings without contextual information. pbos simultaneously considers all possible subword segmentations of a word and derives meaningful subword weights that lead to better composed word embeddings. experiments on segmentation results, affix prediction, word similarity, and pos tagging over 23 languages support the claim. in the future, it would be interesting to see if pbos can also help with the task of learning word embedding, and how hashing would impact the quality of composed embedding while facilitating a more compact model.