Attention Word Embedding (2020)

Abstract: Word embedding models learn semantically rich vector representations of words and are widely used to initialize natural processing language (NLP) models. The popular continuous bag-of-words (CBOW) model of word2vec learns a vector embedding by masking a given word in a sentence and then using the other words as a context to predict it. A limitation of CBOW is that it equally weights the context words when making a prediction, which is inefficient, since some words have higher predictive value than others. We tackle this inefficiency by introducing the Attention Word Embedding (AWE) model, which integrates the attention mechanism into the CBOW model. We also propose AWE-S, which incorporates subword information. We demonstrate that AWE and AWE-S outperform the state-of-the-art word embedding models both on a variety of word similarity datasets and when used for initialization of NLP models.

## introduction word embedding models learn vector representations of words such that words that are semantically related are close to each other in the vector space. popular word embedding models include word2vec @xcite , glove @xcite , and fasttext @xcite . word embedding models are often used to initialize deep learning models for various natural language processing (nlp) tasks such as machine translation @xcite , part-ofspeech tagging @xcite , and sentiment analysis @xcite , leading to improved performance over training the models from scratch. the core idea underlying the training methodology of all the above embedding models is that "a word is characterized by the company it keeps" @xcite . for instance, the continuous bag-of-words (cbow) model of word2vec predicts a randomly selected word in a sentence using the other words in the sentence as a context. the context words are typically treated equally by these models. however, clearly some words in the context will be more predictive of the masked word than others, and intuitively, should be weighted more heavily. a promising remedy that treats each context word differently is the attention mechanism @xcite , which enables learning the relative importance of input features for the prediction of the desired output. attention has become the de facto standard and state-of-the-art in modern nlp for a range of different tasks @xcite @xcite . however, to date, there have been only a few attempts to employ it to improve the performance and interpretability of word embedding models @xcite . ## experiments we demonstrate the superior performance of awe and awe-s against cbow, skip-gram, glove, and fasttext on a variety of datasets, which broadly fall into two main categories: because the conventional application of word embedding models has been to initialize nlp models, we use these tasks to assess the quality of our word embeddings for initializing nlp models for downstream applications. datasets include snli @xcite , sick-e @xcite , sick-r @xcite , sts benchmark @xcite , and mrpc @xcite . unlike datasets for the previous task, all the datasets here have a training set. ## conclusion & future work in this work, we have proposed awe and awe-s, which perform significantly better than cbow, skip-gram, glove, and fasttext on a variety of datasets across a diverse set of tasks. the simple setting of masked word prediction in awe also helped us visually analyze and interpret how the attention mechanism works. our analysis revealed that the attention mechanism can figure out the context words that are most relevant for predicting the masked word. in future, we want to explore more applications of the proposed distributional loss for bigger models like bert on larger datasets.