Embeddings in Natural Language Processing (2020)

Abstract: Embeddings have been one of the most important topics of interest in NLP for the past decade. Representing knowledge through a low-dimensional vector which is easily integrable in modern machine learning models has played a central role in the development of the field. Embedding techniques initially focused on words but the attention soon started to shift to other forms. This tutorial will provide a high-level synthesis of the main embedding techniques in NLP, in the broad sense. We will start by conventional word embeddings (e.g., Word2Vec and GloVe) and then move to other types of embeddings, such as sense-specific and graph alternatives. We will finalize with an overview of the trending contextualized representations (e.g., ELMo and BERT) and explain their potential and impact in NLP.

## description in this tutorial we will start by providing a historical overview on word-level vector space models, and word embeddings in particular. word embeddings (e.g. word2vec @xcite , glove @xcite or fasttext @xcite ) have proven to be powerful keepers of prior knowledge to be integrated into downstream natural language processing (nlp) applications. however, despite their flexibility and success in capturing semantic properties of words, the effectiveness of word embeddings are generally hampered by an important limitation, known as the meaning conflation deficiency: the inability to discriminate among different meanings of a word. a word can have one meaning (monosemous) or multiple meanings (ambiguous). for instance, the noun mouse can refer to two different meanings depending on the context: an animal or a computer device. hence, mouse is said to be ambiguous. in fact, according to the principle of economical versatility of words @xcite , frequent words tend to have more senses. moreover, this meaning conflation can have additional negative impacts on accurate semantic modeling, e.g., semantically unrelated words that are similar to different senses of a word are pulled towards each other in the semantic space @xcite . in our example, the two semantically-unrelated words rat and screen are pulled towards each other in the semantic space for their similarities to two different senses of mouse (see figure 1 ). this, in turn, contributes to the violation of the triangle inequality in euclidean spaces @xcite . accurately capturing the meaning of words (both ambiguous and unambiguous) plays a crucial role in the language understanding of nlp systems. in order to deal with the meaning conflation deficiency, this tutorial covers approaches have attempted to model individual word senses @xcite @xcite @xcite @xcite . sense representation techniques, however, suffer from limitations which hinders their effective application in downstream nlp tasks: they either need vast amounts of training data to obtain reliable representations or require an additional sense disambiguation on the input text to make them integrable into nlp systems. this data is highly expensive to obtain in practice, which causes the so-called knowledge-acquisition bottleneck @xcite . as a practical way to deal with the knowledge-acquisition bottleneck, an emerging branch of research has focused on directly integrating unsupervised embeddings into downstream models. instead of learning a fixed number of senses per word, contextualized word embeddings learn "senses" dynamically, i.e., their representations dynamically change depending on the context in which a word appears. con-text2vec @xcite and elmo @xcite are some of the early examples for this type of representation. these models represent the context of a target word by extracting the embedding of a word in context from a bi-directional lstm language model. the latter further proposed a seamless integration into neural nlp systems, as depicted in figure 2 . more recently, transformers @xcite have proven very effective for encoding contextualized knowledge, thanks to their self-attention mechanism (figure 3 ). bert @xcite , which is based on transformers, has revolutionized the field of representation learning and has impacted many other fields in nlp. many derivatives and subsequent models have followed up, rapidly pushing up the state of the art in different benchmarks. in this tutorial we extensively cover this recent type of representation. we also discuss other types of embeddings, for instance for graph structures which are a popular choice in many scenarios, or for longer units of texts such as sentences and documents. finally, we conclude this tutorial by discussing some of the ethical issues around the implicit gender and stereotypical biases encoded in word embeddings and proposals for reducing these artifacts. ## outline the tutorial is split into seven sections, each of which being roughly self-contained. ## breadth the tutorial is largely based on a recent book written by the instructors published by the synthesis lectures on human language technologies of morgan and claypool, titled "embeddings in natural language processing: theory and advances in vector representations of meaning" (pilehvar and @xcite . this book covers all the various techniques on vectors representations of meaning in detail, while in this tutorial we provide an overview of the main ideas, without getting into details. . ## prerequisites for the attendees no special advanced requirements are required for the attendees, but a certain familiarity with linear algebra, natural language processing and machine learning would be desirable. ## small reading list in addition to the main reference book (pilehvar and camacho-collados, 2020), in the following we present some references that may be helpful for understanding the tutorial. nonetheless, they are not required to read in advance as their main ideas are also covered as part of the tutorial.