An Unsupervised Method for Learning Representations of Multi-word Expressions for Semantic Classification (2020)

Abstract: This paper explores an unsupervised approach to learning a compositional representation function for multi-word expressions (MWEs), and evaluates it on the Tratz dataset, which associates two-word expressions with the semantic relation between the compound constituents (e.g. the label employer is associated with the noun compound government agency) (Tratz, 2011). The composition function is based on recurrent neural networks, and is trained using the Skip-Gram objective to predict the words in the context of MWEs. Thus our approach can naturally leverage large unlabeled text sources. Further, our method can make use of provided MWEs when available, but can also function as a completely unsupervised algorithm, using MWE boundaries predicted by a single, domain-agnostic part-of-speech pattern. With pre-defined MWE boundaries, our method outperforms the previous state-of-the-art performance on the coarse-grained evaluation of the Tratz dataset (Tratz, 2011), with an F1 score of 50.4%. The unsupervised version of our method approaches the performance of the supervised one, and even outperforms it in some configurations.

## introduction multi-word expressions (mwes) are fundamental to language and, as such, having a robust semantic representation for mwes is important for any natural language processing task that involves text understanding such as information extraction, or question answering (e.g., da @xcite @xcite . while mwes have received attention in recent years, leading to considerable progress in learning mwe representations @xcite @xcite @xcite @xcite , we argue that the proposed methods have limitations. first, some methods require concatenating words in specified mwes, and treating the resulting mwe phrases as atomic units. training a set of dedicated distributional embeddings for the new multi-word terms @xcite suffers from language sparsity. for example, the mwe "red flower" is two orders of magnitude less frequent in google search results than the noun "flower," which is likely to affect the quality of its learned mwe representation. these methods additionally have no straightforward way of handling mwes that are out of vocabulary. second, other approaches require supervision for mwe boundaries @xcite , which hinders scalability and portability to different languages. in all situations, the reliance on having determined your entities of interest ahead of time threatens to dramatically reduce the real-world utility of these approaches. here, we propose a method that addresses both limitations by combining recent advances in language modeling @xcite with the simplicity and proven capability of the skip-gram training objective @xcite . our approach is summarized in figure 1 . intuitively, our method has two components. we use a bidirectional long short-term memory network (bilstm) @xcite to encode each mwe. then, we use this encoding to predict the words in the context of the mwe, similarly to the original skip-gram algorithm. importantly, figure 1 : proposed architecture for learning mwe representations using a bilstm. during training of the language model, for a given sentence that contains a mwe, e.g., new york, we use skip-gram to maximize the probability of the words in the right context, e.g., after and two, using the hidden state of york produced by the left-to-right (forward) lstm, and of the words in the left context, e.g., moved and to, using the hidden state of new produced by the right-to-left (backward) lstm. to generate a represention for the mwe for downstream tasks, we average the last bilstm hidden states for the mwe, e.g., the hidden state for york from the left-to-right lstm and the hidden state for new from the right-to-left lstm. specifically, the contributions of this work are: ## related work much like distributional similarity approaches for learning word representations @xcite @xcite , a semantic representation of mwes can be trained using a distributional approach that treats mwes as single tokens @xcite . however, this approach cannot handle out of vocabulary (oov) mwes, and it is likely to suffer from sparsity @xcite , particularly as the mwes grow in length. on the other hand, compositional approaches address these issues by learning a function to compose the representations of the mwe constituent words. @xcite proposes a compositional method which minimizes the l2 distance between the predicted, compositional embedding of the mwe and an observed, distributional one. however, while there are two methods for obtaining the distributional embedding, both suffer from an observer effect. in the first method, a single set of embeddings is trained and the corpus is modified such that sentences that contain mwes are included in their original form as well as a copy with the mwe treated as a single token. this augmentation, however, alters the relative frequency of the words, which consequently affects the embeddings of the mwe constituents. in the second method, two distinct sets of embeddings are trained, one with no alteration and one with mwes treated as single tokens. thus the embeddings of the constituents are unaffected, but since the words in the two corpora have different relative frequencies, the resultant vector spaces are different. our proposed approach does not suffer from this observer effect, as we learn our compositional function indirectly (through skip-gram), without relying on a distributionally learned embedding for mwes for training. @xcite also avoid this reliance on the gold embedding of the multi-word, learning the function indirectly. the compositional function is used to encode the multiword and its paraphrase and is then trained to maximize the cosine similarity between the encodings. the paraphrases are generated either using backtranslation @xcite , or by treating frequent joint co-occurrences as paraphrases. however, this approach is outperformed by much cheaper unsupervised approaches, such as the average of the constituents' emebddings @xcite . furthermore, the backtranslation approach depends on an external system, adding complexity to the model and restricting the languages and domains of application. alternatively, a compositional function can be trained directly with a language modeling objective, leveraging the vast amounts of available unlabeled data. our proposed approach falls into this category, and is similar in nature with the work of @xcite , as both approaches use an adapted skip-gram to learn a composition function. however, there are two key differences: they used a series of hand-crafted features, such as word clusters, while we rely only on word embeddings. second, they assume that the word boundaries are given, while our system can work in a completely unsupervised setting, allowing it to be applied to domains, and, potentially, other languages, with no pre-determined set of entities of interest. further, though they explore a recurrent neural network (rnn) variant, it relies on the availability of a constituency parser (which is unavailable for many domains and languages), is unable to scale (as noted by yu and dredze), and still requires knowing the mwe boundaries ahead of time. we address all of these limitations with our approach. ## approach we propose an approach for extending the skip-gram method @xcite to handle multiword expressions (mwes). @xcite proposed an extension that handles multiwords by treating them as single tokens during training, mwes outside that training vocabulary have no representation. we propose an unsupervised method for learning a composition function capable of producing a representation of a mwe from the embeddings of its components, using bidirectional recurrent neural networks (rnns). ## 4 experiments and results similar to @xcite , we evaluate the quality of the learned mwe representations on the task and dataset introduced in @xcite . this is a multi-class classification problem, where the task is to classify the semantic relation between the head of the mwe and the modifier. (e.g., roof tile is labeled as location, as roof provides the location of the tile). the tratz dataset has two splits between train and test: random (where the mwes are randomly split between train and test), and lexical (where they are assigned based on their head word such that there is no lexical overlap between train and test). each of these splits can also be evaluated with coarse-grained or fine-grained labels. for example, security fear is labeled as topical in the coarse-grained lexical dataset, and as topic of cognition&emotion in the fine-grained dataset lexical. in we report results on both the random and lexical splits of the data for completeness. however, we argue that the random split does not produce a representative evaluation, because methods that rely on in the first setting it has no access to context at evaluation time, i.e., the rnns run only over the mwe constituents. in the second setting, the method has access to context in the form of n sentences that contain the mwe at evaluation time. in this setting, our method averages the embeddings produced from all these n sentences, as illustrated in figure simple memorization perform artificially high due to the lexical overlap between train and test. for example, a simple baseline that predicts the label most commonly seen in training with the last word (typically the head word in a two-word mwe) achieves near state-of-the-art performance in the random split. for this reason, we focus most of the discussion in this paper on results measured on the lexical split of data, where this overlap is avoided. we evaluate and compare different variations of our method against previous work and baselines. as each approach creates a vector representation of the mwe, in order to predict the discrete labels required by the task, we fit a linear classifier on top of this vector. similar to @xcite , we select the best performing classifier on the development partition from 10 possible classifiers. @xcite we implemented five baselines, which fall in two categories: (a) baselines without a classifier, and (b) baselines that employ a linear classifier on the predicted mwe embedding (similar to our actual method). the first class of baselines includes: (i) random, which predicts the label by randomly sampling from the label distribution observed in training; (ii) majority label, which produces the majority label in the dataset, as observed in training; (iii) majority train head, which predicts the label most commonly seen with the last word of the mwe, which is typically its syntactic head. the second category contains (i) fasttext average, where for a given mwe we train a linear classifier on top of the average of the fasttext embedding of each individual word, and (ii) fasttext max pool, which is similar to average, but uses a max pooling function rather than the mean. we performed our empirical analysis under three high-level settings: (a) supervised, where both distributional embeddings and boundaries of mwes are required during training, (b) supervised boundaries, where only mwe boundaries are provided (from the tratz dataset), and (c) no supervision, which does not require either of them (and uses our pos pattern to generate silver boundaries). in the latter setting, we sampled 75 thousand mwes generated using our pos pattern, using the same sampling heuristic as in @xcite . we made sure that these training mwes do not overlap lexically with the mwe in the testing partition. our methods fall under the last two categories: supervised boundaries or no supervision. in each category, we evaluate three variants of our method: (a) mwe-only rnn, which does not use context at all, i.e., it applies the rnns solely on the mwe constituent words, (b) context-aware rnn (train only), which uses contextual information only during training but not in testing, and (c) context-aware rnn (train/test), which has access to context during both training and testing. for the latter method, we considered up to 100 sampled sentences at evaluation time. additionally, we compared our approach against a classifier built on top of bert @xcite , a transformer based @xcite language model. we used the base variant, which has 12 layers and produces an embedding of size 768. to stay close to the previous work on learning representations for mwes, which focuses on the signal encoded in the mwe representations rather than fine tuning on the indirect evaluation task, we froze the underlying transformer model and trained a linear classifier head on top of it. to mitigate the sparsity of context at evaluation time, for context-aware rnn (train/test) we used context during testing only if the number of sentences available in a given context is larger than a threshold t. that is, for mwes with more than t sentences in their context we compute their embedding as explained in section 3.4 and illustrated in figure 2 , namely by averaging the candidate embeddings; for mwes with fewer than t sentences in their context we completely disregard the context, computing the embedding only by running the model over the mwe constituents. 9 we motivate the need for this threshold in the next section. we draw the following observations from the results in ## conclusion we proposed an unsupervised method to learn representations of mwes, which is capable of leveraging vast amounts of unlabeled data. specifically, we indirectly learn a compositional function generated by a bidirectional rnn by training it using the skip-gram training objective. our proposed approach achieves better performance on the coarse-grained lexical task of the tratz dataset @xcite than the previous approaches, and it is only 3% behind a classifier that relies on bert, despite its simplicity, and minimal hyperparameter tuning. code is available at https://github.com/clulab/releases/tree/ master/coling2020-mwe .