Deep Neural Representations for Multiword Expressions Detection (2022)

Abstract: Effective methods for multiword expressions detection are important for many technologies related to Natural Language Processing. Most contemporary methods are based on the sequence labeling scheme applied to an annotated corpus, while traditional methods use statistical measures. In our approach, we want to integrate the concepts of those two approaches. We present a novel weakly supervised multiword expressions extraction method which focuses on their behaviour in various contexts. Our method uses a lexicon of English multiword lexical units acquired from The Oxford Dictionary of English as a reference knowledge base and leverages neural language modelling with deep learning architectures. In our approach, we do not need a corpus annotated specifically for the task. The only required components are: a lexicon of multiword units, a large corpus, and a general contextual embeddings model. We propose a method for building a silver dataset by spotting multiword expression occurrences and acquiring statistical collocations as negative samples. Sample representation has been inspired by representations used in Natural Language Inference and relation recognition. Very good results (F1=0.8) were obtained with CNN network applied to individual occurrences followed by weighted voting used to combine results from the whole corpus. The proposed method can be quite easily applied to other languages.

## introduction multiword expressions (henceforth mwes) have been studied for decades, defined in different ways in literature with different denotations of this term, e.g. see the overview in @xcite . probably, the most genuine, but the least operational, definition is multiword lexemes stored as single lexical units in the mental lexicon ready to be retrieved. in the spirit of this fundamental property, we consider mwes from the lexicographic point of view as lexical units that "has to be listed in a lexicon" @xcite and we seek for methods of automated extraction of mwes from text corpora to expand a large semantic lexicon with multi-word lexical units. summarising a longer definition given in @xcite , mwes are "lexical items decomposable into multiple lexemes", "present idiomatic behaviour at some level of linguistic analysis" and "must be treated as a unit" and, thus, should be described in a semantic lexicon, e.g. from @xcite air corridor (an agreement between two countries), slow food ("traditional food and ways of producing, cooking and eating it"), fast food, fire door, first lady etc. a similar definition was adopted in the parseme shared task resource @xcite ). as we target the construction of a general lexicon expressing good coverage for lexical units occurring frequently enough in a very large corpus, we need also to take into account multiword terms, i.e. @xcite "specialised lexical units composed of two or more lexemes, and whose properties cannot be directly inferred by a non-expert from its parts because they depend on the specialised domain". several mwe characteristics or identifying properties have been postulated, e.g.: arbitrariness, institutionalisation, limited semantic variability (especially non-compositionality and nonsubstitutability), domain specificity, and limited syntactic variability @xcite . among them, semantic non-compositionality seems to be one of the strongest identifying factors. however, the challenge is to trace them using some corpusbased evidence and guide the extraction process. in addition, mwes should be some how correlated with higher or more prominent frequency in language use in order to be worth inclusion in a lexicon. extraction of mwes and their description in a semantic lexicon (e.g. as a reference resource) is important for many nlp applications like semantic indexing, knowledge graph extraction, vector models, topic modelling etc. due to the specific properties of mwes as whole units, their automated description by the distributional semantics method, e.g. embeddings, is not guaranteed, especially in the case of mwes of lower frequency. traditionally, mwes extraction is preceded by finding collocations (frequent word combinations) by statistical or heuristic association measures and filtering them by syntactic patterns. however, in this way mainly the frequency-related aspect is covered. the peculiar behaviour of mwes as a language unit may be observed in linguistic contexts, and methods based on the well-known sequence labelling scheme try to do that. they explore mwe specific behaviour of as a language expressions across text contexts, where the contexts are represented by contextual embeddings (neural language models). however, such approaches require a lot of hard manual work on text annotation. in addition, due to the corpus size limitation, most potential mwes are observed only in a few, if not singular uses, while a lexicon element by a definition is a ready-to-use unit to be included in different contexts and, as such, should be studied. thus, we want to fully explore the expected mwe characteristic aspects, including frequency, and to reduce the amount of manual work required. mwe annotated corpora are very rare and small, e.g. parseme @xcite , but mwes are listed in dictionaries and lexical resources. we propose a weakly supervised approach in which a lexicon of mwes is used to build a kind of silver data on the basis of general text corpus. concerning negative examples, i.e. language expressions rejected to be mwes, that are hardly listed in any lexical resources, we use association measures (frequency aspect) to find collocations very likely not being mwes. next we feed a system combining contextual embeddings, deep neural learning and weighted voting scheme across individual mwe occurrences with the silver data. as a result, the system can be next used to filter potential mwes extracted from a corpus with association measures (the frequency aspect in a positive role). in contrast to many methods from literature, we neither need a corpus laboriously annotated with mwe occurrences, nor language models specially trained for this task. in addition we aim at jointly encompass most of the mwe characteristic aspects with the majority of them recognised in a kind of over-lap of mwe contextual embeddings across their different occurrences. the proposed approach is illustrated with good results achieved on english mwes coming from several dictionaries and the british national corpus. however, our method can be quite easily adapted to any language, the only required elements are: a corpus and an initial lexicon of mwes, and a general contextual embeddings model. ## related work initially statistical association measures calculated on the basis of word co-occurrence statistics in corpora were used for discovering and ranking collocations as potential mwes @xcite . single measures can be also combined into complex ones, e.g. by a neural network @xcite . syntactic information from parsing @xcite or from lexico-syntactic constraints based on morphosyntactic tagging @xcite were used in counting statistics and post-filtering collocations. several systems for mwe extraction were proposed, combining different techniques, e.g. mwetoolkit by ramisch @xcite combines statistical extraction and morpho-syntactic filtering, but also describes collocations with feature vectors to train machine learning (ml) classifiers. lexicosyntactic patterns, measures, length and frequency are used as features in ml-based mwe extraction @xcite . linguistic patterns were used to extract mwes and post-filter the outcome of association measures @xcite . mwes were also detected by tree substitution grammars @xcite or finite state transducers @xcite . recently, attention was shifted to mwe extraction perceived as a sequence labelling problem, e.g. @xcite , where corpora are annotated on the level of words, typically, bio annotation format @xcite : ba word begins an mwe, i is inside, o -outside. sequence labelling approaches can also be combined with heuristic rules @xcite or supersenses of nouns or verbs @xcite . such heuristics are applied to extract linguistic features from texts for training a bayesian network model @xcite . convolutional graph networks and self-attention mechanisms can be used to extract additional features @xcite . there are many challenges related to the nature of the mwes, e.g.: disconti-nuity -another token occurs between the mwe components or overlapping -another mwe occurs between the components of the given mwes. to counteract this, a model based on lstm, the long short-term memory networks and crf is proposed @xcite . the model from @xcite combines two learning tasks: mwe recognition and dependency parsing in parallel. the approach in (kurfalı, 2020) leverages feature-independent models with standard bert embeddings. mbert was also tested, but with lower results. an lstm-crf architecture combined with a rich set of features: word embedding, its pos tag, dependency relation, and its head word is proposed in @xcite . mwes can be also represented as subgraphs enriched with morphological features @xcite . graphs can be next combined with the word2vec @xcite embeddings to represent word relations in the vector space and then used to predict mwes on the basis of linguistic functions @xcite . morphological and syntactic information can be also delivered to a recurrent neural network @xcite . two approaches to mwe recognition within a transition system were compared in @xcite : one based on a multilayer perceptron and the second on a linear svm. both utilise only lemmas and morphosyntactic annotations from the corpus and were trained and tested on parseme shared task 1.1 data @xcite . however, such sequence labeling approaches focus on word positions and orders in sentences, and seem to pay less attention to the semantic incompatibility of mwes or semantic relations between their components. furthermore, sequence labeling methods do not emphasize the semantic diversity of mwe occurrence contexts. thus, they overlook one of the most characteristic mwe factors: components of a potential mwe co-occur together regardless of the context. it allows us to distinguish a lexicalised mwe from a mere collocation or even a term strictly related to one domain. to the best of our knowledge, the concept of using deep neural contextual embeddings to describe the semantics of the mwes components and the semantic relations between them in a detection task has not been sufficiently studied, yet. moreover, due to the sparsity of the mwes occurrences in the corpus, the corpus annotation process is very time consuming and can lead to many errors and low inter-annotator agree-ment. for this reason, we propose a lexicon-based corpus annotation method. we assume that the vast majority of mwes are monosemous, automatically extract the sentences containing the mwe occurrences, and treat all sentences including a given mwe (as a word sequence) as representing the same multiword lexical unit. ## datasets the conducted analysis of the existing resources has shown that it is difficult to find a large annotated dataset for the multiword expressions detection task. parseme shared task and multilingual corpus @xcite is a very valuable initiative, but focused mainly on verbal mwes and quite small, especially its english part. moreover, dictionaries containing mwes follow different definitions and lexicographic practices, which makes it difficult to unambiguously determine whether a given multiword entity is a valid mwe. therefore, in order to obtain a large dataset, we followed our idea of silver dataset and selected the oxford dictionary of english (ode) @xcite as a reference point to obtain the list of correct mwes. the proposed method is in some way parameterised by a selected reference dictionary. concerning language expressions that are not mwes, i.e. negative samples from the ml perspective, they are not listed or mentioned in the dictionaries. having a corpus annotated with mwe occurrences we could extract expressions that are not as negative samples. however genuine mwes are more frequent or statistically specific. thus, 'normal' language expressions would be too obviously different. instead, we noticed that statistical association measures produce very long ranking lists of collocations. further down the ranking, mwe occurrences are quickly dwindling away. in addition, we are interested only in specific structural types of collocations that match structural types of mwes acquired from a dictionary. to generate the list of incorrect mwes, we selected three popular association measures foot_0 : (1) the pointwise mutual information (pmi) @xcite , (2) the sørensen-dice coefficient (dice) @xcite , and (3) pearson's chi-square (chi2) @xcite and used them to extract collocation ranking list from the british national corpus (bnc) @xcite . in order to find relevant examples of multiword units, we decided to select those collocations that were in the third quartile of the list sorted in descending order based on the value of the selected measure. we quickly skimmed the list in order to ensure that it is hard to spot anything looking as a mwe (but we do not exclude the possibility that some mwes may occur, perfect precision does not seem to be necessary). we combined the list of the correct mwes (from the dictionary) separately with the lists of collocations obtained via each of the three selected measures. in all experiments we concentrated on two word mwes and collocations, as the statistical association measures we applied are naturally defined for two word combinations. however, as it will be presented later, some of the mwe representation we propose can be easily expanded to k-word cases. moreover, two word mwes form the vast majority of all in the dictionaries. collocations extracted from the corpus were restricted only to those that represent structural types of mwes from the dictionary. we then used the three resulting lists to search for sentences including collocation or mwe occurrences in the bnc corpus. the searched expressions were simply recognised by comparing lemma sequences. some recognition error may appear, but the potential error ration seems to be very small (single percents). if multiple mwe/collocation lemma sequences were detected among the sentence lemmas, then their occurrences were considered as separate training samples (positive or negative), see alg. 1. in order to evaluate our method of detecting sentences containing mwes, we extracted 4 randomly selected samples containing 100 found sentences each. a linguist conducted the analysis and found that 99% of the sentences contained correct mwe occurrences. the analysis was performed only on sentences corresponding to positive samples -mwes from the dictionary, but similar results can be expected for collocations from the lists. our work resulted in the creation of three datasets of mwe and collocation occurrences, named on the basis of the sources of knowledge: algorithm 1 procedure of obtaining sentences (s) containing mwes from the corpus (c) by comparing sentence word lemmas (l @xmath0, l 1 , . . . , l n ]) to the list (m ) of lemmatised mwes (m @xmath1, m 1 , . . . , m k ]) the obtaining of contextual mwe embeddings is described in eq. 1. an mwe embedding ( ---→ m sent ) in the sentence context (sent) is an average of the wordpiece subtoken (@xmath2. in the next step, the mwe occurrence was replaced subsequently with each of its components in order to obtain their contextual embeddings ( --→ c sent ) by averaging the corresponding subtoken vectors representations ( -→ ν s ) related to the substituted components (s csent ), see eq. 2. the final baseline embedding ( -→ b ) of a training sample related to a sentence (sent) containing mwe (m) and one of its components (c) is described in eq. 3. ## experimental setup for all conducted experiments we selected a singletask binary classification, where the classifier aims to predict the correct label out of 2 possible ones (lexicalised vs non-lexicalised) for the expression represented by one of the vector representations: baseline, diff-emb, prod-emb or mean-emb. in the process of generating the contextual embeddings we used the xlm-roberta @xcite language model as it is considered as one of the best transformer models for english. we decided to use the convolutional neural network (cnn) architecture as the classifier to better extract the knowledge from our vector representations. we used the tensorflow library @xcite to implement the cnn model. our convolutional neural network contains three convolutional layers, each followed by the pooling layer and the dropout layer and is shown in fig. 1 . we used the f1-macro metric to measure the performance of the classifier on each of the representations. to prevent data leakage, we applied the lexical split to avoid the risk of testing on the same multiword unit, which was used in the training procedure (even if the sentence samples are obviously not overlapping). we leveraged the 10-fold cross-validation and used statistical tests to measure the significance of the difference between different experiment configurations. we checked the assumptions and then applied the independent samples t-test with the bonferroni correction if they were met. otherwise we used the mann-whitney u-test. ## results tab. 1 shows the evaluation results for each representation on the ode-pmi dataset. each value is averaged over ten folds. the mean-emb representation combining both the knowledge based on the difference vector and the hadamard product vector achieved the best results. the performance of the cnn model trained on all representations and evaluated on the ode-dice dataset is shown in tab. 2. the best performance can be observed for the mean-emb model. each of the developed representations achieved better results than the baseline vector representation. the evaluation results for the classifier trained on each representation and evaluated on the ode-chi2 dataset are shown in tab. 3. the mean-emb model achieved the best results among other representations. the worst performance can be observed ## discussion the idea of silver dataset enables transformation of any corpus into a dataset for mwe extraction, only if a limited lexicon of mwe examples is provided as a starting point -a kind of seed lexicon to be expanded. we can leverage a mwe annotated corpus, too, in the same way as a lexicon to extract the initial list of mwes, but a large non-annotated corpus stays the basis. several linguistic resources can be also merged, any mwe annotated text, as well as lexicons. time-consuming and expensive corpus annotation is avoided. moreover, it seems to be easier to maintain high quality lexicon than corpus annotation, e.g. due to potential errors and discrepancies between single annotations. a lexicon can be edited by several linguists, and metrics such as inter-annotator agreement can be easily calculated. what is more, such a transformation of lexiconbased knowledge into a dataset enables the use of deep neural network models that require large number of training samples. this is one of the reasons why our cnn method, pre-trained on contextual embeddings with weighted voting, applied to mwe recognition achieved several times better results than methods based on contextual embeddings and recurrent neural in the parseme shared task in general @xcite , not mentioning the english part alone that is very small. our approach may be applied to texts in different languages, both to obtain multilingual collections and to apply transfer learning to facilitate the knowledge about mwes in one language to mwe recognition in another language. this may be particularly relevant for low-resource languages, and it definitely a direction for further research. another advantage of the proposed method is faster training and prediction in comparison to sequence labeling methods. in our case, the model gets the full sample representation only once before prediction. this shortens the inference time. our vector representations support mwes longer than two words. in the case of multiword units containing three and more words, the difference and product vectors calculated between two mwe components can be replaced with the vector obtained via the same operation, but averaged over all mwe component pairs. the obtained results show that non-lexicalised representations, i.e. those that do not include vectors for components and the whole expression 2 perform better independently of the kind of a measure used to extract collocations. all representations except the baseline are built from differences and 2 a contextual vector of the whole expression somehow includes a picture of the particular expression and its lexemes. products of vectors, not the vectors itself. thus they are more focused on representing relations between a potential mwe and its components. it is worth to be emphasised that lexical split was also implemented in order to prevent the models to remember concrete words instead of learning patterns for behaviour of proper mwes. there are no large differences between results for different measure, but, with some caution, we can observe that results obtained with pmi are slightly better, while in the case of pmi the measure is naturally is filtered by 0 threshold and produces potentially more interesting collocations, thus harder to be distinguished from the proper mwes. ## conclusions and future work our three representations allowed classifier to achieve significantly better results in comparison to the baseline approach focused on the component and mwe embedding. the context provided additional information on the mwe semantics, which improved the model performance. this is related to the noncompositional nature of the mwes, which meaning cannot be inferred from their component meanings. our approach based on difference and product vectors forced the models significantly reduced the training time. it may be more important in practice, when the training time and inference time are more important than the quality of prediction. on the other hand, the method based on contextual embeddings allows transforming any set of texts with the use of dictionary knowledge into an annotated corpus containing occurrences of the mwes and their components. the model, by examining the semantic differences between the component and the entire expression, takes into account the variability of the context, which should allow for the extraction of the mwe meaning following the assumption of its monosemous character. in future work, we want to use our methods to generate corpora in other languages, which will be later used to train models in the multilingual mwes detection task and to explore the transfer learning mechanism in a language-independent mwe detection.