Extracting meaning by idiomaticity: Description of the HS em ID system at C og AL ex VI (2020) (2020)

Abstract: The HSemID system, submitted to the CogALex VI Shared Task is a hybrid system relying mainly on metric clusters measured in large web corpora, complemented by a vector space model using cosine similarity to detect semantic associations. Although the system reached ra-ther weak results for the subcategories of synonyms, antonyms and hypernyms, with some dif-ferences from one language to another, it is able to measure general semantic associations (as being random or not-random) with an F1 score close to 0.80. The results strongly suggest that idiomatic constructions play a fundamental role in semantic associations. Further experiments are necessary in order to fine-tune the model to the subcategories of synonyms, antonyms, hy-pernyms and to explain surprising differences across languages. 1 Introduction

## introduction this paper is a system description of hsemid (hybrid semantic extraction based on idiomatic associations), presented at cogalex vi. contrary to most models dedicated to the extraction of semantic associations, hsemid is based on a similar model developed for the extraction of multiword expressions, hmsid, presented at the parseme 1.2. workshop of the coling 2020 conference. from a theoretical point of view, we wished to explore the link between general meaning associations and associations based on idiomaticity, in the general sense of multiword expressions (mwes). for instance, beans may display a general meaning association with food (as many of them are edible) or with coffee, but there is an idiomatic association between spill and beans because of the idiom spill the beans (reveal a secret). thus, general meaning associations are mainly extralinguistic and cultural, whereas idiomatic associations are mainly intralinguistic, as they are just valid for one specific language, although similar associations may exist in other languages because they are cognate or have influenced each other. the implicit link between semantics and idiomaticity has already been mentioned in the literature. @xcite point out that using larger windows with statistical scores yields extraction models that can be adapted from mwes to semantic associations. according to them, 1st-order models (based on co-occurrence statistics such as the log-likelihood, dice score or t-score) and 2nd-order models (based on similar contexts of use, as in the case of cosine similarity in a vector space model) appear to be redundant on the basis of the first experiments and do not really benefit from a combination of both approaches. our model for the extraction of multiword expressions (hmsid, hybrid multi-layer system for the extraction of idioms) yielded promising results for french verbal expressions. in the official results of the parseme 1.2. shared task, our model obtained an f1-score of 67.1, with an f1-score of 36.49 for mwes that were unseen in the training data; in an adapted version proposed just after the workshop, we reached an even better f1-score of 71.86 in the closed track, relying only on the training data, with no external resources, and an f1-score for unseen mwes of 40.15, which makes it by far the best score in the closed track for unseen french mwes. it should be pointed out that the model used for the extraction of mwes is corpus-based and derives from metric clusters used in information retrieval @xcite @xcite , but does not use any machine learning architecture. we adapted this model in a deep learning approach for the cogalex vi shared task, as described in the following section. from a theoretical point of view, we wished to explore the performance of a model used for mwe extraction, in a related but very different context: the extraction of semantic associations. although we realize that the main goal of the cogalex vi shared task was to improve the extraction of the specific categories of synonyms, antonyms and hypernyms, we did not have enough time to train our model for this subcategory distinction, and were mainly concerned with the identification of a semantic association (being random or non-random) on the basis of idiomatic patterns. ## methodology our model was tested for the different languages taking part in the cogalex vi shared task, using the datasets provided for english @xcite , chinese @xcite , german @xcite and italian @xcite . as suggested by the acronym (hsemid, hybrid semantic extraction based on idiomatic associations), our methodology was hybrid, as we used both a vector space model (turney and pantel, 2010) and a co-occurrence model based on metric clusters in a general corpus @xcite . however, most features of the model were derived from the second part, so that the model mainly relies on cooccurrence and therefore on idiomatic meaning, as explained below. for the vector space model, we measured cosine similarity in the wikipedia corpora. we relied on the wiki word vectors foot_1 and on the perl implementation of word2vec, by means of the multiword cosine similarity function foot_2 . for the metric cluster, we used the cpr-score @xcite , a simple co-occurrence score based on the proximity of the ngrams in a large corpus. in order to avoid redundancy with the wikipedia corpora, this score was computed by using other, general-purpose web corpora: the wacky corpora @xcite for english, german and italian. for chinese, we compiled our own web corpus by means of the webbootcat tool provided by the sketch engine foot_3 . as we have only basic knowledge of chinese, we relied for this purpose on the seed words originally used for compiling the english wacky corpus. the english seed words were translated into chinese by google translate foot_4 . all those corpora have a size of about 1.4 billion tokens; for chinese (mandarin, simplified spelling), we reached a comparable size by taking into account the number of chinese words, not the number of chinese characters (hans). in order to train our model, we implemented a neural network (multi-layer perceptron), relying on most of the default options provided by the microsoft cognitive toolkit (cntk) foot_5 . we imported the cntk library in a python script. our neural network used minibatches, had an input dimension of just 11 features (for the 4 output classes), 2 hidden layers (dimension: 7), and we used relu as an activation function. for the loss, we relied on cross entropy with softmax. among the 11 features used for training the model, it should be noted that the vector space approach, represented by the multiple cosine similarity, only played a limited role, as it represented just one of the 11 features to be weighted by the model. the other features were based on the metric clusters. for these, the association score (cpr-score) was measured with a narrow window between the grams composing the pairs from the datasets, and with wider windows for a number of linguistic markers favoring semantic associations (typically or, and, not, and their equivalents in the different languages). the frequencies of the different grams in the wacky corpora were also used as input features. all features were smoothened to real figures between 0 and 1. for measuring the average test error during training, we used 80 percent of the training data as the trainer, and 20 percent (with the correct labels) as the test data. the average test error when training the model was situated around 20 percent. ## results and discussion for lack of time, we didn't have the opportunity of fine-tuning our model to the specific subcategories syn, hyp and ant, as was the main goal of the cogalex vi shared task. as a matter of fact, our objective was to focus the training of the model on the general semantic associations (random or notrandom), in the hope that this would also yield acceptable subcategories syn, hyp and ant. obviously, this was not really the case, although high scores for european languages are hard to reach (the best f1 scores for english, german and italian at the shared task are resp. 0.517, 0.500 and 0.477). a closer analysis of the errors produced by our model reveals that too many idiomatic associations of synonyms and antonyms are similar. for instance, turn right and turn left are equally strong idiomatic associations, and it is unclear how right and left should be considered as antonyms from this point of view. in the same way, hypernyms are very hard to discriminate from synonyms if we pay attention to their idiomatic associations. a further improvement of our model may therefore consist in a more complex neural network, in which the different contexts for syn/ant and syn/hyp would be specified by additional features. in spite of these shortcomings, our model reached pretty good scores for the general task of extracting semantic links, which does not appear in the official results but may be computed by means of the evaluation score provided in the training data, which contains the random category. if we take into consideration the f1 score obtained for the random label, we obviously get a picture of the general ability of the model to extract strong semantic associations, be they cases of synonymy, hypernymy or anything else (such as metaphors or idiomatic meaning). for lack of space, in spite of the rather weak results obtained by our model for the elicitation of the subcategories syn, hyp and ant at the cogalex vi shared task, we therefore come to the conclusion that the hsemid model, relying mainly on the extraction of semantics by means of idiomatic associations, makes it possible to extract general semantic associations, with f1 figures for the random category that were rarely reached by any experiment carried out within distributional semantics. the results strongly suggest that idiomatic constructions play a key role in semantic associations. further experiments should improve the scores obtained for synonyms, antonyms and hypernyms, which clearly remains a daunting challenge in the case of european languages.