HMS id and HMS id2 at PARSEME Shared Task 2020: Computational Corpus Linguistics and unseen-in-training MWE s (2020)

Abstract: This paper is a system description of HMSid, officially sent to the PARSEME Shared Task 2020 for one language (French), in the open track. It also describes HMSid2, sent to the organ-izers of the workshop after the deadline and using the same methodology but in the closed track. Both systems do not rely on machine learning, but on computational corpus linguistics. Their score for unseen MWEs is very promising, especially in the case of HMSid2, which would have received the best score for unseen MWEs in the French closed track.

## introduction although the parseme shared task 2018 @xcite produced very interesting results for the extraction of verbal multiword expressions, one important note of caution has to be made: the participating systems produced poor results for unseen mwes, i.e. expressions that were absent from the training data. as pointed out by the organizers of the new parseme shared task 2020 foot_0 , a possible solution to this issue is the recourse to large mwe lexicons. in this paper, however, we report the results of two systems offering promising results for unseen mwes with no recourse to mwe lexicons: hmsid (hybrid multi-layer system for the extraction of idioms) and hmsid2. both systems are based on computational corpus linguistics: they just used the training data and an additional general linguistic corpus. as the models require a fine-tuned adaptation to each language under study, they were only applied to the french dataset of the parseme shared task 2020. hmsid used as an external corpus the french wacky corpus @xcite and was submitted to the parseme shared task 2020. as there was a recourse to an external corpus, it was logically put in the open track. thanks to the feedback from the organizers of parseme 2020, however, we adapted the system in order to propose it in the closed track: the corpus used was the wikipedia corpus included in the training data. the new version, hmsid2, was sent to the organizers after the official deadline. in this paper, both the official results of hmsid and the new results from hmsid2 are discussed. our theoretical starting point for both systems is that, while deep learning will surpass most techniques for reproducing elements that are somehow present in training sets, it will need additional corpus-based information for unseen-in-training mwes. it should also be pointed out that mwe extraction is a daunting practical task, but that the theoretical background is also very complex, as it is related to grammatical and semantic structure. information retrieval @xcite has shown that semantic relations may be analyzed by very diverse methods, including vector space models and clustering methods. many of its findings are compatible with the distributional hypothesis @xcite ): differences in meaning will be reflected by differences in distribution. however, the distribution of words is also affected by existing mwes, as at least 50 percent of the words from any text will actually be included in mwes, collocations or phraseological units @xcite . in addition, a wide array of studies in construction grammar @xcite strongly suggest that language structure consists of a very complex and probabilistic network of constructions at various levels of abstraction and schematicity. it is no wonder then that very complex techniques are necessary for extracting mwes, in much the same way as for the extraction of semantic links. in particular, the complex interplay between 1 st -order co-occurrence (words appear together) and 2 nd -order co-occurrence (words appear in similar contexts, lapesa and evert, 2014) probably requires a hybrid methodology. while deep learning and in particular neural networks are very efficient ways of gaining information from a training set, it may be complemented by a more traditional, corpus-based approach in the case of the extraction of data that are unseen in the training set. the technical background for hmsid and hmsid2 is a combination of techniques inherited from information retrieval, such as metric clusters @xcite and a query likelihood model, with a big data approach, in this case a large (unparsed and untagged) linguistic corpus: the french wacky for hmsid and the parseme french training corpus (wikipedia) for hmsid2. as described in @xcite , a clustering algorithm based on the average distance between the component parts of the mwes is measured, the cpr-score (corpus proximity ratio): this approach, as opposed to vector models, is a 1 st -order model, as it is based on the co-occurrence of words and not on similar contexts. given a window w of x tokens (depending on the language and the corpus, typically set at 20 for mwes), the score simply measures the ratio between the number of exact occurrences of an n-gram, divided by the number of occurrences with a window between each gram. the main advantages of this metric are that it is not limited to bigrams, and that semantic links may be captured as well by enlarging the window, a point that has also been made by @xcite : larger windows may enable 1 st -order models to capture semantic associations. experiments with large datasets of idiomatic mwes have shown @xcite that most formulaic and idiomatic constructions can be captured by co-occurrence clusters, provided that the corpus used is sufficiently large (at least 1 billion tokens). in order to reach a good compromise between results that could be extracted from co-occurrence in large corpora and recurrent patterns with specific categories of mwes, a hybrid methodology was used, as detailed in the following section. ## methodology used for hmsid and hmsid2 in the parseme shared task 2020 for french, the following categories of verbal mwes had to be extracted from the test set: irv (inherently reflexive verbs, as in the english example to help oneself), lvc.cause (light-verb constructions in which the verb adds a causative meaning to the noun, as in the english to grant rights), lvc.full (light-verb constructions in which the verb only adds meaning expressed as morphological features, as in to give a lecture), mvc (multi-verb constructions, as in to make do) and vid (verbal idioms, e.g. to spill the beans). after a number of preliminary tests, we decided to extract french mwes from the test set in a twostep process. the first step concerned all categories of verbal mwes, as described above, except the last one (vid, verbal idioms). the second step was just devoted to verbal idioms. this two-step approach was motivated by the unpredictable character of verbal idioms: contrary to the other categories of mwes used for the parseme shared task, idioms display a very irregular number of elements, of which the syntactic structure is also diverse. during the first step, we used a perl script and the data::table module 2 for storing each sentence at a time in ram memory. for the categories irv, lvc.cause, lvc.full and mvc, the specific syntactic features of these categories were taken into account by the algorithm: in the case of irv, for instance, the parsed sentences provided by the parseme dataset made it easy to extract all pronouns preceding or following the verbs, and an additional check was performed in order to determine whether those pronouns were indeed french reflexive pronouns, including elision (e.g. the pronominal form s' instead of se). for lvc.cause, a list of french causative verbs was extracted from the training data (for instance apporter, causer, créer, entraîner). in the extraction phase, all objects depending on such causative verbs were measured by our co-occurrence score, the cpr-score @xcite and the highest values were considered as cases of lvc.cause constructions. for lvc.full, a similar methodology was used, taking into account all subjects (for passive constructions) and objects (for direct object constructions) depending on verbs, excluding causative verbs, with a medium-range association between the subject/object and the verb (computed by the cpr-score). in the same way, the mvc category was extracted on the basis of the degree of association between two successive verbs, as in faire remarquer (to point out). in the second step of our extraction methodology, verbal idioms were extracted and added to the results. this made it possible to add the category of verbal idioms in the labels of the final results if and only if the results had not yet received another category label, for instance lvc.full. preliminary tests on the basis of the training data indeed revealed that our algorithm tended to assign the vid category quite often, whereas the annotators of the gold set had been rather strict as to the idiomatic character of verbal mwes. using two separate scripts was a simple way of avoiding interference in the results. in the perl script devoted to the extraction of vids, we also used the data::table module and selected in the parsed data all verbs, all their complements, and all complements of each complement. extensive testing with the training data showed that this approach yielded higher scores than an n-gram based approach, in which the successive grams of each verb were analyzed left and right. ## results and discussion as shown in this is even more obvious with hmsid2, which used exactly the same methodology, as explained in the above section, but relied on the training corpus provided by the shared task (part of the wikipedia corpus), and would therefore be placed in the closed track. among the 3 systems submitted to the closed track for french, hmsid2 would receive rank 2 for the global f1-score (mwe-based or token-based), and rank 1 for unseen mwes, with an f1-score (39.21) far better than those obtained by the other systems in the closed track (with respectively 24.4 and 3.67). the best overall system officially submitted to the french closed track (seen2seen) has an f1-score of 3.67 for unseen mwes. the difference between precision and recall, especially for unseen mwes, should also be relativized by the choices made in the training and gold set. in spite of the excellent quality of the parseme annotated dataset, decisions as to the idiomatic character of a mwe will never be unanimous. in the case of the french dataset, for instance, the notion of verbal idiom (vid) was taken strictly by the annotators, but there are a few notable exceptions. a number of less idiomatic constructions were also labeled as vids. for instance, avoir lieu (to take place), il y a (there is / there are), mettre en pratique (to put into practice), tenir compte de (take into account), are all considered french verbal idioms in the training data, a choice that may be respected but has consequences on the statistical extraction. the statistical metric indeed had to be more tolerant for weaker associations when assigning the label 'vid', which contributed to a fairly good recall but a slightly lower precision. this appears clearly in all results from tables 1 and 2 , and in particular for unseen mwes. in this case, one should bear in mind that the algorithm is looking for recurrent patterns in the linguistic system itself, as there are no similar examples in the training set. many cases of verbal idioms from the gold set are quite obvious, such as tourner le dos à (turn one's back on, lines 5817-19 of the gold set), il pleuvait des cordes (it was raining cats and dogs, lines 7415-17) or sortir le grand jeu (pull out all the stop, lines 12129-32), all three labelled as vid and also recognized by the algorithm because of the very strong association between the grams: a cpr-score of resp. 0.92 / 0.88 / 0.94. in other cases, however, the algorithm and the annotators are at odds. in lines 5868-9, for instance, rester silencieux (remain silent, keep quiet) is not considered as mwe by the annotators, but the cpr-score contradicts this view: 0.81. the same holds true of many other examples, such as trouver un compromis (lines 14387-89), not considered as a mwe in the gold set, but displaying a cprscore of 0.80. in this specific case, it should be reminded that native speakers are not always the best judges of the idiomaticity of their own language. it may be pretty obvious for speakers or french and of english that a compromise may be found but a quick look at other european languages reveals that this is far from being the case: in spanish, for instance, the common construction is llegar a un compromiso. it should also be pointed out that the methodology used for hmsid and hmsid2 is easily applicable to other languages. as a matter of fact, we have already implemented it as an experimental web tool 3 , idiomsearch for english, german, spanish, french, dutch and chinese. measuring associations based on the cpr-score is indeed possible for any language, provided that the necessary web corpus is compiled. the only caveat is the goal of the classification. the parseme shared task 2020, as the previous editions, wanted the systems to target very specific categories of verbal expressions, whereas our experimental tool idiomsearch looks for recurrent statistical associations, whatever the precise category may be. finetuning the algorithm to specific categories expected by the gold set, and annotated as such by native speakers of the language requires sophisticated training algorithms such as those used in deep learning. in conclusion, the most interesting results from hmsid and hmsid2 are those obtained for unseen mwes. due to the well-known phenomenon of overfitting, deep learning models often have problems with unseen data, which suggests that a hybrid approach combining deep learning and our model may be useful for future research.