M ulti V itamin B ooster at PARSEME Shared Task 2020: Combining Window- and Dependency-Based Features with Multilingual Contextualised Word Embeddings for VMWE Detection (2020)

Abstract: In this paper, we present MultiVitaminBooster, a system implemented for the PARSEME shared task on semi-supervised identification of verbal multiword expressions - edition 1.2. For our approach, we interpret detecting verbal multiword expressions as a token classification task aiming to decide whether a token is part of a verbal multiword expression or not. For this purpose, we train gradient boosting-based models. We encode tokens as feature vectors combining multilingual contextualized word embeddings provided by the XLM-RoBERTa language model with a more traditional linguistic feature set relying on context windows and dependency relations. Our system was ranked 7th in the official open track ranking of the shared task evaluations with an encoding-related bug distorting the results. For this reason we carry out further unofficial evaluations. Unofficial versions of our systems would have achieved higher ranks.

## introduction multiword expressions (mwes) are an object of research in various areas of linguistics and nlp. on the one hand, areas such as lexico-semantics and construction grammar have a distinct research interest in the form and semantics of different classes of multiword expressions @xcite ; their automatic detection makes them accessible to large-scale corpus analyses. on the other hand, various nlp-systems, especially in the area of machine translation where the detection of mwes prevents spurious literal translations, can benefit from detecting mwes @xcite , as well. in this paper, we present an approach to the automatic detection of verbal multiword expressions (vmwe), mwes which form around a head verb, which was our contribution to the parseme shared task 2020 on verbal multiword expressions @xcite . the data set provided for the shared task @xcite past approaches in the area of mwe detection rely on the usage of statistical association measures @xcite @xcite and machine learning @xcite @xcite , sometimes combining this work is licensed under a creative commons attribution 4.0 international licence. licence details: http: //creativecommons.org/licenses/by/4.0/ . transformer-based language models such as bert @xcite or roberta @xcite ) are pre-trained on large-scale corpora and are able to achieve state-of-the-art results for various standard tasks in nlp. they can either be fine-tuned to solve a specific task or used to provide contextualised word embeddings. the difference between such contextualised embeddings and the static ones based on traditional methods such as glove @xcite or word2vec @xcite is that the former can account for different local word contexts and encode a given word individually with regard to the observed local context as well as global distributional information. by intuition, this aspect should make contextualised word embeddings a feasible candidate for the detection of mwes, as they should encode a word differently when observed as part of an mwe compared to an occurrence in open distribution, given both cases were reflected during pre-training. as this aspect can, however, not be guaranteed in all cases, we complement the embeddings with regular window-and dependency-based features to make the results of our systems less dependent on the pre-training of the language model used, and, thus, more robust. to be able to account for the multiple languages represented in the shared task without providing a distinct transformer language model for each language, we use the multilingual xlm-roberta @xcite which was trained on common crawl data in 100 languages (including all languages represented in the shared task) and is, as a consequence, able to generate contextualised word embeddings for all of them. ## results and analysis as already stated, our official system (mvb) was ranked last in the official shared task evaluations with an output encoding-related bug distorting results (under a common condition, it was likely that multiple sentences were assigned the same vmwe tags). for this reason, we evaluated a bug-fixed system (mvb (bug free)) for seven out of the 14 languages represented in the shared task (de, eu, ga, hi, it, sv and tr; due to time-related reasons, we only managed to evaluate our system for these languages for the official shared task which is why we decided to focus on them throughout all other evaluations). in addition to our bug-fixed system, we trained a system exclusively on the contextualised embeddings and attention values (emb. att. b.), another system exclusively on the window-and dependency-based linguistic features (ling. feats. b.) and a third system which relies on the same feature set as mvb but uses logistic regression as classification algorithm (we rely on the implementation provided by scikitlearn @xcite for this and train for 1000 iterations) as additional baselines. 2 the official evaluations of the shared task were separated into two tracks. systems participating in the closed track were obliged to only rely on the data sets provided directly by the organisers @xcite , while systems participating in the open track were allowed to use external resources such as external corpora or lexical resources, as well. all our systems except for the one relying solely on window-and dependency-based linguistic features would have participated in the open track due to the usage of the external contextualised embeddings if submitted for the official shared task. the systems were evaluated with regard to different categories with the three most important ones being unseen mwebased, a category that evaluates the performance of systems in respect to vmwes not observed within the training data, global mwe-based, a category that evaluates the general detection of vmwes as connected units, and global token-based, a category evaluating the detection of vmwes on a token level. the bug-free version of multivitaminbooster would have been ranked fifth within the global mwebased and global token-based evaluation categories and sixth within the unseen mwe-based category if it had participated in the official shared task. while this is a huge improvement over the bugged version, these results can be considered subpar, especially in comparison to the winning systems mtlb-struct and seen2seen. two further observations which speak against our form of usage of multilingual contextualised word embeddings for the given task can be made here, as well: on the one hand, the system which was trained solely on them (emb. att. b.) performed by far worst out of all our unofficial systems, and, on the other hand, the system which was trained solely on the window-and dependency-based linguistic feature set (ling. feats. b.) performed best out of all our systems and even manages to put multivitaminbooster into place. if submitted to the shared track, it would have been ranked second in the unseeen mwe-based category and third in the global mwe-based and global token-based categories for the closed track. here, the question whether these results would have turned out more successful if another language model had been used instead of xlm-roberta @xcite or if our results reflect a general inadequacy of the approach to use transformer-based word embeddings for detecting vmwes remains. our model based on logistic regression (mvlr) achieved lower precision but higher recall scores than all our systems based on gradient boosting and, by average, lower f1-scores than the best gradient boosting-based models. this raises the question to what extent the results would differ when applying other classification algorithms. one other important finding is that for all our systems, there is a discrepancy between the precision scores observed for global mwe-based and global token-based. while the precision achieved for the global mwe-based category turned out subpar, multivitaminbooster and the system relying solely on the linguistic feature set achieve the best precision scores out of all our unoffical and all official systems within the global token-based category. we attribute this discrepancy to our heuristics used for reassembling vmwes. a closer look onto an example can explain this: "'es tut weh, die sprache derer ben ützen zu müssen, die dich schinden', heißt es da beispielsweise schon am anfang [...]." ("'it hurts having to use the language of those who maltreat you', it says, for example, at the beginning.") in this german sentence, all tokens marked as bold were recognised as one large vmwe instead of multiple ones, illustrating a problematic pattern which is observable for our systems throughout all languages evaluated. tokens of multiple vmwes of the same category can form connected components within the dependency tree of a given sentence which our heutristics is not able to resolve in a correct way. a solution to this would be to further inspect the dependency relations for which this phenomenon is observable and to try to identify criteria to filter them out under given circumstances. in the closed track. mtlb-struct and ermi are the winning systems of the two tracks of the shared task for this language. we provide their results for reasons of comparability. a further observation is that in the case of the language hindi, the results achieved by our systems show positive outliers. here, multivitaminbooster would have ranked third in the global mwe-based category and second in the global token-based category. the unofficial system trained solely on windowand dependey-based features even manages to achieve unofficial first ranks in the closed track. ## conclusion and future work we presented multivitaminbooster and three unofficial systems implemented for the parseme shared task 2020 on verbal multiword expressions. we evaluated our systems for seven languages. the best of our systems would have ranked fifth in the official shared task. a positive outlier can be observed for the language hindi, where our systems achieved more competitive results. the usage of multilingual contextualized word embeddings for our systems can be considered a failure, as the same deteriorated results and our system relying solely on the linguistic feature set achieved superior results. it is, however, to explore if this would have turned out differently with another language model. to summarise, there remains room for improvement. using statistical association measures induced from large scale corpora as additional features may be a route to further explore this. an improved redesign of the heuristics used for assembling tokens into vmwes built on a more complex rule set could lead to improvements in mwe-based precision scores and close the gap to the token-based ones. different classification algorithms, such as crfs or svms, could be explored as alternatives to gradient boosting and logistic regression, as well as different variations and combinations of training hyperparameters to aim for better regularisation.