TRAVIS at PARSEME Shared Task 2020: How good is (m) BERT at seeing the unseen? (2020)

Abstract: This paper describes the TRAVIS system built for the PARSEME Shared Task 2020 on semi-supervised identification of verbal multiword expressions. TRAVIS is a fully feature-independent model, relying only on the contextual embeddings. We have participated with two variants of TRAVIS, TRAVIS-multi and TRAVIS-mono, where the former employs multilingual contextual embeddings and the latter uses monolingual ones. Our systems are ranked second and third among seven submissions in the open track, respectively. Despite the strong performance of multilingual contextual embeddings across all languages, the results show that language-specific contextual embeddings have better generalization capabilities.

## introduction multiword expressions (mwes) are, most commonly, defined as a group of words which act as a single lexical unit and display idiomaticity at lexical, syntactic, semantic or pragmatic levels @xcite . as the name suggests, verbal mwes (vmwes) are mwes with a verb as the head in their canonical form. identification of vmwes tend to be more challenging than that of other mwes, as they exhibit more syntactic/morphological variation (due to inflection of the verb), their components can be interrupted by other words (he made a serious mistake) and furthermore their order may vary (the decision was hard to take) @xcite . yet, their identification is equally important as it is a prerequisite to fully address a number of downstream tasks, such as machine translation, information retrieval or syntactic parsing. this year's shared task is built upon the observation that the existing models fail when it comes to identify the vmwes which are not seen during the training. hence, the aim of this year's shared task is updated to identify the unseen vmwes in running text and the organizers provide annotated corpora with varying sizes in 14 different languages. in this paper, we present two variants of travis, travis multi and travis mono , which were submitted to the open track of the shared task, where additional resources were allowed. travis follows the tradition of approaching vmwe identification as a token classification task. to this end, it employs the, now standard, contextual embeddings model, bert @xcite , which has seen very limited application to this task. due to the multilingual nature of the shared task, we also pay special attention to the performance on different languages. the variants of travis are named after this concern, highlighting the type of the contextual embeddings in terms of their pre-training languages: travis multi only uses the multilingual-bert, which is trained on 104 languages, whereas travis mono uses the available language-specific bert model for each language. hence, the aim of the current submission is twofold: (i) we investigate the generalizability capabilities of pre-trained language models on identification of vmwes (ii) we provide a thorough comparison of the multilingual-bert against language-specific berts to understand the limitations of the former, if there is any, hoping to guide the future multilingual research on vmwes. ## background treating mwe identification as a sequence tagging problem has been one of the most popular approaches @xcite . to this end, @xcite propose new tagging schemes for vmwe by extending the bio format to allow the annotation of discontinuous and nested mwes. @xcite constitutes the first study which adopts this approach and applies deep learning models including feedforward, recurrent and convolutional networks. later, a number of studies adopting a recurrent neural network with an optional crf classifier have been proposed @xcite @xcite . @xcite further study the effects of different word representations on this architecture by using the veyn model of @xcite . @xcite specifically target the challenge caused by discontinuity of verbal mwes and propose a neural model which combines convolutional network and self-attention mechanism to deal with long-range relations. ## system description below, we briefly introduce the bert language model, which constitutes the backbone of our model, followed by the introduction of the proposed models. ## experimental design our implementation is based on the transformers library of huggingface @xcite . all monolingual bert models as well as mbert are obtained through huggingface's model hub foot_1 . in languages with several available bert models, we opted for the most downloaded cased one. the complete list of the models used in the first submission are provided in appendix a. we train all models for four epochs using the adam optimizer (kingma and ba, 2014) with a learning rate of 5e-5. the sequence length is set to 400 during training, but at the time of the inference we use bert's maximum sequence limit which is 512. as the fine-tuning procedure is prone to high variance, we run all our models four times and used the run with the best development performance to obtain the final predictions for the test sets. ## results and discussion travis variants ranked 2 nd and 3 rd in the general ranking, according to the target metric of the unseen mwe-based f-score. although travis multi ranks higher than travis mono in however, the performance of mbert cannot be simply dismissed, as the travis multi also achieves consistent results across languages. although the results are not directly comparable as the set of the languages is different and the datasets have, possibly, been modified over time, travis multi achieves an average of 10% increase in f-score over shoma @xcite , the best performing system of the previous parseme shared task (2018), in the identification of the unseen mwes. language-wise, travis mono achieved the best performance in the open track of the shared task for the six of the eight languages it covers which are: fr, it, pl, ro, tr, zh. these languages represent various language families suggesting that the performance of travis is stable typologically. as for travis multi , irish (ga) turns out to be the most challenging language with only 2.6% f-score. however, that is probably due to the size of the dataset that contains only 100 vmwes in the training portion which is too limited to fine-tune mbert in a meaningful way. the language-wise comparison of our submissions with the best performing system is provided in figure 1 . finally, a general advantage of employing contextual embeddings is being completely featureindependent. hence, the proposed model only requires a training data annotated for the positions of the target vmwes, rendering it easily adaptable to other low resource languages where obtaining other linguistics features, such as pos-tags or dependency trees, can be challenging. ## conclusion in this paper, we try to answer two questions: (i) how generalizable is the performance of contextual embeddings in vmwe identification, and (ii) if the pre-training language plays an important role or, in other words, the multilingual contextual embeddings are good enough. to this end, we offer a computational model, travis, which treats vmwe identification as a sequence classification task and employs various bert models. the results indicate that language-specific models perform particularly well on the identification of the unseen vmwes by outperforming the multilingual embeddings by 6% in f-score when compared on the same set of languages. yet, the multilingual-bert also exhibits strong multilingual abilities, suggested by the average of 32% f-score in identification of the unseen vmwes which is significantly higher than results obtained in the previous editions of the parseme shared tasks.