MTLB - STRUCT @Parseme 2020: Capturing Unseen Multiword Expressions Using Multi-task Learning and Pre-trained Masked Language Models (2020)

Abstract: This paper describes a semi-supervised system that jointly learns verbal multiword expressions (VMWEs) and dependency parse trees as an auxiliary task. The model benefits from pre-trained multilingual BERT. BERT hidden layers are shared among the two tasks and we introduce an additional linear layer to retrieve VMWE tags. The dependency parse tree prediction is modelled by a linear layer and a bilinear one plus a tree CRF architecture on top of the shared BERT. The system has participated in the open track of the PARSEME shared task 2020 and ranked first in terms of F1-score in identifying unseen VMWEs as well as VMWEs in general, averaged across all 14 languages.

## introduction in addition to other challenges in multiword expression (mwe) processing that were addressed in previous work, such as non-compositionality @xcite , discontinuity @xcite , and syntactic variability @xcite , the parseme shared task edition 1.2 1 has focused on another prominent challenge in detecting mwes, namely detection of unseen mwes. the problem with unseen data is common for many nlp tasks. while rule-based and unsupervised ml approaches are less affected by unseen data, supervised ml techniques are often found to be prone to overfitting. in this respect, the introduction of language modelling objectives to be added to different nlp tasks and their effect on generalisation have shown promising results @xcite . further improvements brought by pre-trained language models made them a popular approach to a multitude of nlp tasks @xcite . one particular advantage of such models is that they facilitate generalisation beyond task-specific annotations @xcite . mwes are inherent in all natural languages and distinguishable for their syntactic and semantic idiosyncracies @xcite . since language models are good at capturing syntactic and semantic features, we believe they are a suitable approach for modelling mwes. in particular, our system relies on bert pre-trained language models @xcite . additionally, we render the system semi-supervised by means of multi-task learning. the most promising feature to be jointly learned with mwes is dependency parse information @xcite . accordingly, we fine-tune bert for two different objectives: mwe detection and dependency parsing. mwe learning is done via token classification using a linear layer on top of bert, and dependency parse trees are learned using dependency tree crf network @xcite . our experiments confirm that this joint learning architecture is effective for capturing mwes in most languages represented in the shared task. 2 ## related work in earlier systems, mwes were extracted using pre-defined patterns or statistical measures that either indicated associations among mwe components or (non-)compositionality of the expressions with regard to the components @xcite . for example, @xcite employed such a system for identifying mwes. while these models can be effective for some frequent mwes, their main disadvantage is that they capture mwe types (as opposed to tokens) and they are unable to take context into account in running texts. the use of supervised machine learning was facilitated by the availability of resources tagged for mwes @xcite @xcite proposed a transition-based system based on an arc-standard dependency parser @xcite which ranked first in the first edition of parseme shared task on automatic identification of verbal mwes (vmwes) @xcite . @xcite proposed a cnn-lstm system which exploited fasttext word representations and ranked first in the open track of the parseme shared task edition 1.1 @xcite . previous systems such as traversal @xcite (ranked first in the closed track of the parseme shared task edition 1.1), and crf-seq/dep @xcite employed tree crf using dependency parse features in non-deep learning settings. they showed strengths of this approach particularly in the case of discontinuous vmwes. in shoma @xcite , using a linear-chain crf layer on top of the cnn-bilstm model did not result in improvements. in this work, we use tree crf, implemented as part of the torch-struct library @xcite , to model dependency trees, and we show that when it is jointly trained with a transformer-based mwe detection system, it improves mwe prediction for a number of languages. recently, @xcite proposed that learning mwe lexicons in an unsupervised setting is an important step that can be used in combination with a supervised model, especially when the latter is trained on a small amount of data. while we do not specifically learn mwe lexicons from external unannotated data, we believe that state-of-the-art pre-trained language representation models can capture crucial information about mwes similar to other nlp phenomena @xcite . for instance, @xcite showed how a semi-supervised system may benefit from pre-trained language model-based embeddings for named entity recognition (ner) and chunking. the joint learning of mwes and dependency parsing has been proved effective in @xcite . they proposed an arc-standard transition-based system which draws on a new representation that has two linguistic layers (a syntactic dependency tree and a forest of mwes) sharing lexical nodes. the closest to our work is @xcite where they have trained a multi-task neural network which jointly learns vmwes and dependency parsing on a small english dataset and uses elmo pre-trained embeddings. our work here is different in that we fine-tune the bert architecture and we use a tree crf for dependency parsing. ## system description we use pre-trained bert for language representation @xcite as the basis for our neural network. the bert architecture is based on standard transformers involving self-attention layers of encoders and decoders. 3 what makes it different from other transformer-based pre-trained language representation models is its capability in encoding the representation in a bidirectional way through a masked language model schema. the reason that we choose bert among other pre-trained models is the availability of multi-lingual pre-trained bert. 4 our model is set up to learn mwes and dependency trees simultaneously. bert weights are shared among the two tasks. a fully connected layer that performs sequence tagging is added as the final layer for mwe objective. parallel to that, linear layers and a dependency crf module are introduced to perform structured prediction for dependency trees. 5 the whole model is trained in an end-to-end manner. figure 1 depicts the overall architecture of the system. we use torch-struct (rush, 2020) for dependency parsing where tree crf is implemented as a distribution object. we first apply a linear followed by a bilinear layer on bert's output to obtain the adjacency matrix structure of the dependency tree. the outputs from these layers are considered as logpotentials (l) for the crf distribution. the distribution takes in log-potentials and converts them into one is a linear classifier layer for mwe tagging and the other consists of a linear layer, a bilinear layer and a tree crf dependency tagger. probabilities crf (z; l) of a specific tree z. we query the distribution to predict over the set of trees using argmax z crf (z; l). the cost for updating the tree is based on the difference between the tree probability and the gold standard dependency arcs. the mwe classification layer is optimised by cross-entropy between the ground truth mwe tags and the predicted ones, while the cost for crf is estimated using log probabilities over the tree structures. note that log probabilities (logprobs) for crf are large negative values which should be maximised, so we multiply them by -1 to get the dependency loss values compatible with mwe ones: loss @xmath0. the overall loss function to be optimised by adam optimiser is a linear combination of the two losses, loss mwe and loss dep which are the losses for multi-word expression and dependency parse tree, respectively, with α being a constant value which is empirically set to 0.001 ≤ α ≤ 0.01. we adapted the sequential labelling scheme of @xcite which is similar to iob with the difference that it introduces a new soft label o-for the tokens that are in between components of an mwe. we preserved mwe categories by suffixing the label with the category name. in this case, the annotations for the idiomatic verbal expression (of type vid) in the sentence i would give this job a go, would be: i [o] would [o] give [b-vid] this [o-vid] job [o-vid] a [i-vid] go [i-vid] with the labels shown as subscripts in brackets. foot_3 in the development phase of the shared task, we trained various configurations of our system and evaluated the performance on development sets. specifically, we examined the performance of our model in two settings: (1) the model is back-propagated only based on loss mwe (single-task), and (2) the learning is based on the multi-task loss (multi-task). we decided on the setting to be used for each language separately based on the performance on development sets. we used bert-base-multilingualcased as the pre-trained model for all languages. 7 due to lack of time and resources, we did not perform any extensive hyper-parameter search. we empirically chose learning rate 3 × 10 -5 and batch size 10 (except for ga where the selected batch size is 1). we trained the models for 10 epochs, and the maximum lengths of sentences for training were chosen for each language separately based on the word piece tokenisation of multilingual bert. 8 ## results and analysis our system is applied to all 14 languages and achieves the highest f1 score overall. the amount of mwes seen in the training data is the largest contributing factor, as the percentage of seen-in-train gold mwes is highly linearly correlated (@xmath1.90) with the global mwe-based f1 score across the languages. we achieve the highest performance in terms of mwe-based f1 score on unseen data for 8 out of 14 languages, with the largest gaps in performance observed on pt, where our system outperforms seen2unseen by 21.59 points, and on hi, where the gap between our system's f1 and that of seen2unseen equals 10.45 points. we note that our system works significantly better than the second best systems for smaller datasets (ga, he, and hi) which also happen to have larger amount of unseen expressions. at the same time, travis-mono outperforms our system on fr, it, pl, tr, and zh, with the largest gap of 5.68 points observed on fr. in addition, our system's performance is balanced across continuous and discontinuous mwes, with the exceptions of hi and tr, where discontinuous mwes amount to 7% and 4% of all mwes, respectively, and our system's performance drops by as much as 30 f1 points compared to its performance on continuous mwes. the distinction between multiand single-token mwes is only applicable to 3 languages, on two of which (de and sv) our system achieves an f1 score above 0.80 on single tokens. finally, the shared task data shows a wide diversity of vmwe categories present in different languages: from just three in eu and tr up to eight in it. once again, we note that our system is applicable to detection of all categories: for instance, it achieves the highest f1 scores among all systems in identification of ls.icv, a rare language-specific category of inherently clitic verbs used only in italian. at the same time, we identify lvc.cause, light-verb constructions with the verb adding a causative meaning to the noun, as the most problematic category on which our system achieves comparatively poorer results, especially on de, el, fr, hi, pt, and sv. it is worth noting that no language specific feature is used in our system and the authors were not involved in the creation of the datatsets. overall, we note that our system is not only cross-lingual, but also robust in terms of its performance and is capable of generalising to unseen mwes. ## conclusions and future work we described mtlb-struct, a semi-supervised system that is based on pre-trained bert masked language modelling and that jointly learns vmwe tags and dependency parse trees. the system ranked first in the open track of the parseme shared task -edition 1.2 and shows the overall state-of-theart performance for detecting unseen vmwes. in future, we plan to augment the dependency parsing architecture to train on dependency relation categories (labels) as well as dependency arcs. we also plan to improve our system by making it more efficient in order to train the dependency parsing module on the extra available unannotated datasets.