ERMI at PARSEME Shared Task 2020: Embedding-Rich Multiword Expression Identification (2020)

Abstract: This paper describes the ERMI system submitted to the closed track of the PARSEME shared task 2020 on automatic identification of verbal multiword expressions (VMWEs). ERMI is an embedding-rich bidirectional LSTM-CRF model, which takes into account the embeddings of the word, its POS tag, dependency relation, and its head word. The results are reported for 14 languages, where the system is ranked 1st in the general cross-lingual ranking of the closed track systems, according to the Unseen MWE-based F1.

## introduction multiword expressions (mwes) are lexical items that consist of multiple lexemes. the challenge of identifying mwes comes from the fact that their properties cannot directly be deducted from the lexical, syntactic, semantic, pragmatic, and statistical properties of their components @xcite . addressing this challenge, the parseme shared task 2020 is a campaign that encourages the development of automatic verbal mwe (vmwe) identification models in a multilingual context. in this third edition of the parseme shared task, the focus is on identifying vmwes that are unseen in training data. for this task, dev, test, train and raw corpora have been provided for 14 languages. ermi (embedding-rich multiword expression identification) is a multilingual system with a bidirectional lstm-crf architecture, which can take as input the embeddings of the word, its pos tag, dependency relation, and its head word. since the main focus of the shared task is to identify unseen vmwes, we experiment with how the addition of the head word embedding affects the prediction results for different languages. in addition, we also take advantage of the raw corpora in a semi-supervised teacher-student neural model carrying the same lstm-crf architecture for two languages (el, tr). we use no external resources in the training of our system, thus participating in the closed track. the results for all 14 languages in the closed track have been submitted where language-specific combinations of the above-mentioned embeddings have been used as input to the system. the system has been ranked 1 st in the general cross-lingual ranking of the closed track systems for the unseen mwe-based f 1 , and 2 nd for the global mwe-based and global token-based f 1 metrics. ## system description named entity recognition (ner) and mwe detection can be considered similar tasks, thus encouraging similar architectures. neural models have been preferred frequently for ner @xcite , and for detecting vmwes in the previous edition of parseme @xcite @xcite @xcite . in order to detect vmwes, we develop a system 1 consisting of three (two supervised, one semisupervised) neural network models, all of which carrying the same, bidirectional lstm-crf architecture, as proposed by @xcite for sequence tagging tasks. all models consist of three layers: the input layer, lstm layer, and crf layer, implemented using keras (chollet and others, 2015) with tensorflow backend @xcite . the architecture of each model is shown in figure 1 . the input of our neural networks is an embedding layer, where we provide the model with the concatenation of the embeddings of the word, its pos tag, its dependency relation to the head, and the head of the word (for some languages). we do not use a pre-trained word embedding model. instead, we exploit the provided raw corpora foot_0 , which are gathered specifically for this task and are in the same domain as the annotated corpora foot_1 ; and train fasttext word embedding models for each of the 14 languages separately, using gensim's fasttext implementation @xcite . the embedding vector dimension for all languages is 300 (for word and head word embeddings), whereas the vocabulary size of the embedding models vary due to different sizes of raw corpora. due to computational limitations, we only use a portion of the raw corpora for fr, pl and sv languages. ## experimental setup tagging scheme: during pre-processing, we adopt the bigappy-unicrossy tagging scheme proposed by @xcite to better represent overlapping (nesting and crossing) and discontinuous mwes. datasets: during the validation runs (results of which are explained in section 4.1), we concatenate the training and development corpora for each language, and randomly split 90% for training and 10% for testing. for the teacher-student model, we also use a portion of the raw corpora (roughly half the size of the training sets). after selecting the best system (out of ermi, ermi-head, and teachermi) for each language, we train our final models using the combined training and development sets, and use the blind test data for testing. for turkish (tr) and greek (el), we develop a teacher-student model, using 10,796 and 9,510 sentences, respectively, of the provided raw corpora in addition to the development and training sets. hyperparameters: we choose the mini batch size and number of epochs with respect to the size of training sets for each language (ref. ## results and discussion we make validation runs on the training and development data, and compare our three neural models for each language. afterwards, we report the official results of the selected systems on the blind test set. ## conclusion in this paper we proposed an embedding-rich bidirectional lstm-crf system. in addition to word, pos and dependency relation embeddings, we exploited head word embeddings, especially to tackle the issue of predicting unseen vmwes. within the closed track, we have used the raw corpora to train word embeddings, as well as proposing a semi-supervised teacher-student model, providing the opportunity of training on unlabeled data for vmwe identification. these methods have increased the generalisation power, enabling our system to perform best in predicting unseen vmwes in the closed track.