COMBO : State-of-the-Art Morphosyntactic Analysis (2021)

Abstract: We introduce COMBO – a fully neural NLP system for accurate part-of-speech tagging, morphological analysis, lemmatisation, and (enhanced) dependency parsing. It predicts categorical morphosyntactic features whilst also exposes their vector representations, extracted from hidden layers. COMBO is an easy to install Python package with automatically downloadable pre-trained models for over 40 languages. It maintains a balance between efficiency and quality. As it is an end-to-end system and its modules are jointly trained, its training is competitively fast. As its models are optimised for accuracy, they achieve often better prediction quality than SOTA. The COMBO library is available at: https://gitlab.clarin-pl.eu/syntactic-tools/combo .

## introduction natural language processing (nlp) has long recognised morphosyntactic features as necessary for solving advanced natural language understanding (nlu) tasks. an enormous impact of contextual language models on presumably all nlp tasks has slightly weakened the importance of morphosyntactic analysis. as morphosytnactic features are encoded to some extent in contextual word embeddings (e.g. @xcite , doubts arise as to whether explicit morphosyntactic knowledge is still needed. for example, @xcite have recently investigated an intermediate fine-tuning contextual language models on the dependency parsing task and suggested that this step does not significantly contribute to advance nlu models. conversely, @xcite reveal the powerlessness of contextual language models in encoding linguistic phenomena like negation. this is in line with our intuition about representing negation in polish sentences (see figure 1 ). it does not seem trivial to differentiate between the con-tradicting meanings of these sentences using contextual language models, as the word context is similar. the morphosyntactic features, e.g. parts of speech part vs. intj, and dependency labels advmod:neg vs. discourse:intj, could be beneficial in determining correct reading. in order to verify the influence of explicit morphosyntactic knowledge on nlu tasks, it is necessary to design a technique for injecting this knowledge into models or to build morphosyntax-aware representations. the first research direction was initiated by @xcite . our objective is to provide a tool for predicting high-quality morphosyntactic features and exposing their embeddings. these vectors can be directly combined with contextual word embeddings to build morphosyntactically informed word representations. the emergence of publicly available nlp datasets, e.g. universal dependencies @xcite , stimulates the development of nlp systems. some of them are optimised for efficiency, e.g. spacy @xcite , and other for accuracy, e.g. udpipe @xcite , the stanford system @xcite , stanza @xcite . in this paper, we introduce combo, an open-source fully neural nlp system which is optimised for both training efficiency and prediction quality. due to its end-to-end architecture, which is an innovation within morphosyntactic analysers, combo is faster in training than the sota pipeline-based systems, e.g. stanza. as a result of applying modern nlp solutions (e.g. contextualised word embeddings), it qualitatively outperforms other systems. combo analyses tokenised sentences and predicts morphosyntactic features of tokens (i.e. parts of speech, morphological features, and lemmata) and syntactic structures of sentences (i.e. dependency trees and enhanced dependency graphs). at the same time, its module, combo-vectoriser, extracts vector representations of the predicted features from hidden layers of individual predictors. combo user guide is in §4 and a live demo is available on the website http://combo-demo.nlp. ipipan.waw.pl . contributions 1) we implement combo ( §2), a fully neural nlp system for part-of-speech tagging, morphological analysis, lemmatisation, and (enhanced) dependency parsing, together with combo-vectoriser for revealing vector representations of predicted categorical features. combo is implemented as a python package which is easy to install and to integrate into a python code. 2) we pre-train models for over 40 languages that can be automatically downloaded and directly used to process new texts. 3) we evaluate combo and compare its performance with two state-of-the-art systems, spacy and stanza ( §3). ## combo architecture combo's architecture (see figure 2 ) is based on the forerunner @xcite implemented in the keras framework. apart from a new implementation in the pytorch library @xcite , the novelties are the bert-based encoder, the eud prediction module, and combo-vectoriser extracting embeddings of upos and deprel from the last hidden layers of combo's tagging and dependency parsing module, respectively. this section provides an overview of combo's modules. implementation details are in appendix a. local feature extractors local feature extractors (see figure 2 ) encode categorical features (i.e. words, parts of speech, morphological features, lemmata) into vectors. the feature bundle is configurable and limited by the requirements set for combo. for instance, if we train only a dependency parser, the following features can be input to combo: internal character-based word embeddings (char), pre-trained word embeddings (word), and embeddings of lemmata (lemma), word lemma char upos ufeats head deprel upos ufeats enhhead enhdeprel tokenised sentences concatenated local features local extractors predictors lemma encoder figure 2: combo architecture. explanations: cnn fc system-trained bilstm optional required parts of speech (upos) and morphological features (ufeats). if we train a morphosyntactic analyser (i.e. tagger, lemmatiser and parser), internal word embeddings (char) and pre-trained word embeddings (word), if available, are input to combo. words and lemmata are always encoded using character-based word embeddings (char and lemma) estimated during system training with a dilated convolutional neural network (cnn) encoder @xcite . additionally, words can be represented using pretrained word embeddings (word), e.g. fasttext @xcite , or bert @xcite . the use of pre-trained embeddings is an optional functionality of the system configuration. combo freezes pre-trained embeddings (i.e. no fine-tuning) and uses their transformations, i.e. embeddings are transformed by a single fully connected (fc) layer. part-of-speech and morphological embeddings (upos and ufeats) are estimated during system training. since more than one morphological feature can attribute a word, embeddings of all pos-sible features are estimated and averaged to build a final morphological representation. global feature encoder the encoder uses concatenations of local feature embeddings. a sequence of these vectors representing all the words in a sentence is processed by a bidirectional lstm @xcite . the network learns the context of each word and encodes its global (contextualised) features (see figure 3 ). global feature embeddings are input to the prediction modules. ## combo performance data combo is evaluated on treebanks from the universal dependencies repository @xcite , preserving the original splits into training, validation, and test sets. the treebanks representing distinctive language types are summarised in by default, pre-trained 300-dimensional fasttext embeddings @xcite are used. we also test encoding data with pre-trained contextual word embeddings (the tested bert models are listed in qualitative evaluation regarding lemmatisation, stanza has an advantage over combo in most tested languages. this is probably due to the fact that stanza lemmatiser is enhanced with a key-value dictionary, whilst combo lemmatiser is fully neural. it is not surprising that a dictionary helps in lemmatisation of isolating languages (english). however, the dictionary approach is also helpful for agglutinative languages (finnish, korean, basque) and for arabic, but not for polish (fusional languages). comparing combo models estimated with and without bert embeddings, we note that bert boost only slightly increases the quality of lemma prediction in the tested fusional and agglutinative languages. for a complete insight into the prediction quality, we evaluate individual upos and udeprel predictions in english (the isolating language), korean (agglutinative) and polish (fusional). result visualisations are in appendix c. combo took part in iwpt 2021 shared task on parsing into enhanced universal dependencies @xcite , where it ranked 4th. 4 in addition to elas and eulas metrics, the third evaluation metric was las. combo ranked 2nd, achieving the average las of 87.84%. the score is even higher than the average las of 86.64% in ## conclusion we have presented combo, the sota system for morphosyntacic analysis, i.e. part-of-speech tagging, morphological analysis, lemmatisation, and (enhanced) dependency parsing. combo is a language-agnostic and format-independent system (i.e. it supports the conll-u and conll-x formats). its implementation as a python package allows effortless installation, and incorporation into any python code or usage in the cli mode. in the python mode, combo supports automated download of pre-trained models for multiple languages and outputs not only categorical morphosyntactic features, but also their embeddings. in the cli mode, pre-trained models can be manually downloaded or trained from scratch. the system training is fully configurable in respect of the range of input features and output predictions, and the method of encoding input data. last but not least, combo maintains a balance between efficiency and quality. admittedly, it is not as fast as spacy, but it is much more efficient than stanza considering the training time. tested on the selected ud treebanks, combo morphosyntactic models enhanced with bert embeddings outperform spacy and stanza models. language language type ud treebank #words #trees reference english isolating english-ewt 254,856 16,622 silveira et al. (2014) arabic fusional arabic-padt 282,384 7,664 hajič et al. (2009) polish fusional polish-pdb 350,036 22,152 wróblewska (2018) finnish agglutinative finnish-tdt 202,453 15,136 haverinen et al. (2014) korean agglutinative korean-kaist 350,090 27,363 chun et al. (2018) turkish agglutinative turkish-imst 57,859 5,635 sulubacak et al. (2016) basque agglutinative (fusional basque-bdt 121,443 8,993 aranzabe et al. (2015) verb morphology)