C ombo NER : A Lightweight All-In-One POS Tagger, Dependency Parser and NER (2021)

Abstract: The current natural language processing is strongly focused on raising accuracy. The progress comes at a cost of super-heavy models with hundreds of millions or even billions of parameters. However, simple syntactic tasks such as part-of-speech (POS) tagging, dependency parsing or named entity recognition (NER) do not require the largest models to achieve acceptable results. In line with this assumption we try to minimize the size of the model that jointly performs all three tasks. We introduce ComboNER: a lightweight tool, orders of magnitude smaller than state-of-the-art transformers. It is based on pre-trained subword embeddings and recurrent neural network architecture. ComboNER operates on Polish language data. The model has outputs for POS tagging, dependency parsing and NER. Our paper contains some insights from fine-tuning of the model and reports its overall results.

## introduction the paradigm for automated language processing was originally based on building dedicated tools for every task. we illustrate it on the example of the polish language, but it is very likely the case also for other languages. the dedicated tools include morphological taggers and disambiguators (a polish language approximation of pos tagging), dedicated dependency parsers and named entity recognizers. for convenience, the tools sometimes were made available as web services and possibly composed into chains. one such example is multiservice @xcite , a linguistic web service for polish, combining several mature offline linguistic tools in a common online platform. multiservice offers a number of pre-defined chains. for example, the dependency parser requires morphological tagging and disam-biguation as the initial step. the two tools are put together in appropriate order in one chain. among universal tools (not dedicated to the polish language) it is worth to mention: recently, with the advances of neural networks in natural language processing, new tools were proposed that simplify the chains by doing several tasks at once. one example is combo @xcite , a bi-directional recurrent neural network (rnn) using word2vec embeddings with multiple outputs for morphological and dependency analysis. the model was submitted to the conll 2018 universal dependency shared task. the combo tool, while doing certain things right, has its own issues. the biggest problem is the fact that it requires word-level word2vec embeddings for every token that it encounters. unfortunately, the number of unique tokens in the polish language is rather large. for example, in the national corpus of polish and wikipedia, the number of unique case-sensitive tokens exceeds 2 millions. this is by far too many to fit into gpu memory. this leads to a situation where only word2vec vectors that are used in input texts are loaded into gpu memory (embedding layer) and an additional preprocessing step of data-driven word vector selection and loading. depending on linguistic productivity, this operation has to be repeated once in a while when processing large amounts of text. to solve this problem, one can apply techniques of vocabulary selection such as those described in @xcite . hopefully, another simple solution exists for this issue: subword level embeddings @xcite . this idea, successfully applied in machine translation, became popular in multiple other scenarios and architectures, including transformer neural networks such as bert @xcite . it allows to represent an unknown word (not present in the dictionary) as a sequence of shorter tokens or subwords. using subwordlevel tokenization and embeddings the problem of vocabulary selection is non-existent. the goal of comboner is to (1) avoid the above issues of the initial combo @xcite by introducing subword tokenization and (2) add named entity recognition as another output. the overall design goal is to reduce the size of the model where possible. ## datasets this section describes the datasets used to train the comboner model. it is only an overview with basic information about data sizes, please refer the cited papers for more details such as label descriptions and annotation principles. ## hyperparameter tuning to tune the parameters we used only pos accuracy and one measure from the dependency parsing: uas score (unlabeled attachment score). this was in order to simplify the analysis and check the balance between types of outputs. therefore, the hyperparameter tuning was carried out only on the universal dependencies dataset. hyper-parameter tuning was done using the hparams api in tensorflow foot_2 and grid-search approach. tuning was subjected to the following hyperparameters: comboner uses pre-trained polish language subword embedding vectors from the bpemb library @xcite . therefore, vocabulary and vector sizes are determined by availability in bpemb. for initial parameter selection only the results of the first training epoch were taken into account. the interestingly, the number of subwords, i.e. vocabulary size (vocab), turned out to be a nonobvious measure. the pos tagger performed better with a large number of subwords (100k vocab), while a dependency parser preferred a smaller number (10k vocab). this effect was present regardless of rnn cell size. we also tested the configuration of the model with two bi-lstm layers. we tested two values of lstm cell sizes for the first layer (100, 200) and two (25, 50) for the second layer. when measuring performance for the first training epoch, a model with a two-layer bi-lstm performs generally worse than a single-layer model with similar settings. as a result, the addition of a second bi-lstm layer does not seem to be advisable. for the named entity recognition, two variants of the model architecture were tried. in the multiservice @xcite (and in many older generation nlp tools), named entity recognition model uses pos tags as an input feature and is therefore located after pos tagging. this observation leads to testing two design choices for the named entity recognition part of comboner. the first one is two independent outputs for ner and pos / dependency: the outputs share only the word embedding layer. the second assumes dependence: named entity branch takes as an input not only embeddings but also hidden states from the lstm in pos / dependency branch. this setting allows named entity recognition to take into account information from layers trained on syntactic data, in manner somewhat similar to using pos features in older nlp models. the results in terms of f1 scores of both variants are presented in ## comboner architecture figure 1 illustrates the design of comboner. data processing starts with subword tokenization and embedding layer. here, pre-trained subword embedding vectors are used (along with the tokenizer), namely monolingual polish language bpemb (heinzerling and strube, 2018) trained on wikipedia. this part is common to all outputs. then, computation is split into two branches: one for pos and dependency, the other for named entities. each of the two branches begins with a bi-directional lstm layer. named entity branch takes as input not only embeddings but also hidden states from the lstm in pos / dependency branch. architecture of this type was selected on the basis of experiments described in activations inside the network are tanh and softmax is used in the final layers. for details of implementation see @xcite ; in particular, dependency parser outputs with the dot product solution follow closely the initial model. ## conclusions this paper describes comboner: a new version of the combo tool @xcite , developed with following assumptions in mind: (1) use subwords to avoid out-of-vocabulary issues (words in model's input not present in memory / the embedding layer), (2) add named entity recognition output, (3) lightweight in terms of memory footprint, as of today's standards. the tool was implemented in tensorflow 2. comboner follows many design choices of the original combo @xcite , but introduces some new solutions. training multi-output models on multiple datasets is a challenging task. technically, it was solved by freezing appropriate part (layers) of the model while training another part, using a dedicated tensorflow data loader class for each corpus. the paper contains the results of hyperparameter optimization, which dictate the final hyperparameter choice for the comboner model. there are several interesting findings. first, the number of subwords (vocabulary size): the pos tagger performed better with a large number of subwords (100k vocab), while dependency parser preferred a smaller number (10k vocab). second, named entity layers greatly benefit from accessing layers pretrained for pos and dependency. this architecture improves the f1 score of named entity recognition by 12,8% on average compared to the variant with fully independent training of both tasks. the results of poleval competition in 2018 are an interesting reference for comparisons regarding dependency parser and named entity recognition quality @xcite foot_3 . it appears that las score achieved by the com-boner is competitive and easily on-par with participating systems. unfortunately, the solutions for named entity recognition outperform the output of comboner in a significant manner. one explanation of this fact is context: effective named entity recognition requires context wider than just the input sentence. most of the corpora and tools assume the supra-sentence level. important clues for identyfing named entities are often a part of preceding sentences. in order to evaluate the pos tagger, one can compare the results of the poleeval competition in 2017 @xcite 5 . in this context, the accuracy of 0.933 is very competitive and on-par with the best of the participating systems. to recap, this paper describes a relatively straightforward yet important modifications over previous work @xcite . our goals were to develop a small system and avoid large scale transformer, at the same time solving out-of-vocabulary problems posed by word-level embeddings. in the future it may be advisable to compare with small transformer models like dis-tilbert @xcite (40% less parameters than the bert base model).