Composing Byte-Pair Encodings for Morphological Sequence Classification (2020)

Abstract: Byte-pair encodings is a method for splitting a word into sub-word tokens, a language model then assigns contextual representations separately to each of these tokens. In this paper, we evaluate four different methods of composing such sub-word representations into word representations. We evaluate the methods on morphological sequence classification, the task of predicting grammatical features of a word. Our experiments reveal that using an RNN to compute word representations is consistently more effective than the other methods tested across a sample of eight languages with different typology and varying numbers of byte-pair tokens per word.

## introduction after its introduction, the transformer model @xcite has emerged as the dominant architecture for statistical language models, displacing recurrent neural networks, in particular, the lstm and its variants. the transformer owes its success to several factors, including the availability of pretrained models, which effectively yield rich contextual word embeddings. such embeddings can be used as is (for so-called feature extraction), or the pre-trained models can be finetuned to specific tasks. at the same time as transformer models became popular, the tokenization of natural language texts have shifted away from methods explicitly oriented towards words or morphemes. rather, statistical approaches are favoured: strings of characters are split into units which are not necessarily meaningful linguistically, but rather have statistically balanced frequencies. for example, the word "scientifically" may be composed of the tokens: "scient", "ifical", "ly" -here the central token does not correspond to a morpheme. that is, rather than identifying complete words or morphemes, one aims to find relatively large sub-word units occurring significantly often, while maximizing the coverage of the corpus (the presence of the "out of vocabulary" token is minimized). approaches for composing words from sub-word units have focused on combining character n-grams @xcite , while other approaches have looked at splitting words into roots and morphemes @xcite @xcite , and then combining them. in this paper, we consider byte-pair encodings (bpe) @xcite . bpe has been popularized by its usage in translation and the bert transformer model @xcite . the bpe algorithm does not specifically look for either character n-grams or morphs, but rather it aims at splitting a corpus c into n tokens, where n is user defined. even though bpe is not grounded in morphosyntactic theory, the characteristics of the sub-word units generated by bpe will be directly influenced by morphosyntactic patterns in a language. in particular, it is reasonable to expect that the statistical characteristics of bpe to be different between languages with different typologies. one issue with this tokenization scheme is that models based on bpe provide vector representations for the bpe tokens (which we call token embeddings from now on), while one is typically interested in representations for the semantically meaningful units in the original texts, words. in sum, one wants to combine token embeddings into word embeddings. this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . our main goal is to explore how to best combine token embeddings in the context of sequence classification on words, that is, the task of assigning a label to every word in a sentence. coming back to our example, we must combine the token embeddings assigned to the bpe tokens "scient", "ifical" and "ly" to form a word representation of "scientifically" (as a vector) which we can then assign a label to. to our knowledge, this is a little-studied problem. for the original bert model devlin et al. (2018) simply state that for named entity recognition the first sub-word token is used as the word representation. for morphological sequence classification @xcite report that only small differences in performance were found between averaging, taking the maximum value or first sub-word token. in this paper we explore the problem in further detail and identify the effect that different methods have on the final performance of a model. additionally, with the increased interest in multilingual nlp it becomes important to explore how different computational methods perform cross-linguistically. that is, because languages are different morphosyntactically, one can expect various computational methods not to be uniformly effective. ## task to investigate composition methods for token embeddings we focus on the task of morphological sequence classification. the task is to assign a tag to a word that represent its grammatical features, such as gender, number and so on. in addition to the word-form, the system can use information from context words as cues. while the grammatical features primarily are given by the word-form, useful information is also found in the context. thus, we have to identify k different tags for a word, each with c i possible classes, making the task a multi-class classification problem. we simplify the classification problem by combining the different tags into a composite tag with up to k i c i classes (instead of making k separate predictions). this task is suitable for our goal as the output space is large, ranging from 100 to 1000 possible tags for a word, depending on the grammatical features present in the language 1 , and is directly linked to the affixes in the word-form. a system must efficiently encode information about the structure of the target words as well as the context words to be able to predict the correct grammatical features. ## data for both training and testing data, we use the universal dependencies dataset @xcite annotated with the unimorph schema @xcite . we are mainly interested in how the accuracy is influenced by different composition methods, but also consider the type of morphology a language uses as a factor in this task. with this in mind, we consider both languages that use agglutinative morphology where each morpheme is mapped to one and only one grammatical feature, and languages that use fusional morphology where a morpheme can be mapped to one or more grammatical features. the fusional languages that we consider are arabic, czech, polish and spanish, and the agglutinative languages that we consider are finnish, basque, turkish, and estonian. we show the size, the average number of bpe tokens per word, and the number of morphological tags for each treebank in the fusional languages were chosen such that two of them (czech and polish) have a higher bpe per word ratio than the other two (arabic and spanish). we make this choice because one factor that impacts the accuracy obtained by a composition method may be the bpe per word ratio. by having both fusional and agglutinative languages with similar bpe per word ratio we can take this variable into account properly in our analysis. ## results even though our aim is to compare the relative performance of various bpe-combination methods rather than to improve on the state of the art in absolute terms, we compare our results against the baseline reported by @xcite . this comparison serves the purpose of checking that our system is generally sound. in particular, the actual state of the art, as reported by @xcite , uses treebank concatenation or other methods to incorporate information from all treebanks available in a language, which means that results are not reported on a strict per-treebank basis and thus our numbers are not directly comparable. we report the accuracy of prediction morphological tags for each of our composition methods, and for our two training regimes in our system performs better than the baseline. as a general trend we see that the rnn method tends to perform better than all other tested methods. this trend is consistent across both language families (agglutinative and fusional) and training regimes showing that, while the advantage of the rnn is small, focusing on the finetuning regime only, we see the largest benefits of the rnn method for basque with an increased performance of 3.25 points, and 2.7 points for turkish over using mean or averaging. the first method for basque and turkish performs worse with a decrease of 4.4 percentage points for basque and 3.6 points for turkish compared to the rnn method. in the bare features extraction regime, we see a larger benefit for the rnn, of 3.7 percentage points (turkish) and 4.95 points (basque). again, this is not unexpected: when finetuning the error rate is smaller, and therefore there is a smaller margin for a subsequent phase to yield and improvement. ## discussion for predicting morphological features, the rnn method is more effective than the other proposed methods (summing, averaging or taking the first bpe token). this holds regardless of training regime (finetuning versus feature extraction) and across languages with different bpe per word ratios. as we see it, the advantage of the rnn over commutative methods (sum, mean) and taking the first bpe token is that it can take the order of elements into account. in broad terms, information about the order of elements in morphology allows a system to determine what is a stem, prefix, or suffix. thus allowing a model to collect more predictive information from token embeddings. we can suspect that the average bpe per word ratio in a language affects the performance of the composition method used. to further control this variable, in figure 3 we plot the average number of bpe tokens per word in each language (x-axis), and compare this average against the gain in accuracy yielded by using the rnn method over summation (y-axis). for finetuning we see that in general the average number of bpe tokens does not matter that much. the two cases where it does matter is for turkish and basque, where we see a substantial improvement of about 3 percentage points. we note however that these are also the languages with the lowest amount of training data. for the other languages the improvements lie in the range .6 to 1.2 percentage points. this indicates that when finetuning, the model can provide information that allows commutative methods to properly compose bpe tokens. however, looking at bare feature extraction we see that there is a larger gap between the low bpe-ratio and the high bpe-ratio languages. our sample of languages contain both fusional and agglutinative languages, and the typology does not appear to have an effect in our experiments. we see about the same trends for the fusional languages with a high bpe per word ratio as the agglutinative languages. ## conclusions and future work in conclusion, our results indicate that using an rnn to compose word representations from token representations, obtained from a large transformer model, is more efficient than two commutative methods, summing and averaging, and also more effective than letting a transformer model automatically pool the predictive word-level information into the first bpe token embedding. we show this for the task of morphological sequence classification, in eight different languages with varying morphology and wordlengths in term of bpe tokens, as well as for two training regimes, finetuning and feature extraction. in future work, we want to continue experimenting with the different bpe token embedding composition methods, specifically looking at more complex syntactic and semantic tasks, such as dependency and/or constituency parsing, semantic role labeling, named entity recognition, and natural language inference. we also wish to run our experiments on the hundreds of available ud treebanks to improve the robustness of our results.