Subword Pooling Makes a Difference (2021)

Abstract: Contextual word-representations became a standard in modern natural language processing systems. These models use subword tokenization to handle large vocabularies and unknown words. Word-level usage of such systems requires a way of pooling multiple subwords that correspond to a single word. In this paper we investigate how the choice of subword pooling affects the downstream performance on three tasks: morphological probing, POS tagging and NER, in 9 typologically diverse languages. We compare these in two massively multilingual models, mBERT and XLM-RoBERTa. For morphological tasks, the widely used ‘choose the first subword’ is the worst strategy and the best results are obtained by using attention over the subwords. For POS tagging both of these strategies perform poorly and the best choice is to use a small LSTM over the subwords. The same strategy works best for NER and we show that mBERT is better than XLM-RoBERTa in all 9 languages. We publicly release all code, data and the full result tables at https://github.com/juditacs/subword-choice .

## introduction training of contextual language models on large training corpora generally begins with segmenting the input into subwords @xcite to reduce the vocabulary size. since most tasks consume full words, practitioners have the freedom to decide whether to use the first, the last, or some combination of all subwords. the original paper introducing bert, @xcite , suggests using the first subword for named entity recognition (ner), and did not explore different poolings. @xcite also use the first subword, for dependency parsing, and remark in a footnote that they tried the first, last, average, and max pooling but the choice made no difference. @xcite report similar findings for constituency parsing, but nevertheless opt for reporting results only using the last subword. @xcite take the average of the subword vectors for syntactic and word sense disambiguation tasks. @xcite use attentive pooling with a trainable norm for news topic classification and sentiment analysis in english. @xcite use hierarchical pooling for sequence classification tasks in english and chinese. here we show that for word-level tasks (morphological, pos and ner tagging), particularly for languages where the proportion of multi-subword tokens (i.e. those word tokens that are split into more than one subword) is high, more care needs to be taken as both pooling strategy, and that the choice of language matters. we demonstrate this clearly for european languages with rich morphology, and in chinese, japanese and korean (cjk). similar to subword pooling, the choice of the lowest layer, the topmost one, or some combination of the activations in different layers has to be made. here our main focus is subword pooling, but we do discuss layer pooling to the extent it sheds light on our main topic. we observe that the gap between using the first and the last subword unit is larger in lower layers than in higher ones. we describe our data and tasks in section 2, and the subword pooling strategies investigated in section 3. our results are presented in section 4, and in section 5 we offer our conclusions. our main contributions are: we investigate pooling through three kinds of tasks. in morphological tasks we attempt to predict morphological features such as gender, tense, or case. in pos tasks we predict the lexical category associated with each word. in ner tasks we assign bio tags @xcite to named entities. we chose word-level, as opposed to syntactic, tasks because they can be tackled with fairly simple architectures and thus allow for a large number of experiments that highlight the differences between subword pooling strategies. our experiments are limited only by the availability of standardized multilingual data. we use universal dependencies (ud) @xcite for morphological and pos tasks, and wikiann @xcite for ner. we pick the largest treebank in each language from ud and sample 2000 train, 200 dev and 200 test sentences for the morphological probes and up to 10,000 train, 2000 dev and 2000 test sentences -often limited by the size of the treebank -for pos. we chose languages with reasonably large treebanks in order to generate enough training data, making sure we have an example from each language family, as well as one from european subfamilies since their treebanks tend to be very large. we use 10,000 train, 2000 dev and 2000 test sentences for ner. preprocessing steps are further explained in appendix a. our choice of languages are arabic, chinese, czech, english, finnish, french, german, japanese, and korean. ud's gold tokenization is kept and we run subword tokenization on individual tokens rather then the full sentences. morphological tasks ud assigns zero or more tag-value pairs to each token such as @xmath0. we define a probe as a triplet of language, tag, pos , i.e. we train a classifier to predict the value of a single tag in a sentence in a particular language. 1 the task english, verbform, verb would be trained to predict one of three labels for each english verb: finite, infinite or gerund. we pick 4 tasks that are applicable to at least 3 of the 6 languages where the task makes sense (there are no morphological tags for chinese and japanese, and korean uses a different tagging scheme). part-of-speech tagging assigns a syntactic category to each token in the sentence. usually treated as a crucial low level task to provide useful features for higher level linguistic analysis such as syntactic and semantic parsing. universal pos tags (upos) are available in ud in all 9 languages. named entity recognition is a classic information extraction subtask that seeks to identify the span of named entities mentioned in the sentence and classify them into pre-defined categories such as person names, organizations, locations etc. ner was the only token level task explored in the original bert paper @xcite . architectures bert and other contextual models use subword tokenizers that generate one or more subwords for each token. in this study we compared mbert and xlm-roberta, two transformer-based large scale language models with support for over 100 languages. we pick these two since they are architecturally similar (both have 12 layers and the same hidden size) making our comparison easier. mbert was trained on wikipedia while xlm-roberta was trained on commoncrawl @xcite . both models have been extensively applied to english and multilingual tasks, but generally at the sentence or sentence pair level, where subword issues do not come to the fore. mbert uses a common wordpiece vocabulary with 118k subword units. when a word is split into multiple subword units, each token that is not the first one is prefixed with ##. xlm-roberta's vocabulary was trained in a similar fashion but with 250k units and a special start symbol (unicode lower eights block) instead of continuation symbols. each word is prefixed with this start symbol before it is tokenized into one or more subword units. these start symbols are often then tokenized as single units, particularly before chinese, japanese and korean characters, therefore artificially increasing the subword unit count. we indicate the proportion of words starting with a standalone start symbol along with other tokenization statistics in as we also observe that the two models' tokenizers work in very different ways. out of the 2800 morphological test examples, only 58 are tokenized the same way and 51 of these are not split into multiple subwords. only 7 words that are in fact tokenized, are tokenized the same way. although the full tokenization is rarely the same, the first and the last subwords are the same in 45.5% and in 44.7% of the cases. ## subword pooling we test 9 types of pooling methods listed in ## results our analysis consisted of two steps. we first performed the first and last tasks at each layer (see figure 2 ). based on the results of this, we picked a single layer, the 6th, to test all 9 subword pooling choices. the full list of results on the 6th layer is listed in appendix c. ## conclusion the key takeaway from our work is that performance on lower level tasks depends on the way we pool over multiple subword units that belong in a single word token. this is more of an issue in languages other than english, where a significantly larger proportion of words are represented by multiple subword units. morphological and pos tasks are both probing word-level attributes, but the results show huge disparity: for the morphological tasks first pooling is the worst strategy, and attn is the best, while for pos tagging attn is almost as bad as first, the best being lstm. the ner task is intermedi-ary between word-and phrase-level, and subword pooling effects are less marked, but still statistically significant (see the full result tables in the appendix). ud's train set. we sample the sentences in a way that avoids overlaps in target words between train, dev and test splits, in other words, if a word is the target in the train set, we do not allow the same target word in the dev or test set. a target word is the word that needs to be classified according to some morphological tag. we also limit class imbalance to 3:1 at max. this results in the removal of rare tags such as a few of the numerous finnish noun cases. these restrictions and the size of the treebanks do not allow generating larger datasets.