Multilingual and Zero-Shot is Closing in on Monolingual Web Register Classification (2021)

Abstract: This article studies register classification of documents from the unrestricted web, such as news articles or opinion blogs, in a multilingual setting, exploring both the benefit of training on multiple languages and the capabilities for zero-shot cross-lingual transfer. While the wide range of linguistic variation found on the web poses challenges for register classification, recent studies have shown that good levels of cross-lingual transfer from the extensive English CORE corpus to other languages can be achieved. In this study, we show that training on multiple languages 1) benefits languages with limited amounts of register-annotated data, 2) on average achieves performance on par with monolingual models, and 3) greatly improves upon previous zero-shot results in Finnish, French and Swedish. The best results are achieved with the multilingual XLM-R model. As data, we use the CORE corpus series featuring register annotated data from the unrestricted web.

## introduction the focus of this paper is on multilingual training and cross-lingual transfer in register classification of web documents. text register (or genre) @xcite , such as discussion forum or encyclopedia article, has been shown to be one of the most important predictors of linguistic variation @xcite , and register affects also the automatic processing of text @xcite @xcite . yet, web data is typically used without register information in many nlp tasks. web register classification studies have suffered from the lack of corpora featuring the full range of registers found on the web, as many datasets are based on a priori selection of register categories instead of unrestricted sampling of the web @xcite . furthermore, despite the availability of web-scale data in hundreds of languages, until recently, the resources for register identification have focused exclusively on english. the data for this study consist of four similarly annotated online register collections featuring the core corpus series in english @xcite , finnish @xcite , french and swedish @xcite . all the datasets have been extracted from the unrestricted open web. while the english core is extensive, with 34k training examples, the other languages feature merely 2.7-4.6% of that (cf. in this paper, we explore how joint training on the four available core corpora can benefit register classification, with a particular interest in improving performance in smaller languages. 1 first, using multilingually pre-trained language models and a custom sampling and training strategy, we compare performance when training on all languages against previous monolingual results on the same corpora, observing gains for the smaller languages. second, with the aim of creating a universal model fit for all languages, we train a multilingual master model that we evaluate in a zero-shot cross-lingual setting, demonstrating results that land within a relatively short distance from monolingual performances (4-6% f1-score for xlm-r). ## related work until recently, register identification from the unrestricted web has achieved only modest performance @xcite @xcite lang. @xcite . most importantly, the challenges are caused by the range of linguistic variation found on the web. texts are written without gatekeepers, and not all registers are equally well-defined with discrete class boundaries @xcite . to this end, @xcite suggest to extend the analysis to hybrid documents combining characteristics of several register classes, and @xcite examines web genres by prototypical genre classes and text dimensions featuring communicative functions, such as argumentation or reporting. despite the difficulty, @xcite show that multi-and cross-lingual modeling of registers between english and finnish is possible at practical levels of performance, as they propose a convolutional neural network (cnn) model with multilingual word embeddings to model registers. further, @xcite demonstrate that pretrained neural language models, especially xlm-r, can achieve strong performance monolingually on the four aforementioned languages, as well as achieve strong cross-lingual transfer in a zero-shot learning setting from english to other languages. the benefits of combining several languages during training has been demonstrated for other nlp tasks. training the multilingual xlm-roberta (xlm-r), @xcite showed that adding more languages to training leads to better cross-lingual performance on lowresource languages. comparing the performance of multiple multilingual models across a number of tasks and languages, @xcite noted as well that adding target language data to training provides higher performance. however, they highlighted that a model's cross-lingual performance varies greatly between languages and tasks -on qa tasks, zero-shot models are very efficient and outperform models trained on 1,000 examples of target-language data. finally, also the positive effect of sampling under-and overpresented languages has been demonstrated previously; in the context of multilingual semantic parsing, @xcite perform up-and downsampling of languages based on frequency as part of their sampling strategy, in order to improve multilingual performance. ## data the four datasets we use in this study-core, fincore, frecore and swecore-all feature the unrestricted web, however, they have been compiled in different ways. the english core is based on unrestricted search queries of extremely frequent n-grams, while the other datasets are randomly sampled from the 2017 conll shared task datasets, originally drawn from common crawl @xcite . the four datasets have all manual register annotations following the same register taxonomy that was developed during the compilation of the english core. the taxonomy is hierarchical, with eight main registers and approximately 30 subclasses, depending on the language-specific version. in this study, we focus on the main register level, which includes the classes narrative (na), informational description (in), opinion (op), interactive discussion (id), how-to/instruction (hi), informational persuasion (ip), lyrical (ly) and spoken (sp) (for a detailed description, see @xcite ). in order to reflect the variation found within the data, hybrid documents combining characteristics of several registers are also annotated. on the main register level, these display 11-15% of all other language-specific datasets but finnish. perhaps because of the different approaches to gathering the corpora, the register distributions differ also for some other classes between core and the others. specifically, the informational persuasion class covers only 2.75% of core, and 16.82-24.15% of the other datasets, and also the opinion class covers 16.23% of core and 15,23% of fin-core, but only 6.63% of frecore and 6,60% of swecore (for details, see @xcite ). ## experiments we first train models jointly on all four languages following the sampling strategy introduced above, and optimize hyperparameters foot_1 for each target language separately, based on development set performance. the optimal model for each language is tested on the respective test set. we compare the multilingual results to the previous state-of-the-art results in monolingual settings, i.e., where one and the same language is used to train, validate and test the models. second, after optimizing on each language individually, we perform another hyperparameter search for training a single multilingual model that should favor each language equally, which we call a master model. in order to train the master model, we create a common development set based on the individual sets of the languages. the development sets differ in size due to different sizes of the corpora and different data split ratios (see third, in order to estimate the performance that can be expected of the master model on an unseen language, we still perform an experiment where each of our four languages is in turn taken as target, and a model is trained with the previously optimized hyperparameters, using the remaining three languages for training and validation (controlling early stopping). the models are tested in each language separately. the results of this zero-shot cross-lingual experiment are listed in finally, figure 2 summarizes the f1-scores from the aforementioned tables in a side-by-side comparison. we especially observe how the zeroshot multilingual results take the lead over the baseline of zero-shot from english, in order to approach the levels of the monolingual and multilingual models for which target language is also used for training. the levels of inter-annotator agreement, as reported by @xcite , were counted prior to any discussions between the annotators. although this level should be considered as a lower bound of human agreement, it sets a theoretical boundary for automatic register identification. ## error analysis in order to gain a more detailed understanding of the types of errors the models are making, we study the confusion matrices in figure 3 . these present the correct classifications (diagonal) and misclassifications (rest), both in a single language setting using the master multilingual model and in a cross-lingual setting using the zero-shot model. the matrices include the six most frequent classes and a separate hybrid class, as the confusions matrix is not defined for the multi-label setting. we observe that hybrid documents overall are difficult to recognize as such, in particular hybrids composed of narrative (na) and another class are often predicted as na only. comparing the master model (left) and the zero-shot models (right), we see that the overall patterns are quite similar, while the cross-lingual performance, for instance, in english and finnish is worse for how-to/instruction (hi) and interactive discussion (id). in swedish, however, id performs better cross-lingually, and swedish generally exhibits the smallest differences between the settings. informational description (in) and informational persuasion (ip) are difficult to distinguish in english for the master model, whereas the crosslingual model handles these classes much better, although there is still room for improvement. distinguishing purely informational texts and those with an intent to persuade is difficult for other zero-shot models as well. comparing the cross-lingual matrices with those reported by @xcite for transfer from english to the other languages, we note that our diagonals are significantly crisper, i.e., the classes more frequently correctly predicted. in their results, especially the classes hi and ip are generally more dispersed, as well as opinion (op) for french, na for swedish and in for finnish (vertically, i.e., other classes are mistaken for in). finally, comparing class-wise f1-score between the master and zero-shot models we observe a 3.1% mean decrease for na (sd. 1.5%), 5.9% for op (sd. 2.6%) and 7.6% for ip (sd. 5.4%). most of the classes are too infrequent in our data for meaningful interpretation of class-wise differences, or the patterns are inconsistent across languages. ## discussion our results show that multilingual training brings clear advantages to web register identification, in particular for the languages with small amounts of training examples. when allowing training on target data, performance is somewhat improved for these languages, while it remains on par in average. in the zero-shot setting, however, the performance is greatly improved compared to the recent and already strong state-of-the-art results. as illustrated in figure 2 , the multilingual zero-shot xlm-r is closing in on its top-performing counterparts trained monolingually or on all languages. the fact that the multilingual performance on english is lagging behind is expected, as its class distribution differs notably from that of the other languages, and the uniform sampling is designed to allow the model to learn a mean distribution across the languages. in the zero-shot experiments, the english-targeted model will see relatively little data compared to the other models, which likely works to its disadvantage. in the context of pre-trained language models, english monolingual models are also known to be high-performing; similar results on a multilingual model outperforming other monolingual models but not english have been reported by @xcite . to test how the multilingual model performs in a zero-shot setting, we experimented with a leaveone-out version of the multilingual setting, where a model was trained on all except for the target language data on which the model was tested. although the results were, as expected, lower than the monolingual and multilingual results where target language was included in training, the gap is closing quickly. with the baseline methods, the average gap between the cross-lingual models and monolingual models has been 12.76% points f1-score-in our study, it is 3.73% excluding english, 5.9% including english (with xlm-r). with an average f1-score of 73.98% for finnish, french and swedish, we demonstrate that applying this multilingual register classification model in zero-shot settings can be done at very practical levels of performance. this indicates that our multilingual model can be applied without significant loss of accuracy on languages without existing register-annotated corpora, which is an important step toward being able to perform register identification on the truly unrestricted web, also in terms of language. in particular, these performances are competitive considering the difficulty of the task. as discussed above, the inter-annotator agreements of 78% for french and 84% for swedish serve as a potential upper bound in modeling. the monolingual models are already very close to this level, and the multilingual zero-shot models are not far. the competitiveness of multilingual training is particularly interesting in the case of registers. although the advantages of this multilingual training have been noted before (see section 2), it is not evident that register identification can benefit from it. registers are specific to the situation and to the culture where they have been produced. for instance, opinion blogs can express their points of view differently depending on cultural context, and the level of formality of speeches and news reports (subregisters of spoken and narrative) may vary according to the culture. also the linguistic means to express functional characteristics associated with registers, such as narration or interaction, differ across languages. these differences can have a drastic effect on the success of the modeling even if the transfer itself works. in the current study, the included languages are all european, which makes also the transfer easier, whereas including more languages and more distant cultures remains a research desideratum. ## conclusion to sum up, our study corroborates the power of multilingual training when modeling registers in languages with a limited amount of training data. we train and make available a multilingual master model for register classification, whose performance is competitive with existing monolingual models. its zero-shot performance is approaching that of monolingual models, as it improves upon already strong state-of-the-art results. considering the estimated level of human agreement on the task, the margin for further improvement is relatively slim. nevertheless, it is our goal to continue this work in order to achieve robust zeroshot performance in a wide range of languages up to the level of monolingual models. furthermore, it would be interesting to test the robustness and generalizability of our models by evaluating them against the prototypical web genre categories and function text dimensions presented in @xcite . finally, in the future, we will also investigate register-specific differences in their transfer. registers differ in terms of how well they are linguistically defined, which naturally also affects their identification @xcite . for instance, while the linguistic characteristics of many blogs can vary extensively, those of encyclopedia articles remain very similar across texts. this tendency concerns also the cross-lingual similarities of registers, and similarities have already been discovered in particular in the spoken register.