Beyond the E nglish Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registers (2021)

Abstract: We explore cross-lingual transfer of register classification for web documents. Registers, that is, text varieties such as blogs or news are one of the primary predictors of linguistic variation and thus affect the automatic processing of language. We introduce two new register-annotated corpora, FreCORE and SweCORE, for French and Swedish. We demonstrate that deep pre-trained language models perform strongly in these languages and outperform previous state-of-the-art in English and Finnish. Specifically, we show 1) that zero-shot cross-lingual transfer from the large English CORE corpus can match or surpass previously published monolingual models, and 2) that lightweight monolingual classification requiring very little training data can reach or surpass our zero-shot performance. We further analyse classification results finding that certain registers continue to pose challenges in particular for cross-lingual transfer.

## data we use four register-annotated corpora representing the unrestricted open web: the english core and finnish fincore, which have been introduced in previous work @xcite , and two new corpora, frecore for french and swecore for swedish. these novel datasets are released under open licences together with this paper. 1 with these new resources, the possibilities for web register identification expand substantially. frecore and swecore are random samples of the 2017 conll datasets @xcite originally drawn from common crawl. both datasets were deduplicated using onion @xcite with 0.7 threshold and n-gram length of 5. all material not belonging to the body of text, such as boilerplate, was removed. titles, however, were preserved. the cleaning and pre-processing steps follow the procedure suggested in @xcite . the register annotation of the datasets was conducted individually by two trained annotators with a linguistics background. uncertain cases were discussed and resolved together with an annotation supervisor. the inter-annotator agreement, counted prior to the discussions, was 78% f1-score for frecore and 84% for swecore. this can be considered as a lower bound. all datasets are similarly annotated across languages, and they all apply the same hierarchical register class taxonomy originally introduced for core. it includes eight main registers (e.g., narrative) and approximately 30 sub-registers (e.g., news report within narrative). the main and subregister categories are illustrated in the appendix. when a document shares characteristics of several registers, it can be assigned several labels both at the main and sub-register level. these documents are called hybrids. as our focus in this paper is on general register categories, we initially pre-process all four corpora to remove the more specific subregister labels. the general register categories and their distributions as well as the average document length and standard deviation for all classes are presented in ## experimental setup the architectures and models we are using are presented below. 2 we perform multi-label document classification, where each document can have zero, one, or several register labels. the experiments are divided into 1) a monolingual setup with training and evaluation on finnish, french, swedish, and english (as reference), and 2) a zero-shot crosslingual setup with training on english and evaluation on the other languages. bert, bidirectional encoder representations from transformers @xcite is a stateof-the-art deep bidirectional language model pretrained on large unlabelled corpora. bert's architecture is a multi-layer transformer encoder that is based on the original transformer architecture introduced by @xcite . we use cased bert models (tensorflow versions) through the huggingface transformers library @xcite with the following language-specific models: the original english bert, finnish finbert @xcite , french flaubert @xcite and swedish kb-bert @xcite . additionally, we use multilingual bert (mbert) @xcite , which was pretrained on monolingual wikipedia corpora from 104 languages with a shared multilingual vocabulary. xlm-roberta (xlm-r, conneau et al. ( 2020 )) is a multilingual language model that follows the cross-lingual language modeling (xlm) approach @xcite and is based on the roberta model @xcite , which shares the architecture of bert. the authors argue that xlm and mbert are undertuned and that the improved and prolonged training procedure of roberta in combination with more data -on average two orders of magnitude more for low-resource languages -is key to improving cross-lingual performance. xlm-r is trained on 2.5tb of filtered common crawl @xcite ) data comprising of monolingual texts in 100 languages. it is claimed to be the first multilingual model to outperform monolingual models, as well as multilingual bert in a number of experiments @xcite @xcite . we also apply a cnn (convolutional neural network) based architecture following @xcite , as our baseline model. we modify the cross-lingual cnn used by @xcite to a multilabel setting. we use the multilingual word vectors introduced by @xcite . the cnn employs a convolution layer with relu activation, monolingual model train-dev test test f1 (%) std. f1 (%) std. cnn fi 59.04 (0.67) 58.04 (1.02) mbert fi 65.91 (0.85) 64.83 (1.16) xlm-r large fi 76.25 (0.45) 73.18 (1.35) finbert fi 76.28 (1.23) 72.98 (0.74) cnn fr 59.78 (1.10) 58.14 (1.10) mbert fr 70.74 (1.67) 68.66 (0.63) xlm-r large fr 77.38 (0.51) 76.92 (0.24) flaubert large fr 73.93 (0.93) 72.56 (1.40) cnn sv 69.43 (0.56) 67.89 (1.01) mbert sv 76.91 (0.45) 76.43 (0.46) xlm-r large sv 82.61 (0.37) 83.04 (0.62) kb-bert sv 80.15 (0.50) 80.75 (0.09) cnn en 64.56 (0.78) 64.03 (0.30) mbert en 72.80 (0.21) 73.06 (0.09) xlm-r large en 75.80 (0.12) 75.68 (0.05) bert large en 74.01 (0.42) 74.07 (0.28) cross-lingual train-dev test test f1 (%) std. f1 (%) std. en-fi 40.53 (1.11) 41.56 (0.20) en-fi 51.02 (2.92) 50.21 (0.74) en-fi 61.60 (2.01) 61.35 (1.26) en-fr 46.44 (0.51) 46.78 (1.80) en-fr 56.73 (1.54) 55.04 (0.66) en-fr 65.66 (0.52) 64.27 (1.58) en-sv 43.74 (0.82) 43.78 (1.00) en-sv 62.37 (0.82) 62.53 (0.78) en-sv 70.49 (0.58) 69.22 (1.66) the french and swedish data were divided into training, development and test sets using stratified sampling with a 50/20/30 split. for bert-based models we used large model size when available to maximize model performance. we used the maximum sequence length of 512 tokens (with truncation at the end) and batch size of 7, and performed a grid search on learning rate (8e -6 -6e -5 ) and number of training epochs (3-7). for the cnn, we performed a grid search on the kernel size (1-2), learning rate (1e -4 -1e -2 ), and prediction threshold (0.4, 0.5, 0.6). ## results in in monolingual settings, xlm-r large performs competitively compared to monolingual models and clearly outperforms both mbert and the cnn baseline. the lead of xlm-r over monolingual models is substantial in all cases except for the fin-bert model, where the two perform within one standard deviation of each other. our results support the claimed competitiveness of xlm-r large with monolingual models, mentioned in section 3. english, finnish and french bert models achieve similar monolingual test results (73-74% f1-score), while the swedish kb-bert achieves the highest f1-score (81%). the finnish classification task is seemingly easier due to smaller number of classes, nevertheless, other factors may cause the difficulty of the task to differ between languages. for instance, the measured human inter-annotator agreements at 78% (fr) and 84% (sv) f1-score (see section 2) represent a theoretical upper bound for the classification task and reflect the tendency of swedish being easier to classify; the level of agreement has not been reported for finnish. although not strictly comparable, our results clearly outperform the previous state-of-the-art results achieved with the cnn @xcite in terms of f1, which in turn outperforms @xcite , who used the same corpus but in multiclass setting. furthermore, in figure 1 , we illustrate the effect of training monolingual xlm-r large models with varying train set sizes and compare the performance against the reported zero-shot performance. the optimal monolingual hyperparameter settings for each language are used, while training the model instances on 100-900 examples each. we see that zero-shot cross-lingual performance is surpassed already with about 150 training instances for french, 225 for swedish and 400 for finnish, while performance seems to converge around 500. previous studies have shown repeatedly that registers vary considerably in terms of how well they are linguistically defined and thus how well they can be automatically identified @xcite @xcite . for instance, while texts in the in (informational description) and na (narrative) classes, such as encyclopedia articles and sports reports, have very distinctive characteristics and can be identified with a very high reliability, others, such as information blogs in the in class or advice in the op (opinion) class receive much lower scores. figure 2 presents confusion matrices on the predictions in monolingual and cross-lingual settings, using the best-performing model. 3 for the sake of simplicity, the multi-label predictions have been 3 see appendix for class-specific f1 results. the hybrids are also frequently misclassified in cross-lingual settings. interestingly, register classes also feature clear differences in the extent to which the cross-lingual transfer affects the identification performance. the register class in tends to be predicted strongly in all zero-shot language pairs. this is probably due to the in class including documents with strong cross-lingual signals. for instance, in includes encyclopedia articles (see appendix), such as wikipedia texts, that tend to be very similar across languages. while most of the non-hybrid classes experience a small drop in performance, the identification rate for ip and hi (how-to/instructions) drops dramatically in cross-lingual settings in all language pairs. the decrease of ip can be linked to its smaller proportion in the english data (see section 2), but the drops experienced by ip and hi can also reflect the variation displayed by registers across languages. @xcite showed that registers, such as spoken texts, display functional similarities across languages, which obviously is needed for highquality transfer in register identification. however, analyzing the english core registers, @xcite noted that some registers, such as many blogs, depend highly on lexical characteristics reflecting the discussion topics. these topics, however, may vary extensively between languages. this, again, may complicate the transfer learning for these classes. ## discussion and conclusions despite the many opportunities that reliable recognition of text register would introduce for the analysis and use of web documents and many efforts to address this task over the years, only limited progress has been made toward unrestricted web document register classification. previous work has also focused almost exclusively on english. in this study, we have introduced manual register annotation compatible with that of the large english core corpus for two languages previously lacking such a resource, namely french and swedish. we also demonstrated that state-of-the-art multilingual neural language models can support zero-shot transfer of register annotations from english to a germanic, romance and finnic language at levels of performance broadly comparable or better to previously published monolingual results on core. moreover, we demonstrated that small amounts of monolingual training data are needed to reach or surpass this level of performance, which attests that reliable register identification in a new language is readily attainable using current pre-trained language models. we further compared and analysed the results for monolingual and cross-lingual register classifiers, finding that certain registers as well as hybrid texts combining several register characteristics continue to pose challenges in particular for cross-lingual transfer. in future work, we will build on these results to extend multi-and cross-lingual modeling in order to create massive multilingual register-annotated web corpora.