M ulti EURLEX - A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer (2021)

Abstract: We introduce MULTI-EURLEX, a new multilingual dataset for topic classification of legal documents. The dataset comprises 65k European Union (EU) laws, officially translated in 23 languages, annotated with multiple labels from the EUROVOC taxonomy. We highlight the effect of temporal concept drift and the importance of chronological, instead of random splits. We use the dataset as a testbed for zero-shot cross-lingual transfer, where we exploit annotated training documents in one language (source) to classify documents in another language (target). We find that fine-tuning a multilingually pretrained model (XLM-ROBERTA, MT5) in a single source language leads to catastrophic forgetting of multilingual knowledge and, consequently, poor zero-shot transfer to other languages. Adaptation strategies, namely partial fine-tuning, adapters, BITFIT, LNFIT, originally proposed to accelerate fine-tuning for new end-tasks, help retain multilingual knowledge from pretraining, substantially improving zero-shot cross-lingual transfer, but their impact also depends on the pretrained model used and the size of the label set.

## introduction multilingual learning is an active field of research in nlp. starting from neural machine translation (stahlberg, 2020), multilingual neural models are increasingly being considered across nlp tasks and multilingual benchmark datasets for cross-lingual language understanding are becoming available @xcite , complementing previous monolingual benchmarks @xcite . the initial paradigm of multilingual word embeddings @xcite was rapidly expanded to pretrained multilingual models @xcite , including work on zero-shot cross-lingual transfer @xcite . multilingual models based on transformers @xcite , jointly pretrained on large 1 ) from 7 families (illustrated per eu country in the map). the uk was an eu member until 2020. the map should not be taken to imply that no other languages are spoken in eu countries. corpora across multiple languages, have significantly advanced the state-of-the-art in cross-lingual tasks @xcite . in another interesting direction, legal nlp @xcite ) is an emerging field targeting tasks such as legal judgment prediction @xcite , legal topic classification @xcite , legal question answering @xcite , contract understanding @xcite , to name a few. generic pretrained language models for legal text in particular have also been introduced @xcite . but despite rapid growth, cross-lingual transfer has not yet been explored in legal nlp. to facilitate research on cross-lingual transfer for text classification and legal topic classification in particular, we introduce a new multilingual dataset, multi-eurlex, which includes 65k european union (eu) laws, officially translated in the 23 eu official languages (fig. 1 ). each document is annotated with multiple labels from eu-rovoc, where concepts are organized hierarchically (fig. 2 ). 1 we use the dataset as a testbed for zero-shot cross-lingual transfer in cases where we wish to exploit labeled training documents in one language (source) to classify documents in another language (target). this would allow, e.g., classifiers trained in resource-rich languages to be reused in languages with fewer or no training instances. we experiment with monolingual and multilingual transformer-based models, i.e., monolingual bert models @xcite , xlm-roberta @xcite , and mt5 @xcite . we find that fine-tuning a multilingual model in a single source language leads to catastrophic forgetting of multilingual knowledge and, consequently, poor zero-shot transfer to target languages. we show that adaptation strategies, namely, not fine-tuning some layers, adapters @xcite , @xcite , and lnfit inspired by @xcite , originally proposed to accelerate fine-tuning for new end-tasks, help retain multilingual knowledge from pretraining, substantially improving zero-shot cross-lingual transfer, but their impact also depends on the particular pretrained model used and the size of the label set. we also compare chronological vs. random splits, highlighting the impact of temporal concept drift in legal topic classification, which causes random splits to over-estimate performance @xcite . our main contributions are: legal topic classification has been studied for eu legislation @xcite in a monolingual setting (english). while there are several legal nlp studies with non-english datasets @xcite @xcite @xcite , cross-lingual transfer has not been studied in the legal domain. cross-lingual transfer is a very active area of wider nlp research, currently dominated by large multilingually pretrained models @xcite @xcite . recent work explores adapter modules @xcite to transfer monolingually pretrained @xcite or multilingually pretrained @xcite ) models to new (target) languages. we examine more adaptation strategies, apart from adapter modules, in truly zero-shot cross-lingual transfer. unlike pfeiffer et al. (2020), we do not train language-specific adapters per target language; we use adapters to fine-tune a single multilingual model on the source language, which is then used in all target languages. in the broader field of multilingual legal studies, @xcite examined legal topic classification with a dataset comprising 2.7k eu laws in 4 languages (english, german, spanish, portuguese). they experimented with monolingual svm classifiers and their combination as a multilingual ensemble. more recently, @xcite transferred sentence-level gold labels from annotated english to non-annotated german sentences, for the task of identifying unfair clauses in terms of service (2.7k sentences) and privacy policy documents (1.8k). they experimented with similarity-based methods aligning the english sentences to machine-translations of the german sentences. we experiment with state-of-the-art multilingual transformer-based models considering many more languages (23) and a much larger dataset (65k eu laws). although multi-eurlex is largely parallel, we use it as a testbed for zeroshot cross-lingual transfer, without requiring parallel training data or machine translation systems. ## the multi-eurlex dataset 2 documents: multi-eurlex comprises 65k eu laws in 23 official eu languages ( multi-granular labeling: eurovoc has eight levels of concepts (fig. 2 illustrates three). each document is assigned one or more concepts (labels). if a document is assigned a concept, the ancestors and descendants of that concept are typically not assigned to the same document. the documents were originally annotated with concepts from levels 3 to 8. we created three alternative sets of labels per document, by replacing each assigned concept by its ancestor from level 1, 2, or 3, respectively. thus, we provide four sets of gold labels per document, one for each of the first three levels of the hierarchy, plus the original sparse label assignment. 5 supported tasks: similarly to eurlex57k @xcite , multi-eurlex can be used for legal topic classification, a multi-label classification task where legal documents need to be assigned concepts (in our case, from eurovoc) reflecting their topics. unlike eurlex57k, however, multi-eurlex supports labels from three different granularities (eurovoc levels). more importantly, apart from monolingual (one-to-one) experiments, it can be used to study cross-lingual transfer scenarios, including one-to-many (systems trained in one language and used in other languages with no training data), and many-to-one or many-tomany (systems jointly trained in multiple languages and used in one or more other languages). data split and concept drift: multi-eurlex is chronologically split in training @xcite , development @xcite , test (5k, 2012-2016) subsets, using the english documents. the test subset contains the same 5k documents in all 23 languages ( to verify that the chronological split of multi-eurlex in training, development, test subsets leads to a temporal concept drift, we compare the kl-divergence between the label distributions of the subsets using the chronological vs. a random split. ## experimental setup configuration of models and training details: we implemented all methods in tensorflow 2, obtaining pretrained models from the hugging face library. we release our code and data for reproducibility. 9 all models follow the base configuration with 12 stacked transformer encoder blocks, each with d @xmath0. we use the adam optimizer (kingma and ba, 2015) across all experiments. we grid-search to tune the learning rate per method, considering classification performance on development data. 10 evaluation: given the large number and skewed distribution of labels, retrieval measures have been favored in large-scale multi-label text classification literature @xcite . following chalkidis et al. (2019, 2020a ), we report mean r-precision (mrp) @xcite . that is, for each document, the model ranks the labels it selects by decreasing confidence, and we compute precision@k, where k is the document's number of gold labels; we then average over documents. for all experiments, we use the chronological data split and report the average across three runs. unless stated otherwise, we use level 3 with @xmath1, which has a highly skewed (long-tail) label distribution and temporal concept drift ( ## experiments and discussion for the main experiments, we mainly use xlm-roberta in a one-to-many setting (fine-tuning in english, testing in all languages). we also report key mt5 results for completeness. as a ceiling for cross-lingual transfer, in section 6.1 we first evaluate monolingual (native) bert models and xlm-roberta, both in a one-to-one manner (finetuning and testing in the same language), which requires annotated training data in the target language. for completeness, in section 6.3, we also report many-to-many results, where xlm-roberta is jointly fine-tuned and tested in all languages.