A Multilingual Approach to Identify and Classify Exceptional Measures against COVID -19 (2021)

Abstract: The COVID-19 pandemic has witnessed the implementations of exceptional measures by governments across the world to counteract its impact. This work presents the initial results of an on-going project, EXCEPTIUS, aiming to automatically identify, classify and com- pare exceptional measures against COVID-19 across 32 countries in Europe. To this goal, we created a corpus of legal documents with sentence-level annotations of eight different classes of exceptional measures that are im- plemented across these countries. We evalu- ated multiple multi-label classifiers on a manu- ally annotated corpus at sentence level. The XLM-RoBERTa model achieves highest per- formance on this multilingual multi-label clas- sification task, with a macro-average F1 score of 59.8%.

## introduction the increasing availability of digitized and publicly available legal documents has boosted political scientists, legal scholars, lawyers and policy-makers to apply human language technologies (hlt) to discover, analyze, digest, and use automatically extracted information. all of these operations fall under the larger area of study that can be labeled as legal artificial intelligence, or legalai @xcite . similarly to any domain that wants to apply hlt to process written documents, legalai requires the development of both domain-specific languages resources (e.g., annotated corpora or embedding representations @xcite @xcite @xcite ) and tools (e.g., language specific natural language processing pipelines @xcite or pretrained language models @xcite ). at the same time, legalai presents an additional challenge when it comes to the development of multilingual systems. with very few exceptions (e.g., eu-level legislation), in the domain of legalai, natural languages are strictly connected to countries, meaning that different legislative systems and practices may be in place. for instance, although french-speaking countries share a common language, their legal traditions widely differ making comparisons across legal systems uneasy. nevertheless, the adoption of a multilingual approach may prove valuable especially to legal practitioners and scholars as well as political scientists and policy-makers who are increasingly interested in comparing legal systems and examining how legal concepts "travel" across time and spaces. commonly used data collection and analysis techniques -largely relying on manual or rule-based coding of legal documents -have so far prevented the development of meaningful and systematic analyses allowing the comparison of fine-grained classes. in this paper we investigate the potential of multilingual pretrained language models in order to facilitate the analysis, exploration, and comparison of legal texts on covid-19 exceptional measures. our major contributions can be summarised as follows: ## the exceptional measures against covid-19 in europe the covid-19 pandemic has led governments around the world to take exceptional measures in order to contain the spread of the virus. such exceptional decision-making has seen executives challenge the scope and legality of their powers, as well as impose restrictions on democratic processes, the rule of law, fundamental rights and civil liberties. these exceptional measures considerably vary from one country to another, even in cases where some forms of coordination are claimed to be in place, like in the european union. countries sharing close political culture and institutions reacted in contrasting ways, as attested by the sharp difference between the belgian and dutch responses to the crisis between march and june 2020. while the dutch government implemented one of the softest approaches in europe relying on people's compliance with governmental recommendations, belgium introduced very early a strict lockdown. this diversity is not only practical but also semantic. while many countries relied on a "lockdown" to contain the spread of the virus, restrictions vary in scope while enforcement modalities are unequally coercive @xcite . this fragmented political response sparked interest from researchers in political science, economics, and law that started to trace exceptional decision-making in times of covid-19 @xcite . to the best of our knowledge, all current data collection efforts are based on manual or rule-based methods applied to press releases @xcite @xcite or on experts survey. yet, in the specific case of the covid-19, such methods suffer from three core limitations: ## corpus collection the corpus collection process has been overseen by four political science experts working in partnership with national legal experts. all documents were retrieved from official governmental websites that publish legal acts. the identification of the relevant documents has been done by means of 4 keywords (i.e., "covid", "covid-19", "coronavirus" and "health emergency"). for each language, the corresponding language specific keywords were used. in this initial phase, we focus on a sample of 19 eea countries on measures adopted at the national level. to do so, we identify publicly available links to relevant documents foot_1 plus uk and switzerland. we could not find corresponding documents for two countries of the eea (i.e., bulgaria and greece). all documents have been collected either by manually downloading them or by automatic scraping. 3 for countries with more than one official language (e.g., switzerland), legal acts were collected in all available languages. 4 a total of 6,449 documents has been collected and stored in text format so far. documents form a homogeneous set of existing covid-19 legislation. such legislation however includes a variety of texts adopting by political authorities acting at different levels. beside legal acts -adopted by national parliaments -the corpus also includes executive acts -adopted by governmental authorities which were granted exceptional powers during the covid-19 crisis -and administrative acts which mainly specifies how the implementation modalities of measures adopted by parliaments and governments. the distribution of the documents per country varies greatly: from 9 on the federal level in germany to 969 in slovenia, with 212.5 being the median. these differences are mainly due to a variability across the countries concerning the institutional levels responsible for taking actions against the spread of covid-19: for example, the low number of german documents reflects the fact that the länder were responsible for enacting covid-related measures. we further process all documents using the spacy udpipe 2 nlp pipeline 5 @xcite using models trained on the universal dependencies project @xcite . although the udpipe models may be sub-optimal for processing legal documents, the availability of models and unified representations (i.e., same sets of labels for parts-of-speech tagging and dependency relations) for all the languages of the corpus is an advantage. the spacy udpipe 2 pipeline successfully processed 6,049 documents, providing sentence splitting, tokenization, part-of-speech tagging, and dependency relations. not processed documents are due to issues in the text format due to the conversion process from pdf or other formats. an full overview of the processed data is reported in table a .1 in appendix a. the corpus covers 17 languages, 16 belonging to the indo-european family and one to the uralic family (i.e., hungarian). the length of the documents is a dimension of variation among the countries, dependent mostly on different archival practices across the countries. some countries include only the changes to legal acts, while others combine the original text and the changes. a further peculiarity concerns the levels of detail that accompany the modalities of the measures taken, with some countries being very precise and others addressing the issue in a much broader manner. these dissimilarities can be observed by looking at the average number of sentences per country / per language. countries such as switzerland, latvia, slovenia, and czechia appear to have the longest documents class id class label # subclasses example e1 state of emergency 18 a restriction or requirement imposed under paragraph (1)-(a)by the secretary of state may be varied (orally or in writing) by the secretary of state; e2 restrictions of fundamental rights and civil liberties 5 the conditions or measures which may be specified under paragraph (2)(d) include (b)a restriction on p's activities; e3 restrictions of daily liberties 10 where paragraph (2) applies, the secretary of state or, as the case may be, registered public health consultant may impose on or in relation to p one or more screening requirements. e4 closures / lockdown 15 during the emergency period, no person may participate in a gathering which-(b)takes place indoors, ## annotating exceptional measures the identification of the exceptional measures has been conducted by applying a taxonomy of 8 classes. note that, although the overall project focuses on a large range of subclasses, the size of the corpus and the dispersion of the subclasses in the initial phase of project presented in this paper did not allow to annotate documents at the subclass level. gathering data on implementation modalities is crucial to capture differences in policy styles that may be hidden between the use of the same term to refer to covid-19 policy responses. annotating the measures a subset of 281 documents in eight languages has been selected for manual annotation. the annotation of the exceptional measures applies at sentence-level. the sample is based on the french, polish, dutch, english, hungarian, belgian, italian, and norwegian subcorpora. annotators were allowed to assign as many subclasses as they consider relevant to each sentence, but with a total of eight main classes of exceptional measures. sentences can potentially entail multiple exceptional classes, making this a multi-label annotation task. the annotation process results in eight binary annotations per sentence, with 0 if the specific class is not identified within the sentence and 1 if it is.the annotation 7 the full list of the subclasses is presented in appendix c. for a new (sub)class to be accepted, the measure should have been independently identified by the majority of the annotators. in this phase, no new classes were proposed. ## experiments we have run a set of experiments to develop a tool to support the work of political scientists and other scholars to analyse the large amount of documents in the corpus. in this paper, we present the first development cycle of this tool that targets the automatic identification of the exceptional measures at class level. we opted for this setting mainly due to the low amount of positive instances and the sparseness of the subclass annotation. in particular, we investigate an array of machine learning algorithms distinguishing between feature-based (section 5.1) and model-based (section 5.2). the goal is to identify which approach works best for this task. following the annotation method, the task is framed as a multi-label sentence classification task. given the distribution of the manually annotated data per language and country, we opted to experiment directly using a multi-lingual setting. we report in ## results comparative results for the supervised multi-label classification task for all methods are presented in we further evaluate the best system, xlm(pre@36), for its language adaptation capacity by performing a series of zero-shot experiments. in particular, we fine-tune the xlm-roberta models using the manually annotated training data from all countries/languages except one that is used for testing. with this experiment we want to identify whether our multilingual model is capable of learning cross-lingual concepts that are general enough to successfully detect the measures in new languages. this is also a strategy to check the expected performance of the trained models on the not-annotated documents of the corpus. the results of this experiment (presented in ## related work legalai has a longstanding tradition with early works dating back from the 1960s @xcite and has seen the development of a variety of tasks ranging from the development of domain specific ontologies and lexica @xcite @xcite @xcite , to automatic classification of legislative documents @xcite @xcite @xcite , automatic summarization @xcite @xcite , court judgment predictions @xcite @xcite , legal entities detection and classification @xcite , and question answering systems @xcite @xcite . most of current work is embedded in the paradigm of deep learning, using embedding representations, neural networks or large pre-trained language models. the emergence of deep learning has been accompanied by a growth in specialized embedding representations for the legal domain. similarly to other areas of applications of hlt, the legal domain has seen two waves of embedding methods. the first is has seen the application of static methods (e.g., word2vec, glove, fasttext, doc2vec, a.o.) based on characters, words, or even documents @xcite @xcite @xcite and their integration in neural network architectures. the second wave has seen the development of contextualized representations and the generation of domain-specific transformer based language models @xcite . our work is related to this second wave of embedding representations, in particular, by using a generic multilingual pretrained model such as xlm-roberta. on the contrary, rather than creating domain specific and multilingual language models from scratch, we applied task adaptive pretraining (tap) on the line of @xcite to the generic xlm-roberta as a strategy to boost early domain adaptation. similarly to previous work, we perform an information extraction task framed as classification problem at sentence level. @xcite introduces a sentence classification task aiming at extracting different modalities from financial legislative document in english. modality plays a pivotal role in order to distinguish between what is permitted, prohibited, obliged. they are able to achieve an f1 score of .79 using a bilstm model and combining an ensemble of domain specific and generic word embedding representations based on word2vec. @xcite improves along different dimensions including a more powerful bi-lstm system, a larger dataset, and the use of more fine-grained classes. other works have applied the same task to sentences in german tenancy law @xcite , and us and italian regulations @xcite . other works have focused on the extraction of contract elements @xcite or text classification @xcite @xcite . to the best of our knowledge, we have identified limited previous work on multilingual or cross-lingual applications to the legal domain @xcite . in this respect, in our work the multilingual dimension plays a pivotal role in the development of our approach. given the homogeneity of the topic of the legislative documents taken into account (i.e., covid-19 legislation), the multilingual dimension has been exploited to account for the limited amount of manually annotated documents in each language and country. ## conclusion and future work in this paper, we have presented a new corpus and a taxonomy to identify exceptional measures implemented to counteract the covid-19 emergency across 21 countries. a subset of the corpus (281 documents, 4,145 sentences) has been manually annotated. the data have been successfully applied to develop a fist version of a classifier based on a domain-adapted multi-lingual language model (xlm-roberta) to support experts in investigation of the measures and their impact in the society. besides the relatively small size of the training data, the final score of the system (f1 score 59.8) indicates promising applicability at the final stage of the project. in the future, we plan to develop the project in two directions. first, we will extend the corpus to include additional countries/languages and extend the data to the regional/municipal levels. this will allow us to further adapt xlm-roberta. active learning methods can be adopted to boost the annotation process, leveraging on the fine-tuned models to auto-tag and manually review documents in batches to accelerate the annotation process. second, we plan to develop a fine-grained version of the classifier to include the taxonomy's subclasses. finally, we wish to further study the predictions of the classification system in the unsupervised data and identify potential cross-lingual keywords and/or topics that relate to each exceptional class separately. the average sentence length for croatia is due to the spacy udpipeline failing to correctly split sentences when end of the sentence punctuation marks are missing.