EUROPA : A Legal Multilingual Keyphrase Generation Dataset (2024)

Abstract: Keyphrase generation has primarily been explored within the context of academic research articles, with a particular focus on scientific domains and the English language. In this work, we present EUROPA, a novel dataset for multilingual keyphrase generation in the legal domain. It is derived from legal judgments from the Court of Justice of the European Union (EU), and contains instances in all 24 EU official languages. We run multilingual models on our corpus and analyze the results, showing room for improvement on a domain-specific multilingual corpus such as the one we present.

## introduction keyphrases are short phrases that describe a text, and have been shown to be useful for many applications, from document indexing @xcite to opinion mining @xcite . while heavily researched for the stem (science, technology, engineering, and mathematics) domains, there has been little investigation on its application to law. this is surprising, as keywords can reduce the workload of legal experts by allowing them to get the gist of lengthy documents @xcite @xcite . the scope of this work is to assess to what extent keyphrases can be automatically generated in the legal domain. our contributions are the following: (1) we collected and curated europa, the first open benchmark for legal kpg (keyphrase generation), spanning 24 languages and extracted from real-world european judgments. 1 (2) we provide in-depth analysis of our corpus, highlighting its differences compared to other existing corpora. ## related work while english kpg has been studied fairly extensively in technical and academic domains, our work bridges two less common fields : legal kpg and multilingual kpg. in this section, we first review the literature surrounding keyphrase extraction (kpe) and generation before covering legal kpe and kpg. for a more complete survey on kpg, we refer to @xcite . ## the creation of the europa dataset cases in the court of justice of the european union (cjeu), which aims at ensuring the consistent interpretation and application of the eu law across all eu institutions (court of justice of the european union, 2023), can be processed in any of the 24 eu official languages, depending on the member state involved, making drafting and translation pivotal and complex tasks. stakeholders and judges communicate in their respective languages and rely on lawyer-linguists for document exchange. once a judgment is rendered, it is translated into other eu languages to ensure consistency in the judgment text, keyphrases, and their legal implications @xcite . through private correspondence, the cjeu explained that keyphrases, drafted by the registry and completed by the reporting judge's cabinet or the advocate general, aim at providing concise case descriptions. their meticulous construction establishes them as high-quality gold references for kpg evaluation. ## dataset analysis as shown in the distribution of keyphrases in our dataset varies across languages due to the fact that the most recent judgments tend to have a higher number of keyphrases attached to them (the average number of keyphrases per language can be found in appendix f). consequently, instances in languages from the most recent member states tend to be biased towards having more keyphrases. another consequence of the temporal evolution of the average number of keyphrases coupled with the chronological split, is that europa's validation and test sets contain more keyphrases per instance on average (8.3 and 10.5 respectively) compared to its training set (5.4 keyphrases on average). this creates a discrepancy between the sets that practitioners should be aware of. however, we advocate that a higher number of keyphrases in the test set of europa will be beneficial to the evaluation of the competing models by providing a larger number of potential gold keyphrases, and by assessing the capacity of models to generalize across time when target keyphrases follow different patterns with respect to the past. ## models as the majority of europa's target keyphrases are absent from input documents, we focus on generative models, as extractive ones such as @xcite are ill-suited here. recent state-of-the-art models in english kpg rely heavily on pretrained models, such as keybart @xcite or scibart @xcite , which are pre-trained and fine-tuned on a massive corpus of english scientific documents. however, such models are not available in a multilingual format, nor are they specific to the legal domain. we therefore choose to use mbart50 @xcite , a variant of mbart @xcite with support for 50 languages instead of 25. we also test mt5 @xcite , which covers 101 languages. most models, including mbart50, have a maximum input sequence length typically set to 1024 tokens. while mt5's input length can be set arbitrarily, it was originally pretrained with a context of 1024. the main caveat is mt5's memory complex- ## analysis and discussion the first striking observation is that mbart models, despite covering less languages compared to mt5s, succeed in outperforming the latter models. one explanation is that mt5 small, base and large variants generated on average significantly fewer phrases per instance (2.1, 2.3 and 2.3, respectively) with respect to mbart50 and mbart50-8k (5.5 and 6.0). consequently, mt5 models are less likely to achieve high scores. the other noticeable result is that the mbart models even succeed in outperforming mt5s in languages that only mt5s support such as bulgarian or greek. this suggests that @xcite 's corpus used for pretraining mbart50 gave it a capacity to deal with much more languages than stated by @xcite . the improvement from mbart50 to mbart50-8k is significant and emphasizes the importance of larger input length. less that 14% of all instances fit into a 1024-tokens context window. around 58% do when that length reaches 8k tokens. building models that can efficiently generate keyphrases from larger documents is therefore crucial in order to achieve further progress (the length reaches 9160 tokens on average, and 17k at the 90th percentile, with mbart50 tokenizer). 6 moreover, kpg models struggle for keyphrases with more tokens (split by whitespaces). in the case of mbart50-8k, a matched target keyphrase contains on average 3.1 tokens, while an unmatched in order to mitigate this issue, we evaluated the models again with a semantic matching metric @xcite allowing us to compare predictions and targets without having to apply any sort of post-processing or stemming (the tool is however only available for english). results in ## conclusion in this paper, we present europa, a novel and open multilingual keyphrase generation dataset in the le- our corpus also highlights the need to efficiently capture larger input context, and will be a suitable testbed for models designed to do so.