Tracking Semantic Shifts in G erman Court Decisions with Diachronic Word Embeddings (2022)

Abstract: Language and its usage change over time. While legal language is arguably more stable than everyday language, it is still subject to change. Sometimes it changes gradually and slowly, sometimes almost instantaneously, for example through legislative changes. This paper presents an application of diachronic word embeddings to track changes in the usage of language by German courts triggered by changing legislation, based on a corpus of more than 200,000 documents. The results show the swift and lasting effect that changes in legislation can have on the usage of language by courts and suggest that using time-restricted word embedding models could be beneficial for downstream NLP tasks.

## introduction languages change over time on different levels, from phonetic and spelling changes to lexical changes, semantic changes, and syntactic changes. semantic shifts, i.e. changes to the meaning of words, have been researched for hundreds of years, and different taxonomies exist for their classification, e.g. by @xcite . these changes are often happening slowly over the course of many years, like the word "dog", which used to refer to a specific breed and now refers to all breeds @xcite , or the word "broadcast" that in the early 20th century meant "casting out seeds" and now refers to transmitting a signal @xcite . arguably, the stability of language is higher and more important in legal documents than in most other contexts. many legal terms, but also terms that are used more freely in everyday language, are well-defined in the context of legal proceedings, either by laws or years of legal precedent. however, language and its meaning and interpretation also change in the context of legal practice. very prominent examples of that can be found in constitutions. in many western democracies, constitutions are very stable documents whose texts are hardly changed in decades or sometimes even centuries. yet, as societies and culture change, the interpretation of these documents by politicians and judges changes as well. since its implementation in 1949, the first sentence of the second paragraph of article 3 of the german constitution reads "männer und frauen sind gleichberechtigt" (men and women have equal rights). however, it was not until 1977, that a wife would not need the approval of her husband anymore to get a job. a practice that, beyond any doubt, would be ruled unconstitutional today based on the very same sentence. the reasons why the use of language in legal practice can change are manifold, just like in everyday language. a reason that is particular to the legal domain are changes in legislation that can lead to semantic shifts that are unseen in other domains in speed and thoroughness. by introducing a new law or changing an existing law, legislators have the power to almost instantaneously change the meaning of a word in legal practice. an example of such a quick shift is the german word "rundfunkbeitrag". before 2013, a "rundfunkbeitrag" was a tv or radio report. however, in january 2013, the "rundfunkbeitragsstaatsvertrag"(broadcast fee state contract) renamed the german public broadcasting license fee from "rundfunkgebühr" to "rundfunkbeitrag", giving the word a new meaning that quickly has become predominant in legal proceedings (see section 5). in everyday language, however, the new "rundfunkbeitrag" is still regularly referred to by its old name. in recent years, diachronic distributional models, and especially diachronic word embeddings, have been successfully used in different contexts to track semantic changes and changes in language use (see section 2). the basic idea behind this approach is to train separate word embedding models based on documents from different time periods and then analyse how the word vectors for chosen terms change between the different models (and hence over time) in relation to other terms @xcite . this paper presents an application of diachronic word embeddings to the legal domain and specifically focuses on the analysis of semantic shifts introduced by legislative changes. we trained multiple word embedding models on different temporal subsets of a corpus of more than 200,000 decisions from german courts provided by open legal data @xcite , containing texts from the 1970s to 2020. the results show that diachronic word embeddings can capture immediate and permanent changes in the language used by courts after relevant legislation comes into force. these significant semantic shifts indicate that it could be beneficial for prediction and classification tasks that are connected to words that have undergone such a shift, to use word embeddings that are temporally aligned with this shift, even if that reduces the overall available data. ## related work in 2018, @xcite a comprehensive survey on works on diachronic word embeddings, including the work of @xcite , @xcite , @xcite , @xcite , and many others. therefore, we will focus on works published after 2018. @xcite found that, as is the case often, most of the existing work focuses on the english language. one notable recent exception from that is the work by @xcite , in which the authors analysed a corpus of german parliamentary proceedings spanning from 1867 to 2020. they were, for example, able to show an increase in antisemitic rhetoric in the years leading to the seizure of power by the national socialists. the data sources that have been used to train diachronic word embeddings in recent years are diverse. @xcite , for example, used a corpus of websites from the uk called dukweb, @xcite used two newspaper corpora in english and german, and many researchers use the google books corpus @xcite @xcite . while multiple works have been published training diachronic word embeddings on political data and debates, including the previously mentioned year(s) # sentences avg. sentences/doc 1970-1979 3,014 26.2 1980-1989 22,102 27.4 1990-1999 170,082 26.4 2000-2009 1,991,396 31.6 2010-2019 4,957,720 38.5 2020 158,705 59.1 1970-2020 7,303,019 36.2 ## data set this work is based on the corpus provided by open legal data @xcite . it consists of over 200,000 german court rulings, published between april 2022 and april 1970, by different german courts, ranging from the "bundesverfassungsgericht" (federal constitutional court) to "amtsgerichte" (district courts). for reasons of reproducibility, we used the latest stable dump from december 2020 foot_0 , which contains data from april 1970 to december 2020. the dump consists of 201,824 documents from 626 courts. figure 1 shows the temporal distribution of the documents and the number of courts that contributed documents in each year. over time, more and more courts started to publish their decisions digitally and courts also increased the number of decisions they published. therefore, most of the documents in the data set are from recent years. and not only are there more decisions available from more recent years, but the average length of the documents also increased, as is shown in such a temporal imbalance is very common in legal corpora and the results of this paper suggest, that diachronic word embeddings could help to mitigate the bias that is introduced by such an imbalanced corpus. however, since the available data before 2000 is very limited, with only three to 61 documents per year, we will focus on semantic shifts that happened after 2000. ## approach in order to be able to identify the swift semantic shifts we expect to see based on changing legislation, we trained 51 word embedding models, one for each year from 1970 to 2020. that differentiates our approach from works like @xcite and @xcite , which focus on long-term shifts and therefore aggregate their data over decades. for reasons of comparability and in order to observe more long-term trends, we also trained five models, each of which is spanning a decade from the 1970s to the 2010s. additionally, we also trained two larger models, one including all data before 2000 and one including all data from 2000 onwards. lastly, we also trained a model based on all available data, i.e. data from the years 1970 to 2020, resulting in a total of 59 models. ## results first, we will look at three examples of rapid semantic shifts that have been caused by changes in legislation (see section 5.1). we selected three legislative changes which came into force between 2001 and 2013, in order to focus on time periods with sufficient data available. afterwards, we will look at slower, more traditional patterns of semantic shift in which words change the context they are used in, based on broader societal changes (see section 5.2). finally, we will also take a brief look at changes on the level of the vocabulary (see section 5.3). ## discussion the results of our analysis show that changes in legislation can cause almost instantaneous semantic changes in the language used by courts and that diachronic word embeddings can be used to track these semantic shifts. for all instances discussed in the paper, the model that was trained on the complete corpus with data from 1970 to 2020 represented the same meaning as the post-2000 model, which was to be expected, given the temporal imbalance of the data. if we would want to work on historical decisions, or if a semantic shift would have happened only in 2020, the model trained on all data would most likely misinterpret the words for which a semantic shift has happened. that suggests that using diachronic word embeddings within downstream nlp tasks, like classification or outcome prediction, could be useful in cases where it is known that a word that is important in the context of the task has changed its meaning, e.g. through a change of the law or other significant events. in such cases, aligning the data with such events could help to improve performance, even if it means a reduced corpus for training. ## conclusion the paper presents an application of diachronic word embeddings to a data set of more than 200,000 german court rulings. 59 different embedding models have been trained, spanning different time spans from years to decades, in order to observe semantic shifts introduced by changes in legislation. the results show that these semantic shifts happen quickly and have a lasting influence on the language that is used by courts. for tasks, like classification or outcome prediction, that are directly connected to legal terms that have undergone such a semantic shift, it could therefore be beneficial to train word embeddings on a time-restricted dataset, to ensure correct interpretation of the terms in questions, even if that means reducing the available data for training. in the future, it would be desirable to conduct a similar experiment with a temporally more balanced data set. another interesting direction for future research would be to connect our findings on the usage of language by courts with the existing literature on semantic shifts in political debates and general language. one could hypothesise that changes in the language used by courts could be predicted by changes in the language used in political debates, which might precede them, and which in turn might be preceded by changes in the general use of language. analysing whether such influences can be seen in diachronic word embeddings could help to develop models to predict when changed language use will start to have an influence on politics and law.