C hronos L ex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks (2024)

Abstract: This study investigates the challenges posed by the dynamic nature of legal multi-label text classification tasks, where legal concepts evolve over time. Existing models often overlook the temporal dimension in their training process, leading to suboptimal performance of those models over time, as they treat training data as a single homogeneous block. To address this, we introduce ChronosLex, an incremental training paradigm that trains models on chronological splits, preserving the temporal order of the data. However, this incremental approach raises concerns about overfitting to recent data, prompting an assessment of mitigation strategies using continual learning and temporal invariant methods. Our experimental results over six legal multi-label text classification datasets reveal that continual learning methods prove effective in preventing overfitting thereby enhancing temporal generalizability, while temporal invariant methods struggle to capture these dynamics of temporal shifts.

## introduction legal classification tasks involve the assignment of legal documents, texts, or cases to specific legal categories, making them essential for legal practitioners, researchers, and organizations seeking efficient legal document management and analysis. however, the dynamic nature of legal domain, characterized by the intricate interplay of laws, precedents, and ever-evolving interpretation of existing jurisprudence, influenced by external forces such as real-world events and shifting societal norms, provides a distinctive challenge for these legal tasks specifically. in this backdrop of a dynamic and non-stationary world, temporal generalization of models emerges as a key necessity, given such systems are deployed to assist users with data that it encounters in the future, whilst being trained on data from the past. most of the works dealing with legal classification often neglect time as a factor and simply split the data randomly into training, validation, and test sets (e.g., @xcite @xcite @xcite @xcite @xcite . recent works have realized that this standard splitting criterion leads to overly optimistic performance estimates, which are far from realistic scenarios when such systems need to be deployed to assist with the new content over time and suffer from degradation of performance due to temporal misalignment i.e., difference in distributions of training and evaluation data instances. they advocate for chronological splitting of datasets to estimate the performance, accounting for the data drift over time (e.g., @xcite @xcite @xcite . furthermore, we posit that using a single fixed chronological test split (eval-fix, fig. 1a ) may yield misleading results if the chosen fixed split coincides with anomalous events, such as brexit, covid-19, lockdown, the russo-ukrainian war, or political regime changes, impacting one side of the splits more than the other. to address this concern, we utilize a streaming evaluation protocol (eval-stream, fig. 1b ) which involves evaluating the model over multiple splits involving distinct time periods which can lead to reliable conclusions about models' robustness. in this protocol, we incorporate all past data into training and assess the model's capacity to adapt to emerging trends in the next time period, which mirrors standard machine learning development pipelines, where models are regularly updated and redeployed to account for temporal drifts @xcite . while legal texts are intrinsically tied to the time of their origin, making it imperative to account for these temporal aspects when developing classification models, these previous methods overlook the chronological context in which legal documents exist during the training process and consider the entire training data spanning across a considerable stretch of time, as a single homogeneous block (fig. 1c ). this oversight hinders model capabilities to adapt to the data drifts that naturally occur over time leading to performance degradation when evaluated over data from the future time @xcite . for instance, shifts in societal attitudes towards fundamental rights like privacy, discrimination, and freedom of expression drive the evolution of legal concepts, broadening or contracting their scope over time. influenced by technological advances and global events, national legal concepts expand to encompass emerging trends, as seen in security related matters to encompass artificial intelligence and cryptocurrency. landmark cases can also introduce new civil/human rights, such as the recognition of environmental rights within the right to life. legislative changes, such as developments in data protection laws keeping pace with technology, further shape concepts, reflecting evolving regulatory needs. overall, data drifts in the legal domain can manifest gradually through the gradual evolution of legal concepts , reflecting societal changes, emerging legal trends, and real-world events or even abruptly through landmark legal decisions, overruling precedents, and legislative changes. in response to these dynamic temporal drifts, we put forth a hypothesis which we subsequently validate: models trained on data from a timeframe closer to the test set tend to yield superior results, under the assumption of the same model and dataset size. motivated by this finding, and with the aim to capitalize on the whole training set -typically associated with better performance with more data, we introduce "chronoslex", an incremental training framework (fig. 1d ). in this approach, the model is systematically trained on data from one time period at a time before progressing to the next chronological split, thus preserving the temporal order of the data the model interacts with during training. our experiments on six multi-label legal classification datasets demonstrate that incremental training paradigm improves temporal generalization of these models on three datasets but also leads to overfitting to recent data on other datasets. to further improve the models' robustness to address the overfitting issue, we assess (i) different families of continual learning algorithms such as regularization-based ewc @xcite , rehearsal-based er @xcite , agem @xcite , and parameter expansion based adapters @xcite , lora @xcite (ii) temporally invariant methods @xcite derived from domain adaptation methods such as groupdro @xcite , deepcoral @xcite , irm @xcite where we treat consecutive windows of time as distinct domains. our experimental results suggest that temporal invariant methods fail to learn these dynamics while continual learning methods effectively prevent models from overfitting to recent data, ultimately enhancing temporal generalization. ## related work temporal drift there has been increasing recent evidence that temporal misalignment results in degradation of model performance due to difference in distribution between the train and test data, caused due to temporal shifts @xcite @xcite . these distribution shifts arise with the passage of time and are a very naturally occurring type of distribution shift due to non-stationarity and evolving world @xcite . temporal misalignment can be caused due to (1) the dynamic nature of language @xcite @xcite @xcite @xcite and (2) the update of factual information @xcite @xcite @xcite . temporal generalization has been explored both in upstream language model pre-training @xcite @xcite @xcite @xcite and in downstream tasks, such as sentiment analysis @xcite @xcite , named entity recognition (rijhwani and preoţiuc-pietro, 2020; @xcite , question answering @xcite , headline generation @xcite , rumor/fake news detection @xcite , spoken language understanding @xcite . model explainability @xcite , document classification @xcite @xcite , abusive language detection @xcite , topic modeling @xcite and readmission prediction in the health care context @xcite . in this work, we assess temporal generalization in multi-label legal text classification, which is challenging given the complexities of managing both the multiplicity of labels and accommodating the evolving legal context over time. this is in contrast to prior studies that predominantly focused on single-label classification. in prior work on temporal generalization in multi-label legal text classification, chalkidis and søgaard 2022 postulated that temporal drift is primarily attributable to shifts in the label distribution over time and explored various group-robust optimization algorithms to mitigate disparities at the label level. we propose an alternative viewpoint, arguing that temporal drift in legal tasks extends beyond label distribution shifts to encompass shifts in the input text as well, driven by the fact that the vocabulary changes with time (tab. 1). furthermore, even vocabulary specific to each label undergoes temporal shifts (tab 1). this complexity was not fully considered in chalkidis and søgaard 2022, where the entire training data was treated as a single unit, overlooking shifts in input text distribution over time. to address this, we introduce an incremental training framework which enables models to adapt to distribution shifts over time. we further improve its robustness by evaluating a range of continual learning and temporal invariant approaches for multi-label legal text classification. continual learning most research in continual learning (or lifelong learning) initially centered around computer vision tasks, and more recently, it has gained attention in the nlp field @xcite @xcite @xcite . the majority of these works adopted traditional task-incremental, domain-incremental, or label-incremental settings. while our temporal incremental setting bears resemblance to the domain-incremental setting, a crucial distinction lies in the assumption of strict boundaries in domain incremental settings, which is not applicable to our temporal adaptation where the boundaries between drifts are blurred @xcite . generally, continual learning algorithms can be categorized into (i) rehearsal-based methods @xcite , which maintain a memory buffer of older data to perform experience replay with actual data (de masson d'autume et al., 2019), automatically generated data @xcite , or previously computed gradients @xcite , (ii) regularization-based approaches @xcite @xcite , which regularizes neural network parameters from drastic updates for new information to preserve the information of older ones, preventing overfitting to the newer data (iii) parameterexpansion methods @xcite @xcite which freeze network architectures on older data and dynamically grow branches for newer ones. these methods have been explored in the pre-training stage to accommodate factual updates @xcite or adaptation to new domains @xcite and in the fine-tuning stage @xcite , which is closely related to our work, but within the context of singlelabel classification tasks. to the best of our knowledge, continual methods have never been explored for multi-label legal classification tasks. temporal invariant domain adaptation domain adaptation (da) primarily addresses the co-variate shift between source and target data distributions @xcite . this involves learning domaininvariant feature representations that can generalize across different domains @xcite . in the legal context, da meth-ods have been applied to tasks such as legal judgement prediction, where reasoning with respect to each article is treated as a domain @xcite and legal text classification tasks, where each label is considered a domain to address label imbalance @xcite . traditional da methods are typically applied to domains with clear distinctions, whereas temporal distribution shifts tend to occur gradually without well-defined boundaries, forming a continuous process. to adapt domain invariant methods to address temporal distribution shifts, we assume instances from consecutive windows of time to constitute a distinct domain, enabling the application of invariant learning approaches to these constructed domains @xcite . to the best of our knowledge, this is the first attempt to assess these methods for multi-label legal texts to handle temporal drift. ## evaluation strategy to assess temporal generalizability, following previous works @xcite , we carry out evaluation using fixed chronological splits, referred to as eval-fix (fig. 1a ). in this setup, we split the entire data chronologically into train (@xmath0, validation (@xmath1, and test (@xmath2, where d t denotes data from time period t, and t 1 < t 2 . we train and validate the model using the respective splits and subsequently report its performance over the entire test set. we posit that such a fixed split may not reflect a true picture as it can be influenced by the interplay between the splits and the data distribution where the anomalous/drift-driven events may overlap or primarily fall within one of the splits, leading to misleading conclusions about the model's ability to handle temporal drifts. to mitigate the impact of dataset-specific traits on the splits, we adopt a streaming evaluation protocol, inspired by @xcite @xcite we refer to as eval-stream (fig. 1b ). we evaluate models using multiple splits, formed at distinct timestamps, considering all the data up to time period t (@xmath3, d t for validation, and d t+1 as the test split. we focus only on the evaluation of the future data rather than the past because the practical need for performance over past data is limited. while it is possible to test the model on data from all subsequent time periods (@xmath4, we restrict the testing to the following time period t + 1, to simplify the result analysis, aligning with practical scenarios where models are updated based on their performance in the next time iteration. ## datasets we conduct our experiments on the six multi-label legal classification datasets from three sources. uklex @xcite this dataset comprises legislative documents of the uk, publicly accessible through the national archives, which are typically categorized into thematic areas such as healthcare, finance, education, transportation and planning, which are outlined in the document preamble and serve as indexes for archival purposes. the dataset contains 36.5k documents and is chronologically split into training @xcite , and test (8.5k, 2009-2018) sets. the labels are provided at two distinct levels of granularity, encompassing 18 and 69 topics, referred to as small (s) and medium (m) respectively. for eval-fix, we use the above splits and for eval-stream, we consider a two-year time period as a unit. due to limited data prior to 1990 and to ensure an adequate number of data instances for model learning, we report eval-stream performance from 1992. eurlex @xcite this dataset contains eu legislation documents accessible via the eur-lex platform and is annotated using concepts from eurovoc, a thesaurus maintained by the eu's publications office. we work with the english portion of this dataset, consisting of 65k documents and split chronologically into training @xcite , and test (5k, 2012-2015) sets which we use for eval-fix. the dataset provides four levels of label granularity. we use the first two levels of the eurovoc taxonomy, as followed in chalkidis and søgaard 2022, encompassing 21 and 127 concepts, denoted as small (s) and medium (m), respectively. for eval-stream, we use two years as one unit and report performance from 1987 due to a lower number of instances in the initial years. echr the dataset by @xcite comprises cases heard by the european court of human rights, which are publicly accessible via hudoc, the official court database. these cases involve the adjudication of complaints by individuals against states for alleged violations of their rights as enshrined in the european convention of human rights. each case includes in-formation about the convention articles that have been alleged to be violated and which the court has found to be violated. the dentification of alleged and violated articles are referred to as task b and a, respectively, by @xcite . it consists of 11k cases, divided chronologically into training @xcite , and test (1k, 2017-2019) sets for eval-fix. we use the 14 and 17 articles related to the core rights as followed in @xcite for task a and b respectively. we use annual data as a unit for eval-stream and report the performance from 2004. while task employed in this work are all effectively text classification, the tasks surveyed here are different in nature. the echr tasks represent legal determinations by judges in specific cases of high complexity and semantic depth, whereas the keyword classification tasks of eurlex and uk-lex represent semantically much shallower analysis. this has consequences for the types and complexity of temporal shifts one would observe. for example, the court changing its jurisprudence on a particular legal issue prompted by world events is a more complex phenomenon than the usage of particular european law database keyword. ## chronoslex previous approaches for legal classification typically fine-tune models on the entire training dataset in a shuffled manner, treating all data as a single homogeneous block. we argue that such an approach neglects the temporal nature of the data which may be crucial to learn the temporal evolution of concepts. to address this limitation, we propose the chronoslex framework, which considers the chronological context of data to capture and adapt models to the natural drifts over time. we hypothesize that recent data holds insights into upcoming distributional shifts and training models with this recent data enables better adaptation to temporal changes. thus, we systematically train the model with data from one time period at a time before progressing to the next chronological split. specifically, at time t, the model m t is initialized with the model obtained from the previous timestamp m t-1 and fine-tuned on data d t from time period t, progressively moving to the next time period t + 1. we term this approach incremental fine-tuning (ift) (fig. 1d ), where the model architecture and the loss function remain similar to traditional fine-tuning, but the model encounters chronological training data incrementally. however, ift may risk overfitting to recent data, potentially forgetting previously acquired knowledge which is crucial for extrapolating to the new distribution based on past data. therefore, a balance between preserving knowledge from the past and accumulating information from recent data is essential to enhance temporal generalization. to explore this balance, we investigate continual learning methods and temporal invariant methods, to adapt to temporally emerging distribution shifts. ## experiments 6.1 base model & metrics for uklex and eurlex, we use a state-of-theart model, bert-lwan @xcite , which has a label-wise attention network on top of the pre-trained model. specifically, we pass the text into legalbert @xcite and employ one attention head per label to generate n document representations where n denotes the number of labels, which are finally passed through a linear layer to get the final predictions. for echr, we employ a state-of-the-art hierarchical version of bert due to long input documents @xcite . specifically, we pass each sentence in the long document into legalbert to obtain @xcite representations which are then passed through a two-layer transformer to obtain final representations. implementation details of all the methods can be found in appendix a. following chalkidis and søgaard 2022, we report, micro-f1, macro-f1, and mean r-precision (m-rp). ## conclusion in this study, we delve into temporal drift in legal multi-label text classification tasks, revealing a crucial oversight in the current model training process that treats the training data as a single, homogeneous block. this neglect results in the degradation of performance over time. to remedy this, we introduce chronoslex, an incremental training paradigm that allows the model to interact with data chronologically, facilitating adaptation to temporal shifts. however, we observe a potential overfitting to recent data with this incremental approach. to address overfitting, we explore mitigation strategies using both continual learning and temporal invariant approaches and found that continual algorithms exhibit promising results by leveraging historical distribution to extrapolate trends into the future. in contrast, temporal invariant methods prove less effective in tackling this. finally, we advocate for streaming evaluation protocol with multiple time splits to draw reliable conclusions, especially when time is involved as a critical axis. in future, we plan to evaluate this incremental strategy across other legal tasks such as prior case retrieval, contract analysis, legal document summarization, regulatory compliance and legal argumentation, where the temporal aspect is crucial in modeling.