The CLC - UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal (2024)

Abstract: This paper explores the intersection of technological innovation and access to justice by developing a benchmark for predicting case outcomes in the UK Employment Tribunal (UKET). To address the challenge of extensive manual annotation, the study employs a large language model (LLM) for automatic annotation, resulting in the creation of the CLC-UKET dataset. The dataset consists of approximately 19,000 UKET cases and their metadata. Comprehensive legal annotations cover facts, claims, precedent references, statutory references, case outcomes, reasons and jurisdiction codes. Facilitated by the CLC-UKET data, we examine a multi-class case outcome prediction task in the UKET. Human predictions are collected to establish a performance reference for model comparison. Empirical results from baseline models indicate that finetuned transformer models outperform zero-shot and few-shot LLMs on the UKET prediction task. The performance of zero-shot LLMs can be enhanced by integrating task-related information into few-shot examples. We hope that the CLC-UKET dataset, along with human annotations and empirical findings, can serve as a valuable benchmark for employment-related dispute resolution.

## introduction in recent years, there has been great interest in adopting natural language processing techniques in the legal domain. one notable application is the prediction of outcomes for legal disputes in various jurisdictions @xcite @xcite @xcite . however, the ai-based prediction of uk court decisions is still under-explored. this paper investigates the prediction of dispute outcomes in the uk employment tribunal (uket). the uket serves a crucial function in the uk justice system, specifically dealing with employmentrelated disputes. cases heard at the uket cover a wide range of issues, such as unfair dismissal, discrimination and breach of contract. the possibility to apply to the uket for a decision ensures that employment rights can be enforced. knowing the likely outcome of a court procedure improves access to justice and facilitates amicable dispute resolution. the contributions of this paper are as follows: the clc-uket dataset and the empirical explorations aim to supplement the standard clc dataset and facilitate future research on employmentrelated dispute resolution in the uk legal system. we will make the clc-uket dataset available via the official clc website foot_0 . ## uk legal system and uket the uk has a special category of judicial body, the employment tribunals, which deal exclusively with employment disputes. the uket is one of the three largest tribunals in the greater tribunals system @xcite . the uket aims to provide a procedure which is easily accessible, informal, speedy and inexpensive @xcite . the form of employment tribunal proceedings is adversarial rather than investigatory, as each party has to present and prove its case @xcite . claimants must comply with procedural and substantive requirements to be successful. for instance, claimants must submit their claims on time, comply with the orders of the tribunal, present required evidence or information in a timely manner, and avoid scandalous, unreasonable or vexatious conduct (which would make a fair trial impossible). these are usually considered as procedural requirements. claimants must also comply with the substantive requirements of the rules supporting the claims. for example, in order to be successful with a discrimination claim on grounds of disability, the claimant must prove their status as an employee, demonstrate their disability and show that they faced discrimination, which are considered as substantive requirements of the case. these procedural and substantive requirements are not necessarily determined at one final hearing or included in one final judgment. instead, they may be iteratively decided at different stages, which can result in multiple decisions. the employee (claimant) and the employer (respondent) submit their claims and responses, respectively, through a standardised form (rules 8 and 16 of the employment tribunals rules of procedure 2013, hereinafter referred to as rules). the tribunal considers these forms and may dismiss a claim for procedural or substantive reasons, e.g., for lack of jurisdiction or for lack of any reasonable prospect of success @xcite . at any stage of the proceedings, the tribunal can determine a preliminary issue, make a procedural order (e.g., a deposit order or require the presentation of additional documents) or make a final decision (e.g., strike out the claim, rules, r. 37). there may be multiple final hearings for different issues, for example, one hearing to determine whether a party is liable, another hearing to determine the remedy and another to determine the costs (rules, r. 57). each of these hearings results in a separate judgment, written out in a separate document. finally, a party may request a reconsideration of a previous judgment, which will lead to another judgment (rules, r. 70). as a consequence, the resolution of a dispute may not be covered by one judgment only, but may be determined by iterative multiple decisions resulting in various case documents. each decision is linked to one or multiple jurisdiction codes. in the case of the uket, there are 54 jurisdiction codes in total, which are used to identify the matter of disputes. by way of example, the jurisdiction code "unfair dismissal" is used when claimants argue that they have been unfairly dismissed. this jurisdiction code is often employed in addition to other jurisdiction codes, such as unlawful deduction from wages, redundancy, protective award, breach of contract and working time regulations. in stark contrast with typical uk judgments, uket decisions are relatively clearly structured, not only because there are no dissenting opinions, but also because there are specific rulings that set out which elements a judgment must contain (rules, r. 62(5)). nevertheless, uket judgments are not always consistent since there are no formal rules on the style to be used in drafting a decision. most english judgments summarise their decisions in a paragraph, although this summary does not need to respect any particular form @xcite . in the case of the uket, the summary is often found at the beginning of the judgment. however, judgments on multiple claims are sometimes divided into chapters, each analysing one claim containing the relevant decision. also, whilst a judgment may contain an initial statement that the claimant is successful, it may not be clear which claim(s) this relates to in cases where there are multiple claims. ## the clc-uket dataset we curated a large-scale dataset focusing on uk employment-related dispute resolution. the resulting clc-uket dataset consists of two components: clc-uket anno consisting of 19,090 cases with detailed legal annotations and clc-uket pred with 14,582 cases curated for case outcome prediction for the uket. the clc-uket dataset is constructed based on the uket subset of the clc @xcite by adding annotations for selected uket cases. a common practice for collecting legal annotations is to ask legal experts to manually annotate texts. however, this can be costly and time-consuming. to alleviate the burden of manual annotation, we explored utilising large language models (llms) for automatic annotation. the dataset curation pipeline of clc-uket anno consists of two steps: a case preparation module and an llm-aided case annotation module. ## case outcome prediction the annotated clc-uket data (i.e., clc-uket anno ) provides a large collection of court decisions augmented with rich legal annotations, which can readily be used for downstream legal ai tasks. in this paper, we showcase a use case of the clc-uket data by investigating a classic task in legal ai, i.e., case outcome prediction. ## conclusion this paper explores the prediction of dispute outcomes for the uk employment tribunal (uket). it also illustrates the utility of llms for automatic annotation to reduce the burden of extensive manual annotation. with llm-aided annotation, we curated the clc-uket dataset with comprehensive, high-quality legal annotations. we showcased how the clc-uket data can be used to construct a prediction task to categorise case outcomes based on sequences of facts and claims. we fine-tuned and evaluated two widely used transformer-based models on this prediction task. we also evaluated llms on the prediction task with a range of settings, and reported human performance on the task to facilitate model calibration. these empirical efforts serve as a useful benchmark for the uket prediction task. we will make the clc-uket dataset publicly available foot_5 to facilitate future research in this field.