ECHR : Legal Corpus for Argument Mining (2020)

Abstract: In this paper, we publicly release an annotated corpus of 42 decisions of the European Court of Human Rights (ECHR). The corpus is annotated in terms of three types of clauses useful in argument mining: premise, conclusion, and non-argument parts of the text. Furthermore, relationships among the premises and conclusions are mapped. We present baselines for three tasks that lead from unstructured texts to structured arguments. The tasks are argument clause recognition, clause relation prediction, and premise/conclusion recognition. Despite a straightforward application of the bidirectional encoders from Transformers (BERT), we obtained very promising results F1 0.765 on argument recognition, 0.511 on relation prediction, and 0.859/0.628 on premise/conclusion recognition). The results suggest the usefulness of pre-trained language models based on deep neural network architectures in argument mining. Because of the simplicity of the baselines, there is ample space for improvement in future work based on the released corpus.

## introduction texts can be categorized into subjects such as law, philosophy, computing, science, etc., and annotated with information appropriate for the purpose of the research (e.g. argument mining, named entity recognition, etc.). this information is collectively called a corpus and forms a critical component of this research. the corpora are a source of knowledge for creating certain rules and regulations (i.e. a model) which are used in statistical and hypothetical tests. although influenced by many other factors, the performance of statistical approaches still depends primarily on the quality and size of the corpus. therefore during the process of creating a corpus, annotators need to prioritize their ability to maintain quality and quantity via inter-annotator agreement @xcite . while creating corpora, it is desirable to ensure that a system will achieve the highest possible accuracy. nevertheless, creating and constructing corpora is labor-intensive, as it is complex, time-consuming and requires experts who need to be well versed in the corresponding corpus field. for example, in the case of legal corpora, the annotating experts must be lawyers and should also be familiar with the legal arguments. with these constraints, there are a limited number of corpora available in the respective fields. in this paper a corpus of echr decisions is released to the public with a well-structured electronic format foot_0 . the corpus is annotated with a compact argumentation type system (3 types). pointers to the work where the corpus has been already used prior to its release here are provided. the annotation process and the tools used are explained in detail. the resulting corpus is described and several tasks where it can be employed are proposed. the rest of the paper is organized as follows: section 2 presents the related work focusing on various corpora that are publicly available. section 3 introduces the echr's case law. section 4 describes the annotation types, the annotation process, and the tools that were used for annotation. section 5 presents the summary statistics related to the resulting corpus. section 6 presents the baselines for three fundamental argument mining tasks. ## related work a corpus is typically a representation of text, language, image, video, audio, and subject that is annotated with certain signs for a specific purpose. the performance of a predictive system based upon a corpus also depends on its quality and size. depending on the goal, corpora are collected from different domains, such as newspaper, legal, political, scientific and persuasive essays and also in various languages like english, portuguese, german, greek. lippi and torroni (2016) mention that annotating corpora is complex, expensive and requires experts who are well versed in the corresponding field to ensure that annotations are correct. further, the paper also describes the corpora of various domains concerning the argument mining area. @xcite deal with the theoretical aspects of the structure present in legal corpora; they highlight the different critical points humans need to encounter when applying theory to real argumentation and emphasize the association between real arguments and the theories that describe those arguments. several research centers in the world are devoted to developing corpora. one such center that plays an important role in many aspects of argumentation, from theoretical to practical, is arg-tech centre foot_1 . its one of the goals is to develop freely available software tools to aid the researcher in the argumentation field. there are more than 50 corpora from different sectors offered in aifdb @xcite . these corpora are available in different formats, such as svg, png, dot, json, lkif, rtnl, rdf, pl. a popular corpus named araucaria @xcite was developed in the arg-tech centre. the granularity of the corpus was 'claim'and 'premise'. the corpus was collected from 19 newspapers (from the uk, us, india, australia, south africa, germany, china, russia and israel), 4 parliamentary records (in the uk, us and india), 5 court reports (from the uk, us and canada), 6 magazines (uk, us and india), and 14 other online discussion boards and cause sources such as the human rights watch (hurw) and the globalwarming.org. another research lab, ubiquitous knowledge processing (ukp) lab foot_2 , tu darmstadt, germany, is dedicated to developing natural language and machine learning tools and techniques. one of their activities is to develop corpora for argument mining in english and german. @xcite created the argument annotated user-generated web discourse consisting of 90,000 tokens from 340 documents with the datasets prepared using toulmin argument classification (backing, claim, premise, rebuttal, refutation). habernal and gurevych also created the ukpconvarg1 corpus, a user-generated web content corpus that consists of 11,650 argument pairs @xcite and a new crowd-sourced benchmark data-set that contains 9,111 argument pairs labeled with 17 categories (habernal and gurevych, 2016a). @xcite developed the corpus based on public comments on the environmental protection agency (epa) standards rules on hazardous pollutants. two annotators were involved in categorizing 630 documents in terms of 9 categories achieving an agreement of 0.60 on cohen's kappa coefficient. @xcite created a corpus from two datasets: 285 livejournal 4 blog spots and 51 wikipedia discussion forums; the datasets were annotated for the purpose of identifying claims (the ratio of claim vs. not claim is 60:40 in livejournal and 64:36 in wikipedia). @xcite present an argumentative structure dataset consisting of 33 controversial topics; the corpus is derived from 586 wikipedia articles and the authors state that the corpus was constructed (manual annotation) with great attention to detail. @xcite annotated 204 documents in greek related to renewable energy. the documents contain 16,000 sentences and were collected from social media, news, blogs, and microblogs. 760 sentences were annotated with premise and claim at the clause-level. similarly, @xcite choose 300 news items (sports, politics, economics, and culture, etc.) in the greek language to annotate; two post-graduate students were assigned to annotate the corpus and since they were only moderately experienced, guidelines were provided to describe the identification of arguments focusing on discourse markers such as because, in order to and but. each annotator was assigned to annotate 150 documents with argument components, and the final version of the corpus contained 1191 argument components. ## echr's case law case-law documents are written using detailed information from the stakeholders of the court, factual information from the defendant, allegations made by the plaintiff, arguments from both parties, and a decision made by the judge. after collecting all information, it needs to be structured to be useful. to structure the data, and also to know the location of the components of the arguments, it is necessary to analyze and determine the content of each section available via case-law. from the website foot_4 , it is known that case-law is divided into seven categories: judgment, decision, communicated case, legal summary, advisory opinion, report and resolution. of these seven categories, two categories (judgment and decision) are found in the echr corpus. in the echr corpus, there are 20 decisions and 22 judgments foot_5 issued by a chamber of seven judges ruling on the admissibility and merits of the cases. both categories represent similar information, however, the 'decision category' presents the information briefly (the average word length is 3500 words) in the corpus whereas, in the case of judgments, more detailed information is available (an average word length of 10000 words). the decision case-law documents are divided into six sections: i) introduction, ii) the facts, iii) complaints, iv) proceedings before the commission, v) the laws and vi) for the reason. judgment case-law documents are divided into eight sections: i) introduction, ii) procedure, iii) as the facts, iv) the circumstances of the case, v) proceedings before the commission, vi) final submissions to the court, vii) as to the law, and viii) for the reason. the case-law documents begin with introducing the stakeholders of the courts (president, judge, registrar and deputy registrar, lawyers, plaintiff, defendants, and their agents), with their designations, plaintiff, defendant and other members involved in the case. after this, procedure and facts regarding the plaintiff are described. the facts describe an overview of the case that includes information from previous cases, the reason for making an allegation and the chronological sequence of events. the structure of the case-law varies depending upon the exact laws. this information is included as necessary, meaning that the case will vary in length depending upon the case-law and any other essential information that is included. after providing facts of the plaintiff and defendant, the case-law includes the discussions held in the court based upon the allegation made by the plaintiff are presented. likewise, the defendant provides the reason and claim from their perspective and returns a response to the claim made by the plaintiff. after several discussions and arguments presented by both parties, the judge renders his decision. detail information about echr case law documents can be found in @xcite . ## annotation the language intelligence and information retrieval research lab at the ku leuven hired two lawyers to annotate the echr case-law documents @xcite . the annotators were given an argumentation scheme formalism and guidelines that describe the arguments. once the annotation was completed, they were compared and the kappa inter-rater agreement tally was found to be 0.58. a third lawyer was selected to analyze the annotations and found that the main reason for the discrepancies was due to a different demarcation of argument boundaries or, put another way, to the ambiguity that is found in argumentative structure. subsequently, a fourth annotator was selected and was given new guidelines, new sets of comments and recommendations. his annotation achieved 80% agreement which is quite a significant gain. additional information regarding the evaluation of the corpus annotation process can be found in @xcite , @xcite . the corpus is composed of 42 decisions. there are 1951 premises and 743 conclusions (note some argument constituents are premises/conclusions for more than one argument, and some constituents are both premises of one argument and conclusion for another). ## resulting corpus summary statistics about the resulting corpus are reported in ## baselines in this section we present the baselines on the three tasks defined in the past that lead from unstructured texts to structured arguments. the tasks are argument clause recognition, clause relation prediction, and premise/conclusion recognition. as a basis for the classification algorithms we employ the roberta pre-trained language model @xcite which is a variation on the original bert model @xcite with improved pre-training phase. a small layer is placed on top of the model to handle the classification. we work with the base version of roberta to make sure the baselines are simple and easily reproducible. the base roberta model runs perfectly fine on the gpus one can access for free on google colaboratory. 7 in all the experiments 5-fold cross-validation was used. the models were fine-tuned on each of the specific tasks for 15 epochs for each of the folds. in each of the iterations 20% of the data set was used for testing. of the remaining 80% documents 20% were used for validation. after the end of each epoch we saved the current version of the model and recorded its performance on the validation set. for the evaluation on the test set we used the version of the model with the best f 1 score as evaluated on the validation set. ## conclusions in this paper, the echr corpus is described in detail. the procedure undertaken for annotating the components of the arguments in the case-law is described, and an analysis of the quality of the corpus, including its statistical nature and structure is offered. it is a characteristic of legal corpora that many, if not all, case law documents are interconnected to each other through citations. for that, as a future work it is necessary to annotate the related content/information provided by the citation link. for example, in the echr corpus excerpt "the notion of security of person has not been given an independent interpretation (see in this respect selc ¸uk and asker v. turkey, nos. 23184/94 and 23185/94, commission's report of 28 november 1996, § § 185-187)." the first phrase in bold is the conclusion, and the text in italics is the premise. in future work, the annotator would check the report mentioned and annotate that sentence as a premise. this work might help in achieving higher classification performances.