Generating Intelligible Plumitifs Descriptions: Use Case Application with Ethical Considerations (2020)

Abstract: Plumitifs (dockets) were initially a tool for law clerks. Nowadays, they are used as summaries presenting all the steps of a judicial case. Information concerning parties’ identity, jurisdiction in charge of administering the case, and some information relating to the nature and the course of the preceding are available through plumitifs. They are publicly accessible but barely understandable; they are written using abbreviations and referring to provisions from the Criminal Code of Canada, which makes them hard to reason about. In this paper, we propose a simple yet efficient multi-source language generation architecture that leverages both the plumitif and the Criminal Code’s content to generate intelligible plumitifs descriptions. It goes without saying that ethical considerations rise with these sensitive documents made readable and available at scale, legitimate concerns that we address in this paper. This is, to the best of our knowledge, the first application of plumitifs descriptions generation made available for French speakers along with an ethical discussion about the topic.

## introduction the right to access judicial information is a fundamental component of canadian democracy and its judicial process (vancouver sun (re), 2004; cbc. v canada (a.g), 2011) 1 . this right has two main purposes. first, to enhance judicial accountability by providing opportunities to the public to scrutinize it and put forward criticisms of the judicial process (sierra club of canada v canada (minister of finance), 2002; cbc. v new brunswick (a.g), @xcite ). second, it has an educational purpose: by accessing judicial information, people acquire a better understanding of the court process (edmonton journal v alberta (a.g), @xcite ). given these purposes, the necessity to provide access to judicial information in an intelligible form cannot be ignored. indeed, getting a copy of a document is not enough; people have to understand its contents. this is particularly crucial in a digital context since citizens face an overload of judicial information online @xcite . as a consequence, litigants have great difficulty in finding relevant information for their case online @xcite . studies show that, in the province of quebec, the plumitif (a public register where one can find an official trace of all the actions taken by the courts) lacks intelligibility @xcite . some users have called it "non-sense" for non-attorneys @xcite ). yet, the plumitif is necessary for every litigant as it provides information concerning the parties' identity, the jurisdiction responsible for administering cases, and information relating to the nature and the course of proceedings. in this work, we aim at leveraging both information extraction and natural language generation to increase the intelligibility of excerpts of the court of quebec's plumitif regarding criminal offenses under the criminal code of canada (ccc). improving the comprehension of textual legal documents has been the subject of several studies in the past. for example, patent claims are long legal pieces of texts that contain complex sentences making it hard for a layperson to reason about. @xcite framed this problem into an automatic text simplification task while @xcite and @xcite proposed extractive summarization techniques to make them easier to understand. the plumitifs, while also lying in the "legal texts" family, take a completely different form; they are not written in a valid grammatical form, and contain many abbreviations and references to the ccc. this makes our use case application rather unique. to handle this type of document, we have de-signed a description generation pipeline, divided into three steps. the first step consists of segmenting a plumitif into different parts. in the second step, we extract, for each part, the relevant information using a named entity recognition (ner) model. for the final step, we generate sentences from the data extracted by the ner model. to this end, we use a template-filling approach to ensure there are no factual fallacies introduced in the generation, an essential concern in legal text generation. moreover, we use a statistical language model in a controlled setting to augment the generation with vital contextual information, namely texts from the ccc, making our approach a hybrid generation model. our contributions, in this work, are twofold; 1. we propose a simple yet robust data-to-text multi-source textual generation pipeline to make plumitifs easier to understand for the litigants (made available through a web application, see appendix i); 2. we bring a discussion on the ethical considerations about privacy and discrimination that such an application may cause. we further describe our architecture, related work and methodology in section 2 and evaluate its generation capabilities in section 3. we bring important ethical considerations in section 4 and open the discussion for future work in section 5. ## generating intelligible plumitifs summaries plumitifs are used as summaries presenting all the steps of a case heard by the court. in the context of criminal proceedings, they contain information about the plaintiff, the accused, different charges along with their associated penalty (if applicable). we present a plumitif example in appendix a, figure 2. plumitifs are freely available in person at any courthouse and are also accessible on the société québécoise d'information juridique (soquij) website 2 where they can be consulted for a fee. in this section, we detail our proposed architecture, which is broken down into three steps; segmenting the plumitif into parts, extracting relevant information from each part, and generating descriptions by also leveraging the ccc. we illustrate the whole architecture in appendix b, figure 4 and further detail each component in the following subsections. ## evaluating the realisation of the summaries since our generation model mostly relies on rules, it is straightforward to evaluate its performance; we first need to make sure all the relevant information is fully extracted (ner step) and that it properly fills in the corresponding template (generation step). we thus quantify our model's performance in terms of "error rate" where a generation error is the lack of realizing a specific part (accused, plaintiff or list of charges paragraphs), instead of evaluating the textual generation. the counts are computed per text. errors are split into two categories; extractionbased errors (ee) and generation-based errors (ge). for clarity, we display the errors rates by districts in in most cases, we find that a wrong extraction of the plaintiff (due to the ner model) causes ee. we can see that granby and sherbrooke have the highest ee rate; this is mostly due to the many different values an organisation can take in these districts 8 . ge are mainly due to edge cases found in plumitifs which our rules do not cover. as we can see from the ge rates in ## ethical considerations there is some ethical considerations regarding our dataset's privacy that ought to be addressed. plumitifs contain sensitive information such as the names, dates of birth, addresses and criminal backgrounds of accused people. the identity of judges, plaintiffs, clerks, and attorneys taking part in a criminal case are also found in the plumitifs. as explained in section 1, all of this information must be publicly accessible. as long as this data is protected by practical obscurity 9 , the actual risks from public access of this information are limited @xcite . however, if this data was to be released in bulk to the scientific community, it would not be "scattered [. . .] bits of information" (us department of justice v. reporters committee for freedom of the press, 1989) that require time and resources to retrieve anymore. information could be easily searched, aggregated or combined with information from other public sources. this poses a risk to the privacy of judicial stakeholders. in this subsection, we explain why we decided not to release our data set publicly (raw or synthesized). to put it in straightforward terms: information collected in public records should not be "up for grabs". its use can result in privacy violations. this is especially true in the digital context where aggregation, linkage and analytics are made easier @xcite . there are several examples of privacy violations that occurred due to the malicious use of judicial information that was publicly accessible. for instance, more than 270 cases of identity theft have been linked to a security lapse in an american municipal court's website. @xcite . the office of the privacy commissioner of canada had to intervene to end an extortion scheme relying on data available from the canadian legal information institute and soquij's websites @xcite . united state's "public access to court electronic records" system made the identity of some cooperating defendants and undercover agents publicly available, which contributed to the intimidation and harassment of witnesses in order to discourage them from testifying @xcite . there have also been some docu9 a term broadly used to explain that documents might be accessible to all in principle, but that the access is hindered by some obstacles such as fees to consult a document or the need to go physically to a location -as is the case for the plumitif. this is now prohibited by the personal information protection and electronic documents act, (office of the privacy commissionner of canada, 2014), but @xcite argues that this regulatory framework is not sufficient to prevent inappropriate uses of judicial data. our team is currently working to develop a framework for the management of personal information contained in digital court records. however, for the moment, since the law provides no satisfactory solution, we chose not to release the dataset used to train our algorithm. ## conclusion and future work in this paper, we introduce a simple yet effective multi-source architecture able to generate digestible plumitifs for canadian citizens. we also show that we are in a position to easily divulge who has been accused of what and the outcome of it, which raises some important ethical concerns. in the future, we plan to explore statistical natural language generation further by using case law, provide more diverse plumitifs descriptions and improve the generation of sentences. finally, we hope that our application will provide better insights to the community and give the right direction for the next applications of not only nlg, but machine learning in general, in the field of law.