Semi-automatic Triage of Requests for Free Legal Assistance (2021)

Abstract: Free legal assistance is critically under-resourced, and many of those who seek legal help have their needs unmet. A major bottleneck in the provision of free legal assistance to those most in need is the determination of the precise nature of the legal problem. This paper describes a collaboration with a major provider of free legal assistance, and the deployment of natural language processing models to assign area-of-law categories to real-world requests for legal assistance. In particular, we focus on an investigation of models to generate efficiencies in the triage process, but also the risks associated with naive use of model predictions, including fairness across different user demographics.

## introduction the number of australians with unmet legal needs is estimated to be over 4 million people per year and growing (out of a total population of around 25 million): each year approximately 8.5 million australians will have a legal problem and only around 4.5 million will access any assistance @xcite the department of justice and regulation, 2012) -an indication that free legal assistance services are critically under-resourced. a bottleneck for free legal assistance providers is the determination of what (if any) specific legal needs the individual has. we investigate the viability of semi-automating this step by building a model that suggests how to categorise lay descriptions of problems/incidents into legal areas. it is critical that we develop models which will perform equally well for users of all backgrounds, generalise well from small amounts of curated data, and potentially dynamically interact with the help-seeker to clarify the nature of the case. however, in this preliminary work, our aim is to develop initial models as a means to ascertain what biases manifest in our given data, and to have a workable model upon which we can make incremental measurable improvements. text classification of any real-world data can be a challenge for many reasons. in the case of legal text classification, the classes themselves or the legal categorisation of a possible case, can vary from organisation to organisation, and also from court to court; there is no universally agreed-upon set of areas of law neatly defined into a taxonomy @xcite @xcite . furthermore, a case can span multiple areas of law -for example, a family law matter could also fall under the umbrella of guardianship and ad-ministration, or a charities law issue may also have aspects regarding employees and volunteers. in addition to the issues surrounding the inherent fuzziness of legal categories, the descriptions of legal issues themselves exhibit a range of language styles: those who seek free legal help are not versed in the legal domain, and may have varying linguistic styles, reflecting their social, cultural, and educational background. we report on an ongoing collaboration between justice connect, a public benevolent institution, as defined by the australian government, 1 that aims to ameliorate social inequalities through legal assistance and community engagement, and melbourne university whose aim is to alleviate the help-seeker intake bottleneck. in section 2, we outline the importance of accessible legal assistance to those most in need, and the barriers to be overcome in providing this service to the community. section 3 details the data collection and corpus creation process. we designed and developed an annotation platform exclusively for volunteer lawyers to annotate online requests for help from the public through the justice connect intake portal. our experiments and results are outlined in section 4 and 5, which describe the initial fine-tuned bert classifiers @xcite on the small curated help-seeker data informally describing issues in their own words on matters they believe require legal assistance. as a starting point, we wanted to leverage the patterns of language usage encoded in bert given our relatively small data set. the main risk is that while robust results can be achieved by fine-tuning over relatively little labelled data in this manner, the data used in developing the pre-trained models can lead to these models implicitly capturing a variety of biases about the world @xcite . in section 6, we reveal how these biases manifest for our given data set, not only in terms of which areas of law the models can, or cannot, reliably predict, but also which demographic groups the model has inherent difficulty in representing. finally, in section 8, we discuss how we can overcome these biases for future iterations of the model while keeping in mind the protection and privacy of the help-seekers who are most vulnerable. unresolved legal problems have been shown to lead to significant life impacts at high levels of frequency, including financial strain (29%), stressrelated illness (20%), physical ill health (19%), relationship breakdown (10%), and having to move home (5%) @xcite . even when a person is eligible for free legal assistance, there are various ecosystem-level barriers that increase the difficulty of finding and engaging with a legal service. one such barrier is the disconnection between a person who has recognised that their problem may have a legal dimension but does not yet know the technical terminology around their issue, and a legal service that can assist that person with the problem they have. this barrier is exacerbated in online settings, as people search via search engines and directories for legal services without the right search terms and technical language required to successfully reach relevant services. in a follow-up survey by justice connect, many applicants, in requesting help online, self-identify this issue: e.g. "[there are] too many seperate courts and unclear what laws do what (sic)", "it's complex and i am not an expert!" and "[i'm unsure of the category] because of the family relationship together with financial issue". legal services also experience this issue as a bottleneck in their processes, where significant resources are required to assist applicants to determine the nature of the problem that they have (especially difficult given that many users of such services have little or no formal experience with the legal system), whether it is legal in nature, and what specific legal services should be provided. lack of knowledge and capability often results in "failing to identify a legal problem, consulting non-legal advisers instead of legal experts, or taking no action to resolve the problem" (the department of justice and regulation, 2016, p120). legal area classification @xcite @xcite can potentially help to alleviate this bottleneck, in providing semiautomatic legal triaging of user-supplied textual descriptions of the issue. ## data set development the corpus upon which the data set is derived comes from requests for help via the justice connect web-based intake tool. our annotators are lawyers from firms that were approached based on their level of engagement with justice connect. a number of firms had previously shown interest in pilot and project opportunities through justice connect's subscription model. annotators were self-elected, or chosen by each firm's pro bono coordinators. they were asked to disclose how many years they had been practicing in total. of these firms, there were 231 lawyersall admitted to practice law in australia -who were signed up for the annotation task. in addition, there were 12 justice connect-based lawyers who selfelected to participate as annotators, taking the total number of lawyers to 243 over 9 firms throughout australia. we derive three different labellings of the data from these annotations: majority-vote: majority-vote labels for each text 4 from 16 november 2020 to 30 april 2021 sample based on a per-class majority vote over the annotators. that is, in order for a label to apply it must have been assigned by at least half of the annotators. confidence-weighted: the weighted mean of the 'certainty' or 'confidence' score for each label, as self-assessed by the annotator on a scale of 1-100 (according to the placement of a slider), by averaging over the confidence scores for all annotators who assigned a given label. annotator-weighted: the proportion of annotators who assigned a given area of law to the instance, divided by the number of annotators (constrained to be at least 3). as an illustration of how confidence-and annotator-weighting work, one text sample (entry id 3085) had 3 annotators: the first annotator labelled it with the tags litigation and dispute res-olution and employees and volunteers, giving a score of 42% and 100%, respectively; the second annotator applied the same labels but rated them both 100; and the third annotator tagged this excerpt with employees and volunteers and per-sonal safety with confidence ratings 79 and 67, re- ## experiments we performed fine-tuning experiments using bert, with early stopping based on the validation loss. in addition, we experimented with various values for the dropout rate during training, with the final value being set to 0.001. the batch size was set to 32 and the number of epochs was set to 50. all experiments are based on 20-fold cross-validation, and were run 10 times and averaged, with each run having the data split into 20 folds randomly. the non-testing portion of each fold was split into 90/10 for training/validation. for experiments over the confidence-weighted and annotator-weighted data sets, we train over the given label representation, but evaluate on the majority-vote data. we do this because the majority-vote labelled data is our approximation of a manually curated gold-standard data set, also for direct model comparability. given that the number of labels is quite large (33 areas of law, including not a legal issue), we wanted to see if grouping the tags into small thematic groups would increase accuracy. we experimented with 2 grouping structures: (1) "legal", based on legal specialisations; and (2) "theme", based on topics or themes that may be shared between the areas of law. these groupings were agreed upon by trained lawyers at justice connect, where the first group was determined by answering the question, in general, if a lawyer specialises in area x, do they often specialise in area y too?, and for the latter, what areas of law have common narratives or topics shared between them, when people describe issues pertaining to these areas of law? we include three baselines to gauge how difficult the task is: the results show that while the confidenceweighted-trained system outperforms the other systems with respect to precision, recall is low and misses out on correctly classifying instances well over half the time. although, when it does label instances, it gets them correct 83.8% of the time. the annotator-weighted-trained model obtains the highest recall, and also obtains the best trade-off between precision and recall, as reflected in the best f1-score. all models perform well above all three baselines, for all of precision, recall, and f1-score. for all 3 bert models, recall is rather low. in the original experiments, if the model predicted a given label with a @xmath0, we output it as a prediction, but it is also possible to adjust the probability threshold to a value other than 50% (with higher values expected to lead to higher precision and lower recall, and lower values expected to lead to higher recall and lower precision). in a follow-up experiment, we learned a threshold per label (area of law) based on the training data. while these experiments, shown in in the previous experiments, the validation data was the same type as the training data (e.g. for 'annotator-weighted', the training and validation data were both labelled with the annotatorweighted approach), but final evaluation was based on the majority-vote labelling. this means that for the models trained on 'annotator-weighted' and 'confidence-weighted', we optimise hyperparameters on the basis of one labelling strategy, and perform our final evaluation based on a separate labelling strategy ('majority-vote'). in the next experiment, we seek to rectify this mismatch by also validating on 'majority-vote' data. the results in summarising the findings of these experiments, the best of the models is trained on 'annotatorweighted' labels, without modifying the probability threshold or majority-vote data validation. it is this model that we experiment with in the remainder of the paper, in terms of scoping out its viability for live deployment in semi-automatically triaging of incoming requests for legal assistance. ## analysis as shown in a greater importance on precision rather than recall. tier iii and iv classes, with an f0.5-score @xmath1.55, are those that are least 'trustworthy' if the system was to emit them as a prediction. even though some classes have a precision of 1.000 (e.g. intel-lectual property and trusts/equity), are still be treated as lower-tier classes for a number of reasons -these classes have far fewer instances and therefore the ability of the model to learn features for these classes is comparatively degraded (and the high precision is perhaps more luck than a reproducible trend). this is reflected in the very poor results in the recall and thus overall f1-score of these classes. the classes in tiers i and ii have higher precision and an f0.5 that range between 0.55 and 0.925. for the areas of law in these tiers with a higher precision, we expect that when a model predicts a text sample as one of these classes, that we can be fairly confident of that prediction. ## fairness from the metadata provided by justice connect, we analyse 6 sub-groups of help-seeker: of these, the sen group fared the worst with an overall f1-score of 0.540, as shown in aboriginal and torres strait islanders make up 1.7% of the total data, and make up a comparable percentage of family law cases (1.8%) and crimi-nal law cases (1.9%), however at a far lower performance than the average when compared to data for all demographics. for example criminal law cases have @xmath2.571, @xmath3.571, and @xmath4.571 (vs. the number of submitted requests by seniors (sen: those who identify as being over 65 years of age) are almost double the number of aboriginal and torres strait islanders, making up slightly over 3% of the total number of samples in the data (3.2% of the data used in the annotator-weighted system). our initial hypothesis for the poor performance of the sen data was that perhaps they made enquiries in certain areas of law that inherently had low performance. for example, if most of the seniors' enquiries fell in the tier iii and iv categories, then this would help explain the overall poorer performance of the sen group. however, our analysis showed otherwise: the vast majority of classes with at least 5 instances from the sen group are tier i or tier ii, and yet for the majority of these, the relative performance is below the overall performance for that class, as seen in many of the enquiries by seniors fall in tier i and ii categories, which make up the best-performing classes in general. even though the sen group only makes up around 3% of the total number of instances, they do however make up almost 21% of all enquires regarding elder law, over 6% of all banking and finance, almost 6% of consumer law, and 10% of planning and local government. while property law and planning and local gov-ernment perform well in comparison to the overall results for these classes, the other classes within tier i and ii categories underperform. in particular, employees and volunteers and family law are overall top-performing classes as shown in ## conclusion the findings of our paper presents a preliminary exploration of the application of nlp for social good. the results of this paper shows the importance of building fairer models. one approach as a future avenue in this endeavour is by incorporating adversarial learning @xcite or null space projection @xcite to learn representations that are invariant to subgroups, so as to limit the model from learning undesirable correlations between the legal categories and sub-group features. the challenge though, is that subgroup information is not always available in the data, particularly in the justice connect intake tool to request help where all demographic information is volunteered and not mandatory. this means that possibly identifying proxy attributes (e.g. postcode and education level as a potential means to identify income status) or areas of law that are highly associated with certain sub-groups (e.g. migration issues may be likely submitted by persons whose main language is not english, and elder law issues may likely be submitted by seniors). in addition to the subgroups already presented in this study, in our next iteration group by similar theme/topic • group 0 -not a legal issue -not a legal issue • group 1 -protecting personal interests -intellectual property -privacy -consumer law -employees and volunteers • group 2 -common law elements -trusts/equity -torts and compensation • group 3 -relating to insurance -insurance -building and construction • group 4 -litigation -litigation and dispute resolution • group 5 -organisations/corporations -corporate and commercial law -banking and finance -tax -charities law -fundraising law -it • group 6 -land and housing -property law -housing and residential tenancies -native title -neighbourhood disputes • group 7 -family or personal affairs -family law -guardianship and administration -wills -health -elder law • group 8 -relating to government -public and administrative law -planning and local government -environment -migration -inquiries • group 9 -relating to crimes -fines and infringements -personal safety -criminal law group by legal expertise • group 0 -not a legal issue -not a legal issue • group 1 -corporate -corporate and commercial law -banking and finance -tax -charities law -fundraising law • group 2 -ip and technology -it -privacy -intellectual property • group 3 -property/real estate -property law -housing and residential tenancies -neighbourhood disputes • group 4 -family law -family law -guardianship and administration -elder law • group 5 -wills and estates -wills -trusts/equity • group 6 -government -public and administrative law -inquiries -migration • group 7 -planning and environment -native title -planning and local government -environment • group 8 -criminal -criminal law -fines and infringements -personal safety • group 9 -personal injury/compensation -torts and compensation -consumer law -health • group 10: -employees and volunteers • group 11: -insurance • group 12: -building and construction • group 13: -litigation&dispute resolution figure 3: step 2: highlighting supporting text for the areas of law chosen