Few-shot and Zero-shot Approaches to Legal Text Classification: A Case Study in the Financial Sector (2021)

Abstract: The application of predictive coding techniques to legal texts has the potential to greatly reduce the cost of legal review of documents, however, there is such a wide array of legal tasks and continuously evolving legislation that it is hard to construct sufficient training data to cover all cases. In this paper, we investigate few-shot and zero-shot approaches that require substantially less training data and introduce a triplet architecture, which for promissory statements produces performance close to that of a supervised system. This method allows predictive coding methods to be rapidly developed for new regulations and markets.

## introduction organizations that are governed by legal and regulatory statues concerning communications with the public are required to comply with principles-based content standards. as such, this involves a significant expense due to having to use highly qualified staff to review, iterate on communications internally and file content with regulators, externally. there is thus a substantial expense in terms of iteration time and specialized staff associated with this process. with recent advances in natural language processing (nlp) technologies, it is increasingly becoming possible to automatically flag high-risk statements by predictive coding and thus reduce the cost of these manual reviews. however, each industry has specific regulatory requirements and modern nlp systems need large training sets to be effective, and as such it is challenging to develop such systems. in this paper, we focus on a single example of such a regulatory compliance in the financial domain under the us regulation finra 2210 1 , which states that "no member may make any false, exaggerated, unwarranted, promissory or misleading statement or claim in any communication." we examine how we can train a system in the following settings: firstly in a traditional dataheavy supervised setting, where a large number of existing examples have been classified. secondly, we investigate a zero-shot training situation, where we have asked a legal expert to provide only rough guidelines for what is not compliant with the legal code. finally, we combine this in a few-shot setting and show that with comparatively little training data, we can achieve performance that is equivalent with the data-heavy supervised setting and thus enables text classification systems for regulatory compliance to be constructed quickly and with little effort allowing them to cover a wide range of industries and national regulatory frameworks. ## related work there has been some work in the area of legal text classification and the application of text classification techniques to legal texts has mostly been successful so far. methods based on counting the words in the text and then classifying using machine learning approaches such as support vector machines @xcite for example by @xcite , where they applied this method to the classification of texts according to the legal area, ruling and time span of the text. deep learning methods such as convolutional neural networks (cnns) have been shown to further improve the performance of such systems @xcite . more recently, the emergence of large pre-trained language models such as bert @xcite has further increased the performance and @xcite showed that these models could be used to classify legal texts according to thousands of labels and even on multiple languages if sufficient training data exists. a criticism of such nlp-based approaches to predictive coding, especially with the emergence of more sophisticated deep learning methods, is that they can appear to be 'black boxes', and thus there has been work in providing explainable systems @xcite that can identify snippets and provides explanations for why they make certain predictions. similarly, some work has gone into the investigation of specific complexities of legal texts, such as in @xcite , who showed that for some legal texts the complex combination of negative and positive statements can confused machine learning approaches. they showed that by combining these machine learning approaches with propositional logic, text classification systems could handle intricate legal wording. ## methodology to solve the problem of legal text classification, we approach this with a triplet architecture @xcite where an input sentence, s, is compared with a positive example s + and a negative example s - as depicted in figure 1 . we begin by describing the model architecture. then we discuss the triplet loss used for training the network. finally, we describe the classification model used for the final classification. ## results in section, we first perform a quantitative analysis of models. we then study a few examples where our approach produces results different from the gold standard dataset. ## conclusion we have investigated the use of few-shot and zeroshot text classification methods for the quick development of predictive coding systems for legal texts. we found that zero-shot systems have a substantial decrease in performance relative to a supervised approach. we then developed a few-shot approach based on a triplet architecture and showed that this model is within a few percentage points of the supervised system in performance but requires much less manual annotation in order to develop the system.