A Free Format Legal Question Answering System (2021)

Abstract: We present an information retrieval-based question answer system to answer legal questions. The system is not limited to a predefined set of questions or patterns and uses both sparse vector search and embeddings for input to a BERT-based answer re-ranking system. A combination of general domain and legal domain data is used for training. This natural question answering system is in production and is used commercially.

## introduction question answering (qa) applications range from simple yes/no systems to complex questions where answers might be synthesized from several sources @xcite . this work concerns a qa system for legal research. the system is designed to answer both factoid @xcite and non-factoid questions. a short answer can satisfy factoid questions, e.g., "what is the burden of proof for breach of contract?". in contrast, non-factoid questions are open-ended and an adequate answer needs opinions or explanations @xcite , e.g., "why does child support increase with income?". a typical system user is a litigator seeking answers to case-specific legal questions. those answers inform the creation of litigation documents, such as pleadings, briefs, and motions. a system should provide complete multiple-sentence answers with context that can be cited. a legal qa system must also handle questions where no single answer exists. for example, the useful answer may be jurisdiction specific, or be time frame dependent because the law evolves. importantly, the best answers can depend on the lawyer's perspective because application of the law can make fine distinctions per the case facts, or recognizes competing principles or mitigating factors. legal practice areas can have distinctive concerns. for example, the scope of legal principles and practice affecting family law legal matters is distinguishable from those applied in bankruptcy law. developing an effective and useful question answering system in this setting faces state of the art challenges, including creation of training collections and performance metrics. we present a retrieval-based legal domain qa system designed to provide useful answers for all legal practice areas. it is designed for customers with real-world tasks while meeting cost, response time, and scalability constraints. this system is in production and serving customers. we also describe an experiment methodology found to have pragmatic value in system development. the methodology can be useful in domains with similar context-laden qa characteristics. ## related work recent deep learning open domain qa research successfully applied a retrieve and read paradigm. the retrieval step selects candidate documents, then a reading component finds answers @xcite @xcite . we adopted a similar approach. often, the systems employ standard retrieval methods based on sparse vector space approaches like tf-idf @xcite and bm25 @xcite . dense vector representations with distributional semantic properties based on lsa @xcite , glove @xcite , and sentence embedding @xcite are also used. custom domain embedding methods such as legalbert have been studied @xcite . the qa retrieval target is usually at the passage level rather than complete documents @xcite . paragraph-based legal domain search has been studied @xcite . answer extraction techniques have used machine reading comprehension (mrc) @xcite and drqa @xcite . several ranking models have been applied in qa systems @xcite . one coliee 1 task is to answer yes/no legal questions and retrieve germane legal documents. competitors have used tf-idf and bm25, and also contextual embedding vectors such as bert and elmo, to score passages against the question @xcite . lex-isnexis legacy answer cards address knowledgebased qa by detecting the user query intent and then serving a previously mined answer if appropriate @xcite . the westsearch qa system categorizes non-factoid questions to different frames. to answer a frame-specific question, a trained framespecific question-answer pair classifier is used to recognize a retrieved passage as an answer @xcite . both of these commercial legal qa system can only handle a limited range of questions. in contrast to the previous work, the system presented here is designed to answer almost all legal content questions without legal practice area restrictions. in particular, the question coverage is not limited by pattern or frame. ## methodology the system selects answers by re-ranking search results obtained using both sparse vector techniques (bm25) and a dense vector approach (semantic embedding). figure 1 shows a simplified system architecture. the search repository contains the passage text and passage embeddings. the search engine retrieves passages by text and embedding similarity. the answer finder re-ranks the retrieved passages. ## results two test sets were used: 100 and 1000 questions. the small set proved valuable for rapid development cycles. both sets consist of 50% actual user questions with additional questions from smes to ensure coverage of legal research goals including content, entity, and analytic questions. the qa system presented here focuses on answering content questions and may not provide good answers for en-score criteria -1 so unrelated that a user will lose patience with the system ('silly') 0 off point and is not reasonably related 1 right topic but not an answer 2 partial answer 3 good answer the input to answer finder is the top 30 retrieved passages for each retrieval method. answer finder re-ranks the list by answer probability and selects the top 3 passages. the top 3 passages for each method were evaluated. smes evaluated these answer sets in a random order. the answer evaluation scale has five levels and the smes agreed that an acceptable answer will score a '2' or '3'. sme evaluator interrater agreement (irr) is excellent, given the complexity of legal question answering task, with an agreement level of 84% for acceptable answers and 91% for unacceptable answers. setting a threshold of '2' in the test set, the system performance was: f1 0.766, precision 0.827, recall: 0.713, and accuracy 0.759. f1 and accuracy provide classifier performance metrics. the production legal research system presents the top three answers to users, so we also use ranked search results evaluation metrics ## af on bm25_mlt an airline corporation is not an insurer of the safety of its passengers. the liability of an airline corporation must be based on negligence. promising system combinations are bolded. production systems have practical constraints and cannot simply optimize performance without considering factors like user experience, cost, and scalability. re-ranking retrieved passages by each retrieval method increased the average dcg, confirming the value of adding answer finder to the system. another advantage of the re-ranker is the ability to combine multiple retrieval methods to improve dcg performance. combining dense and sparse retrieved passages both increases system cost and creates scaling challenges. however, the richer representation can improve user experience because users are able to ask their question using a greater variety of words. another advantage of using a reranker is the capacity to apply a threshold to help filter unrelated passages. ## error analysis case-specific error-analysis helps to identify shortcomings in the training data coverage. categorized dcg analysis indicates the qa system provides good answers for well-defined legal questions such as standard of review foot_2 question (e.g. what is standard of review for marital property allocation decision in florida?) . single topic questions are free format questions about specific legal issue, e.g "can an executor compromise a claim without beneficiary consent?". the qa system performs significantly better than previous systems on single topic questions, although there remains room for improvement. optimizing the real world performance of a legal qa system needs to consider the frequency of each question category, the space for improvement, and the question category importance judged by smes. we identified 16 categories to prioritize for high return on development investment. single topic question is at the top of the list. training data for this category will be created to improve system accuracy. ## discussion and future work various development directions are being explored. direct extensions of this work include use of a smaller siamese bert, vocabulary improvements for the models, training set enhancements targeting low performing legal areas, and training the answer finder component on augmented training data and smaller legal-bert models.