Answering Legal Questions by Learning Neural Attentive Text Representation (2020)

Abstract: Text representation plays a vital role in retrieval-based question answering, especially in the legal domain where documents are usually long and complicated. The better the question and the legal documents are represented, the more accurate they are matched. In this paper, we focus on the task of answering legal questions at the article level. Given a legal question, the goal is to retrieve all the correct and valid legal articles, that can be used as the basic to answer the question. We present a retrieval-based model for the task by learning neural attentive text representation. Our text representation method first leverages convolutional neural networks to extract important information in a question and legal articles. Attention mechanisms are then used to represent the question and articles and select appropriate information to align them in a matching process. Experimental results on an annotated corpus consisting of 5,922 Vietnamese legal questions show that our model outperforms state-of-the-art retrieval-based methods for question answering by large margins in terms of both recall and NDCG.

## introduction question answering in the legal domain is a difficult task due to the complication and topic variety of legal document systems. giving an accurate answer to a legal question usually requires domain expert knowledge, making this task more challenging even with human beings. retrieval-based approaches, which return relevant legal documents respect to the input question, are often feasible and more appropriate. legal documents, however, are usually long and complicated, which takes time to read and locate exact information. relevant legal articles are, therefore, the expected result of retrieval-based legal question answering systems, where users are easier to find a useful answer than the whole documents. similar to an information retrieval system, a typical retrieval-based question answering one usually consists of three main steps: question 1 representation, document representation, and matching, which judges the relevance between the question and documents @xcite . text representation, i.e., question and document representation, therefore, plays a crucial role in such systems. good text representation methods are expected to be able to extract important information from the input question and documents and be able to select appropriate information to align them in the matching step. recently, many successful text representation methods have been proposed thanks to the advancement of deep neural network models. the most popular deep neural architectures for text representation include convolutional neural networks (cnns) @xcite @xcite , recurrent neural networks (rnns) @xcite , and their variations such as long short-term memories (lstms) @xcite @xcite @xcite and gated recurrent units (grus) @xcite , attention mechanisms @xcite , and pre-trained models like bert @xcite , among the others. the main advantage of deep neural networks for text representation, as well as for other tasks, is that they can capture various types of information at different levels by integrating multiple neural architectures in a multi-layer model. in this paper, we deal with the task of retrieval-based legal question answering at the article level, which poses several challenges compared with the task at the document level. the number of legal articles is much more than the number of documents. moreover, differentiating articles in the same document is very challenging because they usually focus on the same topic and share the vocabulary. the contributions of this paper are two-fold: first, we introduce a retrieval-based legal question answering model by learning neural attentive representations of the input question and legal articles. second, we show the effectiveness of the proposed model by introducing an annotated corpus and conducting a series of experiments to compare our model with state-of-the-art methods in the field. in the following, we first describe related work in section 2. section 3 presents our text representation method and retrieval-based question answering model. section 4 describes our datasets and experimental setup. experimental results are presented in section 5. finally, we conclude the paper and discuss future work in section 6. ## related work several approaches have been developed to overcome the challenge of the difference between semantics and query morphology in which lexical overlapping may not mean semantical relevance. non-neural approaches these methods, without using neural networks, mainly determine relevance based on the occurrence and frequency of the terms in the query and the documents. they are oldfashioned approaches for text processing, but the ideas in these methods are essential for current stateof-the-art models. the most basic models for text retrieval are logical models @xcite . it requires users to understand the corpus deeply to create their query for exact logical matching. the first idea for overcoming the problem of exact matching in logical models, @xcite , proposed a statistical matching process, which is the origin of the vector space approaches for textual retrieval, for example, term weighting (tf-idf @xcite ). neural approaches lexical mismatching and ambiguity because of the richness of natural language challenge the non-neural approaches to achieve state-of-the-art results in retrieval tasks. systems, therefore, need to better understand semantics. instead of labor features, authors propose effective neural architectures as well as training methods, and let models extract features of different abstraction levels of semantics from data by itself, which expands system capability by data availability. before bert, most of systems are built based on feeding word embeddings pretrained (word2vec @xcite , or glove @xcite ) or randomized to a neural network (cnn, lstm, etc.). @xcite used an lstm network to construct the semantic vector of a sentence via sentence-pair similarity modeling. @xcite proposed a contextual semantic model based on cnn using both local and global contexts to compresses a word sequence into a lowdimensional vector. @xcite claimed that neural retrieval systems like dssm @xcite don't truly understand relevance. the authors, then, proposed deeprank to mimic the process of humans in assessing relevance. adopting bert, a heavy text-encoding model pretrained on a huge amount of texts, @xcite proposed an ad-hoc retrieval system that can handle document-level retrieval. the system, also, combines lexical matching and bert scores for better performance. the system, however, requires costly computation resource. legal text ad-hoc retrieval digitization of legal documents has created the need for the invention of a more efficient search system. for legal text, using a keyphrase-based search system is a challenge because of contingent on the end-users' expertise in the subject area, such as being aware of complex legal terms, the classification of case laws and statutes. in order to reduce the dependence of system users on legal professionals to interpret results as well as the amount of time taken in retrieving information, attempts have been made to invent systems that can automatically classify legal text and queries. however, the majority of problems identified in legal information retrieval systems is that lengthy texts have to be scrutinized before making a meaningful conclusion. @xcite used neural networks for legal text retrieval. they conducted experiments on a dataset of over 2500 legal cases from various online resources. using deep learning, the authors propose a system with a tf-idf page ranking network to construct document embedding. tran et al. (2020) proposed a deep learning phrase scoring framework to summarize a document into a continuous vector space. base on their experiments, the authors concluded that the summary features extracted by their model could represent selective essential information from a lengthy legal case. @xcite used svm to rank the candidates. in terms of input features for the model, the author used tf-idf, euclidean distance, manhattan distance, jaccard distance, latent semantic indexing, and latent dirichlet allocation. ## conclusion in this paper, we have proposed a method for learning neural attentive text representation for legal text and applied to a retrieval-based question answering system. we choose the appropriate architecture based on the understanding of the characteristic of the legal text. our proposed architecture captures both local dependencies using cnn and long-range context using attention mechanism. with a hierarchical architecture, the models learn the vector representation at different abstract levels of a query and an article and the relevance between the two. our models can make a big gap comparing to elastic search and outperform the bert-based model.