Using O pen W ordnet- PT for Question Answering on Legal Domain (2018)

Abstract: In order to practice a legal profession in Brazil, law graduates must be approved in the OAB national unified bar exam. For their topic coverage and national reach, the OAB exams provide an excellent benchmark for the performance of legal information systems, as it provides objective metrics and are challenging even for humans, as only 20% of its candidates are approved. After constructing a new data set on the exams and doing shallow experiments on it, we now employ the OpenWordnet-PT to verify whether using word senses and relations we can improve previous results. We discuss the results, possible future ideas and the additions to the OpenWordnet-PT that we made.

## introduction automatic analysis of legal content offers opportunities for improving the effectiveness of legal actors, transparency of the system and, ultimately, the welfare of the public. as law is practiced with language itself, linguistic approaches are invaluable. this focus on language and higher demand for precision created by a technical domain makes it natural to try to grow upon and evaluate the performance of a lexical-semantic resource, such as wordnets, in this area. one task for legal technology is question answering: an automatic way of determining the right answer to a question presented in natural language form @xcite . an ideal legal question answering system would take a question in natural language and a corpus of all legal documents in a given jurisdiction, and would return both a correct answer and its legal foundation (answer justification), i.e., which sections (or articles) of which norms provide support for the answer. considering lack of knowledge about facts, incompleteness, inconsistency or disagreements between sources of law, an ideal system would generate each possible answer with corresponding arguments, explanations and confidence value. since such a system is still far from our current capabilities, as the results of recent evaluation tasks such as respubliqa @xcite has shown, we started with a simpler task. in brazil, even after graduating from law school, it is required that one is approved in the oab exam in order to practice a legal profession. the "ordem dos advogados do brasil" (order of attorneys of brazil, oab) is the professional body of lawyers in brazil. the first stage of the exam is a multiple-choice test. we are interested in investigating the performance of simple methods in answering this test correctly, and providing justifications for its answers. we measure the impact of the usage of an open lexical resource such as wordnet, and also promote its expansion into the legal domain by demand. in particular, we use freeling @xcite for linguistic analysis, and evaluate specially the usage of the word sense disambiguation (wsd) module @xcite , which in portuguese, uses openwordnet-pt @xcite (own-pt). we find that the system does not improve considerably over the performance of our previous effort @xcite ; however, this might be because of missing concepts and relations in own-pt, which in turn render some of freeling's processing inaccurate. in section 2 we present the data-set we created and made available for experimentation. in section 3 we discuss our previous experiments with the data-set, while in section 4 we describe the tools and resources we employed for our current experiment: freeling, own-pt, and the word sense disambiguation algorithm ukb @xcite . in section 5 we describe the meth-ods used in our experiments and then discuss its results in section 6. finally, we conclude and debate future works in section 7. ## the oab exams data set among other responsibilities, oab is responsible for the regulation of the legal profession in the brazilian jurisdiction. one of the key ways of regulating the legal practice is through the "exame unificado da oab" (unified bar examination), required for enrolling at oab, which is mandatory to practice law. in order to be approved in the oab exam, candidates need to be approved in two stages. the first phase consists of multiple choice questions, while the second phase involves free-text questions. since 2012, the first phase has 80 multiple choice questions and each question has 4 alternatives. candidates are asked to choose the correct alternative and in order to be approved, candidates need at least a 50% performance. historically, the exam has had a global 80% failure rate, with the first stage being responsible for eliminating the majority of the candidates @xcite . thus, the first stage of the oab exams provides an excellent benchmark for the performance of a system attempting to reason about the law. that is, passing the oab exam would signal that the system has acquired important aspects of legal knowledge, up to a level comparable to a human lawyer. in trying to build such a system, it was necessary to create the appropriate data sets, which includes not only the questions and answer keys in machine readable format, but also the legal literature involved @xcite . in previous work @xcite , we have obtained the pdf files of the all the previous oab exams, extracted their text, cleaned them up and made the data freely available in a public repository foot_0 . along with the 1820 questions (from 22 exams) in plain text and in xml, it contains a golden set of 30 questions which were manually analyzed and annotated with the answer keys' legal basis, i.e., which articles from which norms justify the correct answer to the question. these 30 questions are on a single subject, legal ethics. since 2012, the exams have revealed a pattern for which areas of law the examination board fo-cuses on and in which order the questions appear on the exam. traditionally, the first 10 questions are about legal ethics, that is, the rights, the duties and the responsibilities of the lawyer in regard to brazilian law. we have chosen to provide a golden set on legal ethics because this subject area is the simplest part of the exam with respect to the legal foundations of the questions. it also has a high frequency rate, and the highest performance rate among candidates (65%) @xcite . the key finding from analysis done in our previous work is that, usually, only one article on federal law no. 8906 was enough to justify the answer to the legal ethics questions (15 questions). less often, in four questions, the justification was in "regulamento geral da oab" (oab general regulation), or on the "código de ética da oab" (oab ethics code, 7 questions). three other questions were justified by two articles in law no. 8906 each, and one question only in case law from the superior court of justice about an article from the law no. 8906. federal law no. 8906 has 89 articles, while the oab general regulation has 169 articles, and the oab ethics code has 66 articles. ## the previous work question answering in legal domain is a hard problem. in the last respubliqa evaluation task, the only system that dealt with portuguese texts, the priberam system, has the worst performance among the competitors, obtaining only 0.56 in the c@1 score (peñas et al., 2010). 4 in @xcite the authors report a textual entailment study on the us bar exam material. in the experiment, the authors treat the relationship between the question and the multiplechoice answers as a form of textual entailment. answering a multiple choice legal exam is a more feasible challenge, although it is still a daunting project without restrictions on the input form. that is the reason we have chosen in @xcite to restrict the domain to a single section of the oab exams: legal ethics, one which is governed by only a few legal norms. in @xcite , we conducted 3 experiments in question answering (section 5). in the first experiment, they tried to find the right answer between the multiplechoice alternatives. the last 2 were in shallow question answering (sqa), a form of question answering where a system retrieves documents that justify the already provided answer. they have adapted the methodology described in @xcite to answer multiplechoice exams instead of closed-ended answers. a range of issues on the texts of the questions of the exams was identified. many of the problems are similar to the ones found in the us bar exams and described by @xcite . for instance, some questions do not contain an introductory paragraph defining a context situation for the question. instead of that, they have only meta comments (e.g. "assume that..." and "which of the following alternative is correct?") followed by the choices. some questions are in a negative form, asking the examinee to select the wrong option or providing a statement in the negative form such as "the collective security order cannot be filed by. . . ". moreover, some questions explicitly mention the law under consideration, others do not. many questions present a sentence fragment and ask for the best complement among the alternatives, also exposed as incomplete sentences. even in the presence of such problems, our results in this previous work were not bad, given our system's simplicity. but our initial approach also had its shortcomings: it could not distinguish successfully between two almost identical alternatives which differed only by few words (such as an alternative and its negation), nor could it treat related words in an appropriate manner. the former problem may require deep linguistic processing of the texts for properly obtaining the meaning of the utterances, while the latter can be partly tackled by the use of lexical resource such as the own-pt, as is done in this paper. ## freeling, openwordnet-pt and word sense disambiguation freeling is an open source language processing library developed at the talp research center foot_4 @xcite . it has support for many languages, including english, portuguese, among others. it implements modules for tokenization, sentence splitting, morphological analysis, part-of-speech tagging, word sense disambiguation, parsing and other tasks. freeling distribution includes linguistic data for the supported languages provided by many different projects and collaborators: morphological dictionaries, gazettes, lexical-semantic resources etc. particularly, for portuguese, its word sense disambiguation (wsd) module relies on own-pt, an open freely available wordnet for portuguese @xcite . freeling implements a pipeline-based approach. after tokenization, sentence split and the mogrphological analysis and part-of-speech tagging, the user can choose to execute the wsd module to search for senses in wordnet matching the lemma and part-of-speech tag of each word or multi-word expression. every possible sense is returned and may be weighted by the sense disambiguation module. the disambiguation is an implementation of the ukb algorithm @xcite , an unsupervised graph-based method which uses personalized pagerank to select the right sense of each word in a lexical database such as own-pt. before running our experiment, we did a preliminary survey on the coverage of own-pt for the oab corpus -a proxy for the legal domain as a whole. in princeton wordnet (pwn) @xcite , the synset [08441203n, law/jurisprudence: the collection of rules imposed by authority.] is a general concept about law, and is linked to hundreds of synsets via the classifiesbytopic relation. this suggests that pwn already covers (synset-wise) the relevant context, but it remained to be investigated whether such synsets are properly translated in own-pt with the relevant words, and if the existent concepts indeed encompass notions used in the brazilian legal context, as legal jargon can be language and cultural dependant. in order to further evaluate the coverage of the legal domain in own-pt we have taken a simple approach: after running freeling on our corpus, we have listed the most common words whose senses freeling could not find. we then proceeded to add them to own-pt. some synsets did not seem to exist yet, such as one for "cartório" (notary office). 6 other synsets existed, but the word at hand was not included in it, as in [06532763n, nulidade: nullity]. other cases were those of relations that did not exist in own-pt; if present, these relations would improve the results of the ukb algorithm. one such relation that we included in own-pt was the nominalization (morphosemantic link) between [00664276v, comprovar: authenticate] and [06855035-n, comprovac ¸ão: authentication]. in the end, since we focused only on the possible improvements to our immediate purpose, we have added to own-pt two synsets, eight semantic and lexical relations, and 25 words. after running our experiment (to be described in the next sections), we also reevaluated the legal domain coverage in own-pt. to do so we looked at the difference between the questions answered and justified correctly by our previous system @xcite and the present one. one observation is that even when the wsd was not 6 we will make the data available as part of the own-pt distribution available at http://wnpt.brlcloud . com/wn/. the question below and the first article from law no. 8906 following it illustrate cases where wordnet resources are helpful and a more shallow approach could fail. even though article and question alternative are related, this relation is not captured by our previous algorithm, because it does not take into account anything but the equivalence of tokens. using own-pt, we can exploit the relationship between the action (sign, "visar") and the result of the action (signature, "visto"). constitutive acts and contracts of legal persons, in order to be registered regarding the legal practice statute, must: [. . . ] c) contain the lawyer's [. . . ] signature. (17th ed. oab exam, question 2) § 2 o the constitutive acts and contracts of legal persons can only be registered in the competent bodies, under a penalty of invalidity, when signed by lawyers. (law no. 8906, article 1) in the example above, however, own-pt was missing the words 'visar" and "visto" in the appropriate synsets: [00996485-v, sign, subscribe: mark with one's signature] and [06404582-n, signature: your name written in your own handwriting]. these missing senses, of course, had to be created before being properly linked by the morphosemantic link result. during our evaluation, we also had to make some changes in the freeling dictionary, some adjectives and their lemmas and part-of-speech tags were introduced. an important attribute of this approach is that it propagates. extending the wordnet and giving the right senses for some words can improve the classification of other words that were not changed directly due to correct partof-speech tagging and adequate linking between senses, tasks which depend on neighboring words. the missing words, synsets and links in own-pt is both a problem and an opportunity: in order to make better use of own-pt for the task at hand one must further extend it to the legal domain @xcite . ## experiment setup the original idea for the experiment was inspired by @xcite , and it runs as follows: one collects legal norms in a corpus and preprocesses them performing tasks such as converting text to lower case, eliminating punctuation and numbers and removing stop-words. after that, the articles of the norms are represented as term frequency -inverse document frequency (tf-idf) vectors in a vector space model (vsm) @xcite . in @xcite , we have adapted this method to deal with exam questions with multiple choice alternatives. in the present article, we relied on freeling to incorporated more linguistic processing in our pipeline. we use the freeling tokenizer, sentence splitting, morphological analyzer (pos tagging and lemmatisation), and the wsd modules to assign own-pt synsets, with a weight value (normalized in order to sum 1), to each token or sequence of tokens. for an input text we thus have a list of key-value pairs (s, w) with a sense key and a weight value, in contrast to a simple list of tokens, as we had in the previous experiment. the intuition behind tf-idf is that the more similar two text fragments are, the lesser is the distance between them. as the articles of the norms are not lists of tokens anymore, we have adapted the tf-idf definition to deal with the weights assigned to each synset, as equation 1 shows. a directed graph is then created, with a node for each article of the used norms. this is the base graph, used for answering all questions. when provided a question-answer pair, our system processes the question statement and the alternatives in the same way as it does to the articles in the base graph: turning them into a list of (s, w) pairs. it then turns them into tf-idf vectors using idf values from the document corpus. 7 the statement node is connected to every article node, and each article node is then connected to every alternative node. in this we differ once more from @xcite , as we have no need for heuristic rules for splitting the questions. the edges are given weights whose value is the inverse cosine similarity between the connected nodes' tf-idf vectors. the system then calculates the shortest path between question statement and answer item using dijkstra's algorithm, and returns the article that connects them as the answer justification. unlike @xcite our graph structure does not allow for more than one node connecting statement and alternative, as we knew from previous analysis that questions were usually justified by only a single article. figure 1 illustrates the types of graphs we construct for each question. ## results using the method described in section 5, we conducted two experiments. as we explained in section 2, our golden answer set was manually created qa qa+j j word system 12 12 18 synset system 14 11 17 by one of the authors and it consists of 30 questions from eleven different editions of the oab exam associated to the article on the appropriate norm that justify the answer of the question. our first experiment aimed to evaluate the main task (qa): choosing the right answer at the multiple choice problem, given the questions and the laws (all three normative documents related to the legal ethics area). the performance of the synset system was of 14 questions, against 12 in the word system. if we require not only correct answer, but a correct justification as well, experiment (qa+j), the synset system achieves 11 correct answers, while the word system scores the same 12 correct answers. in some cases, both systems would find the correct justification article for the correct answer, but would pick as their putative answer another (incorrect) item, because it had a shorter path. other times, they would not be capable of deciding between two (or more) answer items, as they all had a shortest path of the same length. the following exam question is a sample case where this statistical approach to question answering is defective: concerning the expiration of punitive disciplinary infractions, choose the right alternative. [. . . ] a) the punitive aim in regard to disciplinary infractions expires after five years [. . . ] b) the punitive aim in regard to disciplinary infractions expires after three years [. . . ] (15th ed. oab exam, question 4) these two options differ by only one word (the number of years until expiration), and coincidentally both are in the text of the article which justifies the answer key. in the synset system, as "three" and "five" are both hyponyms of [13741022-n, digit: one of the elements that col-lectively form a system of numeration], this difference shouldn't interfere with wsd of the other words. this gives us almost the same distance between the question statement and these two answer choices, and the system is incapable of choosing between them. a similar situation arises when one answer item makes a statement and another item denies this statement: in a question like this a system can only systematically report a correct answer if it has a higherlevel understanding of the texts at hand: no bagof-words model will suffice. although results in the first and second experiments may be humble, we then considered shallow question answering. as our approach tries to find not only the correct answer, but to find through a justification, it's reasonable to evaluate the ability to find the correct justification given the correct answer to the question. therefore in our third experiment (j) the system's task was to determine which article (considered every law it has seen) justified the (already given) answer to the sentence. for each question in our golden set, we again added its statement and correct answer as nodes connected to all article nodes in the graph (see figure 1 ). the word system was able to find 18 while the synset system found 17. the overall results are not very impressive, although they are not bad as well. using part-ofspeech tagging and word sense disambiguation in order to improve the use of tf-idf does not solve important difficulties, such as compositional understanding, pragmatics, etc. nevertheless, the contributions to own-pt can be seen as a benefit by itself and will be valuable in the future planned experiments. these contributions may also improve the synset system to the point that it outperforms the word system noticeably. ## conclusion and future works we tested the coverage and improved own-pt with terms from the legal domain. we also presented a new data set with all brazilian oab exams and their answer keys jointly with three brazilian norms in lexml format. furthermore, we also reported our findings in the course of constructing a system to pass in the oab exams. we obtained reasonable results considering the simplicity of the methods employed and the limited golden data available. for the next steps, many other ideas can be tested. the tf-idf vsm approach was devised as a baseline for the next phases of the project. even so, we can still explore variations on that approach with lemmas and edges between articles, considering that 10% of our golden set includes more than one article as justification. moreover, such approach can be combined with other methods, following classical ideas such as @xcite , since it seems to be sufficient for solving many questions. in another direction, we need to increase the size of the golden set. using crowdsourcing websites to obtain more justifications from humans or crawling data from websites dedicated to discussions about the oab exams is likewise a possibility. many different proposals for encoding laws in a machine readable format are available. why no single standard have been largely adopted yet? we aim to explore the best candidates for the remaining normative documents that we will need to cover all areas of the oab exams. we can considering ideas used in the data preparation of the respubliqa editions @xcite . other techniques for textual entailment could be used as well for the task of answering multiple choice questions. given the legal information (such as statutes, regulations and case law) as background knowledge, inferring the correct answer would amount to selecting the item which is entailed by the question statement and background knowledge (in case of multiple entailed answers, the one with highest confidence). the results of the experiments presented here clearly show that we need 'deep' linguistic processing to capture the meaning of natural language utterances in representations suitable for performing inferences. that will require the use of a combination of linguistic and statistical processing methods, possibly using leveraging experiences from @xcite . in @xcite we begin to explore the use of the logic called ialc @xcite . ialc can be used to represent legal knowledge and it may help in the next steps of our project. we may also explore recent advances in statistical relational learning, specially combining probabilistic and logical methods for semantic tasks, such as done by @xcite . this approach uses syntactical parsing in order to construct a logical form, which is given probabilistic semantics, weighted by linguistic resources (e.g. wordnet). using probabilistic logics @xcite and probabilistic soft logic @xcite ) allows a semantic with clear support for vagueness and ambiguity, as well for a integrated use of lexical resources, handcoded rules and information learned from the data itself. the base of this approach is general: logical forms could be encoded in different formalisms, such as ialc or others intermediary semantic representation formats such as amr @xcite , if suitable probabilistic semantics could be given.