Can AMR Assist Legal and Logical Reasoning? (2022)

Abstract: Abstract Meaning Representation (AMR) has been shown to be useful for many downstream tasks. In this work, we explore the use of AMR for legal and logical reasoning. Specifically, we investigate if AMR can help capture logical relationships on multiple choice question answering (MCQA) tasks. We propose neural architectures that utilize linearised AMR graphs in combination with pre-trained language models. While these models are not able to outperform text-only baselines, they correctly solve different instances than the text models, suggesting complementary abilities. Error analysis further reveals that AMR parsing quality is the most prominent challenge, especially regarding inputs with multiple sentences. We conduct a theoretical analysis of how logical relations are represented in AMR and conclude it might be helpful in some logical statements but not for others.

## introduction legal nlp has become a highly researched topic in recent times because of its many real-world applications @xcite . the field has been concerned with developing tools to help legal practitioners with time-consuming and repetitive tasks such as finding similar court cases. many of these tasks require reading huge amounts of legal documents, which can take a long time and therefore benefit from automation. legal nlp aims to build systems that can help legal experts as well as people without legal knowledge. examples of this could be a qa system that allows consumers to ask questions about their data privacy rights or a system that can tell citizens if they are eligible for a certain social service program by stating their case. regardless of the task, a system needs to be able to capture the semantics of the relevant legal documents. this can be done implicitly by using the text directly, or explicitly with semantic representation frameworks. semantic parsing is the process of converting natural text into a graph-structured representation of sentence meaning @xcite . the idea is to utilise the semantic graphs instead of or in addition to the textual input, which allows the system to better encode the document semantics. abstract meaning representation (amr; @xcite represents the semantics of a sentence as a rooted, directed acyclic graph, where nodes represent concepts and edges encode relations. the advances in amr parsing have been significant in recent years @xcite , with state-of-the-art (sota) amr parsers achieving smatch scores @xcite higher than 84 on the latest amr 3.0 dataset @xcite . 2 this creates the possibility of using amr for downstream tasks including commonsense reasoning @xcite , information extraction @xcite and question answering @xcite . contributions. this paper investigates whether amr can help legal and logical reasoning on mcqa tasks. specifically, we investigate if amr can help capture logical relationships, since understanding the logic in law is a major challenge in legal nlp and amr facilitates the representation of some logical structure in sentences. different models utilising amr are tested and compared with text-only baseline systems on a mcqa task targeting logical reasoning. lastly, we provide an error analysis to identify issues with the proposed ## logical relations in amr to reason about whether amr can help capture logic in law, consider quantifiers, negation, conjunction, disjunction, implication and equivalencelogical operators used in propositional logic @xcite . some logical statements and their corresponding logical connectives are represented consistently in amr, regardless of the specific english expression (surface form). this includes conditional statements with if, unless and in case of, etc., represented using the :condition role. the core concept of the consequence is the root node and the antecedent has the role :condition. for example, "no major traffic accidents will occur if the highway is not closed" is represented as: in this case "no major accident will occur" is the consequence and "the highway is not closed" is the antecedent. negation is represented in a logical sense with the :polarity role. furthermore, amr aims to represent the semantics of a sentence independently of syntax, meaning that the same graph corresponds to multiple sentences. the amr would not change if the consequence and antecedent were reversed, i.e., "if the highway is not closed, no major traffic accidents will occur." in other cases the representation is closer to the surface form. for example, the and concept is used to represent both conjunctive statements and conjunction of entities and therefore does not always represent logical conjunction. it uses the :opn roles for the operands. the sentence "all musicians are capable of reading music and some musicians are capable of improvising" is represented as: the and concept does not always represent a conjunctive statement, similar to how and in a sentence is not only used to connect two statements. this holds also for disjunctive statements and or. other conjunction words, such as moreover, use different roles or concepts. the conjunctions but and however are represented by the concepts contrast-01 or instead-of-91 or the role :concession-of. this is an example of how, besides logical relationships, certain amr concepts and roles correspond to discourse connectives @xcite . 3 to conclude, amr helps capture some logical statements but not others. 4 3 related work @xcite incorporated structured semantic information from amrs for machine translation. they use a graph recurrent network (grn) to encode amrs and a sequential lstm @xcite to encode the source input text. for the decoder model, they use a doubly attentive lstm architecture, taking both the graph and text encoding as attention memory. they show on an english-to-german translation task that using amr as complementary to the source text input improves performance. there have been attempts of using linearised amrs with pretrained language models (plm). for example, @xcite fine-tuned a transformer language model with linearised amr graphs for the amr-to-text task. various methods for graph linearisation and simplifications have been tried such as depth-first traversal through the graph @xcite . linearised amrs have also been used in combination with cnns @xcite and phrase-based models @xcite . the introduction of large plms such as bert @xcite has led to new sota performance in many nlp domains in recent years. in the legal domain, using domain-specific plms for simpler tasks such as text classification has only shown small improvements @xcite . however, bigger gains were achieved for more complex tasks @xcite . other efforts have addressed the issue that legal documents are oftentimes much longer than the input size of standard transformer models such as bert. plms for long sequences such as longformer @xcite ) have shown to be beneficial for such tasks @xcite . to the best of our knowledge, no one has tried to leverage structured semantic information for a reading comprehension task in the legal domain. similar to the legal domain, large plms such 3 capturing discourse relations is in fact often useful for legal reasoning even if they do not correspond to logical operators @xcite . nevertheless, here we focus on logical reasoning and assume the text is semantically self-contained. 4 our full analysis of logical expressions is in appendix a. as bert have struggled with reading comprehension tasks that require logical reasoning @xcite . @xcite proposed a qa model which constructs logical graphs from a set of elementary discourse units (edus), where the edges are discourse relations. @xcite improved this method by introducing logical relations mapped from rhetorical relations using graphene @xcite . in our work, we try to use amr graphs to capture logical relations. other work has explored the use of amr for mcqa tasks. xu et al. utilises amr graphs to fuse semantic concepts between the hypothesis and retrieved evidence facts to find reasoning chains @xcite and create active fact-level connection graphs @xcite for multi-hop science question answering. the reasoning chains/connection graphs are used to select relevant facts and to guide the reasoning process. in comparison, we use amr graphs from the context and answer directly with a pre-trained amr language model to extract embeddings. motivated by a similar research question, @xcite investigated whether supervised syntactic parsing is beneficial for natural language understanding (nlu) tasks. rather than meaning representation, they focused on syntactic representation with universal dependencies (de marneffe et al., 2021) and their methodology is different from ours in that they use fine-tuning on parsing (with a biaffine decoder) as a way to infuse the symbolic representation into the model, whereas we incorporate linearised amr graphs directly into the architecture. furthermore, we focus on specific nlu tasks that require logical or legal reasoning. nevertheless, we reach a similar conclusion, namely that explicit symbolic representation provides negligible impact, if any. ## amr for mcqa in mcqa, each instance consists of a context paragraph, a question and several answer options, among which only one is correct. a model is evaluated by accuracy, that is, frequency of selecting the correct answer. we propose a system for mcqa by utilising semantic encoding with amr, to hopefully better capture semantics than a text-only system. our system is composed of an amr parser for converting the source text into amr graphs, an encoding component for amr based on graph linearisation in combination with a plm (a domain-specific model for legal text) and a feedforward layer that takes the text and graph encoding as inputs and makes predictions. ## experiments we experiment with the model architectures proposed in §4: bert, legalbert and bart, which use text-only input; amrbart and the smatchbased similarity model, which uses amr-only input, and the fusion model, which uses both inputs. ## results we proceed to present the results of our experiments, measuring performance by accuracy. 11 ## error analysis models using only amr input perform overall worse than models with text input. to check if they solve different instances than the text models, for casehold, the percentage of intersecting instances is 86%. this means that most of am-rbart's correct predictions were also correctly predicted by the text model. on the other hand, the percentage of intersecting instances for logiqa is 29%. this means that less than a third of the correctly predicted instances by the amr model were also correctly predicted by the text model. to see if the amr model can consistently solve different instances than the text model, the experiments were run three times for the logiqa dataset. the results (see model input model size accuracy random 0.25 bert base text 139m 0.28 amrbart base amr (spring prepr.) 142m 0.27 fusion model text and amr (spring prepr.) 252m 0.27 ## conclusion we investigated whether amr can help capture logical relations by conducting experiments on legal and logical reasoning datasets with model architectures that utilize amr. in addition, a theoretical analysis was performed to see how logical statements are represented in amr. specifically, we proposed four amr model architectures for casehold, which requires legal reasoning. using only amr input performs worse than using text input. using both amr and text input showed similar performance to the text-only models. amr models have therefore not been able to outperform baselines. we further analysed the performance for logiqa, which requires logical reasoning. again, amr models did not outperform text-only baselines. since only some types of logical statements are represented consistently in amr and certain concepts and roles are not only used to represent logical relationships but are also used to annotate other semantics, we can expect this kind of mixed results. amr might be useful to capture logical relations in some statements but not for others. our fusion model takes the text encoding and graph encoding as inputs and makes predictions. ideally, the model would leverage amr to better capture the semantics than a text-only system. this can be a separate model or simply a task-specific head of a transformer model. future work will further investigate how amr can help understand the logic in law. the challenge of creating accurate document-level amrs remains and alternative graph encoding components @xcite ) may be more appropriate. by addressing this, the utility of amr on downstream tasks in the legal domain, which oftentimes requires the understanding of long documents, could be manifold. besides automated reasoning, nlp can help humans understand logical relationships by e.g., creating simplified versions of legal text. this could in turn be used to enable semi-automatic legal process discovery @xcite .