D eep A 2: A Modular Framework for Deep Argument Analysis with Pretrained Neural T ext2 T ext Language Models (2022)

Abstract: In this paper, we present and implement a multi-dimensional, modular framework for performing deep argument analysis (DeepA2) using current pre-trained language models (PTLMs). ArgumentAnalyst – a T5 model [Raffel et al. 2020] set up and trained within DeepA2 – reconstructs argumentative texts, which advance an informal argumentation, as valid arguments: It inserts, e.g., missing premises and conclusions, formalizes inferences, and coherently links the logical reconstruction to the source text. We create a synthetic corpus for deep argument analysis, and evaluate ArgumentAnalyst on this new dataset as well as on existing data, specifically EntailmentBank [Dalvi et al. 2021]. Our empirical findings vindicate the overall framework and highlight the advantages of a modular design, in particular its ability to emulate established heuristics (such as hermeneutic cycles), to explore the model’s uncertainty, to cope with the plurality of correct solutions (underdetermination), and to exploit higher-order evidence.

## introduction argumentative text analysis is an interpretation method for clarifying arguments @xcite . being studied in argumentation theory, logic, or epistemology, it is widely taught and applied as a key critical thinking skill in, e.g., law @xcite , the humanities @xcite , social sciences @xcite , policy advice @xcite , or public debate @xcite . this paper presents a computational approach for deep argument analysis, i.e., for reconstructing naturallanguage arguments from a given text, as in the following example (adapted from @xcite : the literature on argument reconstruction (cf. @xcite @xcite @xcite characterizes deep argument analysis as: given these special characteristics, deep argument analysis poses many challenges for machine models of natural language understanding. in this paper, we introduce a novel modular modeling approach for analysing complex argumentation that builds on recent pre-trained text2text transformers @xcite . our approach -deepa2 (illustrated in figure 1 ) -works by systematically decomposing a complex reconstruction problem to smaller text2text sub-tasks (see section 3), which allows for emulating the types of interpretation strategies and heuristics studied in argument theory. referring to the different components of a comprehensive argumentative analysis, we may also define tailor-made metrics for assessing argument reconstructions. to demonstrate the benefits of our approach, we construct a new argumentation dataset (aaac) that exhibits several complex interpretive dimensions, show how to map other existing datasets into our framework (section 4), and train and evaluate our main model, referred to as argu-mentanalyst, within deepa2 (section 5). our empirical results show: 1. argumentanalyst generates -out-of-domainsemantically meaningful argument reconstructions, 70% of which are logically valid. by pooling alternative reconstructions, virtually every source text in the synthetic dataset can be reconstructed as a valid argument. in concluding this paper, we sum-up and interpret these findings as general vindication of deepa2's modular, multi-angular design (section 6). ## related work taking transformers as soft reasoners, recent work, pioneered by @xcite , has shown that pre-trained language models (ptlms) possess basic deductive and abductive reasoning capabilities on diverse domains @xcite @xcite , but are equally prone to fallacies and biases @xcite . besides drawing the correct conclusion, transformers are able to generate correct reasoning chains that justify an answer, which in turn further increases answer accuracy @xcite @xcite @xcite . neural semantic parsing uses sequence models to formalize natural language sentences (kamath and das, 2019). @xcite show that ptlms are zero-shot parsers, and that intermediate steps which rephrase and streamline the original input before parsing it to a formal language improve accuracy. argument mining is an active research field that studies computational methods for retrieving argumentative components from a text corpus @xcite @xcite . recently, work in this field has started to use ptlms: ein-dor et al. (2020) and @xcite succeed in retrieving relevant pro-or con-arguments for a given topic from a large corpus with a fine-tuned bert model @xcite . using bert, @xcite map argumentative texts to key points that succinctly summarize the argument's gist. akiki and potthast (2020) explore abstractive argument retrieval by means of text generation with gpt2 @xcite . similarly, @xcite deploy bart @xcite to generate conclusions of argumentative texts on a challenging corpus compiled from reddit and various online debate corpora. @xcite , revisiting the argument comprehension task @xcite , demonstrate that identifying implicit premises -and deep argument analysis a fortiori -remains a hard, unsolved task. recently, @xcite have shown that augmenting training data with discourse-aware commonsense knowledge improves the plausibility of automatically identified implicit premises. such a knowledge-driven perspective is orthogonal to, and may eventually complement the logical approach adopted in this paper. ## datasets for the experiments reported below, we synthetically create two artificial argument analysis corpora that comply with the deepa2 framework (see also appendix a): aaac01 and aaac02. in addition, we translate the synthetic ruletaker @xcite and the manually compiled entailmentbank @xcite datasets into our framework. in argument analysis, one proceeds from a source text to its reconstruction. creating the synthetic corpora, we reverse-engineer this process: step 1. we sample, first of all, a possibly complex argument (a) from a set of valid inference schemes. in doing so, we use a multi-step templating strategy (inspired by @xcite to translate symbolic forms into natural language schemes (which were generated by local domain experts) and to substitute natural language terms for placeholders. premises (p), conclusion (c) and their formalization (f, o, k) are side-products of such a construction of an argument. step 2. given the fully explicit argument (a), we compose a text (s) that presents the argument in a more or less transparent and faithful way. such text creation involves: rendering the argument tree as a linear story, leaving out premises or conclusions (implicit premises and conclusions), inserting irrelevant material (distractors), using templates that obfuscate the logical form of a sentence, limiting the use of premise and conclusion indicators (such as "therefore"), applying rule-based and automatic paraphrasing. in composing the argumentative text (s), we may record its reasons (r) and conjectures (j). given the synthetic and controlled nature of our dataset, which involved eliciting rule templates from a group of local domain experts, all data is assumed to be correct by construction. as an additional check of correctness on the logic of our examples, we ran a symbolic theorem prover @xcite over the argument formalizations to verify their validity. to ensure the fluency of the underlying language templates, all templates were hand verified by the authors. our two datasets aaac01 and aaac02 differ in the following ways: each dataset contains diverse texts and arguments. broadly speaking, data records may differ in terms of properties of the argument (step 1 above) and properties of the argument's presentation (step 2). along these two dimensions, we define five homogeneous subsets of the data: simple inference: arguments with a single inference step that neither involves negation nor compositional predicates; complex inference: arguments with four inference steps that heavily rely on syntactically intricate schemes (e.g., transposition, or de morgan); plain presentation: all premises and conclusions are explicit in the source text which, in addition, contains no distractors; mutilated presentation: at least two premises and one conclusion are implicit, while the text contains two distractors and explicitly states the final conclusion; c&m: the argument's inference is complex, plus the text contains at least two distractors. the ruletaker and entailmentbank datasets contain multi-hop inference trees (a). to import these into the deepa2 framework, we create source texts (s) for the given arguments by means of simple templates (such as "{theory} all this entails: {hy-pothesis}") and record reasons (r) and conjectures (j) on the fly. unlike aaac and entailmentbank, ruletaker (as updated in tafjord et al., 2020) contains an equal share of arguments for which (i) the conclusion follows from the premises, (ii) the conclusion contradicts the premises, (iii) the conclusion is independent of the premises. ## experiments and results as first and main experiment we train our base model (see section 3.5) on the aaac01 corpus, and evaluate the resulting argumentanalyst model out-of-domain on aaac02. argumentanalyst undergoes multi-task training on 21 generative modes, which are interpreted as sequence-tosequence tasks (the training set-up is further described in appendix b). the evaluation of argumentanalyst on aaac02 proceeds in two steps: (1.) prediction: produces output in accordance with 16 different generative chains (appendix c); (2.) metrics application: assesses the quality of the generated output by means of the systematic and exegetic metrics of the deepa2 framework (see section 3.4). moreover, argumentanalyst produces much better reconstructions of simple inferences and plain presentations -compared to complex inferences and mutilated presentations, i.e., difficult problems (cf. ## conclusion in this paper, we have presented and implemented a multi-angular, modular framework for deep argument analysis (deepa2). it allows for defining a large variety of generative modes by combining different dimensions of the data. these modes, in turn, can be concatenated into complex generative chains. argumentanalyst -a text-to-text model set up and trained within the deepa2 frameworkyields plausible reconstructions of argumentative texts. our empirical findings vindicate the overall framework and highlight the following advantages of a multi-angular, modular design in general: first of all, modular chains may emulate established, well-proven, typically piece-meal, scholarly techniques for text analysis (heuristics), which hence may provide normative, methodological guidance in setting up nlp systems. secondly, by defining and implementing different modular chains, and investigating the plurality of generated solutions, one can systematically explore the system's uncertainty as well as the tasks's underdetermination. thirdly, monitoring the sys-tem during modular computation yields diagnostically useful information (e.g., intermediary results) which not only describes the model's performance on the given problem, but which additionally allows us -as higher-order evidence -to characterize (e.g., classify) the original problem in the first place. fourthly, breaking down a complex task into sub-tasks with intermediary results that can be further processed and re-combined helps to overcome input size limitations of neural language models. fifthly, modular generation with meaningful modes allows users to follow the system, comprehend generated solutions, verify sub-steps and detect errors -the nlp system becomes a transparent, explainable ai @xcite . finally, modular nlp systems as described by deepa2 may be connected to a user-interface which promises fine-grained interactive control of modular generations and seamless cognitive cooperation of ai and human experts in analysing texts. with generalized dilemma (neg variant) from ( 1 ) (2) (3) -(4) if someone admires chico, then they haven't visited monterey. premises: if someone is an admirer of chico, then they are an admirer of laguna beach or a visitor of stockton. (ref: (1)) | if someone admires laguna beach, then they haven't visited monterey. (ref: (2)) | if someone has visited stockton, then they haven't visited monterey. (ref: (3)) conclusion: if someone admires chico, then they haven't visited monterey. (ref: (4)) premises_form: (x): fx -> (g x v h x) (ref: (1)) | (x): g x -> not i x (ref: (2)) | (x): h x -> not i x (ref: (3)) conclusion_form: (x): f x -> not i x (ref: (4)) keys: f: admirer of chico | g: admirer of laguna beach | h: visitor of stockton | i: visitor of monterey