Critical Thinking for Language Models (2021)

Abstract: This paper takes a first step towards a critical thinking curriculum for neural auto-regressive language models. We introduce a synthetic corpus of deductively valid arguments, and generate artificial argumentative texts to train CRiPT: a critical thinking intermediarily pre-trained transformer based on GPT-2. Significant transfer learning effects can be observed: Trained on three simple core schemes, CRiPT accurately completes conclusions of different, and more complex types of arguments, too. CRiPT generalizes the core argument schemes in a correct way. Moreover, we obtain consistent and promising results for NLU benchmarks. In particular, CRiPT’s zero-shot accuracy on the GLUE diagnostics exceeds GPT-2’s performance by 15 percentage points. The findings suggest that intermediary pre-training on texts that exemplify basic reasoning abilities (such as typically covered in critical thinking textbooks) might help language models to acquire a broad range of reasoning skills. The synthetic argumentative texts presented in this paper are a promising starting point for building such a “critical thinking curriculum for language models.”

## introduction pre-trained autoregressive language models (lm) such as gpt-2 and gpt-3 achieve, remarkably, competitive results in a variety of language modeling benchmarks without task-specific fine-tuning @xcite ). yet, it is also widely acknowledged that these models struggle with reasoning tasks, such as natural language inference (nli) or textual entailment @xcite . actually, that doesn't come as a surprise, given the tendency of humans to commit errors in reasoning @xcite , their limited critical thinking skills @xcite , and the resulting omnipresence of fallacies and biases in texts and the frequently low argumentative quality of online debates @xcite @xcite : neural language models are known to pick up and reproduce normative biases (e.g., regarding gender or race) present in the dataset they are trained on @xcite @xcite , as well as other annotation artifacts @xcite ; no wonder this happens with argumentative biases and reasoning flaws, too @xcite . this diagnosis suggests that there is an obvious remedy for lms' poor reasoning capability: make sure that the training corpus contains a sufficient amount of exemplary episodes of sound reasoning. in this paper, we take a first step towards the creation of a "critical thinking curriculum" for neural language models. critical thinking can be loosely defined as "reasonable reflective thinking that is focused on deciding what to believe or do." @xcite generally speaking, our study exploits an analogy between teaching critical thinking to students and training language models so as to improve their reasoning skill. more specifically, we build on three key assumptions that are typically made in critical thinking courses and textbooks: first, there exist fundamental reasoning skills that are required for, or highly conducive to, a large variety of more specific and advanced critical thinking skills (e.g., @xcite . second, drawing deductive inferences is one such basic ability (e.g., @xcite . third, reasoning skills are not (just) acquired by learning a theory of correct reasoning, but by studying lots of examples and doing "lots of good-quality exercises" @xcite , typically moving from simple to more difficult problems (e.g., @xcite . these insights from teaching critical thinking translate, with respect to our study, as follows (see fig. 1 ). first of all, we design and build 'lots of good-quality exercises': a synthetic corpus of de- evaluating cript's ability to correctly complete conclusions of arguments, we observe strong transfer learning effects/generalization (section 5): just training cript on a few central core schemes (generalized modus ponens, contraposition and chain rule) allows it to accurately complete conclusions of different types of arguments, too (e.g., complex argumentative forms that involve dilemma and de morgan). the language models appear to connect and generalize the core argument schemes in a correct way. in addition, cript is equally able to apply learned argument patterns beyond the training corpus' domain. moreover, we test cript on different reasoning benchmarks. because we are particularly interested in transfer learning effects, we do so in a zero-shot set-up (i.e., evaluating our argumentation models on entirely unrelated nlu tasks, which follows recent work by @xcite al. (2019); shwartz et al. (2020); ma et al. (2020) ). we obtain consistent and promising results for the glue diagnostics @xcite and @xcite benchmarks (section 5), finding that training on core schemes clearly improves the nlu skills of pre-trained models. all these transfer learning effects observed strengthen the analogy between teaching critical thinking and training language models: a variety of reasoning skills are improved by generic, inter-mediary pre-training on high-quality texts that exemplify a basic reasoning skill, namely simple deductive argumentation. obviously, drawing correct inferences is just one of the elementary skills typically covered in critical thinking courses @xcite . critical thinking involves more than deduction. and it would hence, by analogy, be unreasonable to expect that intermediary pre-training on the synthetic argument corpus suffices to turn language models into accomplished reasoners. however, we have shown that argumentative texts (with valid syllogistic arguments) are certainly a good starting point when building a more comprehensive dataset for initial or intermediary pre-training that might help language models to acquire a broad range of reasoning skills. or, to put it differently, the synthetic argumentative texts might belong to the core of a "critical thinking curriculum for language models." in the final section, we advance some ideas for complementing the artificial argument corpus so as to further improve the performance of lms with regard to different reasoning benchmarks. ## related work to our knowledge, this paper is, together with @xcite , among the first to show that autoregressive language models like gpt-2 can learn to reason by training on a text corpus of correct natural language arguments. by contrast, previous work in this field, described below, has typically modeled natural language reasoning problems as classification tasks and trained neural systems to accomplish them. for example, @xcite ; @xcite find that a masked language model with classification head achieves remarkable nlu performance by pre-structuring the training data. this paper explores the opposite route: we start with highly structured (synthetic) data, render it as unstructured, plain text and train a uni-directional language model on the synthetic text corpus. over and above the methodological novelty of our approach, we discuss, in the following, related reasoning benchmarks and explain what sets our synthetic argument corpus apart from this work. rule reasoning in natural language various datasets have been developed for (deductive) rule reasoning in natural language. one-step rule application (cf. @xcite @xcite closely resembles the conclusion completion task for gen-eralized modus ponens and generalized modus tollens schemes described below. however, we go beyond previous work in investigating the ability of lms to infer conclusions that have a more complex logico-semantic structure (e.g., existential or universal statements). ruletaker, arguably the most general system for rule reasoning in natural language so far, is a transformer model for multi-hop inference @xcite . prover @xcite extends ruletaker by a component for proof generation and is able to construct valid proofs and outperforms ruletaker in terms answer accuracy in a zero-shot setting. benchmarks for enthymematic reasoning an 'enthymeme' is an argument whose premises are not explicitly stated, e.g.: "jerry is a mouse. therefore, jerry is afraid of cats." the following studies involve such reasoning with implicit assumptions, whereas our synthetic argument corpus doesn't: all premises are transparent and explicitly given. comet generates and extends commonsense knowledge graphs @xcite . trained on seed data, the model is able to meaningfully relate subject phrases to object phrases (by doing the type of completion tasks we introduce in section 4). the argument reasoning comprehension (arc) dataset @xcite comprises simple informal arguments. the task consists in identifying which of two alternative statements is the missing premise in the argument (see also @xcite . clutrr is a task generator for relational reasoning on kinship graphs @xcite . cluttr takes a set of (conceptual) rules about family relations as given and constructs set-theoretic possible worlds (represented as graphs) which instantiate these rules. the task consists in inferring the target fact from the base facts alone -the conceptual rules remain implicit. @xcite show that transformers do not only learn to draw the correct conclusion (given a cluttr task), but also seems to acquire the ability to generate valid proof chains. finally, training on synthetic knowledge-graph data from scratch, @xcite find that bert @xcite is able to correctly infer novel facts implicit in the training data. critical thinking tasks logiqa @xcite is a collection of publicly available critical thinking questions, used by the national civil servants examination of china to assess candidates' critical thinking and problem solving skills. its scope is much broader than our highly specific and carefully designed argument corpus. ## an artificial argument corpus this section describes the construction of a synthetic corpus of natural language arguments used for training and evaluating cript. 1 the corpus is built around eight simple, deductively valid syllogistic argument schemes (top row in fig. 2 ). these eight base schemes have been chosen because of their logical simplicity as well as their relevance in critical thinking and argument analysis @xcite @xcite . each of these eight base schemes is manually varied in specific ways to create further deductively correct variants, which are verified for correctness using an off-the-shelf theorem prover. negation variants of base schemes are created by substituting a sub-formula with its negation (e.g., f x ¬f 1 x) and/or by applying duplex negatio affirmat. complex predicates variants build on base schemes or their respective negation variants and are obtained by substituting atomic predicates with compound disjunctive or conjunctive ones (e.g., f x f 1 x ∨ f 2 x). de morgan variants of base schemes are finally derived by applying de morgan's law to the respective variants created before (a de morgan variant of modus ponens is, for instance: ∀x : ¬(f x ∨ gx) → hx; ¬f a; ¬ga ⇒ ha). with 2-3 different versions for each of these variations of a base scheme (parameter n in fig. 2 ), we obtain, in total, 71 distinct handcrafted argument schemes. in view of their simplicity and prominence in natural language argumentation, three of the eight base schemes are marked as core schemes: generalized modus ponens, generalized contraposition, hypothetical syllogism 1. natural language instances of the argument schemes can be created by means of a first-orderlogic domain (with names and predicates) and natural language templates for the formal schemes. in order to obtain a large variety of realistic natural language arguments, we have devised (i) a this process can be split into five consecutive steps. in step 1, the argument scheme, which serves as formal template for the natural language argument, is chosen at random. in step 2, each sentence in the formal scheme (premises and conclusion) is individually replaced by a natural language pattern in accordance with a randomly chosen template. for example, the formula "∀xf @xmath0: "every f is a g", "whoever is a f is also a g", "being a g is necessary for being a f", "if someone is a f, then they are a g". some of these patterns (e.g., the fourth one in the above list) are reserved for generating an out-of-domain test dataset, and are not used for training. in step 3, the entity-and property-placeholders in the resulting argument scheme are replaced argument-wise with names and predicates from a domain. we hence obtain an instance of the formal argument scheme as premise-conclusion list. each domain provides hundreds of entity-names, which can be paired with different binary predi-cates to create thousands of different unary predicates. for example, the text in fig. 1 is obtained by substituting predicates from the domain female relatives, which includes predicates like being a "sister of anna", "granddaughter of elsa", "cousin of sarah", . . . once more, some domains are used for testing only, and not for training (see below and section 4.2). in step 4, the premises of the natural language argument are randomly re-ordered. in step 5, the premise-conclusion list is packed into a text paragraph by adding an argument intro, framing the premises, and adding an inference indicator. again, multiple templates are available for doing so, which yields a large variety of textual renderings of an argument. following this pipeline, we generate natural language instances of each formal argument scheme, thus creating: this represents the artificial argument text corpus we use to train and evaluate cript. ## experiments with cript our basis for training and evaluating cript are three compact versions of gpt-2 with 117m, 345m and 762m parameters, as implemented by @xcite . we note that all of these models fall short of the full-scale model with 1542m parameters. in an attempt to avoid over-fitting, we blend the training arguments with snippets from reuters news stories @xcite and the standardized project gutenberg corpus @xcite , trying a mixing ratio of 1:1 and thus doubling training size to @xmath1, 18k, 36k, 72k. 3 training the base model (pre-trained gpt-2) on train01-train03 yields three corresponding cript models (see @xcite . for purpose of comparison, we have similarly trained three randomly initialized transformer models (structurally identical with gpt-2) -none of these random models gains any performance through training on our critical thinking corpus. ## conclusion this paper has taken a first step towards the creation of a critical thinking curriculum for neural language models. it presents a corpus of deductively valid, artificial arguments, and uses this artificial argument corpus to train and evaluate cript -a transformer language model based on gpt-2. as our main finding, we observe strong transfer learning effects/generalization: training cript on a few central core schemes allows it to accurately complete conclusions of different types of arguments, too. the language models seem to connect and to generalize the core argument schemes in a correct way. moreover, cript is equally able to apply learned argument patterns beyond the domain it has been trained on, and there is evidence that generic language modeling skill facilitates the successful generalization of learned argument patterns as randomly initialized models fail to acquire any inference skill by critical thinking pre-training. (accordingly, we expect our approach to scale to even larger versions of gpt-2.) these findings are consistent with previous work on rule reasoning @xcite . moreover, cript has been tested on different reasoning benchmarks. we obtain clear and promising results for the glue ax and snli benchmarks. all this suggests that there exist (learning-wise) fundamental reasoning skills in the sense that generic intermediary pre-training on texts which exemplify these skills leads to spillover effects and can improve performance on a broad variety of reasoning tasks. the synthetic argumentative texts might be a good starting point for building such a "critical thinking curriculum for language models." there are different directions for advancing the approach adopted in this paper and further improving the general reasoning skill of neural language models: in conclusion, designing a critical thinking curriculum for pre-training neural language models seems to be a promising and worthwhile research program to pursue. ity of cript to correctly complete arguments of the corresponding scheme (given the out-of-sample test dataset). reported accuracy values that fall within gray background areas are attained by models which have seen the corresponding scheme during training. vice versa, thick lines on white background visualize model performance on unknown schemes. fig. 5 reveals, first of all, that even the base models (only pre-training, no fine-tuning) display a significant ability to correctly complete conclusions of some kinds of arguments. for example, gpt-2-762m achieves 50% accuracy (split task) in completing contrapositions, 30% accuracy in completing generalized modus ponens, and still 20% accuracy in completing disjunctive syllogism and dilemma arguments. these findings further corroborate the hypothesis that nlms learn (basic) linguistic and reasoning skills "on the fly" by training on a large generic corpus @xcite .