LogiGAN: Learning Logical Reasoning via Adversarial Pre-training
Abstract
We present LogiGAN, an unsupervised adversarial pre-training framework for improving logical reasoning abilities of language models.
Upon automatic identification of logical reasoning phenomena in massive text corpus via detection heuristics, we train language models to predict the masked-out logical statements.
Inspired by the facilitation effect of reflective thinking in human learning, we analogically simulate the learning-thinking process with an adversarial Generator-Verifier architecture to assist logic learning.
LogiGAN implements a novel sequential GAN approach that (a) circumvents the non-differentiable challenge of the sequential
GAN by leveraging the Generator as a sentence-level generative likelihood scorer with a learning objective of reaching scoring consensus with the Verifier; (b) is computationally feasible for large-scale pre-training with longer target length.
Both base and large size language models pre-trained with LogiGAN demonstrate obvious performance improvement on 12 datasets requiring general reasoning abilities, revealing the fundamental role of logic in broad reasoning, as well as the effectiveness of LogiGAN.
Ablation studies on LogiGAN components reveal the relative orthogonality between linguistic and logic abilities and suggest that reflective thinking’s facilitation effect might also generalize to machine learning 2 .
Introduction
“ Learning without thinking is labor lost; thinking without learning is perilous.”
– Confucius
Pre-trained Language Models (PLMs) (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2020) are approaching human-level performance in numerous tasks requiring basic linguistic abilities (Rajpurkar et al., 2016; Wang et al., 2018), setting off a huge wave of interest in Natural Language Processing (NLP).
Despite the emerging fervor, researchers soon realized that PLMs are relatively incompetent in their reasoning abilities, which seems to be an insurmountable bottleneck for PLMs with even better linguistic abilities (Kassner & Schütze, 2019; Helwe et al., 2021).
Following this, researchers delve into reasoning from multitudinous aspects, striving to improve PLMs’ reasoning abilities.
From our perspective, reasoning (in natural language) is essentially an inferential process where an unstated statement is drawn based on several presented statements, and Logic is the systemic set of principles that provides reasoning with correctness and consistency assurance (Hurley, 1982).
Regardless of the variability of contents, logical reasoning generally incorporates two invariant forms: drawing conclusions based on some premises (aka.
deduction & induction, (Reichertz, 2013)), or hypothesizing premises to explain some conclusions (aka.
abduction (Douven, 2021)).
Most existing tasks requiring general reasoning ability, such as natural language inference (Nie et al., 2019) and complex machine reading comprehension (Lai et al., 2017), can be readily interpreted by this criterion.
Other tasks requiring specialized reasoning skills can be considered either as (i) providing sufficient premises but requiring specific ways of premise extraction to draw conclusions, such as multi-hop (Yang et al., 2018b) or hybrid (Chen et al., 2020) reasoning; or (ii) requires external knowledge, such as commonsense (Sap et al., 2020) or numerical (Dua et al., 2019) knowledge, as premises to draw conclusions, hence could also be interpreted by the two forms of logical reasoning.
Following this analysis on the relation between logic and reasoning, Logic ability will be an essential foundation for a broad scope of reasoning, and should be prioritized in improving PLMs’ reasoning abilities 3 .
Conventional pre-training via randomized M asked L anguage M odeling (MLM) and auxiliary tasks are generally developed upon Firth (1957)
’s distributional hypothesis of semantics – "a word is characterized by the company it keeps.
" Under this paradigm, models efficiently learn to capture grammatical structures and contextualized semantics.
However, since logical consistency is beyond correctness on a linguistic level, it is less obvious how MLM could help with logical reasoning abilities.
Do models harvest logic ability for free from MLM?
Or is that something that needs further learning beyond language acquisition?
Motivated by these questions, we propose an unsupervised pre-training method aiming at enhancing the logical reasoning ability of PLMs : we automatically identify occurrences of logical reasoning phenomena in large corpus via detection heuristics, and then require PLMs to predict the masked-out logical statements made in the original context (Section 3).
For example, in the case “Bob recently made up his mind to lose weight.
Therefore , [MASK] ”, the prediction goal is the masked original statement “he decides to go on a diet”.
However, statements different from the original statement could also be logically consistent with respect to a given context.
For example, “he decides to exercise from today on” is also a reasonable inference in the case above.
Since Generators trained merely from recovering original statements are not encouraged to explore the possibilities of other reasonable statements, their overall learning effectiveness of logic could potentially be degraded.
Therefore, it makes sense to provide additional feedback based on the degree of logical consistency between statements predicted beforehand and the original context – we realize this much resembles humans’ reflective thinking process.
Inspired by research from cognitive psychology (Moon, 2013; Boud et al., 2013; Di Stefano et al., 2016) advocating for the vital role of reflective thinking in improving the experiential efficiency of human learning, we hypothesize that machines might also benefit from reflective thinking in their learning processes.
Following this hypothesis, we analogically simulate humans’ learning-thinking process with a Generator-Verifier architecture, and propose LogiGAN , a novel adversarial training approach tailored for sequential GAN training to further facilitate the learning of logical reasoning.
In LogiGAN’s design, the Generator learns not only to recover the original masked statements, but also to score candidate statements (based on their generative likelihood) in a manner that could reach scoring consensus with the Verifier, who learns to make judgments on the logical consistency between premises and conclusions.
The more logically consistent the Verifier thinks of a statement w.r.t.
the input context, the higher generative likelihood score is expected to be assigned by the Generator.
To encourage the exploration of broader possibilities of reasonable statements other than the original one, we also apply a diversified sampling strategy for candidate statement generation.
Both Generator and
Verifier scoring processes are continuous throughout the adversarial training, thereby circumventing the non-differentiable barrier in sequential GAN posed by the discrete beam-search.
Moreover,
LogiGAN does not involve token-wise Monte Carlo Search for policy gradient estimation, and scoring processes of Generator and Verifier are decoupled, so that parallel score computation is possible.
This makes large-scale pre-training with longer target length computationally feasible.
To test the effectiveness of LogiGAN, we extensively experiment on 12 datasets requiring general reasoning ability.
The apparent performance improvements of both base and large size PLMs across all tasks reveal models’ harvest of logic ability, shoring up the fundamental role of logic in general reasoning.
We also carry out ablation studies to understand the functionality of LogiGAN components, the results of which shed light on the relative orthogonality between linguistic and logic ability and suggest that the facilitation effect of reflective thinking is also generalizable to machine learning.
Logic Pre-training
In this work, we primarily focus on improving PLMs’ ability of informal logic (Groarke, 2021).
We include the three most classical types of logical reasoning: deductive, inductive, and abductive reasoning conducted in the form of natural language (Reichertz, 2004; Kennedy & Thornberg, 2018;
Reichertz, 2007; Douven, 2021) in our consideration.
Note that our coverage is broader than the informal logic strictly defined in the philosophy community (Munson, 1976) that primarily focuses on analyzing the soundness and cogency of the application of the aforementioned reasoning in real-life arguments.
The other half of logic investigation – the normative study of formal logic (typically conducted in a symbolic form), such as truth-function logic (Buvac & Mason, 1993), modal logic (Priest, 2008), and fuzzy logic (Dote, 1995), is beyond the scope of this paper.
Logic Indicators as Detection Heuristics.
To set up an unsupervised pre-training aiming at improving models’ logic ability, the very first step will be to identify logical reasoning phenomena from a vast-scale unstructured text corpus.
While invocations of logic are not explicitly stated in most cases, writers’ usage of logic indicators usually marks their logical reasoning processes with high precision (Hurley, 1982), thereby serving as an ideal heuristic device for our detection purpose.
We consider two standard types of logic indicators: (i) conclusion indicator such as “Therefore”, “We may infer that”, which denotes drawing conclusion deductively or inductively from given premises; And (ii) premise indicator such as “Due to”, “The reason that”, which denotes abductively hypothesizing premises that explain or provide evidence to some stated conclusions.
Corpus Construction.
For a text corpus, we detect every occurrence of pre-defined logic indicators (listed in App. C), and mask out (i.e., replace with [MASK] ) the entire statement subsequent to the indicator (each training example will have exactly one masked-out statement).
Then models’ task will be to perform language modeling and predict the masked statement.
We emphasize that statements are declarative sentences or declarative clauses, owning complete subject and predicate structures, and are capable of being factually true or false.
To supply sufficient context information for these predictions, we keep x complete sentences previous to the [MASK] , as well as y sentences after the [MASK] , where x and y can be sampled from a geometric distribution with pre-defined hyper-parameters.
Fig.
1 illustrates the input and output format, and we discuss details in Sec. 4.
Masked Logical Statement Prediction.
In the simplest setting, the Generator learns to infill the masked statement via a single-task pre-training, which fulfills the training process of a typical masked language modeling task.
The only difference is that models no longer predict randomly masked tokens or spans but instead logic-targeted masked complete statements .
Models are trained to perform Max Likelihood Estimation (MLE) for masked statements under a typical teacher forcing loss.
Practically, generative pre-trained language models such as T5 (Raffel et al., 2020) could take up the position of Generator G .
Given a single input context / output statement pair ( c, s ) , the teacher forcing loss can be mathematically expressed as 4 :
[Equation]
The Adversarial Training Framework
Since Generators trained merely from recovering masked original statements miss out opportunities of exploring other reasonable statements, LogiGAN implements an adversarial mechanism for providing
Generators with extra signals based on logical consistency between pseudo-statements (sampled from
Eq. 3) and context to encourage explorations.
The adversarial framework has two major components: (i) a Verifier V that learns to judge logical consistency between statements and context; (ii) a Generator
G that learns both to recover masked original statements, and scores pseudo-statements (based on their generative likelihood) in a manner that could reach scoring consensus with the Verifier – The more logically consistent the Verifier thinks of a statement w.r.t.
the input context, the more likely the
Generator is expected to generate the statement under the input context (i.e., assign higher generative likelihood score).
The overall objective of LogiGAN can be expressed as the minimax objective:
[Equation]
[Figure]
where G θ and V ϕ denote the Generator and the Verifier with model parameters θ and ϕ , respectively.
s + /s − represents ground-truth statements from original text / sampled pseudo-statement 5 .
We discuss sampling details of pseudo-statements later in this section in Eq.
Classical GAN settings (Goodfellow et al., 2014; Zhu et al., 2017) fall short in sequential generation because the gradient propagation from the Verifier to the Generator is blocked by a non-differentiable beam-search during text generation.
Previous approaches such as (Yu et al., 2017) address this challenge by token-wise policy gradient estimation via Monte Carlo Search.
However, since the sampling run-time grows exponentially with the length of the target sequence, their original implementations are not applicable to million-scale pre-training with relatively longer target length as in our scenario.
Different from them, LogiGAN omits the token-wise Monte Carlo Search for policy gradient estimation, and realizes a similar goal via measuring the similarity of scoring distributions between
Verifier and Generator.
The main procedures of LogiGAN can be summarized in four steps: (a) several candidate pseudo-statements are sampled on a sentence level; (b) the Verifier assigns the logical consistency scores V score based on how logical consistent these candidates are w.r.t the original context; (c) the Generator assigns the sentence-level generative likelihood score G score to each candidate to reflect how likely it will produce the pseudo-statement under the given context.
(d) The similarity between Generator and Verifier score distributions is computed as a new signal to encourage the Generator to reach scoring consensus with the Verifier – i.e., the more logically consistent the Verifier thinks of the statement, the higher likelihood score the Generator is expected to assign.
Since both scoring processes are continuous, the non-differentiable barrier is successfully bypassed.
Meanwhile, this design does not involve sequential token-level sampling and decouples
the
Generator and Verifier scoring processes, thereby enabling parallel score computations.
This makes large-scale pre-training with relatively longer target sequence length computationally feasible.
The overall framework overview is illustrated in Fig.
1, and the detailed training procedure is summarized in Algorithm 1.
To diversify the candidate pseudo-statements, we sample pseudostatements from two sources: (i) self-sampling via diversified beam-search from the Generator; or (ii) retrieving similar statements from the corpus, and the sampling process can be summarized as:
[Equation]
where G θ ( .
| c ) denotes self-sampled statement s α given context c from Generator G θ , and R ( s + ) denotes a retriever 6 that retrieves textually similar statements s β with ground-truth statement s + from the corpus.
Note that this process is conducted separately for the corpus of Verifier and Generator.
3.1 Training of Verifier
The Verifier serves as a critic to judge whether a statement is logically consistent w.r.t.
the context.
Therefore, the training task of Verifier can be viewed as a binary classification problem.
Pre-trained language models that could perform discriminative classification tasks such as BERT (Devlin et al., 2018), ALBERT (Lan et al., 2019), and RoBERTa (Liu et al., 2019), will be suitable for the role of
Verifier.
With y = 1 for both ground-truth and logically consistent pseudo-statements, and y =for other pseudo-statements, the binary classification loss for a single pair of context/statement/label ( c, s, y ) of Verifier can be mathematically expressed as (omitting average):
[Equation]
3.2 Training of Generator
The Generator targets both to recover the original masked statements, and to score pseudo-statements based on their generative likelihood in a manner that could reach sentence-level scoring consensus with the Verifier.
This corresponds to the two sources of learning signals received by the Generator: (i) the original generative objective with teacher forcing loss defined in Eq.1 as a signal; and (ii) the distribution similarity between sentence-level generative likelihood score assigned by Generator and logic consistency score assigned by Verifier.
To achieve the goal of (ii), we first sample pseudostatements { s − 1 , ..., s − n } from p neg ( .
| G θ , c, s + ) .
Then the Verifier assigns logical consistency score
V score based on how logically consistent the pseudo-statements are w.r.t.
the context, expressed as:
[Equation]
After this, the Generator assigns a sentence-level generative likelihood score G score for each pseudostatement to reflect how likely the pseudo-statement will be produced under the given context:
[Equation]
where ℓ θ ( s | c ) is the accumulated log-likelihood of the statement s conditioned on the context c .
Afterward, each statement with a high Verifier score V ϕ ( s, c ) is also expected to receive a high generative score ℓ θ ( s | c ) to facilitate the Generator’s capturing of the Verifier’s judgment criterion based on logic consistency.
KL-divergence (Kullback & Leibler, 1951)
D KL is therefore a appropriate measure for the similarity between the score distribution of V score and G score .
For the purpose of smoothing the gradient to stabilize the GAN training process, we gather both the ground-truth
[Figure]
F r e qu e n c y F r e qu e n c y
(learned with teacher-forcing loss) and pseudo statements (learned with KL loss) inside the same batch w.r.t.
a single input context c .
In our case, there is exactly one ground-truth statement and n pseudo-statements for each input context c .
For a batch of ( c ; s + , s − 1 , ..., s − n ) , the overall objective of the Generator is defined as (in App.
F we show how Eq.
7 commits to the optimization of Eq. 2):
[Equation]
Experiment Setup 4.1 Datasets
To test the effectiveness of LogiGAN, we extensively experiment on 12 datasets requiring reasoning via natural language.
Specifically, ReClor (Yu et al., 2020), LogiQA (Liu et al., 2021a), Adversarial
NLI - ANLI, (Nie et al., 2019), focuses especially on logical reasoning, TellMeWhy (Lal et al., 2021) on abuductive reasoning, HotpotQA
(Yang et al., 2018a) on multi-hop reasoning, QuoRef (Dasigi et al., 2019) on reasoing with co-reference resolution, MuTual (Cui et al., 2020), DREAM (Sun et al., 2019)), SAMSum (Gliwa et al., 2019) on reasoning in conversational scenarios, and NarrativeQA (s Koˇ ciský et al., 2018), RACE (Lai et al., 2017),
XSum (Narayan et al., 2018) on general verbal reasoning.
These datasets make most, if not all, necessary premises for drawing logically consistent conclusions available in their provided context, and require few external premises like commonsense or numerical knowledge.
Hence, they fit nicely for testing our hypothesis that LogiGAN brings PLMs logic ability beyond their intrinsic linguistic ability, which could benefit general reasoning processes.
4.2 Pre-training Corpus
We apply the corpus construction methodology (§ 2) on the widely used BookCorpus (Kobayashi, 2018), which consists of e-books and movies with topics crawled from general domains.
Although some corpus featuring debates and arguments (Walker et al., 2012; Abbott et al., 2016; Swanson et al., 2015) appears to be more suitable for our emphasis on logic, we do not elect them due to their high domain specificity in fields such as politics, law, and economics.
We discard overly short statements and instances where indicators do not indicate logical reasoning (e.g., “ since 2010” indicating a time point rather than premises, “ so happy” indicating degree of the subsequent adjective rather than conclusions).
This results in 3 .
14 million ( 1 . 43 and 1 .
71 million from conclusion and premise indicators, respectively) instances.
Corpus statistics are visualized in Fig.
4.3 Models
Baseline Choice.
Since our primary goal of the experiment is to test the effectiveness of LogiGAN and test our hypothesis that logic ability can be further enhanced beyond PLMs’ intrinsic linguistic ability, we only compare models pre-trained with LogiGAN against their vanilla versions.
After
LogiGAN pre
-training, we discard the auxiliary Verifier (discussed in Sec. 6) and employ the
Generator only to solve all downstream tasks in a purely end-to-end manner.
For our main experiments, we initialize Generators from both base and large size pre-trained T5 (Raffel et al., 2020), and Verifier from pre-trained ALBERT-large
(Lan et al., 2019).
We leave discussions of the rest implementation details and hyper-parameter settings of pre-training and downstream fine-tuning in Appendix D.
Elastic Search vs. Self Sampling.
As stated earlier in section 3.2, candidate pseudo-statements have two possible sources – they could either be sampled via beam search from the Generator’s selfdistribution, or could be retrieved from some external resources.
We carry out two variant versions of LogiGAN whose Generator is trained purely from self-sampled sentences as pseudo-statements ( LogiGAN base (ss) ), and from extra pseudo-statements retrieved from corpus by Elastic Search
Gormley & Tong (2015) ( LogiGAN base (ss+es)).
For the large model, we use LogiGAN large (es+ss) as default.
Our database consists of 3.14 million sentences discovered by the corpus construction process, and we keep the top-5 similar retrieved sentences along with self-samples from Generator.
Experiments 5.1 Experimental Results
[Table]
[Table] [Table]
As presented in Table
[Table]
both base and large size PLMs further pre-trained with LogiGAN surpass their vanilla baselines across both discriminative and generative task formats, through a wide scope of downstream tasks requiring general reasoning abilities.
We can make the following observations:
Among all observed improvements, those on tasks with particular emphasis on logic (ReClor, LogiQA, and ANLI) are most noticeable.
These positive results manifest the effectiveness of LogiGAN in injecting logic ability into PLMs, while testifying to our primary hypothesis that logic ability is fundamental to general reasoning as well.
This conclusion answers the two questions in the intro section 7 , suggesting that randomized MLM pre-training might fall short in endowing language models with logic ability, and a logic-targeted pre-training approach like LogiGAN may further assist logic learning beyond language acquisition.
Furthermore, extra retrieved pseudo-statements (ss+es) bring some additional performance improvement compared with the pure self-sampling (ss)
LogiGAN variant, revealing the important role of pseudo-statements’ diversity in adversarial training.
5.2 Ablation Study and Analysis
Observing the apparent performance enhancement, we now aim at pinpointing the truly functional components of LogiGAN through ablation studies and deriving the origins of observed improvements.
For fair comparison purposes, we hold all pre-training and downstream settings (including hyperparameters, implementation designs, and evaluations) unchanged from full LogiGAN.
All variations are initialized from T 5 base , and we report performance variance on 7 datasets.
[Table]
I. Random Masked Sentence Prediction Pre-training.
To explain the observed improvements, our first hypothesis is: Models harvest extra linguistic ability from masked statement prediction compared with masked token (or span) prediction.
Quite intuitively, filling entire sentences with complete subject-predicate structures might put additional demands on models to capture more abundant syntactic information beyond the coverage of masked token (or span) prediction.
Since LogiGAN involves recovering masked sentences , it is then necessary to determine to what degree, if any, that the observed performance gain is attributable to models’ plausible linguistic ability improvement.
We therefore carry out a variant pre-training where the prediction objects are randomly masked sentence .
Results (shown in Table 2) displays that masked sentence prediction training barely brings improvement against the vanilla baseline.
This suggests it is unlikely that masked sentence prediction empowers PLM trained from masked token prediction significantly better linguistic ability, nor likely that the extra pre-training corpus per se significantly raises the performance.
Therefore, we reject the first hypothesis and conclude that observed improvements should derive from somewhere else.
II.
MLE-only Logic Pre-training.
Our second hypothesis is that logic-guided masked statement prediction enhances models’ intrinsic ability of logical reasoning, thereby lifting the downstream performance.
Having addressed the potential impact of learning randomized complete sentence generation, we next aim to check how learning logic-targeted statement generation affects models’ behavior.
We ablate the entire adversarial training process, and train models to perform maximum likelihood estimation (MLE) with teacher-forcing loss only on masked-out logical statements.
Results 2 of MLE-only logic pre-training reveals quite a notable improvement across almost all datasets against both vanilla baseline and I., suggesting that learning to generate logical statements indeed injects extra abilities into the model.
Since results of I. eliminate the possibility that models harvest stronger linguistic abilities from complete sentence prediction, it is safe to partially ascribe the better downstream performance to models’ enhanced ability in modeling logical reasoning.
This reveals the relative orthogonality between logic ability and models’ inherent linguistic ability, suggesting that logic ability could be enhanced through further logic-targeted pre-training.
III.
Iterative Multi-task Pre-
training.
Since II.
only partially explains the observed improvements, here is our last hypothesis: the adversarial training procedure of LogiGAN explains the unexplained rest part beyond the coverage of II.
Here a multi-task pre-training with both generation and verification tasks will be the most natural intermediate setting between the single-model generation-only setting of II.
and LogiGAN’s dual-model adversarial setting .
However, since the verification task relies on
Generator’s self-sampled statements, we adopt an iterative self-critic pre-training manner following
Nijkamp et al.
Unlike typical multi-tasking training that simultaneously carries different tasks and then sums the losses, our generation and verification tasks happen alternately 8 .
Surprisingly, the iterative multi-task pre-training barely brings any positive effects to models’ downstream performance compared with II.
One possible explanation for this might be that the drastically different mechanisms between the verification and generations task intervene with each other, making the single-model & multi-task setting non-beneficial.
Now that we have confirmed that an extra verification task fails to explain the rest improvement, we can accept our final hypothesis and conclude that it is indeed the adversarial mechanism between the Generator and Verifier that truly facilitate learning of logical reasoning, thereby further improving the downstream performance beyond II.
Discussion
A Psycholinguistic Interpretation of Logic-oriented MLM From the first glance, the idea of Logic-oriented MLM seems to be naive and simply. However, we argue that to fully appreciate the value and potential of logic-oriented MLM, going beyond the superficial appearance of masked text and touching down to their underlying psycholinguistic essence is necessary. It is neither the linguistic patterns of the masked-out text, nor the masking technique to corrupt them that makes logic-oriented MLM (and its potential follow-ups) unique.
What truly matters is the distinctive cognitive processes proceeding in the minds of writers when they put down different pieces of text – language is a window into human minds (Pinker, 2007).
Consider the following examples when a human is filling in the [MASK]’s:
(1) “19 + 69 = [MASK].
" (Numerical cognition).
(2) “Windows is founded by [MASK].”
(Declarative memory retrieval).
(3) “Socrates is a mortal, so he will eventually [MASK]” (Logical reasoning).
(4) “If I feed my dog more than 2 treats per day, it will get [MASK].”
(Causal inference).
A crow immediately stands out in swans because it’s [MASK]."
(Common sense reasoning).
(6) “Mike deeply bows to his teacher to show his [MASK].”
(Social perception).
Though answers to these [MASK]’s are similar in string length, they nevertheless involve different information pathways and substantially distinctive cognitive processes in writers’ minds.
Logic-oriented MLM shines in that it consistently captures exactly one type of such cognitive process, and trains LMs to model humans’ logic reasoning mechanism, which is well beyond modeling language per se.
On the contrary, Randomized MLM does not capture consistent underlying cognitive processes, which could significantly lower LMs’ efficiency of learning advanced intelligence mechanisms beyond language itself.
The empirical effectiveness of Logic-oriented MLM provides positive evidence for the practicability of this paradigm, suggesting that LMs might be able to learn various advanced human cognitive processes other than logical reasoning via a similar approach.
Combined with LogiGAN’s natural analogy of humans’ learning-thinking mechanism during logic development, LogiGAN made an encouraging attempt to unify cognitive modeling and language model pre-training.
Adversarial Training Might Assist Downstream Generation Tasks.
Although in our experiments, we discard the Verifier and solve downstream tasks with the Generator only, some previous works (Shen et al., 2021; Cobbe et al., 2021) reveal that the Verifier can be used for ranking multiple generation results, thereby effectively enhancing overall downstream accuracy.
However, in their paradigm, the information propagates unidirectionally from the Generator to the Verifier, and the
Generator cannot directly benefit from the Verifier’s discriminative feedback.
In contrast, our LogiGAN adversarial training paradigm surmounts the non-differentiable obstacle and could potentially enlighten a new paradigm of both pre-training and downstream fine-tuning.
Improving Logical Pre-training.
Our paper demonstrates that PLMs’ logic ability can be further enhanced beyond their inherent linguistic ability, and adversarial training may bring extra benefits beyond the learning of logic-targeted masked statement prediction.
However, our heuristic-based approach to identifying logical phenomena in a text corpus and the single mask prediction setting can be further improved.
Logic recognition methods with higher recall and better unsupervised task designs (e.g., logical indicator prediction , or logic-guided sentence shuffling ) are worthwhile to explore in the further work.
Besides, since we are adopting a general domain pre-training corpus (i.e.,
BookCorpus ) with bare emphasis on logic, understanding the impacts of extending pre-training to the domain-specific corpus (e.g., law corpus) or others emphasizing logical reasoning is also substantial.
Related Works
Generative Adversarial Training in NLP.
Unlike conventional GAN (Goodfellow et al., 2014;
Mirza & Osindero, 2014; Zhu et al., 2017) that generates continuous output such as images, sequential
GAN generates discrete sequences via non-differential searches.
This makes feedback from the discriminator not propagatable to the generator.
To tackle this challenge, SeqGAN (Yu et al., 2017) borrows an idea from reinforcement learning, treating each output token as a single action, and estimates token-wise policy gradient via Monte Carlo search.
RankGAN (Lin et al., 2017) adopts a similar approach but breaks the binary-classification assumption of discriminator task design, and a ranker provides feedback to the generator.
Their generator attempts to generate verisimilar sentences to deceive the ranker into ranking synthetic sentences higher over multiple human-written ones.
In our scenario, however, the gold ranking is hard to determine because measuring which statements are more logically consistent w.r.t.
context than others is non-trivial, and multi-gold cases are possible.
While successfully enabling communication between generator and discriminator, the original designs of SeqGAN, RankGAN, as well as other works such as (Guo et al., 2017; Fedus et al., 2018; Caccia et al., 2018; Rekabdar et al., 2019), generally formulate text generation as a sequential action decision problem, thereby involving heavy sampling for policy gradient estimation, and are sensitive to the length of the target sequence.
Since large-scale pre-training (with arbitrary target length) puts a high demand on scalability and computational efficiency, the above approaches are not readily applicable in our scenario.
Furthermore, previous work leverages adversarial training to improve qualities of generated examples , whereas our focus is on enhancing models’ intrinsic logic ability .
A recent work, AR2 (Zhang et al., 2021), leverages adversarial training to improve dense document retrieval.
With a retriever-ranker architecture, the learning objective of retriever is to maximize the agreeableness between its own score assignment and that of the ranker for input documents.
This is conceptually similar to LogiGAN, as our Generator also aims at reaching consensus with
Verifier.
However, AR2 does not fall into the sequential GAN paradigm, since it does not involve any sequential text generation, and there is no non-differentiable barrier between the retriever and ranker.
Pre-training for Reasoning Ability Improvement.
Previous works have extensively investigated the possibility of injecting specific type of reasoning via pre-training, such as numerical (Geva et al., 2020; Yoran et al., 2021; Pi et al., 2022), commonsense (Zhong et al., 2019; Tamborrino et al., 2020; Staliunaite et al., 2021), formal logic (Wang et al., 2021; Pi et al., 2022), multi-hop
(Deng et al., 2021; Zhong et al., 2022), and tabular (Liu et al., 2021b) reasoning.
Different from them, LogiGAN focuses on logic reasoning, which plays a fundamental role in general reasoning via natural language.
Conclusion
In this work, we hypothesize that (i) logic ability plays a key role in a wide scope of tasks requiring general reasoning; and (ii) PLMs’ logic ability can be further improved beyond their original linguistic ability.
We correspondingly propose LogiGAN, an unsupervised adversarial pre-training framework for logical reasoning enhancement.
LogiGAN circumvents the non-differentiable challenge of sequential GAN via a novel Generator-Verifier scoring consensus mechanism, and enables largescale pre-training with longer target length.
Extensive experiments and ablation studies reveal the effectiveness and functional components of LogiGAN, providing evidence to our major hypothesis.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2. If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A] (b) Did you include complete proofs of all theoretical results? [N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We will release the code and data upon acceptance.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No] (d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Thinking Straight about Reasoning in NLP
The following argument is tentative and is deduced with bounded rationality, and is for communication purposes only .
We realize and well acknowledge that researchers observing different sets of evidence could hold fundamentally different but reasonable views from ours.
A.1 Conundrums of Reasoning
Along with the increasing interest in reasoning, multiple reasoning terms are proposed – hybrid reasoning, commonsense reasoning, numerical reasoning, multi-hop reasoning, and unspecified general reasoning, to name a few.
However, among all these scattered and distinctive types of reasoning, what is varying?
What essence remains constant regardless of the variety of forms?
If we were to arrange these reasoning into a hierarchical structure much like a biological taxonomy or to group them into categories, what standard should we follow?
How should we put boundaries between each type of reasoning?
To the best of our knowledge, few works from the NLP community articulated these queries well.
There seems to be a conceptual conundrum of reasoning.
Moreover, the ethereal and shapeless nature of reasoning makes it not as visible or concrete as tokens or spans that are readily accessible to the masked language modeling pre-training paradigm.
How could we then systematically inject reasoning ability into models via pre-training?
What is the nature of this ethereal ability are we truly pursuing?
More fundamentally, is pre-training the correct way to add reasoning abilities into language models?
Should reasoning abilities be acquired during the pre-training stage, or should it be subsequently tackled by outsourcing symbolic modules (i.e., with neural-symbolic models)?
There seems to be a methodological conundrum of reasoning.
These questions are non-trivial to answer, and these morals are hard to tell.
However, it is quite unlikely that we can solve the reasoning challenge before articulating what the challenge truly is.
A.2 Reasoning in our Sense
As defined in the introduction section, reasoning (via natural language) is an inferential process where an unstated statement is drawn based on several presented statements.
Specifically for deductive and inductive reasoning, a conclusion is drawn from provided premises under the guidance of logic.
Here is an example of the most trivial cases of such reasoning: given premises “Bob’s daughter is called
Lily.
Lily is now 3 years old.”, concluding “Thus, Bob’s daughter is 3 years old.” requires shallow synthesis between exactly-matched subject and predicate terms.
However, in complex machine reading comprehension tasks, such single-step synthesis can be arbitrarily long-chained (e.g., requires 5 syntheses, and combined with semantic-invariant linguistic transformations such as synonym replacements and syntactic transformations.
As a result, reasoning over long and rhetorically sophisticated articles becomes non-trivial, demanding highly on both linguistic and logic abilities.
Apart from linguistic transformations, synthesizing among statements usually requires specific synthesis rules of statements.
For example, the synthesis rule of degree comparison “Elephants are larger than dogs.
Tigers are larger than dogs.”
→ “Elephants are larger than dogs.” does not apply to the case “In Pokemon, the water type is strong against the fire type, and the fire type is strong against the grass type.”
since the conclusion is guided by the same rule “Therefore, the water type is strong against the grass type.”
fallaciously contradicts reality.
Putting into broader domains, mathematics (e.g., arithmetic operators, set operators), formal logic (e.g., quantifiers, logic operators), and most artificial symbolic systems implement their own synthesis rules as standards of correctness.
Therefore, while the forms of reasoning remain relatively invariant, the underlying synthesis mechanism could be drastically different from system to system.
Among all these symbolic reasoning, the reasoning in
NLP focuses primarily on reasoning via natural language – i.e., linguistics as synthesis rules.
A.3 General Reasoning and Specialized Reasoning
Following the definition of reasoning we make in the previous subsection, we are now able to tentatively categorize all investigations of reasoning in NLP into two families: Specialized Reasoning and General Reasoning.
We discuss them separately below:
I. Specialized Reasoning can be further divided into sub-categories:
(a) Reasoning requiring special way of premise extraction , such as multi-hop reasoning (Yang et al., 2018b), tabular reasoning (Zhong et al., 2020), and hybrid reasoning (Chen et al., 2020).
The foremost assumption in this scenario is that the input context has already provided all premises necessary to draw targeted conclusions.
If we humans are trying to answer a question correctly (i.e., with both correct answers and reasons), we will have to first seek back-and-forth across multiple documents or paragraphs (multi-hop) or rows and columns (tabular) to extract premises for answering this question.
Based on these extracted premises, we then follow the logic rules and synthesize a conclusion from these premises to answer the question.
Not drastically different from humans, in such complex reasoning scenarios where spurious patterns are mostly unreliable, machines will also have to identify the necessary premises correctly to reach correct conclusions.
With all relevant and necessary premises extracted, the rest of the reasoning are reducible to general reasoning described below in II.
(b) Reasoning requiring external premises , such as numerical reasoning (Dua et al., 2019), symbolic reasoning (Zhong et al., 2021), domain-specific reasoning (Wang et al., 2022; Gao et al., 2021), and commonsense reasoning (Sap et al., 2020).
This family of reasoning requires external knowledge that is harder to be acquired via typical language modeling pre-training.
Here we emphasize that knowledge can either be declarative or procedural , following the theory from psychology (Ten Berge & Van Hezewijk, 1999; Ullman, 2001).
For example, the commonsense knowledge “There aredays in a year on earth.”, or knowledge of historical events can be primarily declarative (i.e., can be articulated with language).
Other not readily articulable knowledge, such as swimming, and performing complex arithmetics by hand (e.g. 1969 ∗ 331 ) are primarily procedural.
In fact, humans need extra learning to acquire specialized knowledge.
For example, we do not acquire knowledge of mathematics or domain expertise for free from language learning.
Just as “import
PyTorch” is a necessary dependency to implement our neural models in Python, we have to implicitly invoke established mathematics rules to perform calculations, and invoke domain-specific knowledge or laws as premises to make scientific arguments.
Machines are generally no different.
One special type of premise is commonsense, a large set of knowledge that human effortlessly obtains from multi-modal life experiences.
Suppose all necessary premises are readily invoked and stated in natural language, the rest of the reasoning proceeds just as general reasoning, which is discussed below in II.
II.
General Reasoning General reasoning covers a broad unspecified form of reasoning that involves recognizing and understanding relevant concepts framed in plain text as premises and then synthesizing them into conclusions.
Typically, it assumes stated premises are mostly self-contained (i.e., sufficient for drawing certain conclusions), and require little or no external knowledge to be additionally invoked beyond the given context.
Moreover, since premises are presented in a plain text format, general reasoning does not require special ways of extracting premises from context (e.g., structured data such as tables).
The general reasoning is pervasive in tasks emphasizing natural language understanding (NLU), such as machine reading comprehension (Gao et al., 2020) and natural language inference (Nie et al., 2019).
General reasoning also serves as an underlying foundation of other specialized reasoning tasks, since most of them can be conditionally reducible to general reasoning, as discussed earlier.
During general reasoning processes, premises can be organized into a logic chain via procedures such as alignment of semantically similar concepts, understanding relations among sentences, and synthesizing sub-conclusions from specific subsets of premises.
Logic is the systemic set of principles created for providing correctness assurance to conclusions inferred following such logic chains, upon examination of coherence and consistency of these chains 9 .
Invocation of logic is therefore necessary for reasoning to be correct (i.e., drawing correct conclusions from correct reasons) – although humans and machines usually carry this out implicitly.
Conclusively, logic is the core engine for general reasoning.
This completes our analysis on logic and reasoning from the introduction section.
Some Comments on our Categorization Regarding I-(b) , we realize it might be less intuitive or even controversial to consider procedural knowledge as “premises”.
In contrast with crystallized declarative knowledge that resembles more of String type variables in computer programs, fluid procedural knowledge resembles more of functions – although functions have well-defined function bodies while procedural knowledge is usually beyond words.
Therefore, tasks requiring external declarative knowledge (e.g., commonsense, domain-specific expertise) and procedural knowledge (e.g. symbolic calculations formal logic, arithmetics) might be further divided into two sub-categories.
Apart from the above, our coverage bounds to deductive and inductive reasoning via natural language.
Abductive reasoning is the reverse reasoning process where premises are hypothesized to explain or support some stated conclusion, and symbolic reasoning is beyond the coverage of linguistics.
B Bridging the Gap between Pseudo and Logical Inconsistency
As mentioned in the footnote of Sec.
3 (The Adversarial Training Framework) from the main text, there is a gap between pseudo-statements that are either self-sampled from the Generator or retrieved and logically inconsistent statements.
i.e., there might be logically consistent statements that should have received a positive label when training the Verifier assigned with a negative label since they are pseudo.
This could potentially introduce noise signals in the training of the Verifier.
To bridge this gap, we propose a trick that leverages off-the-shelf Natural Language Inference (NLI) models to make judgment on the textual entailment between the pseudo-statement and the groundtruth statement.
Our basic intuition here is that: suppose a pseudo-statement implies or is implied by the ground-truth statement, then this pseudo-statement is also expected to be logically consistent w.r.t.
the original context, just as the ground-truth one.
For example, in the example of Fig.
1 (LogiGAN Overview), with context, “ All men are mortal, and
Socrates is a man.
Therefore , [MASK] .”, the ground-truth statement “ Socrates is mortal. ” implies the self-sampled pseudo-statement “ Socrates will eventually die .”
from Generator, vice versa.
Hence this pseudo-statement should receive a positive label (i.e., y = 1 ) for training Verifier, who learns to discriminate logical consistency of statements w.r.t.
input context, instead of merely capturing fake examples.
In contrast, the first retrieved pseudo-statement “ a mortal can never be a god. ”
does not entail, nor is entailed by, the ground-truth statement.
Therefore this pseudo-statement will have a negative label for training Verifier.
Technically, we employ “ynie/albert-xxlarge-v2-snli_mnli_fever_anli_R1_R2_R3-nli” from Huggingface, a well-trained NLI model (denoted as F ), to determine the textual entailment relationship between ground-truth and pseudo-statements.
The process can be formally expressed as:
[Equation]
where e ( s + , s − ) represents the entailment score between the ground-truth statement s + and a pseudostatement s − .
To determine the final label for training Verifier, we set a hard threshold of 0 .
50 – above results in y = 1 below turns to y = 0 .
According to our statistical study, this extra NLI mechanism flips around 12% of the pseudo-statements (whose default labels are negative) to y = 1 .
With our human evaluation, the flipped pseudo-statements are indeed logically consistent w.r.t.
the original context in most cases.
As an emphasis, the NLI model and the Verifier are making judgment on different things.
The NLI does not consider the original context and merely judges the entailment of a statement pair, whereas the Verifier judges one statement’s logical consistency w.r.t.
the context at a time.
The signal from the NLI serves noisy and indirect supervision in Verifier’s learning process, and we are leveraging the intrinsic denoising ability of large pre-trained language models.
Ablation of this mechanism results in a minor performance drop, but not significant enough to include in our main ablation study, so we are omitting this in the main text.
C List of Logic Indicators
Conclusion Indicators (41 in total): therefore, thereby, wherefore, accordingly, we may conclude, entails that, hence, thus, consequently, we may infer, it must be that, whence, so that, so, it follows that, implies that, as a result, it can be inferred that, suggests that, can conclude, proves that, it can be shown, as a conclusion, conclusively, which implies that, for that reason, as a consequence, on that account, that being said, in conclusion, to that end, for this reason, on account of, because of this, that being so, because of that, ergo, in this way, in this manner, in such a manner, by such means.
Premise Indicators (20 in total): since, on account of, considering, because of, because, due to, now that, in order, as indicated by, because, may be inferred from, given that, owing to, by virtue of, owing to, on account of, in view of, for the sake of, thanks to, reason that.
D Implementation Details
LogiGAN Details.
We randomly sample 2 million and 0.5 million for source training examples of
Generator and Verifier, respectively (i.e., M = 2 million, N = 0 .
5 million in Algorithm 1 in the main text).
We partition the source Generator training corpus into to two 1 million subsets for warmup and
GAN training (i.e., M α = M β = 1 million).
In each iteration of GAN training, 10% or 0 .
05 million and 0 .
1 million examples are sampled for Verifier and Generator training (i.e., m = 0 .
05 million, n = 0 .
1 million).
The warm-up epoch E is set to 5 , whereas the max number of GAN iteration is set to 10 .
For our main experiments, we initialize the Generator from pre-trained “google/t5-v1_1-base” or “google/t5-v1_1-large”, and the Verifier as “albert-large-v2” from HuggingFace (Wolf et al., 2020).
We use 8 V100 GPUs for model training, and set the maximum iterations of adversarial training as 15, and set batch size as 8 and 64 for Generator and Verfier training.
During Generator training, we put both the only one ground-truth statement, and 5 candidate statements for each instance within the same batch.
By default, we adopt learning rate as 5e-5, 1e-5 for the training of Generator and Verifier during adversarial process, respectively.
Downstream Details.
Our tested downstream datasets mainly belong to two types: generation-based datasets (like extractive QA, abstractive QA, summarization, etc.), and classification datasets (like natural language inference, and multiple-choice QA).
We introduce how we process the inputs and outputs for each downstream tasks and the hyper-parameters for fine-tuning.
For the generation-based datasets, we adopt simple hard prompts to write the task input.
For example, for the generative QA task, we formulate the input with “ The question is: {question}.
The context is: {context}, please give the answer ”.
For the classification datasets, with the context (optionally question and options) given as the inputs, the target sequence is one of the options, like “entailment” for NLI datasets or one candidate answer for MRC datasets.
For example, for the multiple-choice
QA task, we prompt the input as “ The question is: {question} .
The options are: {options}.
The context is: {context}, please select the best option.”, where the output is the specific content of the option.
We make the final choice of option by selecting the option with the highest text similarity score calculated by the model output and the context of each option.
During fine-tuning, we don’t perform exhaustive parameter searches for each task.
We adopt the learning rate as either 1e-4 or 5e-5 for each task, depending on which one will lead to stable performance.
We adopt 8 V100 GPUs for fine-tuning and set batch size as 32 for the base model, and 8 for the large model. [Table]
E Few-shot Experiments
As a supplementary experiment, we also explore LogiGAN’s impact under a data-scarce setting.
For each dataset, we randomly select 32 samples and fine-tune models until convergence.
To decouple the complete sentence generation and logic learning in the logic-targeted masked statement prediction process, we also add the random sentence pre-training setting into the comparison.
This could reveal to what extent the performance variance is explained by the linguistic logic ability improvement separately.
Results of provide more evidence for our hypothesis that logic ability and linguistic ability have non-overlapping components.
F LogiGAN Training with Policy Gradient
This is to show the overall optimization goal of the adversarial training (Eq. 2 in the main):
[Equation]
is reducible to the optimization problem of KL-divergence in Generator training in (Eq. 7).
First of all, we can discard the irrelevant terms of Eq. 2:
[Equation]
Since the sampling process of s − is discrete, we cannot directly optimize the G ∗ with gradient descent.
Following previous works in Sequential GAN [Equation] we apply the policy gradient approach.
[Equation]
which is equivalent to minimizing L gen as decribed in Eq.
7 from the main text.