LogiGAN: Learning Logical Reasoning via Adversarial Pre-training
Abstract
We present LogiGAN, an unsupervised adversarial pre-training framework for improving logical reasoning abilities of language models.
Upon automatic identification of logical reasoning phenomena in massive text corpus via detection heuristics,we train language models to predict the masked-out logical statements.
Inspiredby the facilitation effect of reflective thinking in human learning, we analogicallysimulate the learning-thinking process with an adversarial Generator-Verifier architecture to assist logic learning.
LogiGAN implements a novel sequential GANapproach that (a) circumvents the non-differentiable challenge of the sequential
GAN by leveraging the Generator as a sentence-level generative likelihood scorerwith a learning objective of reaching scoring consensus with the Verifier; (b) iscomputationally feasible for large-scale pre-training with arbitrary target length.
Both base and large size language models pre-trained with LogiGAN demonstrateobvious performance improvement on 12 datasets requiring general reasoningabilities, revealing the fundamental role of logic in broad reasoning, as well asthe effectiveness of LogiGAN.
Ablation studies on LogiGAN components revealthe relative orthogonality between linguistic and logic abilities and suggest thatreflective thinking’s facilitation effect might also generalize to machine learning 1 .
Introduction
“ Learning without thought is labor lost; thought without learning is perilous.”
– Confucius
Pre-trained Language Models (PLMs) (Devlin et al., 2018; Brown et al., 2020; Raffel et al., 2020) areapproaching human-level performance in numerous tasks requiring basic linguistic abilities (Wanget al., 2018; Rajpurkar et al., 2016), setting off a huge wave of interest in Natural Language Processing(NLP).
Despite the emerging fervor, researchers soon realized that PLMs are relatively incompetentin their reasoning abilities which seems to be an insurmountable bottleneck for PLMs with evenbetter linguistic abilities (Helwe et al., 2021; Kassner & Schütze, 2019).
Following this, researchersdelve into reasoning from multitudinous aspects, striving to improve PLMs’ reasoning abilities.
From our perspective, reasoning (in natural language) is essentially an inferential process wherean unstated statement is drawn based on several presented statements, and Logic is the systemicset of principles that provides reasoning with correctness and consistency assurance (Hurley, 1982).
Regardless of the variability of contents, logical reasoning generally incorporates two invariant forms:drawing conclusions based on some premises (aka.
deduction & induction, (Reichertz, 2013)), orhypothesizing premises to explain some conclusions (aka.
abduction (Douven, 2021)).
Most existingtasks requiring general reasoning ability, such as natural language inference (Nie et al., 2019) andcomplex machine reading comprehension (Lai et al., 2017), can be readily interpreted by this criterion.
Other tasks requiring specialized reasoning skills can be considered either as (i) providing sufficientpremises but requiring specific ways of premise extraction to draw conclusions, such as multi-hop(Yang et al., 2018b) or hybrid (Chen et al., 2020) reasoning; or (ii) requires external knowledge, suchas commonsense (Sap et al., 2020) or numerical (Dua et al., 2019) knowledge, as premises to drawconclusions, hence could also be interpreted by the two forms of logical reasoning.
Following thisanalysis on the relation between logic and reasoning, Logic ability will be an essential foundation fora broad scope of reasoning, and should be prioritized in improving PLMs’ reasoning abilities 2 .
Conventional pre-training via randomized M asked L anguage M odeling (MLM) and auxiliary tasksare generally developed upon Firth (1957)’s distributional hypothesis of semantics – "a word ischaracterized by the company it keeps.
" Under this paradigm, models efficiently learn to capturegrammatical structures and contextualized semantics.
However, since logical consistency is beyondcorrectness on a linguistic level, it is less obvious how MLM could help with logical reasoningabilities.
Do models harvest logic ability for free from MLM?
Or is that something that needs furtherlearning beyond language acquisition?
Motivated by these questions, we propose an unsupervisedpre-training method aiming at enhancing the logical reasoning ability of PLMs : we automaticallyidentify occurrences of logical reasoning phenomena in large corpus via detection heuristics, andthen require PLMs to predict the masked-out logical statements made in the original context (Section3).
For example, in the case “Bob recently made up his mind to lose weight.
Therefore , [MASK] ”, theprediction goal is the masked original statement “he decides to go on a diet”.
However, statements different from the original statement could also be logically consistent withrespect to a given context.
For example, “he decides to exercise from today on” is also a reasonableinference in the case above.
Since Generators trained merely from recovering original statementsare not encouraged to explore the possibilities of other reasonable statements, their overall learningeffectiveness of logic could potentially be degraded.
Therefore, it makes sense to provide additionalfeedback based on the degree of logical consistency between statements predicted beforehand andthe original context – we realize this much resembles humans’ reflective thinking process.
Inspiredby research from cognitive psychology (Di Stefano et al., 2016; Moon, 2013; Boud et al., 2013)advocating for the vital role of reflective thinking in improving the experiential efficiency of humanlearning, we hypothesize that machines might also benefit from reflective thinking in their learningprocesses.
Following this hypothesis, we analogically simulate humans’ learning-thinking processwith a Generator-Verifier architecture, and propose LogiGAN , a novel adversarial training approachtailored for sequential GAN training to further facilitate the learning of logical reasoning.
In LogiGAN’s design, the Generator learns not only to recover the original masked statements, butalso to score candidate statements (based on their generative likelihood) in a manner that couldreach scoring consensus with the Verifier, who learns to make judgments on the logical consistencybetween premises and conclusions.
The more logically consistent the Verifier thinks of a statementw.r.t.
the input context, the higher generative likelihood score is expected to be assigned by the Generator.
To encourage the exploration of broader possibilities of reasonable statements other thanthe original one, we also apply a diversified sampling strategy for candidate statement generation.
Both
Generator and Verifier scoring processes are continuous throughout the adversarial training, therebycircumventing the non-differentiable barrier in sequential GAN put by the discrete beam-search.
Moreover, LogiGAN does not involve token-wise Monte Carlo Search for policy gradient estimation,and scoring processes of Generator and Verifier are decoupled, so that parallel score computation ispossible.
This makes large-scale pre-training with arbitrary target length computationally feasible.
To test the effectiveness of LogiGAN, we extensively experiment on 12 datasets requiring generalreasoning ability.
The apparent performance improvements of both base and large size PLMs acrossall tasks reveal models’ harvest of logic ability, shoring up the fundamental role of logic in generalreasoning.
We also carry out ablation studies to understand the functionality of LogiGAN components,the results of which shed light on the relative orthogonality between linguistic and logic ability andsuggest that the facilitation effect of reflective thinking is also generalizable to machine learning.
Logic Pre-training
In this work, we primarily focus on improving PLMs’ ability of informal logic (Groarke, 2021).
We include the three most classical types of logical reasoning: deductive, inductive, and abductive
[Figure]
reasoning conducted in the form of natural language (Reichertz, 2004; Kennedy & Thornberg, 2018;
Reichertz, 2007; Douven, 2021) in our consideration.
Note that our coverage is broader than theinformal logic strictly defined in the philosophy community (Munson, 1976) that primarily focuses onanalyzing the soundness and cogency of the application of the aforementioned reasoning in real-lifearguments.
The other half of logic investigation – the theoretical study of formal or mathematicallogic (typically conducted in a symbolic form), which usually deal with propositional logic (Buvac &
Mason, 1993) (Smullyan, 1968), and fuzzy logic (Dote, 1995), is beyond the scope of this paper.
Logic Indicators as Detection Heuristics To set up an unsupervised pre-training aiming at improvingmodels’ logic ability, the very first step will be to identify logical reasoning phenomena from a vastscale unstructured text corpus.
While invocations of logic are not explicitly stated in most cases,writers’ usage of logic indicators usually marks their logical reasoning processes with high precision(Hurley, 1982), thereby serving as an ideal heuristic device for our detection purpose.
We considertwo standard types of logic indicators: (i) conclusion indicator such as “Therefore”, “We may inferthat”, which denotes drawing conclusion deductively or inductively from given premises; And (ii)premise indicator such as “Due to”, “The reason that”, which denotes abductively hypothesizingpremises that explain or provide evidence to some stated conclusions.
Corpus Construction For a selected text corpus, we detect every occurrence of pre-defined logicindicators (listed in App. C), and mask out (i.e., replace with [MASK] ) the entire statement subsequentto the indicator (each training example will have exactly one masked-out statement).
Then models’task will be to perform language modeling and predict the masked statement.
Here we emphasize thatstatements are declarative sentences or declarative clauses, owning complete subject and predicatestructures, and are capable of being factually true or false.
To supply sufficient context informationfor these predictions, we keep x complete sentences previous to the [MASK] , as well as y sentencesafter the [MASK] , where x and y can be sampled from a geometric distribution with pre-definedhyper-parameters.
Fig.
1 illustrates the input and output format, and we discuss details in Sec. 4.
Masked Logical Statement Prediction
In the simplest setting, the Generator learns to infill themasked statement via a single-task pre-training, which fulfills the training process of a typical maskedlanguage modeling task.
The only difference is that models no longer predict randomly maskedtokens or spans but instead logic-targeted masked complete statements .
Models are trained toperform Max Likelihood Estimation (MLE) for masked statements under a typical teacher forcingloss.
Practically, generative pre-trained language models such as T5 (Raffel et al., 2020) could takeup the position of Generator G .
Given a single input context / output statement pair ( c, s ) , the teacherforcing loss can be mathematically expressed as 3 :
[Equation]
The Adversarial Training Framework
Since Generators trained merely from recovering masked original statements miss out opportunities ofexploring other reasonable statements, LogiGAN implements an adversarial mechanism for providing
Generators with extra signals based on logical consistency between pseudo-statements (sampled from
Eq. 3) and context to encourage explorations.
The adversarial framework has two major components:(i) a Verifier V that learns to judge logical consistency between statements and context; (ii) a Generator
G that learns both to recover masked original statements, and scores pseudo-statements (based ontheir generative likelihood) in a manner that could reach scoring consensus with the Verifier – Themore logically consistent the Verifier thinks of a statement w.r.t.
the input context, the more likely the
Generator is expected to generate the statement under the input context (i.e., assign higher generativelikelihood score).
The overall objective of LogiGAN can be expressed as the minimax objective:
[Equation]
where G θ and V ϕ denote the Generator and the Verifier with model parameters θ and ϕ , respectively.s + /s − represents ground-truth statements from original text / sampled pseudo-statement 4 .
Wediscuss sampling details of pseudo-statements later in this section in Eq.
Classical GAN settings (Goodfellow et al., 2014; Zhu et al., 2017) fall short in sequential generationbecause the gradient propagation from the Verifier to the Generator is blocked by a non-differentiablebeam-search during text generation.
Previous approaches such as (Yu et al., 2017) address this challenge by token-wise policy gradient estimation via Monte Carlo Search.
However, since the samplingrun-time grows exponentially with the length of the target sequence, their original implementationsare not directly applicable to million-scale pre-training with arbitrary target length as in our scenario.
Different from them, LogiGAN omits the token-wise Monte Carlo Search for policy gradient estimation, and realizes a similar goal via measuring the similarity of scoring distributions between
Verifier and Generator.
The main procedures of LogiGAN can be summarized in four steps: (a)several candidate pseudo-statements are sampled on a sentence level; (b) the Verifier assigns thelogical consistency scores V score based on how logical consistent these candidates are w.r.t theoriginal context; (c) the Generator assigns the sentence-level generative likelihood score G score toeach candidate to reflect how likely it will produce the pseudo-statement under the given context.(d)
The similarity between Generator and Verifier score distributions is computed as a new signalto encourage the Generator to reach scoring consensus with the Verifier – i.e., the more logicallyconsistent the Verifier thinks of the statement, the higher likelihood score the Generator is expectedto assign.
Since both scoring processes are continuous, the non-differentiable barrier is successfullybypassed.
Meanwhile, this design does not involve sequential token-level sampling and decouples
the
Generator and Verifier scoring processes, thereby enabling parallel score computations.
This makeslarge-scale pre-training with arbitrary target sequence length computationally feasible.
The overall framework overview is illustrated in Fig.
1, and the detailed training procedure issummarized in Algorithm 1.
To diversify the candidate pseudo-statements, we sample pseudostatements from two sources: (i) self-sampling via diversified beam-search from the Generator; or (ii)retrieving similar statements from the corpus, and the sampling process can be summarized as: [Equation]
[Equation]
where G θ ( .
| c ) denotes self-sampled statement s α given context c from Generator G θ , and R ( s + )denotes a retriever 5 that retrieves textually similar statements s β with ground-truth statement s + fromthe corpus.
Note that this process is conducted separately for the corpus of Verifier and Generator.
3.1 Training of Verifier
The Verifier serves as a critic to judge whether a statement is logically consistent w.r.t.
the context.
Therefore, the training task of Verifier can be viewed as a binary classification problem.
Pre-trainedlanguage models that could perform discriminative classification tasks such as BERT (Devlin et al.
,2018), ALBERT (Lan et al., 2019), and RoBERTa (Liu et al., 2019), will be suitable for the role of
Verifier.
With y = 1 for both ground-truth and logically consistent pseudo-statements, and y = 0for other pseudo-statements, the binary classification loss for a single pair of context/statement/label( c, s, y ) of Verifier can be mathematically expressed as (omitting average):
[Equation]
3.2 Training of Generator
The Generator targets both to recover the original masked statements, and to score pseudo-statementsbased on their generative likelihood in a manner that could reach sentence-level scoring consensuswith the Verifier.
This corresponds to the two sources of learning signals received by the Generator:(i) the original generative objective with teacher forcing loss defined in Eq.1 as a signal; and (ii)the distribution similarity between sentence-level generative likelihood score assigned by Generatorand logic consistency score assigned by Verifier.
To achieve the goal of (ii), we first sample pseudostatements { s − 1 , ..., s − n } from p neg ( .
| G θ , c, s + ) .
Then the Verifier assigns logical consistency score
V score based on how logically consistent the pseudo-statements are w.r.t.
the context, expressed as:
[Equation]
After this, the Generator assigns a sentence-level generative likelihood score G score for each pseudostatement to reflect how likely the pseudo-statement will be produced under the given context:
[Equation]
where ℓ θ ( s | c ) is the accumulated log-likelihood of the statement s conditioned on the context c .
Afterward, each statement with a high Verifier score V ϕ ( s, c ) is also expected to receive a highgenerative score ℓ θ ( s | c ) to facilitate the Generator’s capturing of the Verifier’s judgment criterionbased on logic consistency.
KL-divergence (Kullback & Leibler, 1951)
D KL is therefore a appropriatemeasure for the similarity between the score distribution of V score and G score .
For the purpose ofsmoothing the gradient to stabilize the GAN training process, we gather both the ground-truth(learned with teacher-forcing loss) and pseudo statements (learned with KL loss) inside the samebatch w.r.t.
a single input context c .
In our case, there is exactly one ground-truth statement and npseudo-statements for each input context c .
For a batch of ( c ; s + , s − 1 , ..., s − n ) , the overall objective ofthe Generator is defined as (in App.
F we show how Eq.
7 commits to the optimization of Eq. 2):
[Equation]
[Figure]
F r e qu e n c y F r e qu e n c y
Experiment Setup
[Figure]
4.1 Datasets
[Figure]
To test the effectiveness of LogiGAN, we extensively experiment on 12 datasets requiring reasoningvia natural language.
Specifically, ReClor (Yu et al., 2020), LogiQA (Liu et al., 2021a), Adversarial
NLI - ANLI, (Nie et al., 2019), focuses especially on logical reasoning, TellMeWhy (Lal et al.,
2021)on abuductive reasoning, HotpotQA
(Yang et al., 2018a) on multi-hop reasoning, QuoRef (Dasigiet al., 2019) on reasoing with co-reference resolution, MuTual (Cui et al., 2020), DREAM (Sun et al.,2019)), SAMSum (Gliwa et al., 2019) on reasoning in conversational scenarios, and NarrativeQA(s Koˇ ciský et al., 2018), RACE (Lai et al., 2017), XSum (Narayan et al., 2018) on general verbalreasoning.
These datasets make most, if not all, necessary premises for drawing logically consistentconclusions available in their provided context, and require few external premises like commonsenseor numerical knowledge.
Hence, they fit nicely for testing our hypothesis that LogiGAN brings PLMslogic ability beyond their intrinsic linguistic ability, which could benefit general reasoning processes.
4.2 Pre-training Corpus
[Figure]
We apply the corpus construction methodology (§ 2) on the widely used BookCorpus (Kobayashi,2018), which consists of e-books and movies with topics crawled from general domains.
Althoughsome corpus featuring debates and arguments (Walker et al., 2012; Abbott et al., 2016; Swanson et al.,2015) appears to be more suitable for our emphasis on logic, we do not elect them due to their highdomain specificity in fields such as politics, law, and economics.
We discard overly short statementsand instances where indicators do not indicate logical reasoning (e.g., “ since 2010” indicating atime point rather than premises, “ so happy” indicating degree of the subsequent adjective ratherthan conclusions).
This results in 3 .
14 million ( 1 . 43 and 1 .
71 million from conclusion and premiseindicators, respectively) instances.
Corpus statistics are visualized in Fig.
4.3 Models
Baseline Choice.
Since our primary goal of the experiment is to test the effectiveness of LogiGANand test our hypothesis that logic ability can be further enhanced beyond PLMs’ intrinsic linguisticability, we only compare models pre-trained with LogiGAN against their vanilla versions.
After
LogiGAN pre
-training, we discard the auxiliary Verifier (discussed in Sec. 6) and employ the
Generator only to solve all downstream tasks in a purely end-to-end manner.
For our main experiments,we initialize Generators from both base and large size pre-trained T5 (Raffel et al., 2020), and Verifierfrom pre-trained ALBERT-large (Lan et al., 2019).
We leave discussions of the rest implementationdetails and hyper-parameter settings of pre-training and downstream fine-tuning in Appendix D.
Elastic Search vs. Self Sampling.
As stated earlier in section 3.2, candidate pseudo-statementshave two possible sources – they could either be sampled via beam search from the Generator’s selfdistribution, or could be retrieved from some external resources.
We carry out two variant versionsof LogiGAN whose Generator is trained purely from self-sampled sentences as pseudo-statements( LogiGAN base (ss) ), and from extra pseudo-statements retrieved from corpus by Elastic Search
Gormley & Tong (2015) ( LogiGAN base (ss+es)).
For the large model, we use LogiGAN large (es+ss)as default.
Our database consists of 3.14 million sentences discovered by the corpus constructionprocess
[Table]
and we keep the top-5 similar retrieved sentences along with self-samples from Generator
[Table]
Experiments
5.1 Experimental Results
[Table]
As presented in Table 1, both base and large size PLMs further pre-trained with LogiGAN surpasstheir vanilla baselines across both discriminative and generative task formats, through a wide scopeof downstream tasks requiring general reasoning abilities.
We can make the following observations:
Among all observed improvements, those on tasks with particular emphasis on logic (ReClor, LogiQA,and ANLI) are most noticeable.
These positive results manifest the effectiveness of LogiGAN ininjecting logic ability into PLMs, while testifying to our primary hypothesis that logic ability isfundamental to general reasoning as well.
This conclusion answers the two questions in the introsection 6 , suggesting that randomized MLM pre-training might fall short in endowing languagemodels with logic ability, and a logic-targeted pre-training approach like LogiGAN may furtherassist logic learning beyond language acquisition.
Furthermore, extra retrieved pseudo-statements(ss+es) bring some additional performance improvement compared with the pure self-sampling (ss)
LogiGAN variant, revealing the important role of pseudo-statements’ diversity in adversarial training.
5.2 Ablation Study and Analysis
Observing the apparent performance enhancement, we now aim at pinpointing the truly functionalcomponents of LogiGAN through ablation studies and deriving the origins of observed improvements.
For fair comparison purposes, we hold all pre-training and downstream settings (including hyperparameters, implementation designs, and evaluations) unchanged from full LogiGAN.
All variationsare initialized from T 5 base , and we report performance variance on 7 datasets.
I. Random Masked Sentence Prediction Pre-training To explain the observed improvements, our
first hypothesis is: Models harvest extra linguistic ability from masked statement prediction comparedwith masked token (or span) prediction.
Quite intuitively, filling entire sentences with completesubject-predicate structures might put additional demands on models to capture more abundantsyntactic information beyond the coverage of masked token (or span) prediction.
Since LogiGAN
[Table]
involves recovering masked sentences , it is then necessary to determine to what degree, if any, thatthe observed performance gain is attributable to models’ plausible linguistic ability improvement.
Wetherefore carry out a variant pre-training where the prediction objects are randomly masked sentence .
Results (shown in Table 2) displays that masked sentence prediction training barely brings improvement against the vanilla baseline.
This suggests it is unlikely that masked sentence predictionempowers PLM trained from masked token prediction significantly better linguistic ability, nor likelythat the extra pre-training corpus per se significantly raises the performance.
Therefore, we reject thefirst hypothesis and conclude that observed improvements should derive from somewhere else.
II. MLE-only Logic Pre-training
Our second hypothesis is that logic-guided masked statementprediction enhances models’ intrinsic ability of logical reasoning, thereby lifting the downstreamperformance.
Having addressed the potential impact of learning randomized complete sentencegeneration, we next aim to check how learning logic-targeted statement generation affects models’behavior.
We ablate the entire adversarial training process, and train models to perform maximumlikelihood estimation (MLE) with teacher-forcing loss only on masked-out logical statements.
Results 2 of MLE-only logic pre-training reveals quite a notable improvement across almost alldatasets against both vanilla baseline and I., suggesting that learning to generate logical statementsindeed injects extra abilities into the model.
Since results of I. eliminate the possibility that modelsharvest stronger linguistic abilities from complete sentence prediction, it is safe to partially ascribethe better downstream performance to models’ enhanced ability in modeling logical reasoning.
This reveals the relative orthogonality between logic ability and models’ inherent linguistic ability,suggesting that logic ability could be enhanced through further logic-targeted pre-training.
III. Iterative Multi-task Pre
training
Since II.
only partially explains the observed improvements,here is our last hypothesis: the adversarial training procedure of LogiGAN explains the unexplainedrest part beyond the coverage of II.
Here a multi-task pre-training with both generation and verificationtasks will be the most natural intermediate setting between the single-model generation-only settingof II.
and LogiGAN’s dual-model adversarial setting .
However, since the verification task relies on
Generator’s self-sampled statements, we adopt an iterative self-critic pre-training manner following
Nijkamp et al.
Unlike typical multi-tasking training that simultaneously carries different tasksand then sums the losses, our generation and verification tasks happen alternately 7 .
Surprisingly, the iterative multi-task pre-training barely brings any positive effects to models’ downstream performance compared with II.
One possible explanation for this might be that the drasticallydifferent mechanisms between the verification and generations task intervene with each other, makingthe single-model & multi-task setting non-beneficial.
Now that we have confirmed that an extraverification task fails to explain the rest improvement, we can accept our final hypothesis and concludethat it is indeed the adversarial mechanism between the Generator and Verifier that truly facilitatelearning of logical reasoning, thereby further improving the downstream performance beyond II.
Discussion
Adversarial Training Might Assist Downstream Generation Tasks
Although in our experiments,we discard the Verifier and solve downstream tasks with the Generator only, some previous works(Shen et al., 2021; Cobbe et al., 2021) reveal that the Verifier can be used for ranking multiplegeneration results, thereby effectively enhancing overall downstream accuracy.
However, in theirparadigm, the information propagates unidirectionally from the Generator to the Verifier, and the
Generator cannot directly benefit from the Verifier’s discriminative feedback.
In contrast, our Logi-
GAN adversarial training paradigm surmounts the non-differentiable obstacle and could potentiallyenlighten a new paradigm of both pre-training and downstream fine-tuning.
Improving Logical Pre-training Our paper demonstrates that PLMs’ logic ability can be furtherenhanced beyond their inherent linguistic ability, and adversarial training may bring extra benefitsbeyond the learning of logic-targeted masked statement prediction.
However, our heuristic-basedapproach to identifying logical phenomena in a text corpus and the single mask prediction settingcan be further improved.
Logic recognition methods with higher recall and better unsupervisedtask designs (e.g., logical indicator prediction , or logic-guided sentence shuffling ) are worthwhile toexplore in the further work.
Besides, since we are adopting a general domain pre-training corpus (i.e.,
BookCorpus ) with bare emphasis on logic, understanding the impacts of extending pre-training to thedomain-specific corpus (e.g., law corpus) or others emphasizing logical reasoning is also substantial.
Related Works
Generative Adversarial Training in NLP.
Unlike conventional GAN (Goodfellow et al., 2014;
Mirza & Osindero, 2014; Zhu et al., 2017) that generates continuous output such as images, sequential
GAN generates discrete sequences via non-differential searches.
This makes feedback from thediscriminator not propagatable to the generator.
To tackle this challenge, SeqGAN (Yu et al., 2017)borrows an idea from reinforcement learning, treating each output token as a single action, andestimates token-wise policy gradient via Monte Carlo search.
RankGAN (Lin et al., 2017) adopts asimilar approach but breaks the binary-classification assumption of discriminator task design, and aranker provides feedback to the generator.
Their generator attempts to generate verisimilar sentencesto deceive the ranker into ranking synthetic sentences higher over multiple human-written ones.
Inour scenario, however, the gold ranking is hard to determine because measuring which statements aremore logically consistent w.r.t.
context than others is non-trivial, and multi-gold cases are possible.
While successfully enabling communication between generator and discriminator, the original designsof SeqGAN, RankGAN, as well as other works such as (Rekabdar et al., 2019; Fedus et al., 2018;
Caccia et al., 2018; Guo et al., 2017), generally formulate text generation as a sequential actiondecision problem, thereby involving heavy sampling for policy gradient estimation, and are sensitiveto the length of the target sequence.
Since large-scale pre-training (with arbitrary target length) putsa high demand on scalability and computational efficiency, the above approaches are not readilyapplicable in our scenario.
Furthermore, previous work leverages adversarial training to improvequalities of generated examples , whereas our focus is on enhancing models’ intrinsic logic ability .
A recent work, AR2 (Zhang et al., 2021), leverages adversarial training to improve dense documentretrieval.
With a retriever-ranker architecture, the learning objective of retriever is to maximizethe agreeableness between its own score assignment and that of the ranker for input documents.
This is conceptually similar to LogiGAN, as our Generator also aims at reaching consensus with
Verifier.
However, AR2 does not fall into the sequential GAN paradigm, since it does not involve anysequential text generation, and there is no non-differentiable barrier between the retriever and ranker.
Pre-training for Reasoning Ability Improvement.
Previous works have extensively investigatedthe possibility of injecting specific type of reasoning via pre-training, such as numerical (Pi et al.
,2022; Geva et al., 2020; Yoran et al., 2021), commonsense
(Tamborrino et al., 2020; Staliunaite et al.
,2021; Zhong et al., 2019), formal logic (Wang et al., 2021; Pi et al., 2022), multi-hop
(Deng et al.
,2021; Zhong et al., 2022), and tabular (Liu et al., 2021b)
reasoning.
Different from them, LogiGANfocuses on logic reasoning, which plays a fundamental role in general reasoning via natural language.
Conclusion
In this work, we hypothesize that (i) logic ability plays a key role in a wide scope of tasks requiring general reasoning; and (ii) PLMs’ logic ability can be further improved beyond their original linguistic ability.
We correspondingly propose LogiGAN, an unsupervised adversarial pre-training framework for logical reasoning enhancement.
LogiGAN circumvents the non-differentiable challenge of sequential GAN via a novel Generator-Verifier scoring consensus mechanism and enables large- scale pre-training with arbitrary target length.
Extensive experiments and ablation studies reveal the effectiveness and functional components of LogiGAN, providing evidence to our major hypothesis
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We will release
the code and data upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]