L OGIC I NFERENCE : A N EW D ATASET FOR T EACHING L OGICAL I NFERENCE TO SEQ 2 SEQ M ODELS
A BSTRACT
Machine learning models such as Transformers or LSTMs struggle with tasks that are compositional in nature such as those involving reasoning/inference.
Although many datasets exist to evaluate compositional generalization, when it comes to evaluating inference abilities options are more limited.
This paper presents
L
OGIC I NFERENCE , a new dataset to evaluate the ability of models to perform logical inference.
The dataset focuses on inference using propositional logic and a small subset of ﬁrst-order logic, represented both in semi-formal logical notation as well as in natural language.
We also report initial results using a collection of machine learning models to establish an initial baseline in this dataset
I NTRODUCTION
It is well known that machine learning models such as Transformers (Vaswani et al., 2017) or
LSTMs (Hochreiter & Schmidhuber, 1997) struggle with tasks that are compositional in nature (Li ˇ ska et al., 2018; Hupkes et al., 2020;
Keysers et al., 2019; Ontan ´ on et al., 2021) such as those involving reasoning/inference.
Many datasets have been proposed to show this effect such as
SCAN (Lake & Baroni, 2018) or PCFG (Hupkes et al., 2020) among many others.
However, when it comes to evaluating inference, options are more limited.
This paper presents L OGIC I NFERENCE 1 , a new dataset to evaluate the ability of models to perform logical inference.
The dataset focuses on inference using propositional logic and a small subset of ﬁrst-order logic, represented both in semiformal logical notation, and in natural language.
L OGIC I NFERENCE has two main long-term goals: (1) to evaluate the ability of models to perform logical inference, and the degree to which inference chains are real or hallucinated, and (2) to assess whether learning logical inference abilities in the abstract (e.g., getting better in this dataset) would then transfer to other real-world tasks.
Some datasets already exist to assess inference in machine learning models.
L OGIC I NFERENCE is designed to complement those.
For example, natural language entailment datasets, such as
SNLI (Bowman et al., 2015), involve only single step inferences, as the model only needs to establish the relation between two pieces of text (entails/contradicts/unrelated), but it is not asked to present the reasoning chain (although an extension with simple explanations was recently proposed in Camburu et al.
CLUTRR (Sinha et al., 2019) asks models to reason about family relation (given a story in natural language with a collection of characters, where some of their family relations are mentioned, the model needs to infer a family relation that was not mentioned).
The main drawback of this dataset is that the types of reasoning it covers are limited to reason about family trees/graphs.
Another dataset is E NTAILMENT B ANK (Dalvi et al., 2021), which asks models to construct an entailment tree that supports a hypothesis given some text/corpus.
While this dataset is a great resource for evaluation, its limited size (1840 instances in total) limits the types of experiments that can be performed on it.
Also related are datasets designed to evaluate mathematical reasoning abilities.
For example, the mathematics (Saxton et al., 2019) dataset is a relatively large dataset (2m examples) concerning math problems.
One difference with our dataset is that the mathematics dataset only contains inputs and ﬁnal answers, but no reasoning chain that the model has to produce.
The GSM8k dataset (Cobbe et al., 2021) is a similar dataset, but that contains reasoning chains for about 8k simpler math word problems, and the Math32k dataset (Wang et al., 2017) is a middle ground, where there is no reasoning chain, but the equation used to produce the ﬁnal answer is provided as part of the dataset.
The main features of L OGIC I NFERENCE are:
• It covers propositional logic and a subset of ﬁrst-order logic 2 ; • The model needs to learn a variety of tasks such as translate language to logic, one-step or multi-step inferences, and doing so in semi-formal logic notation or natural language; • The model is asked for the step-by-step inference chains used to reach the conclusions;
• The dataset includes corner cases, such as contradictory premises, questions that do not follow from the premises, or that are obvious (answer already stated in the premise), so that the model learns to handle these cases and learns not to hallucinate inference chains when asked to infer something that cannot be inferred or that is obvious; • The dataset contains several data splits and conﬁgurations for better evaluation.
Moreover, we acknowledge the dataset has also several important limitations, which we plan to address in the future.
First, only a subset of ﬁrst-order logic is covered, and only symbolic reasoning is involved (no numerical or mathematical reasoning).
But second, and most important, the main limitation of the dataset is that the natural language it contains is automatically generated, and hence it does not contain the expected variety and complexity found in natural language written by humans.
In the remainder of this paper, we present the dataset in Section 2, a brief description of how it is generated in Section 3, and ﬁnally Section 4 presents the performance of a collection of machine learning models to establish baseline performance for future reference.
T HE L OGIC I NFERENCE D ATASET
L OGIC I NFERENCE is a sequence-to-sequence dataset, where both the input and output are strings of text.
The input is a question, and the model needs to provide the answer, including any reasoning chain used to generate it.
The model may be asked to provide the name of the inference rule used in each step.
Examples in the dataset can belong to one of 5 different types: 1 (language to logic), 2a (one step inference), 2b (one step inference in natural language), 3a (inference chains) and 3b (inference chains in natural language) (examples in Table 1).
The main motivation for having both 2a/3a and 2b/3b is that hopefully the model can learn the general principles of inference from both forms of examples, and hence generalize better when confronted with real world tasks after training in this dataset.
Appendix B shows more examples of the dataset, illustrating interesting cases.
Splits.
The dataset is synthetically generated and the generating script can generate a dataset of arbitrary size.
Moreover, we provide support for three different data splits: IID, OOD, and length split.
In the IID split, data is randomly split between training and test (we expect models to do well in this split).
In the OOD split, data is split such that the model sees different reasoning chains during training and during testing, and hence, it should be a harder generalization challenge.
Speciﬁcally, it targets the ability of the model to exhibit systematicity (Hupkes et al., 2020).
Finally, the length split has the model training on shorter examples, and then test on longer examples.
This type of length split targets the ability of the model to exhibit productivity (Hupkes et al., 2020), and is known to be hard for machine learning models (Lake & Baroni, 2018).
Answer Position.
Finally, notice that in the examples in Table 1 the model is asked to provide the ﬁnal answer of problem types 3a/3b in the very ﬁrst token of the output.
This requires the model to assess whether or not an inference chain exists to prove the target clause, with a single forward pass in the case of seq2seq models.
One hypothesis we wanted to evaluate is whether giving the model the chance of performing more computation for harder problems (by asking it to generate the inference chain ﬁrst, and only once the chain is generated, answer yes or not) would allow the model to generalize better.
Hence, two versions of the dataset exist, L OGIC I NFERENCE b (which has the answer at the b eginning) and L OGIC I NFERENCE e (which has the answer at the e nd)
[Table]
Input: Translate the following inference to logic notation: If James were rich, then Susan is playing squash.
James is rich.
Therefore Susan is playing squash.
Output: p -> q. p.
Therefore q. 2a Input: What can be inferred from the following premises in a single inference step (ignoring inferences that add new predicates or constants)?
Name the inference rule being used: forall x 2: Q(x 2).
exists x 2: P 2(x 2) -> Q 2(x 2).
forall x 2: P 2(x 2).
Output: exists x 2: Q 2(x 2) can be inferred via the existential modus ponens rule.
forall x 2: Q(x 2) and P 2(x 2) can be inferred via the universal conjunction rule. forall x 2: P 2(x 2) and Q(x 2) can be inferred via the universal conjunction rule.
2b Input: What can be inferred from the following premises in a single inference step (ignoring inferences that add new predicates or constants)?
David is working if and only if it is cloudy.
David is not working.
Output: If David works, then it is cloudy.
If it is cloudy, then David is working.
It is not cloudy.
3a Input: Consider the following premises.
p <-> p 2.
p 2 -> ~q 2.
q 2.
Can we infer p from them?
If possible, name the inference rules being used at each step.
Output:
Yes, via the following inference chain.
From p 2 -> ~q 2, q 2 we can infer ~p 2 via modus tollens.
Finally, from p <-> p 2, ~p 2 we can infer ~p via biconditional elimination.
3b Input: Consider the following premises
There
is at
least
one x for
which
if
x
were
an
astronaut
, then x is
climbing a
mountain.
For all
x
if
x
climbs
a mountain, then x is an astronaut.
Can we infer the following from them?
If we can, name the inference rule being used
: There is at least one x for which if x were an astronaut, then x is climbing a mountain.
Output:
Yes, that is one of the premises.
D ATASET G ENERATION
In order to generate L OGIC I NFERENCE , the following procedure was used:
Let R be a set of inference rules R , such as modus ponens , modus tollens , etc.
Rules include propositional logic (17 rules) and rules with universal and existential quantiﬁers (49 rules) (see Appendix A for examples and details).
Given the rules in R , we generate a set of inference problems P , where an inference problem p
= ( P, I, C, U ) is deﬁned by a set of premises p.P , a set of potential inference chains
p.I and p.C
( p.I containing inference chains to prove some statements, and p.C to disprove some statements), a set of unrelated clauses p.U (clauses that cannot be proven or disproven from the premises).
This is done by starting from an inference rule, and randomly choosing other rules that could be used to infer any of the current premises, creating arbitrarily long inference chains/trees.
We control for contradictions introduced by this process, but allow some in the dataset to test the model’s ability to detect them.
For each inference problem p , we generate a set of renaming variations by renaming the propositions, constants and variables appearing in them (to prevent models memorizing patterns associated with variable names).
For example, p → q , could be renamed to r → p 2 .
This results in an enlarged set of problems P v . 4.
To generate a training example, we pick a variation p ∈ P v at random, and then we stochastically pick one of 5 the different problem types to generate an example: Type 1: language to logic : given a natural language representation of the premises and a potential inference, the model is asked to translate it to a more formal logical notation.
[Table]
Type 2a: one step inference : given a set of premises, the model is asked to predict all the possible one step inferences that can be done from them.
Type 2b: one step inference in natural language : same as 2a, but the input and output are in natural language, rather than in logical notation.
Type 3a: inference chains : given a set of premises and a potential inference, the model is asked whether we can prove the inference or not from the premises, and to provide the inference chain in either case.
Type 3b: inference chains in natural language : same as 3a, but using natural language, rather than formal notation.
The IID split dataset is generated by trying to generate up to n (n = 200k in our experiments) examples in this way and ﬁltering for duplicates, and then splitting them randomly 90% / 10% between train / test.
The OOD split is generated by randomly splitting the set P into two subsets (one for training, one for testing), and then continuing the process of generation separately for each of those subsets.
In this way, we can ensure that, given inference problem, all the examples derived from it are either in the training set or they are all in the test set.
This creates a harder generalization challenge, as problems in the test set will have reasoning patterns (sequence of rule applications) not seen during training.
Finally, the length split is generated by splitting the examples based on the number of premises their corresponding inference problems have: less or equal to k ( = 4 in our experiments) premises for training, and more than k for testing.
With the default settings, the dataset has 118931 instances for training and 13215 for testing in the IID split,/ 13266 in the OOD split, and 114790 / 17356 in the length split (See Appendix D for more detailed statistics).
Considerations, In problems of type 3a and 3b, the potential inference may be selected from the unrelated set (e.g., Consider the following premises.
p. p -> q.
Can we infer r? ).
This is to prevent the model from making up answers when users ask questions that cannot be answered from the premises.
With some probability, problems contain contradictions in the premises (either direct contradictions, or their inferences reach a contradiction).
This can happen in problems of types 2a, 2b, 3a and 3b.
In this case, the model is supposed to identify this, and respond accordingly (anything can be inferred from a contradiction).
Some inference rules can produce an inﬁnite number of inferences, e.g.: if we know p , we also know p ∨ q , p ∨ r , p ∨ s , etc.
In problems of types 2a and 2b, the model is explicitly told not to introduce new variables or predicates, to prevent this.
Finally, sometimes, the potential inference is directly present in the premises.
Again these problems are introduced to force the model to detect this and respond accordingly.
B ASELINES
In order to establish some baseline performance in this dataset, we evaluated the performance of a collection of T5 (Raffel et al., 2019) models.
Speciﬁcally, we evaluated the small , base and large conﬁgurations of T5.1.1, which have 77m, 248m and 783m parameters respectively.
Moreover, as previous work has shown that pre-training signiﬁcantly helps in compositional generalization (Furrer et al., 2020), we evaluated both ﬁnetuning the public pre-trained checkpoints (for 20k steps), as well as training from scratch (for 50k steps).
Although performance was still increasing at the end of training, it was starting to taper off, as shown in the training curves in Appendix E. We used batch size 128, constant learning rate of 0.001, the AdaFactor (Shazeer & Stern, 2018) optimizer, and sequence lengths of 256 tokens for inputs, and 512 for targets.
Table 2 show the performance of these models evaluated using sequence level accuracy (percentage of times they predicted the exact ground truth output sequence).
The results show that pre-training (which was known to help compositional generalization) signiﬁcantly helps in the length split, where the pre-trained models signiﬁcantly outperforming the non-pre-trained versions.
Pre-training also seems to help the large model in all three splits. Interestingly, the small and base models do not seem to beneﬁt from pre-training for the IID or OOD splits.
The OOD split (measuring systematicity) was harder than the IID set as expected, but the length split (measuring productivity) was even harder for models.
Surprisingly, we did not see any clear difference in between L OGIC I NFERENCE e and L OGIC I NFERENCE b (the large model achieves higher performance in IID/OOD with the L OGIC I NFERENCE e setup, but the other results are mixed).
Moreover, detailed analysis shows that trained-from-scratch models struggle in problems of typein the length split, while this is not as pronounced in the pre-trained models (more detailed results in Appendix E). Finally, when training from scratch, we saw the large models perform worse than the smaller models, which might be due to overﬁtting.
C ONCLUSIONS AND F UTURE W ORK
In this paper we presented L OGIC I NFERENCE , a new dataset designed to evaluate the inference abilities of seq2seq models, and to investigate if models trained in this dataset transfer inference abilities to real world natural language tasks (which is part of our future work).
We evaluated a collection of T5.1.1 models on our dataset in order to establish an initial baseline performance as a reference for future work.
As part of our future work, we would like to improve the dataset by including a larger subset of ﬁrst order logic, and improve the natural language in the dataset by having humans rephrase a subset of the automatically generated inputs/output pairs in the dataset.
A A PPENDIX : I NFERENCE R ULE D EFINITION
There are 17 propositional rules and 2 base quantiﬁed rules deﬁned in the dataset generation script.
Each rule is deﬁned as a 5-tuple: ( P, I, C, U, name ) , where P , I , C , and U have the same meaning as for the inference problem deﬁnition above (premises, inferences, contradictions, unrelated), and name is the name of the inference rule.
For example, modus ponens is deﬁned as follows:
[Equation]
 which, in the Python deﬁnition looks as follows:
[Equation]
"->", ["p"], ["q"]], ["p
[Equation]
]], [["q"]], [["˜",["q"]]], [["r"],["˜",["r"]]], ["p", "q", "r"], "modus ponens")
Notice that in the Python version there is an additional ﬁeld (right before the name) that contains a list of all the atomic propositions involved in the rules.
This is included for convenience, to prevent having to search them every time.
In addition to the 17 propositional rules, there are 2 quantiﬁed rules ( universal instantiation , and existential generalization ).
Universal instantiation looks as follows:
([["forall", "var x", ["P", "var x"]]], [["P", "a"]], [["˜", ["P", "a"]]], [["Q", "a"], ["˜", ["Q", "a"]]], [["P", "var x"], ["P
", "a"], ["Q", "a"]], "universal instantiation")
Finally, from each of the 17 propositional rules, we generate one universally quantiﬁed rule automatically by turning each predicate p into P ( x ) and adding a universal quantiﬁer.
We also generate several existentially quantiﬁed rules by universally quantifying all premises but one, and existentially quantifying one of them.
Then the conclusions are existentially quantiﬁed.
This generatesadditional rules, resulting in 17 + 2 + 47 = 66 total rules used for generation of inference problems, which form the set R .
B A PPENDIX : M ORE D ATASET E XAMPLES
We include here a few more examples from the dataset, to illustrate some of its features.
B. L OGIC I NFERENCE b VS L OGIC I NFERENCE e
Here is the same training example for both versions of the dataset.
This is how it looks in L OGIC I NFERENCE b (with the answer at the beginning):
Input: Consider the following premises.
forall x 3: Q 1(x 3).
forall x 3: ~P 3(x 3).
forall x 3: P 3(x 3) or Q 3(x 3).
Can we infer exists x 3: ~Q 3(x 3) from them?
Output: No, we can see why via the following inference chain.
From forall x 3:
P 3(x 3) or Q 3(x 3), forall x 3: ~P 3(x 3) we can infer forall x 3: Q 3(x 3).
From forall x 3: Q 3(x 3), forall x 3: Q 1(x 3) we can infer forall x 3: Q 3(x 3) and
Q 1(x 3).
Finally, from forall x 3: Q 3(x 3) and Q 1(x 3) we can infer forall x 3:
Q 3(x 3), which contradicts exists x 3: ~Q 3(x 3).
This is how it looks in L OGIC I NFERENCE e
(with the answer at the end):
Input: Consider the following premises.
forall x 3: Q 1(x 3).
forall x 3: ~P 3(x 3).
forall x 3: P 3(x 3) or Q 3(x 3).
Can we infer exists x 3: ~Q 3(x 3) from them?
Output: From forall x 3: P 3(x 3) or Q 3(x 3), forall x 3: ~P 3(x 3) we can infer forall x 3: Q 3(x 3).
From forall x 3: Q 3(x 3), forall x 3: Q 1(x 3) we can infer forall x 3: Q 3(x 3) and Q 1(x 3).
Finally, from forall x 3: Q 3(x 3) and
Q 1(x 3) we can infer forall x 3: Q 3(x 3), which contradicts exists x 3: ~Q 3(x 3).
Therefore, the answer is no.
Notice that the input is the same, and the only change in the output is that the answer (“No”) appears either in the ﬁrst token, or in the last token of the output.
The main goal of doing this variation is as follows: if the answer is in the ﬁrst token, the model is forced to already predict (with a single forward pass of the Transformer encoder in our experiments) the ﬁnal answer of the problem, regardless of how long is the inference chain required to assess this answer.
If a human were to solve this problem, she would probably use some scrap paper to write down some inferences before reaching the ﬁnal answer.
We hoped to use the inference chain as this “scrap paper”, before reaching the ﬁnal decision (although our experiments show that this had only a small effect in performance).
B. C ONTRADICTIONS , U NRELATED AND O BVIOUS A NSWERS
As mentioned above, some examples do not require any inference chain, but contain unrelated or obvious questions, or even contradictory premises.
We added these examples (in a small proportion) to teach the model that it is not always the case that it has to look for an inference chain, and some times the question is either obvious or does not make sense.
Hopefully, this contributes to minimize hallucination when training on this dataset and trying to transfer logical inference skills to another task.
This is an example where the answer is obvious, as it is stated as one of the premises:
Input: Consider the following premises.
There is at least one x 1 for which x 1 is a lawyer.
For all x 1, x 1 will go running.
Can we infer the following from them?
If we can, name the inference rule being used: There is at least one x 1 for which x 1 is a lawyer.
Output: Yes, that is one of the premises.
This is an example, where the question is unrelated to the premises:
Input: Consider the following premises.
It is not overcast.
It is overcast if and only if Joseph is happy.
Can we infer the following from them?
If we can, name the inference rule being used: Karen is not an electrician.
Output: No, we cannot infer that from the premises.
And ﬁnally, this is an example where the premises contain a contradiction:
Input: Consider the following premises.
For all x 4, x 4 is an electrician.
Joseph is not an electrician.
Can we infer the following from them?
If we can, name the inference rule being used: Joseph is not making tea.
Output: Yes, the premises are contradictory, so we can infer anything from them.
C A PPENDIX : D ATASET G ENERATION
This section presents some additional details on dataset generation.
Speciﬁcally, we provide some additional details on how inference problems are created, and how clauses are translated to natural language.
C. I NFERENCE P ROBLEM G ENERATION
As mentioned in the paper, an inference problem is deﬁned as a tuple p =
( P, I, C, U ) , with a set of premises, inferences, contradictions and unrelated clauses.
Inference problems are the starting point from which training examples are generated.
To generate an inference problem, we basically create random inference chains by chaining some of the deﬁned inference rules above.
Speciﬁcally, we use the following procedure:
One inference rule r 0 is selected at random.
From r 0 , we can form an inference problem p as follows: the premises of the problem are the same as those in the rule: p.
P = r 0 .P .
For each inference in the rule, we construct an inference chain of length one in the problem: p.
I = { ( c, [( r 0 .P, c, r 0 . name )]) | c ∈ r 0 .I }
(each inference chain consists of the ﬁnal conclusion, and a list of triples: (premises, conclusion, inference rule name) with the inference steps).
We do the same with the contradictions: p.
C
= { ( c, [( r 0 .P, c, r 0 )])
| c ∈ r 0 .C } .
The set of unrelated clauses are also taken directly from the rule: p.
U =
r 0 .U .
At this point, we can now take any of the premises c ∈ p.P , and choose a random rule r 1 ∈ R that could be used to infer c .
Then, we remove c from p.P , and add the premises of r 1 to p.P .
We then extend all the inference chains in p.I and p.C by adding one initial step using r 1 All variables in r 1 are renamed to unique names before using it to update the inference problem, to prevent any name clashes.
This step 3 can be iterated as many times as desired to create arbitrarily long inference chains.
Finally, as the previous process just creates arbitrary inference chains, it is possible that the generated chains are redundant (inferring things that are already known),
or even introducing contradictions.
Thus, a ﬁnal step to simplify inference chains (any inference step that infers something that was already known is removed, and if the statement that wants to be proven or contradicted is inferred earlier, then the inference chain is cut at that point).
As mentioned, inference problems might be created in a way that their premises lead to contradictions.
If any such problem is generated, it is stored on a separate buffer, and at the end of inference problem generation, some of those problems with contradictions (up to a user speciﬁed proportion) are added back to the set of inference problems, so that we have, up to the desired fraction of problems with contradictions (by default, we limit to at most 10% of inference problems containing contradictions, but in reality many less than that are generated.
The process that generates an inference problem receives a hand-deﬁned probability distribution over the possible problem lengths (number of times we iterate step 3 above) that we want, and this is used for generating inference problems.
By default, we use the following distribution: [0.425, 0.3, 0.2, 0.05, 0.025] (for 0, 1, 2, 3, 4 and 5 applications of step 3 respectively).
C. T RANSLATION TO N ATURAL L ANGUAGE
Translation of a clause to natural language follows a set of patterns:
• Atoms of the form p , q , etc.
get translated to one of these forms: – “subject verb-action” (e.g., Mary plays tennis ), – “Subject predicate” (e.g., Mary is happy ), – “Impersonal-action” (e.g., It is raining ) •
There is a set of predeﬁned subjects, verb-actions, predicates, and impersonal actions and they are sampled randomly (but without replacement within the same training example, to prevent repetitions).
• When an atom is of the form P(c) , Q(c) , etc.
then only the patterns with subjects above are used and c is mapped to the subject, and P / Q to the verb-action/predicate.
• When an atom is of the form P(x)
, Q(x) , etc.
then only the patterns with subjects above are used and the subject is just rendered as x (since x is a variable here).
We acknowledge that this step could use better translation to natural language.
• Each atom can be rendered in several modes (present, past, negated, etc.) to be used in the patterns below.
• Or, and, implication and biconditional, have patterns like X or Y , if X then Y , etc.
• Quantiﬁed clauses also have patterns: For all x, X and There is at least one x for which X .
Finally, there is a special case for existentially quantiﬁed rules of the form exists x, P(x) and Q(x) that renders it as some Xs are Y (where X and Y are the predicates associated with P and Q respectively).
In the generation script there are currently 20 possible subjects (the 10 most common male andmost common female names in English), 30 possible predicates, 15 possible actions, and 8 possible impersonal-actions.
For example, p -> q , could be translated to If John plays Tennis then it will snow .
C. A DDITIONAL C ONSIDERATIONS
During the development of L OGIC I NFERENCE , we observed that models obtained much better performance in the ﬁrst versions of the dataset, which was surprising to us.
We realized that the generation pipeline had certain biases which models were latching onto for predictions.
For example:
• Due to the way problems are generated, variable names were indicative of the particular inference chain that had used.
Each time a rule is applied for generation, there is an internal counter (that starts at 1), and all variables in a rule are then renamed based on that counter ( p becomes p 1 , etc.).
We noticed that
models were able to identify patterns based on these numbers, and that’s why we introduced the variable renaming variations.
This made the problem harder, as removed these types of clues.
• In addition to variable renaming variations, we had to randomly shufﬂe the premises of a problem each time a new example was being generated, as the order in which the premises appear in a problem also contained clues that models were latching onto.
• For problems of type 1, there are many ways in which a problem can be translated to semiformal notation.
For example, if a natural language
statement, can be translated to p -> q , then it could also be translated to r -> s , as it’s just a matter of which names do we decide to use to assign to each predicate.
This was making evaluation complex, as models were proposing translations that were correct, but different from that present in the dataset.
To solve this problem, we deﬁned a canonical naming convention, where the ﬁrst atomic proposition that appears is always called p , the second always called q , etc.
This was used only for problems of type 1.
D A PPENDIX : D ATASET S TATISTICS
The dataset generation script can generate datsets or arbitrary sizes.
In this section, we report statistics using the default conﬁguration, which is:
• Inference chain distribution: [0 . 425 , 0 . 3 , 0 . 2 , 0 .
• Try to generate up to 5000 inference problems.
• Try to generate 25 renaming variations.
• Try to generate 200000 training examples.
• For IID/OOD split train/test with a 0.9/0.1 ratio.
• For the length split, use problems with 4 premises or less as train, and 5 or more as test.
With these parameters, the resulting dataset (using random seed 0), has the following statistics (notice that all the splits have less than 200000 training examples, due to duplication removal):
• IID: 177543 / 19728 instances for train / test.
• OOD: 177578 / 19743 instances for train / test
• length: 175179 / 22092 instances for train / test.
More detailed statistics for one of the splits (IID):
• The script generated 4814 different inference problems generated (168 with contradictions).
In these 4814 inference problems, we ﬁnd (note that these do not add up to 4814, since each inference problem might have more than one inference chain, and also, we are counting chains in both the inferences and contradictions set): – 1079 inference chains of length
(appear in premises).
– 607 inference chains of length 1 (application of a single inference rule).
– 1347 inference chains of length 2.
– 4227 inference chains of length 3.
– 2127 inference chains of length 4.
– 852 inference chains of length 5.
• 77199 renaming variations were generated.
• The example distribution among the different problem types was (notice that the low count of problems of type 2a is because, as they do not depend on inference chains, but only on premises, many were removed due to duplication removal): – 1:– 2a:– 2b:– 3a:– 3b:• Once tokenized using the T5 default vocabulary, we see the following token length distribution in the training set (min / median / 90% percentile max): – Input: 18 / 97 / 147 /– Output: 2 / 70 / 269 /
Notice that we use input/output sequence lengths of 256 and 512 respectively for our experiments, so, the longest examples will get cropped.
This only happens for less than 100 examples in the IID set, so we decided not to increase the sequence length beyond 256/512 for efﬁciency.
Moreover, concerning the tokenization, we also would like to note that the characters “ < ” and “ ~ ” are not part of the T5 vocab, and hence are parsed as “unknown“.
Moreover, since the “ < ” always appears before a “ - ”, and “ ~ ” can never appear before a “ - ”, we can perform a simple string replacement of unknown tokens by these two tokens easily.
However, we note that this might cause some small problems for T5, and hence, an extended vocab with these two tokens might obtain slightly better results.
E A PPENDIX : D ETAILED R ESULTS
Learning Curves.
Figures 1 and 2 show the learning curves corresponding to the results in Table 2.
As the curves show, evaluation sequence accuracy still continued to grow at the end of training (except for the length split when training from scratch, which had stabilized).
Hence, training for longer would result in better results.
Also, although we do not show it, training set accuracy reaches above 0.99 very quickly (even when training from scratch), usually after 20k iterations when training from scratch.
However, as noted by Csord´as et al. (2021), test accuracy continues to grow in compositional generalization tasks even after training set accuracy (or even IID validation set accuracy) has saturated.
Accuracy by Problem Type.
It is also interesting to see in which types of problems do errors occur.
Table 3, shows the performance divided by problem type in the L OGIC I NFERENCE e version.
We report results for the base model, which generally achieved good results.
Table 3 reveals an interesting effect: when training from scratch, models signiﬁcantly struggle in task 1 (0.075 accuracy).
Moreover, task 3b seems to be the harder task overall in the IID and OOD splits.
[Figure] [Figure]
pre-trained checkpoints (pre-trained for 1m steps).
Accuracy by Problem Length.
Figure 3 plots the accuracy as a function of the problem input length (in tokens).
As the ﬁgure shows, examples with shorter inputs are easier to predict (higher accuracy), and errors are concentrated in the longer examples.
This shows that models struggle to generalize via productivity .
Moreover, notice that in the length split, since examples are split by the From Scratch (50k) [Table] [Figure]
number
Results Variance.
As we were only trying to present example baseline performance, we only report accuracy of a single run per model conﬁguration.
However, we observed a some degree of variance in the results in our experiments.
In some preliminary runs with larger size models ( xl size), we observed that while the best runs achieved better results than those in the table (0.940 accuracy in the IID split), the variance problem was more pronounced ( small and base seemed to be more stable), and we decided not to include xl results, as that would require multiple runs and reporting averages, increasing the computation cost of the experiments signiﬁcantly.
Previous work on compositional generalization tasks has already reported on this type of variance in some tasks, and hence, this is not particularly surprising.
However, the consequence is that future work using this dataset should probably report the average of several runs to prevent this issue.