Original: LIME: L EARNING I NDUCTIVE B IAS FOR P RIMITIVES OF M ATHEMATICAL R EASONING
Revised: LIME: L EARNING I NDUCTIVE B IAS FOR P RIMITIVES OF M ATHEMATICAL R EASONING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While designing inductive bias in neural architectures has been widely studied, we hypothesize that transformer networks are ﬂexible enough to learn inductive bias from suitable generic tasks.
Revised: While designing inductive bias in neural architectures has been widely studied, we hypothesize that transformer networks are ﬂexible enough to learn inductive bias from suitable generic tasks.

Original: Here, we replace architecture engineering by encoding inductive bias in the form of datasets.
Revised: Here, we replace architecture engineering by encoding inductive bias in the form of datasets.

Original: Inspired by Peirce’s view that deduction, induction, and abduction form an irreducible set of reasoning primitives, we design three synthetic tasks that  require the model to have these three abilities.
Revised: Inspired by Peirce’s view that deduction, induction, and abduction form an irreducible set of reasoning primitives, we design three synthetic tasks that are intended to require the model to have these three abilities.

Original: We speciﬁcally design these synthetic tasks in a way that they are devoid of mathematical knowledge to ensure that only the fundamental reasoning biases can be learned from these tasks.
Revised: We speciﬁcally design these synthetic tasks in a way that they are devoid of mathematical knowledge to ensure that only the fundamental reasoning biases can be learned from these tasks.

Original: This deﬁnes a new pre-training methodology called “LIME” (Learning Inductive bias for Mathematical rEasoning).
Revised: This deﬁnes a new pre-training methodology called “LIME” (Learning Inductive bias for Mathematical rEasoning).

Original: Models trained with LIME signiﬁcantly outperform vanilla transformers on three very different large mathematical reasoning benchmarks.
Revised: Models trained with LIME signiﬁcantly outperform vanilla transformers on three very different large mathematical reasoning benchmarks.

Original: Unlike dominating the computation cost as traditional pre-training approaches, LIME requires only a small fraction of the computation cost of the typical downstream task.
Revised: Unlike dominating the computation cost as traditional pre-training approaches, LIME requires only a small fraction of the computation cost of the typical downstream task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Inductive bias is essential for successful neural network learning.
Revised: Inductive bias is essential for successful neural network learning.

Original: Many of the breakthroughs in machine learning are accompanied by new neural architectures with better inductive biases, such as locality bias in convolutional neural networks (LeCun et al., 1999), recurrence and memory in
Revised: Many of the breakthroughs in machine learning are accompanied by new neural architectures with better inductive biases, such as locality bias in convolutional neural networks (LeCun et al., 1999), recurrence and memory in

Original: LSTMs (Hochreiter and Schmidhuber, 1997), and structural bias in graph neural networks (Scarselli et al., 2008).
Revised: LSTMs (Hochreiter and Schmidhuber, 1997), and structural bias in graph neural networks (Scarselli et al., 2008).

Original: However, existing designs of inductive biases need to be explicitly encoded in neural architecture.
Revised: However, existing designs of inductive biases need to be explicitly encoded in neural architecture.

Original: This is sometimes difﬁcult as one may not know the exact mechanism for an abstract ability, in order to describe the architectural bias explicitly.
Revised: This is sometimes difﬁcult as one may not know the exact mechanism for an abstract ability, in order to describe the architectural bias explicitly.

Original: In particular, designing proper inductive bias for abstract concepts such as mathematical reasoning becomes an extremely challenging task.
Revised: In particular, designing proper inductive bias for abstract concepts such as mathematical reasoning becomes an extremely challenging task.

Original: Moreover, attempts to design elaborate architectures for reasoning often fall short of the performance of more generic transformer architecture.
Revised: Moreover, attempts to design elaborate architectures for reasoning often fall short of the performance of more generic transformer architecture.

Original: In this work, we aim to avoid the search for new architectures and investigate whether one can learn useful inductive bias for mathematical reasoning through pretraining .
Revised: In this work, we aim to avoid the search for new architectures and investigate whether one can learn useful inductive bias for mathematical reasoning through pretraining .

Original: Large-scale unsupervised pretraining of language models revolutionized the ﬁeld of natural language processing (NLP), improving the state-of-the-art in question answering, name entity recognition, text classiﬁcation, and other domains, e.g. (Radford et al., 2018;
Revised: Large-scale unsupervised pretraining of language models revolutionized the ﬁeld of natural language processing (NLP), improving the state-of-the-art in question answering, name entity recognition, text classiﬁcation, and other domains, e.g. (Radford et al., 2018;

Original: Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020).
Revised: Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Raffel et al., 2020; Brown et al., 2020).

Original: As a result, pretraining has become a common practice for modern neural network based NLP.
Revised: As a result, pretraining has become a common practice for modern neural network based NLP.

Original: One plausible explanation for the beneﬁt of pretraining is that the model can learn world knowledge by memorizing the contents of the natural language corpus.
Revised: One plausible explanation for the beneﬁt of pretraining is that the model can learn world knowledge by memorizing the contents of the natural language corpus.

Original: This can be useful in various natural language downstream tasks, such as question answering and text classiﬁcation.
Revised: This can be useful in various natural language downstream tasks, such as question answering and text classiﬁcation.

Original: However, there is another potential advantage of pre-training—it may distill inductive biases into the model that are helpful for training on downstream tasks (Brown et al., 2020; Warstadt and Bowman, 2020).
Revised: However, there is another potential advantage of pre-training—it may distill inductive biases into the model that are helpful for training on downstream tasks (Brown et al., 2020; Warstadt and Bowman, 2020).

Original: We focus on the latter and design pre-training tasks that are intentionally devoid of knowledge and only allow the model to learn inductive bias for reasoning.
Revised: We focus on the latter and design pre-training tasks that are intentionally devoid of knowledge and only allow the model to learn inductive bias for reasoning.

Original: Inspired by the logician Charles Peirce (Peirce, 1992), we believe that the following three primitives are the most crucial for reasoning:
Revised: Inspired by the logician Charles Peirce (Peirce, 1992), we believe that the following three primitives are the most crucial for reasoning:

Original: Deduction : the ability to deduce new truths from given facts and inference rules.
Revised: Deduction : the ability to deduce new truths from given facts and inference rules.

Original: Induction : the ability to induce general inference rules from a set of known facts.
Revised: Induction : the ability to induce general inference rules from a set of known facts.

Original: Abduction : the ability to explain the relationship between the evidences and inference rules.
Revised: Abduction : the ability to explain the relationship between the evidences and inference rules.

Original: To endow the models with an inductive bias for mathematical reasoning, we design a synthetic task for each of the three inductive biases.
Revised: To endow the models with an inductive bias for mathematical reasoning, we design a synthetic task for each of the three inductive biases.

Original: We hypothesize that the transformer networks are ﬂexible enough to learn strong inductive bias from the three synthetic reasoning tasks and consequently improving the downstream tasks.
Revised: We hypothesize that the transformer networks are ﬂexible enough to learn strong inductive bias from the three synthetic reasoning tasks and consequently improving the downstream tasks.

Original: Although such inductive bias may be useful in general reasoning tasks (e.g., NLP tasks), in this work, we focus on mathematical reasoning benchmarks, for which we expect to observe the largest gains.
Revised: Although such inductive bias may be useful in general reasoning tasks (e.g., NLP tasks), in this work, we focus on mathematical reasoning benchmarks, for which we expect to observe the largest gains.

Original: We call training on these tasks LIME – an acronym for “Learning Inductive Bias for Mathematical rEasoning”.
Revised: We call training on these tasks LIME – an acronym for “Learning Inductive Bias for Mathematical rEasoning”.

Original: Note that there is only a limited amount of pretraining data available for formal mathematical benchmarks, therefore the study of generic pre-training techniques is particularly important for the success of machine learning in mathematical reasoning.
Revised: Note that there is only a limited amount of pretraining data available for formal mathematical benchmarks, therefore the study of generic pre-training techniques is particularly important for the success of machine learning in mathematical reasoning.

Original: We demonstrate that LIME pretrained models provide signiﬁcant gains across three large mathematical reasoning benchmarks: IsarStep (Li et al., 2020), HOList Skip-tree (Rabe et al., 2020) and MetaMathStep (Polu and Sutskever, 2020).
Revised: We demonstrate that LIME pretrained models provide signiﬁcant gains across three large mathematical reasoning benchmarks: IsarStep (Li et al., 2020), HOList Skip-tree (Rabe et al., 2020) and MetaMathStep (Polu and Sutskever, 2020).

Original: Notably, on the IsarStep benchmark, pre-training improved the top-1 accuracy from 20 .
Revised: Notably, on the IsarStep benchmark, pre-training improved the top-1 accuracy from 20 .

Original: 4% to 26 .
Revised: 4% to 26 .

Original: 9% and top-10 accuracy from 33 . 1% to 41 .
Revised: 9% and top-10 accuracy from 33 . 1% to 41 .

Original: Compared to the traditional pre-training tasks, there are two major differences.
Revised: Compared to the traditional pre-training tasks, there are two major differences.

Original: First, we do not load the input embeddings or the weights in the output layer for ﬁnetuning on downstream tasks.
Revised: First, we do not load the input embeddings or the weights in the output layer for ﬁnetuning on downstream tasks.

Original: This allows us to use the same pre-trained model for a variety of downstream tasks, which can have vastly different vocabularies due to language or tokenization differences.
Revised: This allows us to use the same pre-trained model for a variety of downstream tasks, which can have vastly different vocabularies due to language or tokenization differences.

Original: Also, it prevents the transfer of content knowledge from the pretraining to downstream tasks, supporting the evidence of learning inductive biases.
Revised: Also, it prevents the transfer of content knowledge from the pretraining to downstream tasks, supporting the evidence of learning inductive biases.

Original: Furthermore, pretraining on synthetic tasks require only a fraction of the computational cost of downstream tasks.
Revised: Furthermore, pretraining on synthetic tasks require only a fraction of the computational cost of downstream tasks.

Original: With only about two hours of training on a single modern
Revised: With only about two hours of training on a single modern

Original: GPU, one already obtains all the beneﬁts, in contrast to days of training on a large natural language corpus with hundreds of GPUs/TPUs.
Revised: GPU, one already obtains all the beneﬁts, in contrast to days of training on a large natural language corpus with hundreds of GPUs/TPUs.

Original: Our method can also be regarded as a form of curriculum learning, in which the model is taught basic, extremely generic but general skills before being trained on the speciﬁc problem domain.
Revised: Our method can also be regarded as a form of curriculum learning, in which the model is taught basic, extremely generic but general skills before being trained on the speciﬁc problem domain.

Original: To summarize, the contributions of the paper are:
Revised: To summarize, the contributions of the paper are:

Original: Providing the ﬁrst method to design inductive biases in the form of datasets for mathematical reasoning.
Revised: Providing the ﬁrst method to design inductive biases in the form of datasets for mathematical reasoning.

Original: Demonstrating signiﬁcant improvements in the reasoning performance of transformer models on three large mathematical reasoning benchmarks with negligible extra computation cost.
Revised: Demonstrating signiﬁcant improvements in the reasoning performance of transformer models on three large mathematical reasoning benchmarks with negligible extra computation cost.

Original: By showing how pretraining brings beneﬁts other than learning content knowledge, disentangling the study of its working mechanism.
Revised: By showing how pretraining brings beneﬁts other than learning content knowledge, disentangling the study of its working mechanism.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Learning Models Applied to Mathematics There has been increasing interest in applying deep learning methods to Interactive Theorem Provers (ITP) (Bansal et al.; 2019; Gauthier et al., 2020;
Revised: Learning Models Applied to Mathematics There has been increasing interest in applying deep learning methods to Interactive Theorem Provers (ITP) (Bansal et al.; 2019; Gauthier et al., 2020;

Original: Huang et al., 2019; Yang and Deng, 2019; Wu et al., 2020; Li et al., 2020; Polu and Sutskever, 2020).
Revised: Huang et al., 2019; Yang and Deng, 2019; Wu et al., 2020; Li et al., 2020; Polu and Sutskever, 2020).

Original: The work that is most related to ours is GPT- f (Polu and Sutskever, 2020).
Revised: The work that is most related to ours is GPT- f (Polu and Sutskever, 2020).

Original: The authors performed pretraining on several natural language corpora and showed signiﬁcant improvements for an ITP system – MetaMath.
Revised: The authors performed pretraining on several natural language corpora and showed signiﬁcant improvements for an ITP system – MetaMath.

Original: Different from ours, they used GPT-style large-scale language modeling pretraining, which dominates the computation cost compared to the downstream task.
Revised: Different from ours, they used GPT-style large-scale language modeling pretraining, which dominates the computation cost compared to the downstream task.

Original: We, on the other hand, propose pretraining on a few lightweight synthetic tasks costing only a minor fraction of the computation spent on the downstream task.
Revised: We, on the other hand, propose pretraining on a few lightweight synthetic tasks costing only a minor fraction of the computation spent on the downstream task.

Original: Lample and Charton (2020) have demonstrated that transformer models can be used for symbolic mathematics by successfully predicting the integrals of formulas from a randomly generated dataset.
Revised: Lample and Charton (2020) have demonstrated that transformer models can be used for symbolic mathematics by successfully predicting the integrals of formulas from a randomly generated dataset.

Original: Similar observations are made for logical problems relevant to veriﬁcation: that transformer networks can learn the semantics of logics (Hahn et al., 2020).
Revised: Similar observations are made for logical problems relevant to veriﬁcation: that transformer networks can learn the semantics of logics (Hahn et al., 2020).

Original: Rabe et al.
Revised: Rabe et al.

Original: (2020) have shown that mathematical reasoning can emerge from self-supervised training alone.
Revised: (2020) have shown that mathematical reasoning can emerge from self-supervised training alone.

Original: Li et al.
Revised: Li et al.

Original: (2020) show that language models can learn to synthesize missing high-level intermediate propositions given a local context.
Revised: (2020) show that language models can learn to synthesize missing high-level intermediate propositions given a local context.

Original: Piotrowski and Urban (2020) used RNNs in automated theorem provers for ﬁrst-order logic.
Revised: Piotrowski and Urban (2020) used RNNs in automated theorem provers for ﬁrst-order logic.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) explored the use of machine translation to translate between synthetically generated natural language descriptions of proofs and formally represented proofs.
Revised: (2020) explored the use of machine translation to translate between synthetically generated natural language descriptions of proofs and formally represented proofs.

Original: Urban and Jakub˚uv (2020) present initial experiments on generating mathematical conjectures with a Transformer model.
Revised: Urban and Jakub˚uv (2020) present initial experiments on generating mathematical conjectures with a Transformer model.

Original: Saxton et al.
Revised: Saxton et al.

Original: (2019) suggest a dataset for the analysis of mathematical reasoning skills.
Revised: (2019) suggest a dataset for the analysis of mathematical reasoning skills.

Original: In contrast to the datasets considered here, their dataset is synthetic, focuses on calculation with concrete numbers, and only contains relatively few symbolic tasks.
Revised: In contrast to the datasets considered here, their dataset is synthetic, focuses on calculation with concrete numbers, and only contains relatively few symbolic tasks.

Original: Language Model Pretraining The advent of the transformer architecture (Vaswani et al., 2017) and the BERT style pretraining (Devlin et al., 2019) represented a huge improvement in the quality of language modeling.
Revised: Language Model Pretraining The advent of the transformer architecture (Vaswani et al., 2017) and the BERT style pretraining (Devlin et al., 2019) represented a huge improvement in the quality of language modeling.

Original: Since then, an explosion of research activity in the area pushed the quality of language models through better pretraining tasks.
Revised: Since then, an explosion of research activity in the area pushed the quality of language models through better pretraining tasks.

Original: Where BERT (Devlin et al., 2019) masks out a fraction of the input tokens, later works demonstrated the advantages of masking out subsequences (Song et al., 2019; Dong et al., 2019; Joshi et al., 2020; Raffel et al., 2020; Conneau and Lample, 2019) and whole sentences (Zhang et al., 2020).
Revised: Where BERT (Devlin et al., 2019) masks out a fraction of the input tokens, later works demonstrated the advantages of masking out subsequences (Song et al., 2019; Dong et al., 2019; Joshi et al., 2020; Raffel et al., 2020; Conneau and Lample, 2019) and whole sentences (Zhang et al., 2020).

Original: Besides the choice of pretraining tasks, the scale of language models is also an important factor.
Revised: Besides the choice of pretraining tasks, the scale of language models is also an important factor.

Original: Language models improve in quality and develop new abilities as they grow larger while trained on the same data (Radford et al., 2018; Raffel et al., 2020; Brown et al., 2020).
Revised: Language models improve in quality and develop new abilities as they grow larger while trained on the same data (Radford et al., 2018; Raffel et al., 2020; Brown et al., 2020).

Original: Inductive Biases in General There have been works studying learning inductive biases in other contexts.
Revised: Inductive Biases in General There have been works studying learning inductive biases in other contexts.

Original: In particular, McCoy et al.
Revised: In particular, McCoy et al.

Original: (2020) studied whether one can learn linguistic inductive biases on synthetic datasets via meta-learning.
Revised: (2020) studied whether one can learn linguistic inductive biases on synthetic datasets via meta-learning.

Original: Papadimitriou and Jurafsky (2020) shows inductive biases learned in music data can be useful for natural language.
Revised: Papadimitriou and Jurafsky (2020) shows inductive biases learned in music data can be useful for natural language.

Original: They further designed several synthetic tasks and showed similar kind of improvements for natural language tasks.
Revised: They further designed several synthetic tasks and showed similar kind of improvements for natural language tasks.

Original: From a more theoretical point of view, Xu et al.
Revised: From a more theoretical point of view, Xu et al.

Original: (2020) formalize an aspect of inductive (architectural) bias under the context of GNNs, with a notation called architectural alignment .
Revised: (2020) formalize an aspect of inductive (architectural) bias under the context of GNNs, with a notation called architectural alignment .

Original: The architecture is aligned when the architecture can perfectly simulates the ground truth solution.
Revised: The architecture is aligned when the architecture can perfectly simulates the ground truth solution.

Original: But their work is limited to showing alignment in combinatorial problems, whose ground truth solutions are known.
Revised: But their work is limited to showing alignment in combinatorial problems, whose ground truth solutions are known.

Original: In contrast, our work tries to learn architectural bias by relying on the ﬂexible Transformer architecture and training on synthetic datasets.
Revised: In contrast, our work tries to learn architectural bias by relying on the ﬂexible Transformer architecture and training on synthetic datasets.

Original: Inductive Biases for Mathematics
Revised: Inductive Biases for Mathematics

Original: Previous work studying inductive biases for logical reasoning has focused on encoding bias in the neural architecture.
Revised: Previous work studying inductive biases for logical reasoning has focused on encoding bias in the neural architecture.

Original: Initial works focused on encoding the tree structure of expressions using TreeRNNs (Evans et al., 2018).
Revised: Initial works focused on encoding the tree structure of expressions using TreeRNNs (Evans et al., 2018).

Original: Graph neural networks are shown to provide a much stronger performance than tree models in premise selection (Wang et al., 2017) and theorem proving (Paliwal et al., 2020).
Revised: Graph neural networks are shown to provide a much stronger performance than tree models in premise selection (Wang et al., 2017) and theorem proving (Paliwal et al., 2020).

Original: GNNs also scale to larger formulas in SAT (Selsam et al., 2019;
Revised: GNNs also scale to larger formulas in SAT (Selsam et al., 2019;

Original: Selsam and Bjørner, 2019; Han, 2020), QBF (Lederman et al., 2020), and #SAT (Vaezipoor et al., 2020).
Revised: Selsam and Bjørner, 2019; Han, 2020), QBF (Lederman et al., 2020), and #SAT (Vaezipoor et al., 2020).

Original: Crouse et al.
Revised: Crouse et al.

Original: (2019) have shown that pooling mechanisms can have an impact on the performance of GNNs on logical formulas as well.
Revised: (2019) have shown that pooling mechanisms can have an impact on the performance of GNNs on logical formulas as well.

Original: Closely related, Hellendoorn et al.
Revised: Closely related, Hellendoorn et al.

Original: (2020) have shown that it can be helpful to hard-code the tree structure of programs in the attention mask of transformers.
Revised: (2020) have shown that it can be helpful to hard-code the tree structure of programs in the attention mask of transformers.

Original: Schlag et al.
Revised: Schlag et al.

Original: (2019) developed an architecture for encoding relational information using tensor product representation for mathematical reasoning.
Revised: (2019) developed an architecture for encoding relational information using tensor product representation for mathematical reasoning.

Original:  M ETHODS
Revised:  M ETHODS

Original: In this section, we ﬁrst discuss the primitives of reasoning, inspired by Peirce’s views, and design one synthetic task for each reasoning primitive.
Revised: In this section, we ﬁrst discuss the primitives of reasoning, inspired by Peirce’s views, and design one synthetic task for each reasoning primitive.

Original: 3. R EASONING P RIMITIVES
Revised: 3. R EASONING P RIMITIVES

Original: In Peirce’s view, there are exactly three kinds of reasoning: deduction, abduction, and induction.
Revised: In Peirce’s view, there are exactly three kinds of reasoning: deduction, abduction, and induction.

Original: Deduction is known as the workhorse for mathematics.
Revised: Deduction is known as the workhorse for mathematics.

Original: It is the process of deriving new facts by applying logical inference rules to known facts or premises.
Revised: It is the process of deriving new facts by applying logical inference rules to known facts or premises.

Original: On the other hand, abduction and induction can be thought of as the inverses of deduction.
Revised: On the other hand, abduction and induction can be thought of as the inverses of deduction.

Original: If we call the premise used in deduction as
Revised: If we call the premise used in deduction as

Original: Case , its logical rule as Rule , and its conclusion as Result , then abduction is equivalently the inference of a Case from a Rule and a Result, while induction may be said to be the inference of a Rule from a
Revised: Case , its logical rule as Rule , and its conclusion as Result , then abduction is equivalently the inference of a Case from a Rule and a Result, while induction may be said to be the inference of a Rule from a

Original: Case and a Result.
Revised: Case and a Result.

Original: We summarize the three reasoning primitives in the following table:
Revised: We summarize the three reasoning primitives in the following table:

Original: Reasoning Primitives Inference Map Deduction Rule, Case → Result Abduction Rule, Result → Case Induction Case, Result → Rule
Revised: Reasoning Primitives Inference Map Deduction Rule, Case → Result Abduction Rule, Result → Case Induction Case, Result → Rule

Original: To give an example, we let Rule be “All the beans in this bag are white”, Case be “These beans are from this bag”, and Result be “These beans are white”.
Revised: To give an example, we let Rule be “All the beans in this bag are white”, Case be “These beans are from this bag”, and Result be “These beans are white”.

Original: Deduction is to derive the fact that these beans are white (Re) from knowing all the beans from this bag are white (R) and these beans are from this bag (C).
Revised: Deduction is to derive the fact that these beans are white (Re) from knowing all the beans from this bag are white (R) and these beans are from this bag (C).

Original: Abduction explains why the beans are white (Re) from knowing that all the beans in the bag are white (R) – because these beans must be from the bag (C).
Revised: Abduction explains why the beans are white (Re) from knowing that all the beans in the bag are white (R) – because these beans must be from the bag (C).

Original: Lastly, induction aims to provide a general principle to observing the fact that the beans are white (Re) and they come from this bag (C), which is that all the beans in the bag must be white (R).
Revised: Lastly, induction aims to provide a general principle to observing the fact that the beans are white (Re) and they come from this bag (C), which is that all the beans in the bag must be white (R).

Original: We refer to Peirce (1992) and Bellucci and Pietarinen (2015) for more elaborate discussions on the primitives of reasoning.
Revised: We refer to Peirce (1992) and Bellucci and Pietarinen (2015) for more elaborate discussions on the primitives of reasoning.

Original: Mathematical reasoning exhibits nontrivial uses of these reasoning primitives.
Revised: Mathematical reasoning exhibits nontrivial uses of these reasoning primitives.

Original: Deduction happens when one needs to derive new valid statements from the given premise (Case) and theorems in the library (Rule).
Revised: Deduction happens when one needs to derive new valid statements from the given premise (Case) and theorems in the library (Rule).

Original: Abduction is used to postulate conjectures from the known facts and theorems, allowing one to decompose the challenging theorem into subgoals for proof.
Revised: Abduction is used to postulate conjectures from the known facts and theorems, allowing one to decompose the challenging theorem into subgoals for proof.

Original: Induction, the ability to extract general principles from known facts and theorems is also one of the major activities of mathematical reasoning.
Revised: Induction, the ability to extract general principles from known facts and theorems is also one of the major activities of mathematical reasoning.

Original: It is used when one derives theorems from special cases and proposes new deﬁnitions and general frameworks to encapsulate existing knowledge.
Revised: It is used when one derives theorems from special cases and proposes new deﬁnitions and general frameworks to encapsulate existing knowledge.

Original: 3. LIME S YNTHETIC T ASKS F OR R EASONING P RIMITIVES
Revised: 3. LIME S YNTHETIC T ASKS F OR R EASONING P RIMITIVES

Original: We design three synthetic tasks inspired by the three reasoning primitives.
Revised: We design three synthetic tasks inspired by the three reasoning primitives.

Original: As discussed in the previous section, all of the reasoning primitives consist of three essential elements: Rule, Case, and
Revised: As discussed in the previous section, all of the reasoning primitives consist of three essential elements: Rule, Case, and

Original: Result.
Revised: Result.

Original: Inspired by this, we ﬁrst design a method to generate those elements.
Revised: Inspired by this, we ﬁrst design a method to generate those elements.

Original: Once they are generated, we can construct tasks that predict one element from the other two.
Revised: Once they are generated, we can construct tasks that predict one element from the other two.

Original: In the following, we describe one simple way to generate those three elements, though we acknowledge that there are many other possible approaches.
Revised: In the following, we describe one simple way to generate those three elements, though we acknowledge that there are many other possible approaches.

Original: We require two types of symbols: 1. math symbols , 2.
Revised: We require two types of symbols: 1. math symbols , 2.

Original: rule symbols .
Revised: rule symbols .

Original: In general, these symbols can take any forms (e.g., integer representations).
Revised: In general, these symbols can take any forms (e.g., integer representations).

Original: But for the ease of discussion, we will think of math symbols as the union of those operators used in mathematics (e.g., “ + − ∗ =
Revised: But for the ease of discussion, we will think of math symbols as the union of those operators used in mathematics (e.g., “ + − ∗ =

Original: ()& ”) and lower case letters (e.g., a , b , c ...), and rule symbols as upper case letters (e.g., A , B , C ...).
Revised: ()& ”) and lower case letters (e.g., a , b , c ...), and rule symbols as upper case letters (e.g., A , B , C ...).

Original: We now construct
Revised: We now construct

Original: Rule, Case, and Result in order:
Revised: Rule, Case, and Result in order:

Original: Rule is a randomly sampled string that consists of i) rule symbols and ii) math symbols.
Revised: Rule is a randomly sampled string that consists of i) rule symbols and ii) math symbols.

Original: The length of the string is randomly sampled from a range.
Revised: The length of the string is randomly sampled from a range.

Original: For instance, a randomly sampled rule can be: A ∗ A + B = C with rule symbols A , B , and C . 2.
Revised: For instance, a randomly sampled rule can be: A ∗ A + B = C with rule symbols A , B , and C . 2.

Original: Case is a dictionary that represents substitutions.
Revised: Case is a dictionary that represents substitutions.

Original: For each rule symbol used in the Rule string, we sample a random string of random length that consists of math symbols.
Revised: For each rule symbol used in the Rule string, we sample a random string of random length that consists of math symbols.

Original: This forms a dictionary, whose keys are all rule symbols, and the values are the corresponding sampled string.
Revised: This forms a dictionary, whose keys are all rule symbols, and the values are the corresponding sampled string.

Original: To illustrate, following the previous example, for each A , B and C , we sample a random string to form a dictionary as: { A : a, B : b, C : d + e } . 3.
Revised: To illustrate, following the previous example, for each A , B and C , we sample a random string to form a dictionary as: { A : a, B : b, C : d + e } . 3.

Original: Result is the outcome of the substitution.
Revised: Result is the outcome of the substitution.

Original: For each rule symbol in the Rule string, we replace it with the corresponding value stored in the Case dictionary.
Revised: For each rule symbol in the Rule string, we replace it with the corresponding value stored in the Case dictionary.

Original: This gives rise to the Result string.
Revised: This gives rise to the Result string.

Original: As per the previous example, we now substitute A with a , B with b , and C with d + e into the Rule string, generating the Result string: a ∗ a + b = d + e .
Revised: As per the previous example, we now substitute A with a , B with b , and C with d + e into the Rule string, generating the Result string: a ∗ a + b = d + e .

Original: After Rule, Case, and Result are generated, we can construct three tasks for deduction, abduction, and induction respectively.
Revised: After Rule, Case, and Result are generated, we can construct three tasks for deduction, abduction, and induction respectively.

Original: We deﬁne the three synthetic tasks as follows:
Revised: We deﬁne the three synthetic tasks as follows:

Original: • Deduct : Source: Rule string and Case dictionary.
Revised: • Deduct : Source: Rule string and Case dictionary.

Original: Target:
Revised: Target:

Original: Result string.
Revised: Result string.

Original: Abduct : Source: Rule string and Result string.
Revised: Abduct : Source: Rule string and Result string.

Original: Target:
Revised: Target:

Original: Case dictionary.
Revised: Case dictionary.

Original: • Induct : Source: Case dictionary and Result string.
Revised: • Induct : Source: Case dictionary and Result string.

Original: Target:
Revised: Target:

Original: Rule string.
Revised: Rule string.

Original: We also consider a task called Mix , which is a uniform mix of three tasks.
Revised: We also consider a task called Mix , which is a uniform mix of three tasks.

Original: Namely, during generation, we randomly select a task and sample an example from that task.
Revised: Namely, during generation, we randomly select a task and sample an example from that task.

Original: To formulate them as sequence to sequence tasks, we represent the Case dictionary also as a string, e.g., “ { A : a, B : b, C : d + e } ”.
Revised: To formulate them as sequence to sequence tasks, we represent the Case dictionary also as a string, e.g., “ { A : a, B : b, C : d + e } ”.

Original: An example of Abduct using the examples of Rule, Case, and Result above is to predict the target { A : a, B : b, C : d + e } from the source A ∗
Revised: An example of Abduct using the examples of Rule, Case, and Result above is to predict the target { A : a, B : b, C : d + e } from the source A ∗

Original: A + B = C 
 a ∗
Revised: A + B = C 
 a ∗

Original: a + b = d + e .
Revised: a + b = d + e .

Original: Pre-training on our synthetic tasks can be seen as a form of skip-component learning.
Revised: Pre-training on our synthetic tasks can be seen as a form of skip-component learning.

Original: There are three essential components: Rule, Case and Result, and we skip one of them and use the remaining two elements to reconstruct the missing one.
Revised: There are three essential components: Rule, Case and Result, and we skip one of them and use the remaining two elements to reconstruct the missing one.

Original: Past work has shown that learning to predict missing words (Devlin et al., 2019), subsequences (Song et al., 2019; Raffel et al., 2020), or subtrees (Rabe et al., 2020) are strong pre-training tasks.
Revised: Past work has shown that learning to predict missing words (Devlin et al., 2019), subsequences (Song et al., 2019; Raffel et al., 2020), or subtrees (Rabe et al., 2020) are strong pre-training tasks.

Original: 3. S YMBOL -A GNOSTIC R EPRESENTATION
Revised: 3. S YMBOL -A GNOSTIC R EPRESENTATION

Original: In order to solve the synthetic tasks, the model needs to distinguish which set of symbols can be substituted (rule symbols).
Revised: In order to solve the synthetic tasks, the model needs to distinguish which set of symbols can be substituted (rule symbols).

Original: As a result, the model may memorize information about the symbols that is irrelevant to the inductive biases encoded in the task.
Revised: As a result, the model may memorize information about the symbols that is irrelevant to the inductive biases encoded in the task.

Original: To prevent such memorization, we propose a way to make the synthetic tasks agnostic to the choice of symbols.
Revised: To prevent such memorization, we propose a way to make the synthetic tasks agnostic to the choice of symbols.

Original: We ﬁrst note that the choice of symbols is irrelevant to our synthetic tasks.
Revised: We ﬁrst note that the choice of symbols is irrelevant to our synthetic tasks.

Original: To avoid symbol-speciﬁc memorization, for each training and evaluation example, we randomly sample two sets of symbols to be used in Rules and in the rest of the example.
Revised: To avoid symbol-speciﬁc memorization, for each training and evaluation example, we randomly sample two sets of symbols to be used in Rules and in the rest of the example.

Original: But for the Abduct task, the model needs to know which symbols are replaced by the Rule part of the example and which symbols are in the
Revised: But for the Abduct task, the model needs to know which symbols are replaced by the Rule part of the example and which symbols are in the

Original: Result language.
Revised: Result language.

Original: We simply list the split of the symbols used in the example at the beginning of the input string, marked by two special symbols, <Rule> and <
Revised: We simply list the split of the symbols used in the example at the beginning of the input string, marked by two special symbols, <Rule> and <

Original: Math> .
Revised: Math> .

Original: They are followed by the original source string.
Revised: They are followed by the original source string.

Original: The target string remains unchanged.
Revised: The target string remains unchanged.

Original: For example, the previous example in the Abduct task becomes,
Revised: For example, the previous example in the Abduct task becomes,

Original: [Equation]
Revised: [Equation]

Original: In our implementation, we use integers to represent symbols.
Revised: In our implementation, we use integers to represent symbols.

Original: Speciﬁcally, for each example, we sample two disjoint sets of integers from the set { 1 , . . .
Revised: Speciﬁcally, for each example, we sample two disjoint sets of integers from the set { 1 , . . .

Original: , S } to represent the math symbols and the rule symbols, where S is the size of the vocabulary.
Revised: , S } to represent the math symbols and the rule symbols, where S is the size of the vocabulary.

Original: In our experiments, we sample 44 math symbols and 24 rule symbols for each problem.
Revised: In our experiments, we sample 44 math symbols and 24 rule symbols for each problem.

Original: The complete pseudo-code of generating the symbols, Rule,
Revised: The complete pseudo-code of generating the symbols, Rule,

Original: Case, and Result for one task example is provided in Appendix Algorithm 1.
Revised: Case, and Result for one task example is provided in Appendix Algorithm 1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present results on three large mathematical reasoning tasks that are especially useful in the context of automated theorem proving.
Revised: In this section, we present results on three large mathematical reasoning tasks that are especially useful in the context of automated theorem proving.

Original: Our results show signiﬁcant gains in learning inductive biases from synthetic tasks.
Revised: Our results show signiﬁcant gains in learning inductive biases from synthetic tasks.

Original: We have selected three tasks to cover three different styles of interactive theorem provers: The HOL-Light (skip-tree) corpus was created from very high-level tactic-based proofs, but it is less interpretable than IsarStep’s declarative style corpus.
Revised: We have selected three tasks to cover three different styles of interactive theorem provers: The HOL-Light (skip-tree) corpus was created from very high-level tactic-based proofs, but it is less interpretable than IsarStep’s declarative style corpus.

Original: We also evaluate the next proof-step prediction task on the set.mm library of MetaMath, which consists of very granular, basic proof steps.
Revised: We also evaluate the next proof-step prediction task on the set.mm library of MetaMath, which consists of very granular, basic proof steps.

Original: Namely, the proof steps are more predicable and average proof lengths have signiﬁcantly increased.
Revised: Namely, the proof steps are more predicable and average proof lengths have signiﬁcantly increased.

Original: 4. E XPERIMENT D ETAILS
Revised: 4. E XPERIMENT D ETAILS

Original: LIME Pretraining We generate datasets of our synthetic tasks for pretraining: Deduct , Abduct ,
Revised: LIME Pretraining We generate datasets of our synthetic tasks for pretraining: Deduct , Abduct ,

Original: Induct , Mix .
Revised: Induct , Mix .

Original: For pretraining of IsarStep, we used a vocabulary size S of 1000 .
Revised: For pretraining of IsarStep, we used a vocabulary size S of 1000 .

Original: For the other two downstream tasks, we used a vocabulary size of 100 .
Revised: For the other two downstream tasks, we used a vocabulary size of 100 .

Original: The reason we used different vocabulary sizes was that we found (cf. appendix) the discrepancy in vocabulary size affects the performance of a downstream task if it has a very large vocabulary size (IsarStep has 28 K).
Revised: The reason we used different vocabulary sizes was that we found (cf. appendix) the discrepancy in vocabulary size affects the performance of a downstream task if it has a very large vocabulary size (IsarStep has 28 K).

Original: We use 44 math symbols and 24 rule symbols.
Revised: We use 44 math symbols and 24 rule symbols.

Original: The length of the Rule string is sampled from 5 to 20 , the length of the string for each substitution (the values of Case dictionary) is sampled from 2 to 8.
Revised: The length of the Rule string is sampled from 5 to 20 , the length of the string for each substitution (the values of Case dictionary) is sampled from 2 to 8.

Original: We used word-level tokenization for all the tasks.
Revised: We used word-level tokenization for all the tasks.

Original: We pretrained the model for 20 K updates.
Revised: We pretrained the model for 20 K updates.

Original: For tasks with larger vocabulary size (i.e., 1000 ), we found the learning became more difﬁcult.
Revised: For tasks with larger vocabulary size (i.e., 1000 ), we found the learning became more difﬁcult.

Original: Hence we used a curriculum learning scheme: we ﬁrst trained the model for 10 K steps on the same task with a vocabulary size of 100 , then continue training for another 10 K step on vocabulary size of 1000 .
Revised: Hence we used a curriculum learning scheme: we ﬁrst trained the model for 10 K steps on the same task with a vocabulary size of 100 , then continue training for another 10 K step on vocabulary size of 1000 .

Original: The pretraining was done on a single Nvidia Tesla T

Revised: The pretraining was done on a single Nvidia Tesla T

Original: GPU with 4 CPU cores for 2 hours.
Revised: GPU with 4 CPU cores for 2 hours.

Original: We set the maximum number of tokens in a batch to 4096 , and accumulate four batches of gradients for one parameter update.
Revised: We set the maximum number of tokens in a batch to 4096 , and accumulate four batches of gradients for one parameter update.

Original: We used the Adam optimizer (Kingma and Ba, 2015) with learning rate 3 · 10 − 4 .
Revised: We used the Adam optimizer (Kingma and Ba, 2015) with learning rate 3 · 10 − 4 .

Original: We used a dropout rate of 0 . 1 and label smoothing (Szegedy et al., 2016) with a coefﬁcient 0 . 1 .
Revised: We used a dropout rate of 0 . 1 and label smoothing (Szegedy et al., 2016) with a coefﬁcient 0 . 1 .

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Fine-tuning For all the downstream tasks in this section, when loading the pretrained models for ﬁne-tuning, we do not load in the vocabulary embeddings nor the output layer weights.
Revised: Fine-tuning For all the downstream tasks in this section, when loading the pretrained models for ﬁne-tuning, we do not load in the vocabulary embeddings nor the output layer weights.

Original: For the downstream task IsarStep and MetaMathStep, we used four Nvidia Tesla T4 GPU with 16 CPU cores for training.
Revised: For the downstream task IsarStep and MetaMathStep, we used four Nvidia Tesla T4 GPU with 16 CPU cores for training.

Original: We set the maximum number of tokens in a batch to 4096 , and accumulated four batches of gradients for one parameter update.
Revised: We set the maximum number of tokens in a batch to 4096 , and accumulated four batches of gradients for one parameter update.

Original: We trained the model for 200 K updates.
Revised: We trained the model for 200 K updates.

Original: We used the
Revised: We used the

Original: Adam optimizer
Revised: Adam optimizer

Original: [Table]
Revised: [Table]

Original: and we searched over the learning rates { 3 · 10 − 4 , 7 · 10 − 4 }
Revised: and we searched over the learning rates { 3 · 10 − 4 , 7 · 10 − 4 }

Original: [Table]
Revised: [Table]

Original: and warmup steps { 4000 , 8000 } .
Revised: and warmup steps { 4000 , 8000 } .

Original: We used a dropout rate of 0 . 1 and label smoothing with a coefﬁcient 0 . 1 .
Revised: We used a dropout rate of 0 . 1 and label smoothing with a coefﬁcient 0 . 1 .

Original: For the
Revised: For the

Original: HOList skip-tree task, we used TPUs for running the experiments.
Revised: HOList skip-tree task, we used TPUs for running the experiments.

Original: We used a batch size ofsequences and trained the model for 1 million updates.
Revised: We used a batch size ofsequences and trained the model for 1 million updates.

Original: Architecture All experiments used the transformer base model from Vaswani et al.
Revised: Architecture All experiments used the transformer base model from Vaswani et al.

Original: (2017), i.e.hidden size, 2048 ﬁlter size, 8 attention heads.
Revised: (2017), i.e.hidden size, 2048 ﬁlter size, 8 attention heads.

Original: For the IsarStep and MetaMathStep task, we usedlayers for both the encoder and decoder, implemented using fairseq (Ott et al., 2019).
Revised: For the IsarStep and MetaMathStep task, we usedlayers for both the encoder and decoder, implemented using fairseq (Ott et al., 2019).

Original: For the HOList skip-tree experiment, we used a somewhat modiﬁed transformer architecture with 8 encoder anddecoder layers of the same size as above in which the self-attention and attention over the encoder output were merged.
Revised: For the HOList skip-tree experiment, we used a somewhat modiﬁed transformer architecture with 8 encoder anddecoder layers of the same size as above in which the self-attention and attention over the encoder output were merged.

Original: Evaluation During training, we kept track of the best validation tokenized BLEU score 1 , and we used the model with validation BLEU for evaluation on the test set.
Revised: Evaluation During training, we kept track of the best validation tokenized BLEU score 1 , and we used the model with validation BLEU for evaluation on the test set.

Original: We report top-1 and topaccuracies.
Revised: We report top-1 and topaccuracies.

Original: We consider an output sequence as correct if it matches the target sequence exactly.
Revised: We consider an output sequence as correct if it matches the target sequence exactly.

Original: We performed a beam search with width 10.
Revised: We performed a beam search with width 10.

Original: The top-1 accuracy is then deﬁned as the percentage of the best output sequences that are correct.
Revised: The top-1 accuracy is then deﬁned as the percentage of the best output sequences that are correct.

Original: The top- n accuracy is deﬁned as the percentage of target sequences appearing in the top n generated sequences.
Revised: The top- n accuracy is deﬁned as the percentage of target sequences appearing in the top n generated sequences.

Original: 4. I SAR S TEP
Revised: 4. I SAR S TEP

Original: The IsarStep task is taken from Li et al.
Revised: The IsarStep task is taken from Li et al.

Original: IsarStep is a task of predicting the missing intermediate propositions given surrounding propositions to bridge the gap between the goal and the current state of the proof.
Revised: IsarStep is a task of predicting the missing intermediate propositions given surrounding propositions to bridge the gap between the goal and the current state of the proof.

Original: The dataset was mined from the public repository of formal proofs of the Isabelle proof assistant (Paulson, 1994).
Revised: The dataset was mined from the public repository of formal proofs of the Isabelle proof assistant (Paulson, 1994).

Original: Unlike HOList and MetaMath, IsarStep contains mostly declarative proofs, a proof style close to humans’ prose proofs.
Revised: Unlike HOList and MetaMath, IsarStep contains mostly declarative proofs, a proof style close to humans’ prose proofs.

Original: The dataset has a broad coverage of undergraduate and research-level mathematics and computer science theorems.
Revised: The dataset has a broad coverage of undergraduate and research-level mathematics and computer science theorems.

Original: There are 820K, 5000, 5000 sequence pairs for the training, validation, and test sets with a maximum of 800 tokens in source sequences and 200 tokens in the target sequences.
Revised: There are 820K, 5000, 5000 sequence pairs for the training, validation, and test sets with a maximum of 800 tokens in source sequences and 200 tokens in the target sequences.

Original: Following Li et al.
Revised: Following Li et al.

Original: (2020), during training, we use 512 as the
Revised: (2020), during training, we use 512 as the

Original: [Table]
Revised: [Table]

Original: maximum length for both the source and target, and truncated those that exceed the length to 512.
Revised: maximum length for both the source and target, and truncated those that exceed the length to 512.

Original: For reporting, we evaluate all 5000 test examples regardless of their lengths.
Revised: For reporting, we evaluate all 5000 test examples regardless of their lengths.

Original: [Figure]
Revised: [Figure]

Original: The results on the IsarStep task for four pretrained models and the baseline transformer model without pretraining is shown in Table 1.
Revised: The results on the IsarStep task for four pretrained models and the baseline transformer model without pretraining is shown in Table 1.

Original: We also include another baseline, HAT transformer introduced in Li et al.
Revised: We also include another baseline, HAT transformer introduced in Li et al.

Original: (2020), which is a specially designed hierarchical transformer architecture tailored to this task.
Revised: (2020), which is a specially designed hierarchical transformer architecture tailored to this task.

Original: We see the pretrained model achieved substantial improvement over the model trained from scratch as well as HAT.
Revised: We see the pretrained model achieved substantial improvement over the model trained from scratch as well as HAT.

Original: Notably, the model that was pretrained on Abduct improved the top-10 accuracy from 33 . 1% to 41 .
Revised: Notably, the model that was pretrained on Abduct improved the top-10 accuracy from 33 . 1% to 41 .

Original: 0% , for almost 8% absolute improvement.
Revised: 0% , for almost 8% absolute improvement.

Original: The model pretrained on Mix performed the best on top-1 accuracy, improving the baseline by 6 . 5% accuracy.
Revised: The model pretrained on Mix performed the best on top-1 accuracy, improving the baseline by 6 . 5% accuracy.

Original: We also showed the validation
Revised: We also showed the validation

Original: BLEU scores along training in Figure 1.
Revised: BLEU scores along training in Figure 1.

Original: We can see that the pretrained models learned much faster than the model trained from scratch.
Revised: We can see that the pretrained models learned much faster than the model trained from scratch.

Original: With around 50 K steps of updates, the pretrained model already obtained better BLEU scores than the best score achieved by the un-pretrained model.
Revised: With around 50 K steps of updates, the pretrained model already obtained better BLEU scores than the best score achieved by the un-pretrained model.

Original: Moreover, since the downstream task requires 200 K steps of training with 4 GPUs, the amount of computation spent on pretraining is only 2 .
Revised: Moreover, since the downstream task requires 200 K steps of training with 4 GPUs, the amount of computation spent on pretraining is only 2 .

Original: 5% of the downstream task, strongly demonstrating the efﬁciency of the proposed
Revised: 5% of the downstream task, strongly demonstrating the efﬁciency of the proposed

Original: pretraining method.
Revised: pretraining method.

Original: 4. HOL IST S KIP -T REE
Revised: 4. HOL IST S KIP -T REE

Original: As the second mathematical reasoning benchmark we consider the HOList skip-tree evaluation tasks by Rabe et al.
Revised: As the second mathematical reasoning benchmark we consider the HOList skip-tree evaluation tasks by Rabe et al.

Original: These tasks include two variants of type inference, predicting under which assumptions theorems hold, and completing equalities.
Revised: These tasks include two variants of type inference, predicting under which assumptions theorems hold, and completing equalities.

Original: All source expressions for these tasks are taken from the validation set of the theorem database of the HOList proof logs (Bansal et al.).
Revised: All source expressions for these tasks are taken from the validation set of the theorem database of the HOList proof logs (Bansal et al.).

Original: The evaluations are done on a random sample of 1000 instances from the full evaluation sets.
Revised: The evaluations are done on a random sample of 1000 instances from the full evaluation sets.

Original: We initialized the model parameters with the pretrained weights and then repeated the experiments by Rabe et al.
Revised: We initialized the model parameters with the pretrained weights and then repeated the experiments by Rabe et al.

Original: That is, we trained the models for up to 1M parameter updates on the training set with batch size 256 and repeat the evaluation every 100K steps.
Revised: That is, we trained the models for up to 1M parameter updates on the training set with batch size 256 and repeat the evaluation every 100K steps.

Original: In Table 2 we present the best result from these 10 evaluation runs.
Revised: In Table 2 we present the best result from these 10 evaluation runs.

Original: We see a signiﬁcant improvement in these reasoning tasks when the models are initialized with the pretrained weights.
Revised: We see a signiﬁcant improvement in these reasoning tasks when the models are initialized with the pretrained weights.

Original: Notably, on equation completion and missing assumptions task, we improved the beam search (with width 8 ) exact match rate performance from 46 .
Revised: Notably, on equation completion and missing assumptions task, we improved the beam search (with width 8 ) exact match rate performance from 46 .

Original: 3% to 51 .
Revised: 3% to 51 .

Original: 7% and 41 .
Revised: 7% and 41 .

Original: 8% to 47 .
Revised: 8% to 47 .

Original: Note that this is despite the amount of pretraining compute cost being negligible: it takes less than 1 percent of the cost of the downstream task training.
Revised: Note that this is despite the amount of pretraining compute cost being negligible: it takes less than 1 percent of the cost of the downstream task training.

Original: Pretraining used 1 / 20 number of the update steps ( 50 K vs 1 M) with 8 (and 4 ) times smaller batches (pretraining has much shorter sequence lengths, 128 vs. 1024 and 512 , respectively).
Revised: Pretraining used 1 / 20 number of the update steps ( 50 K vs 1 M) with 8 (and 4 ) times smaller batches (pretraining has much shorter sequence lengths, 128 vs. 1024 and 512 , respectively).

Original: 4. M ETA M ATH S TEP
Revised: 4. M ETA M ATH S TEP

Original: Compared to other ITPs, MetaMath is a low-level proving system: each proof step makes only a small step towards the goal.
Revised: Compared to other ITPs, MetaMath is a low-level proving system: each proof step makes only a small step towards the goal.

Original: As such, each proof contains many more proof steps than in other ITPs: with 37 , 000 theorems in the human-written theorem library, there are around 3 million proof steps.
Revised: As such, each proof contains many more proof steps than in other ITPs: with 37 , 000 theorems in the human-written theorem library, there are around 3 million proof steps.

Original: [Table]
Revised: [Table]

Original: We extract the proof steps and use them to construct a sequence-to-sequence task following Polu and
Revised: We extract the proof steps and use them to construct a sequence-to-sequence task following Polu and

Original: Sutskever (2020) (their proof step training objective).
Revised: Sutskever (2020) (their proof step training objective).

Original: In this task, the model is asked to generate PROOFSTEPS given a GOAL, namely, the GOAL string is the source input, and PROOFSTEPS is the target output.
Revised: In this task, the model is asked to generate PROOFSTEPS given a GOAL, namely, the GOAL string is the source input, and PROOFSTEPS is the target output.

Original: We follow Polu and Sutskever (2020) and use their string representation for the GOAL and the PROOFSTEPS.
Revised: We follow Polu and Sutskever (2020) and use their string representation for the GOAL and the PROOFSTEPS.

Original: Instead of using subword tokenization in Polu and Sutskever (2020), we use a character-level representation for our task.
Revised: Instead of using subword tokenization in Polu and Sutskever (2020), we use a character-level representation for our task.

Original: Following Polu and Sutskever (2020), we split theorems into train/valid/test theorems of size 35 K, 1 K, 1 K, and associate all proof steps of a theorem with that split.
Revised: Following Polu and Sutskever (2020), we split theorems into train/valid/test theorems of size 35 K, 1 K, 1 K, and associate all proof steps of a theorem with that split.

Original: For each dataset, we ﬁlter examples with lengths longer than 1024.
Revised: For each dataset, we ﬁlter examples with lengths longer than 1024.

Original: This reduced the total number of proof steps to 1 .
Revised: This reduced the total number of proof steps to 1 .

Original: 4 million.
Revised: 4 million.

Original: For validation and test set, we randomly sample 3000 examples out of 40 K (after ﬁltering) and perform validation and test evaluations on them.
Revised: For validation and test set, we randomly sample 3000 examples out of 40 K (after ﬁltering) and perform validation and test evaluations on them.

Original: In Table 3 we present the impact of pretraining on our synthetic reasoning tasks on MetaMathStep.
Revised: In Table 3 we present the impact of pretraining on our synthetic reasoning tasks on MetaMathStep.

Original: We also observe gains from pretraining on this dataset, with the model trained on Induct task achieving 2 . 2% top- 1 and 1 .
Revised: We also observe gains from pretraining on this dataset, with the model trained on Induct task achieving 2 . 2% top- 1 and 1 .

Original: 5% top- 10 test accuracy improvement.
Revised: 5% top- 10 test accuracy improvement.

Original: Similarly, as for the IsarStep task, the computation spent on pretraining is only 2 .
Revised: Similarly, as for the IsarStep task, the computation spent on pretraining is only 2 .

Original: 5% of the downstream task.
Revised: 5% of the downstream task.

Original:  A BLATION S TUDIES
Revised:  A BLATION S TUDIES

Original: In this section, we perform ablation studies.
Revised: In this section, we perform ablation studies.

Original: Additional ablation studies can be found in Appendix C.
Revised: Additional ablation studies can be found in Appendix C.

Original: 5. P RETRAINING ON F ORMAL R EASONING AND N ATURAL L ANGUAGE T ASKS
Revised: 5. P RETRAINING ON F ORMAL R EASONING AND N ATURAL L ANGUAGE T ASKS

Original: Here we investigate how LIME compares to pretraining on natural language or existing formal reasoning datasets.
Revised: Here we investigate how LIME compares to pretraining on natural language or existing formal reasoning datasets.

Original: In this set of experiments, we pretrained three models on Mix , MetaMathStep, and on the WMT 2016 English-to-Germany (WMT En-De) translation task, and then we ﬁne-tuned and evaluated these models on the IsarStep task.
Revised: In this set of experiments, we pretrained three models on Mix , MetaMathStep, and on the WMT 2016 English-to-Germany (WMT En-De) translation task, and then we ﬁne-tuned and evaluated these models on the IsarStep task.

Original: We pretrained the model on MetaMathStep and
Revised: We pretrained the model on MetaMathStep and

Original: WMT EN-DE for 200 K steps with 4 GPUs, which is 40 times more computation spent than on LIME.
Revised: WMT EN-DE for 200 K steps with 4 GPUs, which is 40 times more computation spent than on LIME.

Original: Due to the mismatch between vocabularies of the pretraining task and the downstream task, we do not load the vocabulary embeddings nor output layer weights.
Revised: Due to the mismatch between vocabularies of the pretraining task and the downstream task, we do not load the vocabulary embeddings nor output layer weights.

Original: The results in Table 4 show that pretraining on MetaMathStep did provide gains, though signiﬁcantly smaller than gains provided by LIME Mix , despite their 40 times higher computational cost.
Revised: The results in Table 4 show that pretraining on MetaMathStep did provide gains, though signiﬁcantly smaller than gains provided by LIME Mix , despite their 40 times higher computational cost.

Original: Moreover, pre-training on WMT translation had even a negative effect on the performance.
Revised: Moreover, pre-training on WMT translation had even a negative effect on the performance.

Original: We also conducted an analogous experiment with an evaluation on the MetaMathStep, which we present in Appendix C.
Revised: We also conducted an analogous experiment with an evaluation on the MetaMathStep, which we present in Appendix C.

Original: 5. D O WE NEED VOCABULARY EMBEDDINGS FOR FINE - TUNING ?
Revised: 5. D O WE NEED VOCABULARY EMBEDDINGS FOR FINE - TUNING ?

Original: As mentioned earlier, we did not load in the vocabulary embeddings from the pretrained models when we switched to ﬁne-tuning on downstream tasks.
Revised: As mentioned earlier, we did not load in the vocabulary embeddings from the pretrained models when we switched to ﬁne-tuning on downstream tasks.

Original: Even without loading the vocab embeddings, the pretrained models still improved the performance.
Revised: Even without loading the vocab embeddings, the pretrained models still improved the performance.

Original: In this ablation study, we investigate how much this decision has affected the results and whether vocabulary embeddings can help improve the performance even further.
Revised: In this ablation study, we investigate how much this decision has affected the results and whether vocabulary embeddings can help improve the performance even further.

Original: We performed the comparisons on IsarStep.
Revised: We performed the comparisons on IsarStep.

Original: The task contains a token vocabulary of size 28336.
Revised: The task contains a token vocabulary of size 28336.

Original: We generated new synthetic tasks for the same vocabulary size, such that we can load the vocabulary embeddings and output layers when initializing the model for IsarStep.
Revised: We generated new synthetic tasks for the same vocabulary size, such that we can load the vocabulary embeddings and output layers when initializing the model for IsarStep.

Original: Table 5 shows that this led to similar performance.
Revised: Table 5 shows that this led to similar performance.

Original: This aligns with our expectation that the model should not learn content speciﬁc knowledge that is potentially stored in the vocabulary.
Revised: This aligns with our expectation that the model should not learn content speciﬁc knowledge that is potentially stored in the vocabulary.

Original: These weights turn out to be non-essential for the ﬁnal performance, supporting the evidence that the transformer learns inductive biases from the pretraining task.
Revised: These weights turn out to be non-essential for the ﬁnal performance, supporting the evidence that the transformer learns inductive biases from the pretraining task.

Original: [Table]
Revised: [Table]

Revised:  D O THOSE TASKS ENCODE I NDUCTION , DEDUCTION AND ABDUCTION ?

Revised: Although LIME has shown to achieve substantial improvements across various benchmarks, it is not entirely clear that the speciﬁc synthetic tasks necessarily enforce the reasoning ability of induction, deduction and abduction.

Revised: We would like to note that deduction, induction, and abduction are highlevel and philosophical concepts, and serve only as an inspiration for us to design the synthetic tasks.

Revised: We do not expect the model will necessarily learn exactly these three capabilities.

Original:   C ONCLUSION
Revised: After all, we have chosen a particular implementation of "Case", "Rule" and "Result". Furthermore, we also design tasks mimic proof steps in formal theorem proving (see the rewrite task in Appendix B.1), which also achieved excellent results. Nevertheless, we believe LIME is a ﬁrst step towards building reasoning inductive biases, and provides many inspirations and directions for future work.  C ONCLUSION

Original: In this work, we encoded inductive biases for mathematical reasoning in the form of datasets.
Revised: In this work, we encoded inductive biases for mathematical reasoning in the form of datasets.

Original: We created three synthetic tasks corresponding to the three reasoning primitives of deduction, induction, and abduction and demonstrated that pretraining on these tasks (LIME) signiﬁcantly improved the performances across three mathematical reasoning benchmarks.
Revised: We created three synthetic tasks inspired by three reasoning primitives of deduction, induction, and abduction. We demonstrated that pretraining on these tasks (LIME) signiﬁcantly improved the performances across three mathematical reasoning benchmarks.

Original: Notably, LIME requires negligible computation compared to the downstream task, unlike being the dominating factor in previous pretraining methods.
Revised: Notably, LIME requires negligible computation compared to the downstream task, unlike being the dominating factor in previous pretraining methods.

Original: Our work naturally poses many future research questions.
Revised: Our work naturally poses many future research questions.

Original: Could the primitive tasks provide similar gains for NLP tasks?
Revised: Could the primitive tasks provide similar gains for NLP tasks?

Original: Are there similar primitive tasks for natural language reasoning?
Revised: Are there similar primitive tasks for natural language reasoning?

Original: We also look forward to disentangling the effects of pretraining between learning content knowledge and inductive bias for all downstream tasks to better understand pre-training.
Revised: We also look forward to disentangling the effects of pretraining between learning content knowledge and inductive bias for all downstream tasks to better understand pre-training.

Original: A PPENDIX A S YNTHETIC T ASK G ENERATION P SEUDOCODE Algorithm
Revised: A PPENDIX A S YNTHETIC T ASK G ENERATION P SEUDOCODE Algorithm

Original: Math set M
Revised: Math set M

Original: Rule set
Revised: Rule set

Original: SAMPLE
Revised: SAMPLE

Original: C
Revised: C

Original: Result s do 12:
Revised: Result s do 12:

Original: SUBSTITUTE
Revised: SUBSTITUTE

Original: 13: end
Revised: 13: end

Original: 15: end
Revised: 15: end

Original: A PPENDIX B O THER SYNTHETIC TASKS
Revised: A PPENDIX B O THER SYNTHETIC TASKS

Original: In this section, we give descriptions of other variants of the synthetic tasks we considered than the ones introduced in the main paper.
Revised: In this section, we give descriptions of other variants of the synthetic tasks we considered than the ones introduced in the main paper.

Original: A PPENDIX B. R EWRITE AND R EWRITE _ MULTISTEP
Revised: A PPENDIX B. R EWRITE AND R EWRITE _ MULTISTEP

Original: We propose a rewrite task, inspired by the rewrite tactic used in interactive theorem provers.
Revised: We propose a rewrite task, inspired by the rewrite tactic used in interactive theorem provers.

Original: The
Revised: The

Original: Rewrite task requires the model to rewrite a string according to a rule transformation.
Revised: Rewrite task requires the model to rewrite a string according to a rule transformation.

Original: One example of the task is:
Revised: One example of the task is:

Original: Source
Revised: Source

Original: [Equation]
Revised: [Equation]

Original: Target:
Revised: Target:

Original: b + a − c
Revised: b + a − c

Original: “ A + B = B + A “ is the rule transformation, which is applied to the LHS string “ a + b − c ”.
Revised: “ A + B = B + A “ is the rule transformation, which is applied to the LHS string “ a + b − c ”.

Original: The model needs to predict the RHS string as the result of the rule application, i.e., b + a − c .
Revised: The model needs to predict the RHS string as the result of the rule application, i.e., b + a − c .

Original: Besides rule symbols and math symbols, we also require the third set of symbols, named as "string symbols".
Revised: Besides rule symbols and math symbols, we also require the third set of symbols, named as "string symbols".

Original: For the ease of our discussion, we we will think of math symbols as the union of those operators used in mathematics (e.g., “ + − ∗ =
Revised: For the ease of our discussion, we we will think of math symbols as the union of those operators used in mathematics (e.g., “ + − ∗ =

Original: ()& ”), rule symbols as upper case letters (e.g., A , B , C ...), and string symbols as lower case letters (e.g., a , b , c ...).
Revised: ()& ”), rule symbols as upper case letters (e.g., A , B , C ...), and string symbols as lower case letters (e.g., a , b , c ...).

Original: We ﬁrst sample a random string as the LHS string, consisting of math symbols and string symbols (e.g., a + b − c ).
Revised: We ﬁrst sample a random string as the LHS string, consisting of math symbols and string symbols (e.g., a + b − c ).

Original: We sample a sub-string of the LHS string, and replace the string symbols in the sub-string with rule symbols.
Revised: We sample a sub-string of the LHS string, and replace the string symbols in the sub-string with rule symbols.

Original: For example, we sample and obtain the substring a + b from a + b − c , and we replace a , b with rule symbols A , B .
Revised: For example, we sample and obtain the substring a + b from a + b − c , and we replace a , b with rule symbols A , B .

Original: This then forms the LHS of the rule transformation, A + B , with the substitution dictionary { A : a, B : b } .
Revised: This then forms the LHS of the rule transformation, A + B , with the substitution dictionary { A : a, B : b } .

Original: We then sample the RHS of the rule transformation from the union of rule symbols A and B , and all math symbols, e.g., B + A .
Revised: We then sample the RHS of the rule transformation from the union of rule symbols A and B , and all math symbols, e.g., B + A .

Original: This gives the rule transformation A + B = B + A .
Revised: This gives the rule transformation A + B = B + A .

Original: We substitute the value of the substitution dictionary for each rule symbol in the RHS rule, and then substitute back to the original LHS string to obtain b + a − c .
Revised: We substitute the value of the substitution dictionary for each rule symbol in the RHS rule, and then substitute back to the original LHS string to obtain b + a − c .

Original: The task example is constructed by using the LHS string and the rule transformation as the source input, and use the result of the rule transformation as the target.
Revised: The task example is constructed by using the LHS string and the rule transformation as the source input, and use the result of the rule transformation as the target.

Original: We further introduce a multi-step version of the rewrite task: Rewrite_multistep .
Revised: We further introduce a multi-step version of the rewrite task: Rewrite_multistep .

Original: In this task, the source may contain more than one rewrite rule, and the target is the result of applying all the rewrite rules in a sequence.
Revised: In this task, the source may contain more than one rewrite rule, and the target is the result of applying all the rewrite rules in a sequence.

Original: This task is motivated from the need to perform multi-step planning in
Revised: This task is motivated from the need to perform multi-step planning in

Original: [Table]
Revised: [Table]

Original: mathematical reasoning tasks.
Revised: mathematical reasoning tasks.

Original: During pre-training, for each training example
Revised: During pre-training, for each training example

Original: [Table]
Revised: [Table]

Original: we uniformly sample the number of rewrite steps from 1 to 5.
Revised: we uniformly sample the number of rewrite steps from 1 to 5.

Original: A PPENDIX B. O THER VARIANTS OF I NDUCT T ASK
Revised: A PPENDIX B. O THER VARIANTS OF I NDUCT T ASK

Original: We introduce three other variants of the Induct task.
Revised: We introduce three other variants of the Induct task.

Original: Induct_v2 : We move the Case dictionary from the source input to the target output.
Revised: Induct_v2 : We move the Case dictionary from the source input to the target output.

Original: This makes the task signiﬁcantly harder, which requires the agent to synthesize a rule and a possible explanation (Case) to explain the Result.
Revised: This makes the task signiﬁcantly harder, which requires the agent to synthesize a rule and a possible explanation (Case) to explain the Result.

Original: Induct_v3 : Instead of providing the Case dictionary, we provide two Result strings, coming from the same Rule.
Revised: Induct_v3 : Instead of providing the Case dictionary, we provide two Result strings, coming from the same Rule.

Original: Namely, we sample two Case dictionaries, and applying each to the Rule string to obtain two Result strings.
Revised: Namely, we sample two Case dictionaries, and applying each to the Rule string to obtain two Result strings.

Original: Both Result strings are used as source, and the target is the Rule string.
Revised: Both Result strings are used as source, and the target is the Rule string.

Original: 3. Induct_rewrite
Revised: 3. Induct_rewrite

Original: : We also create a “induction” version of the Rewrite task.
Revised: : We also create a “induction” version of the Rewrite task.

Original: In this task, the source is the LHS string concatenated with the RHS string, that is the result of the rewrite.
Revised: In this task, the source is the LHS string concatenated with the RHS string, that is the result of the rewrite.

Original: The target is the rewrite rule that is used to do the rewrite.
Revised: The target is the rewrite rule that is used to do the rewrite.

Original: A PPENDIX B. A FULL COMPARISON OF ALL SYNTHETIC TASKS
Revised: A PPENDIX B. A FULL COMPARISON OF ALL SYNTHETIC TASKS

Original: In this section we present a full comparison for all synthetic tasks.
Revised: In this section we present a full comparison for all synthetic tasks.

Original: We followed the training protocol in 4.1 and evaluate the method on IsarStep.
Revised: We followed the training protocol in 4.1 and evaluate the method on IsarStep.

Original: The results are reported in Table 6.
Revised: The results are reported in Table 6.

Original: We can see that the Rewrite_multistep achieved the best performance across all synthetic tasks, surpassing the baseline by 8 . 2% for Top-1 accuracy and 10 . 8% for Top-10 accuracy.
Revised: We can see that the Rewrite_multistep achieved the best performance across all synthetic tasks, surpassing the baseline by 8 . 2% for Top-1 accuracy and 10 . 8% for Top-10 accuracy.

Original: This indicates the inductive bias for long horizon reasoning encoded in Rewrite_multistep is very useful for the reasoning task.
Revised: This indicates the inductive bias for long horizon reasoning encoded in Rewrite_multistep is very useful for the reasoning task.

Original: A PPENDIX C M ORE A BLATION S TUDIES A PPENDIX C. D OES THE VOCABULARY SIZE MATTER ?
Revised: A PPENDIX C M ORE A BLATION S TUDIES A PPENDIX C. D OES THE VOCABULARY SIZE MATTER ?

Original: In this section, we investigate whether the vocabulary size S in the synthetic task generation algorithm has an effect on the performance.
Revised: In this section, we investigate whether the vocabulary size S in the synthetic task generation algorithm has an effect on the performance.

Original: We used the REWRITE task for the experiment in this section.
Revised: We used the REWRITE task for the experiment in this section.

Original: We generated datasets of various vocabulary sizes, 100 , 512 , 1000 , 5000 , 25000 .
Revised: We generated datasets of various vocabulary sizes, 100 , 512 , 1000 , 5000 , 25000 .

Original: We used the same curriculum learning for pre-training as described in 4.1 on larger vocabulary sizes: ﬁrst training on the Rewrite task of vocabulary size 100 for 10 K steps, then training on each individual dataset for another 10 K steps.
Revised: We used the same curriculum learning for pre-training as described in 4.1 on larger vocabulary sizes: ﬁrst training on the Rewrite task of vocabulary size 100 for 10 K steps, then training on each individual dataset for another 10 K steps.

Original: We compare the performance on the downstream task Isarstep.
Revised: We compare the performance on the downstream task Isarstep.

Original: The results are presented in Table 7.
Revised: The results are presented in Table 7.

Original: We see that when the vocabulary size is equal or larger than 512 , the performance were similar.
Revised: We see that when the vocabulary size is equal or larger than 512 , the performance were similar.

Original: The smallest vocabulary size 100 obtained the worst performance among all, and all the other four models achieved similar BLEU scores.
Revised: The smallest vocabulary size 100 obtained the worst performance among all, and all the other four models achieved similar BLEU scores.

Original: The model trained on the largest vocabulary achieved best performance on top-1 accuracy and top-10 accuracy.
Revised: The model trained on the largest vocabulary achieved best performance on top-1 accuracy and top-10 accuracy.

Original: The results show there is a non-trivial effect of the vocabulary size of the synthetic task to the performance of the downstream task.
Revised: The results show there is a non-trivial effect of the vocabulary size of the synthetic task to the performance of the downstream task.

Original: Hence we use vocabulary size of 1000 for all the experiments in the main paper.
Revised: Hence we use vocabulary size of 1000 for all the experiments in the main paper.

Original: We leave investigations of the causes to future work.
Revised: We leave investigations of the causes to future work.

Original: [Table]
Revised: [Table]

Original: A PPENDIX C. P RE - TRAINING ON I SAR S TEP FOR M ETA M ATH S TEP
Revised: A PPENDIX C. P RE - TRAINING ON I SAR S TEP FOR M ETA M ATH S TEP

Original: Following Section 5.1, we performed pre-training on IsarStep for MetaMathStep.
Revised: Following Section 5.1, we performed pre-training on IsarStep for MetaMathStep.

Original: The result is shown in Table 8.
Revised: The result is shown in Table 8.

Original: In contrast to MetaMath helping IsarStep, we see that pretraining on IsarStep task did not help the downstream task MetaMathStep.
Revised: In contrast to MetaMath helping IsarStep, we see that pretraining on IsarStep task did not help the downstream task MetaMathStep.

Original: We hypothesize that this could be due to MetaMathStep task is closer to the LIME tasks than IsarStep, and hence providing more gains than the opposite direction.
Revised: We hypothesize that this could be due to MetaMathStep task is closer to the LIME tasks than IsarStep, and hence providing more gains than the opposite direction.

Original: We leave investigations to the future versions.
Revised: We leave investigations to the future versions.

Original: [Table]
Revised: [Table]

Original: A PPENDIX C. D OES LIME HELP LSTM S ?
Revised: A PPENDIX C. D OES LIME HELP LSTM S ?

Original: In this section, we investigate if LIME also helps other architectures than transformers.
Revised: In this section, we investigate if LIME also helps other architectures than transformers.

Original: In particular, we applied LIME to two LSTM based architectures: 1. vanilla LSTM, 2.
Revised: In particular, we applied LIME to two LSTM based architectures: 1. vanilla LSTM, 2.

Original: LSTM with attention mechanism.
Revised: LSTM with attention mechanism.

Original: The vanilla LSTM is a stacking LSTM with 4 layers, each with 1000 cells, and 1000dimensional embeddings.
Revised: The vanilla LSTM is a stacking LSTM with 4 layers, each with 1000 cells, and 1000dimensional embeddings.

Original: The LSTM with attention architecture is taken from Luong et al.
Revised: The LSTM with attention architecture is taken from Luong et al.

Original: (2015), also with 4 layers, 1000 cells and 1000-dimensional embeddings.
Revised: (2015), also with 4 layers, 1000 cells and 1000-dimensional embeddings.

Original: We evaluate on the IsarStep task, and compared a model trained from scratch and a model pre-trained on LIME abduct task.
Revised: We evaluate on the IsarStep task, and compared a model trained from scratch and a model pre-trained on LIME abduct task.

Original: We used the same training protocol as described in 4.1.
Revised: We used the same training protocol as described in 4.1.

Original: The results are shown in Table 9, along with the results on transformer.
Revised: The results are shown in Table 9, along with the results on transformer.

Original: We observe that LIME improved LSTM as well as LSTM with attention, but the improvements were small compared to transformer.
Revised: We observe that LIME improved LSTM as well as LSTM with attention, but the improvements were small compared to transformer.

Original: Speciﬁcally, if we compare Top-1 accuracy, we can see that LIME improved LSTM from 5 .
Revised: Speciﬁcally, if we compare Top-1 accuracy, we can see that LIME improved LSTM from 5 .

Original: 5% to 6 .
Revised: 5% to 6 .

Original: 9% , LSTM with attention from 12 .
Revised: 9% , LSTM with attention from 12 .

Original: 3% to 13 .
Revised: 3% to 13 .

Original: 4% , and transformer from 20 .
Revised: 4% , and transformer from 20 .

Original: 4% to 26 .
Revised: 4% to 26 .

Original: This observation is aligned with our hypothesis that the transformer is a malleable architecture and hence it is capable of learning architectural inductive biases from datasets.
Revised: This observation is aligned with our hypothesis that the transformer is a malleable architecture and hence it is capable of learning architectural inductive biases from datasets.

Original: This is mainly attributed to the potential of learning dynamic attention graphs in self-attention layers.
Revised: This is mainly attributed to the potential of learning dynamic attention graphs in self-attention layers.

Original: We note that this still warrants further investigation as the performance of these architectures are not at the same level, and that may also lead to different improvements.
Revised: We note that this still warrants further investigation as the performance of these architectures are not at the same level, and that may also lead to different improvements.

Original: [Table]
Revised: [Table]