  While designing inductive bias in neural architectures has been widely
studied, we hypothesize that transformer networks are flexible enough to learn
inductive bias from suitable generic tasks. Here, we replace architecture
engineering by encoding inductive bias in the form of datasets. Inspired by
Peirce's view that deduction, induction, and abduction form an irreducible set
of reasoning primitives, we design three synthetic tasks that are intended to
require the model to have these three abilities. We specifically design these
synthetic tasks in a way that they are devoid of mathematical knowledge to
ensure that only the fundamental reasoning biases can be learned from these
tasks. This defines a new pre-training methodology called "LIME" (Learning
Inductive bias for Mathematical rEasoning). Models trained with LIME
significantly outperform vanilla transformers on three very different large
mathematical reasoning benchmarks. Unlike dominating the computation cost as
traditional pre-training approaches, LIME requires only a small fraction of the
computation cost of the typical downstream task.
