Mathematical Reasoning via Self-supervised Skip-tree Training
Authors: Markus Norman Rabe, Dennis Lee, Kshitij Bansal, Christian Szegedy
Venue: ICLR 2021 Spotlight
Keywords: self-supervised learning, mathematics, reasoning, theorem proving, language modeling
Abstract: We demonstrate that self-supervised language modeling applied to mathematical formulas enables logical reasoning. To measure the logical reasoning abilities of language models, we formulate several evaluation (downstream) tasks, such as inferring types, suggesting missing assumptions and completing equalities. For training language models for formal mathematics, we propose a novel skip-tree task. We find that models trained on the skip-tree task show surprisingly strong mathematical reasoning abilities, and outperform models trained on standard skip-sequence tasks. We also analyze the models' ability to formulate new conjectures by measuring how often the predictions are provable and useful in other proofs.