LILA : A Unified Benchmark for Mathematical Reasoning (2022)

Abstract: Mathematical reasoning skills are essential for general-purpose intelligentsystems to perform tasks from grocery shopping to climate modeling.Towards evaluating and improving AI systems in this domain, we proposeLILA, a unified mathematical reasoning benchmark consisting of 23 diversetasks along four dimensions:(i) mathematical abilities e.g., arithmetic, calculus (ii) language format e.g., question-answering, fill-in-the-blanks (iii) language diversity e.g., no language, simple language (iv) external knowledge e.g., commonsense, physics. We construct our benchmark by extending 20 datasets benchmark by collecting task instructions and solutions in the form of Python programs,thereby obtaining explainable solutions in addition to the correct answer.We additionally introduce two evaluation datasets to measure out-of-distribution performance and robustness to language perturbation.Finally, we introduce BHASKARA,a general-purpose mathematical reasoning model trained on LILA. Importantly, we find that multi-tasking leads to significant improvements (average relative improvement of 21.83% F1 score vs. single-task models),while the best performing model only obtains 60.40%,indicating the room for improvement in general mathematical reasoning and understanding.

## introduction mathematical reasoning is required in all aspects of life, from buying ingredients for a recipe to controlling the world economy. given the fundamental nature of mathematical reasoning, a number of works propose datasets to evaluate specific mathematical reasoning abilities of ai agents, e.g., @xcite (algebra word problems), @xcite (arithmetic reasoning), @xcite (templated math reasoning spanning algebra, calculus, probability, etc.) since evaluating high-capacity models on narrowly scoped mathematical reasoning datasets risks overestimating the reasoning abilities of these ai systems, creating the need for a unified benchmark for systematic evaluation over diverse topics and problem styles. to this end, we introduce l īla foot_0 , a unified mathematical reasoning benchmark that consists of 23 mathematical reasoning tasks. l īla is constructed by extending 20 existing datasets spanning a wide range of topics in mathematics, varying degrees of linguistic complexity, and diverse question formats and background knowledge requirements. importantly, l īla extends all of these datasets to include a solution program as opposed to only an answer, and instruction annotations to enable instructionbased learning @xcite @xcite . in order to accurately assess the mathematical reasoning ability of models, evaluating the chain of reasoning that leads to the correct solution is equally important (if not more important) to evaluating the final answer or expression. we therefore collect python programs that serve as reasoning chains for each question in the benchmark. we achieve this by automatically converting domainspecific language (dsl) annotations into python programs and by manually collecting expert annotations when no dsl annotations are available. by incorporating program annotations, l īla unifies various mathematical reasoning datasets under a single problem formulation i.e., given an input problem in natural language, generate a python program that upon execution returns the desired answer. this formulation allows neural approaches to focus on the high-level aspects of mathematical problem solving (e.g., identifying potential solution strategies, decomposing the problem into simpler sub-problems), while leveraging external solvers (e.g., python builtins, sympy) to perform precise operations like adding huge numbers or simplifying expressions. figure 1 illustrates a sample from our l īla benchmark that illustrates the question, answer, program, instructions, and category tags. in addition to evaluating high-level problem solving, we also facilitate two other key ways to make a fair assessment of models on mathematical reasoning tasks. in line with @xcite , @xcite and @xcite , we evaluate generalization e.g., alternate formulations of a problem ("2+2=?" vs. "what is two plus two?") using an out-of-distribution evaluation set (l īla-ood) containing datasets requiring the same underlying mathematical reasoning skills, but were collected independently of the training datasets. further, we collect a robustness split l īla-robust, that introduces linguistic perturbations (e.g., active vs. passive voice) via crowd-sourcing. the evaluation scheme is a combination of the performance on all three sets: l īla-test, l īla-ood and l īla-robust. contributions 1. we present l īla, a holistic benchmark for mathematical reasoning. l īla extends 20 existing datasets with solutions in the form of python programs and instruction annotations, and categorizes questions into 23 tasks based on their language complexity, question format and need for external knowledge. our benchmark measures performance on out-of-distribution examples and robustness to language perturbations in addition to standard test-set. 2. we introduce bh āskara, a multi-task model fine-tuned on our dataset. our best-performing model achieves comparable performance to a 66× larger model pre-trained on both code and language. 3. we provide an analysis of our models' performance and find that (1) multitasking improves considerably over task-specific learning both in in-distribution and out-of-distribution evaluation (2) program synthesis substantially outperforms answer prediction, (3) few-shot prompting with codex has the strongest performance. we also identify areas for improvement for future work, e.g., data gaps in l īla categories. a separate line of work in mathematical reasoning includes datasets testing mathematical theorem proving (e.g., @xcite @xcite @xcite . we do not, however, consider theorem proving in our work, choosing instead to focus on numerical reasoning. task hierarchy and multi-tasking in numerical reasoning. we take inspiration from the success of multi-task learning in nlp @xcite , including benchmarks (e.g., @xcite @xcite and multitasking models (e.g., @xcite @xcite . numglue @xcite has been proposed as a multi-tasking numerical reasoning benchmark that contains 8 different tasks. l īla expands numglue to provide wider coverage of mathematical abilities, along with evaluation that captures out-of-domain, robustness, and instruction-following performance. our introduction of mathematical reasoning categories and the evaluation setup is inspired by task hierarchies in other domains such as vision @xcite and nlp @xcite which appear in large scale benchmarks (e.g., @xcite . ## l īla l īla is composed of 23 tasks across 4 dimensions, curated from 44 sub-datasets across 20 dataset sources. here we discuss the construction and composition of the benchmark and provide descriptive statistics of the datasets. ## experiments in this section, we introduce our modeling contributions for the l īla benchmark and discuss the overall experimental setup. data partition and evaluation. for the iid setup, we randomly partition the data in each task into training (70%), development (10%) and test (20%) sets. additionally, we also evaluate on l īla-ood and l īla-robust settings; thus, the final evaluation scheme is a combination of the performance on all three evaluation setups fine-tuning we fine-tune a series of gpt-neo-2.7b causal language models @xcite ) on l īla. we choose gpt-neo because it was pretrained on both natural language and code @xcite , as opposed to solely on natural language. to assess the capabilities of gpt-neo on various aspects of the dataset, we fine-tune singletask models on each of the 23 tasks in l īla. we also evaluate the benefit of transfer learning by finetuning a single multi-task gpt-neo baseline on all the tasks simultaneously. prompting. we also use few-shot prompting to evaluate gpt-3 and codex foot_2 @xcite . for the iid setting, we prompt the model with a random input-output examples from the same dataset as the input. in the ood setting, we take examples from other datasets ( the bold score refers to the best score among models with the same supervision method; the underlined score refers to the best score among all models. gpt-3 and codex performance is computed on 100 uniformly distributed examples owing to their cost and usage limit. fine-tuned model performance is calculated on the full test set. ## results and analysis a summary of all key results on our l īla benchmark are shown in ## conclusion in this work, we introduce l īla, a unified mathematical reasoning benchmark for a holistic evaluation of ai agents. l īla consists of 23 tasks across 4 dimensions (i) mathematical abilities, (ii) language format, (iii) language complexity, (iv) external knowledge. it builds on 20 existing mathematical reasoning datasets to collect instructions and python programs. further, it also supports measuring out-of-distribution performance and robustness to language perturbations via l īla-ood and l īla-robust respectively. we also introduce bh āskara, a 2.7b-parameter fine-tuned multitask model. we find that multi-tasking improves over single-task performance by 21.83% f1 score on average, and that our model is a strong starting point for further fine-tuning on new math reasoning tasks. the best performing model we evaluate achieves only 60.40% f1 indicating the potential for improvement on the proposed benchmark.