Injecting Numerical Reasoning Skills into Language Models (2020)

Abstract: Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 –> 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.

## introduction recently, models trained on large amounts of data with a language modeling (lm) objective, have shown great promise in natural language processing, exhibiting surprising amounts of knowledge and information @xcite @xcite @xcite . however, high-level skills, such as the ability to perform numerical reasoning over text, can be challenging to capture with a lm objective only. consider the example in to make the task more manageable, state-of-theart models have employed specialized architectures, restricting the space of possible numerical computations to a limited set. modules were designed for counting (but only until '9') and for addition and subtraction (but of 2-3 numbers only). such models perform well on existing datasets, such as drop @xcite , but do not generalize to unsupported computations, which will require modifying the model architecture. moreover, current models marginalize at training time over all numerical expressions that evaluate to the correct answer. since the number of such expressions grows exponentially, scaling these approaches to arbitrary computations entails using non-differentiable operations (sampling or computing top-k numerical expressions), which can lead to training difficulties. passage: taunton has four art galleries... hughes/ donahue gallery founded in 2007, a local community gallery serving local taunton artists... art euphoric founded in 2008 has both visual and craft exhibits... q1: how many years after founding of hughes/ donahue was art euphoric founded? a1: 1 (number) q2: which gallery was founded later, hughes/ donahue or art euphoric? a2: art euphoric (span) in this work, we propose that reasoning skills, such as numerical reasoning, are amenable to automatic data generation. hence, one can inject that skill directly into the model by adding additional pre-training steps, allowing the model to learn the skill in an end-to-end fashion. this results in a fully-differentiable training procedure over a standard and general-purpose architecture, where the output space can be easily controlled through the data generation procedure. specifically (figure 1 ), we add to a large pre-trained lm two pre-training steps over automatically-generated synthetic data. first, we generate numerical data of the form 3 + 4 + 11 = 18. training on this data teaches the model to compute the value of numbers from their tokens and to perform numerical operations. second, we automatically generate question-passage pairs that require numerical reasoning using a compact grammar (textual data). training on this data endows the model with the ability to understand computations expressed in pseudo-natural language. in both pre-training steps, the model, gen-bert, generates output numbers token-by-token. thus, the model has a standard architecture, where an answer can either be extracted from the input question and passage or generated from a decoder. pre-training is done in a multi-task setup with a standard lm objective, in order to avoid "catastrophic forgetting" @xcite of the linguistic information in the original lm. after pre-training, the model has sufficient language and numerical skills to be directly fine-tuned on a target numerical reasoning dataset, without resorting to specialized architectures. augmenting more numerical skills does not require changing the model, only generating additional data. we demonstrate the validity of our approach by a series of experiments showing that: (a) genbert is able to solve pre-training tasks for numerical reasoning. our code and data can be downloaded from https://github.com/ag1988/ injecting_numeracy . ## numerical reasoning over text numerical reasoning over text (nrot) is commonly set up as a reading comprehension (rc) task. given a training set of question-context-answer triples {(q i , c i , a i )} n @xmath0, the goal is to learn a function that returns the answer a to a question q given a context c. however, in nrot the answer generally requires to internally perform some numerical computation using the entities and numbers in the context. specifically, the answer is either: (a) a span (or list of spans) from the context c or question q, or (b) a number that is the result of some computation (see examples in two natural, yet opposing, approaches lend themselves to tackling nrot: (a) a symbolic approach: a model can read the question and context, output a numerical expression and evaluate the answer with an external symbolic calculator. this approach is a particular case of semantic parsing @xcite , and was common in early nrot datasets @xcite @xcite . how-ever, it suffers from several drawbacks. first, because numerical expressions are discrete and their space grows combinatorially, the model must learn to search in this space using non-differentiable operations, which are usually difficult to optimize. second, numerical expressions are limited to numerical answers, while in drop often a numerical computation is required but the final answer is a text span. (b) a distributed approach: have a model directly generate the answer given (q, c). when the answer is a text span, the model can extract it from the input, and when the answer is a number that is not in q or c, the model must generate it. while this makes training straightforward, the model must learn to perform numerical computations from the relatively small target dataset. we empirically show in §3 that this leads to low performance in general. as a compromise, most nrot models @xcite @xcite have taken a hybrid approach: they augment standard extractive qa models with specialized modules for handling a limited set of numerical computations. we briefly describe this architecture, as it is the basis for our model in §3. given a question with n 1 tokens @xmath1, . . . , q n 1 ) and a context with n 2 tokens @xmath2, . . . , c n 2 ), the hybrid model first computes contextualized representations for the n 1 + n 2 + 3 tokens [cls] q [sep] c[sep] using a pretrained lm, such as bert @xcite : the representations l are then passed to multiple heads, which are small neural networks that estimate p(a | q, c, h), that is, the probability of the answer given the input and conditioned on a head h, corresponding to a particular answer type: • context span head: computes a distribution over all spans in the context using a feed-forward network (ffn) ff c (l). finally, for deciding which answer head to use for a given input, a type head ff typ (l) outputs a probability distribution p head (h | q, c) (using a ffn). thus the model probability for an answer is training is done by enumerating all of the ways in which the answer can be obtained using all of the heads, and maximizing this marginal probability. while existing models perform well on drop, the aforementioned architecture is not flexible. first, the output space is severely constrainedthe model can only count up to '9', and numerical computations are restricted to signed combinations of a few numbers. second, expanding the space of supported numerical computations is non-trivial, because training involves marginalizing over all expressions that lead to the correct answer. since the space of numerical expressions grows exponentially, expanding this space quickly leads to a difficult search problem. third, delegating numerical computations to an external symbolic calculator leads to modeling challenges, since there could be interactions between text and numerical computation: consider the drop question "how many total yards did phil dawson throw for touchdowns?". current models handle such questions by computing a sum from numbers in the text and returning the result. however, if the question was "who threw 45 total yards for touchdowns?", the model would have to compute the sum internally, and then find the relevant span in the text. this is impossible when the computation itself is delegated to an external calculator. thus, training models to handle such numerical questions is desirable. motivated by the above arguments, we wish to push the frontier of end-to-end differentiable models for numerical reasoning. thus, we will automatically generate large amounts of data that endow a pre-trained lm with numerical skills. ## genbert: a bert-based model for generating arbitrary outputs we now describe a simple bert-based generative model that performs numerical computations internally, termed genbert. the model combines the transformer encoder-decoder architecture @xcite with a pre-trained lm, specifically, bert. our architecture is illustrated in figure 2 . our encoder is a standard transformer, initialized with bert weights. to also enjoy bert's representations at decoding time, we tie the weights of the decoder and the encoder. because the transformer decoder has source attention weights (weights for attending to the encoder representations at decoding time) that are not present in bert, we tie these source-attention weights to the self-attention weights of the encoder (which are tied to the selfattention weights of the decoder). this fully initializes the transformer model with bert weights. since the encoder and decoder weights are tied, we make them learn distinct representations by adding a ffn ff enc that transforms the encoder contextualized representations l enc as finally, the probability of an answer is defined in the usual manner: let @xmath3. the decoder outputs the probability p dec (a i+1 | a 0 , ..a i , c, q), and the probability of an answer is: as we have a generative model, we can remove the specialized count and arithmetic heads from §2. thus, the type head ff typ (h enc ) outputs a distribution (p q , p c , p dec ) over the context span, question span, and decoder heads. to improve pre-training on the numeric data ( §4), we make two additional modifications. digit tokenization (dt) conventional wordpiece tokenization treats numbers no differently than any other token. however, computing the value of numbers should be simpler when using digits directly @xcite . hence, we tokenize numbers digit-by-digit. for example, a wordpiece ##d 1 • • • d k where d @xmath4,...,9} is further split into ##d 1 , ..., ##d k . we show in §5.1 that this substantially improves sample complexity when training to perform numerical operations. random shift (rs) the original transformer uses absolute positional embeddings for each token. however, in §4, we train on short inputs such as "1086.1 -2.54 + 343.8". thus, the model can potentially over-fit and learn to perform numerical reasoning only when numbers are at the beginning of an input. to prevent this, when the input length n 1 + n 2 + 3 < 512, we shift all position ids by a random integer in (0, 1, . . . , 512 -(n 1 + n 2 + 3)). training for each span (i, j), a span extraction head h outputs its probability p h ((i, j) | c, q, h) of being the answer. let s be the set of spans in the input corresponding to the gold answer. the model loss l model marginalizes over all ways in which the answer can be predicted: to evaluate the ability of genbert to perform numerical reasoning, we initialize it with bert and fine-tune it on drop. genbert obtains 46.1 em and 49.3 f 1 , roughly 20 points lower than prior models. thus, we conclude that acquiring numerical reasoning skills from drop data only is difficult. to remedy this, we will automatically generate training data that will endow genbert with numerical skills before training it on drop. bridget adopted 4 brown dogs cont verb-pos num attr ent the king recruited 1337 irish soldiers jackson scored 3 running touchdowns . abstraction instantiation with vocab. figure 3 : template extraction and instantiation. a template (in red) is extracted from a mwp sentence, using categories for containers, entities, verbs, attributes and numbers, according to @xcite . for generation, the categories are instantiated with a domain-specific vocabulary. we now describe two automatically-generated datasets and the multi-task training procedure. ## experimental evaluation we now evaluate our two pre-training steps and their applicability for numerical reasoning tasks. we consider the following variants, aiming to investigate the contributions of nd and td, the importance of mlm loss, and techniques like dt and rs. in all cases, we initialize genbert with bert-base-uncased, use dt and rs, and include the mlm loss, except where noted: ## related work most nrot models designed for drop are extractive qa models augmented with specialized modules ( §2). two recent work @xcite ) take a more symbolic approach and output a symbolic program augmented with operations over text. in our work, numerical computations are latent and performed internally by the model. a related line of work has been analyzing the mathematical reasoning abilities of neural models over text @xcite @xcite , and on arithmetic problems @xcite @xcite . designing pre-training tasks to teach lms additional skills has been applied by @xcite , who designed cross-lingual pre-training tasks to teach better mappings between languages, and @xcite , who introduced the inverse cloze task to pre-train an information retriever. ## conclusions large pre-trained lms lack high-level skills such as numerical reasoning. consequently, current models that perform numerical reasoning over a pretrained lm resorted to customized modules with limited flexibility. in this work, we propose a general method for injecting additional skills into lms, assuming automatic data generation is possible. we apply our approach to the task of numerical reasoning over text, using a general-purpose model called genbert, and a simple framework for generating large amounts of synthetic examples. our experiments demonstrate the effectiveness of our method, showing that genbert successfully learns the numerical skills, and performs on par with state-ofthe-art nrot models of the same size.