Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Models (2022)

Abstract: The ability to extrapolate, i.e., to make predictions on sequences that are longer than those presented as training examples, is a challenging problem for current deep learning models. Recent work shows that this limitation persists in state-of-the-art Transformer-based models. Most solutions to this problem use specific architectures or training methods that do not generalize to other tasks. We demonstrate that large language models can succeed in extrapolation without modifying their architecture or training procedure. Our experimental results show that generating step-by-step rationales and introducing marker tokens are both required for effective extrapolation. First, we induce a language model to produce step-by-step rationales before outputting the answer to effectively communicate the task to the model. However, as sequences become longer, we find that current models struggle to keep track of token positions. To address this issue, we interleave output tokens with markup tokens that act as explicit positional and counting symbols. Our findings show how these two complementary approaches enable remarkable sequence extrapolation and highlight a limitation of current architectures to effectively generalize without explicit surface form guidance. Code available at https://anonymous.4open.science/r/induced-rationales-markup-tokens-0650/README.md

## introduction the lack of compositional generalization of neural networks has been a long-standing limitation known for decades @xcite @xcite @xcite @xcite . this is often associated with their failure to extrapolate, i.e., the ability to work on sequences that are longer than those presented as training examples. modern architectures such as the transformer @xcite , which is the core component of state-of-the-art nlp models, perform poorly on this class of problems @xcite @xcite @xcite @xcite . in figure 1-(a) , we illustrate how recent large language models such as gpt-3 fail at this task, even when fine-tuned on thousands of examples. architectures and training methods that target this specific problem are often developed based on synthetic tasks whose creation rules are known @xcite @xcite @xcite @xcite @xcite . thus, they resort to techniques such as augmenting the training data or biasing the model's architecture to internally represent these rules. however, improvements obtained on one compositional generalization benchmark do not transfer to others @xcite , i.e., they lose their ability to be used as competitive generalpurpose models in real tasks, as these can seldom be solved with a small set of rules. we study the behavior of transformer models and demonstrate that this problem is not due to an intrinsic limitation of their training algorithm. we show that inducing autoregressive models to rationalize before making a prediction @xcite is not enough to extrapolate on long sequences: to solve it, we introduce markup tokens @xcite . the two general approaches together allow the models to achieve remarkable extrapolation generalization without requiring changes to the model or architecture. these findings provide evidence that general-purpose models have the ability to both improve their effectiveness and interpretability at the same time. the need to markup tokens also suggests there are fundamental issues that need to be addressed in the transformer architecture, particularly the need for better positional representations. thus, our study confirms and supports recent results from previous work that positional embeddings used in current state-of-the-art transformer models cannot precisely track of token positions or perform precise counting @xcite @xcite . ## related work a long list of architectures and training methods attempt to improve the extrapolation capabilities of deep learning models. for instance, some are specifically designed to solve only a handful of tasks @xcite @xcite @xcite @xcite . pretrained word embeddings find it difficult to extrapolate to unseen numbers in training @xcite . alternatives to improving the extrapolation ability of neural models include building neural models with a pre-training corpus of numerical text @xcite or using scientific notation to represent numbers @xcite ). likewise, better numerical and compositional skills can be achieved by supplementing input texts with pre-computed numerical calculations @xcite or explicitly assuming rules or mathematical equations from natural language texts @xcite @xcite @xcite . many of these models are capable of adding numbers larger than those seen during training. in contrast, more general-purpose architectures fail to extrapolate on numerical tasks @xcite @xcite . our work derives from recent findings that show that inducing the model to generate explanations in natural language leads to better performance in a wide variety of tasks @xcite @xcite @xcite @xcite . in particular, the work proposed by @xcite achieves state-of-the-art results in the extrapolation of tasks involving symbolic manipulation, compositional generalization and numerical reasoning. tasks are solved via few-shot learning applied to a large language model (e.g. text-davinci-002) in two main steps. the first step consists of reducing the question into sub-questions, then, in the second phase, a new interaction is made with the model, now solving sequentially the sub-questions generated in the previous step. the results shown in zhou et al. @xcite corroborate our intuition that explanations alone are not enough to achieve extrapolations. by inducing the model to generate explanations and markup tokens, we provide evidence that compositional generalization can be achieved without sacrificing the general applicability on other tasks, which is often a feature that is lost with architectural modifications. however, a limitation of zhou et al.'s and our method is that both require a programmatic postprocessing step: zhou et al. use a python script to convert the model output (e.g., 3*["left"]), which is in python notation, into the expected format of the final answer (e.g., left left left); in our method, we programmatically remove the markup tokens from the final answer. we argue that the need to call an external script exposes a limitation in the current transformer architecture, namely, that it cannot handle long sequences of repeated tokens. ## methodology in this section, we describe our proposed method for inducing explanations and markup tokens using in-context learning with a few examples. we first create a prompt ic||oc that concatenates incontext training examples ic with a test example oc. the ic examples consist of n triples of "instruction", "explanation" and "output", i.e., @xmath0 this carry 0 is summed with digits in position e, which are e 4 and e 0. that is 0 + 4 + 0, which equals to 4. because 4 is less than or equal to 9, this results in carry 0 and e 4. output: e 4 d 3 c 5 b 0 a 7 (...) example 6: instruction: add the numbers d7 c7 b8 a5 and d1 c7 b1 a7 input explanation: this first number goes until position d, whereas the second goes until position d. both numbers have equal length, so we sum digits in positions a, b, c, d. ## experimental setup we evaluate our method in two tasks that require extrapolation: 1) the length split of the scan data @xcite and 2) the addition of two numbers. in all experiments, we used the text-davinci-002 model, available via a paid api provided by openai. we report the accuracy of the test set. ## results in we also show results for gpt-3's ada (300m parameters), curie (6b parameters) and davinci (175b parameters) models fine-tuned on all 16,990 training examples of the scan dataset for 3 epochs. in these cases, we do not use in-context examples, explanations, or markup tokens. our methodology of providing prompts with detailed explanations was shown to be more effective than finetuning on thousands of examples. the same behavior is also observed in the addition task, as seen in figure 3 . our approach with explanations and markup tokens (rationale + markup) shows that even with as few as 5 examples, the model can perform the task of adding numbers with more than 5 digits, reaching a performance of around 60% in numbers with up to 14 digits and an average accuracy of 73% considering all 400 examples in the test set. we also investigated the performance of finetuning a general-purpose model on this task. we trained a t5-base with 100k samples on numbers with 2 to 5 digits per 10 epochs without adding explanations. we observe that the model reaches 100% accuracy with numbers of up to 5 digits, but fails to add numbers with more than 6 digits. instruction: look left thrice and run around left thrice markup-only output: left look 1 left look 2 left look 3 || left run 1 left run 2 left run 3 left run 4 left run 5 left run 6 left run 7 left run 8 left run 9 left run 10 left run 11 left run 12 rationale-only explanation: because of the word "and", the instruction can be divided in two subinstructions: 1) "look left thrice" and 2) "run around left thrice" 1) "look left thrice" results in 3 left look commands. 2) "run around left thrice" results in 3 x 4 = 12 left run commands. output: left look left look left look left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left run left ## conclusion in this work, we show how step-by-step rationales and positional markup tokens enable generalpurpose architectures to extrapolate to sequences that are significantly longer than those provided as training examples. rationales before the answer break down the problem into small executable chunks and markup tokens track the working progress as the output is generated. importantly, we show how these methods are complementary and, when used together, enable remarkable extrapolation results on two synthetic tasks. however, we note the use of markup tokens as a limitation of current models and subword tokenizers. future models should be able to count tokens and keep track of individual tokens in long sequences without resorting to additional supporting tokens. as our qualitative analysis shows, most failure cases are due to one or two tokens generated incorrectly. we see the ability to automatically verify these errors, as proposed by @xcite , as a promising direction to improve the extrapolation capabilities of current models.