D yna-b A b I : unlocking b A b I ’s potential with dynamic synthetic benchmarking (2022)

Abstract: While neural language models often perform surprisingly well on natural language understanding (NLU) tasks, their strengths and limitations remain poorly understood. Controlled synthetic tasks are thus an increasingly important resource for diagnosing model behavior. In this work we focus on story understanding, a core competency for NLU systems. However, the main synthetic resource for story understanding, the bAbI benchmark, lacks such a systematic mechanism for controllable task generation. We develop Dyna-bAbI, a dynamic framework providing fine-grained control over task generation in bAbI. We demonstrate our ideas by constructing three new tasks requiring compositional generalization, an important evaluation setting absent from the original benchmark. We tested both special-purpose models developed for bAbI as well as state-of-the-art pre-trained methods, and found that while both approaches solve the original tasks (99% accuracy), neither approach succeeded in the compositional generalization setting, indicating the limitations of the original training data. We explored ways to augment the original data, and found that though diversifying training data was far more useful than simply increasing dataset size, it was still insufficient for driving robust compositional generalization (with 70% accuracy for complex compositions). Our results underscore the importance of highly controllable task generators for creating robust NLU systems through a virtuous cycle of model and data development.

## introduction considerable progress has been made recently in natural language understanding (nlu), driven largely by advances in model pre-training @xcite low task configurability leads to static datasets, benchmark saturation & unreliable model development. (b) we propose a dynamic benchmarking approach; developing models and tasks in a tight feedback loop using (c) dyna-babi task generator. dyna-babi provides fine-grained control over task structure, composition and difficulty, yielding challenging new test sets exposing limitations of stateof-the-art models. @xcite and the development of large-scale nlu benchmarks across a wide range of tasks @xcite @xcite . such successes, however, have coincided with the discovery of various shortcomings in existing human curated datasets, largely related to annotation artifacts @xcite , or systematic biases that create shortcuts that can inflate model performance and harm generalization. in order to overcome these issues, two avenues of research have recently gained traction: 1) development of dynamic benchmarks @xcite where, in contrast to conventional static benchmarks, evaluation and data collection are conducted interactively with humans and models in a rapidly evolving feedback loop and; 2) renewed interest in synthetic benchmarks @xcite @xcite ) that allow for absolute control over the data creation process in order to help understand the strengths and weaknesses of existing models on targeted tasks and language phenomena. story understanding is a particularly important domain for research on dynamic and synthetic benchmarks; it is a core competency for nlu systems @xcite , but the scale and annotation detail required make human data collection prohibitively costly. however, the main synthetic resource for story understanding remains the babi task suite @xcite , which is saturated by models reaching near-perfect performance @xcite , and further limited by exploitable biases in the data @xcite . despite its creators' initial intentions, babi has largely remained a static benchmark limited to a small subset of the tasks potentially possible to generate within the babi "micro-world". accordingly, two natural questions arise: (q1) is near-perfect model performance on the original babi tasks a reliable indicator of story understanding competence?; (q2) are there still interesting challenges to discover inside the broader babi task space that help identify weaknesses in current models and drive modelling innovation? to answer these questions, we employ a dynamic synthetic benchmarking approach on babi, combining the benefits of the agile approach of recent dynamic benchmarks with the scale and control provided by synthetic datasets. as illustrated in figure 1 , in dynamic synthetic benchmarks the data generator itself is designed for agile development, enabling experimentation with increasingly complex tasks and a wider range of linguistic phenomena. 2 constructing 2 while our framework does not enable automatic collection challenging tasks is a challenge in and of itself, requiring precise control over the reasoning patterns underlying each question. to meet these requirements, we developed a new task generator for babi called dyna-babi 3 . using dyna-babi, we first devise new splits that systematically test compositional generalization across tasks; as shown in fig. 1c , we test models on novel combinations (right side, line 10) of concepts seen at training, like co-reference and object tracking @xcite . we find that training on the original babi tasks (hereafter: babi 1.0) is not sufficient for models to attain good compositional generalization. though general purpose pre-trained models far outperform special-purpose (non-pretrained) architectures developed for babi, they still suffer a 20-50% drop in accuracy compared to the non-pre-trained models which suffer a 50-80% drop. both types attain near perfect performance on the original tasks, suggesting that babi 1.0 is not challenging enough to differentiate between the two classes of models (q1). we next investigate how different enhancements of training data affect compositional generalization: (a) injecting more questions into babi 1.0, and (b) generating new, more diverse training samples. compared to question injection, we find that diverse training data better facilitates compositional generalization, as well as being more data efficient. however, neither approach drives reliable compositional generalization; a representative state-of-the-art (sota) model, t5 @xcite , demonstrates a lack of robustness to novel combinations and also exhibits knowledge inconsistency, for example, by correctly answering certain types of questions but systematically failing to answer equivalent paraphrases. these results suggest that there remain many important challenges within the broader babi task space (q2) which can be discovered through more careful control of task generation. to sanity-check the quality of our new tests compared with babi 1.0, we employ the notion of concurrence proposed by @xcite ; of new data based on model errors as in other dynamic benchmarks, we still chose the term "dynamic" to highlight their important common function: data generation frameworks that enable easily "moving the goalposts" in meaningful directions (in our case, for probing models' systematic generalization capacities). 3 implemented in python for improved accessibility compared with the original lua implementation ( https://github. com/facebookarchive/babi-tasks ). giving the continued interest in using babi 1.0 to evaluate new modelling approaches @xcite @xcite , our new challenge splits and the dyna-babi task generator contribute to more reliably guiding future efforts. while we focused on babi, our results apply more generally, telling a cautionary tale about the limits of static synthetic datasets, and motivating the development of controllable task generators for dynamic synthetic benchmarking. ## related work our work brings together two promising areas of current research: dynamic benchmarking such as dynabench @xcite that address many existing issues with static benchmarks (bowman and dahl, 2021), and synthetic benchmarking, which is widely used for high-precision and dataintensive problems such as relational and logical reasoning @xcite @xcite , robot planning @xcite , instruction following and language grounding @xcite among many others @xcite . most approaches to synthetic benchmarking focus on model development on a static benchmark, and are not designed to facilitate agile and highly controlled task space exploration, which is our focus here. the recent gscan dataset @xcite ) and later extensions @xcite can be seen as an example of a synthetic benchmark "going dynamic". our work differs in terms of target domain (story understanding as opposed to multi-modal language grounding), and we further focus attention on a more general research direction of intentional, a-priori design of nlu benchmarks for agile development. in this regard, our work can be seen as part of a trend towards data-centric research efforts in response to prevailing modelcentric research, which generally focuses heavily on architectural design and novelty @xcite , at the expense of work on the data side @xcite . we address the domain of story understanding as a particularly core (and data-intensive) capacity underlying language use @xcite , thought to require constructing and manipulating situation models of entities and their relations as they unfold throughout discourse @xcite . procedural text datasets @xcite are closely related in that they provide detailed annotation of entities and state changes, and have mostly focused on relatively small and static benchmarks using human collected data. overall, recent works identify a lack of benchmarks which systematically probe the situation models constructed by systems processing discourse-level texts @xcite . the babi benchmark @xcite ) is seen as highly relevant in terms of objective (targeting situation modelling) @xcite , but has been viewed critically due to its constrained nature and exploitable artifacts @xcite . our work focuses on improving the evaluation in babi through compositional generalization, widely used across nlp to more rigorously probe model robustness @xcite @xcite , but to our knowledge still not applied to story understanding or babi. ## experiments with the controllable task generation afforded by dyna-babi, we can now create datasets probing deeper story understanding capabilities of models. we present two main experiments targeting the following questions: ## future work & conclusions our work opens up multiple new directions for future research. our new tool, dyna-babi is readily extendable for systematic probing of more diverse linguistic phenomena. a beneficial first step could include integration of additional babi tasks. that said, our experience suggests that the design of truly scalable synthetic and dynamic benchmarks poses significant theoretical and engineering challenges, warranting deeper research on their own right. our results raise new questions about the viability of learning robust situation models using standard question-answering training, and our datasets present new challenges for future efforts. additionally, dyna-babi can naturally complement parallel work probing the the situation representations constructed by neural language models @xcite by facilitating tailored data generation for specific questions, thus broadening and deepening the scope of possible research. in conclusion, we introduced dyna-babi, a new framework for highly controllable babi task generation. we used it to create compositional generalization datasets providing new modelling challenges for state-of-the-art neural language models. more broadly, our results underscore the importance in development of benchmarks themselves, beyond only the models solving them.