D i N e R : A Large Realistic Dataset for Evaluating Compositional Generalization (2023)

Abstract: Most of the existing compositional generalization datasets are synthetically-generated, resulting in a lack of natural language variation. While there have been recent attempts to introduce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate compositional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DiNeR) task and create a large realistic Chinese dataset. Given a recipe instruction, models are required to recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 dishes and 228,114 recipes, and involves plenty of linguistic phenomena such as anaphora, omission and ambiguity. We provide two strong baselines based on T5 and large language models (LLMs). This work contributes a challenging task, baseline methods to tackle the task, and insights into compositional generalization in the context of dish name recognition.

## introduction compositional generalization, the capacity to comprehend and create new combinations of observed components @xcite , has been shown to be an aspect of human intelligence both empirically @xcite and experimentally @xcite . taking three recipes in figure 1 as an example, by learning the instructions of cooking braised pork (红烧肉) and marinated beef brisket (卤牛腩), we can acquire knowledge about components in dish names, like the use of light soy sauce and dark soy sauce usually indicates the action braise, and beef brisket is more likely to be a staple ingredient than spices. given such knowledge, when reading the instructions of cooking braised beef brisket (红烧牛腩), we can find the features corresponding to summarizing the staple ingredient beef brisket and abstract the action braise, thereby compose them into the name of this dish. a bunch of synthetically-generated datasets have been created for assessing compositional generalization (lake and @xcite @xcite , and plain sequence-to-sequence(seq2seq) models exhibit significant out-of-distribution (ood) compositional generalization performance loss comparing to in-distribution (id) setting. while effective methods have been proposed to overcome the difficulty in ood compositional generalization @xcite @xcite , most of them mainly focus on semantic parsing, where some important abilities like summarization dataset # of samples generalization forms geoquery @xcite 880 3 spider-ssp @xcite 4,376 3 smcalflow-cs @xcite 28,054 2 counterfactual @xcite 2,500 1 diner(ours) @xcite recently, non-synthetic datasets have been proposed to evaluate compositional generalization on more realistic scenario @xcite @xcite . however, as shown in to address the aforementioned issues, we propose the dish name recognition (diner) task, and build large-scale chinese dataset with a diverse collection of combinations. in this task, given the instructions of a recipe, models are required to predict the dish name foot_0 , which is often composed of up to three kind of components: food, actions and flavor. as visualized in figure 1 , models will be tested on unseen dish names composed by observed components. this task is not trivial even without compositional generalization setting. firstly, it takes summarization and abstracting to figure out which ingredients, actions and flavors should be included in dish name. secondly, the presence of anaphora, omission and ambiguity in recipes further increase the difficulty. we discuss such difficulty in §3.5 with a more comprehensive analysis. based on the xiachufang recipe dataset @xcite , we collect 3,803 dish names that can be parsed into ingredients, actions and flavors, with 223,581 corresponding recipes. as shown in we develop baseline methods to solve the task, as existing compositional generalization models cannot fit in the dish name prediction task form. the first baseline, seq2seq trained t5, suffers from a significant id-ood performance gap. inspired by the chain of thought prompting method @xcite , we propose to fine tune with compositional prompting (cp-ft). we first train a component model for predicting each kind of components, then we add the predictions of components to the input, and fine tune the model with the compositional prompt. to better fit the t5 model to the recipe domain, we also perform continual pretraining on a large-scale recipe corpus to equip models with domain knowledge. the proposed cp-ft method and continual pretraining improve the ood f1 score by a large margin. with large scale data and distribution-based split, the controllability of our dataset is good. we demonstrate how to evaluate the compositional generalization ability of models on different training size and different level of distribution shift. we first find an inverse-scaling phenomenon between ood performance and data scale. according to @xcite , on large distribution shift, fine tuning can distort pretrained features and underperform ood. we control the level of distribution shift, and empirically prove that the inverse-scaling phenomenon between ood performance and data scale is caused by large distribution shift. our contribution is three-fold: 1) we propose a challenging compositional generalization task, dish name recognition (diner), and create a large-scale realistic dataset. 2) we develop strong baseline: t5 with continual pretraining and fine tuning with compositional prompting and gpt-3.5 with selective demonstrations. 3) we demonstrate the controllability of our dataset and provide in- category # of dishes base dishes target dish combtwo 43 beef, bun beef bun addone 11 noodles with gravy, steamed perch steamed noodles with gravy recotwo 619 braised pork, marinated beef brisket braised beef brisket recothree 21 spicy rice noodle, pork with mushroom, braised chicken rice noodle with mushrooms and chicken ## related work compositional generalization is the ability to understand and create new combinations of elements and structures that have been seen before. to measure the compositional generalization ability of models, a series of studies have proposed syntheticallygenerated datasets based on semantic parsing tasks (lake and @xcite @xcite . however, the synthetic data lack natural language variation @xcite such as anaphora, omission and ambiguity @xcite . recently, some works has proposed to evaluate the compositional generalization ability of the model on realistic data. @xcite proposed two non-synthetic semantic parsing datasets, geoquery and spider-ssp, and divided the data based on template, length and tmcd, but there are only hundreds or thousands of samples in these two datasets. @xcite proposed a larger scale semantic parsing dataset smcalflow-cs, but this dataset only involves a compositional generalization from skills involving a single domain to skills involving two domains. @xcite proposed a counterfactual recipe generation task based on different combinations of ingredients, flavors and other components in a recipe, and examined the model's ability to modify a base recipe according to the change of an ingredient. however, this task only involved 2500 pieces of data and two forms of compositional generalization. inspired by @xcite , we also study compositional generalization based on the three components of food, action and flavor in recipes. our dataset is based on large-scale real-world recipe data generation. compared with synthetic dataset, our dataset contains rich linguistic phenomena and combination forms. our data splits result in four types of compositional generalization forms. comparing to exsiting compositional generalization datasets, our dataset has the advantage of data size and diversity of compostional generalization forms. ## demonstrations of data controllability with the large data scale and distribution-based data splits, the diner dataset has good controllability, which enable us to study the model's performance under different data scales and different levels of distributional shift. in this section, we demonstrate such controllability by evaluating the continue pretrained model under multiple conditions. to generate data splits of different levels of distributional shift, we early stop the splitting algorithm and generate another two splits with compound divergence of 0.8 and 0.6. for each data splits, we sample different proportion p of training data for as revealed in figure 5 , under all the three data splits, there is an inverse-scaling phenomenon between ood performance and data scale, which is opposite to the usual case when the training data and test data are sampled from the same distribution. according to @xcite , on large distribution shift, fine tuning can distort pretrained features and underperform ood. we propose that the inverse-scaling phenomenon between ood performance and data scale is caused by large distribution shift. we empirically verify this by comparing model's performance under different data splits. as the compound divergence decrease, the downward trend ease, indicating that the degree of inversescaling has a positive correlation with the level of distributional shift. ## conclusion we propose dish name recognition (diner) task to evaluate natural language understanding ability under compositional generalization settings, and create a large-scale realistic dataset with rich combinations and generalization forms. we provide two strong baselines: 1) t5 with continual pretraining and fine tuning with compositional prompting; 2) gpt-3.5 with selective demonstrations. our t5-based approach outperforms the plain seq2seq trained version by a large margin (8.4%). however, the id-ood performance gap remains unsolved. we also demonstrate the controllability of our dataset and provide insights into compositional generalization in the context of dish name recognition. by proposing this task, we provide a realistic and flexible benchmark for evaluating generalization.