Improving Cross-task Generalization of Unified Table-to-text Models with Compositional Task Configurations (2023)

Abstract: There has been great progress in unifying various table-to-text tasks using a single encoder-decoder model trained via multi-task learning (Xie et al., 2022).However, existing methods typically encode task information with a simple dataset name as a prefix to the encoder. This not only limits the effectiveness of multi-task learning, but also hinders the model’s ability to generalize to new domains or tasks that were not seen during training, which is crucial for real-world applications. In this paper, we propose compositional task configurations, a set of prompts prepended to the encoder to improve cross-task generalization of unified models. We design the task configurations to explicitly specify the task type, as well as its input and output types. We show that this not only allows the model to better learn shared knowledge across different tasks at training, but also allows us to control the model by composing new configurations that apply novel input-output combinations in a zero-shot manner. We demonstrate via experiments over ten table-to-text tasks that our method outperforms the UnifiedSKG baseline by noticeable margins in both in-domain and zero-shot settings, with average improvements of +0.5 and +12.6 from using a T5-large backbone, respectively.

## introduction table-to-text tasks, such as table-based question answering @xcite , summarization @xcite , or fact verification @xcite , are of high interest to the nlp community and have been applied in many real-world applications. traditionally, these tasks have been studied individually, with methods commonly optimized for one or a few tasks @xcite . however, with the recent popularity of pre-trained transformer models @xcite @xcite ., * work done during an internship at aws ai labs. short-form table qa (train) table summarization (train) long-form table qa (test) ## method & tasks prompting is a natural and feasible way to impose explicit control over the behaviors of pre-trained language models @xcite @xcite . in this work, we implement the task configurations as prompts of an encoder-decoder model. each task configuration contains the following four aspects: task type, input type, output type, and dataset name. the task type is the end goal of a task, e.g., qa and summarization, as shown in figure 2 . input and output types specify the inputs of the encoder and the outputs of the decoder of table-to-text models, respectively. these types can be compositional, for example, both long-form and short-form table qa in figure 2 require the decoder to output a set of relevant cells and the final answer. the dataset name specifies the dataset used for training. as different datasets can share the same task type, input and output types, we assume that the model is able to learn the shared and the unique knowledge across different datasets by adding the dataset names as configurations. when testing the model on a new dataset, we can simply omit the dataset name since it is not trained. one of the major advantages of having explicit task configurations is that it enables the model to learn the mapping between a configuration and its behavior. at test time, we can compose a new set of configurations which suits best for an unseen task using the trained configurations. figure 2 demon- title: list of awards and received by masterpiece year work award result 2010 merindang ke bintang the album of the year won … 2016 ngap sayot best vocals won long-form table qa (test) answer: masterpiece received nine awards from the dayak music awardswinning the album of the year twice in 2010 and 2016, and best vocal once in 2016. table summarization (train) short-form table qa (train) question: how many times did the masterpiece won the album of the year? 2016 ngap sayot best vocal won … the album of the year won the album of the year won 2010 the album of the year won 2016 the album of the year won 2016 best vocal won answer: two times answer: ngap sayot won the best vocals award in 2016. unified model question: what were the awards received by the masterpiece in 2010 and 2016? short-form table qa (train) table summarization (train) composi'onal configs encoder input decoder output long-form table qa (test) tasks: … input: … output: … … qa fact-check summarization table text query cell short answer long answer qa table query cell short answer summarization table cell long answer summarization table cell long answer query + + + figure 2: our method allows us to reformulate the unseen task of long-form table qa as a query-based summarization task by composing a new configuration using the existing ones from the two training tasks. at test time, this composed configuration allows the model to first identify relevant cells in the table based on the input question using the cell generation skills learned from short-form table qa, and then generates a long-form answer by utilizing the knowledge acquired from table summarization. for all three tasks, the model also takes the linearized table as input; for simplicity, we omit the linearized table and dataset name configuration in the figure. strates that by training on short-form table qa, the model learns the ability to generate a set of relevant table cells according to the question and then derive the answer based on those cells. by training on table summarization, the model learns to produce a summary based on a set of table cells. foot_0 at test time, by reformulating the long-form table qa as a query-based summarization task, our model is able to first generate a set of relevant cells (learned through short-form table qa) and then synthesize those cells to yield a long-form answer (learned from table summarization). note that our method is an efficient extension to the original unifiedskg model. it only requires a small input prefix, comprising less than 5% of the total sequence length, making it flexible for generalization to more tasks and datasets. ## experiments experimental settings we evaluate our method by following the experimental setup shown in baseline we mainly compare our method against unifiedskg @xcite , a strong baseline that was shown to achieve state-of-the-art results on many table-to-text tasks via multi-task training. in unifiedskg, for each task, a dataset name is prepended to the encoder during multi-task finetuning as a pseudo-task configuration. for fair comparisons, we re-trained unifiedskg models on the five in-domain datasets by using the authors' implementation. 2 evaluation for the in-domain tasks, we simply train on their training sets and evaluate on their test sets. for the test-only tasks, we evaluate our method in two settings: 1) a zero-shot setting, where we directly apply the model trained on indomain datasets and use a new set of task configs designed for each test dataset; 2) a few-shot setting, where for each test dataset, we further fine-tune the model using n randomly sampled training examples (where n is small). since we observed that the few-shot training is unstable and heavily depends on the sampled examples, we report average performance from 5 different random seeds (each with a different set of few-shot examples). 2 we in fact found that our version of the unifiedskg model fine-tuned over the five in-domain datasets outperforms the original authors' version on several datasets, establishing a more competitive baseline. for example, our version achieves 82.9 on wikisql and 77.1 on tabfact, whereas the original model by @xcite achieves 81.9 and 71.2, respectively. ## conclusion we introduced compositional task configurations for unified table-to-text models. compared to existing unified encoder-decoder models that sim-ply use dataset names as input prefix, compositional task configurations allow us to specify the task type, input, and output types at a finer level, which improve multi-task learning effectiveness and cross-task generalization. further, we showed that our method allows fine-grained control over the model's generation at test time, enabling the model to generalize to unseen tasks and improving explainability via generating high-quality supporting table cells.