Parameter-Efficient Abstractive Question Answering over Tables or Text (2022)

Abstract: A long-term ambition of information seeking QA systems is to reason over multi-modal contexts and generate natural answers to user queries. Today, memory intensive pre-trained language models are adapted to downstream tasks such as QA by fine-tuning the model on QA data in a specific modality like unstructured text or structured tables. To avoid training such memory-hungry models while utilizing a uniform architecture for each modality, parameter-efficient adapters add and train small task-specific bottle-neck layers between transformer layers. In this work, we study parameter-efficient abstractive QA in encoder-decoder models over structured tabular data and unstructured textual data using only 1.5% additional parameters for each modality. We also ablate over adapter layers in both encoder and decoder modules to study the efficiency-performance trade-off and demonstrate that reducing additional trainable parameters down to 0.7%-1.0% leads to comparable results. Our models out-perform current state-of-the-art models on tabular QA datasets such as Tablesum and FeTaQA, and achieve comparable performance on a textual QA dataset such as NarrativeQA using significantly less trainable parameters than fine-tuning.

## introduction information seeking systems over diverse contexts require model capabilities to reason over unstructured and structured data such as free-form text, tables, and images @xcite @xcite @xcite . such systems might have the additional requirement of generating natural language responses if deployed as task-oriented conversational agents @xcite @xcite . recent work on open-domain question answering (qa) predominately addresses these challenges by fine-tuning question question encoder lm head table adapter decoder text adapter nx table document natural answer table adapter text adapter figure 1 : parameter-efficient transfer learning using modality-specific (table/text) adapters for abstractive question answering massive pre-trained language models on different modalities such as tables and text @xcite @xcite @xcite . however, each model trained on a specific input type is incompatible with other modalities and requires modality-specific fine-tuning. for example, in tabular qa @xcite , the structure of the table is learnt by training additional position embeddings (row and column identifiers) to identify which row and column a table cell belongs to. this renders such modality specific models incompatible with free-form text-based models. multi-modal models @xcite can reason over both tables and text by concatenating the textual context and the flattened table, leading to longer input sequences and limiting the length of the context that can be encoded. to address these challenges, we study parameterefficient transfer learning for abstractive qa over tables and over text. we are motivated to use adapter-layers that inject small bottle-neck layers between frozen pre-trained transformer layers as they achieve comparable performance to fine-tuning on a variety of tasks such as multi-lingual translation @xcite @xcite , classification @xcite ), text-to-text generation @xcite , domain-adaptation in dialogue state tracking, and response generation @xcite . ablation studies on adapter layers @xcite on masked language models such as bertbase and roberta over the glue benchmark demonstrate that removing beginning adapter layers leads to a minimal drop in performance. extending adapter layer ablation over separate encoder and decoder modules is non-trivial as the conventional approach of sequential pruning of layers does not extend to consecutive encoder and decoder modules. our work explores the interaction of adapter layers from both modules in the context of abstractive qa. @xcite explore the impact of the adapter bottle-neck dimension for various language generation tasks over an auto-regressive model such as gpt-2 @xcite . they do not study tabular data nor ablate adapter layers, which is crucial in understanding impact of individual adapters in sequential transformer module architectures such as encoder-decoder. our analysis is complementary to @xcite as we ablate adapter layers to study parameter-performance trade-off whereas they only focus on adapter bottleneck size. also, we generalize beyond the text-totext setting and explore language generation from structured or unstructured input such as tables and text. this introduces domain-shift in both the task and structure of the downstream data. we propose a system, named parameter, efficient, abstractive question answering (peaqa), shown in figure 1 , which learns to reason over unstructured and structured input using a shared pre-trained language model and modality-specific adapter layers. we automatically transform hierarchical tables to regular tables to have a uniform representation without breaking associations between table cells. in addition, we extend the study of ablating adapter layers over both encoder and decoder modules. our main contributions are summarized as: (1) we perform parameter-efficient abstractive question answering over multi-modal context using only additional 1.5% of trainable parameters for each modality. our adaptertuned model outperforms existing work by a large margin on tabular qa datasets and achieves comparable performance on a textual qa dataset. (2) we study tabular qa as a new modality that introduces massive input domain shift to pretrained language models. we propose a 2step transformation of hierarchical tables to sequences, which produces a uniform representation to be used by a single, shared pre-trained language model and modality-specific adapter layers. to the best of our knowledge, this is the first work that explores tabular qa question answering in a parameter-efficient manner. (3) we ablate adapter layers in both encoder and decoder modules to study their impact and show that beginning layers from both encoder and decoder can be eliminated without significant drop in performance. we also demonstrate that last encoder adapter layers are indispensable and have greater contribution than decoder layers at the same level. textual question answering. question answering over text measures a system's ability to comprehend free-form text in the user question and context passage(s) and predict an answer. the answer predicted can be extractive in nature, where the system identifies short text spans in the context passage to answer the user query @xcite @xcite , or it can be abstractive, where it is required to generate a free-form answer @xcite @xcite . transfer learning. transfer learning techniques such as fine-tuning pre-trained models for down-stream tasks, require a new set of parameters to be learnt for each new task. to avoid such memory intensive transfer learning methods, adapters have been proposed as a parameter-efficient method of adapting to new domains @xcite . adapters have been extended to language generation in a variety of generative tasks such as translation, summarization, multiturn dialogue, and task-oriented natural language generation @xcite . our work combines all the aforementioned aspects to generate abstractive answers from both tables and text with only 0.7%-1.0% trainable parameters without compromising performance. ## model we focus on encoder-decoder models for the task of abstractive question answering. we use a bart @xcite encoder-decoder architecture which comprises of a bidirectional encoder and an auto-regressive decoder. the input sequence consists of the question, the context title and context sequence preceded with prompts indicating the beginning of the each sub-sequence. formally, the input sequence is represented @xmath0. . . q @xmath1. . . t @xmath2. . . c n , where q i is the i-th question token, t j is the j-th title token, and c k is the k-th context token. the context can either be a text passage or a flattened table. the parameters of the pre-trained bart model are frozen during training. modality specific adapter layers added to the model are trained on either tabular context or textual context to generate natural answers. ## textual question answering to study multi-modal abstractive qa, we first focus on free-form text as context to the system. we train adapter layers for textual context on the narra-tiveqa dataset @xcite . narrativeqa is a complex abstractive question answering dataset over stories. the dataset contains 32, 747 samples in the training set, 3, 461 samples in the validation set, and 10, 557 samples in the test set. for our task, we have selected the input context passage to be the human annotated summary of each sample which is the wikipedia page summary of the story and represented as a paragraph. the input to the model is the question, title and summary of each passage and the target is the abstractive answer. ## tabular question answering we study tabular qa as a new modality which introduces massive input domain shift to pre-trained language models. tables enforce structural constraints in their representation which is incompatible with the expected input format of pre-trained language models. to achieve our goal of parameter efficiency by utilizing a uniform pre-trained language model, we only train table specific adapter layers while keeping the pre-trained model frozen. however, this necessitates a uniform input representation for both tables and text. an additional challenge is introduced to maintain uniformity across different table types (regular, hierarchical). for our task, we explore 2 tabular qa datasets, namely, tablesum @xcite and fetaqa @xcite . tablesum consists of 200 unique wikipedia tables over which questions and abstractive answers are manually annotated; 40% of the samples are questions over hierarchical tables but the tables in their released data are missing information in the hierarchical cells and their work do not handle hierarchies. we address this issue by extracting the wikitables from the respective wikepedia pages and release a clean version of the dataset. 1 fetaqa @xcite is a larger abstractive tabular qa dataset consisting of question and free-form answers over 10, 330 regular tables. the dataset consists of 7, 326 samples in the training set, 1, 001 in the validation set, and 2, 003 in the test set. fetaqa consists of human-annotated answers containing explanations involving entities and relations. ## experimental setup we seek to answer the following research questions with our experiments: (rq1) how does adaptertuning perform compared to fine-tuning in the context of multi-modal input? (rq2) do all adapter layers across the encoder and decoder contribute equally to performance across tasks/modalities? ## results we compare the results of our baseline fine-tuned models with the state-of-the-art fine-tuned mod-els in section 7.1. we address (rq1) "how does adapter-tuning perform compared to fine-tuning in the context of multi-modal input?" in section 7.2 and (rq2) "do all adapter layers across the encoder and decoder contribute equally to performance across tasks/modalities?" in 7.3. ## conclusion we are the first to study parameter-efficient transfer learning over tables and text for abstractive question answering using adapters. we demonstrate that parameter efficient adapter-tuning outperforms finetuning on out-of-domain tabular data and achieves comparable results on in-domain textual data. we propose a transformation from hierarchical tables to regular ones and further into a sequential form compatible with pre-trained model. we extend an existing ablation study of adapter layers to encoder-decoder setting and demonstrate that adapter layers from the end of the encoder is indispensable to encoding modality specific information than decoder adapter layers at the same level. our results are useful for exploring scalability of qa models in memory constrained situations with comparable performance while scaling across modalities using light-weight adapters. one of the limitations of our work is that our models do not explicitly reason and aggregate over the table cells. this might lead to fluent but factually incorrect answers on challenging tablesum dataset. addressing this limitation is left as future work.