Analyzing Multi-Task Learning for Abstractive Text Summarization (2022)

Abstract: Despite the recent success of multi-task learning and pre-finetuning for natural language understanding, few works have studied the effects of task families on abstractive text summarization. Task families are a form of task grouping during the pre-finetuning stage to learn common skills, such as reading comprehension. To close this gap, we analyze the influence of multi-task learning strategies using task families for the English abstractive text summarization task. We group tasks into one of three strategies, i.e., sequential, simultaneous, and continual multi-task learning, and evaluate trained models through two downstream tasks. We find that certain combinations of task families (e.g., advanced reading comprehension and natural language inference) positively impact downstream performance. Further, we find that choice and combinations of task families influence downstream performance more than the training scheme, supporting the use of task families for abstractive text

## introduction self-supervised learning has been a significant success driver for generating high-quality abstractive summaries @xcite @xcite @xcite . through self-supervision, language models implicitly learn intrinsic language features (e.g., syntax) from unlabeled data that they can use to solve downstream tasks @xcite . however, skills necessary to perform specific tasks often can be learned from an existing set of labeled data, requiring fewer training iterations @xcite . for example, to perform text summarization, a helpful skill is the ability to answer questions about texts @xcite . the multi-task learning paradigm and its variations aim to acquire multiple skills simultaneously to succeed on the downstream tasks, e.g., t5 @xcite , and are independent of a specific training stage @xcite . while studies on the effects of multi-task learning on a large scale exist @xcite @xcite and are evaluated on broad natural language understanding benchmarks @xcite , they are lacking insight on the influence on abstractive text summarization. furthermore, multi-task learning approaches are diverse in their methods (e.g., training scheme, mixing strategy, task families), hampering their comparison. in this work, we investigate the role of multi-task learning on english abstractive text summarization. therefore, we organize 18 pre-selected training tasks into six higher-level, modular task families. further, we compare three training schemes for the pre-finetuning stage and their respective mixing strategies through changes of multiple scores. our experiments show that families' choice significantly impacts text summarization, while different training schemes have little influence. moreover, pairing a text summarization task family with any other helps to stabilize the overall performance when transferring to unknown data. in some cases, we also found that a text summarization task family can be substituted by other family pairs, e.g., advanced reading comprehension and classification. to summarize our contributions: task selection and relationship. @xcite conduct an empirical investigation on 33 tasks across three broad groups (i.e., text classification, question answering, and sequence labeling) to ex-plore their inter-and intra-group training for different group sizes. their experiments suggest that positive transfers between task groups are possible when the source dataset is small, and intergroup transfers are sensitive to group sizes. ext5 @xcite analyzes the correlation of task family representatives and shows, that summarization tasks (i.e., cnn/daily mail @xcite , xsum @xcite , wikilingua @xcite ) generally reduce performance on most other task families and that cbqa tasks (i.e., natural questions @xcite , trivia qa @xcite , hotpot qa @xcite ) are sensitive to multi-task learning. for the task relationship and transfer analysis, @xcite train on two families simultaneously and evaluate the first one. we expand the study of @xcite by adapting task families and respective representative tasks to be related to the text summarization task (section 3.1), considering different family combinations, training approaches (section 3.2), and tracking their performance through additional metrics for different unseen datasets (section 4). multiple works leverage algorithms for the selection of training tasks, e.g., @xcite use bayesian optimization to learn similarity measures (i.e., jensen-shannon divergence @xcite and rényi divergence @xcite ) and a beta-bernoulli multi-armed bandit with thompson sampling @xcite is used by autosem @xcite . conversely, ext5 @xcite does not rely on automatic training task selection approaches as described by the preceding works and instead chooses an empirical approach to select tasks for higherlevel task families. we follow the approach of @xcite 's task representative selection when choosing our tasks as the training task correlation analysis in ext5 indicates which families could positively influence text summarization. ## methodology we name our study toasts, a task-oriented analysis for text summarization to investigate the effects of different task family combinations on english abstractive text summarization via a multi-task learning architecture. toasts groups selected pre-training tasks into task families and explores the correlation of these families, their influence on two downstream tasks, and their aggre- task family task dataset source characteristics classification sentiment classification goemotion (2020) reddit multi-label cls [cls] sentiment classification imdb (2011) imdb binary cls topic classification ag news (2015) cometomyhead multi-class cls commonsense fill-in-the-blank winogrande (2021) wsc dataset binary options [cmns] question answering physicaliqa (2019) instructables.com binary options question answering socialiqa (2019) crowdsourced ternary options natural language inference textual entailment cls mnli (2018) snli corpus multi-label cls [nli] textual entailment cls anli (2020) human-and-modelin-the-loop dataset multi-label cls textual entailment cls qnli (2018) wikipedia binary classification reading comprehension binary qa boolq (2019) google yes/no answer [rc] extractive qa squad (2016) wikipedia extractive answers abstractive qa tweetqa (2019) twitter abstractive answers allowed advanced rc rc + information retrieval hotpotqa (2018) wikipedia multi-hop question answering [rc + ] rc + open domain qa natural questions (2019) google, wikipedia answer information seeking questions rc + cmns record (2018) cnn/dailymail and internet archive extractive machine rc summarization extractive sum xsum (2018) bbc one-sentence summary [sum] abstractive sum wikilingua [eng] (2020) wikihow one-sentence summary abstractive sum aeslc (2019) e-mail subject line generation the task family setup groups different tasks and related datasets into broader families according to their primary objective. the tasks of these families are then combined following a training strategy and evaluated into a final task. figure 1 illustrates the components of toasts, which are detailed in the following sections. ## experimental setup model. for all experiments, we use bart-large @xcite to probe combinations of task families, mixing, and training strategies in toasts. bart is a two-stage denoising autoencoder that corrupts its input text and reconstructs it through a sequence-to-sequence model. we chose bart because of its ability to perform a wide range of downstream tasks, such as paraphrase detection (wahle et al., 2022b), fake news identification @xcite , and text summarization @xcite . additionally, in our preliminary experiments, bart also performed better than other candidate models such as pegasus @xcite and t5 @xcite (comparison in tables 9 and 10 in appendix b). ## conclusion & future work in this work, we studied the influence of multi-task learning combinations of task families during the pre-finetuning stage for english abstractive text summarization. we trained three different training strategies, six task families composed of 18 tasks, and evaluated two downstream tasks. our experiments show that non text summarization task families, e.g., advanced reading comprehension, can be used as a substitute for the summarization task (rq2) or the combination of all task families (rq1). however, including the summarization task family in the training process positively impacts the downstream performance compared to non text summarization family combinations. further, our analysis shows that training strategies have little influence on the overall performance compared to the task family selection. we see this analysis as the first step to understanding training strategies and task families for text summarization. in the future, we want to investigate more tasks (both in number and diversity) per task family, training schemes, and mixing strategies. we also plan to include psychological studies comparing the similarities of textual understanding tasks as a starting point for task family preselection.