TEAM UFAL @ C reative S umm 2022: BART and S am S um based few-shot approach for creative Summarization (2022)

Abstract: This system description paper details TEAM UFAL’s approach for the SummScreen, TVMegasite subtask of the CreativeSumm shared task. The subtask deals with creating summaries for dialogues from TV Soap operas. We utilized BART based pre-trained model fine-tuned on SamSum dialouge summarization dataset. Few examples from AutoMin dataset and the dataset provided by the organizers were also inserted into the data as a few-shot learning objective. The additional data was manually broken into chunks based on different boundaries in summary and the dialogue file. For inference we choose a similar strategy as the top-performing team at AutoMin 2021, where the data is split into chunks, either on [SCENE_CHANGE] or exceeding a pre-defined token length, to accommodate the maximum token possible in the pre-trained model for one example. The final training strategy was chosen based on how natural the responses looked instead of how well the model performed on an automated evaluation metrics such as ROGUE.

## introduction creative summarization as a field is rather novel, which neatly exists between document summarization and conversation summarization. the task of summarization focuses on extracting relevant information from the entire document where the task of minuting includes an additional objective of getting rid of redundancies in the dialogues as well as extracting relevant information based on different boundaries within the text i.e. topic switching. it is written, proof-read and mostly contains of coherent and grammatically correct sentences, and since it is also supposed to mimic how people speak, it also contains grammatically incorrect sentences as well as people speaking over each other. thus, this track of research has a unique opportunity to leverage the recent advances in document summarization and automatic minuting. unlike the dataset used for automated minuting, the dataset in this subtask carries a special property where the conversation changes are marked with special tokens such as @xcite with on average a transcript containing 20±14, scene breaks across training, test and dev dataset splits. since the dataset consists of transcripts from different shows, which presumably are written by different screenplay writers resulting in different writing style, which explains the very high standard deviation on how often scene_change is there in a transcript. while this approach makes it easier to split a transcript into multiple parts, it does not guarantee that the segments will not exceed the tokens limits of the pre-trained model, 1024 in our case. the task for summarizing a tv show episode introduces a unique challenge compared to the task of summarizing a conversation. the transcripts for tv shows not only contain the dialogues, it also contains visual cue descriptions which are absent from the minuting summarization task as shown in . however, on a broad sense these tasks share some similarities as a summary can constructed from the perspective of one character in the show can be polar opposite to the summary generated from the perspective of a different character. previous approaches to the multi-party summarization includes modeling intra-speaker and inter-speaker topics with random walk in a graph @xcite , leveraged word-embeddings @xcite by using wordnet @xcite to make summary more abstractive. the word-embedding based approach was further incorporated by @xcite , and @xcite with different model architecture with @xcite incorporating information from visual modality into their summarization model. our approach in this subtask builds upon the ideas introduced in these papers, in training and inferences. we incorporate a pipeline with three components, model, data cleaning and inference ## dataset creatievsumm shared task recommends using summscreen @xcite for training, evaluation and testing purposes. it is divided into two parts based on the source of collection, i.e. the tv megasite (tms) and foreverdreaming(fd). for our system, we chose to work with foreverdreaming part of the summscreen dataset. this split in turn is released into two forms, one is anonymized where character names are replaced by "entity", and another is the normal transcript with character names present. for our submission, we chose to work with anonymized version of the dataset, which was in-line with automin @xcite dataset. figure 1 provides a glimpse of how data looked in its original phase. we first began by sanitizing the data and removing information which was not relevant to us. our sanitization process included removing @@, and using mosestokenizer @xcite to fix the tokenization in the dataset. we also implemented various regex expansion to convert i'@xmath0, ..shouldn'@xmath1..should not. after a qualitative overview of the dataset in all splits, we implemented more rule-based text processing such as removing the additional information which did not include a character's action. among such rules, one was to remove the line if it did not start with "entity" after cleaning all the punctuations and extra space. we also change dialogues such as "entity1 laughs" to "entity1 : laughs" to reduce the number of lines removed from the dataset for not being a conversational utterance. we added 16 examples from the dataset with high number of @xcite and sentence count in summaries. the mixture of dataset is visualized in figure 3 . we manually split the summaries based on the relevant splits of the transcript. we also implement similar measures for the au-tomin dataset, where we also sampled eight transcripts and their corresponding summaries. the relevant statistics for the summscreen foreverdreaming dataset is included in ## result ## methodology for our experiment, we use pre-trained bart @xcite from facebook research, released as a pre_trained model on xsum @xcite on hugging-face @xcite . we further fine-tuned the model on samsum dataset. the exact implementation for our models is released publicly on github foot_0 . we are also releasing the performance of different pre-trained models, such as t5 for zero-shot and few-shot learning on summscreen foreverdreaming dataset for inference, we use the same strategy as @xcite 's model for automin2021. we split the transcript into multiple chunks of dialogues. this split is done either on [scene_break] occurrence or when the token count exceeds a pre-defined limit. this helps us in extracting all the relevant information from the data without losing any information in truncation. bart is a denoising autoencoder used to pretrained seq-to-seq models for natural language generation among other tasks. during the training of this model, a random denoising function is used to corrupt the text and then it learns how to recover the original text. it is also capable of operating bi-directionally on sequence generation unlike bert @xcite . we primarily focused on this training strategy because of its proven results on automin shared task. during inference as depicted in figure 2 , we split the conversation into chunks, and concatenate the output to construct final output. ## conclusion in this system description paper, we explain the training regime we used to participate in the cre-ativesumm shared task using summscreen foreverdreaming dataset. our system incorporates mixing of training samples from novel datasets into an existing dataset, which resembles a few-shot approach. in section 6, we propose the direction we would like to take for incorporating the shared task training dataset into training.