Incremental temporal summarization in multi-party meetings (2021)

Abstract: In this work, we develop a dataset for incremental temporal summarization in a multiparty dialogue. We use crowd-sourcing paradigm with a model-in-loop approach for collecting the summaries and compare the data with the expert summaries. We leverage the question generation paradigm to automatically generate questions from the dialogue, which can be used to validate the user participation and potentially also draw attention of the user towards the contents then need to summarize. We then develop several models for abstractive summary generation in the Incremental temporal scenario. We perform a detailed analysis of the results and show that including the past context into the summary generation yields better summaries.

## introduction in meetings, distractions by stimuli such as an email, text messages, slack messages, or in virtual at-home meetings by a child or a pet requiring immediate attention impact the concentration negatively. this exacerbates 'zoom fatigue' (fatigue caused by participating in too many virtual meetings) @xcite and impacts productivity negatively. one of the approaches suggested to optimize the concentration levels is to take frequent notes, which helps maintain engagement @xcite . however, some distractions require immediate attention and are unavoidable, or the participant may just tune-out during the meetings. a note-taking tool designed to help capture the notes for the time the user was distracted could be useful for the participants. such a tool that produces notes taking the past notes from the users and incrementally updating the notes for the time missed from the meeting could be useful. the goal of this work is to develop a dataset that helps us move towards the development of such an automatic dialogue summarizer that captures the notes for the chunks of time using the transcriptions and the past notes. the task of incremental temporal summarization in dialogue that is developed in this work has two main aspects to it, i) the content being summarized has a temporal order, meaning the information evolves over time. all conversations are temporal in nature, however, the current datasets on dialogue summarization @xcite @xcite @xcite consist of summaries that are written for the entire dialogue or parts of it (not in a sequence). thus the summaries are not in temporal order. ii) the summaries build upon or use the past context (transcriptions, summaries, or human notes) to generate the summaries for the current dialogue. to the best of our knowledge, current datasets on dialogue summarization do not possess incremental property. the incremental temporal summarization task bears a resemblance to the tasks of temporal summarization (ts) and incremental update summarization (ius) of news articles @xcite @xcite . these tasks are set up as a summarization task that utilizes news articles/summaries from the past along with the current newly available article to which the summary needs to be generated under the assumption that the user is aware of the past contents. incremental temporal summarization (its) for dialogue introduced in our work highlights challenges that are associated with processing human dialogue due to its incremental nature @xcite @xcite . for instance, the information (utterances, visual and prosodic signals) comes continuously and in smaller increments of time and at a much faster rate than news articles. contents to summarize also depend on dyadic exchanges (question and answers). disfluencies and the dynamic nature of dialogue introduces new challenges. to the best of our knowledge, while the corpora for ts and ius exist for the news/twitter feed summarization, a corpus for multi-party meeting scenarios does not exist. the first contribution of this work is towards providing a dataset for 'incremental temporal summarization' in a meeting scenario. our second contribution is that of providing a model-in-the-loop approach for summary data collection using crowd-sourcing. crowd-sourcing summaries data collection has proven to be a challenging task as the task is non-trivial, subjective, and often ambiguous. in this work, motivated by a promising multi-step approach developed by @xcite for crowd-sourcing summary data collection, we extend the literature by developing a model-in-the-loop approach for collecting summaries. the participants first read the context, mark extractives highlighting important utterances, answer automatically generated multiple-choice questions, and then provide an abstractive summary. we evaluate this approach by comparing the summaries generated by crowd-workers with those created by experts. our third contribution is towards the development and evaluation of baselines for its task and showing that the models, when provided with the context, generate better summaries than the counterparts which do not have access to the past context. while the focus of this work is not to provide new models, we develop the baselines using the recent transformer-based architectures that have performed well in the summarization tasks @xcite @xcite . ## related work dialogue summarization corpora @xcite @xcite @xcite @xcite have helped accelerate the research in the area of conversational summarization and helped identify the differences in the dialogue and news article summarization @xcite . our dataset could help progress the field by identifying similar differences and developing summarization model for incremental scenarios. ## data collection in this work, we extend the ami meetings corpus @xcite with the incremental temporal summaries. ami is a multi-modal corpus consisting of conversations between 4 role-playing participants (project manager (pm), industrial designer (id), user interface expert (ui), and marketing expert (me)) in a remote-control design scenario. each group of four participants meet four times and continue the conversation forward from the previous sessions but often on a new agenda. the ami corpus also consists of extractive and abstractive summaries for the conversation annotated by experts. one important thing to note is that the summaries are not temporal and incremental. summaries are often independent and can have overlapping or shared utterances with other summaries and correspond to variable time duration. for collecting data for its scenario, we split the conversation videos into 100 second time duration (called dialogue chunks) and collect extractive and abstractive summaries for each of these dialogue chunks. we use amazon mechanical turk (mturk) for data collection. our task on mturk was avail-able to participants in the us and canada with an acceptance rate of above 85% in a minimum of 50 tasks. we pay the users @xmath018.00 per hour) we describe the process of setting the pay in appendix a.2. ## models for summarization we also develop models for abstractive summarization in our work. our primary focus is on abstractive summarization for the incremental temporal scenario. the incremental temporal summarization module takes as input the utterances in the current time window along with the past summaries (context) to generate the summaries. however, it is not clear how important these contexts are. we thus mainly set out to answer this question as we develop the abstractive summarization models. ## discussion & future work in this work, we developed a corpus for incremental temporal summarization in dialogue using crowdsourcing. we showed that our approach to collect summaries yields summaries of comparable quality to experts. the dataset also @xmath1. recent developments in the summarizations have developed approaches that utilize such q-a (question-answer) approaches to facilitate summary generation @xcite . in this work, we use the q-a pairs for validating the cw summaries; however, the dataset developed in this work could help facilitate the development of similar approaches for conversational summarization. we developed models for automatic abstractive summarization and showed that models, when provided with past context summaries, helps generate better summaries. the crowd-workers in the study also indicated 94.6% times that the context helped them better understand the context of dialogue. we showed through the statistical tests that the bart model generated better summaries (measured in terms of r-1 and r-l scores) and showed that pretraining interacts with different models differently. hence, we could not conclude that the pre-training alone will help achieve better performance. this information could benefit model builders to test different combinations of a model with the training procedures to get the best performance. yet another avenue for the future work is the development and evaluation of the summaries using metrics that capture the incremental nature of the summaries generated.