Context or No Context? A preliminary exploration of human-in-the-loop approach for Incremental Temporal Summarization in meetings (2021)

Abstract: Incremental meeting temporal summarization, summarizing relevant information of partial multi-party meeting dialogue, is emerging as the next challenge in summarization research. Here we examine the extent to which human abstractive summaries of the preceding increments (context) can be combined with extractive meeting dialogue to generate abstractive summaries. We find that previous context improves ROUGE scores. Our findings further suggest that contexts begin to outweigh the dialogue. Using keyphrase extraction and semantic role labeling (SRL), we find that SRL captures relevant information without overwhelming the the model architecture. By compressing the previous contexts by ~70%, we achieve better ROUGE scores over our baseline models. Collectively, these results suggest that context matters, as does the way in which context is presented to the model.

## introduction in meetings, especially in a virtual setting, distractions are common place and can last anywhere from a few seconds to minutes, impacting concentration and participation in the remainder of the meeting negatively. a note-taking tool designed to provide temporally relevant summaries of what has happened in the last 2-3 minutes may mitigate the negative effects of distractions and interruptions. missing a few minutes of content, rather than the whole meeting, provides unique challenges for current summarization tools. instead of summarizing the main points of the meeting, a temporallyrelevant summarization aid must instead capture relevant meeting content given previous events, even if those events would not be included in the full meeting summary. such a tool may benefit from taking the past notes or summaries from meeting participants as context and incrementally updating the summaries for a specific time interval to capture relevant information that a distracted individual would need to know to reintegrate into the meeting. the goal of this work is to investigate the ability to incrementally summarize meetings, specifically focusing on how a summarization tool may make use of past summaries to increase the accuracy of temporally-relevant abstractive summarization. the task of incremental temporal summarization in dialogue has two main aspects to it, i) the content being summarized has a temporal order-the information evolves over time. ii) summaries build upon or use the past context (transcriptions, summaries, or human notes) to generate the summaries for the current dialogue. a new dataset based on incremental temporal summarization of the ami dataset, which we call the ami-its, provides a means to investigate incremental temporal summarization of meeting dialogues. temporal summarization has been studied in the context of summarizing news articles @xcite @xcite . in such a setting, the input news articles that evolve over time are streamed in chunks. the summarizer needs to either summarize the new content or update the earlier generated summary with the new information. while similar to incremental temporal summarization (its) in meetings scenario, additional challenges are associated with the properties of human conversation such as disfluencies and dyadic exchanges @xcite where a contributions to the summaries are from multiple interlocutors @xcite .the information also comes in smaller increments of time, and at a much faster rate than news articles. limited work has been done on temporal summarization and incremental summarization in multi-party meeting scenarios. the main contribution of this work is to quantify the impact of previous human generated sum-maries in improving meeting summarization. we specifically focus on how to best use previous summaries from earlier temporal summarization. this mimics the use of the meeting notes of individuals to generate up to date summaries of meeting dialogue and provides the basis for an incremental summarization tool that works jointly with meeting participants in real time. we ask fundamental questions about how to use previous summaries by humans including whether meeting summaries or meeting dialogues should be prioritized as input to the model. we then look at how many summaries the model requires to most accurately summarize the most recent temporal chunks and conclude by showing that extracting meaningful information from past summaries through semantic role labeling can further improve temporal summarization. collectively this work shows that temporal summarization benefits from having a human in the loop and suggests ways to use human input most effectively. ## related work because of the differences between news articles and human dialogue, incremental summarization for meetings/dialogues provides unique challenges and requires novel approaches. ## data our primary focus is on abstractive summarization for incremental temporal scenarios. the incremental temporal summarization module takes the utterances in the current time window as input. in this work, we focus on how best to use the past summaries (context) as input. models are evaluated on temporal summaries capturing the last 100 seconds of the meeting. while it has been shown that having previous temporal summaries is helpful in accurately summarizing a specific context @xcite , we investigate this question further by asking how much context is relevant and how to best use past context. we use these results to draw conclusions about the role of human summarization in model performance. ## methods and results we focus our exploration on bart as the baseline model as this model has been investigated both in incremental summarization and dialogue summarization. for fine-tuning of abstractive mod-els, we fine-tune for a maximum of 25 epochs and choose the model resulting in the best rouge-1 f-measure on the validation set. we use the following configuration for all baseline models: learning @xmath0.0001, training batch @xmath1, and label smoothed negative log-likelihood loss. the maximum sequence length is set to 1024. the models can generate summaries of the max length of 142 tokens. for model training and inference, we use multiple machines with a combination of either an intel(r) xeon(r) or intel(r) xeon(r) platinum 8280 cpu and nvidia titan x or titan xp gpu. all models were trained on 2 gpus. for the pretrained models, we use the bart-large-cnn model, from the huggingface @xcite library. we retain the default model configurations. for all experimental conditions, we input the transcriptions of the extractives marked by crowd workers in the ami-its dataset and n previous contexts. the order of the previous contexts are randomly shuffled when building the dataset. we evaluate models on their ability to generate the abstractive summaries similar to those provided by the crowd workers in the ami-its dataset. ## discussion & future work in this work we present an analysis of the role of past context on summarizing 100 seconds of temporal meeting dialogue. we explore, in depth, the way in which past summaries can be used by a summarization model to generate abstractive summaries. our work strongly suggests that context impacts model performance. we also find the way in which we represent previous summaries can impact metrics related to the quality of the abstractive summaries. we show that in certain conditions human generated summaries can improve over models with no contextual information. we then show that extracting meaningful content from past summaries can further boost model performance. specifically, we found the verb arguments of a semantic role labeler provides the most performance improvement over our baseline models. we believe that this result provides a new direction for temporal summarization by suggesting that contextual information preceding the specific dialogue may be informative for the model in generating summaries. to further analyze the summaries generated by the models we compare the summaries to the extractive text that was provided as input. there are limitations and clear future directions of this work. first, the model architecture we explored here is the standard bart summarization architecture. more recent models have achieved impressive performance on meeting summarizations @xcite @xcite . exploring these architectures and adapting them for its scenario remains a promising avenue for the future work. this work also suggests that an architecture specifically aimed to capitalize on past summary information may be a promising line for our future work. when inspecting model performance, specifically when the role labels were not present, we found that the model tended to overattend to previous contextual information. this may be mitigated by building an architecture that keeps dialogue and context information separate. our work provides a rather simplistic hitl (human in the loop) approach for summarization. in this work, we integrate the summaries from the past as input to the models. while, the approach is simple, we have demonstrated that such a method of integrating context information could help improve the performance of the summarizer. integrating human inputs into the inference pipeline is an interesting area for future work. eventually, this system should be able to integrate human information seamlessly, requiring more experiments and analysis to understand how individuals are generating temporal summaries and how the model makes use of the past context for prediction. one of the challenges is evaluating the quality of summaries in a scalable and automatic fashion. the rouge metrics are widely adopted for the purposes of summary evaluation @xcite . while numerous automated evaluation metrics exist for measuring how closely the generated summary matches with the ground-truth (fabbri et al., 2021a) a metric for its scenario needs further research. human evaluations are commonly adopted for measuring the summary quality. however, such an approach can be expensive and could also prove to be noisy when deployed over crowdsourcing environment. recently @xcite have highlighted the issue and provided an interactive evaluation of multi-document summaries. we intend to explore other types of evaluations and human judgements on its datasets in the future. incremental temporal summarization is an emerging area of research and thus limited by data. we base all our analysis on the ami-its dataset @xcite . one aspect of this dataset is that summaries are generated by individuals who are seeing the 3 previous summaries generated by other crowdsource workers. these workers may be influenced by these previous summaries when generating their summaries of the last 100 seconds. because of this, the summaries themselves may contain information about previous context making the addition of other contexts redundant and altering the extendability of these results. in the future, we intend to analyse and better understand how transformer models use previous context as well as how individuals determine what aspects of a meeting are important for incremental summarization.