PREME : Preference-based Meeting Exploration through an Interactive Questionnaire (2023)

Abstract: The recent increase in the volume of online meetings necessitates automated tools for organizing the material, especially when an attendee has missed the discussion and needs assistance in quickly exploring it. In this work, we propose a novel end-to-end framework for generating interactive questionnaires for preference-based meeting exploration. As a result, users are supplied with a list of suggested questions reflecting their preferences. Since the task is new, we introduce an automatic evaluation strategy by measuring how much the generated questions via questionnaire are answerable to ensure factual correctness and covers the source meeting for the depth of possible exploration.

## introduction in recent years, video conferencing technology has gained substantial improvements, and thus, online meetings have become easily accessible and more prominent. primarily due to the pandemic and working from home, the need for video calling has grown significantly. therefore, the high volume of online meetings necessitates automated tools for managing and organizing essential information for attendees. especially when an attendee has missed an online meeting, it is critical to access the required information since quickly reading through the transcript is quite time-consuming. providing meeting summaries is a promising direction @xcite @xcite . however, recent studies show that 1) users' needs do not fully align with current approaches to automatic text summarization @xcite and 2) approaches designed for document summarization could not effectively apply to meetings transcripts @xcite @xcite due to the following potential reasons: (r1) structure: standard documents are well structured compared to meeting transcripts; figure 1 : an example of exploring one of the meetings from the collection @xcite based on user preferences through an interactive questionnaire. this work is motivated by the fact that asking questions is a more efficient way for humans to ac- meeting transcript meeting segmentation question generation subject and aspect extraction subject normalization questionnaire questionnaire generation {s 1 , a 1 } {s 2 , a 2 } {s 1 , a 1 } {s 2 , a 2 } {s 3 , a 3 } {s 1 , a 1 } {s 2 , a 2 } {s norm , a s norm } {s norm , a s norm } {s norm , a s norm } 𝑆 𝑛𝑜𝑟𝑚 𝑆 𝑛𝑜𝑟𝑚 ❑ 𝑆 𝑛𝑜𝑟𝑚 𝐴 1 𝑛𝑜𝑟𝑚 𝐴 2 𝑛𝑜𝑟𝑚 which aspect of 𝑆 𝑛𝑜𝑟𝑚 are you interested in? which subjest are you interested in? questions you might be interested in: quire information than notes in plain text @xcite @xcite . thus, we address preference-based meeting exploration by automatically generating a structured interactive questionnaire for a transcript that covers most of the discussed topics and quickly walks users through the discussed content. an example of the desired questionnaire is shown in fig. 1 . first, the user has the ability to express their preferences regarding subjects that have been discussed @xcite @xcite . next, the questionnaire interactively suggests narrowing down their exploration if possible by displaying a list of possible related aspects. as a result, a ranked list of questions reflecting user preferences is generated. next, the user can pick a question that demonstrates their seeking needs the most and is redirected to the meeting part containing an answer. interactively asking for preferences in the questionnaire is beneficial because the user oversees what has been covered during the meeting they have missed. in section 4.2 we elaborate on a user study on a number of professionals who find such application useful for their daily job. hence, the goal of proposed questionnaires is twofold: (g1) to compactly represent the discussed content; (g2) to guide users to form questions that express their preference regarding the transcript. we require the generated questionnaire to satisfy the following properties: p1 coverage: coverage is the amount of the information from the source text that a questionnaire points to. the generated questionnaire must cover the meeting as much as possible; p2 answerable: a given meeting transcript should contain the answers to the questions generated as a result of the questionnaire. to address the defined challenge, we propose a framework, preme, which consists of several concrete sequential steps highlighted in fig. 2 . we start by enchaining the method to extract meeting segments @xcite . due to the conversational nature of the meeting, topic detection from the segments is challenging @xcite @xcite . thus, we indirectly extract the topics as follows. first, we generate questions from each segments (brown et al., 2020) since extracting topics from the questions is much more well studied. further, we employ a trained conditional random field (crf) model to tag subjects and aspects (fig. 1 ) from generated questions originated from each segments @xcite . once we got each segment's topic list, we proposed a strategy to normalize them to reduce the number of options in the questionnaire. recently, deutsch et al. (2020) demonstrated that qa-based evaluation is strongly correlated with human opinion. thus, to evaluate preme, we employ a similar qa-based strategy. to summarize, the main contributions are: c1 we propose preme, a novel framework to enable meetings exploration based on user's preferences through an interactive questionnaire; c2 we propose a new method for subject normalization which returns the most informative subject from a set of phrases and keywords; c3 we introduce a new automatic evaluation strategy for measuring the effectiveness of the proposed questionnaire to assess the required properties p1 and p2, which according to (deutsch et al., 2020) has a strong correlation with human judgments; and c4 we open-source a dataset that includes 1000 questions comprehensively annotated with subject to their subjects and aspects at https: //github.com/microsoft/preme 2 related work 2.1 automatic textual summarization automatic text summarization task has attracted lots of attention across natural language processing (nlp) community recently. many systems are proposed to summarize documents in different domains, including news (rush et al., 2015; @xcite @xcite @xcite , academic papers @xcite and books @xcite . meeting summarization has also emerged as a widespread need recently. due to the unique discourse structure of dialogues, conventional document summarization systems are facing challenges when summarizing meetings @xcite . thus, new models are proposed for tackling this task. wang and cardie (2013) employ decisions, and action items in dialogues to generate the summary progressively. oya et al. (2014) propose a template-based meeting summarization system by learning the relationship between summaries and their source meeting transcripts. shang et al. (2018) design an unsupervised meeting summarization model with multi-sentence compression techniques. li et al. (2019) introduce multi-modal information into meeting summarization with a hierarchical attention mechanism. zhu et al. (2020) propose a hierarchical meeting summarizer that can process both word-level and turn-level information of dialogues. furthermore, the community noted that due to the lengthy content and distributed information, a general summary of the meetings does not necessarily satisfy what users seek. thus, query-based summarization methods have become more prevailing for generating concise and specific summaries. @xcite @xcite @xcite @xcite . recently, @xcite proposed a new framework of query-based summarization for meetings, in which they annotate qmsum, a query-based multi-domain meeting dataset. each qmsum meetings come along with a set of queries with different levels of abstractness, i.e., general queries and specific queries. human annotators write these queries, and the summaries align with these queries after reading the meeting transcripts. while query-based summarization can be a proper path to provide users with meeting information at different specificity levels, we argue that issuing such specific queries still requires a certain degree of background knowledge. in reallife scenarios, users might not be equipped with that knowledge and issue informative queries, especially when they did not attend the meeting. hence, they can not benefit from query-based summarization techniques to explore the meetings. we address the drawbacks of query-based summarizers by providing users with an interactive questionnaire which provides them with potential queries and allows them to explore the meetings more flexibly. ## proposed framework: preme this section explains preme, our proposed novel methodology to explore meetings based on users' preferences through an interactive questionnaire. an overview of our methodology is shown in fig. 2 in which we first apply a topic segmentation method @xcite on meeting transcript to retrieve segments with different topics (section 3.1). then, we generate a set of all possible questions from each segment (section 3.2). further, we extract the most informative part of the questions, i.e., the subject and aspect of each question (section 3.3). in the last step, we map the normalized subjects and aspects with generated questions and form the questionnaire (section 3.4). ## evaluation methodology for experiments, we use the qmsum dataset @xcite , which includes 232 product, academic, and committee meetings @xcite . each meeting comes with a set of general and specific questions; the general ones are out of the scope of this work since they refer to very broad concepts, e.g., "summarize the whole meeting.". further evaluations are conducted on the qmsum test set. ## conclusions and future work we proposed an end-to-end framework, called preme, that allows automatically building a questionnaire that will enable users to explore the most of discussed subjects and their aspects if desired. as a result, users are supplied with questions about the meetings that express their information needs, and answers can be found in the transcript. since simulating actual users' preferences is challenging and requires hired annotators, we have ran a small user study as well running an automatic end-to-end evaluation strategy to demonstrate the desired properties (p1 and p2) of the generated questionnaires. we publicly release the collected dataset of annotated questions concerning its subjects and aspects, the code for questionnaires generation, and our evaluation procedure to carry forward the proposed state-of-the-art for the newly formulated problem. in future, and by proposing a new method for questionnaire generation will allow us to run a user study for pair-wise comparison of the methods and reveal the correlation between human and automatic evaluation metrics for the suggested task.