How Domain Terminology Affects Meeting Summarization Performance (2020)

Abstract: Meetings are essential to modern organizations. Numerous meetings are held and recorded daily, more than can ever be comprehended. A meeting summarization system that identifies salient utterances from the transcripts to automatically generate meeting minutes can help. It empowers users to rapidly search and sift through large meeting collections. To date, the impact of domain terminology on the performance of meeting summarization remains understudied, despite that meetings are rich with domain knowledge. In this paper, we create gold-standard annotations for domain terminology on a sizable meeting corpus; they are known as jargon terms. We then analyze the performance of a meeting summarization system with and without jargon terms. Our findings reveal that domain terminology can have a substantial impact on summarization performance. We publicly release all domain terminology to advance research in meeting summarization.

## introduction a vast number of meetings are being held and recorded everyday, far more than can ever be comprehended. with this explosion of meetings comes a pressing need to develop summarization techniques to assist in browsing meeting archives @xcite . a meeting summarization system takes a meeting recording and its transcript as input and produces a concise text summary as output, which preserves the most important content of the meeting discussion @xcite @xcite . the techniques hold great potential to make large archives of meetings substantially more efficient to browse, search and facilitate information sharing. we envision an automated summarizer that is capable of generating meeting minutes by identifying salient utterances from transcribed meeting recordings. neural text summarization has seen significant progress @xcite @xcite @xcite @xcite , but most prior work focused on written texts. in contrast, recent years have seen a growing interest in summarizing spoken texts @xcite . particularly, the characteristics of meetings, domain terminology and limited annotated data pose novel challenges to neural summarization models. we favor extractive over abstractive models as the latter are prone to hallucinate content that is unfaithful to the input @xcite . in this paper, we investigate how domain terminology impacts meeting summarization performance, especially in the context of neural extractive summarization. jargon is the specialized terminology associated with a particular domain @xcite . it is employed in a communicative context and may not be well understood outside that context. because meetings are usually held among professionals, jargon is ubiquitous in meeting discussions. in we present an assessment of the meeting summarization performance by comparing models trained with and without jargon. a collection of jargon terms are meticulously compiled by our expert annotators from start end spoken utterance 247.255 252.672 with andreas' help um andreas put together a sort of no frills recognizer which is uh 252.672 258.837 gender-dependent but like no adaptation, no cross-word models, no trigramsa bigram recognizer 258.837 262.221 and that's trained on switchboard which is telephone conversations. 263.983 267.154 and thanks to don's help wh-who -don took 267.154 270.431 the first meeting that jane had transcribed 270.431 277.520 and um you know separated -used the individual channels we segmented it in-into a meeting corpus containing multi-party conversations on the topic of speech and signal processing @xcite . such jargon terms are distinct from speech recognition errors; the latter substitutes one word for another similar-sounding word during automatic transcription. the users can eliminate transcription errors using a modern interactive transcript editor. however, there remains a pressing need to understand how domain terminology affects the meeting summarization performance. our contributions are twofold. first, we create gold-standard annotations for domain terminology on a large meeting corpus; they are known as jargon terms. prior work has not explored such domain-specific thesauri and thus there is limited knowledge of the target domain. second, we analyze the performance of a meeting summarization system with and without jargon. due to the nature of sound, such a summarizer is highly desirable to aid users in navigating through meeting recordings. our findings suggest that domain terminology has a substantial impact on summarization performance, which should not be overlooked. ## data and annotation we extend the icsi meeting corpus @xcite for this study, which contains 75 meetings recorded at the international computer science institute, berkeley. 2 the meetings are primarily between speech group members of icsi. an average meeting lasts an hour and has up to 10 participants. each participant wore a close-talking microphone and they sat around a meeting table equipped with far-field microphones. the corpus is one of the larger resources in this area @xcite . it contains rich annotations including human transcripts, segmentation of utterances and further annotations of extractive summaries foot_1 , making the corpus suitable for summarization. we have chosen icsi over the ami corpus @xcite ; both are natural conversations, but the scenarios in ami meetings are artificial. annotating domain terminology is non-trivial as there lacks a universal definition. instead, we solicit annotations from undergraduate students majoring in computer science and designate words and expressions that are beyond the scope of their knowledge as domain terminology. interestingly, modern deep neural models often acquire such generic knowledge through unsupervised pretraining @xcite . the annotators are instructed to identify words and expressions from human transcripts; they are called jargon terms and usually have particular meaning in the speech and language processing field. ## meeting summarization jargon term tokenization smartkom system smart-ko-m system discourse annotations discourse ann-ota-tions situational context factors situation-al context factors modifiers, auxiliaries mod-ifiers , aux-ilia-ries javabayes belief-net java-bay-es belief -net a real wizard system a real wizard system the l_d_c the l _ d _ c the near field mikes the near field mike-s utterance with jargon she wanted to display the stylized f_ zeroes, i think they're called? utterance without jargon she wanted to display the [mask] i think they're called? the very first step that one must take to build a meeting summarizer is tokenization, which transforms an input utterance to a sequence of sub-word units. wordpiece @xcite and bpe @xcite are two modern methods for tokenization. we use wordpiece that has a total vocabulary of 30,522 sub-words. the method builds a vocabulary of the desired size by iteratively combining word parts into a sub-word if doing so increases the language model likelihoods. given the vocabulary and any input word, it uses a greedy longest-match-first algorithm to tokenize it into sub-word units; the longest sub-word will be matched first. we provide example tokenization outputs in our meeting summarizer takes as input an utterance and outputs a binary label indicating if the utterance should be included in the summary. due to data scarcity, we refrain from using sequential prediction or a more sophisticated approach that may overfit, but focus primarily on demonstrating the impact of domain terminology on model performance. our summarizer is based on bert-large that contains 24 layers of transformer blocks, 16 attention heads and 1024-dimensional hidden vectors @xcite . the top-layer hidden vector of the [cls] token is used as the representation of the input utterance. we apply a linear and a softmax layer to predict a binary label. importantly, jargon terms can be masked-out of the input utterances by replacing each term with [mask] token prior to training ( ## results and analysis our experimental results are presented in we observe that summarizing with jargon terms yields substantially better performance (an absolute gain of +4.3% r-2 f-score) on human transcripts, comparing to the alternative that masks jargon out of input utterances. the performance gap has narrowed on asr transcripts, as domain terminology contains infrequent entities and terms, which are subject to transcription errors. our findings suggest that domain terminology plays a significant role in determining utterance salience. its impact on summarization and other downstream meeting applications should not be underestimated. in ## related work generating meeting summaries is a challenging problem with a great application potential. a significant number of techniques have been attempted in the past, including extraction of utterances and keyphrases from transcripts @xcite @xcite and taking advantage of prosodic and speaker-related features @xcite @xcite . as spoken utterances are verbose with low information density, some methods further compress and merge utterances @xcite @xcite . despite these valuable contributions, a closer investigation remains necessary to develop an understanding of how domain terminology affects meeting summarization performance. recent years have seen a renewed interest in summarizing meeting transcripts @xcite @xcite and other types of online and transcribed conversations @xcite @xcite . in particular, @xcite create a corpus containing 22 public meetings including their automatic transcriptions from audio recordings and meeting reports written by a professional. @xcite develop a multi-modal hierarchical attention mechanism for abstractive summarization, where attention is applied to topics, utterances and words to narrow the focus to salient content; their experiments were performed the ami corpus, thus results are not directly comparable. our work excludes prosodic and speaker-related features to focus solely on domain terminology. it provides a new baseline for future research toward building effective meeting summarizers. ## conclusion we seek to better understand how domain terminology impacts meeting summarization performance in the context of neural extractive summarization. we solicit quality annotations from expert annotators to compile a list of jargon terms from a sizable meeting corpus, which is a valuable resource to investigate a variety of research questions regarding domain adaptation. our extensive experiments show that domain terminology has a substantial impact on summarization performance that should not be neglected. future work may address the questions of how to obtain domain terminology in a semi-automatic way and inject domain knowledge into a meeting summarization system.