Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization? (2024)

Abstract: Reference summaries for abstractive speech summarization require human annotation, which can be performed by listening to an audio recording or by reading textual transcripts of the recording. In this paper, we examine whether summaries based on annotators listening to the recordings differ from those based on annotators reading transcripts. Using existing intrinsic evaluation based on human evaluation, automatic metrics, LLM-based evaluation, and a retrieval-based reference-free method, we find that summaries are indeed different based on the source modality, and that speech-based summaries are more factually consistent and information-selective than transcript-based summaries. Transcript-based summaries are impacted by recognition errors in the source, and expert-written summaries are more informative and reliable. We make all the collected data and analysis code public to facilitate the reproduction of our work and advance research in this area.

## introduction speech summarization @xcite is the task of analyzing long speech recordings and generating a short textual summary that concisely conveys this information. abstractive speech summarization @xcite aims to produce summaries that a human would write @xcite . prior work in abstractive speech summarization @xcite @xcite has focused on developing automatic methods. however, to the best of our knowledge, there has been little work examining how humans perform speech summarization. such research is particularly important because the goal of abstractive summarization is to produce human-like summaries, and having high-quality human summaries is crucial for learning how to automatically summarize, i.e., for model training and evaluation. also, information about how humans summarize may facilitate improved automatic approaches, and hence it is important to explore the annotation process. in this paper, we delve deeper into the process of human annotation for abstractive speech summarization. fig. 1 presents an overview of our paper. humans can summarize speech when it is presented by (a) listening to the speech directly and then constructing the summary based on this, or (b) reading a textual transcript of the content spoken in the recording, and using that to construct the summary. it is unclear whether summaries based on listening to audio differ from summaries based on reading transcripts. for transcript-based summarization, the method used to obtain the transcription, either automatic speech recognition (asr) or manual transcription, may also have a significant effect. additionally, the quality of a summary, regardless of the source modality, is likely to be determined by the level of expertise of the summarizer. to assess the impact of source modality, transcription, and expertise, we formulate the following research questions: past work in automatic speech summarization suggests that there are acoustic characteristics in speech beyond the textual context that are useful for speech summarization @xcite . this implies that both transcript and raw speech carry complementary information that is useful for summarization. the fact that end-to-end models @xcite @xcite ) that use speech features as input perform differently from cascade models @xcite @xcite that use speech transcripts @xcite suggests that there are differences in automatic summaries resulting from textual or speech input. in this paper, we study whether the same holds for human annotators. however, any observations on the impact of reading versus listening for annotation are difficult to make from existing corpora like how2 @xcite , slue-ted @xcite , interview @xcite , and ami @xcite , since it is unclear how human annotations were obtained from speech. therefore, we select 1002 recordings from the interview corpus test set and create a new evaluation set. we collect two expert human annotations for transcript-based summarization and two for speech-based summarization per recording and assess how the resulting summaries differ. comparing two given summaries is challenging. broadly, we conduct three types of evaluation: (a) comparing the summary to its source transcript (source-based evaluation) to establish how extractive and factual the summary is, (b) evaluating the summary independently for its structure and grammar (structure evaluation), and (c) by comparing different summaries (summary comparisons) to as-sess relevance and which represents the input better using semantic similarity, llm, and human evaluation. experiments show that speech-based summaries are more information-selective and factually consistent while transcript-based summaries are preferred by human and llm evaluators. in real-world settings, human-annotated manual transcripts are not available, and, therefore, automatic speech recognition (asr) models may be used to generate such transcriptions. asr transcripts may contain errors, and we investigate the impact of these errors on transcript-based summarization. to do this, we consider 100 recordings from the interview test set, obtain speech transcriptions using whisper @xcite , and perform human summarization using 2 expert annotators for each modality. experiments show that asr errors in the transcript degrade the quality of human summaries, resulting in lower coherence, fluency, and factual consistency with the source. prior work in human evaluation of abstractive text summarization highlights the importance of expertise for summary evaluations @xcite . however, to the best of our knowledge, there is no prior work that examines the need for expertise in human speech summarization. therefore, in this work, we examine whether crowd-sourced non-expert summaries are different from expert summaries. to address this question, we obtain 2 non-expert annotations using amazon mechanical turk (amt) for each of the 1002 recordings from the interview test set. from these summaries, we analyze (a) differences between expert and non-expert annotations, and (b) whether any differences can be attributed to the approach taken for summary annotation. analysis shows that non-expert annotations are less fluent and coherent compared to expert summaries but as factually consistent as expert summaries. in summary, through this work, we find that transcript-based annotation is valuable if errors are minimal and longer, more informative summaries are desired, while speech-based annotation is desired for higher information selectivity, factual consistency, and resilience to transcription errors. ## related work for this study, we describe related work in three areas: (a) metrics for comparing summaries, (b) other work investigating the impact of source modality on human performance, and (c) investigations of asr error impact on spoken language understanding. comparing summaries -metrics: numerous metrics have been proposed to evaluate the quality of summarization @xcite . earlier works use metrics like rouge-h, rouge-l @xcite , rouge-we @xcite , bleu @xcite , sentence recall/precision is commonly used for simple sentence based extraction summaries @xcite . some studies have found that rouge and bleu-based metrics correlate well with human judgments on summarization. however such scores are not always the best metric for summarization, especially when the input is longer like in meeting summarization and summarization of scientific articles. there have been scores of other metrics which are either variants or combinations of the earlier mentioned metrics like bertscore @xcite , sentence mover's similarity (sms) @xcite , summaqa @xcite , su-pert @xcite , chrf @xcite , meteor (banerjee and lavie), cider @xcite , or model generated ones like s3 @xcite , blanc @xcite , summaccy @xcite , which are trained to predict the quality of an input summary. llms have been used as an alternative method for summary evaluation. while some work expresses doubt that llms have reached a human level of summary evaluation capabilities @xcite , other methods have yielded more optimistic results @xcite . llm evaluation methods usually take the form of automatic replacements for human evaluations such as likert scale scoring, pairwise comparison, and binary factuality comparison @xcite . llms can also be used to answer qualitative questions about the factual consistency of a summary and explain the reasoning behind the provided answers @xcite . source modality -transcript vs. speech: we are also interested in studying whether input modalities have an impact on task performance. cognitive psychology literature has studied this question under various settings. @xcite ) studies cognitive performance and information retention, when note-taking is performed using either voice or text. the paper finds that when the notes are taken using voice versus when they are written down by hand, voice leads to a higher conceptual understanding of the topic. @xcite studies how input modality effects learning of the english language, specifically whether learning from lecture audio vs reading lecture notes impacts retention. other papers examine the interaction between input modality (text vs speech) and what users prefer when performing a task. studies have reached various conclusions regarding the most useful/important modality based on the task at hand, what we do find is that the difference exists and is significant. impact of asr on downstream tasks: asr errors are known to have an impact on downstream task performance, like emotion recognition @xcite , spoken language understanding @xcite , and natural language understanding and dialog management @xcite . therefore, it is important to assess their impact on human summarization. ## data collection datasets that are used for speech summarization today do not have information on which of the two approaches, i.e., reading a textual transcript of the recording or listening to the recording to summarize was employed to obtain annotation. further, they only have one annotation per recording, which may inhibit analysis due to annotator-specific biases or tendencies. therefore, to conduct our experiments, we curate a multi-reference dataset from an existing dataset. we considered a subset from the test partition of the interview @xcite @xcite corpus for speech summarization. interview is the largest opendomain corpus with around 5-minute-long speech recordings containing spontaneous speech along with real audio noises representing music, and events inside and outside the studio corresponding to discussion topics 2 . speech recordings contain spontaneous speech along with real audio noises that represent music, and events inside and outside the studio corresponding to discussion topics. the dataset has 49.4k recordings that span multiple topics and domains, with an average recording length of 5.28 minutes. the transcription on average contains 886.4 words and the summary contains 40.1 words on average, representing a compression ratio of 22.09 between the transcript and summary. around 48% of words in the summary are not present in the transcription, making this a relatively challenging task. on average, each recording has 24 turns and 4 speakers. ensuring high-quality summary annotations is crucial to making reliable observations. therefore, we took the necessary steps to ensure a fair and reliable data collection pipeline. initially, we consulted with an expert from the in-house data collection team and conducted small pilot studies to make design decisions. audio length: we conducted in-house pilot studies comprising six questions by trying to summarize audio recordings that were 2 minutes and 5 minutes long. as expected, summarizing 5-minute recordings consumed more time and effort, and was harder due to the high compression ratio. we concluded that limiting the audio length to 2 minutes would provide sufficient content to summarize without increasing the difficulty level of summarization significantly. using forced alignment, we obtained audio reference transcripts up to the first 2 minutes. summary length: a hard limit was not set on the length of the summary, but it was suggested that summaries should contain at least 2 sentences to convey the essence of the conversation. it was recommended that summaries contain between 50 to 80 words for the 2-minute long recordings. data selection: to obtain reasonable summaries, it is important that audio recordings and textual transcripts are understandable, and have a sufficient amount of information to summarize. in cases where background noise or music is present, or when a large portion of the recording does not contain speech, these criteria are violated, lead2 the authors acquired permission from npr to use npr data for this research ing to sub-optimal recordings for summarization. therefore, audio recordings were transcribed using whisper-medium @xcite , and the resulting word error rate (wer) was used to remove unsuitable audio recordings from consideration. based on manual inspection, a threshold of 25% wer was set, which eliminated recordings with only music or large amounts of noise. expert and non-expert: our data comprises 1,002 recordings in total, with each recording having a speech recording and its corresponding reference transcript. for each recording, we collected two expert summaries and two non-expert summaries by reading the text or listening to the audio, which gave us a total of 4 summaries based on the text and 4 summaries based on the audio on the same recording. additionally, we solicited summary annotations from speech recognition transcripts for 100 of these 1002 recordings, sampled at random so the word error rate with whisper-medium is non-zero, i.e., the transcript has asr errors. annotator overlap: it is worth noting that each annotator was assigned to work on either the transcript or audio recording for a particular recording, but not both. this approach ensured that no annotator saw audio or transcription for a recording they had already annotated. expert summary annotation: expert summary annotations were obtained through a third-party vendor that passed in-house qualitative assessments of summary outputs from a pilot annotation. the third-party vendor had proven their quality of annotation by consistently delivering similar summary annotations for other projects of the team. detailed annotation guidelines are in appendix a. ## evaluating differences across summaries summary evaluation is an inherently challenging problem for general settings since there is often no clear definition of what constitutes a good summary. within this paper, we deem a summary good if it is factually faithful to and representative of the input source while being abstractive, coherent, and fluent. therefore, in this section, we outline three types of evaluations. ## conclusion in this paper, we take the first steps towards understanding human annotation for speech summarization by proposing and assessing different approaches. annotators can either read a transcript of spoken content or listen to the audio recording to produce an abstractive speech summary. we formulate three research questions to assess differences in summaries resulting from variations in source modality (listening to audio versus reading a transcript of spoken content), the nature of transcription (manual versus asr-based), and the expertise of annotators. extensive analysis using automatic and human evaluation reveals that speech-based sum-maries are more information-selective, factually consistent, and abstractive compared to transcriptbased summaries. analysis of the impact of asr on transcript-based summarization shows that errors in transcript generation decrease the informativeness and coherence of transcript-based summaries. finally, analysis of expert and non-expert annotations demonstrates higher informativeness, coherence, and fluency across expert annotations. our experiments with non-expert human annotation of transcripts showed that using platforms like amazon mechanical turk may result in less reliable human annotations owing to annotators' tendency to use generative models like chatgpt for annotations. further, our work reaffirms that errors in speech transcription not only impact automatic approaches for summarization but also human annotation. this demonstrates the need for reliable transcripts for speech summarization annotation. through this paper, we also develop and release a new human-annotated dataset to enable reproduction and encourage further research in the area. we hope that this study galvanizes further research into human summarization, and powers informed model design for automatic speech summarization.