S tream H over: Livestream Transcript Summarization and Annotation (2021)

Abstract: With the explosive growth of livestream broadcasting, there is an urgent need for new summarization technology that enables us to create a preview of streamed content and tap into this wealth of knowledge. However, the problem is nontrivial due to the informal nature of spoken language. Further, there has been a shortage of annotated datasets that are necessary for transcript summarization. In this paper, we present StreamHover, a framework for annotating and summarizing livestream transcripts. With a total of over 500 hours of videos annotated with both extractive and abstractive summaries, our benchmark dataset is significantly larger than currently existing annotated corpora. We explore a neural extractive summarization model that leverages vector-quantized variational autoencoder to learn latent vector representations of spoken utterances and identify salient utterances from the transcripts to form summaries. We show that our model generalizes better and improves performance over strong baselines. The results of this study provide an avenue for future research to improve summarization solutions for efficient browsing of livestreams.

## introduction one of the most powerful communication mediums is livestreaming. new platforms such as youtube live, twitch, instagram live and tiktok encompass a variety of topics, ranging from video games to social media to professional sports. we are particularly interested in livestreams that are distinguished by three characteristics: excessive length, the recordings could last from several minutes to several hours; verbal communication, the use of natural language is the primary means of communication, in contrast to gestures or facial expressions; informal nature, the streamers' language is mostly informal and unplanned, unlike news broadcasts. without an effective mechanism to summarize such streamed content, livestreaming platforms may not fully meet the needs of their customers. figure 1 : an example of streamed content on bēhance, a streaming platform for artists and designers to showcase creative work related to adobe photoshop, illustrator, fresco, ui/ux, photography and more. top: the videos are each 27 minutes long. bottom: one video is being broadcast live, the other @xmath0. our goal in this work is to create a text preview of the streamed content. when a user hovers over the thumbnail or scrolls past a video, they are shown a preview of the content. we present a dataset of over 500 hours of video footage, which were streamed live on a social media platform (behance.net) created to showcase and discover creative work. figure 1 shows an example of the streams, where the artists showcase the use of adobe photoshop and illustrator in designing holiday cards and posters. it is necessary to point out that video analysis is not suitable here, as the video only mirrors the artists' screen content. as a first step towards automatic creation of a text preview, we focus on identifying salient utterances to produce an extract from the livestream transcript. we make use of vector-quantized variational autoencoders (vq-vae; @xcite to identify salient utterances. the model has been applied successfully to opinion summarization that learns in-domain sentence representations @xcite , which is essential for adaptation of general-domain models. we refrain from using sequential methods for utterance selection. first, it is difficult to scale up sequential prediction to process transcripts that exceed the maximum allowed length, even with models that handle long text @xcite . second, sequential methods @xcite may not give enough flexibility to select salient utterances on-the-fly when content is being streamed live, thus they are unsuitable for our case. there has been a shortage of annotated datasets that are necessary for livestream transcript summarization. we build a browser-based user interface for summary annotation that provides to the annotators a clip of the livestream recording alongside a synchronized display of the transcript. the interface allows annotators to conveniently label summary utterances and write an abstractive summary using their own words (figure 3 ). with a total of 500 hours of annotated video footage, our dataset is notably larger than existing annotated corpora for transcript summarization @xcite . we compare our summarization approach with strong baselines on the dataset and shed light on the task of livestream transcript summarization. our contributions are as follows. ## related work closed captions are often provided onscreen, turning streaming videos into text on an unprecedented scale @xcite . however, there are very few summarization studies that attempt to generate text previews of streaming videos to help users browse or refind information that has been watched before. neural text summarizers have focused primarily on written text, including news articles, reviews, scientific papers and book chapters @xcite ; tan 1 our annotations and source code are available at https: //github.com/ucfnlp/streamhover et @xcite @xcite @xcite @xcite @xcite . despite their success, it remains unclear as to if and how the summarizers can be extended to spoken text, whose utterances may have very low information density. it is crucial to identify salient content from transcripts where a substantial number of utterances are devoted to informal chit-chats in an attempt to connect with the audience (figure 2 ). we investigate extractive rather than abstractive approaches as the latter are prone to generate hallucinated content that does not exist in the source text @xcite @xcite . the problem could be exacerbated by ungrammatical spoken utterances and transcription errors. instead, we consider vq-vae, an unsupervised representation learning technique @xcite @xcite for content extraction. unsupervised training of the vq-vae model and its inference could potentially be performed at the same time, allowing important utterances to be extracted from a transcript segment on-the-fly during streaming, without interrupting the learning process. it is also easier to tailor the model to specific domains compared to contemporary extractive methods @xcite @xcite . our work contributes to a refined understanding of transcript summarization, which is understudied relative to its importance and potential. the transcripts may be obtained from channels such as movies and tvs @xcite , interviews @xcite , multiparty meetings @xcite @xcite @xcite , telephone speech @xcite and more. the main thrust distinguishing our work with others is the combination of a benchmark summarization dataset, novel summarization methods and a challenging new domain where salient content is scattered throughout the transcript and mixed with substantial chit-chats. we do not make use of video event detection or multi-modal fusion @xcite @xcite as little information could be gleaned from videos that mirror the artists' desktop. instead, we focus on generating short descriptions from transcripts and leave for future work crossmodality research. we describe our data annotation process in the following section. ## our dataset we aim to create a large and representative corpus containing transcripts and summaries of streamed videos. we explore a leading social media platform (behance.net) supported by adobe creative cloud that features livestreams of creative work by artists and designers. the website boasts over 10 million users, who watch artists and designers livestream when they create. our data are extracted from this website, containing a large quantity of streamed videos (>5,000), the length of which ranges from minutes to several hours. the streamers' language is unplanned, instead of rehearsed as that of ted talks @xcite . we obtain a total of 5,398 streamed videos. the metadata of a video includes its id, duration, title, a short description and the transcript. automatic transcription was provided by microsoft automatic speech recognition which helps make videos accessible to a wide audience. each transcript contains a set of segments, each corresponds to about 30 seconds of audio. each segment contains a set of utterances. foot_0 figure 2 shows an example of the segments and utterances. the offset of the segment indicates the number of minutes since the beginning of the recording. when a user hovers over the thumbnail or scrolls past a video, we expect a textual summary to give a glimpse of the verbal content. this view of summarization leads us to annotate salient content across the video in an equally detailed manner. it naturally avoids lead bias that is ubiquitous in news @xcite . we segment a video into 5-minute clips and annotate each clip for summary-worthy content. a clip contains an average of 51 utterances and 460 words. due to time and budget constraints, we select 370 streamed video for summary annotation. 3 dataset type duration extr. abst. switchboard telephone 300 hrs no † no icsi meeting 75 hrs yes yes ami meeting 100 hrs yes yes streamhover livestream 500 hrs yes yes we recruit 12 workers from upwork.com and validate their language skills for summary annotation. upwork is a freelancing platform that allows us to reach out to workers directly to ensure our instructions are fully understood. each worker is asked to write a concise summary for a given clip using their own words (task a) and identify summary utterances (task b) using the graphical user interface (figure 3 ), which shows a clip of the streamed video alongside a synchronized display of the transcript. additionally, our guidelines suggest a good summary of task a should have at least 100 characters and that of task b should have between 50 and 80 words (∼15% compression). as is the case with meeting summaries @xcite , a clip is annotated by a single worker owing to an expensive annotation process. the worker can also identify a clip to be chitchat, in which case it will not be annotated for summaries. the bēhance corpus total number of annotated videos 370 total annotated duration in hours 500 total number of utterances 331,928 average number of utterances in a clip 61.23 average duration of utterances in seconds 3.04 average number of words in an utterance 9.48 total number of annotated clips 6,003 total number of chitchat clips 582 total number of human annotators 12 avg. # of (sentences) words in a human abstract (3.0) 36 avg. # of (utterances) words in a human extract (5.5) 80 percentage of duration of summary utterances 8.57% ## summarization let x denote a sequence of spoken utterances from a segment of the transcript. our summarizer aims to extract a subset of utterances y ⊂ x that convey the essential content of the input. we experiment with an unsupervised summarizer that leverages vector-quantized variational autoencoders (vq-vae; van den @xcite to learn utterance representations and identifies summary utterances. the method was explored for opinion summarization @xcite and machine translation @xcite . we are interested in using the method to account for domain characteristics of livestreams, which showcase new and creative work of artists and designers on their use of photoshop, illustrator, and other tools. foot_3 vq-vae is a powerful framework for learning latent variable models using deep neural networks. it learns discrete vector representations for an utterance, which is then used to categorize the utterance along various dimensions. e.g., "good morning hi everybody" suggests a greeting and opens up a dialogue; "i had probably 3 or 4 different customers on youtube and ... on facebook asked me how the heck do you burn an audio cd in adobe audition" engages the audience and introduces the input text output text embed generate convnet encoder convnet decoder code embeddings straight-through estimator @xmath1 figure 4: our summarizer embeds an input utterance using bert, transforms bert's semantic space to a set of latent codes, then reconstructs the utterance using the code embeddings. we identify summary utterances as those associated with prominent latent codes/topics. the model is trained using a dictionary learning algorithm for code embeddings (e) and backpropagation with a straight-through estimator for model parameters θ, ϕ and φ. we employ an embedding function embed θ (•) to map an input utterance x into a semantically meaningful space. the space is subsequently discretized according to a codebook. to achieve this, we prefix x with a [cls] token and append a @xcite token, pass it into a bert model, then obtain the vector corresponding to @xcite as a pooled representation of the utterance, denoted by @xmath2. ( 1 )). we use a convencoder ϕ (•) with a set of d filters to convolve the input h. the output is a sequence of feature vectors [q 1 , • • • , q h ] where q @xmath3. ( 2 )). we define a codebook @xmath4, • • • , e k ], where k is the number of latent codes and e @xmath5. the i-th feature q i is assigned to the latent code z i whose embedding e z i has the minimum euclidean distance with it (eq. ( 3 )). our method essentially discretizes the h-dimensional semantic space by producing latent codes {z i } h @xmath6, one for each dimension of the semantic space. with the latent code embeddings [e z 1 , • • • , e z h ], we seek to reconstruct the input utterance, which is achieved by generating a dense vector h using a convdecoder ϕ (•) (eq. ( 4 )). h is then fed to a transformer decoder to reconstruct the original utterance x (eq. ( 5 )). in this process, the code embeddings serve as "topic vectors" that group dimensions of the semantic space into clusters relevant to the application domain. our model parameters include those used by the bert encoder and transformer decoder (θ and φ), the convolutional encoder and decoder that use tied parameters (ϕ), and embeddings of the codebook e. we next describe the loss function used to learn these parameters. the loss function of our model comprises of three parts, including a cross-entropy term between the original and reconstructed utterance xent(x, x) that optimizes the bert embedder θ, transformer generator φ, and convolutional encoder and decoder ϕ, as shown in figure 4 . the gradients will, however, bypass the latent code embeddings due to the straight-through estimator @xcite . to learn code embeddings in an end-to-end manner, we use a dictionary learning algorithm @xcite that moves code embeddings e z i towards feature vectors q i by minimizing the l 2 -distance between the two vectors e z i -sg(q i ) 2 2 , where sg(•) is a stop-gradient operator that constrains its operand to be a non-updated constant during backpropagation, i.e., it stops q i from being updated. as illustrated in eq. ( 6 ), we additionally apply a commitment loss to encourage the feature vector q i to commit to a code embedding. sg(e z i )q i 2 2 prevents q i from deviating too much from the code embedding e z i . this loss term is associated with a coefficient β ∈ at test time, we define summary utterances as those associated with prominent latent codes/topics. given a set of n utterances, we obtain latent codes from the n-th utterance using eq. ( 3 ), denoted by {z our method draws on the convolutional encoder and decoder to transform bert's semantic space to map each dimension to a latent code. the summary selection process is deterministic and our encoder takes full advantage of a large, pre-trained model to produce initial utterance representations. this design sets our method apart from that of @xcite . moreover, the method has the potential for modelling topic transitions between utterances to improve summarization of livestreams, which we leave for future work. we call our summarizer "streamhover." when a user hovers their mouse over a video's timeline, a summary preview is shown and keeps updating. as a first attempt, streamhover focuses on extracting salient utterances from individual clips instead of whole streams to encourage selected utterances to be mostly evenly distributed across the stream. when the content is provided live, the stream can be divided into short clips and our algorithm consumes one clip at a time to produce summaries on-the-fly. it is important to note that extracting summary utterances remains challenging even for modern neural summarizers. e.g., @xcite reveal that summarizers may not effectively identify summary content without a dependency on intentional lead bias in news writing. our setting is challenging as not only are there few utterances deemed to be summary-worthy but such utterances can occur anywhere in a video clip. baselines. we compare streamhover with stateof-the-art extractive and abstractive summarizers. the abstractive summarizers generate an abstract from the transcript of a clip without tuning. 6 these include bart-large, bart-large-cnn @xcite and t5 @xcite , which are some of the strongest performing neural abstractive summarizers that are pre-trained on language modeling and summarization tasks. the unsupervised extractive summarizers extract salient utterances from a clip. lexrank @xcite and textrank @xcite are graph-based models that extract relevant sentences based on eigenvector centrality. sum-basic @xcite ) assigns higher scores to sentences containing frequently occurring content words. we further compare to a novel unsupervised graph-based summarization method for speech transcripts: flucovrank @xcite groups utterances into clusters, generates an abstractive sentence from each cluster, then selects the best elements from abstractive sentences under a budget constraint. finally, we compare our approach with the quantized transformer @xcite , which uses a clustering interpretation of the quantized space and two-step sampling algorithm to extract summary sentences from reviews. settings. we use pretrained bert-base as our embedder embed θ (•). the model has 12 layers, 12 heads per layer and a hidden size (h) of 768. a 6layer transformer decoder is used as the generator generate φ (•) to reconstruct the original utterance. the model has 8 heads per layer, a hidden size of 768, and randomly initialized parameters. the convolutional encoder and decoder use a kernel size of 3. because our embedder is pretrained and the remaining parameters are not, we divide them into two groups @xmath7, ϕ}, then apply separate training schedules. following @xcite we use two adam optimizers: all of our models are trained for 30 epochs on dual nvidia v100 gpus with gradient accumulation every ten steps. we experiment with different numbers of filters, @xmath8, 100, 128}, for the convolutional encoder and decoder. the number of latent codes are varied in @xmath9, 1024, 2048}. the coefficient β used for commitment loss is set to 0.25 (eq. ( 6 )). these hyperparameters are tuned on the validation set. we keep only utterances that @xmath10. the final training set contains 168,111 utterances. ## conclusion we present streamhover, a new framework for annotating and summarizing livestream transcripts. our dataset contains over 500 hours of videos annotated with extractive and abstractive summaries. we explored an extractive method leveraging vq-vae to identify salient summary utterances and obtained strong results. future work includes boosting summarization solutions to provide users a concentrated overview of streamed content.