Keyphrase Prediction from Video Transcripts: New Dataset and Directions (2022)

Abstract: Keyphrase Prediction (KP) is an established NLP task, aiming to yield representative phrases to summarize the main content of a given document. Despite major progress in recent years, existing works on KP have mainly focused on formal texts such as scientific papers or weblogs. The challenges of KP in informal-text domains are not yet fully studied. To this end, this work studies new challenges of KP in transcripts of videos, an understudied domain for KP that involves informal texts and non-cohesive presentation styles. A bottleneck for KP research in this domain involves the lack of high-quality and large-scale annotated data that hinders the development of advanced KP models. To address this issue, we introduce a large-scale manually-annotated KP dataset in the domain of live-stream video transcripts obtained by automatic speech recognition tools. Concretely, transcripts of 500+ hours of videos streamed on the behance.net platform are manually labeled with important keyphrases. Our analysis of the dataset reveals the challenging nature of KP in transcripts. Moreover, for the first time in KP, we demonstrate the idea of improving KP for long documents (i.e., transcripts) by feeding models with paragraph-level keyphrases, i.e., hierarchical extraction. To foster future research, we will publicly release the dataset and code.

## introduction keyphrases are one or multiple consecutive words that could represent the main ideas in a document. keyphrases are commonly categorized as present or absent. a present keyphrase explicitly appears in the document, while an absent keyphrase does not exist in the document. keyphrases can serve as concise summary for a document, hence benefiting various nlp applications information retrieval @xcite and text summarization @xcite . due to their usefulness, in the more than two decades, kp has been studied in many re-search works @xcite @xcite @xcite @xcite . whereas traditionally feature engineering has been used for kp @xcite , recently deep learning is proved to be more efficient for this task @xcite . however, one limitation in the current works is that they are mainly limited to the formal text such as scientific papers @xcite and web-logs @xcite . as such, the challenges in other domains are still unresolved. among others, video transcript is one of the less-explored domains that could significantly benefit from kp. for instance, it could be used for video summarization and retrieval or benefit people who are deaf and hard of hearing (dhh) @xcite . on the other hand, kp for transcripts that are automatically obtained are more challenging than the formal written documents as these transcripts involve noisy text, incomplete/repeated sentences and phrases, informal vocabulary, and noncohesive information flow. although there have been a few related attempts to evaluate feature engineering methods on meeting transcripts @xcite , the available resources, with a handful of transcripts and keyphrases, are not useful to train/evaluate the recent advanced deep models. to address such limitations, we propose a large manually-labeled dataset for the domain of video transcripts. specifically, we collect 500+ livestream videos from the behance platform. the videos are automatically transcribed by microsoft automatic speech recognition (asr) tool. since the video transcripts might be lengthy, summarizing the entire transcript into a few keyphrases might be challenging. moreover, such keyphrases might not be helpful for partial retrieval where a part of the transcript is requested. as such, we annotate the collected transcripts in two levels: (1) para-graphs: a paragraph, consisting of multiple consecutive sentences, is a chunk of a transcript that provides a single point. annotators first identify the paragraphs in a transcript. next, the representative keyphrases for every paragraph are annotated; (2) chapter: in addition to the paragraphs, we ask the annotators to provide a few keyphrases that could summarize multiple consecutive paragraphs that convey a single topic (e.g., how to make a special edit on an image). we call these units "chapter", which are comparable to documents in other kp datasets. annotators will first find the boundaries for chapters, then provide the keyphrases for each chapter. we conduct extensive analysis on both levels of the kp task on the prepared dataset. our analysis shows that kp in transcripts is a challenging task and more research is required. more importantly, for the first time for kp, we show that extracting keyphrases of long documents in a hierarchical order could result in better performance on document level kp. specifically, our analysis shows that obtaining paragraph-level keyphrases and providing them to chapter-level kp systems could significantly boost the performance. the provided dataset and analysis could bring forth opportunities for more research on transcripts for kp. ## data annotation data collection: this work aims to annotate kp data for the domain of asr text. to this end, we employ live-stream videos released on the social media platform behance.net. the videos are streamed by artists and designers to share/discuss their creative projects. as such, verbal content from the speakers (in english) is important for video understanding. while the videos have initial subjects, their content is unplanned, hence the streamer might cut sentences, discuss multiple topics, and employ informal phrases. the videos have an average length of 48 minutes. to obtain the verbal content of the streamed videos, we employ the microsoft asr tool. in total, 361 videos with a total length of more than 500 hours are transcribed. a transcript, on average, contains 7,219 words. annotation: as presented in the introduction, the lengthy nature of transcripts motivates us to annotate keyphrases at two levels. first, at the paragraph level, we define a paragraph in a transcript to have the same role as paragraphs in formal written documents. concretely, a paragraph is defined as a chunk of text that conveys a particular point or idea. a transcript consists of multiple disjoint paragraphs. since the asr text does not provide paragraph information, we manually annotate the collected transcripts with paragraphs. afterward, for each paragraph of the transcript, the important keyphrases are selected. to this end, a keyphrase for a paragraph should have the following features: to annotate data for each level, we hire 10 annotators from the upwork.com platform which is a website for hiring freelancers with different expertise. since the collected videos are related to photo editing software, e.g., photoshop, we require the annotators to have experience both in data annotation and in using major photo editing tools. we train the annotators for kp at each level. to prevent chapter-level keyphrases to be biased toward paragraph-level keyphrases, we split annotator pool for paragraph and chapter level annotation (five for each). the transcripts are distributed evenly to the five annotators at each level for annotation. as such, a transcript is annotated entirely by a paragraph annotator and a chapter annotator (including boundary annotation). chapter annotation is done after and uses outputs from paragraph annotation. annotation agreement: following prior work @xcite , we assess the task difficulty of kp over video transcripts by evaluating the agreements of annotators at different levels. for each annotation level, we ask all the five annotators to independently annotate a sample of 5% of the tran ## experiments we randomly split the 361 transcripts into train/development/test sets with the ratio 80/10/10, respectively. the paragraphs and chapters of the transcripts in each split are then employed for our experiments in this section. specifically, we first assess the challenges of kp at each level. next, we empirically study how paragraph-level information can be helpful for chapter-level kp. baselines: we evaluate the performance of the following baselines on the proposed dataset: (1) generative language models: the content of a paragraph or chapter are prompted to a generative language model (lm) to produce the keyphrases. specifically, the language models are trained in an auto-regressive manner on sequence @xmath0, . . . , w n , [sep ], kp 1 , . . . , kp m ], where w i is the i-th word in the input paragraph or chapter and kp i is the i-th keyphrase. we employ gpt-2 (radford et al., 2019), bart @xcite and t5 @xcite as three different versions of this baseline foot_0 . note that for the chapters, since the transformer-based lms impose a length limit, we truncate the input to the length of the maximum size of the lms; (2) seg-net @xcite : in this baseline, salient sentences in the input text are first selected, then keyphrases are predicted by a generative model consuming the selected salient sentences. to select important sentences, a binary classifier is trained to distinguish sentences that contain a present keyphrase or partially overlap with an absent keyphrase; and (3) one2set @xcite : the prediction of keyphrases is modeled as a set prediction task. instead of imposing an order on the output of a transformer-based decoder, the model predicts keyphrases in parallel. we evaluate the models based on the macro-averaged f1@3 and f1@m. in the former, the predictions are truncated/padded at cutoff 3 while in the latter all model model paragraph chapter p@3 r@3 f1@3 p@m r@m f1@m p@3 r@3 f1@3 p@m r@m f1@m one2set 39.54 31. ## related works keyphrase prediction (kp) has been studied extensively in the past @xcite @xcite @xcite . prior works can be categorized into extraction-based and generation-based solutions. in the former, keyphrases are extracted from input text, using either rule-based methods @xcite or deep learning models @xcite via sequence labeling @xcite . in the generation-based models, deep generative models are employed to encode input documents and generates keyphrases @xcite @xcite . however, existing works on kp are mostly trained and evaluated on formal text. to this end, our work introduces a large-scale hierarchical kp dataset for video transcripts with informal and non-cohesive texts. we also note some related attempts to evaluate kp systems on meeting transcripts @xcite ; however, the small size of these datasets hinders their relevance to deep learning era. ## conclusion we present a novel hierarchical kp dataset over live-stream video transcripts. the dataset contains transcripts of 361 videos that are annotated at both paragraph and chapter levels. our experiments show that kp in video transcripts is challenging and hierarchical extraction is helpful for kp in long documents. in the future, we will include more tasks in our dataset for video transcripts.