B ehance PR : A Punctuation Restoration Dataset for Livestreaming Video Transcript (2022)

Abstract: Given the increasing number of livestreaming videos, automatic speech recognition and post-processing for livestreaming video transcripts are crucial for efficient data management as well as knowledge mining. A key step in this process is punctuation restoration which restores fundamental text structures such as phrase and sentence boundaries from the video transcripts. This work presents a new human-annotated corpus, called BehancePR, for punctuation restoration in livestreaming video transcripts. Our experiments on BehancePR demonstrate the challenges of punctuation restoration for this domain. Furthermore, we show that popular natural language processing toolkits like Stanford Stanza, Spacy, and Trankit underperform on detecting sentence boundary on non-punctuated transcripts of livestreaming videos. The dataset is publicly accessible at http://github.com/nlp-uoregon/behancepr .

## introduction livestreaming is a powerful broadcasting medium that catches the attention of millions of users. many video-sharing platforms have supported livestreaming for a wide range of topics such as twitch for gaming, tiktok for short entertainment videos, behance for visual creative work, and youtube/facebook live accepting any topics. among these videos, there are a substantially high number of videos that provide useful knowledge with exceptional visual demonstration. to this end, livestreaming videos are becoming a potential knowledge base waiting for being explored. mining videos on video/audio format directly is extremely hard and expensive because of their high data load and complexity in processing images and audio signals. instead, mining video transcripts, transcribed by either human or machine, is much easier with the existing hardware and software. as such, livestreaming videos should be transcribed at high quality to facilitate future data mining research. as video transcription can be done using existing automatic speech recognition (asr) systems, a reasonable step to improve the quality of transcribed texts for livestreaming videos involves post-processing produces to remove noises and restore correct language structures and texts from asr-generated texts. in this paper, we are particularly interested in punctuation restoration (pr) for livestreaming video transcripts. punctuation restoration is the task to restore fundamental text structures such as sentences and phrases by inserting punctuation marks into non-punctuated text, e.g. text generated by an automatic speech recognition system for livestreaming videos in our paper. punctuation restoration is an important post-processing step to improve the readability of asr texts. moreover, in natural language processing (nlp), pr is even more important as it enables the use of advanced techniques to process texts at the sentence level to achieve optimal performance for various tasks, e.g., part-of-speech tagging and dependency parsing. prior studies have shown that with proper sentence split and punctuation, a downstream application can tolerate the word error rate of 25% @xcite , which is extremely high compared to the current state-of-the-art asr. figure 1 demonstrates how punctuation restoration improves the readability of asr-generated texts. in the literature, pr is considered as a subtask of asr, in which pr annotation is done as part of the asr datasets such as the ami @xcite and ted corpus @xcite . however, the speech recorded in these audios involves multi-speaker meetings, as in ami corpus, or single-speaker talks, as in ted corpus. our work is different from those work as we consider livestreaming videos that feature many distinctive characteristics that are essential to study. in particular, the number of speakers in livestreaming use the marquee tool to draw a selection around the empty space on one side then hold shift and add the other areas to the selection too go to edit and fill then change the drop down menu to content aware photoshop should automatically generate a completely new background but it might make a couple of small mistakes use the marquee tool to draw a selection around the empty space on one side. then hold shift and add the other areas to the selection too. go to edit and fill, then change the drop down menu to content aware. photoshop should automatically generate a completely new background. but it might make a couple of small mistakes. videos varies greatly, ranging from one to a few main speakers along with up to thousands of audiences. the audiences might participate in question answering and commenting during the whole duration of the video, hence, constantly changing the information flow of the videos. furthermore, the speech in livestreaming is much more spontaneous than those in the planned meetings of the ami corpus and the well-scripted talks of the ted corpus. an issue with the research of punctuation restoration for livestreaming videos is the lack of a humanannotated dataset for model development and evaluation. this is even more critical when livestreaming has become one of the most powerful communication mediums for not only entertainment but also education purpose. to this end, we introduce a new dataset for behance livestreaming video punctuation restoration, called behancepr. the dataset is annotated by skilled transcription annotators for 4 types of punctuation markers. our experiments reveal the challenges of the behancepr dataset where the performance of current state-of-the-art models for pr on behancepr lags far behind those existing pr datasets (e.g., the ted dataset). our further experiments on cross-domain generation for pr shows that models that are trained on a pr dataset of a different speech scenario perform much worse than those trained on behancepr even with a much larger training set. annotation: we recruit 8 annotators from the upwork.com crowdsourcing platform. as upwork allows its freelancers to submit resumes, we can choose the most experienced annotators with prior experience on audio transcribing. a detailed annotation guideline with many examples is provided to train the annotators. we also develop a customized web-based annotation tool that allows the annotators to work most efficiently with the transcripts and annotation. appendix a shows the interface and description for our designed annotation tool. after self-practicing on the provided guideline and tool, the annotators are further trained by performing actual pr annotation on transcripts of a 2-hour audio from behance. feedback is provided to each annotator in this process to improve the quality. after the training process, the 8 annotators first co-annotate 10% of the documents, leading to the inter-annotation cohen-kappa agreement score of 0.59 (i.e., a moderate to substantial agreement level). afterward, the annotators discuss to resolve the conflicts over the annotated data so far. finally, the remaining documents are distributed to the 8 annotators for separate annotation to produce a final version of our dataset. to facilitate model development and evaluation, we split the dataset into 3 portions for training/development/test data. ## dataset challenges compared to existing pr datasets, e.g., ted @xcite , ami @xcite , our dataset behancepr features several unique challenges. first, as behancepr's documents are obtained from livestreaming video transcripts, they introduce the unique characteristics of spontaneous speech. this is very different from ted talks, in which the talks are heavily scripted beforehand, and ami meetings, where the talks are also well prepared. as such, livestreaming video transcripts have a much lower cohesion level as they might present sudden changes of topic and incomplete syntax (among others). besides, they come with much more verbal pause and repetition of words and phrases, which are the results of hesitation and stutter of the speakers, thus causing a new level of challenges for pr models. second, as the documents in behancepr are generated by an asr system, it is expected that there is a certain number of word errors (e.g., incorrect transcription, missing words) in the texts. as such, word errors can hinder the language understanding ability, and thus pr performance, for the models on behancepr. third, our introduced exclamation mark is a brand new label that are not captured in existing pr datasets, e.g., the ted and ami datasets where emotion is rarer. to appropriately restore exclama-verbal pause so, this is what we got for the site map. we talked about six of these being virus killers maps triall assets forum another so, those will be that. so, then will also have a footer. um, so this will be the home page going to, um, start grab some assets. ## related work early studies on pr have explored a wide range of features such as lexical, acoustic, prosodic, and their combination @xcite @xcite @xcite . graphical models such as crf have been widely used for this task @xcite before the emerging of neural networks. recently, a variety of deep neural network architectures have been explored for pr such as lstm @xcite , convolutional network @xcite , and transformers @xcite . corpora for pr are usually created as part of asr datasets in various domains such as meetings @xcite , ted talks @xcite , audio books @xcite , and film subtitles @xcite . among these, the ted corpus is widely used as the benchmark corpus for pr. however, livestreaming video transcripts have not been explored for pr in prior work. ## conclusion we present behancepr, the first dedicated corpus for punctuation restoration for livestreaming video transcripts. behancepr is manually annotated for 4 markers and present unique challenges for pr. our experiments with state-of-the-art models show the challenges of pr for livestreaming videos and call for more research effort in this important area.