Original: BehancePR: A Punctuation Restoration Dataset for Livestreaming Video Transcript
Revised: BehancePR: A Punctuation Restoration Dataset for Livestreaming Video Transcripts

Original: Anonymous ACL submission

Original: Abstract
Revised: Abstract

Original: Given the increasing number of livestreaming videos, automatic speech recognition andpost-processing for livestreaming video transcripts are crucial for efficient data management as well as knowledge mining.
Revised: Given the increasing number of livestreaming videos, automatic speech recognition and post-processing for livestreaming video transcripts are crucial for efficient data management as well as knowledge mining.

Original: A keystep in this process is punctuation restorationwhich restores fundamental text structures suchas phrase and sentence boundaries from thevideo transcripts.
Revised: A key step in this process is punctuation restoration which restores fundamental text structures such as phrase and sentence boundaries from the video transcripts.

Original: This work presents a newhuman-annotated corpus, called BehancePR,for punctuation restoration in livestreamingvideo transcripts.
Revised: This work presents a new human-annotated corpus, called BehancePR, for punctuation restoration in livestreaming video transcripts.

Original: Our experiments on BehancePR demonstrate the challenges of punctuation restoration for this domain.
Revised: Our experiments on BehancePR demonstrate the challenges of punctuation restoration for this domain.

Original: Furthermore,we show that popular natural language processing toolkits like Stanford Stanza, Spacy,and Trankit underperform on detecting sentence boundary on non-punctuated transcriptsof livestreaming videos.
Revised: Furthermore, we show that popular natural language processing toolkits like Stanford Stanza, Spacy, and Trankit underperform on detecting sentence boundary on non-punctuated transcripts of livestreaming videos.

Original: The dataset will bemade available to the public to foster researchin this area.
Revised: The dataset is publicly accessible at http://github.com/ nlp-uoregon/behancepr .

Original:  Introduction
Revised:  Introduction

Original: Livestreaming is a powerful broadcasting mediumthat catches the attention of millions of users.
Revised: Livestreaming is a powerful broadcasting medium that catches the attention of millions of users.

Original: Many video-sharing platforms have supportedlivestreaming for a wide range of topics such as
Revised: Many video-sharing platforms have supported livestreaming for a wide range of topics such as

Original: Twitch for gaming, TikTok for short entertainment videos, Behance for visual creative work,and Youtube/Facebook Live accepting any topics.
Revised: Twitch for gaming, TikTok for short entertainment videos, Behance for visual creative work, and Youtube/Facebook Live accepting any topics.

Original: Among these videos, there are a substantially highnumber of videos that provide useful knowledgewith exceptional visual demonstration.
Revised: Among these videos, there are a substantially high number of videos that provide useful knowledge with exceptional visual demonstration.

Original: To thisend, livestreaming videos are becoming a potentialknowledge base waiting for being explored.
Revised: To this end, livestreaming videos are becoming a potential knowledge base waiting for being explored.

Original: Mining videos on video/audio format directly isextremely hard and expensive because of their highdata load and complexity in processing images andaudio signals.
Revised: Mining videos on video/audio format directly is extremely hard and expensive because of their high data load and complexity in processing images and audio signals.

Original: Instead, mining video transcripts,transcribed by either human or machine, is mucheasier with the existing hardware and software.
Revised: Instead, mining video transcripts, transcribed by either human or machine, is much easier with the existing hardware and software.

Original: Assuch, livestreaming videos should be transcribedat high quality to facilitate future data mining research.
Revised: As such, livestreaming videos should be transcribed at high quality to facilitate future data mining research.

Original: As video transcription can be done using existing automatic speech recognition (ASR) systems, a reasonable step to improve the quality of transcribed texts for livestreaming videos involves post-processing produces to remove noises and restore correct language structures and texts from
Revised: As video transcription can be done using existing automatic speech recognition (ASR) systems, a reasonable step to improve the quality of transcribed texts for livestreaming videos involves post-processing produces to remove noises and restore correct language structures and texts from

Original: ASR-generated texts.
Revised: ASR-generated texts.

Original: In this paper, we are particularly interested in punctuation restoration (PR) for livestreaming video transcripts.
Revised: In this paper, we are particularly interested in punctuation restoration (PR) for livestreaming video transcripts.

Original: Punctuation restoration is the task to restore fundamental text structures such as sentences and phrases by inserting punctuation marks into non-punctuated text, e.g. text generated by an automatic speech recognition system for livestreaming videos in our paper.
Revised: Punctuation restoration is the task to restore fundamental text structures such as sentences and phrases by inserting punctuation marks into non-punctuated text, e.g. text generated by an automatic speech recognition system for livestreaming videos in our paper.

Original: Punctuation restoration is an important post-processing step to improve the readability of ASR texts.
Revised: Punctuation restoration is an important post-processing step to improve the readability of ASR texts.

Original: Moreover, in natural language processing (NLP), PR is even more important as it enables the use of advanced techniques to process texts at the sentence level to achieve optimal performance for various tasks, e.g., part-of-speech tagging and dependency parsing.
Revised: Moreover, in natural language processing (NLP), PR is even more important as it enables the use of advanced techniques to process texts at the sentence level to achieve optimal performance for various tasks, e.g., part-of-speech tagging and dependency parsing.

Original: Prior studies have shown that with proper sentence split and punctuation, a downstream application can tolerate the word error rate of 25% (Alam et al., 2015), which is extremely high compared to the current state-of-the-art ASR.
Revised: Prior studies have shown that with proper sentence split and punctuation, a downstream application can tolerate the word error rate of 25% (Alam et al., 2015), which is extremely high compared to the current state-of-the-art ASR.

Original: Figure 1 demonstrates how punctuation restoration improves the readability of ASR-generated texts.
Revised: Figure 1 demonstrates how punctuation restoration improves the readability of ASR-generated texts.

Original: In the literature, punctuation restoration is considered as a subtask of ASR, in which PR annotation is done as part of the ASR datasets such as the AMI (McCowan et al., 2005) and TED corpus (Federico et al., 2012).
Revised: In the literature, PR is considered as a subtask of ASR, in which PR annotation is done as part of the ASR datasets such as the AMI (McCowan et al., 2005) and TED corpus (Federico et al., 2012).

Original: However, the speeches recorded in these audios are multi-speaker meetings, as in AMI corpus, or single-speaker talks, as in TED corpus.
Revised: However, the speech recorded in these audios involves multi-speaker meetings, as in AMI corpus, or single-speaker talks, as in TED corpus.

Original: Our work is different from this work in that we consider livestreaming videos that feature many distinctive characteristics that are essential to study.
Revised: Our work is different from those work as we consider livestreaming videos that feature many distinctive characteristics that are essential to study.

Original: In particular, the number of speakers in  use the marquee tool to draw a selection around the empty space on one side then hold shift and add the other areas to the selection too go to edit and fill then change the drop down menu to content aware photoshop should automatically generate a completely new background but it might make a couple of small mistakes
Revised: In particular, the number of speakers in livestreaming use the marquee tool to draw a selection around the empty space on one side then hold shift and add the other areas to the selection too go to edit and fill then change the drop down menu to content aware photoshop should automatically generate a completely new background but it might make a couple of small mistakes

Original: Use the marquee tool to draw a selection around the empty space on one side.
Revised: Use the marquee tool to draw a selection around the empty space on one side.

Original: Then hold shift and add the other areas to the selection too.
Revised: Then hold shift and add the other areas to the selection too.

Original: Goto edit and fill, then change the drop down menu to content aware.
Revised: Goto edit and fill, then change the drop down menu to content aware.

Original: Photoshop should automatically generate a completely new background.
Revised: Photoshop should automatically generate a completely new background.

Original: But it might make a couple of small mistakes.
Revised: But it might make a couple of small mistakes.

Original: livestreaming videos varies greatly ranging from1 to a few main speakers together with up to thousands of audiences.
Revised: videos varies greatly, ranging from one to a few main speakers along with up to thousands of audiences.

Original: The audiences might participate in question-answering and commenting duringthe whole duration of the video, hence, changingthe topic of the video.
Revised: The audiences might participate in question answering and commenting during the whole duration of the video, hence, constantly changing the information flow of the videos.

Original: Furthermore, the speech inlivestreaming is much more spontaneous than thosein meetings in  AMI corpus and the well-scripted  TED talk.
Revised: Furthermore, the speech in livestreaming is much more spontaneous than those in the planned meetings of the AMI corpus and the well-scripted talks of the TED corpus.

Original: An issue with the research of punctuation restoration for livestreaming videos is the lack of a humanannotated dataset for model development and evaluation.
Revised: An issue with the research of punctuation restoration for livestreaming videos is the lack of a humanannotated dataset for model development and evaluation.

Original: This is even more critical when livestreaminghas become one of the most powerful communication mediums for not only entertainment but alsoeducation purpose.
Revised: This is even more critical when livestreaming has become one of the most powerful communication mediums for not only entertainment but also education purpose.

Original: Toward this end, we introducea new dataset for Behance Livestreaming Video[Figure] Punctuation Restoration, called BehancePR .The
Revised: To this end, we introduce a new dataset for Behance Livestreaming Video Punctuation Restoration, called BehancePR .

Original: [Figure] dataset was annotated by skilled transcription anno- [Figure] tators for 4 types of punctuation.
Revised: The dataset is annotated by skilled transcription annotators for 4 types of punctuation markers.

Original: Our experiments

Original: [Figure]

Original:  reveal the challenges of the BehancePR dataset [Figure] that the performance of the existing state-of-the-art [Figure]
Revised: Our experiments reveal the challenges of the BehancePR dataset where the performance of current state-of-the-art models for PR on BehancePR lags far behind those existing PR datasets (e.g., the TED dataset).

Original: models for punctuation restoration on BehancePR

Original: [Figure]

Original: lags far behind one on the TED dataset.

Original: Our further

Original: [Figure]

Original: experiment of cross-domain punctuation shows that

Original: [Figure] models that are trained on a PR dataset of a differ- [Figure] ent speech scenario perform much worse than thosetrained onthe BehancePRdataset even with a muchlarger training set.
Revised: Our further experiments on cross-domain generation for PR shows that models that are trained on a PR dataset of a different speech scenario perform much worse than those trained on BehancePR even with a much larger training set.

Original:  Data Annotation
Revised:  Data Annotation

Original: Preparation: The livestreaming videosthat weannotate in this work are derived from  Behance.net .
Revised: Preparation: The livestreaming videos in this work are collected from the public source of Behance.net .

Original: Behance is an online platform to showcaseand discover creative work such as digital drawing,graphic design, and photo/video editing.
Revised: Behance is an online platform to showcase and discover creative work such as digital drawing, graphic design, and photo/video editing.

Original: In those videos, one or a few creators stream their work on graphic design tools in English. The topics in the videos relate to design theories, graphical ideas, and tutorials to usethose graphic design tools.
Revised: In those videos, one or a few creators stream their work on graphic design tools in English, covering a wide range of topics such as design theories, graphical ideas, and tutorials to use graphic design tools.

Original: The videos are split  into shorter clips of 5 minutes per clip.
Revised: In our dataset, we split the videos into shorter clips of 5 minutes.

Original: Then, the videos are transcribed by the Microsoft ASR system.
Revised: Next, the clips are transcribed by the Microsoft ASR system.

Original: The video and the aligned automatic-generated transcript are  presented to the annotators.
Revised: The automatically generated transcripts for each clip (called documents) are then presented to the annotators.

Original: To prepare for theannotation, we also created a taxonomy for livestreaming video punctuation restoration.
Revised: To prepare for the

Original: Similar to prior studies in punctuation restoration, we inherit the taxonomy of three most popular markers: i.e. period , comma , and question mark  (Federico et al., 2012).
Revised: PR annotation in livestreaming video transcripts, we inherit the set of three most popular markers, i.e. period , comma , and question mark in prior PR datasets (Federico et al., 2012).

Original: However, as livestreaming videos of creative works involve a lot of emotional expressions such as excitement, we include exclamation mark  to better present strong feelings and emphasis.
Revised: In addition, as livestreaming videos of creative works involve a lot of emotional expressions (e.g., excitement), we include exclamation mark as a new annotation label to better capture strong feelings and emphasis in this area.

Revised: Our instruction guidline is presented in Appendix B.

Revised: To accommodate our annotation budget, we randomly select 2,314 transcribed documents for PR annotation.

Original: Annotation: We recruit 8 annotators from 
Revised: Annotation: We recruit 8 annotators from the

Original: Upwork.com crowdsourcing platform.
Revised: Upwork.com crowdsourcing platform.

Original: As Upwork allows the freelancers to submittheir resumes, we can choose the most experienced annotators with prior practices with audio transcribing.
Revised: As Upwork allows its freelancers to submit resumes, we can choose the most experienced annotators with prior experience on audio transcribing.

Original: A detailed annotation guideline with many examples is provided to  the annotators.
Revised: A detailed annotation guideline with many examples is provided to train the annotators.

Original: We developed a customized web-based annotation tool that allows the annotator to work most efficiently with the material.
Revised: We also develop a customized web-based annotation tool that allows the annotators to work most efficiently with the transcripts and annotation.

Revised: Appendix A shows the interface and description for our designed annotation tool.

Original: To make sure the annotators understand the task andbe familiar with the annotation tool, they are further trained  on 2 hours of  audio on the Behance videos, equivalent to approximately 6 hours of training and practice.
Revised: After self-practicing on the provided guideline and tool, the annotators are further trained by performing actual PR annotation on transcripts of a 2-hour audio from Behance. Feedback is provided to each annotator in this process to improve the quality.

Original: We group 6 annotators into 3 pairs ofannotators that work together in the same set of documents. The inter-annotation Cohen-Kappa agreement score is 0.59 , which indicates a moderate to substantial agreement.
Revised: After the training process, the 8 annotators first co-annotate 10% of the documents, leading to the inter-annotation Cohen-Kappa agreement score of 0.59 (i.e., a moderate to substantial agreement level).

Original: Toward the end, the annotatorsare allowed to discuss to make the  final version of the dataset. 
Revised: Afterward, the annotators discuss to resolve the conflicts over the annotated data so far. Finally, the remaining documents are distributed to the 8 annotators for separate annotation to produce a final version of our dataset. To facilitate model development and evaluation, we split the dataset intoportions for training/development/test data.

Original: Table 1 showsthe detailed statistics and label distribution of the BehancePR dataset.
Revised: Table 1 shows detailed statistics and label distribution of our BehancePR dataset.

Original: Dataset Challenges

Original: Compare to existing punctuation restoration datasets, e.g., TED (Federico et al., 2012), AMI (McCowan et al., 2005), our dataset BehancePR features several unique challenges for punctuation restoration.

Original: First, as its documents are obtained from live streaming video on the Behance platform, it features unique characteristics of spontaneous speech.

Original: Train
Revised: Train

Original: Dev
Revised: Dev

Original: Test
Revised: Test

Original: [Table]
Revised: [Table]

Revised:  Dataset Challenges

Revised: Compared to existing PR datasets, e.g., TED (Federico et al., 2012), AMI (McCowan et al., 2005), our dataset BehancePR features several unique challenges.

Revised: First, as BehancePR’s documents are obtained from livestreaming video transcripts, they introduce the unique characteristics of spontaneous speech.

Original: They are much different from TED talks, in whichthe talks were heavily scripted, and AMI meetings, where the talks were also well prepared.
Revised: This is very different from TED talks, in which the talks are heavily scripted beforehand, and AMI meetings, where the talks are also well prepared.

Original: Assuch, the live streaming video transcript text hasa much lower cohesion. As it might be presentedin a sudden change of topic and incomplete syntax.
Revised: As such, livestreaming video transcripts have a much lower cohesion level as they might present sudden changes of topic and incomplete syntax (among others).

Original: Besides, they come with a substantially highfrequency verbal pause, repetition of words andphrases, which are the results of hesitation and stutter of the speakers.
Revised: Besides, they come with much more verbal pause and repetition of words and phrases, which are the results of hesitation and stutter of the speakers, thus causing a new level of challenges for PR models.

Original: Second, as the documents in this dataset are generated from an automatic speech recognition system, it is expected that there is a certain number ofword errors  in the text even though the automaticspeech recognition technology has improved significantly recently.
Revised: Second, as the documents in BehancePR are generated by an ASR system, it is expected that there is a certain number of word errors (e.g., incorrect transcription, missing words) in the texts. As such, word errors can hinder the language understanding ability, and thus PR performance, for the models on BehancePR. Table 2 shows the examples for different types of noisy texts including verbal pauses, duplicate words and phrases, incomplete syntax, instructional steps, and word errors. In the word error examples, as the streamer has just hurt herself, the ASR system cannot detect the word “ Oww ”.

Original: In theworst case, word errorsmight affect punctuation prediction.
Revised: Instead, it generates “ Oh ” and “ How ”. This error is highly adverse as it might turn a declarative sentence into a WH question starting with the word “ How ”. Third, our introduced exclamation mark is a brand new label that are not captured in existing

Original: Examples ofthese challenges are presented in the Appendix A.
Revised: PR datasets, e.g., the TED and AMI datasets where emotion is rarer.

Original: Easy Owe my goodness
Revised: To appropriately restore exclamaVerbal pause So , this is what we got for the site map. We talked about six of these being virus killers maps triall assets forum another So , those will be that.

Revised: So , then will also have a footer.

Original: [Table]
Revised: Um, so this will be the home page going to, um , start grab some assets... Duplicate words and phrases Alright , alright , alright So, but there are all set

Original: I ’am excited for that
Revised: All of these are all set

Original: [Table]
Revised: Incomplete syntax Um so this will be the homepage going to hum start grab some assets I guess image that google “survivor” Instructional steps

Original: That’s an interesting pick [Table]
Revised: What if I click it open 2 xbox hub

Original: Imagine
Revised: Inspect

Original: [Table] boom [Table] It’s in here
Revised: Header 1920 by

Original: [Table]
Revised: Copy

Original: I found it
Revised: I got it

Original: [Table]
Revised: Word error

Original: Let’s
Revised: Oh definitely just stubbed my toe.

Original: [Table] into [Table]
Revised: And in not very fun pain.

Original: He [Table]
Revised: How just making sure there will be no bleeding.

Original: Third, we introduce the exclamation mark foremotional sentences. This exclamation mark is abrand new challenge compared to other datasets,e.g., the TED and AMI talks in which emotion israre. To detect these sentences, a  model needs notonly the textual features but also acoustic featuressuch as frequency, the strength ofthe excitement. 
Revised: tion mark, a PR model needs to encode not only textual content, but also acoustic features such as frequency and strength of excitement. However, as BehancePR and current PR datasets do not provide access to audio features, the new label for exclamation mark will introduce a new challenging dimension that makes BehancePR an unique PR dataset.

Original: As such, without access to the audio, a model mightfind this a challenging task. Especially, this becomes even harder in a substantial amount of casesthat emotional words do not appear in the sentence. Table 2 shows some examples of easy and hard sentences with and without the appearance of emotional words, respectively.
Revised: We also note that future work can extend BehancePR to include audio features to achieve multi-modal PR.

Original:  Experiments
Revised:  Experiments

Original: Supervised learning: To reveal the complexity of the punctuation restoration for the livestreaming video, we evaluate the performance of the state-ofthe-art model for punctuation restoration.
Revised: Supervised Learning: To reveal the complexity of BehancePR, we evaluate the performance of the state-of-the-art (SOTA) model for PR on this dataset.

Original: Similar to prior work in this task, we model the task as a sequence labeling task on the token level.
Revised: Similar to prior work, we model PR as a sequence labeling task at the token level that aims to assign one of the five punctuation labels

Revised: (i.e., 4 designed labels and 1 special labels for non-punctuation) to every space in input texts.

Original: We investigate two major  model architectures: neural-based model withBi-Directional Long Short-Term Memory (BiLSTM)
Revised: In particular, we investigate two major SOTA model architectures for PR: a neural-based model with

Original:  (Alam et al., 2020), and graphicalbased model with Conditional Random Field (CRF) (Makhija et al., 2019).
Revised: BiLSTM in (Alam et al., 2020), and graphicalbased model with Conditional Random Field in (Makhija et al., 2019).

Original: We also investigate the recent advance in data augmentationtechnique for punctuation restoration (Alam et al., 2020).
Revised: We also investigate the recent advances in data augmentation for PR to automatically produce more training data in (Alam et al., 2020).

Original: These leave us four combinations of model and technique as presented in table 3.
Revised: Applying the data augmentation to two SOTA models leads to four possible model combination as presented in Table 3.

Revised: We fine-tune the hyper-parameters for the models on the development data of BehancePR.

Original: All of these models leverage a pre-trained language model RoBERTa  (Liu et al., 2019) to obtain representation vectors. Table 3(a) presents the  performance of four models on the development and the testing sets of the
Revised: We find that the pre-trained language model RoBERTa (large version) (Liu et al., 2019) delivers the best performance among RoBERTA, AlBERT, and bert-largeuncased version of BERT. This confirms prior results by Alam et al.

Original: BehancePR dataset.
Revised: The texts are split into sequences of 256 word pieces. The best batch size is 64.

Original: First, the graphical-based models with CRF show a slight improvement over the neural-based models in case no data augmentation is applied. In contrast, the performance on the test set decreases.
Revised: The selected learning rate is 3 e -5 for the Adam optimizer. We use a single BiLSTM layer with 200 hidden units for the models.

Original: This result is consistent with the prior study in punctuation restoration (Alam et al., 2020) that performance gain is not significant when
Revised: The augmentation rate is set to 0.2 following previous research (Alam et al., 2020).

Original: CRF is used, which is opposite to the Named Entity Recognition task that CRF works very well.
Revised: Table 3(a) presents the performance of four models on the development and the test sets of BehancePR.

Original: We attribute this to the difference between the PR and theNER tasks.
Revised: The first observation from the table is the

Original: In punctuation restoration, an entity, which is a sentence, is much longer than an entity in the NER task. As such, it is harder to model the dependency between tokens in such a long sequence.
Revised: CRF component can improve the performance of BiLSTM when no data augmentation is applied, thus suggesting the effectiveness of capturing dependencies between labels with CRF for PR.

Original: Hence, the CRF is incapable of producing significant improvement. Second, the data augmentation technique slightly improves the performance of the BiLSTM model but slightly decreases the  performance  of the CRF-based models.
Revised: We also observe that data augmentation has zero or little contribution to the performance of the models on BehancePR. In all, the best PR performance on BehancePR is achieved when CRF is applied on top of the BiLSTM model (without using data augmentation).

Original: Importantly, we find that the performance of the PR models on BehancePR is far below on the TED talk dataset (F-score=84%).
Revised: Importantly, we find that the performance of current PR models on BehancePR is far behind that on the TED talk dataset (with F score of at least 84%) and perfect performance.

Original: This suggests this dataset is much more challenging  and calls for further study on this domain.
Revised: It thus indicates the more challenging nature of PR on livestreaming video transcripts with BehancePR and calls for further study on this domain.

Original: Domain adaptation: As punctuation restoration data is abundant, we further explore the cross-
Revised: Domain Adaptation: To understand the domain difference between BehancePR and current

Original: [Table] domain evaluation setting where the models aretrained on a different source domain and evaluatedon the BehancePR dataset.
Revised: PR datasets, we further explore the cross-domain evaluation setting where the models are trained on a different source domain and evaluated on BehancePR as the target domain.

Original: In particular, we choosethe TED corpus as the source domain because TEDtalks are monologues, which is the closest to the Behance videos.
Revised: In particular, we choose the TED corpus as the source domain as TED talks [Table] are monologues, which is closer to the Behance videos.

Original: Table 3(b) presents the out-ofdomain performance of the models.
Revised: Table 3(b) presents the the out-of-domain performance of the models.

Original: It is clear fromthe table that the performances of all punctuationrestoration models degrade significantly when theyare trained on TED talk and evaluated on Behance.
Revised: It is clear from the table that the performance of all PR models on BehancePR degrades significantly when they are trained on TED talks.

Original: This demonstrates the considerable difference indomains even they are almost monologual.
Revised: This demonstrates the considerable difference between the domains in TED and BehancePR.

Revised: It also highlights the benefit of the annotated BehancePR dataset to achieve better PR performance for video transcripts.

Original: Sentence split: We evaluate the performance ofthe models on the sentence splitting task.
Revised: Sentence Splitting: We conduct an additional experiment to demonstrate another benefit of BehancePR on evaluation sentence splitting toolkits.

Original: In thistask, the model is trained and evaluated to predictwhere the sentence ends.
Revised: In this task, the models need to predict where the sentences end.

Revised: As such, we transform the BehanceED dataset by removing comma labels and converting the other labels into a single label of sentence ending.

Original: We trained the modelspresented in the supervised learning setting on thesame BehancePR dataset.
Revised: We then train the four models in the supervised learning experiment on the transformed training dataset of BehancePR to detect sentence ending label for sentence splitting. The models are then evaluated on the transformed test set.

Original: We further examine theperformance of existing NLP toolkit for  this taskincluding Stanza (Qi et al., 2020), SpaCy (Honnibaland Montani, 2017), and Trankit (Nguyen et al.,2021).
Revised: In addition, we examine the performance of existing NLP toolkits for sentence splitting in this new dataset, including Stanza (Qi et al., 2020), SpaCy (Honnibal and Montani, 2017), and Trankit (Nguyen et al., 2021).

Original: The performance of the models and toolkitsare presented in Table 4.
Revised: The performance of the models and toolkits are presented in Table 4.

Original: The published toolkits perform very poorly onthis kind of text, with the highest F-1 score of30.9% by SpaCy due to their low recall performances.
Revised: As can be seen, existing toolkits perform very poorly on this domain, with the highest F-1 score of only 30.9%.

Original: Stanza’s sentence splitter is the worst withan extremely low recall of 1.4%, Trankit and Spacyare slightly better with higher recalls of 7.8% and21.9%, respectively.

Original: The reason  is that these models were trained on  punctuated text (Nivre et al.,2016), hence, the model is heavily dependent onthe presence of punctuation to trigger the sentenceboundary prediction.
Revised: One potential reason for this poor performance is that existing toolkits are trained on perfectly punctuated text (Nivre et al., 2016), making them unfit for our text domain with missing punctuation.

Original: In another word, they barelylearn the context to predict the sentence boundary. Secondly, the models trained on the BehancePRdataset outperform the published toolkits  with large
Revised: As such, the models trained on the transformed BehancePR dataset significantly outperform existing toolkits for sentence splitting with substantial gaps. This demonstrates the ability of

Original: [Table]
Revised: [Table]

Original: margins at consistent F-1 scores above 72%. Without thepunctuation in the text, these models learn the contextual information to predict the sentence ending, hence, successfully performing the task in this extreme case.
Revised: the models to effectively encode contextual information to infer sentence ending.

Original: This suggests the importance ofthe training data for even fundamental tasks like sentence splitting.
Revised: It also suggests the importance of training data for even basic tasks such sentence splitting in challenging domains.

Original:  Related work
Revised:  Related work

Original: Early studies on the punctuation restoration task explored a wide range of features such as lexical, acoustic, prosodic, and their combination (Gravano et al., 2009; Levy et al., 2012; 
Revised: Early studies on PR have explored a wide range of features such as lexical, acoustic, prosodic, and their combination (Gravano et al., 2009; Levy et al., 2012; Xu et al., 2014;

Original: Xu et al., 2014; Che et al., 2016a; Szaszák and Tündik, 2019).
Revised: Che et al., 2016a; Szaszák and Tündik, 2019).

Original: Graphical models such as conditional random field have been widely used for this task (Lu and Ng, 2010; Zhang et al., 2013) before the emerging of neural networks.
Revised: Graphical models such as CRF have been widely used for this task (Lu and Ng, 2010; Zhang et al., 2013) before the emerging of neural networks.

Original: Recently, a variety of deep neural network architectures have been explored  such as long short-term memory (Gale and Parthasarathy, 2017), convolutionalneural network (Che et al., 2016b), and transformer (Alam et al., 2020).
Revised: Recently, a variety of deep neural network architectures have been explored for PR such as LSTM (Gale and Parthasarathy, 2017), convolutional network (Che et al., 2016b), and transformers (Alam et al., 2020).

Original: Corpora for punctuation restoration are usually created as part of automatic speech recognition in various domains such as meetings (McCowan et al., 2005), TED talks (Federico et al., 2012), audio books (Panayotov et al., 2015), and film subtitles (Tiedemann, 2016).
Revised: Corpora for PR are usually created as part of ASR datasets in various domains such as meetings (McCowan et al., 2005), TED talks (Federico et al., 2012), audio books (Panayotov et al., 2015), and film subtitles (Tiedemann, 2016).

Original: Among these, the TED corpus is widely used as the benchmark corpus for punctuation restoration.
Revised: Among these, the TED corpus is widely used as the benchmark corpus for PR.

Original: However, livestreaming videos have not been explored for this task.
Revised: However, livestreaming video transcripts have not been explored for PR in prior work.

Original:  Conclusion
Revised:  Conclusion

Original: We presented BehancePR, the first dedicated corpus for punctuation restoration.
Revised: We present BehancePR, the first dedicated corpus for punctuation restoration for livestreaming video transcripts.

Original: BehancePR was manually annotated for 4 markers.
Revised: BehancePR is manually annotated for 4 markers and present unique challenges for PR.

Original: The comprehensive experiments with state-of-the-art models show the challenges of the punctuation restoration for livestreaming video, as well as the poor performance of existing NLP toolkits for non-punctuated text.
Revised: Our experiments with state-of-the-art models show the challenges of PR for livestreaming videos and call for more research effort in this important area.

Original: Ethical Considerations
Revised: Ethical Considerations

Original: In this work we present a dataset on the transcriptsof a publicly accessible video-streaming platform,i.e., “ Behance “ 1 .
Revised: In this work we present a dataset on the transcripts of a publicly accessible video-streaming platform, i.e., “ Behance ” 1 .

Original: Complying with the discussionpresented by Benton et al.
Revised: Complying with the discussion presented by Benton et al.

Original: (2017), research withhuman subjects information is exempted from therequired full Institutional Review Board (IRB) review if the data is already available from publicsources or if the identity of the subjects cannot berecovered.
Revised: (2017), research with human subjects information is exempted from the required full Institutional Review Board (IRB) review if the data is already available from public sources or if the identity of the subjects cannot be recovered.

Original: However, to protect the identity of thestreamer and any other person whose informationare shared in the video transcript, we impose extra consideration on the presented dataset.
Revised: However, to protect the identity of the streamer and any other person whose information are shared in the video transcript, we impose extra processing on the transcribed documents before presenting them to annotators and publicly releasing it later.

Original: First,in this dataset, we exclude the username or anyother identity-related information of the streamerin the transcripts to prevent disclosing their identity.
Revised: First, in this dataset, we remove username or any other identity-related information of the streamers in the transcripts to prevent disclosing their identity.

Original: Moreover, the proposed dataset only provides textual data (i.e., paragraphs), hence the other contentof the videos (e.g., images, audios) are not revealedto protect human identity.
Revised: Moreover, the proposed dataset only provides textual data (i.e., documents), hence the other content of the videos (e.g., images, audios) are not revealed (to annotators or users) to protect human identity.

Original: Finally, to reduce the riskof disclosing the information of the people in thetranscripts, in the final version of the dataset, weexclude the transcripts that explicitly or implicitlyrefer to the identify of the target people.
Revised: Finally, to reduce the risk of disclosing the information of the people in the transcripts, in the final version of the dataset, we exclude the transcripts that explicitly or implicitly refer to the identify of the target people.

Original: A Noisy texts in BehancePR
Revised: Acknowledgement

Original: Table 5 shows the examples of different typesof noisy texts including verbal pauses, duplicatewords and phrases, incomplete syntax, instructionalsteps, and word errors as presented in Section 3. In the word error example, the streamer has justhurt herself, the ASR system was incorrect at detecting the words “ Oww ”. Instead, it generated“ Oh ” and “ How ”.

Original: This error is highly adverse as itmight turn a declarative sentence into a WH question starting with the word “ How ”.

Original: Verbal pause

Original: So , this is what we got for the site map. We talked about six of these being virus killers maps triall assets forum another So , those will be that. So , then will also have a footer.
Revised: This research has been supported by the Army Research Office (ARO) grant W911NF-21-1and the NSF grant CNS-1747798 to the IUCRC Center for Big Learning.

Original: Um, so this will be the home page going to, um , start grab some assets...

Original: Duplicate words and phrases

Original: Alright , alright , alright So, but there are all set All of these are all set Incomplete syntax
Revised: This research is also based upon work supported by the Office of the Director of National Intelligence (ODNI),

Original: Um so this will be the homepage going to hum start grab some assets I guess image that google “survivor” Instructional steps What if I click it open 2 xbox hub Inspect
Revised: Intelligence Advanced Research Projects Activity (IARPA), via IARPA Contract No.

Original: Header 1920 by Copy I got it Word error
Revised: 201919051600006 under the Better Extraction from Text

Original: Oh definitely just stubbed my toe. And in not very fun pain. How just making sure there will be no bleeding. B Reproducibility Checklist
Revised: Towards Enhanced Retrieval (BETTER) Program.

Original: Dataset : The statistics of the created dataset BehancePR (including training/development/test portions) and the annotation process are presented in Section 2. We will publicly release the datasetupon the acceptance of  the paper. A URL to the publishing site will be included in the paper.
Revised: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ARO, ODNI, IARPA, the Department of Defense, or the U.S. Government.

Original: Source code with the specification of all dependencies, including external libraries : We will publicly release the code to run the models upon the acceptance of the paper. Description of computing infrastructure used : All the experiments were run on a machine with 2 Intel Xeon E5-2620 v4 CPUs, 128GB of RAM, and 4 NVIDIA RTX 2080 Ti GPUs with 11GB RAM.
Revised: The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

Original: We only train the models with a single GPU. The amount of GPU memory is from 8GB-10GB depending on the variant being used. Average runtime for each approach : We train the models for 20 epochs; the whole training takes 2 hours. The best epoch is chosen based on the performance on thedevelopment set.
Revised: This document does not contain technology or technical data controlled under either the

Original: Number of parameters in the model :

Original: Our models have total 360M trainable parameters.

Original: Explanation of evaluation metrics used, with links to code : Following prior work in punctuation restoration, we use the precision, recall, and F  scores for performance metrics.

Original: Hyperparameter bounds and configurations for best-performing models : We found that RoBERTa large version delivered the best performance among RoBERTA, AlBERT, and bert-largeuncased version of BERT. This confirms prior result by Alam et al. The texts are split into sequences of 256 word pieces, searched in  the range of { 64 , 128 , 256 , 384 } .
Revised: U.S. International Traffic in Arms Regulations or the U.S. Export Administration Regulations.

Original: The best batch size is 64 searched from { 16 , 32 , 64 } . We use a single BiLSTM layer to encode the text. The augmentation rate is set to 0.2 following previous research’s result (Alam et al., 2020).
Revised: We thank the anonymous reviewers and Tracy King for their helpful feedback.

Original: C Annotation Tool
Revised: A Annotation Tool

Original: We developed a customized web-based annotation tool for this work.
Revised: We develop a customized web-based annotation tool for this work.

Original: The annotation tool focuses on improving the readability of the annotated text, as the result, improves the annotation quality.
Revised: The annotation tool focuses on improving the readability of the annotated text, as the result, improves the annotation quality.

Original: Toward this end, we use color coding for punctuation markers.
Revised: Toward this end, we use color coding for punctuation markers.

Original: More importantly, whenever a sentence ending marker is assigned, such as period, question mark, and exclamation, it automatically creates a new line to separate sentences.
Revised: More importantly, whenever a sentence ending marker is assigned, such as period, question mark, and exclamation, it automatically creates a new line to separate sentences.

Original: Figure 2 shows the interface and the annotated text using our tool.
Revised: Figure 2 shows the interface and an annotated text using our tool.

Original: D Annotation guidelines
Revised: B Annotation Guideline

Original: This section summarizes the taxonomy, annotation guideline, and annotation examples.
Revised: This section summarizes the taxonomy, annotation guideline, and annotation examples.

Original: Examples are shown in figure 2.
Revised: Examples are shown in figure 2.

Original: A period is used for:
Revised: A period is used for:

Original: • Marking the end of a declarative sentence .
Revised: • Marking the end of a declarative sentence .

Original: • Separating independent clauses without a con- junction when a semi-colon is usually used (to distinguish with the case that a comma is used when a conjunction presents).
Revised: • Separating independent clauses without a conjunction when a semi-colon is usually used (to distinguish with the case that a comma is used when a conjunction presents).

Original: A question mark is used for:
Revised: A question mark is used for:

Original: • Marking the end of a question .
Revised: • Marking the end of a question .

Original: An exclamation mark is used for:
Revised: An exclamation mark is used for:

Original: • Exclaiming something.
Revised: • Exclaiming something.

Original: They are commonly used after interjections (words or phrases that are used to exclaim, command, or protest like “wow” or “oh”).
Revised: They are commonly used after interjections (words or phrases that are used to exclaim, command, or protest like “wow” or “oh”).

Original: • Express the following emotions: excitement, surprise, astonishment, emphasizing a point, and other types of strong emotions.
Revised: • Express the following emotions: excitement, surprise, astonishment, emphasizing a point, and other types of strong emotions.

Original: A comma is used for:
Revised: A comma is used for:

Original: • Separating independent clauses when they are joined by any of these seven coordinating con- junctions: and, but, for, or, nor, so, yet.
Revised: • Separating independent clauses when they are joined by any of these seven coordinating conjunctions: and, but, for, or, nor, so, yet.

Original: • Separating introductory clauses, phrases, or words from the main clause.
Revised: • Separating introductory clauses, phrases, or words from the main clause.

Original: • Setting off clauses, phrases, and words that are not essential to the meaning of the sen- tence.
Revised: • Setting off clauses, phrases, and words that are not essential to the meaning of the sentence.

Original: Use one comma before to indicate the beginning of the pause and one at the end to indicate the end of the pause.
Revised: Use one comma before to indicate the beginning of the pause and one at the end to indicate the end of the pause.

Original: • Separating three or more words, phrases, or clauses written in a series.
Revised: • Separating three or more words, phrases, or clauses written in a series.

Original: • Separating two or more coordinate adjectives that describe the same noun.
Revised: • Separating two or more coordinate adjectives that describe the same noun.

Original: Be sure never to add an extra comma between the final adjec- tive and the noun itself or to use commas with non-coordinated adjectives.
Revised: Be sure never to add an extra comma between the final adjective and the noun itself or to use commas with non-coordinated adjectives.

Original: • Separating contrasted coordinate elements or to indicate a distinct pause or shift near the end of a sentence.
Revised: Separating contrasted coordinate elements or to indicate a distinct pause or shift near the end of a sentence.

Original: • Setting off phrases at the end of the sentence that refers back to the beginning or middle of the sentence.
Revised: • Setting off phrases at the end of the sentence that refers back to the beginning or middle of the sentence.

Original: Such phrases are free modifiers that can be placed anywhere in the sentence without causing confusion.
Revised: Such phrases are free modifiers that can be placed anywhere in the sentence without causing confusion.

Original: • Setting off all geographical names, items in dates (except the month and day), addresses (except the street number and name), and titles in names.
Revised: • Setting off all geographical names, items in dates (except the month and day), addresses (except the street number and name), and titles in names.

Original: • Shifting between the main discourse and a quotation.
Revised: Shifting between the main discourse and a quotation.

Original: • Preventing possible confusion or misreading.
Revised: • Preventing possible confusion or misreading.