M ulti T urn C leanup: A Benchmark for Multi-Turn Spoken Conversational Transcript Cleanup (2023)

Abstract: Current disfluency detection models focus on individual utterances each from a single speaker. However, numerous discontinuity phenomena in spoken conversational transcripts occur across multiple turns, which can not be identified by disfluency detection models. This study addresses these phenomena by proposing an innovative Multi-Turn Cleanup task for spoken conversational transcripts and collecting a new dataset, MultiTurnCleanup. We design a data labeling schema to collect the high-quality dataset and provide extensive data analysis. Furthermore, we leverage two modeling approaches for experimental evaluation as benchmarks for future research.

## introduction spontaneous spoken conversations contain interruptions such as filled pauses, self-repairs, etc. @xcite . these phenomena act as noise that hampers human readability @xcite and the performance of downstream tasks such as question answering @xcite or machine translation @xcite on transcripts of human spoken conversations. state-of-the-art disfluency detection methods @xcite ) identify and remove disfluencies in order to improve the readability of spoken conversational transcripts @xcite . for instance, figure 1(a) shows that disfluency detection methods can remove self-repairs of single turns. however, these models focus on removing interruptions and errors that commonly occur within single-turn utterances and cannot handle discontinuities across multiple turns. for example, in figure 1 following along ("a: exactly"). b continues their train of thought ("b: just in the last little while. because...") by paraphrasing their own last sentence ("...just in the last generation."). the result is an exchange that is longer and more difficult to follow than necessary to understand what b is conveying. this paper aims to "clean up" spoken conversation transcripts by detecting these types of multiturn "discontinuities" inherent in spontaneous spoken conversations. once detected, they can be removed to produce transcripts that look more like hand-written conversations conducted over text messaging, social media, or e-mail as shown in figure 1 @xcite . given that this is a novel task, with no pre-existing labeled data or benchmarks, we first define a taxonomy of non-disfluency discontinuities (see figure 1 ). then we collect a new dataset, multiturncleanup, for the multi turn spoken conversational transcript cleanup task, based on the switchboard corpus @xcite and label according to the proposed taxonomy. finally we develop two baseline models to detect these discontinuities which we evaluate as benchmarks for future multi-turn cleanup task studies. our data analysis suggests that the multiturncleanup dataset is of high quality. we believe it will help to facilitate research in this under-investigated area. ## data collection and analysis we propose an innovative multi-turn cleanup task and collect a novel dataset for this task called mul-titurncleanup foot_0 . this section presents the task definition, data collection process, and analysis. ## multi-turn cleanup models given the collected multiturncleanup dataset, we leverage two different bert-based modeling approaches, including a two-stage model and a combined model, for the multi-turn cleanup task to remove both single-turn disfluencies and multiturn discontinuities. ## related work recent disfluency detection studies develop bertbased models @xcite @xcite and show significant improvement over lstm-based models @xcite @xcite in disfluency detection tasks. prior studies also show the importance of data augmentation methods, by leveraging extra transcript sources to improve disfluency detection performance @xcite . while most of the research has been focused on improving single-turn disfluency detection accuracy, little exploration has been done in detecting multi-turn transcript discontinuities. obtaining reliable annotated datasets via crowdsourcing is challenging and expensive @xcite @xcite . to collect qualified dataset for multi-turn cleanup task, this work designs a data labeling schema which efficiently collects qualified dataset via the mturk. ## limitation and conclusion we are aware that, in some specific scenarios, it might be undesirable to remove some multi-turn discontinuities because they convey social meaning in human interactions (e.g., engagement). we address this issue by providing category labels. as a result, future research can flexibly select subsets of the multiturncleanup labels to train the model and clean up multi-turn discontinuities. this study defines an innovative multi-turn cleanup task and collects a high-quality dataset for this task, named multiturncleanup, using our presented data labeling schema. we further leverage two modeling approaches for experimental evaluation as the benchmarks for future research. in the next stage, we conservatively remove the following types of disfluencies recursively: by removing all the markers and disfluencies described above, we present the remaining contents of switchboard corpus to mturk workers for labeling the multiturncleanup dataset for multi-turn cleanup task.