Curricular Next Conversation Prediction Pretraining for Transcript Segmentation (2023)

Abstract: Transcript segmentation is the task of dividing a single continuous transcript into multiple segments. While document segmentation is a popular task, transcript segmentation has significant challenges due to the relatively noisy and sporadic nature of data. We propose pretraining strategies to address these challenges. The strategies are based on “Next Conversation Prediction” (NCP) with the underlying idea of pretraining a model to identify consecutive conversations. We further introduce “Advanced NCP” to make the pretraining task more relevant to the downstream task of segmentation break prediction while being significantly easier. Finally we introduce a curriculum to Advanced NCP (Curricular NCP) based on the similarity between pretraining and downstream task samples. Curricular NCP applied to a state-of-the-art model for text segmentation outperforms prior results. We also show that our pretraining strategies make the model robust to speech recognition errors commonly found in automatically generated transcripts.

## introduction text segmentation is the task of identifying segment breaks to organize a continuous text into semantically independent segments. prior research in text segmentation has largely focused on segmenting documents such as wikipedia articles (document segmentation) @xcite @xcite or dialogues such as chat or text messages (dialogue segmentation) @xcite @xcite . in this paper we address text segmentation of transcripts (transcript segmentation). figure 1 shows examples of segments in transcript data. transcript segmentation can help summarize long videos, podcasts or meetings by segmenting and summarizing the transcript such as "video * work done during internship at adobe research. chapters" in youtube videos or "outline generation" @xcite from documents. however, only few works have addressed segmentation of transcripts @xcite @xcite . as shown in figure 1 , transcripts consist of a mix of short sentences, utterances, interjections and long form document style answers. unlike wikipedia articles or chat, the sporadic and non uniform flow of text in transcripts makes annotation of segment breaks hard even for humans @xcite . furthermore, transcripts often involve automatic speech recognition errors such as insertions, deletions, replacement and lack of proper punctuation which add to the challenges. as a result of these challenges, most labeled transcript segmentation datasets are small in size making it difficult for models to be trained on them. to address this issue, we propose pretraining strategies that can be useful in resource constrained settings where huge labeled datasets are not available. our first strategy is next conversation prediction (ncp). in this strategy, pairs of conversations are classified into 1 or 0 based on whether they contiguously in the transcript or not. we hypothesize that the effectiveness of this pretraining task relies on its similarity and relevance to the segmentation task. our second strategy, advanced next conversation prediction (advanced ncp), introduces conditions on the ncp pretraining data to increase the relevance of the pretraining strategy to the segmentation task. our third strategy is curricular ncp where we pretrain the model in two distinct phases based on which pretraining samples are closest to the task of transcript segmentation. our experiments show that the application of the proposed pretraining strategies on multiple segmentation architectures outperforms their corresponding non pretrained versions. also, ncp does not rely on segmentation labels. we show that it is a strong unsupervised approach that outperforms state-of-the-art unsupervised model for transcript segmentation. finally, we observe that the pretraining strategies makes the model more robust to noise and better at predicting highly segmented regions of a transcript. our contributions are: ## related work text segmentation has been addressed in both unsupervised @xcite and supervised manner @xcite @xcite with early works focusing on unsupervised techniques @xcite @xcite . however, since the definition of a segment, could be highly domain and data dependant, supervised learning is desirable. @xcite and @xcite use lstms to identify if a sentence ends a segment or not. similarly, @xcite use grus and pointer-generator networks for this task. these works in segmentation propose a hierarchical approach, where the sentences are encoded into a fixed size representation followed by mapping the representations to a sequence of binary labels whether the current segment is ending at this sentence or not. @xcite use attention based cnn-lstms and phrase the task differently by providing inputs of a median sentence and its right and left context to identify segment breaks. @xcite simplify the new paradigm for this task by using the left and right contexts with respect to an end of a sentence as input. they were also the first to use large pretrained language models for this task. they establish a new sota in text segmentation. hence, we use their model as the base model in our pretraining experiments. very few works have focused on transcript segmentation. @xcite provide a podcast dataset for research in this domain and propose an lstm and universal sentence encoder @xcite based sequence labeling model. @xcite identify introductions in podcast transcripts using bert @xcite . @xcite propose an unsupervised technique for meeting transcript segmentation. they use large language model representations including sentence bert (reimers and gurevych, 2019) and bert to compute cosine similarity between subsequent conversations and estimate segment breaks. we present a new pretraining strategy aimed towards addressing transcript segmentation but not specific to any one model architecture. our work is also related to curriculum learning @xcite . it has gained popularity among nlp tasks such as sentiment analysis @xcite question answering @xcite , nlg @xcite and the glue benchmark @xcite . more recently, some works have used curriculum learning in the pretraining process of large language models. @xcite propose curriculum learning for pretraining the encoder of their speech translation system on multiple speech based tasks of varying difficulty. @xcite gradually introduce longer samples to bert's pretraining to observe performance improvements in resource poor settings. in our curriculum learning setting, we order pretraining samples based on their similarity to the downstream segmentation task. ## text segmentation as a binary classification task there are two major ways supervised text segmentation has been addressed in the past. first, by phrasing it as a sequence labeling problem, where each sentence has a label indicating if it ends the segment or not @xcite @xcite . as a result an entire transcript forms a single training instance consisting of a sequence of binary labels. second, by phrasing text segmentation as a binary classification task where we provide left context and right context around a sentence end and predict if the two contexts belong to the same segment or not. in particular, the input in this task is two text segments -left context (l) and right context (r) of the end of a sentence, each t tokens in length. the output in this task is 0 -if l and r belong to two different segments (segment break) and 1 if l and r belong to the same segment (not a segment break). in this setting, the number of instances is proportional to the number of segments. @xcite note in their experiments that the second setting outperforms the first on transcript segmentation. we follow the second setting and refer to it as the segmentation task in the rest of this paper. ## curricular next conversation prediction in this section, we explain the proposed pretraining strategy. first, we explain our basic pretraining strategy -next conversation prediction (ncp). then, we present improvements on this strategy to make the pretraining task easier and more relevant to the transcript segmentation task (advanced ncp). finally, we introduce curriculum learning to our pretraining strategy (curricular ncp) that presents the pretraining instances in an order that is more helpful for the segmentation task. ## experimental details in this section, we describe the dataset, the base model upon which our pretraining is tested, the implementational details, metrics and baselines. ## conclusion in this paper, we propose novel pretraining strategies for transcript segmentation. our pretraining strategies address major challenges associated with transcript data. the pretraining strategies are based on the idea of next conversation prediction. this strategy by itself also forms a strong unsupervised baseline for segmentation. additional improvements make ncp more relevant and useful to the segmentation task. we further introduced a curriculum in the pretraining strategies based on similarity of pretraining samples to the segmentation samples. our results showed that our proposed pretraining strategies are robust to noise in training data and they are effective for improving performance of multiple model architectures for segmentation.