Improving Topic Segmentation by Injecting Discourse Dependencies (2022)

Abstract: Recent neural supervised topic segmentation models achieve distinguished superior effectiveness over unsupervised methods, with the availability of large-scale training corpora sampled from Wikipedia. These models may, however, suffer from limited robustness and transferability caused by exploiting simple linguistic cues for prediction, but overlooking more important inter-sentential topical consistency. To address this issue, we present a discourse-aware neural topic segmentation model with the injection of above-sentence discourse dependency structures to encourage the model make topic boundary prediction based more on the topical consistency between sentences. Our empirical study on English evaluation datasets shows that injecting above-sentence discourse structures to a neural topic segmenter with our proposed strategy can substantially improve its performances on intra-domain and out-of-domain data, with little increase of model’s complexity.

## introduction topic segmentation is a fundamental nlp task with the goal to separate textual documents into coherent segments (consisting of one or more sentences), following the document's underlying topical structure. the structural knowledge obtained from topic segmentation has been shown to play a vital role in key nlp downstream tasks, such as document summarization @xcite @xcite ), question answering @xcite and dialogue modeling @xcite . the aim of topic segmentation makes it tightly connected to related research areas aiming to understand the latent structure of long and potentially complex text. specifically, understanding the semantic and pragmatic underpinnings of a document can arguably support the task of separating continuous text into topical segments. to this end, figure 1 : an example article about cholinergic urticaria (cu) sampled from the en_disease portion of wiki-section dataset @xcite . left: discourse dependency structure predicted by the sent-first discourse parser @xcite . we further motivate the synergistic relationship between topic segmentation and discourse analysis/parsing in figure 1 , showing anecdotal evidence of the alignment between the document's topical structure and the respective rst-style discourse dependency graph. starting from a sequence of sentences, the task of topic segmentation addresses the problem of splitting the given wikipedia article into an ordered set of topical-coherent fragments (here: t1, t2 and t3) by predicting topical boundaries. as shown in the example, the document discourse tree is indicative of the topical structure of the document, as discourse dependencies occur considerably more often within a topic segment than across topic segments. given significant influence on a variety of realworld tasks, topic segmentation is an active research area in the field of nlp. as such, modern, neural methods for monologue topic segmentation are proposed by formulating the task as a sentencelevel sequence labeling problem, trained and evaluated on the large-scale wikipedia dataset @xcite @xcite . these wikipedia articles are well-suited for the task of topic segmentation, providing natural section marks which can be reasonably used as ground-truth segment boundaries @xcite , superseding previously proposed unsupervised methods @xcite @xcite . despite the significant improvements achieved by neural supervised topic segmentation models, it remains unclear if these topic segmenters effectively learn to cluster sentences into topical-coherent pieces based on the (document-level) topical consistency, or solely exploit superficial patterns (e.g., simple linguistic cues) in the training domain. to address this challenge, in this paper, we propose a more discourse-aware neural topic segmentation model. we thereby inject above-sentence discourse structures into basic topic segmenter to encourage the model to base its topic boundary prediction more explicitly on the topical consistency between sentences. more specifically, we propose to exploit a discourse dependency parser pre-trained on out-of-domain data to induce inter-sentential discourse dependency trees. subsequently, we convert the dependency tree into a directed discourse graph with sentences as nodes and discourse dependencies as edges. with the generated discourse graph, a graph attention network (gat) @xcite is used to encode sentences as discoursecontextualized representations by aggregating information from neighboring sentence nodes in the graph. finally, the discourse-infused sentence representations are concatenated with standard encodings for segment boundary prediction. in our empirical study conducted on english evaluation datasets, we show that: (i) injecting discourse structures can substantially improve the performance of the basic neural topic segmentation model on three datasets. (ii) our novel, discourseenhanced topic segmenter is more robust compared to the basic neural model in settings that require domain transfer, showing superior performance on four challenging real-world test sets, to confirm the improved domain-independence. (iii) even if our proposal has inferior accuracy against a state-ofthe-art segmenter sharing the same basic architecture, it does achieve significantly better efficiency assessed by model's parameter size and speeds for learning and inference, which makes it potentially more favorable in real-world use. more recently, the data sparsity issue has been alleviated by the proposal of large-scale corpora sampled from wikipedia (e.g., wiki-727k @xcite and wiki-section ( arnold et al., 2019) ), in which well-structured articles with their section marks are used as gold labels for segment boundaries. as a result, neural supervised topic segmenters started to gain attention by reaching greater effectiveness and efficiency compared to previously proposed unsupervised approaches. these supervised topic segmenters typically follow a common strategy which formulates the task as a sentencelevel sequence labeling problem. more specifically, by assigning binary labels to each sentence, models infer the likelihood of a sentence to be a topic segment boundary @xcite @xcite . however, we believe that current models, besides reaching promising performance, potentially favour simple linguistic cues over effective measurements for semantic cohesion, restricting their application to narrow domains. some recent works have attempted to address this limitation via explicitly integrating coherence modeling components into segmenters @xcite . however, compared to our objective in this work, these proposed coherence modeling strategies are either (i) only taking two adjacent sentences into account, limiting the additional module to extremely local contexts, or (ii) discriminating real documents from artificially "incoherent" texts, resulting in implicit and synthetic negative training samples and heavy parameter size caused by modeling multiple tasks simultaneously. in contrast, we propose an effective method to integrate the document discourse (dependency) structure into neural topic segmentation frameworks, following the intuition that above-sentence discourse structure are indicative of text coherence and topical consistency, providing a more global and interpretable source of information for better topic transition prediction. ## methodology as shown in figure 2 , our proposed discourseaware neural topic segmentation model comprises two components: the hierarchical topic segmenter and discourse graph modeling, highlighted in green and red respectively. discourse graph modeling further comprises of a discourse graph construction and graph modeling component. ## experiments in order to quantitatively evaluate the effectiveness, generality and efficiency of our proposal, we conduct three sets of experiments to compare our topic segmentation approach against a variety of baselines and previous models. namely, we assess the performance of our model in regards to the intra-domain segment inference performance, domain transfer segment inference performance, and conduct an additional efficiency analysis. ## conclusion and future work in this paper, we present a neural topic segmentation model with injection of above-sentence discourse dependency structures inferred from a stateof-the-art discourse dependency parser. different from previously proposed methods, our segmenter leverages the discourse signal by encoding the topical consistency between sentences from a more global and interpretable point of view. experiments on multiple settings (intra-domain, domain transfer and efficiency comparison) show that our system achieves comparable performance to one of the current top-performing topic segmenters, with much less model size increase and speed degradation. in the near future, we plan to investigate the synergy between topic segmentation and discourse parsing more comprehensively, by incorporating the type of inter-sentential rhetorical relations and analyzing whether and how this discourse knowledge can enhance supervised topic segmentation frameworks. in the long run, we intend to explore the possibility for discourse parsing to benefit segment topic labeling, which is another important task usually coupled together with topic segmentation to provide the coarse-grained structural information for documents. particularly, we believe discourse parsing can potentially enhance the step of key phrase extraction in segment topic labeling due to the significant improvement it brings to the related task of name entity recognition (ner) (jie and lu, 2019).