Improving Context Modeling in Neural Topic Segmentation (2020)

Abstract: Topic segmentation is critical in key NLP tasks and recent works favor highly effective neural supervised approaches. However, current neural solutions are arguably limited in how they model context. In this paper, we enhance a segmenter based on a hierarchical attention BiLSTM network to better model context, by adding a coherence-related auxiliary task and restricted self-attention. Our optimized segmenter outperforms SOTA approaches when trained and tested on three datasets. We also the robustness of our proposed model in domain transfer setting by training a model on a large-scale dataset and testing it on four challenging real-world benchmarks. Furthermore, we apply our proposed strategy to two other languages (German and Chinese), and show its effectiveness in multilingual scenarios.

## introduction topic segmentation is a fundamental nlp task that has received considerable attention in recent years @xcite @xcite . it can reveal important aspects of a document semantic structure by splitting the document into topical-coherent textual units. taking the wikipedia article in a wide variety of techniques have been proposed for topic segmentation. early unsupervised models exploit word statistic overlaps @xcite , bayesian contexts @xcite or semantic relatedness graphs @xcite to measure the lexical or semantic cohesion between the sentences or paragraphs and infer the segment boundaries from them. more recently, several works have framed topic segmentation as neural supervised learning, because of the remarkable success achieved by such models in most nlp tasks @xcite @xcite @xcite . despite minor architectural differences, most of these neural solutions adopt recurrent neural network @xcite and its variants (rnns) as their main framework. on the one hand, rnns are appropriate because topic segmentation can be modelled as a sequence labeling task where each sentence is either the end of a segment or not. on the other hand, this choice makes these neural models limited in how to model the context. because some sophisticated rnns (eg., lstm, gru) are able to preserve long-distance information @xcite @xcite , which can largely help language models. but for topic segmentation, it is critical to supervise the model to focus more on the local context. as illustrated in in this paper, we propose to enhance a state-ofthe-art (sota) topic segmenter @xcite based on hierarchical attention bilstm network to better model the local context of a sentence in two complementary ways. first, we add a coherence-related auxiliary task to make our model learn more informative hidden states for all the sentences in a document. more specifically, we refine the objective of our model to encourage smaller coherence for the sentences from different segments and larger coherence for the sentences from the same segment. secondly, we enhance context modeling by utilizing restricted self-attention @xcite , which enables our model to pay attention to the local context and make better use of the information from the closer neighbors of each sentence (i.e., with respect to a window of explicitly fixed size k). our empirical results show (1) that our proposed context modeling strategy significantly improves the performance of the sota neural segmenter on three datasets, (2) that the enhanced segmenter is more robust in domain transfer setting when applied to four challenging real-world test sets, sampled differently from the training data, (3) that our context modeling strategy is also effective for the segmenters trained on other challenging languages (eg., german and chinese), rather than just english. more recently, neural-based supervised methods have been devised for topic segmentation because of their more accurate predictions and greater efficiency. one line of research frames topic segmentation as a sequence labeling problem and builds neural models to predict segment boundaries directly. @xcite proposed a simple bilstm model to label if a sentence is a segment boundary or not. they demonstrated that along with engineered features based on cue phrases (eg., 'first of all', 'second'), their model can achieve marginally better performance than early unsupervised methods. later, @xcite proposed a hierarchical neural sequence labeling model for topic segmentation and showed its superiority compared with their selected supervised and unsupervised baselines. around the same time, @xcite proposed an attention-based bilstm model to classify whether a sentence was a segment boundary or not, by considering the context around it. the work we present in this paper can be seen as pushing this line of research even further by encouraging the model to more explicitly consider contextual coherence, as well as to prefer more information from the neighbor context through restricted self-attention. another rather different line of works first trains neural models for other tasks, and then uses these models' outputs to predict boundaries. @xcite trained a convolutional neural network (cnn) network to predict the coherence scores for text pairs. sentences in a pair with large cohesion are supposed to belong to the same segment. however, their "learning to rank" framework asks for the pre-defined number of segments, which limits their model's applicability in practice. our selected framework overcomes this constraint by tuning a confidence threshold during the training stage. a sentence with the output probability above this threshold will be predicted as the end of a seg-ment. following a very different approach, @xcite introduced a topic embedding layer into a bilstm model. after training their model to predict the sentence topics, the learned topic embeddings can be utilized for topic segmentation. however, one critical flaw of their method is that it requires a complicated pre-processing pipeline, which includes topic extraction and synset clustering, whose errors can propagate to the main topic segmentation task. in contrast, our proposal only requires the plain content of the training data without any complex pre-processing. coherence modeling early works on coherence modeling merely predict the coherence score for documents by tracking the patterns of entities' grammatical role transition @xcite . more recently, researchers started modeling the coherence for sentence pairs by their semantic similarities and used them for higher level coherence prediction or even other tasks, including topic segmentation. @xcite demonstrated the strong relation between text-pair coherence modeling and topic segmentation. they assumed that (1) a pair of texts from the same document should be ranked more coherent than a pair of texts from different documents; (2) a pair of texts from the same segment should be ranked more coherent than a pair of texts from different segments of a document. with these assumptions, they created a "quasi" training corpus for text-pair coherence prediction by assigning different coherence scores to the texts from the same segment, different segments but the same document, and different documents. then they proposed the corresponding model, and further use this model to directly conduct topic segmentation. following their second assumption, we propose a neural solution in which by injecting a coherence-related auxiliary task, topic segmentation and sentence level coherence modeling can mutually benefit each other. ## neural topic segmentation model since rnn-based topic segmenters have shown success with high-quality training data, we adopt a state-of-the-art rnn-based topic segmenter enhanced with attention and bert embeddings as our basic model. then, we extend such model to make better use of the local context, something that cannot be done effectively within the rnn framework @xcite . in particular, we add a coherence-related auxiliary task and a restricted self-attention mechanisms to the basic model, so that predictions are more strongly influenced by the coherence between the nearby sentences. as a preview of this section, we first define the problem of topic segmentation and introduce the basic model. in the next section, we motivate and describe our proposed extensions. ## experimental setup in order to comprehensively evaluate the effectiveness of our context modeling strategy of adding a coherence-related auxiliary task and a restricted self-attention mechanisms to the basic model, we conduct three sets of experiments for evaluation: we split choi and rules into 80% training, 10% validation, 10% testing. for section, we follow @xcite and split it into 70% training, 10% validation, 20% testing. ## conclusions and future work we address a serious limitation of current neural topic segmenters, namely their inability to effectively model context. to this end, we propose a novel neural model that adds a coherence-related auxiliary task and restricted self-attention on top of a hierarchical bilstm attention segmenter to make better use of the contextual information. experimental results of intra-domain on three datasets show that our strategy is effective within domains. further, results on four challenging real-world benchmarks demonstrate its effectiveness in domain transfer settings. finally, the application to other two languages (german and chinese) suggests that our strategy has its potential in multilingual scenarios. as future work, we will investigate whether our proposed context modeling strategy is also effective for segmenting dialogues @xcite rather than just standard articles. secondly, we will explore how to capture even more accurate and informative contextual information by integrating document structures or sentence dependencies obtained from other nlp tasks (e.g., discourse parsing @xcite or discourse role labeling @xcite ).