A Joint Model for Document Segmentation and Segment Labeling (2020)

Abstract: Text segmentation aims to uncover latent structure by dividing text from a document into coherent sections. Where previous work on text segmentation considers the tasks of document segmentation and segment labeling separately, we show that the tasks contain complementary information and are best addressed jointly. We introduce Segment Pooling LSTM (S-LSTM), which is capable of jointly segmenting a document and labeling segments. In support of joint training, we develop a method for teaching the model to recover from errors by aligning the predicted and ground truth segments. We show that S-LSTM reduces segmentation error by 30% on average, while also improving segment labeling.

## introduction a well-written document is rich not only in content but also in structure. one type of structure is the grouping of content into topically coherent segments. these segmented documents have many uses across various domains and downstream tasks. segmentation can, for example, be used to convert unstructured medical dictations into clinical reports @xcite , which in turn can help with medical coding (since a diagnosis mentioned in a "medical history" might be different from a diagnosis mentioned in an "intake" section @xcite ). segmentation can also be used downstream for retrieval @xcite @xcite , where it can be particularly useful when applied to informal text or speech that lacks explicit segment markup. topically segmented documents are also useful for pre-reading (the process of skimming or surveying a text prior to careful reading), thus serving as an aid for reading comprehension @xcite . uncovering latent, topically coherent segments of text is a difficult problem because it requires solving a chicken-and-egg problem: determining the segment topics is easier if segment boundaries are given, and identifying the boundaries of segments is easier if the topic(s) addressed in parts of the document are known. prior approaches to text segmentation can largely be split into two categories that break the cycle by sequentially solving the two problems: those that attempt to directly predict segment bounds @xcite , and those that attempt to predict topics per passage (e.g., per sentence) and use measures of coherence for post hoc segmentation @xcite @xcite @xcite . the benefit of the topic modeling approach is that it can work in unsupervised settings where collecting ground truth segmentations is difficult and labeled data is scarce @xcite . recent work uses wikipedia as a source of segmentation labels by eliding the segment bounds of a wikipedia article to train supervised models @xcite . this enables models to directly learn to predict segment bounds or to learn sentence-level topics and perform post hoc segmentation. our work is motivated by the observation that the segment bounds and topicality are tightly interwoven, and should ideally be considered jointly rather than sequentially. we start by examining three properties about text segmentation: (1) segment bounds and segment labels contain complementary supervisory signals, (2) segment labels are a product of lower level (e.g. sentence) labels which must be composed, and (3) the model should not only learn to label from ground-truth segmentations at training time, but instead the labeler should learn to be robust to segmentation errors. these properties build on previous work discussed in section 2. we experimentally evaluate and verify each of these properties in section 5 with respect to a document segmentation and segment labeling task. taking advantage of these properties, we propose a neural model that jointly segments and labels without committing to a priori segmentations, segment pooling lstm (s-lstm). it consists of three components: a segment proposal lstm (discussed in section 3.2), a segment pooling layer (section 3.3), and a segment aligner for training and evaluation (section 3.4). our main contribution is a model that performs segmentation and labeling jointly rather than separately. by virtue of joint inference, our model takes advantage of the complementary supervisory signals for segmentation and topic inference, considers the contribution of all sentences to the segment label, and avoids committing to early errors in low-level inference. our approach improves over neural and nonneural baselines of a document segmentation task. we use a dataset of wikipedia articles described in section 5 for training and evaluation. we show that s-lstm is capable of reducing segmentation error by, on average, 30% while also improving segment classification. we also show that these improvements hold on out-of-domain datasets. ## related work coherence-based segmentation. much work on text segmentation uses measures of coherence to find topic shifts in documents. @xcite introduced the texttiling algorithm, which uses term co-occurrences to find coherent segments in a document. @xcite introduced bayesseg, a bayesian method that can incorporate other features such as cue phrases. @xcite later introduced topictiling, which uses coherence shifts in topic vectors to find segment bounds. @xcite proposed graphseg, which constructs a semantic relatedness graph over the document using lexical features and word embeddings, and segments using cliques. @xcite proposed sits, a model for topic segmentation in dialogues that incorporates a per-speaker likelihood to change topics. while the above models are unsupservised, arnold et al. introduced a supervised method to compute sentence-level topic vectors using wikipedia articles. the authors created the wiki-section dataset and proposed the sector neural model. the sector model predicts a label for each sentence, and then performs post hoc segmentation looking at the coherence of the latent sentence representations, addressing segmentation and labeling separately. we propose a model capable of jointly learning segmentation boundaries and segment-level labels at training time. our segmentation does not rely on measures of coherence, and can instead learn from signals in the data, such as cue phrases, to predict segment bounds, while still performing well at the segment labeling task. ## modeling in order to jointly model document segmentation and segment classification, we introduce the segment pooling lstm (s-lstm) model. s-lstm is a supervised model trained to both predict segment bounds and pool over and classify the segments. the model consists of three components: a sentence encoder (section 3.1), a segment predictor lstm (section 3.2), and a segment pooling network which pools over predicted segments to classify them (section 3.3). the segment predictor is allowed to make mistakes that the labeler must learn to be robust to, a process which we refer to as exploration, and accomplish by aligning predicted and ground truth segments (section 3.4). the full architecture is presented in figure 1 , and the loss is discussed in section 3.5. ## experimental setup we follow the experimental procedure of @xcite to evaluate s-lstm for the tasks of document segmentation and segment labeling. ## results and analysis there are five major takeaways from the experimental results and analysis. first, the jointly trained s-lstm model shows major improvement over prior work that modeled document segmentation and segment labeling tasks separately. second, segment alignment and exploration during training reduces error rates. third, the segment pooling layer leads to improvements for both segmentation and segment labeling. fourth, s-lstm outperforms an iob-tagging crf-decoded model for single label segment labeling, and also generalizes easily wikisection-topics single-label classification en_disease 27 topics de_disease 25 topics en_city 30 topics de_city 27 topics model configuration @xcite , textseg @xcite , and c99 @xcite , and the best neural sector models from arnold et al.. wikisection-headings multi-label classification en_disease 179 topics de_disease 115 topics en_city 603 topics de_city 318 topics model configuration and tractably to multi-labeling. fifth, a deeper analysis of the joint modeling demonstrates that segment labeling and segment bound prediction contain complementary information. ## conclusion and future work in this paper we introduce the segment pooling lstm (s-lstm) model for joint segmentation and segment labeling tasks. we find that the model dramatically reduces segmentation error (by 30% on average across four datasets) while improving segment labeling accuracy compared to previous neural and non-neural baselines for both singlelabel and multi-label tasks. experiments demonstrate that jointly modeling the segmentation and segment labeling, segmentation alignment and exploration, and segment pooling each contribute to s-lstm's improved performance. s-lstm is agnostic as to the sentence encoder used, so we would like to investigate the potential usefulness of transformer-based language models as sentence encoders. there are additional engineering challenges associated with using models such as bert as sentence encoders, since encoding entire documents can be too expensive to fit on a gpu without model parallelism. we would also like to investigate the usefulness of an unconsidered source of document structure: the hierarchical nature of sections and subsections. like segment bounds and headers, this structure is naturally available in wikipedia. having shown that segment bounds contain useful supervisory signal, it would be interesting to examine if segment hierarchies might also contain useful signal.