Training and Domain Adaptation for Supervised Text Segmentation (2021)

Abstract: Unlike traditional unsupervised text segmentation methods, recent supervised segmentation models rely on Wikipedia as the source of large-scale segmentation supervision. These models have, however, predominantly been evaluated on the in-domain (Wikipedia-based) test sets, preventing conclusions about their general segmentation efficacy. In this work, we focus on the domain transfer performance of supervised neural text segmentation in the educational domain. To this end, we first introduce K12Seg, a new dataset for evaluation of supervised segmentation, created from educational reading material for grade-1 to college-level students. We then benchmark a hierarchical text segmentation model (HITS), based on RoBERTa, in both in-domain and domain-transfer segmentation experiments. While HITS produces state-of-the-art in-domain performance (on three Wikipedia-based test sets), we show that, subject to the standard full-blown fine-tuning, it is susceptible to domain overfitting. We identify adapter-based fine-tuning as a remedy that substantially improves transfer performance.

## introduction organizing long texts into coherent segments facilitates human text comprehension as well as downstream tasks like text summarization @xcite , passage retrieval @xcite @xcite , and sentiment analysis @xcite ). text segmentation is very important in the educational domain as it enables large-scale passage extraction. educators, for example, need to extract coherent passage segments from books to create reading materials for students. similarly, test developers must create reading assessments at scale by extracting coherent segments from a variety of sources. most segmentation models allow for (i.e., sequential) segmentation @xcite @xcite @xcite , though methods for hierarchical segmentation have been proposed as well @xcite @xcite . owing to the absence of large annotated datasets, (linear) text segmentation has long been limited to unsupervised models, relying on various measures of lexical and semantic sentence overlap @xcite @xcite @xcite and topic modeling @xcite @xcite . more recently, @xcite automatically created a large segment-annotated dataset wiki727 by leveraging the headings structure in wikipedia articles, effectively enabling supervised text segmentation; they then trained a hierarchical recurrent neural segmentation model on wiki727. in subsequent work, @xcite improved on their segmentation performance by (i) replacing recurrent components of the hierarchical model with transformer networks @xcite and (ii) adding an auxiliary self-supervised coherence objective. although both @xcite and @xcite report massive gains over unsupervised baselines, their models have mostly been subject to in-domain evaluation on test sets also derived from wikipedia. in this work, in contrast, we concern ourselves with domain transfer of supervised text segmentation models, with a focus on the educational domain. to investigate the effects of domain transfer in supervised text segmentation, we first introduce k12seg -a segment-annotated dataset which we automatically created from educational texts designed for grade-1 to college-level student population. we then benchmark a hierarchical neural text segmentation model (hits) in a range of indomain and domain-transfer segmentation experiments involving wiki727 @xcite and our new k12seg dataset. our hits model, illustrated in figure 1 , though similar to the hierarchical segmentation models of @xcite , differs in two crucial aspects. first, we initialize the parameters of the lower (token-level) transformer with the weights of the pretrained roberta @xcite . secondly, aiming to prevent both (1) forgetting of distributional information captured in roberta's parameters and (2) overfitting to the training domain -we augment the layers of the token-level transformer with adapter parameters @xcite before segmentation training. adapter-based fine-tuning only updates the additional adapter parameters and the original transformer parameters are frozen: this fully preserves the distributional knowledge obtained in transformer's pretraining. encoding out-of-domain segmentation knowledge (e.g., from the wiki727 dataset) separately from the distributional information (original roberta parameters) allows to combine the two types of information more flexibly during the secondary segmentation training in the target domain (e.g., on k12seg), resulting in more effective domain transfer. experimental results confirm the above expectations. our adapter-augmented hits model trained on wiki727, besides yielding state-of-the-art indomain (wikipedia) segmentation performance, facilitates domain transfer and leads to substantial gains in our target educational domain (k12seg). adapter-based training. unlike @xcite ; glavaš and somasundaran (2020), we initialize the lower transformer with roberta weights, encoding general-purpose distributional knowledge. full fine-tuning, in which all transformer parameters are updated in downstream training, may overwrite useful distributional signal with domain-specific artefacts, overfit the model to the training domain, and impede domain transfer for the downstream task @xcite . alternative, adapter-based fine-tuning @xcite @xcite , injects additional adapter parameters into transformer layers and updates only them during downstream fine-tuning, keeping the original transformer parameters unchanged. we adopt the bottleneck adapter architecture of @xcite , reported effective for a wide range of downstream task. concretely, in each layer of the lower transformer, we insert two bottleneck adapters: one after the multi-head attention sublayer and another after the feed-forward sublayer. let @xmath0, input to the adapter layer. the adapter then yields the following output: the parameter matrix w @xmath1, and w @xmath2; g is the non-linear activation function. training instances and inference. we train the model with sequences of n sentences as instances which we create by sliding the window of size n over document's sentences, with a sliding step of n/2. at inference, for each sentence s, we make predictions for all of the windows that contain s. this means that we obtain (at most) n segmentation probabilities for each sentence (for the i-th sentence, we get predictions from windows [i-n +1 : i], [i-n +2 : i+1], . . . , [i : i+n -1]). we average the sentence's segmentation probabilities obtained across different windows and predict that it starts a new segment if the average is above the threshold t. we treat the sequence length n and threshold t as hyperparameters and optimize them using the development datasets. for brevity, we describe the optimization details in the appendix. ## conclusion supervised text segmentation has been limited to a single large-scale segmentation dataset, wiki727, built automatically from wikipedia. in this work, we studied domain transfer for supervised text segmentation: we introduce k12seg, a new dataset for supervised text segmentation built from educational reading materials (grade-1 to college-level students), and use it together with wiki727 in our domain transfer experiments. our hierarchical segmentation model (hits) couples the pretrained roberta with the upper-level transformer that provides sentence contextualization. we show that hits obtains state-of-the-art performance on standard wikipedia-based evaluation datasets, but overfits to the training domain (wikipedia). we finally substantially improve model's transfer capabilities through adapter-based fine-tuning.