Transformer over Pre-trained Transformer for Neural Text Segmentation with Enhanced Topic Coherence (2021)

Abstract: This paper proposes a transformer over transformer framework, called Transformerˆ2, to perform neural text segmentation. It consists of two components: bottom-level sentence encoders using pre-trained transformers, and an upper-level transformer-based segmentation model based on the sentence embeddings. The bottom-level component transfers the pre-trained knowledge learnt from large external corpora under both single and pair-wise supervised NLP tasks to model the sentence embeddings for the documents. Given the sentence embeddings, the upper-level transformer is trained to recover the segmentation boundaries as well as the topic labels of each sentence. Equipped with a multi-task loss and the pre-trained knowledge, Transformerˆ2 can better capture the semantic coherence within the same segments. Our experiments show that (1) Transformerˆ2$manages to surpass state-of-the-art text segmentation models in terms of a commonly-used semantic coherence measure; (2) in most cases, both single and pair-wise pre-trained knowledge contribute to the model performance; (3) bottom-level sentence encoders pre-trained on specific languages yield better performance than those pre-trained on specific domains.

## introduction text segmentation is an nlp task that aims to break text into topically coherent segments by identifying natural boundaries of changes of topics @xcite @xcite . it is critical in the sense that many downstream tasks can benefit from the resulting structured text, including text summarization, keyword extraction and information retrieval. both supervised and unsupervised learning have been applied to text segmentation. with the lack of large-quantity labels on supervised train-ing @xcite , unsupervised modeling based on clustering @xcite , bayesian methods @xcite @xcite and graph methods @xcite have been proposed. however, with the advancement of self-learning and transfer learning on deep neural networks, there are more recent supervised modeling approaches proposed that aim to predict labeled segment boundaries on smaller datasets. @xcite @xcite to the best of our knowledge, the most straightforward remedy to the above problems is knowledge transfer and distillation from pre-trained models. the rich pre-trained knowledge enables the training of a more general segmentation model on a small labeled dataset. in this paper, we propose a transformer over pre-trained transformer framework that allows different types of pre-trained information regarding sentences to be distilled to their classification for text segmentation. more specifically, the contributions of our paper are as follows: ## related work in this section, we review the past literature on the text segmentation models. these models can further be categorized into being unsupervised and supervised. ## transformer 2 architecture our proposed model adopts the popular two-level network architecture for text segmentation, which consists of a lower-level sentence encoder and an upper-level segment boundary classifier. our model aims to enhance the learning of semantic coherence between sentences from two aspects; 1) different pre-trained embeddings, generated from different nlp tasks on large external corpora, for the same sentences can capture rich and diverse information that the target corpus does not contain; 2) sentences within same segment(i.e. sharing same topic label) tend to be semantically more coherent than those across segments (i.e. with different topic labels). the above enhancements can further improve the segmentation performance of the transformer-based classifier. ## conclusion and future work in this paper, we propose a transformer over pre-trained transformer framework, called transformer 2 , for text segmentation with a focus on enhancing the learning of the semantic coherence between sentences. the bottom level of transformer 2 combines (untrainable and fixed) sentence embeddings outputted respectively from transformers pre-trained with both the single-sentence and the pairwise-sentence nlp tasks. an upper-level transformer is trained upon the combined sentence embeddings to minimize both the binary segmentation loss and the auxiliary topic prediction loss. the empirical results show that the best variants of transformer 2 outperform several state-of-the-art segmentation models, including the deep neural models, across four real-world datasets in terms of a commonly-used segment coherence measure p k . we have also conducted a comprehensive ablation study which shows that in most cases, each component of transformer 2 is helpful for boosting the segmentation performance. we have also found that using language-specific pre-trained transformers at the bottom level is more useful than using domain-specific ones. for the future work, we will investigate the efficacy of transformer 2 on helping the downstream nlp tasks such as text summarisation, keyword extraction and topic modelling.