Neural Sequence Segmentation as Determining the Leftmost Segments (2021)

Abstract: Prior methods to text segmentation are mostly at token level. Despite the adequacy, this nature limits their full potential to capture the long-term dependencies among segments. In this work, we propose a novel framework that incrementally segments natural language sentences at segment level. For every step in segmentation, it recognizes the leftmost segment of the remaining sequence. Implementations involve LSTM-minus technique to construct the phrase representations and recurrent neural networks (RNN) to model the iterations of determining the leftmost segments. We have conducted extensive experiments on syntactic chunking and Chinese part-of-speech (POS) tagging across 3 datasets, demonstrating that our methods have significantly outperformed previous all baselines and achieved new state-of-the-art results. Moreover, qualitative analysis and the study on segmenting long-length sentences verify its effectiveness in modeling long-term dependencies.

## introduction sequence segmentation, as an important task in natural language understanding (nlu), partitions a sentence into multiple segments. the first two rows of in previous works, there are two dominant approaches to sequence segmentation. the most common is to regard it as a sequence labeling problem with resorting to iob tagging scheme @xcite @xcite . this method is simple yet very effective, providing tons of state-of-the-art performances. for example, @xcite present bidirectional lstm-crf for named entity recognition (ner) and pos tagging, which adopts bilstm (hochre-sentence tangible capital will be about @xmath0 115 million", consumes 6 iob tags or 8 transition actions. this ill-posed design certainly limits the full potential of segmentation models to capture the long-term dependencies among segments. in this paper, we introduce a novel framework that incrementally segments a sentence at segment level. the segmentation process is iterative and incremental. at each iteration, the proposed framework determines the leftmost segment of the remaining unprocessed sequence. under this scheme, we don't resort to token-level labels and enjoy linear time complexity. the implementation contains two stages. firstly, we utilize lstm-minus @xcite technique to construct the representations for all the phrases. secondly, we adopt lstm to model the iterative segmentation process, which captures the strong correlation among segments. at every step, the input consists of the previous segment and the remaining unprocessed sequence, and the output is the leftmost segment. figure 1 depicts how our framework segments the sentence in extensive experiments have been conducted on syntactic chunking and chinese part-of-speech (pos) tagging across 3 datasets. the proposed framework has obtained new state-of-the-art performances on all of them. besides, qualitative study and the results on segmenting long-length sentences confirm its effectiveness in capturing long-term dependencies. our contributions are as follows: the source code of this work is available at https://github.com/leepleased/leftmostseg . ## architecture we denote a n-length input sentence as @xmath1, x 2 , • • • , x n ], where x i is a token (such as a word or a character). the output segments are represented as @xmath2, y 2 , • • • , y m ], where m is the amount of segments. every segment y k is denoted as a triple (i k , j k , l k ). (i k , j k ) is the span of the segment, corresponding to the phrase ## while x is not empty do 6 get lstm hidden state h d k by using equation 5. ## experiments extensive experiments have been conducted on syntactic chunking and chinese pos tagging across 3 datasets. firstly, our models have obtained new state-of-the-art performances on all the datasets. then, we have investigated ablation studies to understand the importance of each component. lastly, case study and the results on segmenting longlength sentences confirm the effectiveness of the proposed framework in capturing the long-term dependencies among segments. ## related work there are two mainstream approaches to sequence segmentation. one of them treat sequence segmentation as a sequence labeling problem by using iob tagging scheme @xcite @xcite @xcite . each token in a sentence is labeled as b-tag if it's the beginning of a segment, i-tag if it is inside but not the first one within the segment, or o otherwise. this method is extensively studied by prior works and provides tons of state-of-the-art results. @xcite introduce a funnel-shaped convolutional architecture that learns a better internal structure for the tokens. @xcite propose an efficient characterlevel framework that uses pretrained character embedding. @xcite adopt semi-supervised learning to train lstm encoder using both labeled and unlabeled corpora. despite the effectiveness, these models are at token level, relying on multiple token-level labels to represent a single segment. this limits their full potential to capture the longterm dependencies among segments. the other uses transition-based systems as the backbone to incrementally segment and label an input sequence @xcite @xcite . for example, @xcite design special transition actions to jointly segment, tag, and normalize a sentence. these models have many advantages, such as theoretically lower time complexity and capturing non-local features. however, they are still token-level models, which predict transition actions to shift a token from the buffer to the stack or assign a label to a span. recently, there is a surge of interest in developing span-based models, such as segmental rnn @xcite and lua @xcite . these methods circumvent using token-level labels and directly label the phrases in a sentence. span-based models also enjoy great popularity in language modeling @xcite , ner @xcite , and constituent parsing @xcite . however, its underperforms current token-level models (see section 3.2 and section 3.3) and is very slow in terms of running time (see section 3.7). ## conclusion in this work, we present a novel framework to sequence segmentation that segments a sentence at segment level. the segmentation process is iterative and incremental. for every step, it determines the leftmost segment of the remaining sequence. implementations involve lstm-minus to extract phrase representations and rnn to model the iterations of leftmost segment determination. extensive experiments have been conducted on syntactic chunking and chinese pos tagging across 3 datasets. we have achieved new state-of-the-art performances on all of them. case study and the results on segmenting long-length sentences both verify the effectiveness of our framework in modeling long-term dependencies.