Segmenting Natural Language Sentences via Lexical Unit Analysis (2021)

Abstract: The span-based model enjoys great popularity in recent works of sequence segmentation. However, each of these methods suffers from its own defects, such as invalid predictions. In this work, we introduce a unified span-based model, lexical unit analysis (LUA), that addresses all these matters. Segmenting a lexical unit sequence involves two steps. Firstly, we embed every span by using the representations from a pretraining language model. Secondly, we define a score for every segmentation candidate and apply dynamic programming (DP) to extract the candidate with the maximum score. We have conducted extensive experiments on 3 tasks, (e.g., syntactic chunking), across 7 datasets. LUA has established new state-of-the-art performances on 6 of them. We have achieved even better results through incorporating label correlations.

## introduction plenty of tasks in natural language understanding (nlu), such as syntactic chunking, are essentially a sequence segmentation problem, which partitions a sequence of lexical units into multiple labeled segments. a classical approach to sequence segmentation is to cast it into a sequence labeling task using iob tagging scheme @xcite @xcite . every token in a sentence, according to its position in the corresponding segment, is labeled with a tag (e.g., b-per). a representative work is bidirectional lstm-crf @xcite . recently, there is a surge of interest in developing span-based models @xcite @xcite @xcite . they regard spans rather than tokens as the basic units for labeling. for example, @xcite model named entity recognition (ner) as machine reading comprehension (mrc) @xcite , where entities are extracted as retrieving answer spans. while span-based models have achieved promising performances, they are locally normalized at span level, and therefore suffered from the label bias problem @xcite . moreover, some of them @xcite rely on heuristic rules to correct invalid predictions (e.g., span conflicts between two entities). early span-based models @xcite @xcite based on semi-markov crf @xcite adopts dynamic programming (dp) @xcite to search for the optimal segmentation of a sentence. unlike their counterparts @xcite @xcite , these methods all train the sentence encoders from scratch, without exploiting the knowledge from unlabeled corpora. hence, none of them is even competitive with current best sequence labeling model. in this paper, we propose lexical unit analysis (lua), a unified and effective span-based model that circumvents all above problems. our segmentation of a natural language sentence contains two steps. firstly, we utilize bert @xcite , a powerful pretraining language model, to get contextualized token representations, and with them we embed every span of the sentence, inspired by the finding that pretraining language models are very robust to rare tokens and the low-resource setting @xcite . then, we assign a score to every segmentation candidate and use dp to globally search for the candidate with the maximum score. the score of a segmentation is computed from the segment scores predicted by lua. we minimize the hinge loss, instead of cross-entropy, to train our models. we have performed extensive experiments on syntactic chunking, chinese part-of-speech (pos) tagging, and ner across 7 datasets. our model has achieved state-of-the-art results on 6 of them and performed competitively on the remaining one. besides, we have obtained slightly better performances by explicitly modeling the label dependencies. we also show that lua is very efficient in terms of running time. ## architecture output segments are represented as the segmentation @xmath0, y 2 , • • • , y m ] with each segment y k being a triple (i k , j k , t k ). n and m are respectively the numbers of lexical units and segments. (i k , j k ) is a span that corresponds to the phrase a segmentation is valid if all its segments are non-overlapping and fully cover the input sentence. an example from @xcite : ## experiments we have performed a series of studies to show the effectiveness and efficiency of lua. ## related work the traditional method to sequence segmentation converts it into a sequence labeling tasks with iob tagging scheme. this method is simple and effective, which has inspired a lot of well-performed models @xcite @xcite . for example, @xcite present flair embeddings that pre-trains character embedding in a large corpus and directly use it, instead of word representation, to encode a sentence. @xcite use hierarchical contextualized representations to incorporate both sentence-level and document-level information. recently, span-based models have received much attention. they treat a span, instead of a token, as the basic unit for labeling. for instance, @xcite ; @xcite rank all the spans in terms of the scores predicted by a biaffine model @xcite . spanbased models also emerge in other fields. @xcite integrate lstm-minus feature into constituent parsing models. ## conclusion this work presents a unified span-based model, lua, for neural sequence segmentation. given a natural language sentence, we use bert to encode it and apply dp to extract the segmentation candidate with the maximum score. extensive experiments have been conducted on 3 tasks across 7 datasets. lua has established new state-of-the-art results on 6 of them. we have gained further improvements through explicitly modeling the label dependencies among segments. lua is now adopted as an ner option in our online text understanding system, texsmart @xcite .