A Language Model-based Generative Classifier for Sentence-level Discourse Parsing (2021)

Abstract: Discourse segmentation and sentence-level discourse parsing play important roles for various NLP tasks to consider textual coherence. Despite recent achievements in both tasks, there is still room for improvement due to the scarcity of labeled data. To solve the problem, we propose a language model-based generative classifier (LMGC) for using more information from labels by treating the labels as an input while enhancing label representations by embedding descriptions for each label. Moreover, since this enables LMGC to make ready the representations for labels, unseen in the pre-training step, we can effectively use a pre-trained language model in LMGC. Experimental results on the RST-DT dataset show that our LMGC achieved the state-of-the-art F1 score of 96.72 in discourse segmentation. It further achieved the state-of-the-art relation F1 scores of 84.69 with gold EDU boundaries and 81.18 with automatically segmented boundaries, respectively, in sentence-level discourse parsing.

## introduction textual coherence is essential for writing a natural language text that is comprehensible to readers. to recognize the coherent structure of a natural language text, rhetorical structure theory (rst) is applied to describe an internal discourse structure for the text as a constituent tree @xcite . a discourse tree in rst consists of elementary discourse units (edus), spans that describe recursive connections between edus, and nuclearity and relation labels that describe relationships for each connection. figure 1 (a) shows an example rst discourse tree. a span including one or more edus is a node of the tree. given two adjacent non-overlapping spans, their nuclearity can be either nucleus or satellite, denoted by n and s, where the nucleus represents a more salient or essential piece of information than the satellite. furthermore, a relation label, such as attribution and elaboration, is used to describe the relation between the given spans @xcite . to build such trees, rst parsing consists of discourse segmentation, a task to detect edu boundaries in a given text, and discourse parsing, a task to link spans for detected edus. in this paper, we focus on discourse segmentation and sentence-level discourse parsing, which are indispensable in rst parsing @xcite @xcite @xcite and are applicable to many downstream tasks, such as machine translation @xcite and sentence compression @xcite . in discourse segmentation, @xcite proposed a method for using lexical information and syntactic parsing results. many researchers @xcite @xcite ) utilized these clues as features in a classifier although automatic parsing errors degraded segmentation performance. to avoid this problem, @xcite used bilstm-crf @xcite to handle an input without these clues in an end-to-end manner. @xcite jointly performed discourse segmentation and sentence-level discourse parsing in their pointer-network-based model. they also intro-duced multi-task learning for both tasks and reported the state-of-the-art results for discourse segmentation and sentence-level discourse parsing in terms of f 1 scores. despite these achievements, there is still room for improvement for both tasks due to the scarcity of labeled data. it is important to extract more potential information from the current dataset for further performance improvement. under this motivation, in this research, we propose a language model-based generative classifier (lmgc) as a reranker for both discourse segmentation and sentence-level discouse parsing. lmgc can jointly predict text and label probabilities by treating a text and labels as a single sequence, like figure 1 @xcite . therefore, different from conventional methods, lmgc can use more information from labels by treating the labels as an input. furthermore, lmgc can enhance label representations by embedding descriptions of each label defined in the annotation manual @xcite , that allows us to use a pre-trained language model such as mpnet @xcite ) effectively, since we can already have the representations for labels, that were unseen in the pre-training step. experimental results on the rst-dt dataset @xcite show that lmgc can achieve the state-of-the-art scores in both discourse segmentation and sentence-level discourse parsing. lmgc utilizing our enhanced label embeddings achieves the best f 1 score of 96.72 in discourse segmentation. furthermore, in sentence-level discourse parsing, lmgc utilizing our enhanced relation label embeddings achieves the best relation f 1 scores of 84.69 with gold edu boundaries and 81.18 with automatically segmented boundaries, respectively. ## related work discourse segmentation is a fundamental task for building an rst discourse tree from a text. @xcite proposed a method for using lexical information and syntactic parsing results for detecting edu boundaries in a sentence. @xcite ; xuan @xcite ; feng and hirst (2014b) utilized these clues as features in a classifier, while @xcite utilized bilstm-crf @xcite in an endto-end manner to avoid performance degradation caused by syntactic parsing errors. sentence-level discourse parsing is also an important task for parsing an rst discourse tree, as used in many rst parsers @xcite @xcite @xcite . recently, @xcite tried to jointly perform discourse segmentation and sentence-level discourse parsing with pointer-networks and achieved the state-ofthe-art f 1 scores in both discourse segmentation and sentence-level discourse parsing. in spite of the performance improvement of these models, a restricted number of labeled rst discourse trees is still a problem. in the discourse segmentation and parsing tasks, most prior work is on the basis of discriminative models, which learn mapping from input texts to predicted labels. thus, there still remains room for improving model performance by considering mapping from predictable labels to input texts to exploit more label information. to consider such information in a model, @xcite introduced a generative model-based parser, rnng @xcite , to document-level rst discourse parsing. different from our lmgc, this model unidirectionally predicts action sequences. in this research, we model lmgc for the discourse segmentation and sentence-level discourse parsing tasks. lmgc utilizes a bert-style bidirectional transformer encoder @xcite to avoid prediction bias caused by using different decoding directions. since lmgc is on the basis of generative models, it can jointly consider an input text and its predictable labels, and map the embeddings of both input tokens and labels onto the same space. due to this characteristic, lmgc can effectively use the label information through constructing label embeddings from the description of a label definition @xcite . furthermore, recent strong pre-trained models such as mpnet @xcite are available for any input tokens in lmgc. ## base models our lmgc reranks the results from a conventional discourse segmenter and parser, which can be constructed as discriminative models. in this section, we explain these base models and introduce our mathematical notations. ## language model-based generative classifier (lmgc) in this section, we introduce our generative classifier, lmgc, that utilizes a masked and permuted language model to compute sequence probabilities in both discourse segmentation and sentencelevel discourse parsing tasks. more specifically, as we mention in section 5, we can utilize our lmgc in three tasks, ## experiments in this section, we present our experiments in three tasks, (a) discourse segmentation, (b) sentencelevel discourse parsing with gold segmentation, and (c) sentence-level discourse parsing with automatic segmentation. ## conclusion in this research, we proposed a language modelbased generative classifier, lmgc. given the top-k discourse segmentations or parsings from the base model, as a reranker, lmgc achieved the state-of-the-art performances in both discourse segmentation and sentence-level discourse parsing. the experimental results also showed the potential of constructing label embeddings from token embeddings by using label descriptions in the manual. in the future, we plan to apply lmgc to other diverse classification tasks. zhilin yang, zihang dai, yiming yang, jaime carbonell, ruslan salakhutdinov, and quoc v le. 2019. xlnet: generalized autoregressive pretraining for language understanding. arxiv preprint arxiv:1906.08237. zining zhu, chuer pan, mohamed abdalla, and frank rudzicz. 2020. examining the rhetorical capacities of neural language models. in proceedings of the third blackboxnlp workshop on analyzing and interpreting neural networks for nlp, pages 16-32.