A Transformer Based Approach towards Identification of Discourse Unit Segments and Connectives (2021)

Abstract: Discourse parsing, which involves understanding the structure, information flow, and modeling the coherence of a given text, is an important task in natural language processing. It forms the basis of several natural language processing tasks such as question-answering, text summarization, and sentiment analysis. Discourse unit segmentation is one of the fundamental tasks in discourse parsing and refers to identifying the elementary units of text that combine to form a coherent text. In this paper, we present a transformer based approach towards the automated identification of discourse unit segments and connectives. Early approaches towards segmentation relied on rule-based systems using POS tags and other syntactic information to identify discourse segments. Recently, transformer based neural systems have shown promising results in this domain. Our system, SegFormers, employs this transformer based approach to perform multilingual discourse segmentation and connective identification across 16 datasets encompassing 11 languages and 3 different annotation frameworks. We evaluate the system based on F1 scores for both tasks, with the best system reporting the highest F1 score of 97.02% for the treebanked English RST-DT dataset.

## introduction discourse is defined as a coherent, structured group of sentences @xcite . discourse parsing enables the creation of models for further downstream natural language processing tasks such as question-answering, text summarization, information retrieval and extraction, sentiment analysis, and argument mining. discourse segmentation is a fundamental task in discourse parsing, which involves identifying the minimal chunks of text that combine to form a coherent discourse. in the rhetorical structure theory (rst) @xcite framework, these basic chunks of texts are known as elementary discourse units (edus). these edus are linked together by discourse relations which may be explicit (when explicitly marked in the text by a discourse connective) or implicit. segmentation refers to the task of identifying these edus. there have been several approaches towards the identification of these discourse segments. another popular framework is the segmented discourse representation theory (sdrt) @xcite . both these frameworks segment the text into non-overlapping spans covering entire documents. the discourse segmentation task, in this case, corresponds to identifying the starting point of each discourse unit. in 2008, the penn discourse treebank (pdtb) @xcite was released with a corpus of over 1 million words. the unit identification task for this framework corresponds to identifying the spans of discourse connectives that explicitly identify the existence of a discourse relation. the task of edu segmentation has been widely researched in the past due to its importance as a building block for further downstream natural language processing tasks. most of the previous research concerning edu segmentation has relied on the information obtained from syntactic elements of the text such as syntactic parse trees @xcite . however, recent works have explored the task of segmentation using systems based on neural networks using the bilstm -crf framework @xcite or the attention mechanism @xcite . another important factor in all the previous works has been the presence/absence of gold sentence boundaries @xcite in the data. the disrpt 2021 shared task introduces the second iteration of a cross-formalism shared task on discourse unit segmentation and connective detection, and the first iteration of a cross-formalism discourse relation classification task. the organizers provided 16 datasets in the rst @xcite , sdrt @xcite and pdtb @xcite formalisms with each dataset consisting of a training, development, and test set. the data has been provided in two formats -one is the treebanked format which provides the gold syntax and the other is the plain tokenized data without the gold syntax. this encourages the system design to be able to adapt and deal with data when the gold syntax is not provided. in this paper, we describe our approach towards the discourse unit segmentation and connective detection tasks using a model based on the transformers architecture @xcite with a classification layer on top to provide the outputs. we report scores for discourse segmentation and connective identification on both settings -data with gold sentence boundaries as well as document level boundaries. our best results improve the previous task iterations' @xcite best results on 15 out of the 16 available datasets. ## related work early work in the domain for discourse parsing involved rule-based models which used syntactic information for the prediction of discourse segments and connectives @xcite . @xcite used syntactic information and cue phrases to segment sentences into edus and integrated constraints about textual adjacency and textual organization in a beam search for text level segmentation. @xcite used probabilistic models for edu identification on the rst-dt corpus @xcite . building on this, @xcite developed a neural network model for discourse segmentation using the same rst-dt dataset @xcite . a token based classifier was introduced by @xcite which used pos tags, syntactic chunks, and clause information as features for the segmentation task. fisher and roark (2007) investigated the approach towards segmentation using sentence level syntactic parse trees on the rst-dt corpus. on a similar note, jain and sharma (2016) introduced a hybrid pipeline for discourse connective and argument identification in hindi using sub-tree extraction and linear tagging approaches. the data for this system was obtained from the hindi discourse relation bank (hdrb) @xcite . the importance of dependency information was investigated by @xcite by introducing a segmentation model that only relies on part-ofspeech information. @xcite proposed a method of using syntactic features for the identification of discourse connectives and the disambiguation of the connective in terms of its usage and the relation they mark. recent studies have been moving towards machine learning based approaches with most involving the use of sequential neural network models. @xcite used a bidirectional recurrent neural network along with a pointer network to select text boundaries in the input sequence. an endto-end neural segmenter was proposed by @xcite based on the bilstm-crf framework. their system reported new state-of-the-art performance on the rst-dt corpus @xcite with an f1 score of 94.3%. @xcite investigate a transformers based approach towards document and discourse level segmentation with the rst-dt corpus being used for discourse segmentation and the wiki-727k dataset @xcite for document segmentation experiments. @xcite introduced the first multilingual segmenter across 5 languages and 3 non-newswire english domains using language independent tools. segmenters have also been developed for other languages with @xcite proposing a discourse segmenter for german, @xcite for basque, @xcite for french, van der vliet (2010) for dutch, @xcite for brazilian @xcite for spanish and @xcite for chinese. ## datasets in this section, we describe the datasets provided by the organizers of the codi-disrpt2021: discourse relation parsing and treebanking shared task at emnlp 2021 foot_0 . the data provided consists of 16 datasets comprising of 11 languages (german, english, basque, persian, french, dutch, portuguese, russian, spanish, turkish, and mandarin chinese). this is the first iteration of the persian rst corpus @xcite being included for the task of discourse segmentation. the chinese pdtb dataset @xcite is not available freely. hence, the organizers provided the scores on this dataset after running the model on the cdtb dataset during the evaluation phase. ## system overview in this section, we present our approach towards the tasks of discourse unit segmentation and connective identification. we establish a baseline using a bidirectional lstm classifier and propose a final system using the transformers architecture @xcite with a classification layer on top. ## results our system, segformers, improves the best scores of the previous iteration's results for 15 out of the 16 provided datasets for the discourse unit segmentation and the connective identification tasks. in this section, we present the detailed results of all our systems. the final precision, recall and f1 scores have been provided by the organizers by averaging the scores obtained over 5 runs. ## conclusion and future work in this paper, we looked at the topic of discourse segmentation and its importance in the development of robust models for further downstream natural language processing tasks. we report our approach towards building a system for the edu segmentation and connective identification tasks across 16 datasets consisting of 11 languages and 3 frameworks. we present a multilingual system leveraging the new transformer based approach and fine-tune it to obtain quality results for the two above-mentioned tasks. our results showed that a token classification approach towards the task of segmentation helps the model perform better and results in state-of-the-art scores. we are also currently working on developing a curated dataset and model for the connective identification task for the hindi language. this aims to increase the diversity of the available data and add a dataset to the existing corpora.