Towards Unification of Discourse Annotation Frameworks (2022)

Abstract: Discourse information is difficult to represent and annotate. Among the major frameworks for annotating discourse information, RST, PDTB and SDRT are widely discussed and used, each having its own theoretical foundation and focus. Corpora annotated under different frameworks vary considerably. To make better use of the existing discourse corpora and achieve the possible synergy of different frameworks, it is worthwhile to investigate the systematic relations between different frameworks and devise methods of unifying the frameworks. Although the issue of framework unification has been a topic of discussion for a long time, there is currently no comprehensive approach which considers unifying both discourse structure and discourse relations and evaluates the unified framework intrinsically and extrinsically. We plan to use automatic means for the unification task and evaluate the result with structural complexity and downstream tasks. We will also explore the application of the unified framework in multi-task learning and graphical models.

## introduction a text is not a simple collection of isolated sentences. these sentences generally appear in a certain order and are connected with each other through logical or semantic means to form a coherent whole. in recent years, modelling beyond the sentence level is attracting more attention, and different natural language processing (nlp) tasks use discourse-aware models to obtain better performance, such as sentiment analysis @xcite , automatic essay scoring @xcite , machine translation @xcite ), text summarization @xcite and so on. as discourse information typically involves the interaction of different levels of linguistic phenomena, including syntax, semantics, pragmatics and information structure, it is difficult to represent and annotate. different discourse theories and dis-course annotation frameworks have been proposed. accordingly, discourse corpora annotated under different frameworks show considerable variation, and a corpus can be hardly used together with another corpus for natural language processing (nlp) tasks or discourse analysis in linguistics. discourse parsing is a task of uncovering the underlying structure of text organization, and deep-learning based approaches are used in recent years. however, discourse annotation takes the whole document as the basic unit and is a laborious task. to boost the performance of neural models, we typically need a large amount of data. due to the above issues, the unification of discourse annotation frameworks has been a topic of discussion for a long time. researchers have proposed varied methods to unify discourse relations and debated over whether trees are a good representation of discourse @xcite @xcite . however, existing research either focuses on mapping or unifying discourse relations of different frameworks @xcite @xcite , or on finding a common discourse structure @xcite , without giving sufficient attention to the issue of relation mapping. there is still no comprehensive approach that considers unifying both discourse structure and discourse relations. another approach to tackling the task is to use multi-task learning so that information from a discourse corpus annotated under one framework can be used to solve a task in another framework, thus achieving synergy between different frameworks. however, existing studies adopting this method @xcite do not show significant performance gain by incorporating a part of discourse information from a corpus annotated under a different framework. how to leverage discourse information from different frameworks remains a challenge. discourse information may be used in downstream tasks. huang and kurohashi (2021) and @xcite use both coreference relations and discourse relations for text summarization with graph neural networks (gnns). the ablation study by @xcite shows that using coreference relations only brings little performance improvement but incorporating discourse relations achieves the highest performance gain. while different kinds of discourse information can be used, how to encode different types of discourse information to improve discourse-awareness of neural models is a topic that merits further investigation. the above challenges motivate our research on unifying different discourse annotation frameworks. we will focus on the following research questions: rq1: which structure can be used to represent discourse in the unified framework? rq2: what properties of different frameworks should be kept and what properties should be ignored in the unification? rq3: how can entity-based models and lexicalbased models be incorporated into the unified framework? rq4: how can the unified framework be evaluated? the first three questions are closely related to each other. automatic means will be used, although we do not preclude semi-automatic means, as exemplified by @xcite . we will start with the methods suggested by existing research and focus on the challenges of incorporating different kinds of discourse information in multi-task learning and graphical models. the unified framework can be used for the following purposes: ## research plan rst-dt is annotated on texts from the penn treebank @xcite that have also been annotated in pdtb. the texts are formally written wall street journal articles. the english corpora annotated under the sdrt framework, i.e., the stac corpus @xcite and the molweni corpus @xcite , are created for analyzing multi-party dialogues, making it difficult to be used together with the other two corpora. therefore, in addition to rst-dt and pdtb 3.0, we will use the annodis corpus @xcite , which consists of formally written french texts. we will first translate the texts into english with an mt system and then manually check the translated texts to reduce errors. in the following, the research questions and the approach in our plan will be discussed. these questions are closely related to each other and the research on one question is likely to influence how the other questions should be addressed. they are presented separately just for easier description. rq1: which structure can be used to represent discourse in the unified framework? although there is a lack of consensus on how to represent discourse structure, in a number of studies, the dependency structure is taken as a common structure that the other structures can be converted to @xcite @xcite @xcite @xcite . this choice is mainly inspired by research in the field of syntax, where the dependency grammar is better studied and its computational and representational properties are well-understood foot_0 . the research by @xcite provides a common language for comparing discourse structures of different formalisms, which is used in the transformation procedure presented by @xcite . another possibility is the constrained directed acyclic graph introduced by @xcite . while @xcite focus on the expressivity of different structures, the constrained dag is motivated from the perspective of strong generative capacity @xcite . although neither of the studies deals with the pdtb framework, since they are both semantically driven, we believe it is possible to deal with the pdtb framework using either of the two structures. we will start with the investigation of the two structures. another issue is how to maintain one-to-one correspondence during the transformation of the original structure and the unified structure back and forth. as indicated by @xcite , the transformation from the rst or sdrt structures into dependency structures always produces the same structure, but going back to the initial rst or sdrt structures is ambiguous. @xcite introduces head-ordered dependency trees in syntactic parsing @xcite to reduce the ambiguity. we may start with a similar method. as is clear from section 2, using the dependency structure as a common ground for studying the relations between different frameworks is not new in existing literature, but comparing the rst, pdtb and sdrt frameworks with this method has not yet been done. this approach will be our starting point, and the suitability of the dependency structure in representing discourse will be investigated empirically. the scidtb corpus @xcite , which is annotated under the dependency framework, will be used for this purpose. rq2: 2 what properties of different frameworks should be kept and what properties should be ignored in the unification? we present a non-exhaustive list of properties, which we consider to have considerable influence on the unified discourse structure. 1. nuclearity: @xcite uses the nuclearity principle as the foundation for a formal treatment of compositionality in rst, which means that two adjacent spans can be joined into a larger span by a rhetorical relation if and only if the relation holds between the most salient units of those spans. this assumption is criticized by @xcite . the remedy provided by @xcite is to separate different levels of discourse information, which is in line with the suggestions in @xcite and @xcite . our strategy is to keep this property in the initial stage of experimentation. the existing methods for transforming rst trees to dependency structure @xcite rely heavily on the nuclearity principle and we will use these methods in the transformation and see what kinds of problems this procedure will cause, particularly with respect to the pdtb framework, which does not enforce a hierarchical structure for complete coverage of the text. the differences in the criteria on location and extent for basic discourse unit identification and relation labeling of the rst framework and the pdtb framework may be partly attributed to different annotation procedures. in rst, edu segmentation is performed first and edu linking and relation labelling are performed later. the balance between consistency and granularity is the major concern behind the strategy for edu segmentation @xcite . in contrast, in pdtb, the connectives are identified first, and their arguments are determined afterwards. semantic relatedness is given greater weight and the location and extent of the arguments can be determined more flexibly. on the whole, neither sdrt nor pdtb shows any tendency of sentence-boundedness. we will investigate to what extent the tendency of sentenceboundedness complicates the unification and what the consequences are if entity-based models and lexical-based models are incorporated. rq3: how can entity-based models and lexicalbased models be incorporated into the unified framework? the pdtb framework believes that lexicalbased discourse relations are associated with anaphoric dependency, which is anchored by discourse adverbials @xcite and annotated as a type of explicit relations. as for entitybased relations, pdtb uses the entrel label to annotate this type of relations when neither explicit nor implicit relations can be identified and only entity-based coherence relations are present. in the rst framework, the elaboration relation is actually a relation between entities. however, it is encoded in the same way as the other relations between propositions, which bedevils the framework @xcite . further empirical studies may be needed to identify how different frameworks represent these different kinds of discourse information. the main challenge is to use a relatively simple structure to represent different types of discourse information while keeping the complexity relatively low. rq4: how can the unified framework be evaluated? we will use intrinsic evaluation to assess the complexity of the discourse structure. extrinsic evaluation will be used to assess the effectiveness of the unified framework. the downstream tasks in the extrinsic evaluation include text summarization and document discrimination, which are two typical tasks for evaluating discourse models. the document discrimination task asks a score of coherence to be assigned to a document. the originally written document is considered to be the most coherent, and with more permutations, the document becomes less coherent. for comparison with previous studies, we will use the cnn and dailymail dataset @xcite for the text summarization task, and use the method and dataset foot_3 in @xcite to control the degree of coherence for the document discrimination task. previous studies that use multi-task learning and gnns to encode different types of discourse information will be re-investigated to test the effectiveness of the unified framework. as we may have to ignore some properties, we will examine what might be lost with the unified framework. ## conclusion we propose to unify the rst, pdtb and sdrt frameworks, which may enable discourse corpora annotated under different frameworks to be used jointly and achieve the potential synergy of different frameworks. the major challenges include determining which structure to use in the unified framework, choosing what properties to keep and what to ignore, and incorporating entity-based models and lexical-based models into the unified framework. we will start with existing research and try to find a computationally less expensive way for the task. extensive experiments will be conducted to investigate how effective the unified framework is and how it can be used. an empirical evaluation of what might be lost through the unification will be performed.