Improve Discourse Dependency Parsing with Contextualized Representations (2022)

Abstract: Previous works show that discourse analysis benefits from modeling intra- and inter-sentential levels separately, where proper representations for text units of different granularities are desired to capture both the information of the text units and their relation to the context. In this paper, we propose to take advantage of transformers to encode different contextualized representations of units of different levels to dynamically capture the information required for discourse dependency analysis on intra- and inter-sentential levels. Motivated by the observation of writing patterns shared across articles to improve discourse analysis, we propose to design sequence labeling methods to take advantage of such structural information from the context that substantially outperforms traditional direct classification methods. Experiments show that our model achieves state-of-the-art results on both English and Chinese datasets.

## introduction discourse dependency parsing (ddp) is the task of identifying the structure and relationship between elementary discourse units (edus) in a document. it is a fundamental task of natural language understanding and can benefit many downstream applications, such as dialogue understanding @xcite ) and question answering @xcite . although existing works have achieved much progress using transition-based systems @xcite or graph-based models @xcite @xcite , this task still remains a challenge. different from syntactic parsing, the basic components in a discourse are edus, sequences of words, which are not trivial to represent in a 1 https://github.com/yifeizhou02/improve-discoursedependency-parsing-with-contextualized-representations straightforward way like word embeddings. predicting the dependency and relationship between edus sometimes necessitates the help of a global understanding of the context so that contextualized edu representations in the discourse are needed. furthermore, previous studies have shown the benefit of breaking discourse analysis into intra-and inter-sentential levels @xcite , building sub-trees for each sentence first and then assembling sub-trees to form a complete discourse tree. in this sentence-first (sent-first) framework, it is even more crucial to produce appropriate contextualized representations for text units when analyzing in intra-or inter-sentential levels. automatic metrics are widely used in machine translation as a substitute for human assessment. this is often measured by correlation with human judgement. when applied to a range of metrics across seven language pairs, … tests show that for a high proportion of metrics, there is insufficient evidence to conclude significant improvement over bleu figure 1 shows an excerpt discourse dependency structure for a scientific abstract from scidtb @xcite . the lengths of edus vary a lot, from more than 10 words to 2 words only (edu 12: tests show), making it especially hard to encode by themselves alone. sometimes it is sufficient to consider the contextual information in a small range as in the case of edu 13 and 14, other times we need to see a larger context as in the case of edu 1 and 4, crossing several sentences. this again motivates us to consider encoding contextual representations of edus separately on intraand inter-sentential levels to dynamically capture specific features needed for discourse analysis on different levels. another motivation from this example is the discovery that the distribution of discourse relations between edus seems to follow certain patterns shared across different articles. writing patterns are document structures people commonly use to organize their arguments. for example, in scientific abstracts like the instance in figure 1 , people usually first talk about background information, then introduce the topic sentence, and conclude with elaborations or evaluations. here, the example first states the background of widely used automatic metrics, introduces the topic sentence about their contribution of a significance test followed by evaluation and conclusion. taking advantage of those writing patterns should enable us to better capture the interplay between individual edus with the context. in this paper, we explore different contextualized representations for ddp in a sent-first parsing framework, where a complete discourse tree is built up sentence by sentence. we seek to dynamically capture what is crucial for ddp at different text granularity levels. we further propose a novel discourse relation identification method that addresses the task in a sequence labeling paradigm to exploit common conventions people usually adopt to develop their arguments. we evaluate our models on both english and chinese datasets, and experiments show our models achieve the state-of-the-art results by explicitly exploiting structural information in the context and capturing writing patterns that people use to organize discourses. in summary, our contributions are mainly twofold: (1) we incorporate the pre-training and fine-tuning framework into our design of a sent-first model and develop better contextualized edu representations to dynamically capture different information needed for ddp at different text granularity levels. experiments show that our model outperforms all existing models by a large margin. ## related works a key finding in previous studies in discourse analysis is that most sentences have an independent well-formed sub-tree in the full document-level discourse tree @xcite . researchers have taken advantage of this finding to build parsers that utilize different granularity levels of the document to achieve the state-of-the-art results @xcite . this design has been empirically verified to be a generally advantageous framework, improving not only works using traditional feature engineering @xcite , but also deep learning models @xcite . we, therefore, introduce this design to our dependency parsing framework. specifically, sub-trees for each sentence in a discourse are first built separately, then assembled to form a complete discourse tree. however, our model differs from prior works in that we make a clear distinction to derive better contextualized representations of edus from fine-tuning bert separately for intra-and intersentential levels to dynamically capture different information needed for discourse analysis at different levels. we are also the first to design stacked sequence labeling models for discourse relation identification so that its hierarchical structure can explicitly capture both intra-sentential and intersentential writing patterns. in the case of implicit relations between edus without clear connectives, it is crucial to introduce sequential information from the context to resolve ambiguity. feng and hirst (2014) rely on linearchain crf with traditional feature engineering to make use of the sequential characteristics of the context for discourse constituent parsing. however, they greedily build up the discourse structure and relations from bottom up. at each timestep, they apply the crf to obtain the locally optimized structure and relation. in this way, the model assigns relation gradually along with the construction of the parsing tree from bottom up, but only limited contextual information from the top level of the partially constructed tree can be used to predict relations. besides, at each timestep, they sequentially assign relations to top nodes of the partial tree, without being aware that those nodes might represent different levels of discourse units (e.g. edus, sentences, or even paragraphs). in contrast, we explicitly train our sequence labeling models on both intra-and inter-sentential levels after a complete discourse tree is constructed so that we can infer from the whole context with a clear intention of capturing different writing patterns occurring at intra-and inter-sentential levels. ## task definition we define the task of discourse dependency parsing as following: given a sequence of edus of length l, (e 1 , e 2 , ..., e l ) and a set of possible relations between edus re, the goal is to predict another sequence of edus (h 1 , h 2 , ..., h l ) such that ∀h i , h @xmath0, e 2 , ..., e l ) is the head of e i and a sequence of relations (r 1 , r 2 , ..., r l ) such that ∀r i , r i is the relation between tuple (e i , h i ). ## our model we follow previous works @xcite to cast the task of discourse dependency parsing as a composition of two separate yet related subtasks: dependency tree construction and relation identification. we design our model primarily in a twostep pipeline. we incorporate sent-first design as our backbone (i.e. building sub-trees for each sentence and then assembling them into a complete discourse tree), and formulate discourse relation identification as a sequence labeling task on both intra-and inter-sentential levels to take advantage of the structure information in the discourse. figure 1 shows the overview of our model. ## experiments our experiments are designed to investigate how we can better explore contextual representations to improve discourse dependency parsing. we evaluate our models on two manually labeled discourse treebanks of different language, i.e., discourse dependency treebank for scientific abstracts (scidtb) @xcite in english and chinese discourse treebank (cdtb) @xcite . scidtb contains 1,355 english scientific abstracts collected from acl anthology. averagely, an abstract includes 5.3 sentences, 14.1 edus, where an edu has 10.3 tokens in average. on the other hand, cdtb was originally annotated as connective-driven constituent trees, and manually converted into a dependency style by @xcite . cdtb contains 2,332 news documents. the average length of a paragraph is 2.1 sentences, 4.5 edus. and an edu contains 23.3 tokens in average. we evaluate model performance using unlabeled attachment score (uas) and labeled attachment score (las) for dependency prediction and discourse relation identification. uas is defined as the percentage of nodes with correctly predicted heads, while las is defined as the percentage of nodes with both correctly predicted heads and correctly predicted relations to their heads. we report las against both gold dependencies and model predicted dependencies. we adopt the finegranularity discourse relation annotations in the original datasets, 26 relations for scidtb and 17 relations for cdtb. for both datasets, we trained our dependency tree constructors with an adam optimizer with learning rate 2e-5 for 3 epochs. our relation labeling models are all trained with an adam optimizer until convergence. learning rate is set to one of {1e-5, 2e-5, 4e-5}. ## conclusion in this paper, we incorporate contextualized representations to our sent-first general design of the model to dynamically capture different information required for discourse analysis on intra-and intersentential levels. we raise the awareness of taking advantage of writing patterns in discourse parsing and contrive a paradigm shift from direct classifi-cation to sequence labeling for discourse relation identification. we come up with a stacked bil-stm architecture to exploit its hierarchical design to capture structural information occurring at both intra-and inter-sentential levels. future work will involve making better use of the structural information instead of applying simple sequence labeling. yizhong wang, sujian li, and houfeng wang. 2017. a two-stage parsing method for text-level discourse analysis. in proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: short papers), pages 184-188, vancouver, canada. association for computational linguistics. an yang and sujian li. 2018. scidtb: discourse dependency treebank for scientific abstracts. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 444-449, melbourne, australia. association for computational linguistics. cheng yi, li sujian, and li yueyuan. 2021. unifying discourse resources with dependency framework. in proceedings of the 20th chinese national conference on computational linguistics, pages 1058-1065, huhhot, china. chinese information processing society of china.