Learning Dynamic Representations for Discourse Dependency Parsing (2023)

Abstract: Transition systems have been widely used for the discourse dependency parsing task. Existing works often characterize transition states by examining a certain number of elementary discourse units (EDUs), while neglecting the arcs obtained from the transition history. In this paper, we propose to employ GAT-based encoder to learn dynamic representations for sub-trees constructed in previous transition steps. By incorporating these representations, our model is able to retain accessibility to all parsed EDUs through the obtained arcs, thus better utilizing the structural information of the document, particularly when handling lengthy text spans with complex structures. For the discourse relation recognition task, we employ edge-featured GATs to derive better representations for EDU pairs. Experimental results show that our model can achieve state-of-the-art performance on widely adopted datasets including RST-DT, SciDTB and CDTB. Our code is available at {https://github.com/lty-lty/Discourse-Dependency-Parsing} .

## introduction discourse parsing is the task to study the inner structure of documents by analysing the relationship between text spans known as elementary discourse units (edus). it is an important research topic in natural language processing and benefits many downstream tasks, including sentiment analysis @xcite , text categorization @xcite , summarization @xcite ) and so on. existing methods for discourse parsing include transition-based models @xcite @xcite and graph-based models @xcite . recent works also employ topdown models @xcite @xcite . however, their results are not satisfactory when compared with syntactic parsing, mainly attributed to the difficulty of encoding edus properly and transferring information over long text spans. utilizing hierarchical models is a natural way to alleviate the problem. figure 1 shows the hierarchical parsing process of a document with 5 sentences, a total of 19 edus. if we parse the document in a normal manner, it is challenging to connect edu recent research has demonstrated the advantages of hierarchical discourse parsers across transition-based, graph-based, and top-down models. @xcite utilizes a top-down parsing approach with three levels of granularity: paragraphs in each document, sentences in each paragraph, and edus in each sentence. @xcite ) adopts eisner's algorithm @xcite for structure and relation prediction at intrasentence, intra-paragraph and intra-document levels. @xcite proposes a hierarchical transition model to predict the dependency tree structure at intra-sentence and inter-sentence levels and employs a stacked lstm @xcite model to predict the relations. the previous transition-based and top-down models, despite benefiting from hierarchical structures, suffer from a common issue: they do not effectively utilize lower-level structures for upperlevel parsing. @xcite represents a span as a whole by employing a weighted sum over word representations. @xcite only assigns complete and incomplete scores to each span. @xcite focuses exclusively on root edus of sentences for inter-sentence parsing. these approaches fail to leverage the structural information acquired from preceding lower-level parsing stages during the upper-level parsing process. this may hurt the model performance. for example, during inter-sentence parsing in figure 1 , the key information in edu 13 (monolingual, backtranslation) is crucial when we consider the relationship between edu 4 and edu 14. therefore, it is important for us to leverage the parsing history in subsequent steps. to address this issue, we propose a solution in the form of a hierarchical transition-based model. our approach mitigates the aforementioned problem by employing graph attention networks @xcite to extract features for the sub-trees constructed in the preceding parsing steps. in contrast to @xcite , where an edu becomes inaccessible once it is popped from the stack, our method maintains accessibility to it through the obtained arcs, enabling its reuse in subsequent parsing procedures. for example, when we consider the relationship between edu 4 and edu 8, we can use information from both intrasentence parsing history (arc 4 → 2) and parsing history of the current inter-sentence sequence (arc 4 → 0), instead of relying solely on the isolated root edu of the sentence. furthermore, we improve the performance of discourse relation recognition by employing edge-featured graph attention networks (egats; @xcite . when compared to existing sequence labeling models, egats outperform in leveraging precise structure information and predicting relations between distant edu pairs. experimental results demonstrate that by utilizing more contextual information with gats, our models achieve state-of-the-art performance on rst-dt @xcite , scidtb @xcite and cdtb @xcite for both structure prediction and relation recognition. our main contributions are as follows: (1) we propose a new transition-based model that incorporates a gat-based sub-tree encoder, effectively leveraging the parsing history. (2) we evaluate two different types of graphs and demonstrate that besides edus, the structure information also matters in predicting head-dependant relations, especially when dealing with long text spans with complex structures. (3) we apply egats, instead of sequence labeling models, to discourse relation recognition to better leverage structure information. ## related work pre-trained language models recent works have shown that pre-trained language models(plms) have achieved good results in discourse parsing @xcite @xcite . these works prove that pretrained language models can better capture relationship between edus than hand-crafted features. @xcite points out that plms are trained with sentence-level contexts, rather than the edulevel structure in discourse parsing. they perform second-stage edu-level pre-training with two tasks, next edu prediction and discourse marker prediction, and these adaptations achieve great improvements. for fair comparison, our model and the baselines in this paper use the same plms: bertbase-uncased for english and bert-base-chinese for chinese. discourse dependency parsing compared to discourse constituency parsing, discourse dependency parsing receives relatively less attention. @xcite converts constituency trees in rst-dt to dependency trees, and employ two graph models on the new dataset. @xcite presents scidtb, a discourse dependency treebank for scientific abstracts from acl anthology and implements transition-based and graph-based parsers on the dataset. @xcite adapts three chinese discourse corpora to the dependency framework. we use these datasets to evaluate our model. hierarchical discourse parsing hierarchical models are employed to improve the discourse dependency parsing performance for transition-based @xcite , graph-based @xcite , and top-down @xcite models. these methods improve parsing results but they do not make full use of the parsing history, as is described in the introduction part. we store the predicted arcs and expand each edu to a sub-tree and use a gat-based encoder to merge information from edus in the sub-tree. ## methodology given a document composed of a sequence of edus @xmath0, e 2 , . . . , e n ), a discourse dependency parser is expected to predict a tree structure that represents the relationships between the edus. for each edu e @xmath1, excluding the single root edu of the document, a head edu h i is selected from the set of edus e, and a relation r i is assigned from a pre-defined set of relations r to describe the relationship between the edu pair (e i , h i ). we adopt a pipelined approach @xcite for the task. we construct a dependency tree structure for each document and then leverage this tree structure to predict the relations between edus. ## analysis it is evident that discourse dependency parsing performance declines when applied to lengthy text spans with complex structures. in this section, we conduct experiments on scidtb to show the capability of our model in handling complex structures. additionally, we perform experiments on rst-dt to showcase the superiority of our model in dealing with long text spans. ## conclusion in this paper, we enhance hierarchical transition systems by integrating a gat-based sub-tree encoder, which enables the parser to leverage the parsing history more effectively and have a better understanding of the whole document. additionally, we apply edge-featured gats to predict relations based on the complete discourse structure, where two separate bert models are fine-tuned to capture features for edus and edu pairs, respectively. experimental results demonstrate that our model outperforms existing methods, particularly when applied to long text spans with complex structures.