Dialogue Discourse Parsing as Generation: A Sequence-to-Sequence LLM -based Approach (2024)

Abstract: Existing works on dialogue discourse parsing mostly utilize encoder-only models and sophisticated decoding strategies to extract structures. Despite recent advances in Large Language Models (LLMs), there has been little work applying directly these models on discourse parsing. To fully utilize the rich semantic and discourse knowledge in LLMs, we explore the feasibility of transforming discourse parsing into a generation task using a text-to-text paradigm. Our approach is intuitive and requires no modification of the LLM architecture. Experimental results on STAC and Molweni datasets show that a sequence-to-sequence model such as T0 can perform reasonably well. Notably, our improved transition-based sequence-to-sequence system achieves new state-of-the-art performance on Molweni, demonstrating the effectiveness of the proposed method. Furthermore, our systems can generate richer discourse structures such as directed acyclic graphs, whereas previous methods are limited to trees.

## introduction discourse parsing is a natural language processing task that aims to retrieve a structure from a document. the discursive structure contains clauselike text spans (known as elementary discourse units) and are linked by semantic-pragmatic relations such as elaboration and acknowledgment. it plays a crucial role in natural language understanding and has demonstrated its usefulness in various downstream applications such as summarization @xcite and dialogue comprehension @xcite . existing works on dialogue discourse parsing (ddp) suggest that task-specific models are necessary to achieve state-of-the-art (sota) performance @xcite . they are based on complex architectures constructed on top of encoder-only pre-trained language models (plms) such as bert @xcite and roberta @xcite . these models present a few limitations. first, they require task-specific architectures which oftentimes involve heavy engineering of utterance embeddings and specialized decoding strategies. second, the predicted structures are typically limited to trees, neglecting other rich representations such as directed acyclic graphs @xcite . third, they do not leverage rich latent knowledge in more recent large decoder-only and encoder-decoder language models (llms) @xcite @xcite . such llms have shown remarkable abilities in a wide range of applications, from text understanding and generation to coding to reasoning @xcite , resulting in a shift in focus from relatively small encoder-only plms to large-scale encoder-decoder and decoder-only llms. llms see a great amount of data: t0 model @xcite , for instance, is pretrained on the c4 corpus @xcite containing 356 billion tokens; they are pretrained on a mixture of downstream tasks such as multi-document question answering @xcite and natural language inference @xcite . since many of these tasks require an understanding of the intersentence structure, we hypothesize that llms have good contextual representation for sentence-level reasoning (e.g., discourse analysis). however, in our preliminary experiments, we found that directly prompting llms does not perform well on the ddp task, confirming similar observations by @xcite who applied zero-shot prompting and in-context learning methods but found poor performance with gpt-3.5. in this paper, we ask the question: how to effectively transform the discourse parsing task into a ## related work discourse parsing discourse parsing is a hard task, with low performance especially for multiparty dialogues which involve intricate relations between speakers, such as stac @xcite and molweni @xcite . early approaches to discourse parsing used varied decoding strategies, such as maximum spanning tree @xcite @xcite or integer linear programming @xcite . researchers soon applied neural models such as gated recurrent units @xcite and graph neural networks @xcite to build contextualized embeddings, compared to hand-crafted features from the previous work. more recent works attempted to enhance the parsing task by utilizing pre-trained language models (plms) as backbone @xcite , injecting external information such as speaker interactions @xcite , or joint learning with auxiliary tasks @xcite . due to the small number of annotated examples, some also investigated semi-supervised approaches such as data programming @xcite , bootstrapping @xcite , and signals from the attention matrices in plms @xcite . however, much of this line of work dealt only with structure extraction while ignoring relations. with llms on the scene, chan et al. (2023) evaluated the performance of gpt-3.5 on discourse parsing using zero-shot and few-shot in-contextlearning, but only to find that the model performs abysmally. recently, @xcite employed decoder-only llms for rhetorical structure theory (rst) discourse parsing in monologues, where conventional top-down and bottomup strategies are transformed into prompts. on dialogues, only @xcite have investigated discourse parsing with a fine-tuned t5 model. however, their design of output sequences were overly simplified and we observed poor results with a similar abridged scheme in our experiments. in comparison, we explore the effectiveness of using seq2seq llms for this task with more sophisticated representations, such as an output closer to natural language. ## a formal description of discourse every e i contains at least one pair of edus pointing to the node e i . here, we emphasize the uni-direction of edges given that in a dialogue, there are no "backwards" edges such that an edu e k by speaker a is rhetorically and anaphorically dependent upon a further edu e i of speaker b. this is known as turn constraint in the segmented discourse representation theory (sdrt) @xcite . the combination of all e i is the set of all potential edu pairs in document d. the equation 2 defines what we called discourse structure prediction where a "naked" graph can be extracted from d. for full parsing, each edge must be assigned a relation with the function ℓ. we can expand the pairs in e to triples in f : in a nutshell, discourse parsing takes a document d as input and predicts the triples f as output. assuming we have a training set of n examples, pd i , f i q n ## improve seq2seq-ddp model with transition-based algorithm an inherent drawback of the basic seq2seq-ddp system is the weak supervision in long sequences. the longer the document, the harder it is for the model to retrace previous predictions, as evidenced by the hallucinated or forgotten edus. additionally, the act of consecutive output requires extra attention to some properties such as counting, which llms struggle with @xcite . to provide more guidance during the generation and bypass the counting issue, we improve the seq2seq model with transition-based algorithms. the new seq2seq-ddp+transition system takes a single edu at each step and predicts an action corresponding to a set of links involving that edu. ## conclusion we investigate an effective transformation approach for the ddp task by leveraging seq2seq llms. we adopt the pretrained encoder-decoder model t0 and fine-tune it to produce structured sequences. without using any specific parsing module or modifying llm architecture, our seq2seq-ddp system performs reasonably well on stac and molweni datasets. excitingly, our seq2seq-ddp+transition system yields comparable results with task-specific sota models, with richer discourse structures. building on this work, we intend to explore various generative model architectures and sequence representations, and eventually extend our method to other discourse parsing tasks.