Llamipa: An Incremental Discourse Parser (2024)

Abstract: This paper provides the first discourse parsing experiments with a large language model (LLM) finetuned on corpora annotated in the style of SDRT (Segmented Discourse Representation Theory, Asher (1993), Asher and Lascarides (2003)). The result is a discourse parser, Llamipa (Llama Incremental Parser), that leverages discourse context, leading to substantial performance gains over approaches that use encoder-only models to provide local, context-sensitive representations of discourse units. Furthermore, it is able to process discourse data incrementally, which is essential for the eventual use of discourse information in downstream tasks.

## introduction understanding a conversation requires more than just grasping the contents of its utterances. one must also understand how different utterances relate to one another, for example whether an utterance answers a question or serves to explain or correct something previously said. discourse parsing, the task of predicting such relations, often represents the discourse structure of texts or conversations as graphs in which elementary discourse units or edus (roughly, clause-level contents) are represented by nodes, and semantic relations between clauses are represented as labeled edges. discourse parsing remains a challenge in nlp due to three factors: (i) the complexity of discourse graphs, (ii) the frequent lack of surface cues provided by edus, which forces parsers to rely on deep, semantic information, (iii) the context dependent nature of the inferences required to predict the graph elements. recently, improved discourse parsers have incorporated neural models and encoder-only pretrained language models like bert @xcite which offer richer representations of linguistic content, and can address factor (ii). but given their limited input size, they are confined to look at one small part of a discourse at a time, and so cannot really address factor (iii). generative large language models (llms) can substantially improve discourse parsing. they not only encode large amounts of lexical and conceptual information, addressing problem (ii), but their increased context windows permit an incremental parsing strategy to address problem (iii). such a strategy makes sense cognitively and linguistically, as prior discourse structure has an effect on inferences about where subsequent discourse units are attached (the link task), and how they are attached, i.e., which types of semantic relations link them (the link+relation task) @xcite . further, an incremental approach naturally applies to downstream applications such as conversational agents, in which the information about the previously inferred discourse structure must be available and updated in real time. we develop an incremental parsing model by fine-tuning llama3 @xcite . llamipa (llama incremental parser) uses previously inferred discourse structure to simultaneously predict the links and relation types between new discourse units. we show that llamipa substantially outperforms the state-of-the-art on both the link and link+relation tasks. we demonstrate llamipa's performance using three datasets annotated in the style of sdrt (segmented discourse representation theory @xcite . our central dataset is the minecraft structured dialogue corpus (msdc; @xcite , a corpus of situated, chat-based, task-oriented dialogues, but we also show the generality of our approach by separately fine-tuning on stac, a corpus of situated multiparty chats from an online game @xcite . we further test llamipa fine-tuned on stac on an out of domain dataset, molweni @xcite . not only does llamipa outperform previous discourse parsing models, but it handles particularly complex structural features of our situated, multiparty datasets, addressing problem (i). in particular, llamipa can identify cases in which a single edu has two parent edus, a situation that arises naturally in both multiparty and situated dialogue, where a speaker can respond to multiple channels (conversational or multimodal) at once. most current sdrt based parsers cannot handle such structures @xcite @xcite . in what follows, section 2 provides background on discourse structure. section 3 presents related work, while section 4 details the multiparty corpora we use for training and testing. sections 5 and 6 describe our model and results. section 7 describes our ablation experiments, and we conclude in section 8. ## background discourse representations for a text or conversation often take the form of tree or graph structures that are built recursively from discourse units or edus, where edus are clauses or subclausal units that serve as minimal, linguistic constituents in discourse relations @xcite . these relations, such as explanation, question-answer pair or acknowledgement, are represented as linking edus in a discourse graph. there are two main theories that have investigated complete discourse structures for texts: rhetorical structure theory (rst; @xcite and sdrt @xcite . rst assigns relations between contiguous edus which are connected hierarchically to further edus in a treelike structure. sdrt provides a dependency style, weakly connected directed acyclic graph (dag) in which typed relations are represented as labeled arcs in the graph, and the terms of the relations are edus or elementary event units (eeus). eeus represent salient aspects of the nonlinguistic context in which a conversation takes place, and may be connected to the discourse dag. sdrt also allows subgraphs, or complex discourse units (cdus) to act as terms of semantic relations. sdrt is the only framework to date that handles multiparty conversation. whatever the underlying paradigm, discourse parsing requires addressing three tasks. first, the segmentation task, which involves decomposing a text and (descriptions of) actions into edus and eeus. the link task involves identifying pairs of edus and/or eeus that are semantically related via discourse relations-that is, the arcs in the discourse graph. finally, the link+relation task involves identifying the labels for each attached pair. ## related work traditionally, sdrt style parsers assume gold discourse unit segmentation @xcite @xcite @xcite , and parsing follows a pipeline architecture in which discourse units are first linked, and then labels are predicted for those links. our proposed architecture assumes gold discourse unit segmentation, but combines the attachment and labeling tasks. several strategies have evolved to harness the complex information required for discourse parsing within the dependency based paradigm proposed by sdrt or dltag @xcite . research has shown that semantic features of edus are crucial for solving the link+relation task @xcite . thus, older approaches to multiparty conversation @xcite input manually designed features of the edu pairs to simple ml models to predict link and link+relation label successively. these local models use information local to the pairs of edus that are to be attached, but do not exploit information about the surrounding discourse structure. in order to leverage some more global information, @xcite ; @xcite apply special decoding mechanisms with post hoc constraints on the output edu pairs, following @xcite . although local models exploit little to no information about other components of the discourse structure, such a model can still be competitive @xcite . we do a detailed comparison between an encoder based local parsing model and llamipa. @xcite were the first to obtain competitive discourse parsing results using a deep learning architecture on stac. they attempted to capture incremental and contextual effects in their model by training a supplemental structured encoder to incrementally update attachment paths (sequences of parent-child edus). however, @xcite show that the model obtains similar scores with or without this encoder. @xcite develop an approach that uses a pre-trained roberta model @xcite to provide embeddings for edus. in addition, they use a bi-gru cell @xcite to capture contextual information; but this requires them to limit the size of the input. their model uses two linear layers for link and relation prediction. the model uses a hierachichal gru gate for contextual edu representations. they then apply the ssa-gnn to capture implicit structural information between edus, using a structure-aware scaled dot-product attention @xcite to update edge and edu representations. they further train a teacher network to supplement the standard classification loss with an auxiliary loss. chi and rudnicky (2022) developed a more complex model based on @xcite . none of the models by @xcite ; liu and chen (2021); @xcite ; @xcite ; @xcite predict multiple parents for an edu (or eeu). the only previously developed parser that predicts multiple parents during attachment is @xcite 's model bertline. bertline uses bert embeddings for edus and eeus, along with a linear layer for features like speaker change and distance information, to build a local model and predicts link and link+relation in pipeline fashion. however the relation labeling uses a multitask architecture to capture informational dependencies between link and relation labeling decisions. @xcite show that their model is competitive with approaches that try to build in more contextual knowledge @xcite . @xcite uses bertline on the msdc, which makes it a natural baseline for our work on situated, multiparty conversation. recent work investigates using llms to predict discourse structure by finetuning llama2 7b, 13b, and 70b for rst-style parsing @xcite . the reported results show improvements over the state-of-the-art for rst corpora analogous to those we find using llamipa on sdrt corpora. chan et al. (2024 the msdc represents conversational moves as well as builder move sequences, individuated into block placements and removals. figure 1 shows an excerpt from an msdc dialogue containing a few examples of discourse units with multiple parents, or mpdus. in one such example, after placing two blue blocks, the builder asks, "like that?". the architect's reply, "not quite," not only serves to answer the builder's question, but also corrects the builder's last moves. the edu for "like that?" is an mpdu that is semantically linked to both linguistic and nonlinguistic units. such mpdus are an important feature of the msdc, as they facilitate the interplay of multimodal contributions in a single discourse structure. another noteworthy feature of the msdc is how nonlinguistic actions are annotated. builder moves are usually prompted by architect instructions, but often transpire in longer sequences of block placements and removals, which sometimes overlap with other linguistic chat moves. these action sequences are grouped together in single complex discourse units (cdus) in the msdc, indicated by the dashed boxes in figure 1 . @xcite , @xcite merged these highly repetitive sequences of nonlinguistic moves into a single eeu to prevent the repetitive connections between them from drowning out the more complex and interesting linguistic interactions in the corpus. we used the msdc with this modification. stac. stac 1 is a corpus of multiparty chats from the game settlers of catan, in which players trade or otherwise acquire resources in order to build roads and settlements and thereby score victory points. stac has two versions: stac-l, in which only relations between chat moves (edus) are annotated in the absence of the context provided by nonlinguistic events, and a situated version, stac-sit, in which both chat moves and (descriptions of) game moves (eeus) are annotated in the presence of an integrated linguistic and nonlinguistic context. like msdc, stac contains mpdus due to multimodal interactions, but also features mpdus which arise from multiparty interaction, in which 1 https://https://www.irit.fr/stac/corpus.html one player can simultaneously respond to multiple players at once. relevant statistics for stac and the msdc are found in stac-sit, like the msdc, also contains sequences of moves containing only eeus. @xcite compressed these sequences into individual eeus and we follow this choice. unlike the msdc, however, stac contains numerous cdus for chat moves and for chat/game interactions (as opposed to cdus of action sequences only). following bennis et al. (2023) and most of the literature on sdrt parsing, we do not treat cdus directly, but use the "flattening" procedure outlined in @xcite that converts a structure with cdus into one with only edus and eeus. a final simplification of the original stac corpus involved removing eeus that neither interact with edus nor figure in eeu sequences that interact with edus. this choice, like the one to compress eeu sequences, is made to avoid drowning out the more complex and interesting data involving linguistic edus and their interactions with eeus. molweni. the molweni corpus contains sdrt graphs for multiparty interactions in the ubuntu chat corpus @xcite , a collection of exchanges from an online forum for discussing issues related to ubuntu. unlike the msdc and stac, molweni does not contain cdus, eeus, or mpdus. we use the molweni-clean test set, a version of molweni re-annotated by @xcite . ## llamipa: an incremental parser llamipa leverages the enhanced contextual reasoning capacities of generative llms by introducing an incremental discourse parsing strategy. as it generates a discourse graph for a conversation, the llm bases its link and relation predictions for an elementary unit ϵ based on the graph it has already constructed for units preceding ϵ. formally, let c be a sequence ϵ 0 , ..., ϵ n where ϵ i denotes an edu or eeu. our llm takes as input a set u , containing edus and eeus, and a discourse graph g. at step 0, u is {ϵ 0 }, the head edu, and g, the null discourse graph. after that, u is a set {ϵ l , ..., ϵ m }, where 0 ≤ @xmath0, and g contains links connecting {ϵ l , ..., ϵ m }. at each step, the model extends the previous discourse graph by integrating link and relation predictions for all of the edus or eeus in the current dialogue turn. in the data, we represent relation instances as elementary unit index pairs connected by a relation type, e.g. res(0,1), for the instance of a result relation holding between units 0 and 1. during training, each sample contains the previous edus and eeus within the window size, the gold structure, which is a list of relation instances between these elementary units, and finally, the units of the current turn. the model is trained to predict the gold relation instances that connect the units of the current turn with previous ones. as llms can handle longer input sequences, we use a window size @xmath1. since discourse links follow an inverse logarithmic curve @xcite , there is little statistical support for links of distance longer than 15 in the sdrt corpora. we finetune llama 2 @xcite , and llama 3 @xcite to produce llamipa2 and llamipa3 using qlora @xcite . both the models are trained for 3 epochs. the model parameter details are in we test our models in two settings. in the first setting, the models predict the relation instances for samples containing the gold discourse structure. in the second, the model uses its predicted relation instances instead of the gold structure in the input sample. that is, at each step, the structure fed to the model is just the accumulated relation instances predicted within the window up until that step (see figure 2 ). this incremental setting provides a clear idea of the model's ability to build a complete dialogue structure from scratch. ## results we run finetuning experiments using the msdc corpus, and assess llamipa's performance on link and link+relation prediction under the two test settings described in section 5. we consider only the well-formed outputs during evaluation, but note that llamipa outputs only well-formed relation instances for all experiments. to test the generalizability of our approach, we also finetune llamipa on stac-sit and compare the llamipa3+p outputs to bertline. llamipa3+p shows a gain of 10 f1 points on link prediction and 10.5 f1 points on link+relation prediction. as mentioned in section 5, the predicted context setting best demonstrates the desired functionality of an incremental parser. since llamipa3+p give the best performance in this setting, we use it for all evaluations and henceforward refer to it using the shorthand llamipa. performance on stac-l. in order to compare llamipa to a broader range of sdrt discourse parsers, we tested it on stac-l, as prior work has only used this linguistic version of stac. we also test llamipa on transfer learning. following @xcite , we finetune on stac-l and test on their molweni-clean test set. while llamipa outperforms all models on stac-l, it fares worse on link prediction than @xcite on molweni, though it does better on link+relation prediction; a counterintuitive result given that link+relation prediction is the harder task. taking a closer look at the link prediction results in @xcite , we see their model performs well on links of distance 1, but drops after that (∼ 25% accuracy for distance 2, and 0 for distances greater than 5). given that molweni is a relatively simple corpus where most of the links are adjacent, their model excels on the link task. but for relation type prediction, @xcite use the traditional method, which assigns relation types after establishing the attachment structure, and using only local information. llamipa surpasses their model on relation type prediction, which shows that an approach predicting link and relation simultaneously performs better than a local pipeline approach, even for shallower discourse structures. performance on msdc. llamipa shows impressive gains for predictions of certain relation types. looking at f1 breakdown by relation type in further, llamipa is able to predict many instances of the correction relation, which also has a number of long distance instances in the msdc. we hypothesize that llamipa has a high f1 on narration and correction because it has access to rich semantic information about the discourse context: it can use the previous discourse moves as well as an explicit representation of their discourse structure. this hypothesis coheres with our understanding of how narration functions in the msdc. narrations connect high-level instructions between which participants clarify, question and elaborate on those instructions, as well as correct missteps (see figure 1 ). while there is some statistical similarity in their surface forms, the important cues for isolating a new instruction, and hence a narration attachment, involve certain patterns of result and acknowledgement links in the prior context, as well as the presence of (nonlinguistic) block placements. this pattern is regular enough that one can find most narrations programmatically as with @xcite 's "second pass". this takes predictions from a first pass of the local model for results and acknowledgements, then applies an algorithmic approach to predict long distance narration links. we tried this second pass for llamipa, however there was no improvement on f1 for narration scores, indicating that llamipa has already internalized this pattern during training. llamipa offers improvements over bertline for many other relation types which are typically not considered long distance. overall, llamipa has several advantages. it simplifies previous attach-then-type pipeline methods by predicting link and relation labels simultaneously. previous research has given theoretical arguments that co-predicting attachment and relation labels should give better results @xcite , as well as empirical evidence for this claim @xcite . and as we have demonstrated, llamipa outperforms other parsers on all relevant corpora. ## the role of context so far, the hypothesis has been that llamipa's state-of-the-art performance, especially for longer distance relations, is due to its ability to process richer contextual information. whether this is a function of the superior embeddings conferred by the underlying llm, or a function of a larger context window that allows a longer dialogue history and explicit discourse structure as prediction input, is something we explore further in this section. first ablation: embeddings vs. context size. this ablation tests whether the llm embeddings for edus and eeus account for llamipa's performance. we finetune llama3-8b over pairs of elementary units as input to predict links and relation types, resulting in a local model roughly comparable to bertline. second ablation: perturbing input structure. even if llamipa requires the input available in a larger context window, it is still an open question whether it uses the discourse structure provided in this window. we already have some indication that llamipa does use the structure when we look at llamipa's performance given a gold discourse structure for a context versus its predicted structure, as described in section 5. the results of this comparison are shown in to investigate this further, we perform a second ablation where we perturb the input structure for each test sample by randomizing both the relation type and edu index for each relation instance in the structure. when we run llamipa on this randomized input (denoted as llamipa+rand in third ablation: removing input structure. in this ablation, we removed all structure from llamipa's input context, leaving only a sequence of elementary units. the performance of llamipa on this data set (in the column llamipa+∅ of fourth ablation: targeted perturbations. the results of the previous ablations support the hypothesis that the structure provided in the context window does have an effect on relation instance predictions. this final ablation investigates whether llamipa predicts context dependent relations as well as it does because it leverages (or "reasons" about) the relation instances in the context structure in a manner consistent with our semantic intuitions. here, we focus on instances of question-answer pair (qap) and correction. qap attachments provide a straightforward example of context dependence. in the msdc, it is common that if a prediction qap(y,z) is made for a dialogue turn z then clarifq(x,y), confq(x,y), or qelab(y,z) is in its context. that is, the questions that are answered in the qap relations tend to be questions asked to clarify, confirm or elaborate upon the preceding context. to test whether llamipa's predictions conform to this pattern, we isolate the test samples where llamipa correctly predicts qap ( correction relations are among the most difficult to identify in the msdc, as discussed in section 6. although the language used for corrections usually features some form of negation-no, not like instruction i action x instruction y action z corr(x, y) corr(x, z) res(y, z) figure 4: correction triangles are composed of a correction to an action sequence, followed by a new action sequence that is the result of the corrective move and a correction of the first action sequence. for instance, most corrections come about when the builder makes the wrong moves in response to an instruction, and the architect then gives a further instruction, which results in moves that address this by fixing the last wrong moves. in the discourse structure, these interactions create recurring patterns, "correction triangles" (figure 4 ). these triangles provide a way to check llamipa's sensitivity to corr(x,y) type relations in the context structure in cases where it must predict res(y,z) corr(x,z), the last two relations in a triangle. as with the question-answer pair test, we single out each test sample in which llamipa correctly predicts the second corr; but instead of removing corr(x,y), we change it to ack(x,y) (acknowledgement(x,y)). this leaves the structure intact, but gives it a contrary sense. we re-run the generation step and see a 20 point drop in f1, from 0.71 to 0.52, for correction predictions, and no change for all other relation types. ## conclusions llamipa is an incremental sdrt style discourse parser that uses its prior discourse structure predictions to predict links and relation labels for the discourse units in new conversation turns. as such, it can be naturally assimilated into downstream applications like conversational agents. after finetuning both llama2-13b and llama3-8b models to produce llamipa2 and llamipa3, we show that llamipa3 has substantially higher f1 scores than other parsers that use sdrt data sets on the link and link+relation tasks. we confirm that its performance is due to access to larger swaths of context, which are not available to parsers that use a local model. we further find evidence to support the hypothesis that llamipa uses the explicit representations of structure in its context to predict new relation instances. notably, llamipa excels on data sets that feature conversation containing both linguistic and nonlinguistic moves (actions, in the msdc) that help interpret them. this indicates that a parser like llamipa can furnish important discourse information to builder action models; preliminary demonstrations of this can be found in @xcite .