A simple but effective model for attachment in discourse parsing with multi-task learning for relation labeling (2023)

Abstract: In this paper, we present a discourse parsing model for conversation trained on the STAC. We fine-tune a BERT-based model to encode pairs of discourse units and use a simple linear layer to predict discourse attachments. We then exploit a multi-task setting to predict relation labels. The multitask approach effectively aids in the difficult task of relation type prediction; our f1 score of 57 surpasses the state of the art with no loss in performance for attachment, confirming the intuitive interdependence of these two tasks. Our method also improves over previous discourse parsing models in allowing longer input sizes and in permitting attachments in which one node has multiple parents, an important feature of multiparty conversation.

## introduction discourse parsing, the task of predicting graphs that represent semantic relations (arcs) between elementary discourse units or edus (nodes), is a hard problem in nlp due to the complexity of discourse graphs and the frequent lack of surface cues provided by edus, which forces parsers to rely on deep, semantic information. multiparty, spontaneous conversation is especially tricky as the structure often meanders and a single participant can respond to multiple discourse moves at once, yielding non-tree-like structures that challenge existent parsing techniques @xcite . while it is tempting to attack a complex problem with complex machinery, as recent research on discourse parsing has done (see section 6), we show that a simple model can achieve or surpass state of the art results for discourse relation labeling with a little inspiration from human discourse processing. first, because a decision about whether two edus are attached by a semantic relation generally requires reasoning about their contents together, our transformer-based model encodes embeddings of edu pairs, exploiting a form of message passing simpler than graph neural net models @xcite while achieving better results. next, we draw on the fact that for human annotators, the tasks of discourse attachment and relation labeling are often interdependent: sometimes one sees how to attach two edus but only later how to determine the relation that links them; sometimes, a clue, e.g. an explicit marker like because, makes the relation clear, and the task is to find the second argument to the relation. our model exploits this interdependence with a multi-task architecture for attachment prediction and relation labeling. finally, our model allows for the non-tree-like structures. in section 2, we provide background on discourse structure and data sets. in sections 3 and 4, we describe our model and results. section 6 presents related work in discourse parsing. ## discourse parsing theories and data sets just as a sentence is not a bag of words but comes with a structure that serves to compute its meaning from that of its constituent words, so too a discourse or conversation is not a bag of dialogue moves but comes with a structure that enables an interpreter to compute an overall meaning from its constituents. edus are clauses or subclausal units that serve as the minimal, linguistic constituents upon which discourse structures are built @xcite , and discourse parsing involves finding the recursive structure over edus that exploits their semantic content together with various contextual features. there are two main theories that have investigated complete discourse structures for texts: rst @xcite and sdrt @xcite . only sdrt has been applied to multi party conversation as in the stac and molweni datasets @xcite . given our interest in multiparty conversation, we use sdrt and three versions of the stac corpus, a set of multi-party chats from an online version of the game settlers of catan, in which players trade or otherwise acquire resources in order to build roads and settlements and thereby score victory points. the standard version, s, contains only linguistic (chat) moves made by players; the situated version, s-sit, integrates descriptions of nonlinguistic events (game moves), represented as elementary event units (eeus). in s-sit, both eeus and edus are integrated into discourse structure. we present the third version in section 2.1. in sdrt a conversation provides a number of edus linked together to form a weakly connected, directed acyclic graph (dag). each edu apart from the head has at least one incoming link. backwards links (where an edu attaches to another edu that comes after it in the dialogue) are prohibited if the edus are produced by different speakers @xcite . each edge of the dag is labeled with one of 16 different types of discourse relation, such as explanation (exp), question-answer-pair (qap), or acknowledgement (ack). the dags postulated by sdrt allow one child to have multiple parents in the structure. figure 1 provides a representative example from the stac corpus: with his 'kk' william acknowledges both refusals to his offer and signals that he is moving on before ending his turn. in multiparty dialogue, coordination and negotiation over content is key. edus with multiple parents typically mark moves that signal such a coordination or end of negotiation. multi-parent links can thus have a major impact on conversational structure and its effects on downstream tasks, despite their statistical infrequency. an important advantage of our model over alternative neural parsers is that it is the only one to take these links into account in both model design and evaluation. ## our model the encoder portion of bertline is finetuned using the loss l l 1 on s l , the set of positive and negative attachments. where θ is the parameters to be optimized, s * i,j refers to gold data and s i,j to predictions: we concatenate h with a vector struct (i,j) = [t (i,j) , d (i,j) ] that represents information useful for computing attachments @xcite , namely speaker change and edu distance (i.e., how many edus occurred between i and j in the natural order of the dialogue). the function t i,j returns 1/0 if i and j have the same/different speaker; d i,j returns the distance. our final pair embeddings are: the decoder part of bertline's attachment model is a linear neural network. like previous work, this layer predicts for each edu e j , which of the preceding edus e i , with @xmath0.e., up to a certain limit) are likely parents of e j . it is a binary classification on each pair embedding. p @xmath1,j) , h (j-2,j) , ..., h (j-10,j) ) our loss function is a binary cross entropy over possible attachments for each dialogue d in training set d. below l * i,j indicates if e i and e j are attached in the gold data, and l i,j is a predicted link: the set of predicted attachments l contains all pairs l i,j whose link probability exceeds a given threshold α, optimized at 0.8 through experiments on the validation set; i.e., l i,@xmath2,j ) > α = 0.8. we trained the linear layer for 15 epochs. this set up allows us to predict multiple parents for a given edu. of the 77 multi-parent structures (8 with more than 3 parents) in the test set, we captured about 20%. relation labeling. to test the interdependence of link and relation prediction, we built a bert model with two classification heads. the first head learns to predict links between edus and the second head learns to classify edu pairs in terms of sdrt's 16 relation labels. we compute the loss of each head independently, average the results, and back-propagate the average back to the bert encoder. at inference time, only the relation classification head is used for predictions. we fine-tuned the bert model over s l over 3 epochs, together with training over s r , the set of relation label instances. r ( * i,j) refers to gold relation data, r (i,j) to predictions. all previous work apart from @xcite defines the task of relation prediction as following that of link prediction in a pipeline architecture. our multitask setup allows us to have an embedding containing information about both tasks and improves the relation f1-score by one point over a pipeline relation prediction task. evaluation metric: we used a micro averaged f1-metric over all gold data attachments and relation label instances for the scores in ## results with regard to the different versions of stac, bertline's overall scores improve between s and and s-sit in bertline has a far simpler architecture than the neural models whose code we were able to rerun. it is essentially a local model that uses a minimal amount of message passing from nodes to potential neighbors, a technique from graph neural nets also used to encode entire graphs in @xcite . bertline's main advantage comes from the explicit pair encoding of edus. while broader contextual information and structural constraints on dags like those in @xcite can undoubtedly improve scores for discourse attachment and relation labeling, we have sought to show that the efforts of complex architectures to harness this extra information have failed to lead to substantial gains. how to more successfully exploit it is our main research topic for the future. ## conclusions we have described a simple yet effective discourse parser that provides multiple attachments and codependent learning of the labeling and attachment tasks. our model is the only neural parser that does this. we also wanted to show the power of local information when cleverly used. indeed, discourse parsing requires contextual information, but our results show that current research does not yet leverage that information to achieve gains that convincingly outstrip those of a local model.