Learning Clause Representation from Dependency-Anchor Graph for Connective Prediction (2021)

Abstract: Semantic representation that supports the choice of an appropriate connective between pairs of clauses inherently addresses discourse coherence, which is important for tasks such as narrative understanding, argumentation, and discourse parsing. We propose a novel clause embedding method that applies graph learning to a data structure we refer to as a dependency-anchor graph. The dependency anchor graph incorporates two kinds of syntactic information, constituency structure, and dependency relations, to highlight the subject and verb phrase relation. This enhances coherence-related aspects of representation. We design a neural model to learn a semantic representation for clauses from graph convolution over latent representations of the subject and verb phrase. We evaluate our method on two new datasets: a subset of a large corpus where the source texts are published novels, and a new dataset collected from students’ essays. The results demonstrate a significant improvement over tree-based models, confirming the importance of emphasizing the subject and verb phrase. The performance gap between the two datasets illustrates the challenges of analyzing student’s written text, plus a potential evaluation task for coherence modeling and an application for suggesting revisions to students.

## introduction the clause is a fundamental unit in coherent text. much work in nlp investigates how clauses combine to form larger units, ultimately spanning a whole discourse @xcite ; how to decompose complex sentences into distinct propositions @xcite @xcite ; how to identify explicit or implicit semantic relations between clauses @xcite , or how to select a connective to link multiple clauses into a complex sentence @xcite . in this paper, we @xcite bob cooked tia a burger. p1, q1 alth p2 bob cooked himself a burger. p1, q2 bec q1 bob was hungry. p1, q3 none q2 tia was hungry. p1, q4 alth q3 bob was thirsty. p2, q1 bec q4 tia was thirsty. p2, q2 alth alth(ough): contrast p2, q3 alth bec(ause): precondition p2, q4 none figure 1: for the propositions p m , q n to be joined by although or because, p m and q n . must have some semantic commonality to allow for contrast or causation. four cases allow although. the two cases that allow because have a strong semantic relation between the predicates (make someone a burger, be hungry) and, there is no conflict in the to-object of the first clause and the subject of the second. the remaining two cases have no commonality, and neither connective can occur. we assume that training clause representations based on connective prediction will be useful for developing representations that capture aspects of coherence, such as those shown in fig 1 . @xcite examine a series of coherence evaluation tasks that capture different aspects of coherence. they argue that connective substitution is one of four critical tests of coherence modeling. for example, connectives that express temporal succession should not be substitutable for connectives that express simultaneity, as doing so would change the meaning. studies of students' writing skills look at connectives with respect to quality of students' argumentative writing @xcite , and whether automated assessments differ for low-skilled versus high-skilled writers @xcite . although students can fill in correct connectives eliminated from source texts, they typically do not use connectives as precisely in their own writing @xcite . nlp applications aimed at supporting student revision use connectives as an indication of writing quality @xcite , but do not help students choose correct connectives. to better evaluate model performance in connective selection, and to highlight differences between text from skillful versus developing writers, we provide two large datasets of clauses linked by connectives drawn from published fiction and from students' written text. we demonstrate the potential for a model trained on expert data to identify incorrect uses of connectives in students' writing, where students frequently misuse connectives like and. our contributions are: 1) a data structure we refer to as a dependency-anchor graph that incorporates information from both dependency and constituency trees; 2) dance (dependency-anchor graph representation for clause embedding), a novel neural architecture that exploits bi-lstms at the lower layers for learning inter-word influences, and graph learning of relational structure encoded in the dependency-anchor graph; 3) two datasets for carefully edited versus student text. our approach outperforms the state-of-the-art on connective prediction. ## motivation the question of whether latent representations of sentence meaning can benefit from syntax has been addressed in work that compares recurrence and recursion, and finds the main benefit of recursive models to be better treatment of long-distance dependencies @xcite . two recent works compare tree-based models derived from dependency parses with constituency parses on semantic relatedness tasks, with no clear advantage of one grammar formalism over the other @xcite . as discussed in @xcite , dependency trees provide a more compact structure than constituency trees, through shorter paths from the root to leaf words. further, all of a verb's arguments are its direct dependents. the recursive structure of constituency trees, on the other hand, facilitates identification of subtrees that span more of the leaf words as one moves up the tree, and that have a compositional contribution to the meaning of the sentence. tree-based models take input from syntactic parses and compose the latent vectors through a uni-directional traversal, where the parent node representation is the sum of the child nodes. for both formalisms, many parameters are needed to encode the child-to-parent representations. for this reason, previous work strictly limits the model dimensionality @xcite . to combine advantageous features from both kinds of grammar formalism, we propose dependency-anchor graphs as a compact represen-tation that highlights the core elements of a proposition. we construct a graph with only selected components from two kinds of parse trees, thus limiting the number of parameters to learn. the subject of a clause and the main verb phrase are the two outer nodes in the graph, where we refer to the verb phrase node as the anchor. the subject arc from a dependency parse points from the anchor node to the subject. the anchor node is a subgraph that retains the dependency structure of words within the verb phrase. other syntactic relations (e.g., involving words in the subject phrase or adverbial phrases), are ignored. to encode the graph, we propose dance, which applies graph convolution (gcn) @xcite to encode the arc between the subject and verb phrase. the input to the graph convolution comes from a bidirectional lstm (hochreiter and schmidhuber, 1997) that encodes all the input tokens, including the words outside the subject and verb phrase. the interaction that dance captures between subject and verb phrase has been essential in word representation but missing in tree based models @xcite . we demonstrate the effectiveness of the dependency-anchor graph and dance architecture through its superior performance over baselines, including tree-based models. the rest of the paper is organized as follows: we first present related work and give a detailed discussion of the dependency-anchor graph and dance. then we present the datasets, experiments and discussion. ## related work much research has addressed ways to learn high quality clause representations. @xcite propose a shortest dependency path lstm for sentence representation in the task of relation classification. @xcite propose a bilstm based model that combines paragraph vectors and word vectors into clause embeddings for a situation entity classification task. connective prediction has often been addressed: @xcite and @xcite use recursive neural networks with parse trees as input to predict connectives and discourse relations, with solid improvements on pdtb. @xcite use a decomposable attention model to predict connectives on sentences pairs extracted from wikipedia. our work draws on the idea of incorporating syntax into representation for connective prediction, 2019 ) develop universal sentence embeddings from a connective prediction task, and create a large corpus extracted from published fiction. they achieve state-of-theart performance on predicting connectives, as well as on sentence embedding benchmarks from sen-teval @xcite . our work modifies the corpus from @xcite to restrict the pairs of sentences for connective prediction to simple sentences. our goal is to generate clause embeddings specifically for connective prediction, rather than universal sentence representation. ## dance architecture the input to dance is a graph for each simple sentence that includes syntactic information from a phrase structure parse to identify the vp, and from a dependency parse to identify the grammatical subject, and dependencies within the vp. ## experiments for our experiments to predict connectives, we use the same classifier used in @xcite ). an input pair of sentence vectors representing the clauses to be joined by a connective are concatenated with vectors resulting from three pairwise vector operations: averaging, subtraction and multiplication. the concatenated vectors are fed into three fully-connected layers, then projected to a lower dimension prior to softmax over the classification categories. experiments on book-simpl predict the correct connective, given positive examples of clause pairs. experiments on desse predict the correct connective, given positive and negative examples. we compare dance with four baselines on both datasets, reporting accuracy and f1. student writing is much less coherent than much of the text that applies nlp to tasks related to discourse structure, ## discussion here we discuss the potential to suggest an alternative connective for students when their choice of connective differs from a connective predicted by a model that has been trained on professionally written text. the benefits of this analysis are two-fold: it explores the feasibility of an education application to help students revise their choice of connective, and it allows us to examine dance's ability to model aspects of coherence that pertain to choice of connective. for all pairs of sentences in desse 5 and 8, we compared the observed choice made by the student writer with the prediction from dance-fa trained on book simpl 5 or book simpl 8. ## conclusion this paper presented the dependency-anchor graph, a new data structure emphasizing the propositional structure of clauses, and dance, a neural architecture with a distinct module for learning verb phrase representation, and graph convolution for semantic relation between the verb phrase and its subject.dance shows good performance on two datasets for connective prediction, and introduces a potential application that could help students revise their writing through improved choice of connectives. future work will extend dance for coherence modeling within and across sentences, and for applications to support students' revisions. here we present a detailed statistics of connective distributions on desse and book-simpl. figure 7 presents the connectives from desse with distribution above 1%. among 68 connectives, there are thirteen connectives above the threshold and the rest are low frequency connectives.