Improving Constituent Representation with Hypertree Neural Networks (2022)

Abstract: Many natural language processing tasks involve text spans and thus high-quality span representations are needed to enhance neural approaches to these tasks. Most existing methods of span representation are based on simple derivations (such as max-pooling) from word representations and do not utilize compositional structures of natural language. In this paper, we aim to improve representations of constituent spans using a novel hypertree neural networks (HTNN) that is structured with constituency parse trees. Each node in the HTNN represents a constituent of the input sentence and each hyperedge represents a composition of smaller child constituents into a larger parent constituent. In each update iteration of the HTNN, the representation of each constituent is computed based on all the hyperedges connected to it, thus incorporating both bottom-up and top-down compositional information. We conduct comprehensive experiments to evaluate HTNNs against other span representation models and the results show the effectiveness of HTNN.

## introduction distributed span representations are useful in various natural language processing tasks such as question answering @xcite , coreference resolution @xcite , sentiment classification @xcite and semantic role labeling @xcite . since spans can have arbitrary length, to get fix-dimensional span representations, existing methods are mostly based on simple derivations from word or sub-word representations. these methods either apply some form of pooling over all the words within the target span or utilize only the boundary words of the target span to compute the span representation. there have also been efforts figure 1 : an example binarized constituency parse tree (a) and its corresponding hypertree (b). for each hyperedge, p/l/r denote the parent, left-child, right-child constituent spans respectively. to improve word representation learning using information from text spans, which may at the same time improve span representations built on top of word representations @xcite . further, some previous methods introduce short text spans (phrases or n-grams) as additional inputs to bert and thus learn contextual representations for such spans along with word representations @xcite . however, it is known that natural language text has underlying compositional structure, which can often be represented with constituency parse trees @xcite . what is missing from the aforementioned span representation methods is the utilization of compositional structures both within and outside text spans. specifically, the recursive compositional structure inside a text span suggests more structured bottom-up computation of the span representation from word representations; the compositional structure outside a text span specifies how the span joins its sibling spans to form a larger span, implying top-down computation of the span representation from its parent and siblings 1 . by taking into account such compositional structures, we could obtain better span representations and avoid problems with existing methods such as insufficient modeling of meaning composition @xcite . there are several existing methods of incorporating constituency tree structures in neural networks that can be used to produce constituent span representations. these methods fall into two main categories: recursive neural networks (rvnns) @xcite and graph neural networks (gnns). rvnn-based models recursively compose span representations from their sub-spans following a constituency tree structure @xcite . they hence only model bottom-up composition and miss information from parent and sibling spans in their span representations. some extensions of rvnns @xcite try to tackle this problem by additionally modeling top-down (de-)composition in a similar recursive fashion. however, they separate the representations computed from the directions and disallow them to directly interact with (and potentially disambiguate) each other. gnn-based models, such as graph convolutional networks (gcn) (kipf and welling, 2016) and graph attention networks @xcite , have been applied based on constituency tree structures to compute word and span representations @xcite . one important flaw of gnn-based methods is that a gnn represents each composition with multiple edges that become mixed up with edges from other compositions during gnn updates, thus losing critical information of the compositional structure. more specifically, a gnn does not adequately formulate and differentiate bottomup computation from all the child spans versus top-down computation from the parent and sibling spans. in addition, most gnn-based methods do not model relations between sibling spans. to overcome the above drawbacks, we propose hypertree neural networks (htnn) to improve constituent span representations. specifically, we view a constituency parse tree as a hypertree (figure 1(b) ), in which each node represents a constituent and each hyperedge is a tuple of multiple nodes representing a composition of smaller child constituents into a larger parent constituent. we then build an htnn with the hypertree as its skelethan distituents. see statistics in section 3.4. ton. during iterative update in the htnn, we first visit each hyperedge, computing a representation of each node in the hyperedge from the other nodes using a direction-specific composition function; then we visit each node, aggregating its representations computed from all the hyperedges connected to it. in this way, not only can we keep the hierarchical information during encoding process, but also update the representation of constituents simultaneously. htnns combine the strengths of rvnns and gnns in span representation while avoid their drawbacks. like gnns, htnns compute a single unified representation for each span that integrates information from all the directions in the constituency tree. on the other hand, htnns follow rvnns and utilize direction-specific composition functions to group and integrate information from different directions. to evaluate the effectiveness of our method, we firstly conduct detailed experiments on three probing tasks: semantic role classification, named entity labeling, coreference arc prediction. then we apply our method to the task of semantic role labeling. the results show that our method is superior to word-based, rvnn-based models and gnn-based models. ## related work to get distributed span representations, existing methods are mostly based on simple derivations from word or sub-word representations. @xcite propose the attention-weighted pooling strategy for coreference resolution. @xcite concatenate the sum and the difference of the span endpoints for parsing. @xcite use the "coherent" endpoint-based representation for question answering. all of these methods are originally designed for specific tasks, but because of their simplicity, they can be easily employed for other tasks. @xcite make a comprehensive survey on these methods and conduct experiments to compare their performance. some other studies also focus on the representations of sentences or longer texts @xcite @xcite . instead of representing an arbitrary, some studies consider composing span representations via syntactic parse trees, which are more related to our work. @xcite proposed a recursive model iornn to produce constituent representations for supervised task. @xcite proposed an unsupervised method for constituency parse tree induction, during which span representations are produced as a byproduct. more recently, graph neural networks have been applied with constituency parse trees. @xcite use gcn to encode constituent spans in an srl system. however, they decompose the constituent span representations back to start and end tokens instead of directly using them. @xcite use gat to generate constituent span representations for the sentiment analysis task. ## conclusion and future work in this paper, we propose hypertree neural networks (htnn) to generate better representations of constituent spans following constituency parse tree structures. each node in the htnn represents a constituent span and each hyperedge represents a local composition structure. each htnn layer first computes a representation of each node in a hyperedge using the other nodes from the same hyperedge, and then aggregates the representations of a node from all the connected hyperedges. we empirically compare htnns with previous rvnn-based models and gnn-based models. the outstanding performance of htnns on both probing and srl tasks shows the effectiveness of htnn. for future work, we plan to tackle two related issues of our approach, namely its reliance on highquality constituency parses and its inability to represent distituent spans (some of which may nonetheless be important). we will also extend htnns to more tasks such as sentiment analysis and document classification, and incorporate htnns into