Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling (2022)

Abstract: We examine the extent to which, in principle, different syntactic and semantic graph representations can complement and improve neural language modeling. Specifically, by conditioning on a subgraph encapsulating the locally relevant sentence history, can a model make better next-word predictions than a pretrained sequential language model alone? With an ensemble setup consisting of GPT-2 and ground-truth graphs from one of 7 different formalisms, we find that the graph information indeed improves perplexity and other metrics. Moreover, this architecture provides a new way to compare different frameworks of linguistic representation. In our oracle graph setup, training and evaluating on English WSJ, semantic constituency structures prove most useful to language modeling performance—outpacing syntactic constituency structures as well as syntactic and semantic dependency structures.

## introduction linguistic theories posit that humans can take advantage of hierarchical structure related to some notion of compositionality to produce and comprehend utterances with complex meanings. yet explicit representations of this kind of structure are harder to come by than raw text, and large-scale pretrained neural language models (e.g., @xcite have managed to perform strikingly well at contextually encoding and predicting words from distributional evidence alone. at the same time, there are good reasons to doubt that these models can be said to understand language in any meaningful way @xcite @xcite . to address this conundrum, people have started to explore probing pretrained models @xcite and supplementing training data with linguistic structure guidance @xcite @xcite . a question that has received less attention is which kind of symbolic linguistic representation (slr) is most conducive to guiding neural language models (lms). numerous domain-general candidates exist @xcite @xcite @xcite : some are focused on syntactic structure, others on semantics ( §2; big grey example graphs in the left panels of figure 1 ). frameworks vary along several dimensions, with different label inventories and treatments of specific constructions. formal differences include the type of structure (dependency or constituency, one or multiple parents, projectivity) and its relation to the input string. in general, different design choices may aim to capture different kinds of generalizations or facilitate different kinds of processing, and may make parsing raw text easier or harder. it is often not obvious which framework should be chosen for best results on an external task-or indeed, how to even perform a controlled comparison across frameworks. in this paper we investigate whether structurally guided language modeling can serve as a benchmark task for directly comparing linguistic representations. specifically, we evaluate on next-word prediction-a relatively neutral task in that it does not rely on any artificial test suite, nor does it target a specific downstream application where one linguistic framework may have an advantage. 1 we devise a method for selecting and encoding partial views of linguistic graphs over the preceding context relevant to predicting the next token ( §3 and §4). 2 we call these views slices (small per-token graphs and dashed lines in figure 1 ). our neuro-symbolic encoder statically allocates distinct vector dimensions for different structural ud: syntactic dependency tree with functional labels ptg: deep syntactic / quasi-semantic argument structure 0 1 2 3 am od nsubj:pa ss a u x :p a ss 0 1 2 3 pred ex t pat gpt-2: neural language model with multihead attention n umerous injuries were ???? n umerous injuries were ???? n umerous injuries were ???? 2). as shared tokenization we use gpt-2's byte-pair encoding. slice nodes are color-coded by local relation type (black: target, cyan: parent, blue: child, green: coparent, yellow: sibling, purple: grandparent, brown: aunt). dashed lines indicate the token anchoring of original (big grey) graph nodes and, correspondingly, which previous tokens (empty circles) are accessible for each next-token prediction. in the bottom right we visualize how different models arrive at their prediction ( §3 and §4.3). this invites future research quantifying different formalisms' design choices ( §7). ## background: symbolic linguistic representation following a long tradition in formal linguistics, graph-structured representations of language qualitatively describe grammatical and logical relations among words. the slr paradigm has recently seen a revival in the form of larger-scale treebanking and sembanking for training neural parsers. formally, an slr instance is a directed acyclic graph (dag) = , , , with vertices , labeled edges , and an anchoring function that maps each vertex to a (potentially empty) subset of tokens in the sentence. we broadly distinguish slr frameworks along two dimensions: foot_2 scope. a main goal of syntactic representations is to explain distributional patterns in word order; they tend to be rooted trees with often projective anchoring functions. semantic formalisms are meaning-oriented, aiming to capture the higherlevel logic expressed in a sentence; thus, they may have more complex structures, including reentrant edges and discontiguous anchors. structure. slrs can further be subdivided into dependency and constituency structures. the former are relatively shallow, while the latter contain abstract nodes with no or multiple word anchors. ## overview: language modeling with linguistic graphs our main goal is to quantify the predictive power of different slrs by combining them with a pretrained language model and measuring how this affects next-token generation performance. a language model (lm) assigns probabilities to sentences and can be used to both process existing sentences and generate new ones. as is standard practice, we treat sentences as length-sequences of word tokens, = 0 , 1 , … , 1 . an incremental lm factorizes the joint probability of the sentence in terms of the probability of each word conditioned on previous @xmath0; eq. ( 1 ). here we describe at a high level how we process (oracle) slr graphs for use in this language modeling scenario, i.e., to obtain context-conditional vocabulary distributions from them. in contrast to sequential lms, contexts are now graph-structured, and which context tokens to select as well as in what way they are related to the target token is determined by the underlying slr graph ; eq. ( 2 ). this general idea is closely related to syntactic language modeling @xcite , inter alia). we extend this line of work to arbitrarily complex syntactic and semantic dag structures and, in doing so, take particular care to restrict conditioning contexts from accessing not only future words but also future subgraphs, so effectively top-down and left-to-right. our procedure is as follows: first, we select for each token position to be predicted a subgraph , called the token's slice. slices are both admissible in the language modeling setting, i.e., they do not violate the left-to-right conditioning order, and relevant to the token prediction according to some criteria-here we consider criteria based on structural relationships generally, without relying on formalism-specific labels ( §4.1). consider the small colored subgraphs for each token in figure 1 : the eds-slice for the target 'reported', for example, starts at node 3, and extends to the arg2-child 2, arg1-coparent 1, and bvcoparent 0, which are anchored, respectively, in the spans 'injuries', 'numerous', and 'numerous injuries'). recall from §2 that context @xmath1, to the extent that they are anchored in a node reachable from . inspired by markov assumptions of independence in generative modeling and markov blankets in causal networks, slr graph slicing thus allows us to factorize ( ) as next, we encode each graph slice as a fixedsized vector. prior approaches to encoding linguistic graphs for neural modeling have involved serialization, e.g., as parser transition sequences (qian et al., 2021, inter alia), recursive auto-encoders @xcite , and graphconvolutional networks (gcns; @xcite . however, transition sequences for non-tree graphs are subject to spurious ambiguity; and we find that graph-structured neural networks are impractical in the incremental setting ( §6.5). instead, we propose a computationally inexpensive method for statically and deterministically projecting slices into a high-dimensional space by vector concatenation ( §4.2). finally, we compute output distributions ( ) from the vector representations ( §4.3). ## experimental setup all models are implemented in pytorch and experiments are run on 1 nvidia tesla t4 gpu. model hyperparameters are reported in appendix a.5. ## conclusion we have presented evidence that symbolic linguistic representations of various frameworks have the potential to aid a pretrained incremental transformer in task-neutral next-word prediction. to this end, we have proposed a framework-agnostic neural encoding scheme for linguistic graphs and applied it to an english dataset jointly annotated with 7 different formalisms. the results highlight the importance of appreciating complex linguistic structure and handling its computational representation with nuance. we take the intersection of these sentences and ontonotes 5.0, which contains the gold ptb syntax annotations. 26,719/929 sentences remain in the train-dev/eval set. the mrp graph format operates on raw-text character offsets, while ptb and ud trees operate on word tokens. we are able reconstruct offset-based text anchors for ptb and ud from the raw text strings used in the mrp data for all but 394 train-dev / 8 eval sentences, which leaves us with the final 26,325 train-dev and 921 eval sentences. in a few cases, where the linguistic graph has no edges, we add an artificial edge with a dummy label.