Hidden Schema Networks (2023)

Abstract: Large, pretrained language models infer powerful representations that encode rich semantic and syntactic content, albeit implicitly. In this work we introduce a novel neural language model that enforces, via inductive biases, explicit relational structures which allow for compositionality onto the output representations of pretrained language models. Specifically, the model encodes sentences into sequences of symbols (composed representations), which correspond to the nodes visited by biased random walkers on a global latent graph, and infers the posterior distribution of the latter. We first demonstrate that the model is able to uncover ground-truth graphs from artificially generated datasets of random token sequences. Next, we leverage pretrained BERT and GPT-2 language models as encoder and decoder, respectively, to infer networks of symbols (schemata) from natural language datasets. Our experiments show that (i) the inferred symbols can be interpreted as encoding different aspects of language, as e.g. topics or sentiments, and that (ii) GPT-2-like models can effectively be conditioned on symbolic representations. Finally, we explore training autoregressive, random walk “reasoning” models on schema networks inferred from commonsense knowledge databases, and using the sampled paths to enhance the performance of pretrained language models on commonsense If-Then reasoning tasks.

## introduction much of the developmental and causal theories of human cognition are predicated on relational structures of knowledge that naturally exhibit compositionality. semantic content is intrinsically relational, as one is only able to explain a given unit of knowledge -such as a concept, word or perception -insofar as there are other units of knowledge which relate to it @xcite . thus we can partially construe a concept through its relationships to other concepts (like when we say "a dog is an animal that barks"), just as we can partially construe it through its relationships to our perceptions (when we say "that is a dog", whilst pointing to a dog on the street) or the words we use (when we use the word dog to refer to the concept dog). likewise, we can partially construe words not only through their relationships to concepts or percepts, but also through their relationships to other words, as words that occur in the same context tend to have similar meanings @xcite . note that is precisely this contextual semantic word content what we explicit have access to, when processing our raw text datasets. on the other hand, generalization, reasoning and understanding seem to be inevitably tied to the compositional nature of knowledge. indeed, the ability to compose a set of knowledge units (and their relations) into new, more complex relational units, which can be deployed to understand and reason about unseen data -a feature usually referred to as combinatorial generalization -is regarded as key to human-level intelligence @xcite @xcite . relational structures allowing for compositionality thus seem to comprise not a sufficient, but a necessary attribute of any representation scheme that strives for the generalization power of human cognition. from the computational side, if one is to inform any modern language model with such structural characteristics, one will initially encounter the problem of finding suitable primitives or data structures. distributed continuous word representations @xcite , for example, are routinely leveraged in many different downstream tasks. these representations are trained to encode average contextual semantics -precisely the kind of semantic content typical of word co-occurrence relations we mentioned above -into a semantic space, which allows meaning to change continuously within it @xcite . yet, de-spite earlier attempts @xcite , it is unclear whether such representations can be meaningfully composed into representations of, say, unseen sentences and thus mimic the compositional character of natural language. more recently, contextualized continuous word representations inferred by deep learning architectures have shown spectacular results in many nlp tasks @xcite @xcite . their success stems from those models' ability to infer flexible representations through, inter alia, raw, massive datasets, data-scalable attention mechanisms and minimal inductive biases @xcite . these representations are known to not only contain rich contextual word semantics, but also consistently encode sentence-level grammar @xcite , and the models from which they are obtained have been shown to implement different notions of compositionality too @xcite . nevertheless, it is still unclear whether such representations can be composed into representations of novel sentences @xcite . in fact, most of their syntactic properties are implicit and therefore inferred only a posteriori, typically through probes which neither guarantee their presence, nor establish how they were obtained in the first place @xcite @xcite . roughly following the program outlined by @xcite , we develop a novel, neural language model -the hidden schema network model (hsn) -that enforces, via inductive biases, a discrete, relational structure for sentence representation which allows for compositionality foot_0 onto the output representations of large, pretrained language models (lplm). using a variational autoencoder (vae) framework @xcite , hsn leverages lplms to encode natural language sentences into sequences of symbols, which correspond to the nodes visited by biased random walkers on a global latent graph, while inferring the posterior distribution of the latter. in practice, translating the implicit knowledge encoded by lplms into explicit relational struc-tures has some naturally appealing properties. for example, hsn can support symbolic reasoning via inference of missing semantic connections, through high-order paths along the inferred graphs @xcite @xcite . likewise secondary, autoregressive "reasoning" models can be trained on the symbol sequences inferred by hsn from task-specific natural language sequences, like e.g. question-answer pairs. the reasoning paths sampled from such autoregressive models could then be used to inform gpt-like models and improve their performance on natural language understanding tasks, with which they are known to struggle @xcite @xcite . below we demonstrate that (i) hsn is able to uncover ground-truth graphs from artificially generated datasets of random token sequences, and that (ii) using pretrained bert and gpt-2 language models as encoder and decoder, respectively, hsn is able to infer schema networks from natural language datasets, whose symbols encode different aspects of language (like e.g. topics or sentiments). finally, we also explore training secondary, autoregressive models on the schema networks inferred from commonsense knowledge databases, and using the sampled paths to enhance the performance of lplm on commonsense if-then reasoning tasks. ## related work in cognitive psychology, a schema is (roughly) defined as a large, complex unit of knowledge representing what is typical of a group of instances @xcite @xcite . marvin minsky's frames @xcite are similar in function to a schema, but perhaps more easily characterized in terms of data structures. we use these terms in a loose fashion, however. our aim being only to be suggestive of the general problem of knowledge representation @xcite . we are in fact concerned with representation schemes for natural language processing. within the context of linguistics, @xcite argues that there must be a level of representation -the so-called conceptual structures -at which information conveyed by language must be compatible with information coming from sensory systems. conceptual structures must, he goes on, be able to represent all the conceptual distinctions made by natural language, and provide some degree of compositionality. earlier computational models implementing (some kind of) conceptual structure rely on either hand-coded (semantic) network representations @xcite @xcite or hand-coded databases @xcite . other works focus instead on learning semantic representations from raw text data directly, via topic models @xcite , and even infer latent concept graphs through nonparametric priors @xcite . in sharp contrast with these works, (a line of) modern, neuro-symbolic reasoning approaches in nlp seek to combine the implicit knowledge of lplm with external information, structured as knowledge graphs, in order to solve commonsense question-answering tasks @xcite @xcite @xcite . note that the knowledge graphs here can also be interpreted as (some class of) human-authored conceptual structure (see however @xcite , who shows that lplm can generate commonsense knowledge graphs too). our work runs perpendicular to these approaches, for it leverages lplms to automatically discover, in an unsupervised fashion, knowledge graphs in representation space, and introduces reasoning via random walk processes conditioned on the discovered graphs. this problem necessarily involves the (neural) inference of discrete variables, which has also been successfully treated in the past @xcite @xcite @xcite . ## hidden schema networks we address the problem of learning the joint probability distribution over sequences of words, while inferring symbolic representations capturing their semantics. neural autoregressive language models approximate such distributions with a product over conditional probabilities, such that ## inferring ground-truth graphs before testing the behavior of our methodology on natural language data, we evaluate the ability of the model to infer hidden graph structures from sequential data in a controlled experiment. to this end, we define a synthetic language model with an underlying, ground-truth graph g * as follows: given a graph g * with k nodes, and a vocabulary of random tokens v of size v , we assign one random bag of tokens (i.e. one pdf over v) to each node of the graph. let the k random bags be the set e of k symbols {e 1 , e 2 , . . . , e k } of the synthetic language model. we then sample n uniform random walks of length l over g * , and sample one random token from each symbol (i.e. from each random bag) along the walks. the result is a set of random token sequences of the same length as that of the random walks. a simple proof-of-concept. we consider a problem in which the set of symbols e is known, so that the ground-truth graph g * has a fixed labeling. this setting will allow for simple comparison between g * and our inferred graphs. following the procedure above we generate two datasets from two ground-truth graphs with different topologies. one sampled from the barabási-albert model @xcite , the other from the erdös-rényi model @xcite . we set both graphs to have @xmath0, and the token sequences to have length @xmath1. given these sets of random token sequences, the task is to infer the hidden, ground-truth graphs g * . to solve it we first replace bert in fig. 1 with a 2-block transformer encoder @xcite and note that, since the symbols are known, the likelihood of the model is simply given by l @xmath2, as before, j i denotes the index of the non-zero component of z i . we then train this simplified model by maximizing the hsn objective function. further details about the graph model parameters, the dataset generation procedure and statistics, training procedure, and model sizes and hyperparameters can be found in appendix e. the synthetic datasets are available in the source code too. results. ## natural language hsn in the previous section we demonstrated that hsn can indeed infer hidden graph structure from sequential data in a simple setting. we now move on to our main problem: representation learning from natural language. we leverage pretrained bert and gpt-2 language models as encoder results. as shown in structure of hidden schema networks. we characterize the structure of the schema networks we inferred in terms of five statistics, namely their diameter, average distance, clustering coefficient, number of connected components and degree distribution. we define all these and report their computed values, for all hsn configurations and all datasets, in appendix f. one observation we can immediately make about these results is that the schema networks discovered in each corpora tend to have smaller average distances, and much larger clustering coefficients, than any random graph of the same size, sampled from a maximumentropy (@xmath5.5) erdös-rényi model. interestingly enough, the combination of these two features defines the so-called small-world structure @xcite . now, intuitively speaking, a larger clustering coefficient implies that a random walker starting from a given node "k" will have a larger number of paths bringing it back to "k". in such an scenario, random walkers tend to cluster in neighborhoods around their starting point -a property that could help encode different semantic aspects in different regions of g. similarly, one could also expect schemata composed of repeated symbols. we close this subsection with figure 2 , which displays the difference between the degree distributions of the schema networks and that of purely random graphs. schemata and semantics. taking advantage of the labels available to both yahoo and yelp, we display in figure 3 the schema distributions over the schema networks for 4 subsets of these datasets. similar plots for all subsets of both corpora, extracted with all hsn configurations, can be found in appendix f. note how the "hot" symbols per category reside on different regions of the graphs -as suspected already from the large clustering coefficient of g -and yet, the "science & math" schemata (both nodes and edges) of yahoo are closer to the "education & reference" schemata than to the "sport" schemata, where closer nodes in the figure indicate well-connected nodes in the underlying graph g. these findings (qualitatively) indicate that the schemata indeed encode semantic notions of their corpora. a similar picture holds for yelp. we also introduce and explore "schema interpolations" in appendix f.3, and observe that the translation procedure we employ successfully interpolates between categories of both yelp and yahoo. these experiments hint too at the different semantic notions encoded by hsn. to get a more quantitative evaluation of the semantics encoded in the schema networks, one can also investigate how the distribution of attention weights, in the hsn decoder, and wrt. the symbols changes, as the decoder processes specific words. we exemplify this idea in appendix g. ## commonsense reasoning generation in the previous section we empirically showed that hsn is able to infer schema networks from natural language datasets, whose symbols encode different notions of semantics. next, we explore training "reasoning" models on the schema networks inferred from commonsense knowledge databases, and using these secondary models to sample novel (reasoning) paths and enhance the performance of lplm on commonsense if-then reasoning tasks. let us start by defining this task. it has been recently proposed that lplms finetuned on (natural language) knowledge graph (kg) tuples, can express their encoded knowledge through language generation, thereby providing commonsense knowledge on demand @xcite . consider a kg of natural language tuples of the form (s, r, o), where @xmath6, . the comet framework of @xcite finetunes gpt-like models by maximizing the likelihood of the object, conditioned on the sequence (s, r) = (x s 1 , . . . , x s |s| , x r ). we instead approach the problem of commonsense knowledge generation with a three-step process: (i) we auto-encode the kg tuples using hsn so that the first l 2 symbols of the schemata encode (s, r) only, while their last l 2 symbols encode the complete (s, r, o) tuple; (ii) we train an autoregressive (ar) "reasoning" model, that takes (s, r) and z 1: l 2 (the first l 2 schema symbols inferred in step i) as inputs, and learns to generate the second half of the schema z l 2 +1:l (the half that encodes the object); (iii) given unseen (s, r) pairs and their associated first half-schemata z 1: l 2 , we sample novel, second halfschemata z l 2 +1:l from the trained ar model, and use them to condition the hsn decoder and generate novel objects. to do step i we write the random walk posterior distribution q ϕ (z 1:l |s, r, o) of hsn in the product form q ϕ (z 1: for this preliminary study we focus on the atomic dataset @xcite , evaluate the quality of the generated objects with both, bleu-2 @xcite and bert score @xcite ) metrics, and compare against gpt-2, gpt-2-xl and bart, all trained within the comet framework @xcite . we also fix the number of symbols @xmath7, and the random walk length @xmath8. results. we find that our best ar models only learn about 58% of the second half-schema. that is, about half of the symbols encoding the complete (s, r, o) tuple. ## conclusion we have introduced a novel representation learning algorithm for natural language that (i) enforces, via inductive biases, relational structures which allow for compositionality onto the output representations of lplm, and (ii) allows for symbolic reasoning via random walk processes on the discovered graphs. future work shall investigate replacing both encoding and decoding models with more expressive lplm, and explore training different "reasoning" models and encoding schemes, on natural language understanding tasks.