Distilling Structured Knowledge for Text-Based Relational Reasoning (2020)

Abstract: There is an increasing interest in developing text-based relational reasoning systems, which are capable of systematically reasoning about the relationships between entities mentioned in a text. However, there remains a substantial performance gap between NLP models for relational reasoning and models based on graph neural networks (GNNs), which have access to an underlying symbolic representation of the text. In this work, we investigate how the structured knowledge of a GNN can be distilled into various NLP models in order to improve their performance. We first pre-train a GNN on a reasoning task using structured inputs and then incorporate its knowledge into an NLP model (e.g., an LSTM) via knowledge distillation. To overcome the difficulty of cross-modal knowledge transfer, we also employ a contrastive learning based module to align the latent representations of NLP models and the GNN. We test our approach with two state-of-the-art NLP models on 13 different inductive reasoning datasets from the CLUTRR benchmark and obtain significant improvements.

## introduction the task of text-based relational reasoning-where an agent must infer and compose relations between entities based on a passage of text-has received increasing attention in natural language processing (nlp) @xcite . this task has been especially prominent in the context of systematic generalization in nlp, with synthetic datasets, such as cluttr and scan, being used to probe the ability of nlp models to reason in a systematic and logical way @xcite . more generally, these investigations dovetail with the rising prominence of relational reasoning throughout machine learning and cognitive science @xcite @xcite . however, despite the increased attention and research on text-based relational reasoning, serious challenges remain. perhaps one of the biggest challenges is the persistent gap between the performance that can be achieved using nlp models and the performance of structured models-such as graph neural networks (gnns)-which perform relational reasoning based on structured or symbolic inputs. this gap was made particularly evident in the the clutrr benchmark. clutrr includes relational reasoning problems that can be posed both in textual or symbolic form, and preliminary investigations using clutrr show that gnn-based models-which leverage the structured symbolic input-are able to achieve higher accuracy, better generalization, and are more robust than purely text-based systems @xcite . in this work, we investigate one potential avenue to close this gap. we design an approach to distill the structured knowledge learned by a gnnwhich has access to the underlying symbolic representation of a reasoning problem-into an nlp model. our goal is to do this knowledge distillation @xcite only during training so that the nlp model can achieve higher performance at test time, when only unstructured textual inputs are available. due to the challenges inherent in crossmodel knowledge distillation @xcite , we design an approach that combines both a klbased distillation objective @xcite and a contrastive estimation loss @xcite , which aims to maximize the mutual information between the latent states of text-based nlp and graph-based gnn models. empirical results on 12 different datasets from the clutrr benchmark suite highlight the potential utility of this approach. we find that extending two state-of-the-art nlp models using our structured distillation approach significantly improves performance and that the gains are espe-cially prominent in the context of noisy input data, on which we obtain an 13.6% relative improvement on accuracy. foot_0 2 related work ## task and dataset we use the clutrr benchmark suite as a testbed for our investigations @xcite . clutrr is a relational reasoning dataset that requires an agent to infer family relationships between different characters in a passage of text. importantly, the dataset was constructed in a semisynthetic fashion, which facilitates a principled investigation of text-based relational reasoning. every question-answer pair in clutrr was generated based on underlying family graph structure, where crowd workers were instructed to paraphrase natural language stories from a given set of family relations. to answer a question in the clutrr dataset, the model must infer the family relationship between a pair of entities, whose relationship is not explicitly mentioned. doing so requires extracting the family relationships mentioned in the text and deducing the relationship between the query entities through inductive reasoning (e.g., learning that a parent of a parent is a grandparent. a key element of clutrr is that it provides both text representations and the underlying family graphs used to generate the questions. this allowed @xcite to compare the performance of nlp models, which use only text, with gnn-based models, which reason upon the underlying graph structure, and their analysis revealed a substantial gap in performance between the nlp and gnn models-a gap which we seek to address here. moreover, following @xcite , the semi-synthetic nature of clutrr allows us to evaluate performance in different settings based on the structure of the underlying family graph and the difficulty of the query, including evaluating performance on queries that require a varying number of steps of reasoning and family graphs that include different types of noisy facts (i.e., distractors). ## methodology we now describe our approach for structured distillation, which involves improving the performance of an nlp model by distilling structured knowledge from a gnn (fig. 1 ). graph encoder and text encoder. our base model architectures follow @xcite , with minor improvements. as shown in fig. 1 , we implement both a graph encoder, which generates a vector embedding p graph based on the input family graph, as well as a text encoder, which generates a vector embedding p text of the input text. we use a variant of the graph isomorphism network (gin) architecture @xcite as our graph encoder, since we found this model to outperform the gnn from @xcite . for our text encoders, we experiment with the two top-performing nlp models from @xcite : (1) a variation of an lstm model with attention @xcite and ( 2 ) an adapted version of the mac architecture @xcite . see appendix a for details on the model architectures. integration with knowledge distillation. we utilize knowledge distillation as a surrogate for the structured knowledge transfer from gnns to nlp models. we take the text encoder as the student and a pretrained gnn as the teacher. after generating the representations of the paragraph p text and the question entities (h (m) , h (n) ) , the text encoder sends the concatenation of these embeddings to an mlp decoder to obtain the logits z text . similarly, a pretrained gnn can produce logits z graph from a given underlying graph. we feed the two logits into a kl-based distillation term: integration with contrastive estimation. although knowledge distillation enables nlp models to learn directly from the prediction of gnns, there is no regularization between their latent representations. we mitigate this by using a mutual information (mi) based contrastive learning method to maximize the mi between graph representations from gnns and paragraph representations from nlp models. under our setting, we pair the text representation p text and the graph representation p graph of the same example as positive pairs, and take other graph representations in the same batch as negative pairs. then, following @xcite , we use a jensen-shannon estimator to com-pute the mi, resulting in the contrastive objective: ## experiments our key experimental question is whether an nlp model can be improved by distilling structured knowledge from a gnn. we investigate this question using the gnn and nlp models defined in the previous section, and we follow the experimental protocol from @xcite . we investigate if and how structured distillation can improve generalization and robustness. in all experiments, the nlp models only have access to information from the gnn during training. appendix a contains detailed hyperparameter information. impact on generalization. we first test on the clutrr benchmark tasks where the model must generalize to reasoning problems that require more steps of reasoning than those seen during training. fig. 2 shows the results when we set the number of training reasoning steps to be (2, 3) and 3, 4, and where the test examples require between 2 and 10 reasoning steps. both of the two nlp model obtain higher average performance on test sets with our proposed method. interestingly, however, the positive impact of structured distillation is most apparent when training on examples with longer reasoning paths. appendix b contains results from training on other reasoning path lengths, which are consistent with the trends in fig. 2 . impact on robustness. we next investigated the impact of structured generalization on how robust the nlp models are with respect to noise. following @xcite , we examined settings where different types of noise facts are added into the clutrr reasoning problems. tab. 1 shows the results where we train and test on reasoning problems with different types of noise. here, we see that structured distillation consistently and substantially improves performance of both nlp models, providing an average 13.6% relative improvement on accuracy. the results also shows the distillation and contrastive estimation based on gnns help nlp models ignore noise. however, their fundamental architecture difference limits the extent to which nlp models can learn from gnns. appendix c contains additional results, where the train and test sets do not have the same noise added and which further support this trend. ablation analysis. we found that both knowledge distillation and contrastive estimation (eq. 1-2) losses are necessary in tandem to obtain the benefits of structured generalization. we found no significant gains when adding one loss alone. appendix d contains detailed results on these ablations. ## discussion and conclusion our structured distillation approach achieves promising results. most prominently, the structured distillation approach significantly improved the performance of the nlp models in settings where noisy facts were added to the clutrr reasoning problems. the gnn-based models are particularly strong in this setting (see appendix c), and this suggests that transferring knowledge about the relevancy of facts from structured to unstructured models may be a promising direction. however, at the same time, the improvements for generalization were less substantial, indicating that some reasoning capacities are difficult to distill in this manner. moreover, despite the improvements we observed, the performance of the nlp models is still substantially below the performance of the gnn teacher used for distillation (see appendices b & c), highlighting that significant work that remains to close the gap between the reasoning performance of text-based and gnn-based models. zhilin yang, peng qi, saizheng zhang, yoshua bengio, william cohen, ruslan salakhutdinov, and christopher d. manning. 2018. hotpotqa: a dataset for diverse, explainable multi-hop question answering. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369-2380, brussels, belgium. association for computational linguistics.