Compositional Generalization for Kinship Prediction through Data Augmentation (2022)

Abstract: Transformer-based models have shown promising performance in numerous NLP tasks. However, recent work has shown the limitation of such models in showing compositional generalization, which requires models to generalize to novel compositions of known concepts. In this work, we explore two strategies for compositional generalization on the task of kinship prediction from stories, (1) data augmentation and (2) predicting and using intermediate structured representation (in form of kinship graphs). Our experiments show that data augmentation boosts generalization performance by around 20% on average relative to a baseline model from prior work not using these strategies. However, predicting and using intermediate kinship graphs leads to a deterioration in the generalization of kinship prediction by around 50% on average relative to models that only leverage data augmentation.

## introduction transformer-based large language models @xcite have achieved state-ofthe-art results on numerous nlp tasks such as question answering, reading comprehension, relational reasoning, etc. story (s) jenny and her father rob went to the park. rob's sister lisa was also happy to see her home, back from college. jenny rob lisa task r 32 : sister r 1 3 : f a t h e r r 12 :? s (e 1 , e 2 ) (e 1 , e 3 ) (e 3 , e 2 ) data augmentation intermediate kinship graph s e 1 e 3 e 2 r 12 r 13 r 32 r 12 s (e 1 , e 2 ) + + g r 12 + (e 1 , e 2 ) (e 1 , e 3 ) (e 3 , e 2 ) e 1 e 2 e 3 figure 1 : to improve the compositional generalization of models for the task of kinship prediction between a pair of queried entities (e.g. predicting the relation r 12 given the entities e 1 and e 2 ) from a story (s) we present two strategies (1) data augmentation and (2) predicting and using intermediate structured representation in form of kinship graphs. for data augmentation (first strategy), we utilize the existing ground truth graph (g) to generate more pairs of target relations and query entities (such as predicting r 13 using e 1 and e 3 ) that do not need compositional inference to obtain the answer. in our second strategy, using our augmented data we predict an intermediate kinship graph and reason over it jointly with the story to predict the relation between the queried pair of entities. in this work, we explore two strategies to improve the compositional generalization of models for the task of kinship prediction from stories. in our first strategy, we explore the utility of data augmentation towards compositional generalization. recent works have shown data augmentation to be an effective strategy in improving model performance on different nlp tasks such as neural machine translation (fernando and ranathunga, 2022), semantic parsing @xcite , and text summarization @xcite . our data augmentation strategy focuses on improving a model's ability to extract relations that are explicitly mentioned in the text. in our second strategy, we explore the utility of predicting an intermediate structured representation of the story (as a kinship graph) and then jointly reasoning over it along with the story text for the task of kinship prediction. figure 1 provides an example of this task and also illustrates the two strategies. the strategies are explained in detail in §3. we evaluate the utility of our strategies on a kinship prediction benchmark, clutrr @xcite . overall, we find data augmentation is helpful and boosts the generalization performance (accuracy of predicting correct relation) by around 20% on average relative to a baseline not using these strategies. however, using intermediate kinship graphs deteriorates generalization performance by almost 50% as compared to the model that only uses data augmentation. our code is available at: https://github.com/ weikangda/data-aug-clutrr . jenn y rob lisa r 32 : sister r 1 2 : f a t h e r r 12 :? g e 1 e 2 e 3 roberta text embedding r-gcn graph embedding + roberta classification head output … jenny and @xmath0 figure 2: ssd model illustration: first obtain the graph embedding and text embedding separately using r-gcn and roberta respectively, then adding the embeddings together and feeding through a classification layer to get the final output. ## problem setup each example in cluttr @xcite is a tuple of the form (s, g, e 1 , e 2 ), where s represents the story/passage describing the entities (fictional characters) and relations between them, g represents the kinship graph, e 1 and e 2 represent the pair of query entities (whose relationship is being queried). to aid clarity on these notations, we have illustrated the values of (s, g, e 1 , e 2 ) corresponding to our running example in figure 1 . further, each kinship graph can be considered to be a collection of entity nodes (e) and relation edges (r) (as illustrated in figure 1 ), where @xmath1, e 2 , e 3 ) and @xmath2, r 13 , r 32 ). note that the kinship graph mentions only the relationships clearly stated in the story. for example, in figure 1 , the entity pairs (e 1 , e 3 ) and (e 3 , e 2 ) are explicitly mentioned in story s. the learning task is to predict the relationship between the two query entities. this is framed as a classification task over 20 possible relationship types in the dataset. the number of composition operations/steps required to infer the relationship between the query entities is denoted by k. for example, in figure 1 , @xmath3. in this work, we empirically evaluate the utility of data augmentation and intermediate structured representations towards compositional generalization for the task of kinship prediction from a story. next, we formally describe our model, ssd, where ssd stands for systematic compositional generalization with symbolic representation and data augmentation for kinship prediction. ## method we first describe our base model followed by a description of two strategies explored in this work -(1) data augmentation and ( 2 2019) ). we obtain the representation for each entity by averaging the tokens from the last transformer layer of roberta corresponding to the positions where the entity appeared in the story. 3. classifier for predicting relation: this is a multiclass classification task (with total number of classes as the number of relationships possible in the dataset) using a linear classifier that takes as input the concatenation of representations of the story and two query entities. ## experiments and results all models are trained using cross-entropy loss. every model is trained with 40 epochs and a learning rate of 5e-6. ## conclusion in this paper, we present ssd to empirically evaluate the utility of two strategies (1) data augmentation and (2) predicting and using intermediate kinship graphs, towards compositional generalization of transformer-based models for the task of kinship prediction from a story. while data augmentation boosts the performance of our model, using intermediate kinship graphs leads to a downfall in the overall performance. data augmentation is fruitful even when additional supervision in form of ground-truth kinship graphs is present for a limited set of examples. future work can explore better methods to fuse the information from the intermediate kinship graph and the story instead of simple concatenation as done in this work.