Towards Unsupervised Compositional Entailment with Multi-Graph Embedding Models (2023)

Abstract: Compositionality and inference are essential features of human language, and should hence be simultaneously accessible to a model of meaning. Despite being theory-grounded, distributional models can only be directly tested on compositionality, usually through similarity judgements, while testing for inference requires external resources. Recent work has shown that knowledge graph embeddings (KGE) architectures can be used to train distributional models capable of learning syntax-aware compositional representations, by training on syntactic graphs. We propose to expand such work with Multi-Graphs embedding (MuG) models, a new set of models learning from syntactic and knowledge-graphs. Using a phrase-level inference task, we show how MuGs can simultaneously handle syntax-aware composition and inference, and remain competitive distributional models with respect to lexical and compositional similarity.

## introduction drawing an inference over structured text is considered to be a basic aspect of natural language understanding @xcite . to build structured meaning, humans rely on compositionality @xcite . for this reason, much work has underlined the connection between composition, the construction of complex meaning from smaller units, and inference @xcite @xcite . with respect to recently popularised large language models (llms) like bert @xcite , the literature has produced contrasting evidence, both against @xcite @xcite and in support @xcite of these models being able to simultaneously handle composition and inferences with lit-tle to no supervision. however, most of the work has focused on sentence-level inference. multiple pieces of evidence have shown that, when solving such tasks, models strongly rely on biases and spurious correlations in the benchmarks @xcite @xcite . to address this issue, authors proposed to focus on phrase-level tasks (e.g., @xcite ; @xcite ). in particular, @xcite showed that llms learn to make robust compositional inferences regarding adjective-noun phrases only with direct supervision, and linked this ability to non-lexical subword units. while computationally effective, this solution is poorly grounded in linguistic and cognitive theories. recently, @xcite showed how training knowledge-graph embedding (kge) architectures on syntactic graphs leads to distributional models able to learn syntax-aware compositional representations. while these models theoretically satisfy the compositionality principle @xcite , like llms, they still require external resources or training to be evaluated on inference. in this work, we propose to expand syntactic-graphs distributional models (syg) with knowledge-graph, and propose multi-graph (mug) models. we argue that, by training on both data sources, mug could inherit compositional abilities from sygs, and learn to manipulate the hypernym relation from kge. thus, mug models should be able to handle both composition and inference simultaneously in the form of compositional entailment, in a fully unsupervised manner. since previous results found rotation to better encode hierarchical relations @xcite such as entailment, and reflection to be most suitable to represent syntactic information @xcite , we hypothesize that an attention-based hybrid model will be the best architecture to simulta-neously handle compositionality and inference. our contributions are four-fold. first, we introduce multi-graph (mug) models, a new set of embedding models trained on syntactic and knowledge-graphs. second, we provide evidence that, under the correct combination of training method and architecture, mug models can tackle compositional entailment, using a syntax-aware composition. third, we propose a detailed analysis describing the behaviour of the best mug model, clearly showing how the three macro classes of adjectives and the structure of the inference shape the behaviour of the model. fourth, we investigate which abilities, in terms of distributional and knowledge-based, are inherited by mugs. we show that mugs are competitive distributional models, but struggle under a graph-related task, likely due to an incompatibility with respect to negative samples rate during training. the paper is organised as follows. section 2, reviews the related work on compositional entailment and different embedding models. in section 3, we lay out the methodology behind mug models, in terms of training methods, compositional entailment predictions, and model's parameters (such as the composition strategy). section 4 describes training and evaluation datasets, and other implementation details. in section 5, we present and analyse results on compositional entailment, graph completion and distributional similarity. section 6 presents a discussion on the overall findings of the work, and how they fit in the current literature. ## background and related work compositional entailment a niche of work exists on phrase-level entailment, mostly focusing on adjective-noun (an) phrases (e.g., brown dog entails (|=) animal). @xcite used nonintensional adjectives solely in the form of an |= n. @xcite used an phrases as a data augmentation technique to improve lexical entailment classification. recently, @xcite introduced plane, a benchmark to train and evaluate models on phrase-level adjective-noun entailment, which will be used in this work. all instances of the dataset are built out of true (noun (n), hypernym (h(n))) pairs, modified by an adjective (a). items can take three entailment structures (or inference types (its)): an |= n, an |= h(n), and an |= ah(n). instances are then automatically annotated using rules defined by the three classes of english adjectives: intersective (i), subsective (s) and intensional (o). ## experimental setup our investigation focuses on two main questions. first, can mugs in fact manipulate both composition and inference? to test this, we will compare mug and kge models on a compositional entailment task. second, what ability, if any, will be penalised or completely sacrificed, in order for a model to tackle compositional entailment? to answer this question, mugs will be compared to kge on a standard graph completion task, and to distributional models trained on syntactic graphs (sygs) on multiple similarity benchmarks. ## discussion our work introduced mugs, a set of embedding models learnt from multiple graph-based sources. under specific and predicted conditions (i.e., using an attention-based model and syntax-aware composition), mugs can be shown to simultaneously tackle compositionality and inference with some success. experiments revealed that mugs tuned for compositional entailment are competitive distributional models, with respect to both lexical and compositional similarity, yet struggle with graph completion. our analysis suggests that a considerable part of the trade-off can be explained by the negative samples rate used for training. the best mug model at compositional entailment, attealtern-rh, was tuned on validation data to 35 negative samples. as summarised by the analy-sis in figures 6 , 5 , and 8, whilst similarity tasks, especially compositional ones, also benefit from high negative samples rate, graph completion tends to require low negative samples (and the static training method) to achieve the best performance. of note, the compositional entailment experiment presented in this work can also be interpreted with respect to knowledge-graphs. despite a different evaluation method (accuracy instead of rank), the proposed task is a type of graph completion. the evaluation is still binary and requires the manipulation of hierarchical structures through the hypernym relation. hence, mugs can be interpreted as compositional kge models. indeed, an llm like bert can achieve better results on compositional entailment as defined by plane. however, it can only do so with direct supervision, and relying on an effective yet not theoretically-sound mechanism @xcite . since mug are trained only with uni-grams, our approach to phrase-level inference (i.e., compositional entailment) is fully unsupervised, requires significantly less training data, and has a deeper connection with the principle of compositionality @xcite . on each compositional task, linguistically-sound word encodings composed with a syntax-aware non-linear composition strategy yielded the best performance. moreover, when a model does not present a strongly heuristical behaviour, we found that the three adjective classes pose as many different challenges to the models, similarly to what already observed in @xcite . ## conclusions and future work in this work, we introduced multi-graph embedding models (mugs), a set of models trained on syntactic and knowledge-graphs. under specific conditions, mugs can partially tackle compositional entailment, making use of syntax-aware composition, based on attention. we provided evidence that mugs are competitive with distributional counterparts on lexical and compositional similarity benchmarks. our analysis suggested that compositionality is supported by a higher number of negative samples, and connected this evidence to the low performance of mugs on graph completion. future work will have to primarily focus on developing a training strategy to overcome the negative samples issue, able to obtain a better integration of the two sources of data and produce a more sta-ble performance across tasks. lastly, mug models will have to be tested on other types of compositional entailment (e.g., noun-noun or verb-object phrases), as well as full sentences.