Evaluating Compositionality of Sentence Representation Models (2020)

Abstract: We evaluate the compositionality of general-purpose sentence encoders by proposing two different metrics to quantify compositional understanding capability of sentence encoders. We introduce a novel metric, Polarity Sensitivity Scoring (PSS), which utilizes sentiment perturbations as a proxy for measuring compositionality. We then compare results from PSS with those obtained via our proposed extension of a metric called Tree Reconstruction Error (TRE) (CITATION) where compositionality is evaluated by measuring how well a true representation producing model can be approximated by a model that explicitly combines representations of its primitives.

## introduction compositionality is the principle inherent in human language whereby the meaning of a complex, compound language expression can be deduced from the meanings of its constituent parts and how they are combined. compositionality can be thought of as a key ingredient towards making artificial intelligence more like general human intelligence since it enables understanding of highly complex concepts by breaking them down into simpler, more manageable, and modular components. the last couple of years have seen a breathtaking expansion in the research around transfer learning for natural language understanding. bert @xcite has proven to be a highly successful model for learning general, task-agnostic sentence representations that can equal or outperform taskspecific ones. given the strong intuitive connection between compositionality and generalization of representation learning, but the relative difficulty in often quantifying it, our goal is to propose evaluation metrics for compositional understanding of 1 stanford scpd student sentence encoders and evaluate the level of compositional understanding in the current state-of-the-art sentence encoder models. we propose two new methods to evaluate the compositionality of sentence embedding models. first, we propose a new method called polarity sensitivity scoring (pss) which measures compositionality via the ability of sentence encoding models to be sensitive to minor perturbations in the input that would flip the sentiment polarity of a sentence. next, we extend tree reconstruction error (tre) @xcite to work sentences. ## related work with the rapid improvement of natural language understanding models in recent years, there has simultaneously been a large increase in research on the nuances and pitfalls of these models, especially in the area of compositionality. among other methods, measuring performance in classification tasks targeting semantic understanding @xcite , lexical composition @xcite , synonym substitution @xcite , and divergence @xcite have all been proposed. many researchers have shown evidence that inducing compositionality into deep and shallow models have helped in generalization, data efficiency, and interpretability. @xcite evaluates compositionality at the phrase level to make representations more interpretable. @xcite finds that neural networks are capable of subtle grammar-dependent generalizations, but do not rely on systematic compositional rules. @xcite found that, perhaps counterintuitively, cnns were able to significantly outperform lstms and grus on the more difficult jump and around-right tasks in the scan challenge proposed by lake and @xcite and @xcite . however, they still find that cnns also are not good at learning rule-like compositional generalizations as the mistakes it makes are not systematic and they are evenly spread across different commands. ## polarity sensitivity scoring (pss) our primary contribution is a method we propose is called polarity sensitivity scoring. here, we posit that a model that has strong compositional understanding can adapt to small changes in the constituent components of a sentence such as sentiment polarity. generally, the sentiment of a sentence is localized to a small fraction of the words, which can be separated from the overall content of the sentence that is not sentiment bearing. we hypothesize that if a model can accurately detect a sentiment switch when its thematic content remains constant, but only its tonality changes, then it should have a good semantic understanding of the nuances of composition structure. we define the equation for pss as: the perturbation-driven nature of pss might lead one to question whether pss really captures compositional understanding or is it just a test of the robustness of sentence representations to noise. we believe that, at least with respect to the required compositional understanding to correctly classify sentiment @xcite , it does and might also be more general than that. pss can actually complement the consistency score proposed by @xcite which measures substitutivity, one of the five tests for compositionality. while they replace words with their synonyms and expect the same classification, we replace sentiment bearing words and expect the model to accurately reflect this change in sentiment. since changing a classification label establishes a more direct causal link between change in text and change in label, we believe that our method is better at least for the substitutivity test. ## tree reconstruction error (tre) tre (andreas, 2019) measures the vector space distance between a target vector representation produced by an encoding model and a vector representation that is generated from compositions of its primitive units. in the case of sentences, the target representation is produced by a sentence encoding model and the primitives are generally the words in the sentence. the compositions are represented by syntactic parses of the sentences where at every subtree, the representations of the child nodes are composed using some composition function. the primitive representations (word vectors) are trained using rmsprop, fixing the sentence representation and compositional functions, to minimize the cosine distance between the sentence encoding and the output of the compositional function applied to primitives. we aim to extend tre 2 from phrases to sentences. unfortunately, there are not many open source datasets with human-labeled compositionality scores for sentences that we could find. therefore, using the stanford sentiment treebank (sst) we propose two automated methods to generate 2 https://github.com/jacobandreas/tre ground truth compositionality labels for the sst dataset by using phrase-level sentiment labels in sst. ## discussion even though our observations from pss and tre approaches are not directly correlated, we observe certain consistencies and see that transformer architectures are different compared to others. while they are more compositional as measured by pss, they appear to be less compositional according to tre. we believe this could be because the two methods are quantifying different kinds of compositionality. @xcite described two different senses of compositionality; ontological and functional. tre seems to measure more of the former since it, by nature of its definition, tries to make combination of primitives equal to the whole while pss measures functional compositionality as it calculates a type of sensitivity which only a model with good compositional understanding can grasp. furthermore, phrases that are similar in vector space can have opposite sentiment. for example, the warm and cool could be close in vector space, but could have a high impact on wns. in the current state, we believe that pss is a more mature method to estimate compositionality for sentences especially since our extension of tre to sentences depends on the efficacy of wns as a good estimate of sentence compositionality. furthermore, even if we did not use wns and had humans tag sentences with scores for compositionality, this would still be quite hard to quantify even for humans given how subjective it can be. expert labelers would be needed for such labeling tasks. however, looking at a positive sentence and switching its sentiment to negative or vice-versa is a much easier task for a human, so dataset creation and evaluation for pss is much more practical. ## conclusion we explored two approaches to measure the compositionality of sentence representations. our primary contribution was proposing polarity switching as a possible measure of compositionality which correlated well with empirical results and our knowledge about inductive biases in sentence encoders. we also extended tre as proposed in @xcite beyond bigram phrases to sentence representations. to do this, we needed to come up with a heuristic approximation of a compositional score for a sentence which we did by using weighted node switching. ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n gomez, łukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems, pages 5998-6008.