What Meaning-Form Correlation Has to Compose With: A Study of MFC on Artificial and Natural Language (2020)

Abstract: Compositionality is a widely discussed property of natural languages, although its exact definition has been elusive. We focus on the proposal that compositionality can be assessed by measuring meaning-form correlation. We analyze meaning-form correlation on three sets of languages: (i) artificial toy languages tailored to be compositional, (ii) a set of English dictionary definitions, and (iii) a set of English sentences drawn from literature. We find that linguistic phenomena such as synonymy and ungrounded stop-words weigh on MFC measurements, and that straightforward methods to mitigate their effects have widely varying results depending on the dataset they are applied to. Data and code are made publicly available.

## introduction compositionality is one of the core aspects of language: when speaking, we weave complex propositions out of unanalyzable atoms of meaning. this intricate interplay between symbols and sentences has been an important point of research in areas ranging from philosophy of language (frege, 1884) to distributional semantics @xcite , and has been held as one of the key characteristic differentiating human language from animal communication @xcite . compositionality in natural language is usually captured by the principle of compositionality. this principle, often attributed to frege although this paternity is debatable @xcite , has been worded in different ways; let us quote a classic textbook @xcite : "the meaning of a composite expression must be wholly determined by the meanings of its composite parts and of the syntactic rule by means of which it is formed". one suggestion, that can be traced back to @xcite , and that was fully operationalized in @xcite , is that compositionality can be measured as a correlation between meaning and surface form (i.e., the sequence of tokens). if we consider sentences such as "i saw a black cat" and "i saw a black crow", we see that minute changes in the form of a sentence entail that its meaning does not change too much. compare these examples to sentences such as "i saw a white crow", or "he saw a white crow": the gradual alteration of the form of a sentence also gradually alters its meaning. provided that we can measure similarity between any two sentences forms, and between any two sentences meanings, this observation can be rephrased as follows: the distance between two sentences meanings should correlate with the distance between their forms. we refer to this as meaning-form correlation, or mfc for short. one area where mfc has been employed as a way to both detect and quantify compositionality is the field of emergent communication @xcite @xcite which studies agents (artificial or human) who have to produce messages in order to express well defined meanings. to estimate the compositionality of a set of message-meaning pairs, one can compute a pearson or spearman correlation score between textual distances and meaning distances. additionally, a permutation test (over random permutation of the message-meaning assignment) can produce a p-value indicating whether the correlation is statistically significant as well as a z-score that quantifies in number of standard deviations the difference between the correlation found in the language and the correlation of random assignments. this is done among others by @xcite , @xcite , or spike (2016) using a mantel test @xcite , which is based on pearson's notion of correlation. to the extent of our knowledge, no study has investigated the validity of this methodology. is mfc a coherent method of measuring compositionality, especially when it comes to natural language? or do other factors weigh on the measurements and obfuscate the results? if so, can we identify these factors, and can we control for them? to address this gap, we therefore study how mfc behaves, both in a controlled setup based on artificial data and in more realistic scenarios involving natural language. foot_0 2 mfc and artificial languages mfc-based assessments of compositionality implicitly assume that any change in form should correspond to some change in meaning. this can however be challenged: for instance, synonyms and paraphrases will introduce changes in form that should not entail change in meaning. thus we expect mfc to be sensitive to such phenomena, and this in turn suggests that factors such as synonymy could overpower the effect of compositionality that we wish to detect using mfc. to approach this question, we generate artificial languages containing varying degrees of compositionality as well as potential confounding factors. ## mfc and natural language: definitions the experiments we conducted in section 2 have revealed that confounding factors can impact mfc scores on artificial languages. this raises the question of how mfc behaves on natural instances of compositionality: can the same confounding variables be observed? is it possible to control for them? in short, we now aim to translate our results derived from artificial data to natural language data. computing mantel tests for natural examples of composition requires us to select a set of natural language expressions for which we can compute semantic distances between any pair. taking inspiration from @xcite , we use dictionary definitions. indeed, in dictionaries, the meaning of the word being defined (or "definiendum"; pl. "definienda") is arguably also the meaning of the definition gloss. hence, we can equate the semantic distance between two glosses to the semantic distance between the corresponding definienda. distributional semantics models @xcite allow us to conveniently compute the latter: they represent words as vectors in a space over which we can use metrics such as cosine similarity or euclidean distance, and have moreover been found to match human intuitions @xcite . ## mfc and natural language: sentence encoders the results in section 3 suggest that mfc scores on natural language reflect its compositionality but also confounding structural factors that obfuscate semantic transparency. however, the findings also raise questions about the peculiarity of the dataset, since dictionary definitions do not form a representative sample of language use by any standards. is it possible to replicate the methodology on more naturalistic text data? the main difficulty lies in finding a relevant and practical notion of semantic distance for arbitrary sentences, since human judgments can prove arduous to collect. as an attractive potential strategy, one could use vector distances over semantic representations produced by models of semantic composition called sentence encoders. do these models provide good enough approximations for sentence meaning similarity? if so, we expect to be able to reproduce the effects of confounding factors cosine euclidean -1.00 -.80 -.60 -.40 -.20 0.00 -0.04 -0.22 -0.77 -0.77 -0.43 -0.42 -0.64 -0.66 -0.49 -0.49 correlation random transf. use random lstm infersent skip-thought figure 4: meaning distance metrics evaluated on the sick dataset. (spearman correlation) c o s / l e v . l2 / l e v . c o s / l e v . n . l2 / l e v . n . -.50 -.25 .00 .25 .50 .75 1.00 -0.504 0.754 -0.056 0.238 0.113 0.114 0.342 0.345 correlation random transf. use (a) no control c o s / l e v . l2 / l e v . c o s / l e v . n . l2 / l e v . n . -0.481 0.649 -0.033 0.088 0.090 0.091 0.306 0.311 (b) controlling for stop-words c o s / l e v . l2 / l e v . c o s / l e v . n . l2 / l e v . n . -0.504 0.753 -0.055 0.233 0.114 0.115 0.345 0.348 (c) controlling for synonymy on mfc as predicted by representations derived from sentence encoders. to answer these questions, we replicate our previous experiment on a set of common sentences, rather than definitions. ## related works the core of this work draws upon previous research on mfc @xcite @xcite . this concept finds its roots in works such as the ones of @xcite , @xcite or @xcite , which suggest that while compositionality in language relies on syntactic structures, it could still be measured using surface form. other implementations of mfc include studies centered on correlations between form and meaning at the word or sub-morphemic level-conflicting with the assumption of arbitrariness of the sign-such as the work of gutiérrez et al. (2016), kutuzov (2017), dautriche et al. (2017) or pimentel et al. (2019). more broadly, compositionality has proven to be a fruitful field of research; see for example the overview edited by @xcite . the nlp community has produced models of compositional semantics @xcite @xcite and significant efforts have been made to assess how well they capture human intuitions @xcite @xcite and to analyze their behavior @xcite . it is worth stressing that assessing the degree of compositionality within a corpus or a model, as do these works, is a distinct enterprise from quantifying the degree of similarity between two sentences @xcite @xcite .g.)-more precisely, the methodology of mfc employs the latter to estimate the former. lastly, another related topic of research concerns the use of dictionaries as meaning inventories in nlp. for instance, @xcite , @xcite and @xcite , all employ dictionary entries for purposes ranging from word-sense disambiguation to ontology population and to embedding computation. directly relevant here is the work of @xcite , who leverage the compositional aspect of definitions to compute sentence meaning representations. ## conclusions in this work, we have empirically assessed the validity of meaning-form correlation as a methodology for studying compositionality, both in artificial and natural languages. in all, the relationship between meaning-form correlation and compositionality is not straightforward; experimental setups that would employ the former to represent the latter ought to proceed with caution. confounding factors that we can expect from natural language can also weigh on meaning-form correlation in artificial languages, and their effects are not straightforward to control for. other issues such as the architecture employed to derive meaning representations weigh on the relevance or interpretation of correlation scores. meaningform correlation is very dependent on the actual metrics and dataset under consideration. on the other hand, meaning-form correlation was also shown to be able to detect compositionality, both in controlled artificial setups and with natural language data, suggesting that this methodology can be used to study compositionality as long as due attention is paid to confounding factors. while we have focused on modeling a small number of confounding factors, others can obviously be tested using our methodology. for instance, preliminary experiments suggest that factors such as duality of patterning-i.e., using non-atomic expressions @xcite )-reinforce correlations to some extent, whereas aspects such as free ordering of constituents seem detrimental. this limited study has also focused on a specific implementation of meaning-form correlation, based on mantel tests-however other related setups exist, for example based on mutual information @xcite . we leave them to future studies. another limitation of the present work is that we have focused solely on english: we intend in future experiments to experiment with typologically diverse languages, ranging the full isolating-synthetic spectrum.