AND does not mean OR : Using Formal Languages to Study Language Models’ Representations (2021)

Abstract: A current open question in natural language processing is to what extent language models, which are trained with access only to the form of language, are able to capture the meaning of language. This question is challenging to answer in general, as there is no clear line between meaning and form, but rather meaning constrains form in consistent ways. The goal of this study is to offer insights into a narrower but critical subquestion: Under what conditions should we expect that meaning and form covary sufficiently, such that a language model with access only to form might nonetheless succeed in emulating meaning? Focusing on several formal languages (propositional logic and a set of programming languages), we generate training corpora using a variety of motivated constraints, and measure a distributional language model’s ability to differentiate logical symbols (AND, OR, and NOT). Our findings are largely negative: none of our simulated training corpora result in models which definitively differentiate meaningfully different symbols (e.g., AND vs. OR), suggesting a limitation to the types of semantic signals that current models are able to exploit.

## introduction a current open question in natural language processing is to what extent language models (lms; neural networks trained to predict the likelihood of word forms given textual context) are capable of truly understanding language. @xcite argue that, since such models are trained exclusively on the form of language, they cannot possibly learn the meaning of language. we argue that the question of whether language models can learn meaning cannot be settled a priori. while language models only have direct access to form, linguistic form often correlates with meaning. the strength of the correlation varies across both different aspects of language and different tests of linguistic competence. while several intuitive tests of un-derstanding (e.g., demonstrating knowledge of the word dog by identifying pictures of dogs) are out of scope for lms, many tasks which nlp aspires to solve (e.g., question answering, machine translation) operate entirely on natural language input and output. thus, a relevant question is whether models which operate only on the forms of language can nonetheless learn to differentiate meanings. our goal is to focus on a tractable subproblem in order to improve our intuitions about the types of distributional signals that lms can use to extract information relevant to meaning. we simulate a language modeling setup using propositional logic, in which we can naturally operationalize form to be strings of symbols in the language and meaning to be truth conditions. we define the semantic transparency of a text-only training corpus to be the degree to which an lm trained on that corpus learns to differentiate between aspects of form that affect truth conditions and aspects of form that do not. we have two primary research questions. first, what constraints on corpus generation produce greater semantic transparency? and second, are any such constraints sufficient for an lm to adequately differentiate meanings? ## results and discussion language modeling performance. we first sanity check that the trained models indeed function as lms before evaluating the lexical representations. we compute the models' perplexity on heldout data. however, since perplexity is not comparable across conditions (since each constraint leads to differently distributed corpora) we also sample 1,000 generated sentences from each model and compare by measuring whether the sentences are 1) syntactically well-formed (i.e., parseable) and 2) semantically well-formed (i.e., satisfiable). even in the case of models trained with the syntactic constraint, as seen in representations of logical symbols. again, our first question is: what constraints on corpus generation yield the greatest amounts of semantic transparency? we quantify this by measuring how well the embeddings learned by the trained lms correspond to our truth-theoretic notions of semantic equivalence: e.g., are ∧ 1 and ∧ 2 more similar to one another than ∧ 1 and ∨ 1 ? we use a nearest neighbors probing classifier to evaluate whether models distinguish the operators at the lexical level. we run k-fold cross validation, in each iteration choosing one symbol per class (i.e., one ∧, one ∨, one ¬) as the class exemplars, and then classifying the remaining points using cosine similarity. we set k to 125, so that we observe every symbol combination as exemplars. we report accuracy averaged across folds and random restarts. probing classifier results are shown in figure 1 . figure 2 shows an embedding visualization for one model (medium transformer). we find that training on the syntactic and on the explicit grounding dataset leads to the least and the most distinguishable operators respectively for all models, and the other conditions end up between these values. these results address our first question: there is some difference in semantic transparency between differently constrained datasets. interestingly, the transformer models perform better in the truthfulness condition than in the syntactic condition, which the lstms fail to differentiate. this suggests that, even if it does not necessarily manifest in the models' generations ( in addition, we observe little difference between the quality of the representations learned in the informativity condition and those learned in the truthfulness condition; one exception might be in the medium lstm, though we cannot confirm that this difference is robustly reproducible. thus, random syntactic and synonyms or synonyms not synonyms truthfulness explicit grounding figure 2: pca of the representations created by the medium transformer model. finally, we note that in nearly all cases, models are able to differentiate ¬ from the other operators, likely because it is a unary operator and thus syntactically different from the binary operators. thus the difference in accuracy is almost entirely due to whether the representations of ∧ and ∨ are differentiated (as shown in figure 2 ). this gives a negative answer to our second question concerning whether any constraints are sufficient for an lm to adequately differentiate meaning. apart from the small transformer on the explicit grounding condition, none of the models can completely distinguish between symbols that are similar in form but different in meaning. ## related work it is an open question whether neural models can learn abstract functions @xcite . our work builds upon a large body of research intended to probe which aspects of language and meaning are being captured by large lms. most closely related is work that assesses whether models can perform symbolic reasoning about language @xcite ) e.g., quantifiers or negation @xcite @xcite or by measuring the systematicity of models' inferences @xcite @xcite . such work has tended to find that lms reason primarily contextually as opposed to abstractly. our evaluation method-which asks whether word embeddings cluster according to their truth-conditional meaning-is related to recent work which defines text-only models as "grounded" if the learned embedding space is isomorphic to the similarity function defined over a ground-truth meaning representation @xcite . more distantly related is work on lms' ability to reason about numbers @xcite or perform multi-hop reasoning @xcite . prior work that examines neural networks' ability to perform logical reasoning is superficially related @xcite . in this way, our work builds on past work that uses synthetic rather than natural language datasets in order to probe model behavior in the absence of confounds. notable examples are scan for measuring compositionality and generalization @xcite and @xcite which investigates lm knowledge acquisition and fact memorization using a synthetic dataset of entity-relation tuples. ## conclusion using propositional logic corpora to simulate a controlled language modeling setting, we ask: 1) do properties of the training corpus affect lms' abilities to differentiate the meanings of logical operators? and 2) do any training corpora lead to models that differentiate these meanings to a satisfactory degree? our results imply a positive answer to ( 1 ): models trained on corpora generated with different constraints appear to perform differently at the task of separating ∧ from ∨. however, these differences are a function of both data and model. for example, the transformer architecture seems better able to learn from weaker signal (corpora generated only with a truthfulness constraint), while lstms require more explicit signal (direct access to truth values). on question (2), our results are largely negative for the syntactically similar operators. even the most semantically transparent training data did not enable models to separate the representations of symbols with similar form but different meaning. only the small transformer trained on the explicit grounding condition can perfectly differentiate ∧ from ∨ at the lexical level, despite the task's controlled nature. however, every model did separate ¬ from both ∧ and ∨, illustrating how syntactic differences can support differentiation of meaning. overall, we contribute a novel framework, based on syntax and semantics of propositional logic, via which we can explore questions of the linguistic capabilities and weaknesses of neural lms. our experiments represent a first step in this line of work, but further work is needed to fully appreciate the implications of these results in natural language settings, in particular, how closely the constraints explored here mirror real corpora, and how such learning is influenced by noise and ambiguity found in human language. one specific limitation of our experiments is that we constrain our analysis to the lexical representations-i.e., we assume that differences between the meanings of ∧ and ∨ should be encoded in the lexicon, via contextinvariant type embeddings. while this assumption is commonplace in formal semantics, neural lms open the possibility of alternative representations of lexical and compositional semantics. our results do not rule out the possibility that the relevant semantic distinctions are encoded elsewhere in the model, above the lexical layer. however, we take the combination of the lexical probing results and lm generation results as suggestive but not con-firmational evidence of a more general negative finding.