Transparency Helps Reveal When Language Models Learn Meaning (2023)

Abstract: Many current NLP systems are built from language models trained to optimize unsupervised objectives on large amounts of raw text. Under what conditions might such a procedure acquire meaning? Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations (i.e., languages with strong transparency), both autoregressive and masked language models successfully learn to emulate semantic relations between expressions. However, when denotations are changed to be context-dependent with the language otherwise unmodified, this ability degrades. Turning to natural language, our experiments with a specific phenomenon—referential opacity—add to the growing body of evidence that current language models do not represent natural language semantics well. We show this failure relates to the context-dependent nature of natural language form-meaning mappings.

## introduction despite language models' (lms) centrality to recent progress on nlp benchmarks, a formal characterization of what can be learned from unsupervised training on large text corpora, and of what modern language models actually do learn, remains elusive. empirically, tenney et al. (2019), kovaleva et al. (2019), wu et al. (2021), among others, all discovered that pretrained lms possess unsatisfactory semantic representations. traylor et al. (2021) found co-variation between form and meaning to be insufficient for an lm to represent lexical semantics. li et al. (2021), on the other hand, identified evidence of lms repre-senting dynamic semantics (kamp, 1981; heim, 1982; groenendijk and stokhof, 1991). from first principles, bender and koller (2020) argued that it is a priori impossible for an ungrounded system that has access only to linguistic forms to learn the mapping between those forms and their grounded denotations. they claimed, as a thought experiment, that a learner that has access to all java code (i.e., form) on github can never learn execution (i.e., meaning). they nevertheless acknowledged that the existence of unit tests, which assert the expected output given input to blocks of code, could constitute a weak form of grounding which potentially enables the learning of meaning. formalizing this idea, merrill et al. (2021) theoretically proved the possibility of learning (or more technically, emulating) semantic relations between expressions in a certain class of formal languages-those that are strongly transparent whose expressions have context-independent denotations-using an assertion oracle, analogous to the assertions in unit tests. in addition, with an example, they showed the existence of non-emulatable languages even with an assertion oracle. yet, the practical implications of these theoretical results have not been explored. while assertions enable the emulation of strongly transparent languages, it is unclear if existing lm architectures and objectives achieve emulation given training data with assertions. furthermore, we do not know if natural language (nl) is similarly nonemulatable as merrill et al.'s (2021) constructed example, especially since non-transparency does not always imply non-emulatability. we thus pose two research questions: rq2. can modern lms fully emulate the meaning of natural language which is non-transparent? we answer rq1 in the positive ( §3): on a strongly transparent propositional logic language, autoregressive and masked language models pretrained on only expressions (form), à la gpt-2 (radford et al., 2019) and roberta (liu et al., 2019c), can consistently compare and evaluate their values (meaning). we find that necessary grounding of the pretraining data distribution is crucial to this ability. we also investigate the role of transparency for emulatability in a controlled setting as an intermediate study before analyzing non-transparent natural language. we ablate strong transparency from the logic language while keeping other factors unchanged. we observe a substantial drop in the lms' ability to emulate meaning, highlighting the importance of transparency for emulatability. we then turn to natural language ( §4). referential opacity is an extensively studied phenomenon in semantics and philosophy (quine, 1956; kripke, 1972, among others) but has not been examined in modern nlp. we prove that this phenomenon entails non-transparency and analyze how well existing lms represent it. our analyses based on probing and sentence similarity point to a lack of its representation in the largest gpt-2 and bert (devlin et al., 2019) models (rq2). theoretically, this is a natural language parallel to the emulation difficulty for our nontransparent formal language, and further reinforces the connection between transparency and meaning emulatability. practically, through the lens of strong transparency, our results supplement prior studies that identified pretrained lms' insufficient semantic representations (tenney et al., 2019; yu and ettinger, 2020, 2021; wu et al., 2021, among others). ## background we follow merrill et al.'s (2021) operationalization of the learning of meaning by emulation and their definition of strong transparency. we summarize their nomenclature and theoretical results in this section and provide some examples. we refer readers to merrill et al. (2021) for more details. at a high level, we take an inferential (speaks, 2021, §2.2.3) view of meaning. an lm is taken to understand a language l if it can resolve semantic relations (e.g., equivalence) between expressions in l. foot_0 this is achieved through two procedures: μ l maps expressions into representations based on training data from l, and δ uses the representations of two expressions to resolve a semantic relation between them. ## discussion through analyses based on probing and direct evaluation, we have seen that existing lm architectures and objectives can learn to emulate the meaning of a strongly transparent language l t when the training data reflects equivalence relations. while non-transparency (l n ) causes this ability to decrease, the trained models still outperform a random model in certain setups. we believe this result hints at the strength of current lm architectures and objectives. @xcite there seems to be a limit to this strength, though-in natural language, neither gpt-2 nor bert represents the non-transparent phenomenon of referential opacity well. our results shed light on the relationship between the strong transparency of a language and whether its semantics can be emulated. we observed co-variation between the two: when slightly perturbed to be non-transparent, our logic language becomes harder to emulate; and there is no evidence for lms representing the semantics of a non-transparent nl phenomenon. nevertheless, the above-random emulation performance with l n suggests that there could be language properties that potentially better predict emulatability, leaving room for future theoretical endeavors. we also found that, with a similar size and training procedure ( §3.2), alm is more suitable for representing the meaning of our propositional logic languages than mlm, in our setup. alm achieves better probing accuracy than mlm under both methods of obtaining sentence representations that we explored. also, mlm completely fails to emulate meaning facing non-transparency, but not alm. ultimately, though, we hope to understand if this difference transfers to natural language. our nl investigation reveals that both alm (gpt-2) and mlm (bert) achieve chance-level probing performance on the one phenomenon that we inspected, likely due to its difficulty. it would be interesting for future efforts to further examine their differences, if any, in learning and representing the meaning of other nl phenomena. our results also lead to the question: why can lms achieve above-random results on l n but not referential opacity? while it is entirely possible that the latter is simply more difficult than our synthetic non-transparency, there are other factors at play. first of all, natural language is much more variable than our synthetic language: utterances can be untruthful (though they are in general governed by gricean quality; grice, 1975), subjective (such as our earlier claim about corgis' cuteness, §2.3), intensional (see merrill et al., 2021 for a discussion), etc. but putting these variations aside, we saw from §3 that even the synthetic language requires an explicit grounding @xmath0, and this is missing from nl pretraining. it is certainly not the case that, for every expression such as ''corgis are the cutest dogs.'' that exists in the pretraining corpus, the variations ''the cutest dogs are corgis.'', ''corgis are corgis.'', ''the cutest dogs are the cutest dogs.'' are also guaranteed to appear. so perhaps there needs to be a more foundational change in our pretraining objective. as brown et al. (2020) foretold, ''a more fundamental limitation of [...] scaling up any lm-like model [...] is that it may eventually run into (or could already be running into) the limits of the pretraining objective.'' our results point to one such possibility: we believe research into a more explicit representation of semantic relations in future pretraining processes, such as based on paraphrases, could be fruitful. what we did not investigate, though, is whether partial equivalence grounding enables emulation: what if, for example, only 1% of the pretraining data has this form of grounding, while the rest does not? and the above format already exists for certain sentences in nl. this, too, could be an exciting future research question. ## related work bender and koller (2020) initiated the discussion on the possibility of a learner acquiring meaning from training on linguistic forms alone. from first principles, they argued for its impossibility. empirically, traylor et al. (2021) also found that lms cannot well-represent lexical-level symbols when the pretraining data is distributionally constrained to supply relevant signals. merrill et al. (2021), on the other hand, proved theoretically that it is possible to emulate the meaning of strongly transparent languages with assertion oracle access. we showed in this work that, empirically, lms also attain the capability. the work of patel and pavlick (2022) is also conceptually similar to our work, discovering that the internal representation of lms is to a large extent isomorphic to the conceptual spaces of directions and colors. they adopted in-context learning (brown et al., 2020; among others) to elicit the isomorphism, while we used the more traditional probing paradigm. another line of work has inspected the extent to which pretrained lms encode various types of semantic information. some have examined the representation of lexical semantics: garí soler and apidianaki (2021) found that bert representations reflect polysemy levels, and vulić et al. (2020) showed that they also capture abundant type-level lexical knowledge. on the other hand, ettinger (2020) and ravichander et al. (2020) have discovered that pretrained lms do not satisfactorily encode negation and hypernymy, respectively. moving beyond the lexical level, wu et al. (2021) demonstrated that pretrained bert and roberta models less readily surface semantic dependency information than syntactic dependencies, while li et al. (2021) identified evidence of dynamic semantics representation in these models. ## conclusion we have empirically shown that pretrained language models are able to emulate the meaning of a strongly transparent language through pretraining on an assertion-inspired format, but this ability deteriorates when the language is minimally perturbed to be no longer strongly transparent. furthermore, we found no representation of referential opacity, which is significant for being a non-transparent natural language phenomenon, in pretrained lms. in §c.1. for language entities, for example, one template is '' [pronoun] [verb] to speak [entity].'' a sentence pair is formed by filling in [entity] with a definite description vs. a proper name for a fact. we only consider the pronouns ''she'' and ''he'' in this work. we consider 6 referentially transparent verbs (''starts'', ''begins'', ''ceases'', ''stops'', ''managed'', ''failed'') and 6 referentially opaque verbs (''wants'', ''intends'', ''hopes'', ''begs'', ''preferred'', ''suggested''). the second type of context contains only the main clause. we use the referentially opaque template ''[pronoun] dislikes [entity].'' and an entity category-specific referentially transparent template such as ''[pronoun] speaks [entity].'' in total, we have 64,672 sentence pairs for gpt-2 and 121,768 for bert. for our probing analysis, we also included attractors with coordinated sentences ( §4.4). as there are a quadratic number of possible coordinations, we subsampled 59,548 such sentences for gpt-2 and 119,540 for bert, similar to the number of attractor-less sentences. we split all sentence pairs 8/1/1 for training/validation/testing. for our similarity analysis, for a cleaner significance test, we only consider sentence pairs with an embedded clause. this leaves 58,776 sentence pairs for gpt-2 and 111,312 for bert.