Is a cute puyfred cute? Context-dependent form-meaning systematicity in LLM s (2025)

Abstract: We investigate static and contextualized embeddings for English pseudowords across a variety of Large Language Models (LLMs), to study (i) how these models represent semantic attributes of strings they encounter for the very first time and how (ii) these representations interact with sentence context. We zoom in on a key semantic attribute, valence, which plays an important role in theories of language processing, acquisition, and evolution. Across three experiments, we show that pseudoword valence is encoded in meaningful ways both in isolation and in context, and that, in some LLMs, pseudowords affect the representation of whole sentences similarly to words. This highlights how, at least for most LLMs we surveyed, pseudowords and words are not qualitatively different constructs. Our study confirms that LLMs capture systematic mappings between form and valence, and shows how different LLMs handle the contextualisation of pseudowords differently. Our findings provide a first computational exploration of how sub-lexical distributional patterns influence the valence of novel strings in context, offering useful insights for theories on the form-meaning interface and how it affects language learning and processing.

## introduction recently, a few studies have focused on pseudowords -phonotactically legitimate strings in a language which however lack conventional meaning -, uncovering a rich web of associations between sub-lexical patterns and semantic dimensions @xcite @xcite @xcite . this perspective blurs the distinction between pseudowords and words @xcite @xcite : from a learning perspective, every word a speaker knows used to be a pseudoword. however, the first time speakers experienced a novel word, it likely appeared in context @xcite . nonetheless, no study has yet considered how sentence context influences the semantic connotations of entirely novel words. we aim to start exploring the interplay between systematic form-meaning mappings and linguistic context, relying on computational analyses of current llms. how is lexical meaning constructed for novel words upon first encounter? what role does the linguistic form of words play in the process, if any? how does it interact with context, which is a key source in learning novel words @xcite @xcite ? in addressing these questions, we rely on evidence that has challenged the notion of arbitrary form-meaning mappings @xcite . many studies have observed sound symbolic associations @xcite in typologically diverse languages @xcite @xcite . these observations support the notion that form-meaning mappings are best characterized as at least partly systematic @xcite @xcite . the extent to which systematicity documented in isolated words and pseudowords interacts with the semantics of co-occurring words is an open question we aim to address in this work. we present three experiments which investigate pseudowords by focusing on a specific semantic attribute, i.e., valence, which characterizes words by how positive or negative they are. theoretically, this semantic dimension has been identified as a critical axis along which both people and language models structure semantic representations @xcite . practically, large datasets of human ratings for both words and pseudowords are available for this specific semantic dimension, making it a viable starting point to investigate how semantic connotations of entirely novel words interact with sentence context. our first experiment focuses on static representations learned by different llms and how well they capture human ratings of pseudoword valence. the second experiment studies how such static representations are affected by co-occurring words. finally, the last experiment investigates how pseudowords influence sentence representations. our study provides the following key contributions: llms capture pseudowords' valence: we see a sizable correlation between human valence ratings and static embeddings produced by all llms for pseudowords, confirming that the tokenization in part-words allows llms to encode form-meaning mappings. systematicity interacts with context: llms behave differently, but two cognitively sensible patterns emerge. some models are only influenced by co-occurring words, discounting form-related information; others are influenced by both, suggesting that upon first encounter, pseudowords might retain valence associations conveyed by the linguistic form. pseudowords affect sentence representations: across models, the valence of sentences is systematically influenced by pseudowords. moreover, the influence of pseudowords is similar to that of words, strengthening the hypothesis that words and pseudowords are not qualitatively distinct. ## experiment 1: static representations of novel words we replicate the design from @xcite , investigating to what extent the static representations that five llms learn for pseudowords encode valence in line with human intuitions. ## experiment 2: how novel words are represented in context after establishing that all llms represent isolated pseudowords reflecting human ratings on pseudowords' valence, we turn to investigate how pseudowords' representations encoded by these models are affected by co-occurring words. how do they morph, if they do at all, when context kicks in? do they change entirely according to the other words they co-occur with or do they maintain a stable, even if somewhat blurry, semantics? ## experiment 3: how novel words affect sentence embeddings in this last experiment, we investigate to what extent the valence encoded in pseudowords alters the representation of whole sentences. once again, if pseudowords retain any of the semantics they encode when considering their static embeddings, we expect that sentence valence should change when sentences include different pseudowords. we further consider what happens when plugging words with different valence ratings in the sentences: intuitively, the word plague should make a sentence containing positive words such as we were all incredibly happy to celebrate the funny [open slot]. quite less positive. will the same happen for a negatively rated pseudoword like puyfred? ## general discussion across three experiments, we showed that several llms are sensitive to non-arbitrary form-valence mappings in the input text. in experiment 1, we assessed to what extent the static embeddings each model assigns to pseudowords capture valence in line with human ratings. in experiment 2 we explored how such representations are contextualized, systematically varying the valence of co-occurring words. finally, in experiment 3, we investigated the trace that different pseudowords leave in sentence embeddings derived from sentences manipulated to consist of words with different valence. in this endeavor we leveraged gold standard human valence ratings for both words @xcite and pseudowords @xcite . experiment 1 showed that all models develop representations that capture human ratings across a number of evaluation sets which targeted pseudowords with extreme ratings (either very positive or very negative) as well as pseudowords with no clearly similar words that might have functioned as attractors and influenced the rating. while letter unigrams top the chart when considering all pseudowords, they fail to capture differences in perceived valence across positive and negative pseudowords. on the contrary, neural models fare better, particularly llama 3.2 1b. our evaluation thus complements and further qualifies findings from gatti et al. (2024a), highlights the importance of using appropriate and informative evaluation sets, and speaks to the tension between single letters and broader systematic patterns in language when considering the semantic connotations elicited by entirely novel strings @xcite @xcite @xcite . the slight yet consistent advantage of multilingual models further suggests that systematic form-meaning mappings may be partly cross-linguistic @xcite , benefiting a model which is not limited to learn from english form-meaning patterns. at the same time, experiment 2 shows that m-bert and llama discard pseudowords' valence when these appear in context, possibly indicating that a model trained on multiple languages learns to downplay information coming from the make-up of a new word to prioritize contextual information, highlighting a tension between systematicity and arbitrariness across languages. our results further highlight the role of sub-word tokenization in llms @xcite . fasttext @xcite has been extensively used to gauge the likely meaning of novel strings thanks to its decomposition of every string into its constituent, overlapping n-grams @xcite @xcite . sub-word tokenization offers an alternative approach, which splits only less frequent words into non-overlapping segments: this study confirms that part-words do encode systematic form-meaning mappings @xcite . haslett (2025) has explored the role of part-words produced by sub-lexical tokenization in shaping representations considering semantic radicals in chinese: a model's vocabulary, and hence the degree to which it segments word forms into part-words, has direct influences on the representations the model produces, underscoring how form-meaning mappings play an important role in llms. future work should further investigate differences between these two strate-gies (overlapping n-grams from any string versus non-overlapping tokens derived only from those strings that the model does not store as whole words) in capturing non-arbitrary form-meaning relations. the performance of llama -a multilingual model, with a large vocabulary -hints to the possibility that learning form-meaning mappings from low-frequency words only might provide more signal. finally, our evidence points to small yet systematic effects of both the perceived valence of a pseudoword and the valence of the words it cooccurs with, with two profiles emerging. on the one hand, m-bert, gpt-2 and llama 3.2 almost exclusively rely on sentence context to represent pseudowords in context @xcite . on the other hand, roberta is predominantly sensitive to the valence of the pseudoword itself but is also influenced by the valence of co-occurring words. at the same time, in some models, sentences tend to absorb the valence of pseudowords, which emerge as having a sufficiently strong semantics to survive the contextualisation process. it is in this sense telling that the effect found when using words of varying valence instead of pseudowords is similar, showing that some llms consider novel strings consisting of part-words akin to words. this fits with behavioral evidence showing semantic effects for pseudowords in (primed) lexical decision tasks @xcite , suggesting the existence of a shared cognitive encoding of words and pseudowords. future work should focus on collecting behavioral data from speakers of typologically distinct languages to understand how humans process pseudowords in context. our results provide principled predictions about how novel words might be represented in context, to further study how our cognitive system deals with this very common situation. especially during acquisition, most words are akin to pseudowords for learners @xcite . understanding how humans as well as cutting edge nlp models handle the ever present situation of quickly having to encode an entirely novel string in context and derive flexible and generalizable representations is an important step in the process of modeling human and artificial language learning @xcite , which will further aid in developing theories of how linguistic form interfaces with meaning to shape this process @xcite . ## limitations this study has a few notable limitations which narrow the generalisability of the findings. first, the study only considers english pseudowords plugged in english sentences. this choice is primarily dictated by which resources are available, with large databases of valence ratings for both words and pseudowords being currently only available for english. this evidently limits the scope of our findings and leaves several questions open. closely related, valence is certainly an important dimension of meaning, but does not encompass meaning: future studies should investigate whether the patterns we observed also apply to other salient semantic connotations. moreover, the lack of human behavioral data about processing of pseudowords in context limits the scope of our findings. at this stage, we can only describe what different llms do, but not yet adjudicate which model behavior more closely reflects human behavior. such data would be crucial to develop cognitive theories of how linguistic form and sentence context affect the first stages in the process by which lexical meaning develops and becomes entrenched in semantic memory. at the same time, a comparison with human behavior would allow to understand which learning mechanisms deployed in language models are responsible for the encoding of pseudowords' semantic associations and their update in context. crucially, we do not assume that a model whose behavior fits that of humans is necessarily human-like. we see and study these models as powerful information processing systems that can illuminate the presence and possible influence of statistical patterns in the input. observing similar patterns of behavior in models and humans can then point to interesting directions to better chart how both behave and uncover fundamental similarities and differences in learning mechanisms and how they leverage input information. another limitation has to do with the fact that we probed text-based models and written pseudowords. exploration of systematicity in spoken and signed pseudowords would be necessary to develop precise theories of how meaning is constructed in context for novel, word-like signs. tied to the issue of the role of form-meaning mappings, our study assumes a full-fledged vocabulary: the question of whether developmentally plausible vocabularies in terms of size and content also support stable and useful systematic form-meaning mappings remain open and cannot be answered based on our simulations. related to data availability, we relied on word valence ratings collected approximately 15 years ago. we believe it is unlikely that the valence of a substantial part of the words in this dataset shifted in this time period, but more recent ratings would improve the validity of the estimates and the modeling. ## ethical statements this work uncovers possible biases in llms representations that bear relevance in the following areas. first, because of sub-lexical semantic associations, these models may encode certain names differently than other, with cascading effects. second, for the same reason, this study shows how llms could be leveraged to craft product and brand names with specific semantic associations that could influence people's behavior in subtle ways. the use of automatically generated sentences in experiments 2 and 3 could pose ethical concerns, which we tried to mitigate by manually reviewing all sentences before using them. finally, the pseudoword and word valence ratings were made freely accessible by the authors: no license was stated for either dataset. no dataset used in this study contains unique identifiers that could disclose personal identities. liu zhuang, lin wayne, shi ya, and zhao jun. 2021. a robustly optimized bert pre-training approach with post-training. in proceedings of the 20th chinese national conference on computational linguistics, pages 1218-1227, huhhot, china. chinese information processing society of china.