Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models (2025)

Abstract: Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discussed before, which we call semantic leakage, where models leak irrelevant information from the prompt into the generation in unexpected ways. We propose an evaluation setting to detect semantic leakage both by humans and automatically, curate a diverse test suite for diagnosing this behavior, and measure significant semantic leakage in 13 flagship models. We also show that models exhibit semantic leakage in languages besides English and across different settings and generation scenarios. This discovery highlights yet another type of bias in language models that affects their generation patterns and behavior.

## introduction as language models (lms) become more prevalent @xcite @xcite , we are steadily learning more about their peculiarities and the unique and often unexpected properties of their behavior. phenomena ranging from hallucinations @xcite to sycophancy @xcite and many types of biases @xcite have been revealed in these models' outputs. each such discovery leads to a cycle of in-depth study and development of new methods to mitigate these behaviors as much as possible. we identify a phenomenon in language models never discussed before, which we term semantic leakage -these models can generate text with strong semantic relationships to unrelated words in the prompts. for example, when given the prompt "he likes yellow. he works as a", gpt4o 1 generates the output "school bus driver" (figure 1 ). here we 1 https://platform.openai.com/docs/models/ gpt-4o ## experimental setup we present the experimental setup for automatically evaluating semantic leakage in a wide range of models and model sizes. ## results significant semantic leakage across various use cases and models. leakage is more pronounced in certain model variations. the results in ## human evaluation we now perform a manual evaluation of semantic leakage, which will also validate our automatic metrics and experiments. human evaluation setup human evaluation is based on the same setting as automatic evaluation, described in section 2.2. we recruit two native english speakers who are not involved with the project. these annotators are not provided with the objective of this experiment, but instead are given the following annotation guidelines: consider the word or phrase x. which of the following texts (a or b) is more semantically related to x? (a/b/neither). we map the annotator's choice of (a/b/neither) to (test, control, neither) and then score leak-rate (eq. 1). using these guidelines, the annotators are asked to label 109 test-control generation pairs from each model (ordered randomly). for the human evaluation, we consider the largest model from each model family at the temperature t found to leak the most by automatic metrics: gpt4o (@xmath1. analysis figure 4 compares the semantic leakage detected by the human evaluation for gpt4o against the automatic metric. the values in the human evaluation row are percentages for each category: test is more similar to the concept, control is more similar to the concept, or neither. we expect humans to have higher tolerance for similar scores, i.e., more cases falling under sim @xmath2, where it occurs almost solely when the test and control generations are the same (see equation 1 ). to visually account for this difference we plot the automatic metric results by using colored gradient to depict the difference in similarity values sim test -sim control , with positive values (shades of blue on the right) implying semantic leakage. the results account for all ten generations sampled from each model. we find similar trends on llama3-70b instruct with an average human-annotated leak-rate of 66.7 and automatic evaluation of leak-rate that range from 71.2 to 77.3 across the different embedding types. we also calculate kendall's τ on the human annotations. we find high interannotator agreement between the human annotators (τ = 0.68), indicating that humans generally agree on the cases that constitute semantic leakage. we also compute kendall's τ on the human evaluation vs. similarity differences calculated using bertscore embeddings to evaluate how well these methods correlate, 14 and get a moderate correlation of τ = 0.39 when averaged between the two annotators. ## multilingual and crosslingual semantic leakage in the previous sections we established that semantic leakage is exhibited in various scenarios in english, across different models and model sizes. we now inspect semantic leakage in languages outside of english, as well as in crosslingual settings, by focusing on hebrew and chinese, hebrew-english and chinese-english. as in the previous section, we consider the largest model from each model family at the temperature t found to leak the most: gpt4o (@xmath3. since we found that llama is struggling with hebrew, especially with hebrew generation, we exclude these results. ## open-ended generation we also examine semantic leakage in open-ended scenarios that encourage the model to generate multiple sentences or paragraphs. it is less clear how to quantify the semantic leakage in this setting, as it can manifest in more ways within the longer output; we therefore rely on more qualitative analysis in addition to the automatic metrics. this section focuses on generations from gpt4o with @xmath4, as this model with this temperature was most prone to leakage in previous experiments (section 4). stories a popular setting for open-ended text generation is storytelling: we ask the model to write a story about a child, with the child's name being the concept we expect to leak. figure 6 gives the story that gpt4o generates when prompted to create a story about a child named coral (and also melody in figure 9 in the appendix). these generated stories contain many words and elements directly related to the semantics of each name, and 17 to the best of our knowledge, the model underlying sen-tencebert was trained (predominantly) on english. this is why the metric detects more leakage in cross-lingual settings (with evaluations on english) compared to other languages. the narrative of each story is tailored to those meanings. when measuring semantic leakage in this setting automatically for 23 different names, 18 we get leak-rates of 78.7, 60.9 and 76.1 with bs, sb and oai embeddings respectively, suggesting similar rates of leakage to the other examples. ## conclusion we identified a phenomenon in generative language models never discussed before, which we term semantic leakage: language models are prone to generating text that reflects unrelated semantic information from the prompt, leading to peculiar and sometimes unreasonable outputs. we defined and measured semantic leakage in a range of models, and together with human evaluation show that it is prevalent and consistent across all models we test. we also found that semantic leakage occurs in many different generation settings, including multilingual and crosslingual ones. this leakage reflects associations learned by the model, similar to how different types of biases are learned; therefore, our characterization of semantic leakage broadens the scope we should consider with respect to potential ramifications of learned associations. while the implications of semantic leakage are not fully clear, it is a consistent behavior pattern in lms that merits further study. the finding that instruction-tuned models leak more is of special interest, given that they are the most popular and best performing model variations currently used. we hypothesize that semantic leakage is more dominant in these models because the leaking generations are less generic and seem to provide more information/content, which might be a property that is incentivized under these finetuning processes. we plan to explore this hypothesis more formally in future work.