What Do Language Models Hear? Probing for Auditory Representations in Language Models (2024)

Abstract: This work explores whether language models encode meaningfully grounded representations of sounds of objects. We learn a linear probe that retrieves the correct text representation of an object given a snippet of audio related to that object, where the sound representation is given by a pretrained audio model. This probe is trained via a contrastive loss that pushes the language representations and sound representations of an object to be close to one another. After training, the probe is tested on its ability to generalize to objects that were not seen during training. Across different language models and audio models, we find that the probe generalization is above chance in many cases, indicating that despite being trained only on raw text, language models encode grounded knowledge of sounds for some objects.

## introduction despite being trained only on surface-form strings (i.e., without explicit grounding), language models (lms) have been shown to learn representations of perceptual concepts that plausibly mirror the grounded, physical representations of those same concepts. examples of such concepts that have been investigated so far in the literature include color @xcite , direction @xcite , size @xcite , geography @xcite @xcite , time @xcite , and even visual representations @xcite @xcite . the alignment between an lm's induced representation of a concept (e.g., the space of word embeddings for colors) and its physical (or human perception-like) representation (e.g., rgb space) has direct implications for how much explicit grounding is necessary for an lm to learn about the "real world" referred to by the textual @xcite , visualized via pca. the language representation is from bert @xcite and the audio representation is from passt @xcite . the classes are color-coded based on their parent nodes (i.e., human voice, domestic sounds, animal, music) according to the ontology from the fsd50k @xcite . (bottom) a zoomed-in portion of the blue region of the top figure, which shows the structural similarities between the language and sound representations for the music category. we conduct the sound probing study across 6 language models and 3 audio models. the language representations include those from word vectoronly models (glove @xcite , word2vec @xcite ), encoders (bert @xcite , t5 @xcite ), and decoders (gpt-2 @xcite , llama @xcite ). on the audio side, we experiment with two types of models: self-supervised models that have been pretrained without access to any external labels (audiomae; @xcite , and supervised models that have been pretrained on sound event classification (finetuned au-domae, pann @xcite , passt @xcite ). while all audio models are trained without explicit access to symbolic language data, the representations from supervised models implicitly encode more human perceptionlike priors given that the classification task itself incorporates information about what snippet of sound constitutes a salient-enough signal to humans to warrant its being classified as a distinct event. that is, purely self-supervised models are more likely to encode more physical (i.e., acoustic) representations whereas supervised models are more likely to encode more human perception-like (i.e., auditory) representations. on both acoustic-and auditory-like sound representations, we find that all language models generalize to unseen classes at an above-chance level. we also find that the generalization performance is typically better for sound representations that have been supervised on sound event classification. ## results figure 3 shows the accuracy@3 metric for the different language/sound representation combinations. in the appendix, we report the full numeric values, including accuracy@1 and standard error across the 5 runs. we find that most language models perform well above chance. moreover, within a model family, larger models almost always outperform their smaller siblings (e.g., bert-large vs. bert, gpt-2-xl vs. gpt-2, t5-large vs. t5). however, there is significant variation across families and larger models aren't always better across models (e.g., gpt-2-xl vs. llama-7b). despite their simplicity, word2vec and glove obtain nontrivial performance, sometimes outperforming much more sophisticated models. across the different audio models, we find that alignment is overall better for sound representations from passt, which arguably is most aligned to human perception insofar as it is pretrained as an image classifier, and then finetuned as a sound event classifier. the underperformance of audiomae (which is pretrained only on spectrograms via self-supervision and thus likely to focus only on acoustic information) against audiomae-ft (which is finetuned as a supervised classifier on top of audiomae and thus likely to additionally encode auditory-i.e., human perception-likeinformation) further highlights the importance of human-like representations that emerge from learning to predict human-derived labels. however, despite being trained as a supervised model, pann performs the worst. this may be due to the fact that pann's audio input is in the time domain, as well as the fact that pann uses a cnn architecture instead of a transformer. ## discussion and limitations our work, along with the line of work on aligning language model representations to grounded representations, provides evidence that modeling statistical correlations among surface-form text could lead to learning nontrivial structures about the real world. in hindsight, this is perhaps not so surprising; both language and sound are different "projections" of the same physical world, and thus it is not inconceivable that models trained on the respective modalities represent (some) aspects of the original physical world in a similar way. more generally, the fact that current language models (and foundation models more generally) are trained only on "raw form" (such as word pieces, sound waves, pixels, etc.) is not an inherent limitation on their ability to learn physically grounded conceptual spaces. these models are typically trained (implicitly or explicitly) to compress their training data into their parameters; insofar as good compression can be achieved by learning the underlying generative process, it is possible that aspects of the physical world which were involved in the generation of language could be learned just through form-only training. nonetheless, formonly training is likely to be highly data-inefficient. this work only explored whether sound representations that were learned by an auxiliary audio processing model were encoded through text. here we found that sound representations that are more likely to encode auditory (i.e., human perception-like) information were more aligned to the text representations than sound representations from purely self-supervised models which were just trained on spectrograms, and thus more likely to encode acoustic information. however, even the self-supervised audio models implicitly encode human perception-aligned priors given that the input data consisted of snippets of audio that corresponded to different sound events, which itself is derived from humans. it would be interesting to see whether it is possible to probe out even more low-level representations of objects (e.g., raw spectrograms, pixels) from language models. similarly, as discussed in footnote 7 our audio representations are not completely independent of language as their training sets included a significant amount of human speech. it would therefore be interesting to see if audio models trained without any human speech learn representatons that can be aligned to language models. 6 related work probing language models. language models have been shown to encode much linguistic information in their contextualized representations @xcite @xcite and attention distributions @xcite . building on top of these more linguistically-oriented probes, there has been mounting recent evidence that language models trained on just text are able to meaningfully encode a surprising amount of grounded or extralinguistic information, such as color @xcite , direction @xcite , size @xcite , geography @xcite @xcite , time @xcite , visual representations @xcite @xcite , character-level information of wordpieces @xcite , and representations of meaning @xcite . our work extends this line of to sounds and investigates the extent to which language models trained on textonly can encode auditory representations. our work is also related to the line of work investigating whether a model that has been trained on raw outputs of a synthetic environment can acquire "true" representations of that environment. examples of such environments include othello @xcite , chess @xcite , and toy grid worlds @xcite . meaning in language models. whether language models can acquire meaning and understanding from being trained on form alone is the subject of much debate @xcite @xcite @xcite . in operationalizations of meaning which do not rely on explicit reference to the external world, the fact that the geometry of language models' representation spaces is structurally related to the geometry of grounded representations could be construed as evidence for these models' acquiring meaning in some broad sense. ## conclusion we probe text-only language models for whether their representations of an object contain grounded representations of the sounds of the same object. we find that this is indeed the case, and a contrastive probe can often generalize zero-shot to object classes not seen during training.