How Does an Adjective Sound Like? Exploring Audio Phrase Composition with Textual Embeddings (2024)

Abstract: We learn matrix representations for the fre- quent sound-relevant adjectives of English and compose them with vector representations of their nouns. The matrices are learnt jointly from audio and textual data, via linear regres- sion and tensor skipgram. They are assessed using an adjective similarity benchmark and also a novel adjective-noun phrase similarity dataset, applied to two tasks: semantic similar- ity and audio similarity. Joint learning via Ten- sor Skipgram (TSG) outperforms audio-only models, matrix composition outperforms addi- tion and non compositional phrase vectors.

## introduction natural language data consists of words arranged into phrases and sentences. words have statistical representations and phrases/sentences symbolic forms. the formers, mined from co-occurrence counts, fall within the remit of distributional lexical semantics. the latters, often formalised within logic frameworks, are obtained from rules of grammar. a model of natural language should ideally take both into account. consider a simple adjectivenoun phrase. on the lexical side, statistical vector embeddings are learnt for adjectives and nouns. on the symbolic side, e.g. in combinatory categorial grammar (ccg) @xcite , an adjective is a function applied to a noun. the lexical and the symbolic sides are brought together by providing a statistical representation for the ccg rules. for the adjective-noun phrase rule, this is achieved by representing adjectives as matrices, nouns as vectors, and function application by matrix-vector multiplication @xcite . this unified model has been applied to multimodal image-text data @xcite , but never to other combinations such as audio-text. for example, in an audio-text context, adjectives like "loud" or "soft" can modify nouns like "music," where the meaning is enriched by integrating corresponding audio features with their textual representations. our aim in this paper is to fill this gap. we represent the sounds of adjectives by matrices, the sounds of nouns by vectors, and test whether their matrix-vector multiplication is a good representative of the sound of adjective-noun phrase. to this end, we work with two tasks: a semantic similarity task and an audio similarity one. we develop a new dataset of audio relevant adjective-noun phrases and collect human annotations for them. the matrix representations are from the audio data gathered from freesound foot_0 , a collaborative repository of sounds. the correlation between the model's predictions and human annotations is tabulated. these show that matrixvector adjective-noun composition works better than simple vector addition and non-compositional vectors of adjective-noun phrases. the quality of the audio adjectives significantly improved after auditory and textual data were combined and textual data used as a signal in audio adjective learning. these results show that matrix composition leads to better representations for audio phrases, with potential applications to audio classification @xcite and captioning tasks @xcite . ## related work using vector addition for composing adjectives with nouns was proposed in @xcite . later, in a series of papers @xcite @xcite , it was argued that vector addition is not appropriate for composition as it is commutative. furthermore, an adjective needs to modify the meaning of a noun, thus its representation should be a map, rather than a vector. in finite dimensions, maps are approximated by matrices and adjective-noun phrase composition becomes matrix-vector multiplication, a noncommutative operation. different methodos were put forwards for learning the adjective matrices; @xcite used linear regression and @xcite developed a tensorial extension of the word2vec skipgram model @xcite . learning multimodal image-text embeddings for words was proposed in @xcite ; extended to sound-text in @xcite . matrix composition of images and text was explored in @xcite . ## single and multi modal learning an overview of multimodal phrase composition is presented in figure 1 . to learn the matrices, we used linear regression (lr) and the tensorial extension of skipgram (tsg). for lr, we trained adjective matrices a given observed adjective-noun vectors p and noun vectors v, using the formula @xmath0. the original word2vec skipgram model had the following objective function, where n is a vector, and c and c sets of positive and negative contexts. this model learns a vector for a word w regardless of its grammatical type. its tensorial extension, dubbed as tensor skipgram has an objective function that depends on the grammatical role of the words. for adjective-noun phrases, this is as fol-lows, where a is the adjective matrix, n the vector of the noun it modifies, and the rest is as before. the above function is only for adjective-noun phrases. it generalises to any phrase in @xcite . tsg significantly outperforms lr on text @xcite . the audio and textual representations were combined with two different methods. in the first method, we concatenated their vectors (at-concat) and used the result as an input to training. in the second method, we trained a joint audio-text matrix (at-joint), where one representation was used as a signal to improve the other. ## implementation we implemented an audio-text tsg, by extending the image-text tsg model of @xcite to audio data. the positive context is the audio files representing a target phrase. for instance, for loud melody we had 100 audio files and for loud cat 82. the negative context is determined by random selection of nouns during the training process with each adjective. we treat these nouns as a hyper parameter and choose them by tuning on the validation segment of the dataset. for skipgram models, we learn 50 dimensional phrase vectors with a learning rate of 10 -6 and a batch size of 512, trained for 200 epochs. the models were trained on nvidia t4 and v100 depending on their availability on google colab. the training was done in batches over a period of 3 months, totalling 80 hrs. we used binary cross-entropy loss and the adam optimiser in the training process to refine the performance. ## evaluation tasks and results our main hypothesis is that combining text and audio improves over audio-only learning. to test this, we trained audio-only variants of lr and tsg models. in these, the adjective matrices were learnt using only the audio vectors of their nouns and contexts. a second hypothesis is that non-commutative matrix multiplication models (lr and tsg) outperform simple commutative models. to test this, we implemented an additive model where an adjective's representation is added to its nouns. finally, we hypothesise that compositional models outperform non-compositional ones. for this, we compared the results to the holistic openl3 audio vector of adjective-noun phrases. ## discussion and conclusion we conducted a case study to understand the better performance of compositional multimodal audiotext embeddings using k-means clustering with optimal k values determined via silhouette method @xcite . cosine similarities were computed within each cluster, to find closet neighbours to the random queries from the evaluation split. some examples are provided in figure 2 . we found out that holistic singular text and audio only models predicted either semantic or audio relevance, often getting close to opposite concepts or literal sounds. on the other hand, multimodal composition managed to predict a more accurate phrase meaning. when non-compositional models struggle to predict, e.g. in the second example, the audio-only model predicted resonant piano and big ball as synonyms of big monster, multimodal composition predicted loud squeak and heavy thump and bridged the gap. another example is the prediction of distant firework, distant gun, and high frequency for angry scream by multimodal composition, where a text-only model guessed the opposite, i.e. happy scream. similar is the case for industrial resonance, predicted to be close to percussive banging and loud telephone by the compositional model, improving over the audio-only model which predicted big monster and the text-only model which again predicted opposite, i.e. industrial blast. these findings show that reflecting the textual grammatical structure in adjective-noun composition and considering both audio and text modalities improves the quality of audio data. extending the setting to verb phrases is work in progress.