This paper describes a system for connecting sounds and words in linked multi-dimensional vector spaces. The acoustic space is represented using anchor models and partitioned using agglomer ative clustering. The semantic space is modeled by a hierarchical multinomial clustering model. Nodes in one space are linked by probabilistic models to the other space. With these linked models, users retrieve sounds with natural language, and the system describes new sounds with words. This paper describes a method of connecting sounds to words, and words to sounds. Given a description of a sound, the system finds the audio signals that best fit the words. Thus, a user might make a request with the description "the sound of a galloping horse," and the system would respond by presenting recordi � gs of a horse running on different surfaces, and possibly of musIcal pieces that sound like a horse galloping. Conversely, given a sound recording, the system describes the sound or the environ ment in which the recording was made. Thus, given a recording made outdoors, the system says c�nfidently that the recording was made at a horse farm where several dogs reside. We build a system that has these functions, called SAR (semantic-audio retrieval), by learning the connection between a semantic space and an auditory space. Semantic space maps words into a high-dimensional probabilistic space. Acoustic space describes sounds by a multidimension�1 vector. In general, the 'connection between these two spaces will be many to many. Horse sounds, for example, might include footsteps and neighs. Figure 1 shows one half of SAR; how to retrieve sounds from words. Annotations that describe sounds are clustered within a hierarchical semantic model that uses multinomial models. The sound files, or acoustic documents, that correspond to each node in the semantic hierarchy are modeled with Gaussian mixture models (GMMs). Given a semantic request, SAR identifies the portion of the semantic space that best fits the request, and then measures the likelihood that each sound in the database fits the Figure 1 : SAR models all of semantic space with a hierarchical collection of multinomial models, each portion in the semantic model is linked to equivalent sound documents in acoustic space with a GMM. GMM linked to this portion of the semantic space. The most likely sounds are returned to satisfy the user's semantic request. Figure 2 shows the second half of SAR; how to generate words to describe a sound. SAR analyzes the collection of sounds and builds models for arbitrary sounds called anchors. All sounds in the database are described by how well they are modeled by these anchor sounds. This approach gives us a multidimensional representation of any sound, and a distance metric that permits agglomerative clustering in the acoustic space. Given an acoustic request, SAR identifies the portion of the acoustic space that best fits the request. Each portion of the acoustic space has an associ ated multinomial word model, and from this model SAR gener ates words to describe the query sound. The SAR algorithm is illustrated with a closed set of acoustic and semantic documents about animals. Six CDs (#12, 30, 34, 35, 37,38) from the BBC Sound Effects Library contained 215 sepa rate tracks, with 330 minutes of audio recordings of animal sounds� In addition, the concatenated name of the CD (e.g., "Horses I") and track description (e.g., "One horse eating hay and moving around") form a unique semantic label for each track. The audio from the CD track and the liner notes form a pair of acoustic and semantic documents used to train the SAR system. There are many multimedia retrieval systems that use a combina tion of words and/or examples to retrieve audio (and video) for users. An effective way to find an image of the space shuttle is to enter the words "space shuttle jpg" into a text-based web search engine. The original Google system did not know about images, but, fortunately, many people created web pages with the phrase "space shuttle" that contained a JPEG image of the shuttle. More recently, both Google and AltaVista for images, and Compuson ics for audio, have built systems that automate these searches. They allow people to look for images and sound based on nearby words. The SAR work expands those search techniques by con sidering the acoustic and semantic similarity of sounds to allow an information-retrieval task and argued that the words written by a human annotator describing an image (e.g., "a rose") often pro vide infonnation that complements the obvious infonnation in the image (it is red). ,. SAR improves on two aspects of Barnard s approach. F � r:st, the semantic and image features do not have the same probabIltty distributions. Hofinann 's algorithm assumes that image features can be described by a multinomial distribution, while a Gaussian is probably more appropriate. Second, and perhaps more impor tant, there is nothing in Hoffinan's algorithm that guarantees that the features used to build each stage of the model include both semantic and image features. Thus, the algorithm is free to build a model that completely ignores the image features and clusters the "documents" based on only semantic features. The audio-retrieval problem is easier than image retrieval for two reasons. First, we do not have to solve the foreground-back ground problem (yet). People who want a picture of a tiger usu ally do not care whether the tiger is surrounded by grass, sand, water or even a sunset Fortunately, in the audio world, we can often 'assume that only one sound is present at a time. Second, it is difficult to know which are the features that best characterize an image. The key to SAR is choosing a set of features that al�ows us to t :' lk about mathematical spaces for sounds and semantics. In Section 3.1, we consider what semantic features are, how we describe the position of a concept in semantic space, and how we measure the distance between two concepts. Section 3.2 describes the same proCedure for sounds. In both cases, we have a rich foundation of prior work on which to build. SAR uses multinomial models to represent and cluster a collec tion of semantic documents. The likelihood that a docum 9, nt matches a given multinomial model is described by L = np i' , where p. is the probability that the word i occurs in this type of docume�t, and n. is the number of times that word i is found in this document. The set of probabilities, P" is different for differ ent types of documents. Thus, a model for documents about cows will have a relatively high probability for containing "cow" and ''moo,'' whereas a model for documents that describe birds with have a high probability of containing "feather." SAR uses the same cluster abstraction technique used in Bar nard's work, but only applies it to the semantic data. Cluster .s-A cluster ofdocumenl8 Figure 3 : The cluster abstraction model calculates a hierarchical set of multinomial models (a x )to generate clusters of documents. abstraction builds a hierarchical collection of multinomial models to describe a set of documents. This hierarchical model (Figure 3)
Slaney REF introduced a clustering method in a multi-dimensional vector space for semantic audio retrieval.
Semantic-audio retrieval
2002 IEEE International Conference on Acoustics, Speech, and Signal Processing
2002 IEEE International Conference on Acoustics, Speech, and Signal Processing
Computer Science