Abstract-A novel model is presented to learn bimodally informative structures from audio-visual signals. The signal is represented as a sparse sum of audio-visual kernels. Each kernel is a bimodal function consisting of synchronous snippets of an audio waveform and a spatio-temporal visual basis function. To represent an audio-visual signal, the kernels can be positioned independently and arbitrarily in space and time. The proposed algorithm uses unsupervised learning to form dictionaries of bimodal kernels from audio-visual material. The basis functions that emerge during learning capture salient audio-visual data structures. In addition, it is demonstrated that the learned dictionary can be used to locate sources of sound in the movie frame. Specifically, in sequences containing two speakers, the algorithm can robustly localize a speaker even in the presence of severe acoustic and visual distracters. Index Terms-Audio-visual source localization, dictionary learning, matching pursuit (MP), multimodal data processing, sparse representation. Alternatively, we have recently suggested a different approach to sensor fusion [26] . The idea is to analyze the audio-visual data jointly by extracting typical templates of audio-visual features; see Fig. 1 for an example. These templates represent synchronous transient structures that co-occur in both modalities. Simple template matching can then be used for solving sensor fusion tasks, such as speaker localization. The audio-visual template in Fig. 1 was extracted from a movie showing a speaker: the audio part is the waveform of a spoken digit in English, while the corresponding video part shows a moving edge that could represent the lower lip during the utterance of the digit. The direct extraction of audio-visual templates is interesting because it focuses on relevant bimodal structure rather than first computing the full representations in both modalities separately and then analyzing the joint statistics of features. However, the efficiency of the algorithm in [26] was limited because the template extraction and matching is brittle in the presence of accidental superpositions of separate transient structures. Here we present a novel model of audio-visual fusion that combines the advantages of joint bimodal signal analysis [26] and sparse coding, e.g., [27]-[32]. To combine the two approaches we build on previous work that used unsupervised learning of efficient sparse codes to understand response properties of neurons in various sensory systems. Efficient coding 1045-9227/$26.00
It is also possible to include different correlated modalities such as audio and visual signals in order to learn audio-visual dictionaries REF .
Learning Bimodal Structure in Audio–Visual Data
IEEE Transactions on Neural Networks
IEEE Transactions on Neural Networks
Computer Science Medicine