Abstract-We propose a novel framework for multimodal video indexing and retrieval using shrinkage optimized directed information assessment (SODA) as similarity measure. The directed information (DI) is a variant of the classical mutual information which attempts to capture the direction of information flow that videos naturally possess. It is applied directly to the empirical probability distributions of both audio-visual features over successive frames. We utilize RASTA-PLP features for audio feature representation and SIFT features for visual feature representation. We compute the joint probability density functions of audio and visual features in order to fuse features from different modalities. With SODA, we further estimate the DI in a manner that is suitable for high dimensional features and small sample size (large small ) between pairs of video-audio modalities. We demonstrate the superiority of the SODA approach in video indexing, retrieval, and activity recognition as compared to the state-of-the-art methods such as hidden Markov models (HMM), support vector machine (SVM), cross-media indexing space (CMIS), and other noncausal divergence measures such as mutual information (MI). We also demonstrate the success of SODA in audio and video localization and indexing/retrieval of data with missaligned modalities. Index Terms-Audio-video pattern recognition, multimedia content retrieval, multimodal feature fusion, nonlinear information flow, overfitting prevention, shrinkage optimization.
Chen et al. REF present a mutual information variant to fuse features from audio and video domain to perform action recognition and retrieval.
Multimodal Video Indexing and Retrieval Using Directed Information
IEEE Transactions on Multimedia
IEEE Transactions on Multimedia
Computer Science