Multimodal fusion via cortical network inspired losses (2022)

Abstract: Information integration from different modalities is an active area of research. Human beings and, in general, biological neural systems are quite adept at using a multitude of signals from different sensory perceptive fields to interact with the environment and each other. Recent work in deep fusion models via neural networks has led to substantial improvements over unimodal approaches in areas like speech recognition, emotion recognition and analysis, captioning and image description. However, such research has mostly focused on architectural changes allowing for fusion of different modalities while keeping the model complexity manageable. Inspired by neuroscientific ideas about multisensory integration and processing, we investigate the effect of introducing neural dependencies in the loss functions. Experiments on multimodal sentiment analysis tasks with different models show that our approach provides a consistent performance boost.

## introduction human beings perceive the world as a unified whole, not in individual sensory modalities. while traditionally different sensory models have been studied in isolation, it has been well recognized that perception operates via integration of information from multiple sensory modalities. research in multimodal fusion aims to achieve a similar goal in artificial models: extract and integrate all information from different input modalities. for example, if someone is sarcastic, the facial expression and voice intonation provide information not directly decipherable from the uttered words. if a model only looks at the text of the interaction, then it is unlikely to classify this interaction currently. current research in deep multimodal fusion primarily deals with architectural improvements to create complex feature-rich, yet efficient representations @xcite @xcite . the hope is that more complex models will be able to integrate the complementary information from different unimodal representations into a unified common representation. learning such unified representations, however, is a challenging task. different modalities can present the same information in radically different ways with emphasis on different aspects of the content. these heterogeneities across different modalities mean that learning multimodal representations must deal with feature shifts, distributional effects, nuisance variation and a variety of related challenges @xcite . inspiring from work in multisensory neural processing, we define a loss regularizer that we call synergy to train these models. synergy has a specific meaning in information-theoretic literature @xcite . the synergy between random variables x and y refers to the unique mutual information that x provides about y . while our loss function is not the same as information theoretic synergy, the intuition behind our proposed loss is the same as actual synergy; to try to maximize dependencies between the representations. as our method uses neural networks or kernel-based methods to capture distributional divergences, we expect that this method will allow our model to capture complex dependencies which cannot be captured via techniques like subspace alignment. we test our proposed training loss on different multimodal fusion architectures including lfn @xcite , mfn @xcite , magbert @xcite and mim @xcite . our experiments show that training with synergy maximization improves the result by a significant margin. ## preliminaries in this section, we give an overview of the basic ideas relevant to this work; primarily mutual information, and existing work on deep multimodal fusion and neural synergy. ## dependency coding in multisensory processing a common and vital feature of nervous systems is the integration of information arriving simultaneously from multiple sensory pathways. the underlying neural structures have been found to be related in both vertebrates and invertebrates. the classic understanding of this process is that different sensory modalities are processed individually and then combined in various multimodal convergence zones, including cortical and subcortical regions @xcite , as well as multimodal association areas (rauschecker et al., @xcite . studies in the superior colliculus @xcite showed that multiple sensory modalities are processed in this brain stem region, with some neurons being exclusively unimodal and others being multimodal. hypotheses of encoding of multimodal information include changes in neuronal firing rates (pennartz, 2009) or a combinatorial code in population of neurons @xcite . evidence shows that while multimodal representations are distinct from unimodal ones, there is sufficient overlap between the set of neurons that process different sensory modalities. for example, @xcite show that even in a simple crustacean organism, more than half the neurons in the commissural ganglion are multimodal. moreover, they show that in 30% of these multimodal neurons, responses to one modality were predictive of responses to other modalities. both these facts suggest that the neural representations across different modalities have high information about each other. studies of multisensory collicular neurons suggest that their crossmodal receptive fields (rf) often overlap @xcite . this pattern is also found in multisensory neurons present in other brain regions. as such, a spatiotemporal hypothesis of multisensory integration has been suggested: superadditive multimodal processing is observed when information from different modalities comes from spatiotemporally overlapping receptive fields @xcite @xcite . since multimodal cortical neurons are generally downstream of modality-specific regions, the information about rf overlap is present in their input unimodal neural representations. moreover, the sensory-specific nuclei of the thalamus have been shown to feed multisensory information to primary sensory specific-cortices @xcite . this suggests the existence of explicit feedback connection from the multimodal representations to unimodal representations. cortical and subcortical networks often contain clusters of strongly connected neurons. functionally the existence of such cliques imply highly integrated pyramidal cells that handle a disproportionately large amount of traffic @xcite . in cortical circuits, around 20% of the neurons account for 80% of the information propagation @xcite . timme et al. (2016); faber et al. (2019) demonstrate that multimodal computation tends to concentrate in such local cortical clusters. they also found significantly lower kurtosis in such clusters and that dependence between oscillations was proportional to the amount of information flow. @xcite show that highly kurtotic neural activity positively related when multiple external stimuli are provided. thus, kurtosis in neural firings is a representation of the dependence between inputs. this suggests that when input kurtosis is high there is more significant cognitive processing and information flow required to extract relevant information. ## model for our purposes we will limit ourselves to talk about tasks similar to the mosi dataset. in this setting the input has three modalities viz audio (a), visual (v), and textual language (l). the fusion problem involves learning a representation m f that combined the uni-modal representations of the inputs x a,v,@xmath0, x v , x l ). ## concluding remarks in this paper, we used the idea of regularizing via a term which we label neural synergy maximization. this regularizer is inspired by neural cicruit design in the vertebral cortex. we experimented with different measures of synergy based on discrepancy measures such as kl and mmd. we also show that training with synergy can produce benefit on even sota architectures.