Tutorial on Multimodal Machine Learning (2022)

Abstract: Multimodal machine learning involves integrating and modeling information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, HCI, and healthcare. This tutorial, building upon a new edition of a survey paper on multimodal ML as well as previously-given tutorials and academic courses, will describe an updated taxonomy on multimodal machine learning synthesizing its core technical challenges and major directions for future research.

## introduction multimodal machine learning is a vibrant multidisciplinary research field that addresses some original goals of ai by integrating and modeling multiple communicative modalities, including linguistic, acoustic, and visual messages. with the initial research on audio-visual speech recognition and more recently with language & vision projects such as image and video captioning, visual question answering, and language-guided reinforcement learning, this research field brings some unique challenges for multimodal researchers given the heterogeneity of the data and the contingency often found between modalities. this tutorial builds upon the annual course on multimodal machine learning taught at carnegie mellon university and is a revised version of the previous tutorials on multimodal learning at cvpr 2021, acl 2017, cvpr 2016, and icmi 2016. these previous tutorials were based on our earlier survey on multimodal machine learning, which introduced an initial taxonomy for core multimodal challenges @xcite . the present tutorial is based on a revamped taxonomy of the core technical challenges and updated concepts about recent work in multimodal machine learning @xcite . the tutorial will be cen-tered around six core challenges in multimodal machine learning: 1. representation: a first fundamental challenge is to learn representations that exploit cross-modal interactions between individual elements of different modalities. the heterogeneity of multimodal data makes it particularly challenging to learn multimodal representations. we will cover fundamental approaches for (1) representation fusion (integrating information from 2 or more modalities, effectively reducing the number of separate representations), (2) representation coordination (interchanging cross-modal information with the goal of keeping the same number of representations but improving multimodal contextualization), and (3) representation fission (creating a new disjoint set of representations, usually larger number than the input set, that reflects knowledge about internal structure such as data clustering or factorization). 2. alignment: a second challenge is to identify the connections between all elements of different modalities using their structure and cross-modal interactions. for example, when analyzing the speech and gestures of a human subject, how can we align specific gestures with spoken words or utterances? alignment between modalities is challenging since it may exist at different (1) granularities (words, utterances, frames, videos), involve varying (2) correspondences (one-to-one, many-to-many, or not exist at all), and depend on long-range (3) dependencies. 3. reasoning is defined as composing knowledge from multimodal evidences, usually through multiple inferential steps, to exploit multimodal alignment and problem structure for a specific task. this relationship often follows some hierarchical structure, where more abstract concepts are defined higher in the hierarchy as a function of less abstract concepts. multimodal reasoning involves the subchallenges of capturing this (1) structure (through domain knowledge or discovered from data), the parameterization of (2) concepts (dense vs interpretable and symbolic), and the type of (3) composition (simple vs complex relationships). 4. generation: the fourth challenge involves learning a generative process to produce raw modalities that reflect cross-modal interactions, structure and coherence. we categorize its subchallenges into (1) summarization (summarizing multimodal data to reduce information content while highlighting the most salient parts of the input), (2) translation (translating from one modality to another and keeping information content while being consistent with cross-modal interactions), and (3) creation (simultaneously generating multiple modalities to increase information content while maintaining coherence within and across modalities). we will also cover advances in evaluation and ethical concerns around generated content. 5. transference: a fifth challenge is to transfer knowledge between modalities and their representations, usually to help the target modality, which may be noisy or with limited resources. exemplified by algorithms of ( 1 ) transfer (fine-tuning pre-trained models for a downstream task involving the target modality), (2) representation enrichment (transfer through a joint model sharing representation spaces between both modalities), and (3) model induction (keeping individual unimodal models separate but transferring information across these models), how can knowledge learned from one modality (e.g., predicted labels or representation) help a computational model trained on a different modality? 6. quantification involves a deeper measurement and theoretical study of multimodal models to better understand their (1) output qualities (the extent to which models are predictive, efficient, and robust under natural and targeted modality imperfections), (2) internal mechanics (understanding the internal modeling of multimodal information and cross-modal interactions), and (3) modality tradeoffs (quantifying the utility and risks of each input modality, while balancing these tradeoffs for reliable real-world usage). it is important to obtain a deeper understanding of the data, modeling, and optimization challenges involved when learning from heterogeneous data in order to improve their robustness, interpretability, and reliability in real-world multimodal applications. type of tutorial: this tutorial will begin with basic concepts related to multimodal research before describing cutting-edge research in the context of the six core challenges. target audience and expected background: we expect the audience to have an introductory background in machine learning and deep learning, including a basic familiarity of commonly-used unimodal building blocks such as convolutional, recurrent, and self-attention models. ## tutorial outline this tutorial will be a revised edition of our previously-organized tutorials at cvpr 2022, cvpr 2021, acl 2017, cvpr 2016, and icmi 2016 which were roughly 3-4 hours long. this revision defines a new iteration of the taxonomy that has been updated to help researchers tackle modern multimodal challenges. the tutorial outline is shown below: