What is Multimodality? (2021)

Abstract: The last years have shown rapid developments in the field of multimodal machine learning, combining e.g., vision, text or speech. In this position paper we explain how the field uses outdated definitions of multimodality that prove unfit for the machine learning era. We propose a new task-relative definition of (multi)modality in the context of multimodal machine learning that focuses on representations and information that are relevant for a given machine learning task. With our new definition of multimodality we aim to provide a missing foundation for multimodal research, an important component of language grounding and a crucial milestone towards NLU.

## introduction the holy grail of nlp is natural language understanding (nlu). as previously argued, nlu cannot be achieved by learning from text alone @xcite . instead, an important step towards nlu is grounding language, especially in sight and sounds @xcite . there is thus great interest in the field of nlp to go beyond the textual modality and to conduct multimodal machine learning (ml) research. multimodal ml has made great progress during the last years. neural architectures are employed in tasks that go beyond single modalities. e.g., language is integrated with vision in visual question answering @xcite , visual commonsense reasoning @xcite , visual dialogue @xcite , or phrase grounding @xcite . audio signal processing has made advances in speech recognition @xcite and (visual) speech synthesis @xcite . but ml applications may reach beyond modalities that are familiar to us: astronomical and medical imaging techniques record wavelengths outside of what we call visible light. ge-figure 1 : are these examples instances of the same modality? = = = the same; = = = different. depending on perspective, input data can be judged differently. humanand machine-centered views would agree for (a) speech and @xmath0, (b) images and @xmath1. for (c), an image of text and text, the opinions could differ, while for (d), a visible light vs. infrared picture, humans could not even judge the infrared data, since it is not within their sensory capability. the point we aim to make in this position paper can be crystallized by asking the following questions regarding the input types shown on the left of figure 1 : are the data types shown on the left different from those that appear on the right, or are they instances of the same modality? this question leads to different answers, depending on whether we take a human-centered as opposed to a machine-centered standpoint. in this position paper we reason that either standpoint is insufficient, therefore we develop a definition of multimodality that allows researchers to judge cases as displayed in figure 1 , as involving or not different modalities from a task-specific perspective. our contributions are three-fold: i) we discuss existing definitions of multimodality and argue that they are insufficient to define multimodal tasks in the era of machine learning. ii) we propose a task-relative definition of multimodality and illustrate it for a variety of (multimodal) tasks and data. we argue that a taskrelative definition better suits the current age of diversified data, where the field attempts to create multimodally enabled intelligent machines. iii) by providing a novel definition of multimodality, we offer a foundation for building a roadmap towards multimodality in nlp and beyond. multimodality is a new challenge in nlp with research focusing on grounding language into other modalities. but how can we sensibly choose other modalities to ground language in, if we are not clear about what kinds of modalities language can represent itself, and about what constitutes a modality in the first place? for building a roadmap towards multimodality in nlp, we must establish common ground for what multimodal ml is, what are possible instantiations or specializations of multimodality, and what research questions it gives rise to. pinning down what makes a task multimodal in terms of information diversity, as we set out to do in this paper, is one important research question to ask. and from deeper understanding of what makes a task multimodal, we may -converselyderive novel tasks. ## multimodality and multimedia it is difficult to write about multimodality in ml and not to trigger the question "what about multimedia? isn't multimodal machine learning actually just multimedia in machine learning?" but the use of the terms "multimodality" and "multimedia" in academic literature is very diverse. attempts to compare and better capture these terms focus on different aspects and center on human society, human-to-human interaction and learning environments @xcite @xcite . we, however, focus on multimodal machine learning where the information receiver and processor is a ml system. to decide whether multimodality as used so far is the right term for the field of multimodal ml, we thus consult general definitions for "medium" and "modality" in the oxford advanced learner's dictionary, rather than the academic literature. for "medium" we find: to people" comprising text, (moving) images, graphs, sounds. (b) "a substance that something exists or grows in or that it travels through" being mainly applied in natural sciences. we argue that the term multimodality should be preferred in the context of ml, since multimodal ml aims to enhance world perception through ml systems (cf. def. (a)). finding new ways of presenting information to humans (cf. def. (a)) is an endeavour that can benefit from multimodal ml while not being the main focus of multimodal ml research. for the rest of the paper, we will use the term "medium" or "media" in different contexts, referring to the meaning defined in (b). in the following section 3, we discuss how multimodality has been interpreted by the multimodal ml field and highlight the shortcomings of the corresponding definitions in section 4. ## how multimodality is defined -so far in the multimodal ml literature and beyond, we find three general ways of defining "modality" or "multimodality": not at all or etymologically (bypassing the problem), or by way of a humancentered or a machine-centered definition foot_0 . not at all or etymological especially recent publications, as in @xcite @xcite , bypass a definition, assuming that the term is generally understood. others offer an etymological definition: multimodal research involves not one, but multiple modalities @xcite . clearly, this definition leaves the notion of modality itself unexplicated. human-centered popular definitions of multimodality rely on the human perceptual experience, as found in @xcite al. (2019); lyons (2016); ngiam et al. (2011); kress (2010 @xcite . from this literature, we chose the following illustrative example, because the work focuses specifically on multimodality for ml, as is the interest of this paper: foot_1 "our experience of the world is multimodalwe see objects, hear sounds, feel texture, smell odors, and taste flavors. modality refers to the way in which something happens or is experienced". ## why we need a better definition in this section, we indicate shortcomings of the existing definitions and motivate why time is ripe for a new definition of multimodality. ## a task-relative definition we argue that a definition of multimodality for multimodal ml should relate to the task an ml system needs to solve, since it is the task that determines what information is relevant and in which representation this information can be efficiently stored. the human-and machine-centered definitions try to capture the essence of multimodality in a taskagnostic manner, relating it to categories of human experience, media, representations, and encodings. as shown in section 4, these definitions turn out to be insufficient in view of the plurality of physical sensors, tasks and data types of the present. instead, our task-relative definition aims to answer the question: under which conditions does a multimodal setting reveal crucial insight for the given task -and do we need multiple modalities? in our view, (i) different inputs can contribute specific information, but (ii) what is relevant information can only be determined in relation to the task at hand; and only by taking the task into account (iii) we can determine the status of the inputs as (possibly complementary) modalities. ## conclusion in this paper we have shown how human-centered definitions of multimodality are too restrictive for current and future developments of ml. also, machine-centered definitions focusing on representations only do not capture the crucial trait of multimodal machine learning. we instead propose a new definition of multimodality that focuses on the relation between representations, information and the given task, and that -through the novel dimension of the task -enables us to make much sharper distinctions compared to current standards, while covering a much wider spectrum of multimodal data. with this position paper, we (a) offer a working definition on how to use the term multimodality, (b) aim to raise awareness that defining multimodality is harder than expected, and (c) invite the community to discuss these challenges and (why not?) to provide a better definition.