Multisensory Processes: A Balancing Act across the Lifespan
Authors: Micah M. Murray, David J. Lewkowicz, Amir Amedi, Mark T. Wallace
Journal: Trends in Neurosciences (2016)

Multisensory processes are the product of a dynamic reweighting of physical stimulus characteristics and learned associations. This reweighting occurs across multiple timescales, ranging from long term (i.e., developmental and lifespan) to short-term (i.e., during the learning and encoding of multisensory relations). We propose a novel theoretical framework that combines traditional principles associated with stimulus characteristics (i.e., space, time, effectiveness) with a new principle of dynamic reweighting of stimulus characteristics and learned associations across different timescales. The novel theoretical framework emphasizes the plastic and dynamic nature of multisensory processing across the lifespan and thus accounts for improvements in multisensory perception and behavior under 'normal' circumstances and offers plausible remediation strategies for treating patients in whom sensory or multisensory function is compromised. Multisensory processes are fundamental in scaffolding perception, cognition, learning, and behavior. How and when stimuli from different sensory modalities are integrated rather than treated as separate entities is poorly understood. We review how the relative reliance on stimulus characteristics versus learned associations dynamically shapes multisensory processes. We illustrate the dynamism in multisensory function across two timescales: one long term that operates across the lifespan and one short term that operates during the learning of new multisensory relations. In addition, we highlight the importance of task contingencies. We conclude that these highly dynamic multisensory processes, based on the relative weighting of stimulus characteristics and learned associations, provide both stability and flexibility to brain functions over a wide range of temporal scales. Multisensory processes are fundamental in scaffolding perception, cognition, learning, and behavior. How and when stimuli from different sensory modalities are integrated rather than treated as separate entities is poorly understood. We review how the relative reliance on stimulus characteristics versus learned associations dynamically shapes multisensory processes. We illustrate the dynamism in multisensory function across two timescales: one long term that operates across the lifespan and one short term that operates during the learning of new multisensory relations. In addition, we highlight the importance of task contingencies. We conclude that these highly dynamic multisensory processes, based on the relative weighting of stimulus characteristics and learned associations, provide both stability and flexibility to brain functions over a wide range of temporal scales. Our world comprises a vast array of information that is coded in different sensory modalities. This presents a fundamental challenge for cognitive, perceptual, and motor systems because the multitude of multisensory inputs needs to be seamlessly integrated and appropriately segregated to form coherent perceptual representations and drive adaptive behaviours. Fortunately, the nervous system possesses specialized architectures and processing mechanisms that enable the combination and integration of multisensory information. These mechanisms and brain networks solve fundamental issues associated with sensory 'binding' (see Glossary) and consequently provide marked behavioral and perceptual benefits. Take as an example our ability to comprehend a single speaker in a noisy room. Seeing the speaker's mouth can increase the intelligibility of the auditory signal by 15 dB [1Sumby W.H. Pollack I. Visual contribution to speech intelligibility in noise.J. Acoust. Soc. Am. 1954; 26: 212-215Crossref Scopus (1879) Google Scholar]. This ability to extract speech information from visual cues begins in infancy when infants start babbling and need to learn native speech forms, when they have to disambiguate unfamiliar speech, and when they are learning two languages [2Lewkowicz D.J. Hansen-Tift A.M. Infants deploy selective attention to the mouth of a talking face when learning speech.Proc. Natl Acad. Sci. U.S.A. 2012; 109: 1431-1436Crossref PubMed Scopus (314) Google Scholar, 3Pons F. et al.Bilingualism modulates infants' selective attention to the mouth of a talking face.Psychol. Sci. 2015; 26: 490-498Crossref PubMed Scopus (123) Google Scholar]. Extending this further, deaf individuals can learn to watch mouth movements to understand speech by lip reading, which results in extensive cross-modal plasticity within regions of the cerebral cortex [4Bavelier D. Neville H.J. Cross-modal plasticity: where and how?.Nat. Rev. Neurosci. 2002; 3: 443-452Crossref PubMed Scopus (717) Google Scholar]. Other examples of how multisensory interactions facilitate behavior and perception abound and include faster and more accurate detection, localization, and discrimination (see [5Stein B.E. The New Handbook of Multisensory Processes. MIT Press, 2012Google Scholar, 6Stein B.E. Meredith M.A. The Merging of the Senses. MIT Press, 1993Google Scholar, 7Murray M.M. Wallace M.T. The Neural Bases of Multisensory Processes. CRC Press, 2012Google Scholar] for reviews). The neural architecture and functional rules for multisensory convergence and integration have been described in numerous species and structures. Historically, multisensory functions were considered the bastion of higher-level 'association' cortical regions as well as premotor cortices and sensorimotor subcortical regions (Figure 1). Recent research has changed this view by revealing that sensory systems have the capacity to influence one another even at very early processing stages (reviewed in [8Murray M.M. et al.The multisensory function of the human primary visual cortex.Neuropsychologia. 2016; 83: 161-169Crossref PubMed Scopus (105) Google Scholar, 9Ghazanfar A.A. Schroeder C.E. Is neocortex essentially multisensory?.Trends Cogn. Sci. 2006; 10: 278-285Abstract Full Text Full Text PDF PubMed Scopus (991) Google Scholar, 10Kayser C. Shams L. Multisensory causal inference in the brain.PLoS Biol. 2015; 13: e1002075Crossref PubMed Scopus (72) Google Scholar]). The functional implications of such 'early' multisensory influences are likely to be substantial in that they suggest a previously unrealized level of interaction that could confer tremendous flexibility on sensory function [11van Atteveldt N. et al.Multisensory integration: flexible use of general operations.Neuron. 2014; 81: 1240-1253Abstract Full Text Full Text PDF PubMed Scopus (168) Google Scholar]. To date, much research effort has gone into providing mechanistic and circuit-based descriptions of multisensory functions. Substantially less effort has been directed toward unpacking the relative contributions of lower-level versus higher-level factors in determining the final product of a given multisensory interaction. For purposes of clarity, we consider these lower-level factors as the physical characteristics of the stimuli themselves (e.g., location, timing, intensity) and the higher-level factors the learned associations that are built between the different sensory modalities. Using these operational definitions of stimulus characteristics and learned associations, we seek to highlight the dynamic interplay between them and that are occurring across a host of timescales ranging from the immediate to lifespan. It is well established that physical stimulus characteristics play a central role in shaping multisensory interactions at the neural, behavioral, and perceptual levels. The environmental statistics of these characteristics include the spatial and temporal relationships of the paired stimuli to one another as well as their relative effectiveness in eliciting a response (Box 1). Despite their importance, these stimulus characteristics alone are unlikely to be sufficient to specify fully the outcome of a given multisensory interaction. We now know that these stimulus characteristics do not function in isolation, but rather that they interact with one another [12Cappe C. et al.Looming signals reveal synergistic principles of multisensory integration.J. Neurosci. 2012; 32: 1171-1182Crossref PubMed Scopus (84) Google Scholar, 13Krueger J. et al.Spatial receptive field organization of multisensory neurons and its impact on multisensory interactions.Hear. Res. 2009; 258: 47-54Crossref PubMed Scopus (22) Google Scholar, 14Royal D.W. et al.Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions.Exp. Brain Res. 2009; 198: 127-136Crossref PubMed Scopus (40) Google Scholar, 15Carriere B.N. et al.Spatial heterogeneity of cortical receptive fields and its impact on multisensory interactions.J. Neurophysiol. 2008; 99: 2357-2368Crossref PubMed Scopus (29) Google Scholar]. For example, changing the location of stimuli changes their relative effectiveness in eliciting a response. In addition, we do not know how these stimulus-related factors and their interactions are affected by higher-level factors. For example, task-dependent, goal-dependent, and context-dependent factors have been shown to dramatically alter the nature of certain types of multisensory interactions, even when low-level stimulus features are held constant [11van Atteveldt N. et al.Multisensory integration: flexible use of general operations.Neuron. 2014; 81: 1240-1253Abstract Full Text Full Text PDF PubMed Scopus (168) Google Scholar, 16Talsma D. et al.The multifaceted interplay between attention and multisensory integration.Trends Cogn. Sci. 2010; 14: 400-410Abstract Full Text Full Text PDF PubMed Scopus (512) Google Scholar, 17De Meo R. et al.Top-down control and early multisensory processes: chicken vs. egg.Front. Integr. Neurosci. 2015; 9: 1-6Crossref PubMed Scopus (48) Google Scholar, 18ten Oever S. et al.The COGs (context, object, and goals) in multisensory processing.Exp. Brain Res. 2016; 234: 1307-1323Crossref PubMed Scopus (36) Google Scholar, 19Sarmiento B.R. et al.Contextual factors multiplex to control multisensory processes.Hum. Brain Mapp. 2016; 37: 273-288Crossref PubMed Scopus (16) Google Scholar]. Collectively, this evidence argues that a complex set of interactions between the physical characteristics (including the environmental statistics) of the stimuli and the various learned associations that are acquired through short-term and long-term experience ultimately determines the end product of a given multisensory interaction. Unfortunately, despite the fact that there is now a wealth of evidence for the importance of the separate influences of lower- and higher-level factors on multisensory processing, surprisingly little work has focused on their interplay.Box 1A Primer on Principles of Multisensory IntegrationThe convergence of inputs from multiple sensory modalities has been long known for classic regions of association cortex as well as for subcortical structures (reviewed in [6Stein B.E. Meredith M.A. The Merging of the Senses. MIT Press, 1993Google Scholar]). This traditional view has recently been complemented by evidence of a surprising degree of connectivity between areas traditionally deemed 'unisensory' (Figure 1) [83Falchier A. et al.Anatomical evidence of multimodal integration in primate striate cortex.J. Neurosci. 2002; 22: 5749-5759Crossref PubMed Google Scholar, 84Falchier A. et al.Projection from visual areas V2 and prostriata to caudal auditory cortex in the monkey.Cereb. Cortex. 2010; 20: 1529-1538Crossref PubMed Scopus (89) Google Scholar, 85Rockland K.S. Ojima H. Multisensory convergence in calcarine visual areas in macaque monkey.Int. J. Psychophysiol. 2003; 50: 19-26Crossref PubMed Scopus (438) Google Scholar, 86Beer A.L. et al.Diffusion tensor imaging shows white matter tracts between human auditory and visual cortex.Exp. Brain Res. 2011; 213: 299-308Crossref PubMed Scopus (89) Google Scholar, 87Beer A.L. et al.Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal–occipital network involved in auditory–visual object processing.Front. Integr. Neurosci. 2013; 7: 5Crossref PubMed Scopus (38) Google Scholar] (reviewed in [8Murray M.M. et al.The multisensory function of the human primary visual cortex.Neuropsychologia. 2016; 83: 161-169Crossref PubMed Scopus (105) Google Scholar, 9Ghazanfar A.A. Schroeder C.E. Is neocortex essentially multisensory?.Trends Cogn. Sci. 2006; 10: 278-285Abstract Full Text Full Text PDF PubMed Scopus (991) Google Scholar]). The functional organization and behavioral and perceptual relevance of these cross-modal connectivity patterns are a focus of ongoing research. Functional studies are describing the neurobiological consequences of multisensory convergence spanning levels of analysis ranging from single units in animal models to human imaging studies of large neural network populations [5Stein B.E. The New Handbook of Multisensory Processes. MIT Press, 2012Google Scholar, 7Murray M.M. Wallace M.T. The Neural Bases of Multisensory Processes. CRC Press, 2012Google Scholar]. A recurring finding is that multisensory stimulation often results in response or activation changes different from those predicted based on the addition of the component unisensory responses.Several fundamental processes appear to characterize the integrative features of multisensory neurons. One process revolves around the physical properties of the stimuli to be integrated, including their spatial and temporal proximity. Generally, stimuli presented close together in space or time enhance both neural and behavioral responses (the spatial and temporal principles, respectively). Separating stimuli in these dimensions generally reduces the magnitude of these interactions. Both the response gains and response reductions are often strikingly nonlinear (e.g., [12Cappe C. et al.Looming signals reveal synergistic principles of multisensory integration.J. Neurosci. 2012; 32: 1171-1182Crossref PubMed Scopus (84) Google Scholar, 14Royal D.W. et al.Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions.Exp. Brain Res. 2009; 198: 127-136Crossref PubMed Scopus (40) Google Scholar, 15Carriere B.N. et al.Spatial heterogeneity of cortical receptive fields and its impact on multisensory interactions.J. Neurophysiol. 2008; 99: 2357-2368Crossref PubMed Scopus (29) Google Scholar, 88Cappe C. et al.Selective integration of auditory–visual looming cues by humans.Neuropsychologia. 2009; 47: 1045-1052Crossref PubMed Scopus (92) Google Scholar]). Likewise, pairing weakly effective stimuli generally results in the largest proportionate response gains (the inverse effectiveness principle). These sorts of stimulus-dependent principles make ecological sense; spatially and temporally proximate cues generally originate from a common source and highly effective stimuli need little amplification [6Stein B.E. Meredith M.A. The Merging of the Senses. MIT Press, 1993Google Scholar].The basic patterns of multisensory development look very similar across species and in many different brain structures. Moreover, the global developmental progression is one in which inputs from the individual sensory systems first appear in structures that will eventually become multisensory, and that, as inputs from the different senses converge, neurons that are activated or influenced by inputs from multiple senses (i.e., multisensory neurons) appear and then increase gradually in incidence over postnatal life [89Wallace M.T. et al.Visual experience is necessary for the development of multisensory integration.J. Neurosci. 2004; 24: 9580-9584Crossref PubMed Scopus (150) Google Scholar, 90Wallace M.T. et al.The development of cortical multisensory integration.J. Neurosci. 2006; 26: 11844-11849Crossref PubMed Scopus (98) Google Scholar, 91Wallace M.T. Stein B.E. Development of multisensory neurons and multisensory integration in cat superior colliculus.J. Neurosci. 1997; 17: 2429-2444Crossref PubMed Google Scholar]. Although the same types of invasive recording procedures cannot be (readily) used in humans, the overall developmental pattern obtained from behavioral measures appears remarkably similar and consistent with the pattern derived from animal studies.It has now become evident that interactions based on the physical attributes of multisensory inputs do not fully account for the products of multisensory interactions [92Stevenson R.A. et al.Identifying and quantifying multisensory integration: a tutorial review.Brain Topogr. 2014; 27: 707-730Crossref PubMed Scopus (126) Google Scholar, 93Panzeri S. et al.Neural population coding: combining insights from microscopic and mass signals.Trends Cogn. Sci. 2015; 19: 162-172Abstract Full Text Full Text PDF PubMed Scopus (119) Google Scholar]. Low-level (i.e., stimulus-related) factors only partly account for the multifaceted nature of multisensory processes. Instead, it is now clear that stimulus characteristics work dynamically and in concert with a series of higher-level processes including semantic congruence, attentional allocation, and task demands [11van Atteveldt N. et al.Multisensory integration: flexible use of general operations.Neuron. 2014; 81: 1240-1253Abstract Full Text Full Text PDF PubMed Scopus (168) Google Scholar, 16Talsma D. et al.The multifaceted interplay between attention and multisensory integration.Trends Cogn. Sci. 2010; 14: 400-410Abstract Full Text Full Text PDF PubMed Scopus (512) Google Scholar, 17De Meo R. et al.Top-down control and early multisensory processes: chicken vs. egg.Front. Integr. Neurosci. 2015; 9: 1-6Crossref PubMed Scopus (48) Google Scholar, 18ten Oever S. et al.The COGs (context, object, and goals) in multisensory processing.Exp. Brain Res. 2016; 234: 1307-1323Crossref PubMed Scopus (36) Google Scholar]. The convergence of inputs from multiple sensory modalities has been long known for classic regions of association cortex as well as for subcortical structures (reviewed in [6Stein B.E. Meredith M.A. The Merging of the Senses. MIT Press, 1993Google Scholar]). This traditional view has recently been complemented by evidence of a surprising degree of connectivity between areas traditionally deemed 'unisensory' (Figure 1) [83Falchier A. et al.Anatomical evidence of multimodal integration in primate striate cortex.J. Neurosci. 2002; 22: 5749-5759Crossref PubMed Google Scholar, 84Falchier A. et al.Projection from visual areas V2 and prostriata to caudal auditory cortex in the monkey.Cereb. Cortex. 2010; 20: 1529-1538Crossref PubMed Scopus (89) Google Scholar, 85Rockland K.S. Ojima H. Multisensory convergence in calcarine visual areas in macaque monkey.Int. J. Psychophysiol. 2003; 50: 19-26Crossref PubMed Scopus (438) Google Scholar, 86Beer A.L. et al.Diffusion tensor imaging shows white matter tracts between human auditory and visual cortex.Exp. Brain Res. 2011; 213: 299-308Crossref PubMed Scopus (89) Google Scholar, 87Beer A.L. et al.Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal–occipital network involved in auditory–visual object processing.Front. Integr. Neurosci. 2013; 7: 5Crossref PubMed Scopus (38) Google Scholar] (reviewed in [8Murray M.M. et al.The multisensory function of the human primary visual cortex.Neuropsychologia. 2016; 83: 161-169Crossref PubMed Scopus (105) Google Scholar, 9Ghazanfar A.A. Schroeder C.E. Is neocortex essentially multisensory?.Trends Cogn. Sci. 2006; 10: 278-285Abstract Full Text Full Text PDF PubMed Scopus (991) Google Scholar]). The functional organization and behavioral and perceptual relevance of these cross-modal connectivity patterns are a focus of ongoing research. Functional studies are describing the neurobiological consequences of multisensory convergence spanning levels of analysis ranging from single units in animal models to human imaging studies of large neural network populations [5Stein B.E. The New Handbook of Multisensory Processes. MIT Press, 2012Google Scholar, 7Murray M.M. Wallace M.T. The Neural Bases of Multisensory Processes. CRC Press, 2012Google Scholar]. A recurring finding is that multisensory stimulation often results in response or activation changes different from those predicted based on the addition of the component unisensory responses. Several fundamental processes appear to characterize the integrative features of multisensory neurons. One process revolves around the physical properties of the stimuli to be integrated, including their spatial and temporal proximity. Generally, stimuli presented close together in space or time enhance both neural and behavioral responses (the spatial and temporal principles, respectively). Separating stimuli in these dimensions generally reduces the magnitude of these interactions. Both the response gains and response reductions are often strikingly nonlinear (e.g., [12Cappe C. et al.Looming signals reveal synergistic principles of multisensory integration.J. Neurosci. 2012; 32: 1171-1182Crossref PubMed Scopus (84) Google Scholar, 14Royal D.W. et al.Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactions.Exp. Brain Res. 2009; 198: 127-136Crossref PubMed Scopus (40) Google Scholar, 15Carriere B.N. et al.Spatial heterogeneity of cortical receptive fields and its impact on multisensory interactions.J. Neurophysiol. 2008; 99: 2357-2368Crossref PubMed Scopus (29) Google Scholar, 88Cappe C. et al.Selective integration of auditory–visual looming cues by humans.Neuropsychologia. 2009; 47: 1045-1052Crossref PubMed Scopus (92) Google Scholar]). Likewise, pairing weakly effective stimuli generally results in the largest proportionate response gains (the inverse effectiveness principle). These sorts of stimulus-dependent principles make ecological sense; spatially and temporally proximate cues generally originate from a common source and highly effective stimuli need little amplification [6Stein B.E. Meredith M.A. The Merging of the Senses. MIT Press, 1993Google Scholar]. The basic patterns of multisensory development look very similar across species and in many different brain structures. Moreover, the global developmental progression is one in which inputs from the individual sensory systems first appear in structures that will eventually become multisensory, and that, as inputs from the different senses converge, neurons that are activated or influenced by inputs from multiple senses (i.e., multisensory neurons) appear and then increase gradually in incidence over postnatal life [89Wallace M.T. et al.Visual experience is necessary for the development of multisensory integration.J. Neurosci. 2004; 24: 9580-9584Crossref PubMed Scopus (150) Google Scholar, 90Wallace M.T. et al.The development of cortical multisensory integration.J. Neurosci. 2006; 26: 11844-11849Crossref PubMed Scopus (98) Google Scholar, 91Wallace M.T. Stein B.E. Development of multisensory neurons and multisensory integration in cat superior colliculus.J. Neurosci. 1997; 17: 2429-2444Crossref PubMed Google Scholar]. Although the same types of invasive recording procedures cannot be (readily) used in humans, the overall developmental pattern obtained from behavioral measures appears remarkably similar and consistent with the pattern derived from animal studies. It has now become evident that interactions based on the physical attributes of multisensory inputs do not fully account for the products of multisensory interactions [92Stevenson R.A. et al.Identifying and quantifying multisensory integration: a tutorial review.Brain Topogr. 2014; 27: 707-730Crossref PubMed Scopus (126) Google Scholar, 93Panzeri S. et al.Neural population coding: combining insights from microscopic and mass signals.Trends Cogn. Sci. 2015; 19: 162-172Abstract Full Text Full Text PDF PubMed Scopus (119) Google Scholar]. Low-level (i.e., stimulus-related) factors only partly account for the multifaceted nature of multisensory processes. Instead, it is now clear that stimulus characteristics work dynamically and in concert with a series of higher-level processes including semantic congruence, attentional allocation, and task demands [11van Atteveldt N. et al.Multisensory integration: flexible use of general operations.Neuron. 2014; 81: 1240-1253Abstract Full Text Full Text PDF PubMed Scopus (168) Google Scholar, 16Talsma D. et al.The multifaceted interplay between attention and multisensory integration.Trends Cogn. Sci. 2010; 14: 400-410Abstract Full Text Full Text PDF PubMed Scopus (512) Google Scholar, 17De Meo R. et al.Top-down control and early multisensory processes: chicken vs. egg.Front. Integr. Neurosci. 2015; 9: 1-6Crossref PubMed Scopus (48) Google Scholar, 18ten Oever S. et al.The COGs (context, object, and goals) in multisensory processing.Exp. Brain Res. 2016; 234: 1307-1323Crossref PubMed Scopus (36) Google Scholar]. In this review we describe some of the relative contributions of physical stimulus characteristics and learned associations to multisensory processing (Figure 2). We focus on two very different temporal scales as a means to highlight the dynamic interplay that is continually occurring between these factors. The first temporal scale considers multisensory processing across the lifespan. Evidence indicates that there is a developmental shift from an initial emphasis on stimulus characteristics to a greater emphasis on learned associations (see [20Lewkowicz D.J. Ghazanfar A.A. The emergence of multisensory systems through perceptual narrowing.Trends Cogn. Sci. 2009; 13: 470-478Abstract Full Text Full Text PDF PubMed Scopus (201) Google Scholar] for a review of such evidence from studies of human infants). The second temporal scale is much shorter and refers to the processing of information according to immediate behavioral and perceptual contingencies that are constrained by prior experiences. We argue that multisensory processes occurring within this immediate timescale are highly flexible and dynamic, strongly illustrating the power of learned associations and task contingencies in shaping multisensory processing. At its extreme, such as in the case of using a sensory substitution device (SSD), individuals can be trained to partially or completely adapt to using an intact sensory channel to replace functions of the impaired sense. Here, we suggest that multisensory processes can be dictated almost completely by task demands and contingencies under certain circumstances (reviewed in [21Heimler B. et al.Origins of task-specific sensory-independent organization in the visual and auditory brain: neuroscience evidence, open questions and clinical implications.Curr. Opin. Neurobiol. 2015; 35: 169-177Crossref PubMed Scopus (59) Google Scholar]). Although the development of multisensory function begins before birth [22Lewkowicz D.J. Early experience and multisensory perceptual narrowing.Dev. Psychobiol. 2014; 56: 292-315Crossref PubMed Scopus (64) Google Scholar], studies of its development in animals and humans have focused on the postnatal period [23Lewkowicz D.J. King A.J. The developmental and evolutionary emergence of multisensory processing: from single cells to behavior.in: Stein B.E. In The New Handbook of Multisensory Processing. MIT Press, 2012: 559-572Google Scholar, 24Bremner A.J. et al.Multisensory Development. Oxford University Press, 2012Crossref Scopus (99) Google Scholar, 25Röder B. Wallace M. Development and plasticity of multisensory functions.Restor. Neurol. Neurosci. 2010; 28: 141-142PubMed Google Scholar, 26Polley D.B. et al.Development and plasticity of intra- and intersensory information processing.J. Am. Acad. Audiol. 2008; 19: 780-798Crossref PubMed Scopus (18) Google Scholar]. Collectively, this work has illustrated that multisensory processing and the brain circuits that support it mature over the course of postnatal life and that they depend critically on early experience. Moreover, there appears to be a developmental reweighting as maturation progresses, such that low-level physical characteristics are initially weighted more heavily and later increasingly more complex and learned attributes are prioritized. A common principle that emerges from such work is that multisensory systems pass through three developmental stages: immature, broadly tuned, and narrowly tuned (Figure 3). Studies in human infants have typically manipulated the physical stimulus characteristics of the sensory world and examined how such stimulus changes affect behaviours such as orienting and looking. Preverbal babies will attend preferentially to (multisensory) events that are spatially and/or temporally congruent and thus associated with a common source (e.g., the sight and sound of a bouncing ball). From a developmental perspective, low-level stimulus characteristics such as spatial proximity and temporal coincidence are available before the construction of learned perceptual representations. Because young organisms can perceive low-level stimulus characteristics, they provide the scaffold on which higher-level representations can be built and constitute the essential first ingredients in the assembly of more complex and sophisticated processing architectures. The early dependence on these low-level stimulus characteristics also fits within the developmental chronology of the brain itself. For example, subcortical regions responsible for simple sensorimotor behaviours mature before cortical regions responsible for more complex perceptual and cognitive functions. Furthermore, different brain regions and circuits mature at different rates and with different sensitivities to physical characteristics versus learned associations [this is exemplified below in our discussion of the temporal binding window (TBW)]. The general fact that multisensory processes are founded on an initial