Two ‘what’ pathways for action and object recognition
Authors: Moritz F. Wurm, Alfonso Caramazza
Journal: Trends in Cognitive Sciences (2021)

Object recognition draws on a ventral what pathway. Is there also a what pathway for action recognition? Recent neuroimaging findings suggest that critical stages of action recognition draw on a pathway in lateral occipitotemporal cortex.This lateral action pathway is organized along two main dimensions: a posterior–anterior gradient from perceptual action precursors to conceptual action representations and a dorsal–ventral distinction that segregates animate and inanimate action aspects. This organization mirrors the domain-specific organization of the ventral object pathway.We propose that occipitotemporal cortex subdivides into lateral and ventral what pathways for action and object recognition. This view explains several observed phenomena of functional organization and thereby extends and concretizes previous conceptions of pathways in visual cortex. The ventral visual stream is conceived as a pathway for object recognition. However, we also recognize the actions an object can be involved in. Here, we show that action recognition critically depends on a pathway in lateral occipitotemporal cortex, partially overlapping and topographically aligned with object representations that are precursors for action recognition. By contrast, object features that are more relevant for object recognition, such as color and texture, are typically found in ventral occipitotemporal cortex. We argue that occipitotemporal cortex contains similarly organized lateral and ventral 'what' pathways for action and object recognition, respectively. This account explains a number of observed phenomena, such as the duplication of object domains and the specific representational profiles in lateral and ventral cortex. The ventral visual stream is conceived as a pathway for object recognition. However, we also recognize the actions an object can be involved in. Here, we show that action recognition critically depends on a pathway in lateral occipitotemporal cortex, partially overlapping and topographically aligned with object representations that are precursors for action recognition. By contrast, object features that are more relevant for object recognition, such as color and texture, are typically found in ventral occipitotemporal cortex. We argue that occipitotemporal cortex contains similarly organized lateral and ventral 'what' pathways for action and object recognition, respectively. This account explains a number of observed phenomena, such as the duplication of object domains and the specific representational profiles in lateral and ventral cortex. The ventral stream, specifically ventral occipitotemporal cortex (VOTC) (see Glossary), has been conceptualized as a pathway for the recognition of objects [1.Ungerleider L.G. Mishkin M. Two cortical visual systems.in: Ingle D.J. Analysis of Visual Behavior. MIT Press, 1982Google Scholar,2.Goodale M.A. Milner A.D. Separate visual pathways for perception and action.Trends Neurosci. 1992; 15: 20-25Abstract Full Text PDF PubMed Scopus (4027) Google Scholar] (e.g., scissors, my grandmother, or the Eiffel Tower). Broadly, the ventral stream is organized along two orthogonal axes: a posterior–anterior gradient mediating the transformation from perceptual to more general, conceptual representations [3.Weiner K.S. et al.The cytoarchitecture of domain-specific regions in human high-level visual cortex.Cereb. Cortex. 2017; 27: 146-161Crossref PubMed Scopus (42) Google Scholar] and a medial–lateral specialization for different, evolutionarily salient knowledge domains such as animate and inanimate objects [4.Konkle T. Caramazza A. Tripartite organization of the ventral stream by animacy and object size.J. Neurosci. 2013; 33: 10235-10242Crossref PubMed Scopus (117) Google Scholar,5.Chao L.L. et al.Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects.Nat. Neurosci. 1999; 2: 913-919Crossref PubMed Scopus (943) Google Scholar] and more specific object categories like faces, body parts, tools, and places [5.Chao L.L. et al.Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects.Nat. Neurosci. 1999; 2: 913-919Crossref PubMed Scopus (943) Google Scholar, 6.Bracci S. Peelen M.V. Body and object effectors: the organization of object representations in high-level visual cortex reflects body-object interactions.J. Neurosci. 2013; 33: 18247-18258Crossref PubMed Scopus (0) Google Scholar, 7.Epstein R.A. Baker C.I. Scene perception in the human brain.Annu. Rev. Vis. Sci. 2019; 5: 373-397Crossref PubMed Scopus (11) Google Scholar]. This categorical specialization is thought to be driven by the connectivity to different downstream regions, for example, networks for social cognition, object manipulation, and navigation [8.Mahon B.Z. Caramazza A. What drives the organization of object knowledge in the brain?.Trends Cogn. Sci. 2011; 15: 97-103Abstract Full Text Full Text PDF PubMed Scopus (211) Google Scholar,9.Peelen M.V. Downing P.E. Category selectivity in human visual cortex: beyond visual object recognition.Neuropsychologia. 2017; 105: 177-183Crossref PubMed Scopus (30) Google Scholar]. However, we do not only recognize objects (what is it?) but also the actions an object can be involved in (what is it doing?; e.g., cutting, running, or greeting). Recognizing actions (and perhaps also inanimate and unintentional events, e.g., colliding or slipping) plays a key role in guiding our behavior, as it provides the basis for inferring others' plans and desires, reacting appropriately, and social learning. Hence, it seems plausible that evolutionary pressures led to the development of neural circuits specialized for action recognition. Here, we use a functional approach to address the following questions: Which parts of the brain form a what pathway or pathways for action recognition? Do similar organizational principles apply to actions as has been observed for objects? Since action recognition usually builds on the recognition of objects, in that actions are constrained by the properties of objects as agents, instruments, or recipients of actions, how do the neural representations of objects (e.g., body parts, tools, or fruits) and actions (e.g., running, cutting, and eating) relate to each other? To address these questions, we focus on action aspects that capture what an action is for; typically the effect of an action (e.g., opening as a means to bring something into an open state), and leave aside other kinds of action knowledge, such as knowledge related to action production (how to perform a certain action, e.g., with the hand or foot) or the hidden motivations and reasons of actions (why an action is performed, e.g., drinking to quench thirst), which may rely on complementary neural systems. While action recognition draws on a network of occipitotemporal, parietal, and frontal brain regions, recent proposals emphasized the role of the lateral occipitotemporal cortex (LOTC) in representing various action-related aspects such as tools and body parts and their movements, verbs, and higher-level semantic action knowledge [10.Lingnau A. Downing P.E. The lateral occipitotemporal cortex in action.Trends Cogn. Sci. 2015; 19: 268-277Abstract Full Text Full Text PDF PubMed Scopus (124) Google Scholar, 11.Leshinskaya A. et al.Concepts of actions and their objects.in: Gazzaniga M. The Cognitive Neurosciences. MIT Press, 2020: 757-765Google Scholar, 12.Weiner K.S. Grill-Spector K. Neural representations of faces and limbs neighbor in human high-level visual cortex: evidence for a new organization principle.Psychol. Res. 2013; 77: 74-97Crossref PubMed Scopus (117) Google Scholar]. The LOTC is located in a pathway that appears structurally separate from the traditional dorsal and ventral pathways [12.Weiner K.S. Grill-Spector K. Neural representations of faces and limbs neighbor in human high-level visual cortex: evidence for a new organization principle.Psychol. Res. 2013; 77: 74-97Crossref PubMed Scopus (117) Google Scholar, 13.Pitcher D. Ungerleider L.G. Evidence for a third visual pathway specialized for social perception.Trends Cogn. Sci. 2021; 25: 100-110Abstract Full Text Full Text PDF PubMed Scopus (28) Google Scholar, 14.Boussaoud D. et al.Pathways for motion analysis: cortical connections of the medial superior temporal and fundus of the superior temporal visual areas in the macaque.J. Comp. Neurol. 1990; 296: 462-495Crossref PubMed Scopus (508) Google Scholar, 15.Haak K.V. Beckmann C.F. Objective analysis of the topological organization of the human cortical visual connectome suggests three visual pathways.Cortex. 2018; 98: 73-83Crossref PubMed Scopus (20) Google Scholar]. In the following, we build on these foundations and integrate the different functional roles proposed for this pathway with recent findings pointing toward an action recognition hierarchy in the LOTC that mirrors the domain-specific organization of the ventral object pathway. We argue that the original conception of the ventral stream as a what pathway can be extended to the LOTC. According to this view, lateral and ventral parts of the OTC subserve complementary recognition functions, with subordinate specializations for action and object recognition, respectively. This interpretation explains several observed phenomena, such as the dissociation of response preferences for object properties that are disproportionally relevant for action or object recognition (e.g., motion and color, respectively) and the duplication of response preferences for certain object categories that are important for both action and object recognition (e.g., body parts, faces, and inanimate objects). Action recognition is associated with activation of a network consisting of the LOTC and lateral frontal and parietal brain regions [10.Lingnau A. Downing P.E. The lateral occipitotemporal cortex in action.Trends Cogn. Sci. 2015; 19: 268-277Abstract Full Text Full Text PDF PubMed Scopus (124) Google Scholar,16.Oosterhof N.N. et al.Crossmodal and action-specific: neuroimaging the human mirror neuron system.Trends Cogn. Sci. 2013; 17: 311-318Abstract Full Text Full Text PDF PubMed Scopus (0) Google Scholar,17.Orban G.A. et al.From Observed Action Identity to Social Affordances.Trends Cogn. Sci. 2021; 25: 493-505Abstract Full Text Full Text PDF PubMed Scopus (0) Google Scholar]. Since recognizing an action (e.g., cutting an apple) usually includes the visual analysis of objects (apple, knife, and hands), a straightforward interpretation is that the LOTC processes perceptual action details like body parts and objects and their movements whereas higher-level conceptual action aspects are encoded more anteriorly and dorsally in frontoparietal cortex. However, findings taken to support this view also allow alternative interpretations [16.Oosterhof N.N. et al.Crossmodal and action-specific: neuroimaging the human mirror neuron system.Trends Cogn. Sci. 2013; 17: 311-318Abstract Full Text Full Text PDF PubMed Scopus (0) Google Scholar,18.Caramazza A. et al.Embodied cognition and mirror neurons: a critical assessment.Annu. Rev. Neurosci. 2014; 37: 1-15Crossref PubMed Scopus (143) Google Scholar], and other findings – in particular those based on the performance of brain damaged individuals [19.Tarhan L.Y. et al.Shared and distinct neuroanatomic regions critical for tool-related action production and recognition: evidence from 131 left-hemisphere stroke patients.J. Cogn. Neurosci. 2015; 27: 2491-2511Crossref PubMed Scopus (48) Google Scholar,20.Vannuscorps G. et al.Persistent sparing of action conceptual processing in spite of increasing disorders of action production: a case against motor embodiment of action concepts.Cogn. Neuropsychol. 2016; 33: 191-219Crossref PubMed Scopus (9) Google Scholar] – provide counterevidence against this view (but see [21.Urgesi C. et al.Neuroanatomical substrates of action perception and understanding: an anatomic likelihood estimation meta-analysis of lesion-symptom mapping studies in brain injured patients.Front. Hum. Neurosci. 2014; 8: 344Crossref PubMed Scopus (87) Google Scholar]). Instead of reviewing the arguments resulting from this large body of work, we focus in this section on more recent neuroimaging studies which aimed at characterizing the representational profiles of occipitotemporal and frontoparietal regions with specific regard to conceptual action representation, that is, the 'cardinal stage' of action recognition that captures the core 'what' meaning of an action. We then use this stage as an anchor point from which we move posteriorly to characterize more basic, perceptual levels of action representation. Which parts of the brain represent actions at a conceptual level? We define three criteria that seem necessary (but not sufficient in isolation): (i) an action concept (e.g., opening) should generalize across different exemplars of the action (e.g., the opening of a bottle or of a box); (ii) an action concept should generalize across different input modalities or stimulus formats, for example, it should respond to both the observation of an action and to a verbal description of that action; and (iii) the similarity of conceptual action representations should follow specific semantic principles (see Box 1 for further details). Recent advances in neuroimaging, specifically multivariate pattern analysis (MVPA) and representational similarity analysis (RSA), allow identifying brain regions that meet these criteria. fMRI studies using these approaches revealed exemplar–general action representations in the anterior and posterior LOTC and inferior parietal lobe (IPL). Action representations in these regions generalize across perceptual variants of these actions differing in terms of the objects involved, for example, bottles, boxes, doors, and trash bins [22.Vannuscorps G. et al.Large-scale organization of the hand action observation network in individuals born without hands.Cereb. Cortex. 2019; 29: 3434-3444Crossref PubMed Scopus (10) Google Scholar, 23.Wurm M.F. et al.Decoding concrete and abstract action representations during explicit and implicit conceptual processing.Cereb. Cortex. 2016; 26: 3390-3401Crossref PubMed Scopus (33) Google Scholar, 24.Wurm M.F. et al.Action categories in lateral occipitotemporal cortex are organized along sociality and transitivity.J. Neurosci. 2017; 37: 562-575Crossref PubMed Scopus (33) Google Scholar, 25.Wurm M.F. Lingnau A. Decoding actions at different levels of abstraction.J. Neurosci. 2015; 35: 7727-7735Crossref PubMed Scopus (80) Google Scholar], and the body parts [22.Vannuscorps G. et al.Large-scale organization of the hand action observation network in individuals born without hands.Cereb. Cortex. 2019; 29: 3434-3444Crossref PubMed Scopus (10) Google Scholar] and their movements [25.Wurm M.F. Lingnau A. Decoding actions at different levels of abstraction.J. Neurosci. 2015; 35: 7727-7735Crossref PubMed Scopus (80) Google Scholar] needed to manipulate these different objects. Frontal brain regions, such as premotor cortex, are mostly found for the discrimination of actions that differ not only conceptually but also in terms of involved body parts [26.Hafri A. et al.Neural representations of observed actions generalize across static and dynamic visual input.J. Neurosci. 2017; 37: 3056-3071Crossref PubMed Scopus (17) Google Scholar] or body movements [24.Wurm M.F. et al.Action categories in lateral occipitotemporal cortex are organized along sociality and transitivity.J. Neurosci. 2017; 37: 562-575Crossref PubMed Scopus (33) Google Scholar], suggesting that frontal representations encode effector- and movement-related rather than conceptual action information. Modality–general action representations, which are accessible both via observation and written descriptions, were found using fMRI-based crossdecoding in the left anterior LOTC [27.Wurm M.F. Caramazza A. Distinct roles of temporal and frontoparietal cortex in representing actions across vision and language.Nat. Commun. 2019; 10: 289Crossref PubMed Scopus (24) Google Scholar]. In the same study, RSA revealed that the neural similarity of modality–general action representations in the anterior LOTC can be predicted by models that capture various semantic aspects: whether actions are directed toward persons or not, directed toward inanimate objects or not, or similar in terms of more complex semantic relations. This finding is in line with other studies investigating the representational organization of observed actions: Person and object directedness (also labeled sociality and transitivity, respectively) captures the similarity of activation patterns associated with observed actions in the LOTC [24.Wurm M.F. et al.Action categories in lateral occipitotemporal cortex are organized along sociality and transitivity.J. Neurosci. 2017; 37: 562-575Crossref PubMed Scopus (33) Google Scholar,28.Tarhan L. Konkle T. Sociality and interaction envelope organize visual action representations.Nat. Commun. 2019; 11: 3002Crossref Scopus (9) Google Scholar]. The latter studies also found that frontoparietal and VOTC regions are sensitive to the person and object directedness of actions. However, only the left LOTC is found to be sensitive to semantic action dimensions when perceptual factors that typically covary with person and object directedness, such as reaching/grasping movements and presence of persons and objects, are explicitly modeled and thus controlled for [29.Wurm M.F. Caramazza A. Lateral occipitotemporal cortex encodes perceptual components of social actions rather than abstract representations of sociality.Neuroimage. 2019; 202116153Crossref PubMed Scopus (3) Google Scholar,30.Tucciarelli R. et al.The representational space of observed actions.eLife. 2019; 8e47686Crossref PubMed Scopus (3) Google Scholar]. These findings suggest that conceptual action representations are primarily represented in the left LOTC, whereas frontoparietal regions and the VOTC encode stimulus-specific action-related aspects. It is nonetheless possible that other regions meet conceptual criteria but are not detected with current neuroimaging methods. For example, single-cell recordings in humans have revealed that, in parietal cortex, action representations can be found that generalize across action videos and verbs [31.Aflalo T. et al.A shared neural substrate for action verbs and observed actions in human posterior parietal cortex.Sci. Adv. 2020; 6eabb3984Crossref PubMed Scopus (0) Google Scholar]. However, the neural response profile found in this study were best explained by a model that captured stimulus-specific aspects, pointing toward a role of parietal cortex that is different from conceptual representation (see also [32.Lanzilotto M. et al.Stable readout of observed actions from format-dependent activity of monkey's anterior intraparietal neurons.Proc. Natl. Acad. Sci. U. S. A. 2020; 117: 16596-16605Crossref PubMed Scopus (3) Google Scholar]) (Box 2 for possible roles).Box 1Localizing conceptual action representations in the brainNeural representations of action concepts should fulfil three criteria. (i) An action concept (e.g., opening) should generalize across different concrete instantiations of the action type (Figure IA, left). Note that this criterion is stricter than viewpoint or movement invariance; for example, when observing the same action from different viewpoints or when the same action outcome is achieved with different body movements. Representations that generalize across the latter dimensions could also capture higher-level perceptual features such as viewpoint-invariant motion trajectories or shape. Exemplar–general action representations can be identified by training a classifier to distinguish neural activation patterns associated with the observation of different actions (e.g., of opening and closing bottles) and testing the classifier on its ability to distinguish the same kinds of actions on different objects (e.g., boxes). (ii) An action concept should generalize across different input modalities; for example, observation or language (Figure IA, middle). Modality-general action representations can be identified by training a classifier to discriminate activation patterns associated with observed actions and testing the classifier on its accuracy to discriminate activation patterns associated with corresponding action sentences. (iii) The representational similarity of different action concepts should reflect their semantic similarity rather than similarity along concept-irrelevant dimensions (Figure IA, right). Among different semantic dimensions, we here refer to 'what' aspects that are critical for defining an action's meaning. For example, the core 'what' meaning of hitting is the (forceful) contact with another object rather than the fact that it can be carried out with the foot or with the hand, which reflect narrower meanings related to 'how' aspects. Using RSA, the similarities between activation patterns associated with different actions can be compared with models of similarity that capture, for example, whether actions are directed toward persons or involve an interaction with manipulable objects. fMRI studies that were specifically designed to localize neural representations that meet these criteria reveal converging evidence for conceptual action representation in left anterior LOTC (Figure IB).Box 2The role of frontoparietal cortex in action recognitionThe frontoparietal cortex, specifically premotor cortex and IPL, are typically activated during action recognition. The neuroimaging evidence presented here argues against the view that these brain regions represent actions at conceptual levels that captures the core 'what' meaning of actions. In addition, there is accumulating evidence against the view that sensorimotor representations of actions in frontoparietal cortex are strictly necessary for action recognition [18.Caramazza A. et al.Embodied cognition and mirror neurons: a critical assessment.Annu. Rev. Neurosci. 2014; 37: 1-15Crossref PubMed Scopus (143) Google Scholar,20.Vannuscorps G. et al.Persistent sparing of action conceptual processing in spite of increasing disorders of action production: a case against motor embodiment of action concepts.Cogn. Neuropsychol. 2016; 33: 191-219Crossref PubMed Scopus (9) Google Scholar,111.Thompson E.L. et al.Conceptualizing and testing action understanding.Neurosci. Biobehav. Rev. 2019; 105: 106-114Crossref PubMed Scopus (14) Google Scholar, 112.Vannuscorps G. Caramazza A. Typical action perception and interpretation without motor simulation.Proc. Natl. Acad. Sci. U. S. A. 2016; 113: 86-91Crossref PubMed Scopus (52) Google Scholar, 113.Hickok G. The Myth of Mirror Neurons: the Real Neuroscience of Communication and Cognition. W.W. Norton, 2014Google Scholar, 114.Heyes C. Catmur C. What happened to mirror neurons?.Perspect. Psychol. Sci. 2021; (Published online July 9, 2021)https://doi.org/10.1177/1745691621990638Crossref PubMed Scopus (3) Google Scholar]. What then could be the role of frontal and parietal regions in action recognition and representation? Various studies suggest that frontoparietal regions, in particular anterior IPL, are necessary for knowing how a tool is grasped and manipulated for proper use [104.Goldenberg G. Spatt J. The neural basis of tool use.Brain. 2009; 132: 1645-1655Crossref PubMed Scopus (222) Google Scholar,115.Buxbaum L.J. Kalenine S. Action knowledge, visuomotor activation, and embodiment in the two action systems.Ann. N. Y. Acad. Sci. 2010; 1191: 201-218Crossref PubMed Google Scholar]. In line with this view, the frontoparietal cortex appears mostly sensitive to the kinematics of actions [111.Thompson E.L. et al.Conceptualizing and testing action understanding.Neurosci. Biobehav. Rev. 2019; 105: 106-114Crossref PubMed Scopus (14) Google Scholar]. Besides these 'how' aspects of action knowledge, there are also 'why' aspects, which capture the hidden motives underlying an action, such as desires, goals, and intentions. These inferred aspects of actions might be represented in frontoparietal regions, specifically in posterior IPL [116.Leshinskaya A. Caramazza A. Nonmotor aspects of action concepts.J. Cogn. Neurosci. 2014; 26: 2863-2879Crossref PubMed Scopus (7) Google Scholar] or regions of the mentalizing network such as the temporoparietal junction and anterior STS [117.Spunt R.P. Lieberman M.D. Dissociating modality-specific and supramodal neural systems for action understanding.J. Neurosci. 2012; 32: 3575-3583Crossref PubMed Scopus (0) Google Scholar,118.Thornton M.A. Mitchell J.P. Theories of person perception predict patterns of neural activity during mentalizing.Cereb. Cortex. 2018; 28: 3505-3520Crossref PubMed Scopus (25) Google Scholar]. Additionally, frontoparietal regions are likely involved in action recognition by providing supportive functions related to working memory [119.Vannuscorps G. Caramazza A. Impaired short-term memory for hand postures in individuals born without hands.Cortex. 2016; 83: 136-138Crossref PubMed Scopus (6) Google Scholar] and visuospatial and attentional processes that are disproportionally relevant for action recognition relative to object recognition [16.Oosterhof N.N. et al.Crossmodal and action-specific: neuroimaging the human mirror neuron system.Trends Cogn. Sci. 2013; 17: 311-318Abstract Full Text Full Text PDF PubMed Scopus (0) Google Scholar]. According to a recent proposal, IPL might also play a role in representing the 'social affordances' of actions, by integrating observed body movements and the effects of an action (originating from the LOTC) with potential motor plans to support the selection of behavioral reactions in frontal cortex [17.Orban G.A. et al.From Observed Action Identity to Social Affordances.Trends Cogn. Sci. 2021; 25: 493-505Abstract Full Text Full Text PDF PubMed Scopus (0) Google Scholar]. Finally, frontoparietal regions could be involved in processes that follow action recognition such as activating associated motor representations in the service of learning and imitation [113.Hickok G. The Myth of Mirror Neurons: the Real Neuroscience of Communication and Cognition. W.W. Norton, 2014Google Scholar], and anticipating future actions and events based on the recognized action [120.Wurm M.F. et al.Predicting goals in action episodes attenuates BOLD response in inferior frontal and occipitotemporal cortex.Behav. Brain Res. 2014; 274: 108-117Crossref PubMed Scopus (16) Google Scholar]. Some of these functions also relate to mechanical reasoning, which may be particularly important for inferring the meaning of unfamiliar actions or functions of novel tools [11.Leshinskaya A. et al.Concepts of actions and their objects.in: Gazzaniga M. The Cognitive Neurosciences. MIT Press, 2020: 757-765Google Scholar,104.Goldenberg G. Spatt J. The neural basis of tool use.Brain. 2009; 132: 1645-1655Crossref PubMed Scopus (222) Google Scholar]. Related to this point, understanding how different entities in an observed action scene causally impact with each other might be represented at a perceptually invariant level and independent of object identity. This kind of representation might explain the finding of exemplar-general, but not modality-general action decoding in parietal cortex [22.Vannuscorps G. et al.Large-scale organization of the hand action observation network in individuals born without hands.Cereb. Cortex. 2019; 29: 3434-3444Crossref PubMed Scopus (10) Google Scholar, 23.Wurm M.F. et al.Decoding concrete and abstract action representations during explicit and implicit conceptual processing.Cereb. Cortex. 2016; 26: 3390-3401Crossref PubMed Scopus (33) Google Scholar, 24.Wurm M.F. et al.Action categories in lateral occipitotemporal cortex are organized along sociality and transitivity.J. Neurosci. 2017; 37: 562-575Crossref PubMed Scopus (33) Google Scholar, 25.Wurm M.F. Lingnau A. Decoding actions at different levels of abstraction.J. Neurosci. 2015; 35: 7727-7735Crossref PubMed Scopus (80) Google Scholar,27.Wurm M.F. Caramazza A. Distinct roles of temporal and frontoparietal cortex in representing actions across vision and language.Nat. Commun. 2019; 10: 289Crossref PubMed Scopus (24) Google Scholar]. Neural representations of action concepts should fulfil three criteria. (i) An action concept (e.g., opening) should generalize across different concrete instantiations of the action type (Figure IA, left). Note that this criterion is stricter than viewpoint or movement invariance; for example, when observing the same action from different viewpoints or when the same action outcome is achieved with different body movements. Representations that generalize across the latter dimensions could also capture higher-level perceptual features such as viewpoint-invariant motion trajectories or shape. Exemplar–general action representations can be identified by training a classifier to distinguish neural activation patterns associated with the observation of different actions (e.g., of opening and closing bottles) and testing the classifier on its ability to distinguish the same kinds of actions on different objects (e.g., boxes). (ii) An action concept should generalize across different input modalities; for example, observation or language (Figure IA, middle). Modality-general action representations can be identified by training a classifier to discriminate activation patterns associated with observed actions and testing the classifier on its accuracy to discriminate activation patterns associated with corresponding action sentences. (iii) The representational similarity of different action concepts should reflect their semantic similarity rather than similarity along concept-irrelevant dimensions (Figure IA, right). Among different semantic dimensions, we here refer to 'what' aspects