The idea that there is a fundamental cortical circuit that performs canonical computations remains compelling though far from proven. Here we review evidence for two canonical operations within sensory cortical areas: a feedforward computation of selectivity; and a recurrent computation of gain in which, given sufficiently strong external input, perhaps from multiple sources, intracortical input largely, but not completely, cancels this external input. This operation leads to many characteristic cortical nonlinearities in integrating multiple stimuli. The cortical computation must combine such local processing with hierarchical processing across areas. We point to important changes in moving from sensory cortex to motor and frontal cortex and the possibility of substantial differences between cortex in rodents vs. species with columnar organization of selectivity.  The cerebral cortex performs a wide range of cognitive tasks in mammals -sensory, motor, and everything in between. Yet it processes these diverse tasks with what appears to be a remarkably uniform, primarily six-layer architecture, albeit with significant differences in details across species and cortical areas [1,2,3 ,4-10,11 ,12-14]. Dense connectivity is generally restricted to horizontal extents (parallel to the layers) of a few hundred microns [e.g. 15]. This has long suggested the idea that a piece of six-layer cortex with a surface area on the order of a square millimeter constitutes a fundamental cortical 'processing unit ' [e.g. 16,17]. The cortex varies in surface area by a factor of 10000 across a set of 37 mammalian species, while thickness (the distance across the layers) varies only by a factor of 10 over the same species [18], suggesting that the most salient evolutionary change in cortex has been enormous multiplication of the number of 'units ' [e.g. 14]. The nature of this six-layer computation remains unclear. The classic picture [e.g. 19,20] is that feedforward input to a given area, which either comes from thalamus or from 'lower' cortical areas, comes dominantly into layer 4 (L4); L4 projects strongly to layers 2/3 (L2/3); L2/3 provides feedforward input to L4 of 'higher' cortical areas, and also projects to L5; L5 provides the only output from cortex other than feedback to thalamus, and also projects to L6; L6 projects up to layers 2 through 4, completing a loop through the layers; and L6, and more generally layers other than L4, provide feedback to areas projecting feedforward input to the given area. Layer 1, which contains inhibitory neurons and dendrites of excitatory neurons from lower layers, is thought to integrate various forms of modulatory input from higher cortical and thalamic areas. Several findings challenge this picture [21 ,22,23], including notably that in mouse primary somatosensory cortex (S1), L5 response to whisker movement is directly driven by thalamic inputs independently of layers 2-4 [21 ]. The picture is being filled out by detailed characterization of pathways involving subtypes of excitatory [e.g. 11 ,15,24,25] and inhibitory [e.g. 15,26 ,27 ,28 ,29 ,30 ,31 ,32-35] neurons in mice. The idea of a cortical computational unit is sketchy and not without controversy, but it motivates the idea that there are fundamental or 'canonical' computations performed by cortical circuits [36,37] that, when fed appropriate inputs and arranged in appropriate hierarchies, might go some ways towards accounting for properties of mammalian intelligence. This idea was at least loosely supported by findings that, when visual inputs were fed to primary auditory cortex (A1), that cortex developed many functional properties normally seen in visual cortex [38]. Some support also comes from studies of artificial neural networks arranged in a hierarchy of layers, far removed from biological details but motivated by these basic ideas. These 'deep' networks have recently made great breakthroughs in training machine intelligence [39 ]. Such networks trained to identify objects in visual images are designed so that, within a given layer, they perform the same operations at each point in visual space. The properties found across layers in these trained networks resemble in some detail those seen across the hierarchically arranged cortical areas in the ventral visual stream, which is thought to perform object recognition [40 ]. These are hardly proofs, but at least show that hierarchical repetition of basic distributed processing units can both achieve Neurons in primary sensory cortices respond to stimuli in localized regions of the sensory periphery (the neuron's 'receptive field'). A critical computational task of sensory cortex is to integrate the discrete, localized information carried by individual neurons into unified percepts of objects in the world. Objects are recognized in a manner invariant to almost all particular details of the incoming stimulus, for example, invariant to rotations, occlusions, or changes of luminance, position, distance, apparent size, or contexts -yet of course dependent on those details [41]. This invariant recognition has, until the recent advent of 'deep' neural networks [39 ], been difficult for machines to replicate, as it is not easily captured by rules. A similar interaction between high-level objects and varying lowlevel implementations is seen in motor planning: for example, an intention to point to an object might be instantiated with various fingers of either hand and various arm configurations, or a foot, or a laser pointer. We might speculate that this difficult-to-characterize, dependent-yet-independent relationship between cognitive elements at higher and lower levels reflects something general about many cognitive processes and thus about cortical computation. How is invariant recognition achieved? While we do not know, we can recognize two elements of the computation from studies of the ventral stream in vision. First, there is a hierarchy of cortical areas, in which neurons in 'higher', more anterior areas of the hierarchy are sensitive to the presence of more complex aspects of objects across larger regions of sensory space and with greater ability to recognize these structures in ways invariant to position or pose [19,41]. Action control may similarly be hierarchically organized from posterior to anterior across areas of motor and frontal cortex [42,43]. Second, within any given area, neurons are selective for relevant features of the stimulus, but responses are modulated by other influences. This modulation tends to modify the gain rather than selectivity of responses, whether evoked by additional stimuli within the receptive field [37], context or surrounding stimuli outside the receptive field [44], or top-down influences such as attention [45] (however, attention to stimulus features can boost the gain of responses to the attended features and thus alter a neuron's stimulus selectivity [46]). The role of this dynamic gain modulation in the computations of invariant recognition is not currently understood. Here we review ideas for these two canonical computations within a sensory cortical area: computation of selectivity and computation of gain. In L4 of primary sensory cortex in several modalities and species, the selectivity of neuronal responses (the relative response strength across different stimuli) is primarily established by the pattern of feedforward connections the neurons receive (thalamic input and perhaps thalamicdriven inhibition). Intracortical recurrent input primarily determines the gain (the absolute response strength) but not the selectivity of responses (although a role of cortical recurrence in selectivity may arise in adaptation or perceptual learning [e.g. 47] or blockade of cortical areas lateral to a site [48]). This was first established in cat V1, in which cortex, and thus intracortical input, was silenced either by cooling [49] or by electrical shock (which evoked massive inhibition) [50], leaving only thalamic input. These manipulations did not change the tuning for stimulus orientation of the membrane potential in recorded neurons, but modified the gain of response. The estimated ratio of voltage response for thalamic input alone to that with cortex intact averaged about 1/3 to 1/2, with ratios broadly distributed between 0 and 1 [50,51]. Similar results were obtained for measurements of excitatory conductance in mouse V1 [52 ,53 ] and A1 [54 ], using optogenetic methods to silence cortex by activating parvalbuminexpressing (PV) inhibitory neurons. In rat S1 representing whiskers, the tuning of L4 neurons for direction of whisker movement corresponds closely to the directional tuning of the thalamic inputs they receive [55,56]. In cat V1, many other tuning properties can also be explained from the tuning of feedforward input [57,58], as suggested by earlier models [59][60][61][62]. The role of feedforward inhibitory input in determining tuning remains unclear. Assuming that orientation selectivity in a L4 V1 neuron is determined by the spatial arrangement of thalamic afferents [52 ,63,64], rather than being inherited from a set of orientation-tuned afferents with similar preferred orientations (the latter has been argued as possible in mouse, e.g. [65][66][67]), the mean input to a L4 cell in response to a drifting luminance grating is not tuned for grating orientation. Rather, the modulation of the input about that mean is orientation tuned, with strong peaks and troughs of input at the preferred orientation but little modulation at orthogonal orientations [61,68,69]. This untuned mean and tuned modulation was observed for thalamic excitatory conductance in mice [52 ], arguing against inheritance of orientation tuning from thalamic afferents, and has not otherwise been examined for thalamic input (other studies measured only the modulation, not the mean, of response [49] or used non-periodic stimuli [50,53 ]). The untuned mean thalamic input grows with stimulus contrast, yet the orientation orthogonal to the preferred (the 'null' orientation) typically does not evoke spiking responses [70]. Why does the increasing thalamic input These results largely describe selectivity in L4. In species with columnar organization of selectivity (similar selectivity across the layers at a given cortical point, further discussed below), selectivity in other layers may be inherited in a fairly straightforward way from L4, so accounting for the origin of selectivity in L4 may suffice. However, in rodents, which lack columnar organization, intracortical recurrent input could play a greater role in deriving selectivity outside L4 (e.g. see [85 ,86 ,87 ] for different versions of one proposal). How is gain modulated in the local cortical circuit? In recent years studies of specific cell types in mice have suggested a number of specific circuits that lower or raise gain through inhibition or disinhibition respectively [26 ,27 ,28 ,29 ,30 ,31 ,32-34]. However, these circuits alone do not explain nonlinearities of modulation. For example, responses in visual cortical areas typically grow with increasing stimulus size up until a critical size (the 'summation field' size), with further size increases suppressing responses ('surround suppression'); but the same surround stimulus may facilitate response to a weak center stimulus yet suppress response to a stronger stimulus [88 ,89-92] and the summation field size shrinks with increasing stimulus contrast [44, [93] [94] [95] [96] [97] [98] [99] [100] . Similarly, responses to two stimuli in the receptive field may add linearly or perhaps supralinearly when they are weak but sublinearly when they are stronger ([101]; T. Ohshiro, D.E. Angelaki and G.C. DeAngelis, abstract in Soc Neurosci Abstr 2013, Program No. 360.19). Recently a very general mechanism of gain modulation [102 ,103 ] has been proposed that can explain these nonlinearities and may be instantiated through a variety of specific circuits, including those recently characterized. The mechanism is based on the input/output (I/O) functions of cortical neurons, which specify firing rate as a function of mean input, being supralinear (meaning that the slope of the I/O function is monotonically increasing, Figure 1b) , as judged by a supralinear relationship between mean membrane voltage and firing rate in 30 ms bins in cat V1 [104] . A supralinear I/O function arises if the mean input neurons receive is subthreshold, so that they fire on the fluctuations in their input rather than on integration of the mean input [105, 106] . Importantly, over the full dynamic range of V1 visual responses, the I/O function stays supralinear, not saturating even for the strongest visual stimuli [104] . The supralinear I/O function causes effective synaptic strengths -the change in output firing rate induced by a change in input rate -to increase with increasing postsynaptic activity. The effective strength is the product of the synaptic strength and the gain of the postsynaptic cell -the change in its rate induced by a given change in synaptic input -which is precisely the slope of the I/O function. This change in gain leads to a change in operating regime of the circuit, as follows. For weak activation, the weak effective synaptic strengths cause the input to neurons to be dominated by external drive (Figure 1c ,d, insets) (synapses both from external and network neurons are weak, but network drive involves two or more weak synapses -external drive moves neuronal rates which alters network drive -and so is weak relative to external drive). Thus, neurons are only weakly coupled. Since the I/O function is supralinear, and neurons are roughly responding directly to their external input, responses to multiple stimuli add supralinearly: the response to two simultaneously presented stimuli exceeds the sum of the responses to the individual stimuli. With increasing activation strength, effective synaptic strengths grow and network drive comes to dominate over external (Figure 1c-e) . In particular, connections between excitatory neurons become strong enough that the excitatory subnetwork becomes unstable by itself -subject to runaway activity -but feedback inhibition, if sufficiently strong, keeps the network stable. To stabilize against the increasing effective excitatory strength, however, the network input must 'conspire' to largely cancel the external input received by cells, leaving a residual net input (Figure 1c,d , green) that grows much more slowly than the growth of the external input (Figure 1c,d, dashed red) [102 ]. Thus, when two stimuli are simultaneously presented, much of the external input will be cancelled and the response will be less than the sum of the responses to the two stimuli presented individually: responses will sum sublinearly (Figure 1g ). This is a key property commonly referred to as 'normalization' -the response can be described as if it were divided ('normalized') by some function of the summed strengths of all stimuli presented, in a form that makes summation sublinear -that has been This simple mechanism, instantiated in simple circuit models, can explain the nonlinear changes in surround suppression and normalization mentioned earlier (it is possible that spontaneous activity is already sufficiently strong to eliminate the supralinear regime, so that weak activation leads to linear rather than supralinear summation). With increasing input drive, the cancellation becomes stronger (Figure 1c,d ) and the network input becomes more dominated by inhibition (Figure 1f ). The most direct tests of this circuit mechanism would be to measure the corresponding currents, for example, regarding each layer as an independent network, but there is currently no way to disentangle a layer's external vs. intrinsic inputs except in L4, where external inputs come from thalamus. Recurrent input has been shown to be increasingly inhibition-dominated, as predicted, in optogenetic activation of upper layers of rodent S1 [114] , but the predicted network effect must be disentangled from effects due to facilitation of excitatory synapses to somatostatin-expressing (SOM) inhibitory neurons [115, 116] . A key set of predictions involve the network being an inhibition-stabilized-network (ISN): stabilized by inhibition despite the excitatory subnetwork being unstable. An ISN shows a 'paradoxical' property: increased (decreased) drive to inhibitory cells causes both excitatory (E) and, paradoxically, inhibitory (I) cells to lower (raise) their firing rates in the new steady state [117] . This is one example of 'balanced amplification ' [118], in which, when strong feedback inhibition balances strong excitation, small biases of input towards I (E) cells causes strong decreases (increases) in both E and I firing rates. Given the multiple subtypes of I neurons and complicated circuitry between them [e.g. 15,35], it is possible that different subtypes of I neurons will have opposite responses, but the robust prediction [discussed in 103 ] is that, in an ISN, the net inhibition received by E cells will go down (up) when their firing is suppressed (facilitated) by increases (decreases) in net external drive to inhibitory neurons. The inhibition received by neurons indeed is paradoxically decreased when they are surround suppressed in cats [119] (but see [120], discussed in [103 ] ). In mice, with increasing stimulus size that suppresses E cell firing, the firing rates of PV I neurons decrease while those of SOM I neurons either increase [121] or decrease [122] ; the latter suggests an ISN, the former is equivocal. In mouse A1, locomotion activates layer 1 inhibitory neurons, causing E cells in layers 2/3 to be suppressed while receiving decreased inhibition, suggesting an ISN [123] . Tests of the transition to an ISN with increasing stimulus strength are possible by manipulating both sensory stimulus strength and optogenetic stimulus spatial frequency [discussed in 103 , Fig. 8 ]. The unknowns and challenges greatly outweigh our current understandings. Obviously we need to better understand computations within the local circuit, including those specific for cell type and/or layer; the hierarchical computations across cortical areas, including the role in the latter of feedback [e.g. 19,124-128] as well as feedforward connections; and the role of structural variations between cortical areas and across species [e.g.
More recently, K.Miller has proposed that two operations performed by cortical columns are a "feedforward computation of selectivity" (similar to clustering operations in STAMs) and a recurrent computation of adaptive gain control between external stimuli and competing internally generated signals (similar to the STAM operation of comparing the result of local clustering with predictions from higher-levels) REF .
Canonical computations of cerebral cortex
Current opinion in neurobiology
Current opinion in neurobiology
Biology Medicine