illustrates that certain covariance changes can worsen decoding.
134 estimation and classification 135 In the first part, we will show the effects of noise correlation on population codes in both 136 a neuronal and a voxel population. We simulated multivariate responses with three and 137 two forms of noise correlation in neuronal and voxel populations respectively. We 138 performed two brain decoding tasks a stimuli-classification task and a stimulus-139 estimation task. In the stimulus-classification task, a linear classifier was trained to 140 categorize evoked population responses into one of two discrete orientation stimuli. In 223 We next turned to examine the impact of NC on population codes in fMRI data. We 224 simulated responses of a voxel population using a voxel-encoding model (Fig. 2B ) and 225 attempted to perform the classification and estimation tasks. 226 We again manipulated two types of NCs-cTCNC (Fig. 3D) and SFNC (Fig. 3E ).
The cTCNCs here are similar to above except that they are between voxels rather than 228 neurons. Similarly, cTCNCs here are defined with respect to the similarity of their 229 orientation tuning curves. SFNCs were also generated by randomly shuffling the cTCNCs 230 between voxels (see Methods). Note that we cannot parametrically derive aTCNCs for 231 voxels as we did for neurons since unlike unimodal orientation tuning curves of cortical 232 neurons, orientation tuning curves of voxels might be irregular (i.e., multimodal) due to 233 the mixing of multiple neural populations in a voxel's activity (see Eq. 9). We will return 234 to this point in a later section. In the second part, we will show how to use information-theoretic analyses to 289 support the simulation results above. Especially, we want to highlight the unit tuning 295 Above analyses focused on assessing the population codes from the decoding 296 perspective (i.e., MVPA), the approach that almost all previous fMRI decoding studies 297 used. Here, we propose an alternative approach-directly calculate the amount of Fisher 298 information for the stimulus-estimation task and linear discriminability for the stimulus- For simplicity, we termed both metrics as "information" as they both indicate the 306 accuracy of population codes with respect to the two tasks.
The analysis of information has three major advantages over the conventional 308 MVPA approach. First, in theory two approaches might lead to consistent results as more 309 information in a population usually leads to a higher decoding accuracy. But their 310 relationship is nonlinear. Classification accuracy can reach the floor (e.g., 50% for binary 311 classification) and ceiling (i.e., 100%) but the amount of information has a relatively 312 broad range thus more sensitive to population codes. For example, as we will show, forms. Because of the tuning heterogeneity, TCNCs do not limit information anymore.
The effect of tuning heterogeneity has been studied in some previous theoretical work analysis. 424 We simulated a simple three-voxel scenario for a classification task to illustrate 425 this effect (Fig. 10) . The NC between voxels X and Y improves classification (Fig. 10A) , 426 while the NC between voxels Y and Z impairs classification (Fig. 10B ). The correlation 427 between X and Y, and the correlation between Y and Z are identical in magnitude but as long as there exists at least one pair of "good" voxels whose NC is beneficial, a linear 434 classifier can utilize the NC to achieve a good decoding performance. In other words, the 435 most informative units determine the decoding accuracy that a decoder that can possibly 436 achieve (Fig. 10) .
This winner-take-all principle provides an intuitive explanation for the effects of The assumption of homogeneous tuning curves in early theoretical work is 496 apparently not realistic because in the primate brain it has been known that the shape of Quantifying information in fMRI data 521 In this paper, we focused on two routinely used perceptual tasks-the stimulus-estimation 522 task and the stimulus-classification task. Stimulus estimation is equivalent to a very fine-523 discrimination task as it needs to discriminate the true stimulus value from nearby stimuli 
Our work presents a good example to apply this approach in future fMRI research.
For example, in a typical binary classification task, we usually measure many trials of 531 population responses towards each of two stimuli. There might be some NCs among 532 voxels but whether they impair or improve population codes is hard to delineate as we The nature of noise correlations in fMRI data 566 Although fMRI can naturally measure the activity of many units in the brain, the 567 investigation of NCs in fMRI data has just begun recently. Exploring this issue in fMRI 568 data is, however, non-trivial and we summarize the related issues as follows.
First, the definition of "noise correlation" in fMRI research is still under debate.
The well-accepted definition of "noise correlation" in computational neuroscience is the 571 correlation of trial-by-trial responses between two neurons given the repeated 572 presentation of the same stimulus. This definition emphasizes stimulus-evoked responses.
In this paper, we strictly follow this definition and assume voxel responses as trial-by-574 trial responses estimated from the standard general linear model. This is also called "beta 575 series correlation" in some fMRI literature (Rissman et al., 2004) . In contrast, one recent 576 study defined the noise correlation between two voxels as their resting-state functional 577 connectivity or background functional connectivity during a task (Bejjanki et al., 2017) .
In theory, these definitions deviate from the conventional definition in computational 579 neuroscience and their quantitative relationship remains unclear. Only one recent study 580 suggested that resting-state functional connectivity is highly correlated with the trial-by-581 trial response correlation at the whole-brain level (Di et al., 2018) . Future studies need to 582 examine the relations between resting-, task-based functional connectivity, and trial-by-583 trial variation of responses at the individual voxel level.
Second, the sources of noise correlations in fMRI data are still unclear. On one 585 hand, the conventional term "noise correlation" itself is somewhat misleading since, as 586 shown in this paper, response variability can contain a substantial amount of stimulus 587 information. In other words, response variability is not purely "noise" and might reflect 588 some critical aspects of how neurons process stimulus or task structure (Bondy et al., 
Previous endeavors of brain decoding generally fall into two broad categories: 604 classification of stimuli into discrete categories (Kamitani & Tong, 2005) and estimation 605 of a continuous stimulus variable (Vintch & Gardner, 2014) . We thus evaluated the effect 606 of NC in brain decoding in two tasks-a stimulus-classification task and a stimulus-607 estimation task. We will first introduce the simulation on a neuronal population and then vary trial-by-trial. We posit that the mean of trial-by-trial population responses is G(s). 627 We will detail the covariance in the following section.
Noise correlation and covariance. We proposed three types of NCs for neuronal data: Several theoretical studies assume the NC between a pair of neurons is an 636 exponential function of the angular difference between their preferred orientations, here 637 defined as angular-based tuning compatible noise correlation (aTCNC):
(2) r ij aTCNC is the NC between the i-th and the j-th neurons. 
The second type is the curve-based tuning compatible noise correlation (cTCNC).
In this case, the NC between a pair of neurons is proportional to their SC (i.e., correlation 649 of their orientation tuning curves):
r ij cTCNC = (1− δ ij ) *corr(g i (S), g j (S)) + δ ij where is the Kronecker delta ( =1 if i=j and = 0 otherwise). S indicates all possible orientations between [1°, 180°], and r ij cTCNC is the NC between the i-th and the 653 j-th neurons. g i (S) and g j (S) are their tuning curves (see Eq. 1). We denote R cTCNC as the 654 correlation matrix. Note that unlike aTCNCs, cTCNCs can be negative (see Fig. 3B ).
Also, the key difference between cTCNC and aTCNC is that cTCNC does not rely on the 656 functional form of tuning curves. In other words, cTCNC can be computed given 657 irregular tuning curves, whereas aTCNC can be only computed from unimodal tuning 658 curves. This is important for specifications of voxelwise NCs (see below).
In the third case, we shuffled the NCs between all pairs of neurons in R cTCNC such 660 that the rows and columns are rearranged in the same randomized order but the diagonal 661 of the matrix is kept intact (Fig. 3C) . We term this type of NC as shuffled noise neurons i and j (q neuron_ij as below) can be expressed as:
where c neuron is a parameter that controls the strength of the neuronal NC. The same pool size and c neuron settings as in the stimulus-classification task were used.
For each combination of a pool size and a c neuron value, we calculated the mean circular 713 squared errors (MSE circ ) across all trials between the estimated stimuli ( ŝ ) and the true 714 stimuli (s) across all trials: where Q vxs is the covariance matrix between voxels, which will be detailed in the 740 following section.
Noise correlation and covariance. We evaluate two types of NCs for simulated fMRI 743 data: cTCNC and SFNC. Note that we cannot evaluate aTCNC for voxel populations 744 because voxel tuning curves here are irregular and not unimodal (see Fig. 8C ).
In the first case, we defined cTCNC using a similar method as Eq. 3:
where r ij cTCNC is the NC between voxels i and j. Note that the cTCNC here is based on the 748 tuning curves of two voxels (i.e., h i (S) and h j (S)), not two neurons. is the Kronecker 749 delta.
In the second case, SFNCs were generated using a similar method as in the 751 neuron-encoding model-shuffling the rows and columns in R cTCNC , which is obtained in 752 Eq. 13. 753 We assume the response variances for different voxels (e.g, the j-th voxels as:
where c vxs is the parameter that controls the strength of the voxelwise NCs. the standard deviation of responses of the two voxels (from Eq. 14), respectively. is both neuronal and voxelwise NCs in the voxel-encoding model (Fig. 6) . In this 799 simulation, we kept the same settings as the simulations above in the voxel-encoding Differences between the neuron-and the voxel-encoding models 812 In this section, we summarize three key differences between the two encoding models (Fig. 6) Information-theoretic analyses 829 We calculated Fisher information in the stimulus-estimation task as it is one of the 830 standard methods to quantify information in computational neuroscience (Brunel & 831 Nadal, 1998). Specifically, we used linear Fisher information, which is a linear 832 approximation of Fisher information, and can be expressed as: 
where s is the stimulus. Note that theoretically linear Fisher information above only 849 applies to the stimulus-estimation task or a fine-discrimination task. For a general 850 classification task (i.e., classify two stimuli s 1 and s 2 ), especially for a coarse 851 discrimination task, Eq.16 can be rewritten as the discrete format: confusion in terminologies, we denoted both metrics as "information" in this paper 857 as they indicate the quality of population codes in the two tasks respectively. In the 858 main text, we only show the information in the stimulus-estimation task (Fig. 6) .
Varying voxel tuning heterogeneity 861 To illustrate the effect of tuning heterogeneity, we performed an additional analysis on 862 the voxel-encoding model (Fig. 8) . In this analysis, we calculated the amount of 863 information in the stimulus-estimation task after making the following modifications. 