Original: Efﬁcient coding, channel capacity and the emergence of retinal mosaics
Revised: Efﬁcient coding, channel capacity and the emergence of retinal mosaics

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Among the most striking features of retinal organization is the grouping of its outputneurons, the retinal ganglion cells (RGCs), into a diversity of functional types.
Revised: Among the most striking features of retinal organization is the grouping of itsoutput neurons, the retinal ganglion cells, into a diversity of functional types.

Original: Each of these types exhibits a mosaic-like organization of receptive ﬁelds (RFs) thattiles the retina and visual space.
Revised: Eachof these types exhibits a mosaic-like organization of receptive ﬁelds that tiles theretina and visual space.

Original: Previous work has shown that many features of RGC organization, including the existence of ON and OFF cell types, the structureof spatial RFs, and their relative arrangement, can be predicted on the basis ofefﬁcient coding theory.
Revised: Previous work has shown that many features of retinalganglion cell organization, including the existence of ON and OFF cell types, thestructure of spatial receptive ﬁelds, and their relative arrangement, can be predictedon the basis of efﬁcient coding theory.

Original: This theory posits that the nervous system is organizedto maximize information in its encoding of stimuli while minimizing metaboliccosts.
Revised: This theory posits that the nervous systemis organized to maximize information in its encoding of stimuli while minimizingmetabolic costs.

Original: Here, we use efﬁcient coding theory to present a comprehensive account ofmosaic organization in the case of natural videos as the retinal channel capacity—the number of simulated RGCs available for encoding—is varied.
Revised: Here, we use efﬁcient coding theory to present a comprehensiveaccount of mosaic organization in the case of natural videos as the retinal channelcapacity—the number of neurons available for encoding—is varied.

Original: We show thatmosaic density increases with channel capacity up to a series of critical points atwhich, surprisingly, new cell types emerge. Each successive cell type focuses onincreasingly high temporal frequencies and integrates signals over large spatialareas.
Revised: Using a simplemodel of efﬁcient coding, we show that, beginning with spatially localized temporalsmoothing ﬁlters, mosaic density increases with channel capacity up to a seriesof critical points at which new mosaics focused on increasingly high temporalfrequencies emerge.

Original: In addition, we show we show theoretically and in simulation that a transitionfrom mosaic alignment to anti-alignmentacross pairs of cell types is observed withincreasing output noise and decreasing input noise.
Revised: In addition, we show both experimentally and theoreticallythat a transition from mosaic alignment to anti-alignment is observed not onlywith increasing output noise as previously reported, but also with decreasing inputnoise.

Original: Together, these results offer auniﬁed perspective on the relationship between retinal mosaics, efﬁcient coding,and channel capacity that can help to explain the stunning functional diversity ofretinal cell types.
Revised: Together, these results offer a uniﬁed perspective on the relationship betweenretinal mosaics, efﬁcient coding, and channel capacity that may help to explain thestunning functional diversity of retinal mosaics.

Original:  Introduction
Revised:  Introduction

Original: The retina is one of the most intensely studied  neural circuits, yet we still lack a computationalunderstanding of its organization in relation to its function.
Revised: The retina is one of the most intensely studied of all neural circuits, yet we still lack a computationalunderstanding of its organization in relation to its function.

Original: At a structural level, the retina formsa three-layer circuit, with its primary feedforward pathway consisting of photoreceptors to bipolarcells to retinal ganglion cells (RGCs), the axons of which form the optic nerve [1].
Revised: At a structural level, the retina forms athree-layer circuit, with its primary feedforward pathway consisting of light-sensitive photoreceptorsthat project to bipolar cells that project in turn to retinal ganglion cells (RGCs), the axons of whichform the optic nerve [1].

Original: RGCs can bedivided into 30-50 functionally distinct cell types (depending on species) with each cell responsiveto a localized area of visual space (its receptive ﬁeld (RF)), and the collection of RFs for each typetiling space to form a “mosaic” [2, 3, 4, 5].
Revised: At a functional level, retinal ganglion cells are observed to be divided into30-50 functional types, depending on species, with each cell responsive to a localized area of visualspace (its receptive ﬁeld (RF)) and the collection of receptive ﬁelds of each type tiling visual space toform distinct functional “mosaics” [2, 3].

Original: Each mosaic  represents the extraction of a speciﬁc typeof information across the visual scene by a particular cell type, with different mosaics responding tolight increments or decrements (ON and OFF cells), high or low spatial and temporal frequencies,color, motion, and a host of other features.
Revised: At a computational level, each mosaic thus represents theextraction of a speciﬁc type of information from the visual scene, with different mosaics respondingto light increments or decrements (ON and OFF cells), high or low spatial and temporal frequencies,color, motion, and a host of other variables.

Original: While much is known about the response properties ofeach RGC type, the computational principles that drive RGC diversity remain unclear.
Revised: Yes, while much is known about the response propertiesof each type, the computational principles that drive RGC diversity remain unclear.

Original: Efﬁcient coding theory has proven one of the most powerful ideas for understanding retinal organization and sensory processing.
Revised: For more than sixty years, efﬁcient coding theory has proven one of the most powerful ideas forunderstanding retinal organization.

Original: Efﬁcient coding posits that the nervous system attempts to encode [Figure] sensory input by minimizing redundancy subject to biological costs and constraints [6, 7].
Revised: As originally proposed by Barlow [4], efﬁcient coding positsthat the nervous system attempts to encode the sensory world by minimizing redundancy subject tobiological costs and constraints.

Original: As morecommonly formulated, it seeks to maximize the mutual information between sensory data and neuralrepresentations, with the most common cost in the retinal case being the energetic cost of actionpotentials transmitted by the RGCs.
Revised: As more commonly formulated, it seeks to maximize the mutual information between sensory data and neural representations, with the most common cost in theretinal case being the energetic cost of action potentials transmitted by the ganglion cells.

Original: Despite its simplicity, this principle has proven useful, predictingthe center-surround structure of RFs [8], the frequency response proﬁle of contrast sensitivity [9], thestructure of retinal mosaics [10, 11], the role of nonlinear rectiﬁcation [12], different spatiotemporalkernels [13], and inter-mosaic arrangements [14, 15].
Revised: Despite itssimplicity, this principle, has proven highly effective, having predicted the center-surround structureof receptive ﬁelds [5], the frequency response proﬁle of contrast sensitivity [6], the structure ofretinal mosaics [7, 8], the role of nonlinear rectiﬁcation [9], different spatiotemporal kernels [10],and inter-mosaic arrangements [11, 12].

Original: While previous studies have largely focused on either  spatial or  temporal aspects of efﬁcient coding,we optimize an efﬁcient coding model of retinal processing in both space and time to natural videos[16].
Revised: While previous studies have largely focused on either the spatial or the temporal aspects of efﬁcientcoding, we optimize for the ﬁrst time a model of retinal efﬁcient coding of natural spatiotemporalmovies.

Original: We systematically varied the number of cells available to the system and found that largernumbers of available cells led to more cell types.
Revised: This optimization revealed factors that determine the number of functional types. Speciﬁcally,we found that increasing the number of modeled ganglion cells in the optimization (analogous to thechannel capacity in information theory) increased the number of distinct functional types that resultfrom the optimization.

Original: Each of these functionally distinct types formedits own mosaic of RFs that tiled space.
Revised: Each of these functionally distinct types forms its own mosaic of receptiveﬁelds that approximately tiles space.

Original: We show that when and how new cell types emerge andform mosaics is the result of tradeoffs between power constraints and the beneﬁts of specializedencoding that shift as more cells are available to the system. We  show that cell types begin bycapturing low-frequency temporal information and capture increasingly higher-frequency temporalinformation over larger spatial RFs as new cell types form.
Revised: In the case of linear encoding, we show analytically how atradeoff between information gains from increasing mosaic density and informationg gains fromadding new, specialized cell types leads to the addition of new mosaics capturing higher temporalfrequency information. We then show that the same phenomenon occurs in the case of a nonlinearmodel trained to maximize mutual information subject to ﬁring rate costs.

Original: Finally, we investigated the relativearrangement of these mosaics and their dependence on noise. 
Revised: Finally, we investigatethe relative arrangement of these mosaics, showing that while they exhibit the same phase transitionidentiﬁed for purely spatial encoding [11], different ON-OFF mosaic pairs exhibit different transitionproﬁles as a function of input noise. Related work: As indicated above, this study builds on a long line of work using efﬁcient codingprinciples to understand retinal processing.

Original: We show that mosaic pairs can bealigned or anti-aligned depending on input and output noise in  the system [14].
Revised: In addition, it is related to work examining encodingof natural videos [13, 14, 15] and prediction in the spacetime context. However, the most closelyrelated work to this one is that of [10], which also considered efﬁcient coding of natural videos andconsidered the tradeoffs involved in multiple mosaics. Our treatment here differs from that work inseveral key aspects: First, the linear analytical model of [10] focused on least squares reconstructionerror in a convolutional neural framework with grids of regularly spaced receptive ﬁelds. In otherwords, that study predetermined the number of cell types and optimized their relative spacing.

Original: Together, these resultsdemonstrate for the ﬁrst time how efﬁcient coding principles can explain, even predict, the formationof cell types, and  which types are most informative when channel capacity is limited.
Revised: Here,we follow previous efﬁcient coding models [5, 6, 7, 8] in maximizing mutual information and do notassume an a prior mosaic arrangement, a particular cell spacing, or a particular number of cell types. Second, while the computational model of [10] optimized strides for a pair of rectangular arrays ofganglion cells, we individually optimize receptive ﬁeld locations and shapes, allowing us to studychanges in optimal receptive ﬁeld size and density as new, partial mosaics (up to ﬁve pairs) begin toform. Third, while [10] used zero-padding of natural videos to bias learned temporal ﬁlters towardthose of observed ganglion cells, we link the form of our temporal receptive ﬁelds to biophysicallimits on the ﬁltering properties of bipolar cells, producing temporal ﬁlters with the delay propertiesobserved in experimental data. Finally, while [10] only considered a single noise source in theirmodel, we consider noise in both photoreceptor responses (input noise) and ganglion cell responses(output noise), which allows us to investigate the mosaic alignment transitions that are unique to thiswork.

Original:  Model
Revised:  Model

Original: The model we develop is an extension of [14], a retinal model for efﬁcient coding of natural images,which is based on a mutual information maximization objective proposed in [10].
Revised: The model we develop is an extension of [11], a retinal model for efﬁcient coding of natural images,which is based on a mutual information maximization objective proposed in [7].

Original: The retinal modeltakes D -pixel patches of natural images x 2 R D corrupted by input noise n x ⇠ N (0 , C n in ) , ﬁltersthese with unit-norm linear kernels { w j 2 R D | k w j k = 1 } j =1 , ··· ,J representing J RGCs, and thenfeeds the resulting signals y j = w > j
Revised: The retinal modeltakes D -pixel patches of natural images x 2 R D corrupted by input noise n x ⇠ N (0 , C n in ) , whichare ﬁltered with unit-norm linear kernels { w j 2 R D | k w j k = 1 } j =1 , ··· ,J representing J retinalganglion cells. The resulting signals y j = w > j

Original: ( x + n in )  through softplus nonlinearities ⌘ ( y )
Revised: ( x + n in ) are then fed through softplus nonlinearities⌘ ( y )

Original: = log  

Revised: = log

Original: + e  y  / (we used  = 0 . 25 ) with gain  j and threshold ✓ j . Finally, these signals are further corrupted byadditive output noise n out ⇠ N (0 , C n out ) , to produce ﬁring rates r j :
Revised: + e  y  /  with gain  j and threshold ✓ j and ﬁnally further corrupted by additiveoutput noise n out ⇠ N (0 , C n out ) , to produce ﬁring rates r j :

Original: [Equation]
Revised: [Equation]

Original: Themodel learns parameters w j ,  j , and ✓ j  to maximize the mutual information between the inputsx and the outputs r , under a mean ﬁring rate constraint:maximize
Revised: The parameters w j ,  j , and ✓ j are learned to maximize the mutual information between the inputs xand the outputs r , under a mean ﬁring rate constraint: [Equation] to E [ r j ] = 1 .

Original: [Equation]
Revised: [Equation]

Original: Here C x is the covariance matrix of the input distribution,
Revised: Here C x is the covariance matrix of the input distribution,

Original: W 2 R D ⇥ J contains the ﬁlters w j as itscolumns, the gain matrix G = diag
Revised: W 2 R D ⇥ J contains the ﬁlters w j as itscolumns, the gain matrix G = diag

Original: [Equation] ⌘ , and the noise covariances are C n in =  2 in D ⇥ Dand C n out =  2 out J ⇥ J .
Revised: (  j 1 r j > 0 ) , and the noise covariances are C n in =  2 in D ⇥ D and C n out =  2 out J ⇥ J .

Original: This objective is equivalent to the formulation in [10], which assumesnormally distributed inputs and locally linear responses in order to approximate the mutual informationin a closed form.
Revised: This objective is equivalent to the formulation in [7], which assumes normallydistributed inputs and locally linear responses in order to approximate the mutual information in aclosed form.

Original: Here, we extend this model to time-varying inputs x ( t ) 2 R D representing natural videos (Figure1A-B), which are convolved with linear spatiotemporal kernels { w j ( t ) } j =1 , ··· ,J :
Revised: Here, we extend this model to time-varying inputs x ( t ) 2 R D representing natural videos (Figure1A), which are convolved with linear spatiotemporal kernels { w j ( t ) } j =1 , ··· ,J :

Original: [Equation]
Revised: [Equation]

Original: We additionally assume that the convolution kernels are separable in time and space:
Revised: We additionally assume that the convolution kernels are separable in time and space:

Original: [Equation]
Revised: [Equation]

Original: and the temporal kernels are unit-norm impulse responses taking the following parametric form:
Revised: and the temporal kernels are unit-norm impulse responses taking the following parametric form:

Original: [Equation]
Revised: [Equation]

Original:  where ↵ j , ↵ 0 j , ⌧ j > 0 , ⌧ 0 j > 0 are learnable parameters, and n 2 N is ﬁxed.
Revised: otherwise [Equation] (6) where ↵ j , ↵ 0 j , ⌧ j > 0 , ⌧ 0 j > 0 are learnable parameters and n 2 N is ﬁxed.

Original: Previous work assumedan unconstrained form for these ﬁlters, adding zero-padding before and after the model’s imageinputs to produce the characteristic shape of the temporal ﬁlters in primate midget and parasol cells[13], but this zero-padding represents a biologically implausible constraint, and the results fail tocorrectly reproduce the observed delay in retinal responses [17, 18, 19].
Revised: Previous work assumedan unconstrained form for these ﬁlters, adding zero-padding before and after the model’s imageinputs to produce the characteristic shape of the temporal ﬁlters in primate midget and parasol cells[10], but this zero-padding represents a biologically implausible constraint, and the results fail tocorrectly reproduce the observed delay in retinal responses [16, 3, 17].

Original: Rather, optimizing (2) withunconstrained temporal ﬁlters produces a ﬁlter bank uniformly tiling time (Supplementary Figure 6).
Revised: Rather, optimizing (2) withunconstrained temporal ﬁlters produces a ﬁlter bank uniformly tiling time (Supplementary Figure 1).

Original: By contrast, (6) is motivated by the arguments of [20], which showed that the optimal minimum-phasetemporal ﬁlters of retinal bipolar cells, the inputs to the RGCs, take the form
Revised: By contrast, (6) is motivated by the arguments of [18], which showed that the optimal minimum-phasetemporal ﬁlters of retinal bipolar cells, the inputs to the ganglion cells, take the form

Original: [Equation]
Revised: [Equation]

Original: (7) when !
Revised: when !

Original: ⌧ ⌧ 1 .
Revised: ⌧ ⌧ 1 .

Original: Thus, we model RGC temporal ﬁlters as a linear combination of these forms.
Revised: Thus, we model ganglion cell temporal ﬁlters as a linear combination of these forms.

Original: In practice, we take only two ﬁlters and use n = 6 rather than n = 3 , since these have beenshown to perform well in capturing observed retinal responses [19].
Revised: In practice, we take only two ﬁlters and use n = 6 rather than n = 3 , since these have been shown toperform well in capturing observed retinal responses [17].

Original: The results produced by moreﬁlters or different exponents are qualitatively unchanged (Supplementary Figure 8).
Revised: The results produced by more ﬁlters ordifferent exponents are qualitatively unchanged (Supplementary Figure 2).

Original: For training onvideo data, we use discrete temporal ﬁlters and convolutions with
Revised: Likewise, for training onvideo data, we use discrete temporal ﬁlters and convolutions with

Original: [-P T  1 t =
-]  j [ t ] 2 = 1 .
Revised: [-P T  1 t =
-][Equation]  j [ t ] 2 = 1 .

Original: Finally, whileunconstrained spatial kernels w j converge to characteristic center-surround shapes under optimizationof (2) (Figure 1C), for computational efﬁciency and stability, we parameterized these ﬁlters using aradially-symmetric difference of Gaussians
Revised: Finally, whileunconstrained spatial kernels w j converge to characteristic center-surround shapes under optimizationof (2) (Figure 1B), for computational efﬁciency and stability, we parameterized these ﬁlters using aradially-symmetric difference of Gaussians

Original: [Equation]
Revised: [Equation]

Original: where r measures the spatial distance to the center of the RF.
Revised: where r measures the spatial distance to the center of the receptive ﬁeld.

Original: The parameters a j , b j , c j determining thespatial kernel shape, as well as the center location, are different for each RGC j , allowing individual RGCs to have differentRF sizes and center-surround strengths.
Revised: The parameters a j , b j , c jdetermining the spatial kernel shape, as well as the center location, are different for each neuronj , allowing individual neurons to have different sizes and center-surround strengths.

Original: The result of optimizing (2) usingthese forms is a set of spatial and temporal kernels (Figure 1D-E) that replicate experimentallyobserved shapes and spatial RF tiling.
Revised: The result ofoptimizing (2) using these forms is a set of spatial and temporal kernels (Figure 1C-D) that replicateexperimentally-observed shapes and spatial receptive ﬁelds tiling.

Revised:  Mosaic formation as a function of channel capacity

Original: Before presenting results from our numerical experiments optimizing the model (2, 3), we beginbyderiving intuitions about its behavior by studing the case of linear ﬁlters analytically.
Revised: We begin by analyzing a linear version of the model (2) in the continuum limit, developing intuitionsfor the effects of the number of neurons on mosaic formation.

Original: That is, Weassume a single gain  for all cells,  no bias ( ✓ = 0 ), and a linear transfer function ⌘ ( y ) =
Revised: We assume a single gain  for all cells, [Figure] no bias ( ✓ = 0 ), and a linear transfer function ⌘ ( y ) =

Original: y .
Revised: y .

Original: As wewill see, this linear analysis correctly predicts the same types of mosaic formation and ﬁlling observedin the full nonlinear model. Here, we sketch the main results, deferring full details to Appendix A.
Revised: Again, we follow [5, 7, 8, 11] in optimizingmutual information, though we obtain results in qualitative agreement with [10], which optimizedreconstruction error. We assume an inﬁnite retina on which receptive ﬁelds form mosaics describedby a Bravais lattice R with principal vectors ( a 1 , a 2 ) and reciprocal vectors ( b 1 , b 2 ) in a dual lattice G with a i · b j = 2 ⇡ ij .

Original: For analytical simplicity, we begin by assuming an inﬁnite retina on which RFs form mosaicsdescribed by a regular lattice.
Revised: Put another way, if the original mosaic in image space is periodic under shiftsby elements of R , its Fourier representation is redundant under shifts in G .

Original: Under these conditions, we can  write the log determinants in (2) asintegrals and optimize over the unnormalized ﬁlter v ⌘  w subject to a power constraint:
Revised: Under these conditions,we can then write the log determinants in (2) as integrals and optimize over the unnormalized ﬁlterv ⌘  w subject to a power constraint (Appendix A):

Original: [Equation]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where C x ( k ) is the Fourier transform of
Revised: where C x ( k ) is the Fourier transform of

Original: the stationary image covariance C x
Revised: the stationary image covariance C x

Original: ( z  z 0 ) , the integral isover all frequencies k 2 G 0 unique up to aliasing caused by the spatial regularity of the mosaic, andthe sums  over g account for aliased frequencies (Appendix A.1).
Revised: ( z  z 0 ) . Here, the integralover k is restricted to G 0 , the unit cell of the reciprocal lattice, and the sums are over reciprocallattice cells, the result of aliasing due to the repeated spatial structure in the mosaic.

Original: In [8], the range [  ⇡ , ⇡ ] is used forthe integral, corresponding to a one-dimensional lattice and units of mosaic spacing
Revised: In [5], the range[  ⇡ , ⇡ ] is used, corresponding to a one-dimensional lattice and units of mosaic spacing

Original:  z = 1 .
Revised:  z = 1 .

Original: Now, solving the optimization in (9) results in a spatial kernel with the spectral form (Appendix A.2)
Revised: Now, performing the optimization in (9) results in a spatial kernel with the spectral form (full detailsin Appendix A)

Original: [Equation]
Revised: [Equation]

Original: where k = k k k and ⌫ is chosen to enforce the constraint on total power.
Revised: where k = k k k and ⌫ is chosen to enforce the constraint on total power.

Original: This is exactly the solutionfound in [8], linking it (in the linear case) to the model of [10, 11].
Revised: This is exactly the solutionfound in [5], linking it (in the linear case) to the model of [7, 8].

Original: Note, however, that (10) is onlynonzero within G 0 , since that RF spacing sets an upper limit on the passband of the resulting ﬁlters.
Revised: Note, however, that (10) is onlynonzero within the unit cell of the reciprocal lattice, indicating that receptive ﬁeld spacing sets anupper limit on the passband of the resulting ﬁlters [10].

Revised: As we argue below, it is this restriction thatcritically determines the encoding tradeoffs that result in new mosaic formation as new receptiveﬁelds are added.

Original: The generalization of this formulation to the spacetime case is straightforward.
Revised: The generalization of this formulation to the spacetime case is straightforward.

Original: Given a spacetimestationary image spectrum C x
Revised: Given a spacetimestationary image spectrum C x

Original: ( z  z 0 , t  t 0 ) and radially-symmetric, causal ﬁlter w ( z , t ) , the sameinﬁnite retina limit as above requires calculating determinants across both neuronsi, j and time pointst, t 0 of matrices with entries of the form
Revised: ( z  z 0 , t  t 0 ) and radially-symmetric, causal ﬁlter w ( z , t ) in thecontinuum limit, calculating the information present in ﬁring rates r j ( t ) across both neurons and time [Equation] [Figure] requires computing log determinants of matrices with entries of the form

Original: [Equation]
Revised: [Equation]

Original: Again, such matrices can be diagonalized in the Fourier basis, with the result that the optimalspacetime ﬁlter once again takes the form (10) with the substitutions v ( k ) !
Revised: Again, such matrices can be diagonalized in the Fourier basis, with the result that we again obtain(9) and (10) under the substitutions v ( k ) !

Original: v ( k, ! )
Revised: v ( k, ! )

Original: , C x ( k ) ! C x ( k, ! ) 
Revised: , C x ( k ) ! C x ( k, ! ) .

Original: (Appendix A.3). Figure 2A depicts the frequency response of this ﬁlter in d = 1spatialdimensions, with corresponding spatial and temporal sections plotted in Figures 2B-C.
Revised: Figure 2 depicts thecorresponding spatial and minimum-phase temporal ﬁlters for this solution in d = 1 spatial dimension. As we will argue below, division of this single ﬁlter into multiple specialized cell types as mosaicdensity increases results in the emergence of multiple cell types roughly corresponding to the dots in Figure 2A. A detailed derivation of these results is presented in Appendix A.

Original: 3.2 Multiple cell types and the effects of channel capacity
Revised: 3.2 Multiple mosaics

Original: Up to this point, we have onlyconsidered a singletype of ﬁlter  v ( k, ! ) , corresponding to a singlecell type.
Revised: Of course, the formulation above assumes only a single ﬁlter type, spatially translated by the latticevectors that deﬁne the mosaic. More generally, for multiple ﬁlter types, calculating the determinantsin (2) involves not only terms of the form (11) but also cross-terms involving v ( k, ! ) from differentmosaics. Here, we argue that such cross-terms vanish at the optimum, yielding an informationobjective that is equivalent to a sum of individual mosaic terms like (9). This is because (Appendix

Original: However, multiple cell types might increase the coding efﬁciency of the entire retinaif they specialize, devoting their limited energy budget to non-overlapping regions of frequencyspace.
Revised: A), our power constraint in (9) remains unchanged under an orthogonal transformation of the ﬁlters,while information is subadditive when cross-terms between ﬁlters are nonzero.

Original: Indeed, optimal encoding in the multi-cell-type case selects ﬁlters v and [-v 
-]
Revised: Thus, we can alwaysincrease information by choosing a ﬁlter basis orthogonal with respect to both C x and [-v 
-]C in .

Original:  that satisfyv ⇤ ( k, ! ) 
Revised: Thislogic is similar to the argument of [10], where it was reconstruction performance that was equivalentunder transformations of the ﬁlters but costs were superadditive. However, this reasoning does not fully ﬁx the choice of independent ﬁlters, since there are multipleways to make cross-terms in the determinants vanish. For example, in the discrete (pixel, frame)formulation of the problem, the generalized eigenvectors of ( C x , C in ) represent a special solution(the one that fully diagonalizes both matrices). Yet these ﬁlters need not match the optimal form (10).

Revised: But there is an alternative solution that does respect the optimal ﬁlter shapes: choosing spectrallydisjoint ﬁlters with v ⇤ ( k, ! )

Original: v 0 ( k, ! )
Revised: v 0 ( k, ! )

Original: = {+
 for different mosaics deﬁned by v and v+} 0 , corresponding encoding independent visual information (Appendix A.4).
Revised: = {+
 for different mosaics deﬁned by v and v+} 0 .

Original: This result naturally raises two questions: First, how many ﬁlter types are optimal? And second, howshould a given budget of J RGCs be allocated across multiple ﬁlter types?
Revised: In [10], theauthors considered a form of this with spatial bandwidth partitioning ( v ⇤ ( k ) v 0 ( k ) = 0 ), but as weshow below, this special case encounters a limit beyond the ﬁrst two mosaics, when the spatialpassband spectra of new ﬁlters substantially overlap.

Original: As detailed in Appendix A.5, we can proceed by analyzing the case of a ﬁnite retina in the Fourier domain, approximating theinformation encoded by a mosaic of J RGCs with spatial ﬁlters given by (10) and nonoverlapping [Figure]
Revised: Rather, the more general solution is that mosaicﬁlters are band-limited in both space and time, arranging themselves to tile minimally overlappingregions with highest power in the ( k, ! ) plane (Appendix A).

Original: C
Revised: 3.3 Effects of channel capacity

Original: [-[Figure] For both plots, A = 100 ,  in = 0 . 4 ,  out = 1 . 25 , and log 10 ! 
 = 1 . 5 , 1 . 52 , 1 . 54 , 1 . 56 , 1 . 6 . Details-] of calculations in Appendix A.5.
Revised: [-[Figure] For both plots, A = 100 ,  in = 0 . 4 ,  out = 1 . 25 , and log 10 ! 
 = 1 . 5 , 1 . 52 , 1 . 54 , 1 . 56 , 1 . 6 . Details-]How, then, does this picture change as the channel capacity of the system is increased by addingadditional neurons?

Original: bandpass temporal ﬁlters that divide the available spectrum (e.g., Figure 2B, C). Following [21], weapproximate the correlation spectrum of images by the factorized power law C x ( k, ! ) '
Revised: For a ﬁnite retina of size L d in d dimensions, J receptive ﬁelds lead to a density⇢ ⇠ J/L d ⇠ vol( G 0 ) and a mosaic spacing  z ⇠ ⇢  1 /d .

Original: A k ↵ ! 2 with↵ ⇡ 1 . 3 and ﬁnd that in this case, the optimal ﬁlter response exhibits two regimes as  a function ofspatial frequency (Supplementary Figure 1A): First, below k f =
Revised: Likewise, under changes in density, theprincipal vectors scale as ( a 1 , a 2 ) =

Original: A/  2 in !  1/ ↵ , the optimal ﬁlter isseparable and log-linear, and the ﬁltered image spectrum is white: [Equation]
Revised: ( ˆa 1 , ˆa 2 )  z and the reciprocal vectors of the dual lattice as( b 1 , b 2 ) =

Original: where ⌫ , the Lagrange multiplier in (9) that enforces the power constraint, scales as 1 /P for smallvalues of maximal power P and 1/P 2 for larger values (Supplementary Figure 1D). Second, fork & k f , the ﬁlter response decreases as k  ↵ / 2 until reaching its upper cutoff at k c = k f / ( ⌫ 2out ) 1 / ↵ ,with the ﬁltered image spectrum falling off at the same rate (Supplementary Figure 1B). But what do these regimes have to do with mosaic formation? The link between the two is givenby the fact that, for a ﬁnite retina with regularly spaced RFs, adding RGCs decreases the distancebetween RF centers and so increases the resolving power of the mosaic.
Revised: ( ˆb 1 , ˆb 2 ) 1  z .

Original: That is, the maximal value ofk grows roughly as k ⇠ p J in d = 2 , such that larger numbers of RGCs capture more informationat increasingly higher spatial frequencies (Supplementary Figure 1A).
Revised: In other words, increasing cell number leads to increasing density, whichleads to increases in the size of the unit cell G 0 in spatial frequency space.

Original: However, while informationgain is roughly uniform in the whitening regime, it falls off sharply for k & k f (Supplementary Figure 1C), suggesting the interpretation that the k . k f regime is a “mosaic ﬁlling” phase in whichinformation accumulates almost linearly as RFs capture new locations in visual space, while thek & k f regime constitutes a “compression phase” in which information gains are slower as RFsshrink to accommodate higher numbers (Figure 2D).
Revised: Thus, in the inﬁnite retinalimit, we consider the optimal arrangement of mosaics as density and spatial bandwidth grow.

Original: Indeed, one can derive the scaling of total information as a function of J : [Equation] ↵ [Equation]
Revised: The primary effect of this increase in density on the optimization problem (9) is to raise the cutoff onthe sum over spatial frequencies.

Original: k . k f [Equation]
Revised: That is, denser retinal mosaics offer higher spatial resolving power[10].

Original: JJ f ⌘  ↵ [Equation] [Equation] ↵ k & k f , (12) where P 0 is the power budget per RGC and J f is the RGC number corresponding to k
Revised: Moreover, this change in bandwidth changes the shape (much less so the size) of the optimalﬁlters.

Original: = k f . Thus,mosaic ﬁlling exhibits diminishing marginal returns (Figure 2E), such that new cell types are favoredwhen the marginal gain forgrowing mosaics with lower temporal frequency drops below the gainfrom initiating a new cell type specialized for higher temporal frequencies. Moreover, the differencebetween these gain curves implies that new RFs are not added to all mosaicsat equal rates, but inproportion to their marginal information (Figure 2F).
Revised: Yet increases in high-frequency cutoff for a single ﬁlter type result in diminishing returns interms of information. However, as detailed above, adding ﬁlters covering non-overlapping spectralwindows allows mosaics to specialize to different regions of spatial and temporal frequency, offeringgains in information as density (and thus total spatial frequency bandwidth) increases.

Original: As we demonstrate in the next section, thesefeatures of cell type and mosaic formation continue to hold in the full nonlinear model in simulation.
Revised: As we showbelow, these results continue to hold in the nonlinear, discretized case.

Original:  Experiments
Revised:  Experiments

Original: We analyzed the characteristics of the optimal spatiotemporal RFs obtained from the model (2, 3)trained on videos from the Chicago Motion Database [22].
Revised: We analyzed the characteristics of the optimal spatiotemporal RFs obtained from the model (2, 3)trained on videos from the Chicago Motion Database [14].

Revised: We used square-shaped input patches x ( t )of size D = 18 2 = 324 pixels unless otherwise speciﬁed.

Revised: We applied a circular mask on the inputpatches following [11], which effectively constrained the spatial kernels within a circle bounded bythe input square.

Revised: For computational efﬁciency in large-scale experiments in Sections 4.2 and 4.3, weapproximated the input data distribution as a multivariate Gaussian matched to the statistics of thenatural image data set estimated using one million samples of video patches.

Revised: Input patches consistedof either T = 10 or 20 frames sliced from 30-fps videos, and for computational efﬁciency, we used“valid”-mode convolutions with temporal kernels of the same size T , so each neuron yielded a singleﬁring rate value r j for each input.

Revised: Models using larger convolutions, capturing information acrossboth neurons and time, produced similar results.

Original: Model parameters for spatialkernels,temporal kernels,  and the nonlinearities were jointly optimized using Adam [23] to maximize (2)subject to the mean ﬁring rate constraint (3) using the augmented Lagrangian method with thequadratic penalty ⇢ = 1 [24].
Revised: Model parameters for spatial kernels, temporalkernels, and the nonlinearities were jointly optimized using Adam [19] to maximize (2) subject to themean ﬁring rate constraint (3), enforced using the augmented Lagrangian method with the quadraticpenalty ⇢ = 1 [20].

Original: Further technical details of  model training are in Appendix E.
Revised: Further technical details of analysis and model training are in Appendix E.

Revised: 4.1 Natural video statistics affect learned receptive ﬁelds

Original: As previously noted, the power spectral density of natural videos  can be well approximated by aproduct of spatial and temporal power-law densities, implying an anticorrelation between high spatialand temporal frequency content [21].
Revised: As previously noted, the power spectral density of natural videos is can be well approximated by aproduct of spatial and temporal power-law densities, implying an anticorrelation between high spatialand temporal frequency content [21].

Original: Supplementary Figure 4 shows the data spectrum of the videosin our experiments is also well-approximated by separable power-law ﬁts.

Original: To examine the effect ofthese statistics on the learned RFs, we divided the dataset into four progressively smaller subsetsby the proportion of their temporal spectral content below 3 Hz, their spectral attenuation .
Revised: To examine the effect of these statistics on the learned RFs,we divided the dataset into four progressively smaller subsets by the proportion of their temporalspectral content below 3 Hz, their spectral attenuation .

Original: Usingvalues of 70%, 80%, and 90% then yielded a progression of datasets ranging from most videos toonly the slowest videos (Supplementary Figure 5A, B).
Revised: Using values of 70%, 80%, and 90% thenyielded a progression of data sets ranging from most videos to only the slowest videos (Figure 3A, B).

Original: Indeed, when the model was trained on theseprogressively slower data subsets, it produced only temporal smoothing ﬁlters, whereas the samemodel trained on all videos produced a variety of “fast” temporal ﬁlter types (Supplementary Figure5C).
Revised: As shown in Figure 3C, when the model was trained on these progressively slower data subsets, itproduced only temporal smoothing ﬁlters, whereas the same model trained on all videos produced avariety of “fast” temporal ﬁlter types.

Original: We also note that these experiments used unconstrained spatial kernels in place of (8), yetstill converged on spatial RFs with typical center-surround structure as in [10, 15, 14].
Revised: We also note that these experiments used unconstrianed spatialkernels in place of (8), yet still converged on spatial RFs with typical center-surround structure as in[7, 12, 11].

Original: Thus, thesepreliminary experiments suggest that the optimal encoding strategy—in particular, the number ofdistinct cell types found—depends critically on the statistics of the video distribution to be encoded.

Original: 4.1 Mosaics ﬁll in order of temporal frequency
Revised: 4.2 Mosaics ﬁll in order of temporal frequency

Original: As the number of RGCs available to the model increased, we observed the formation of new celltypes with new spectral properties (Figure 3).
Revised: As the number of neurons available to the model increased, we observed the formation of newmosaics with new spectral properties (Figure 4).

Original: We characterized the learned ﬁlters for each RGCin terms of their spectral centroid, deﬁned as the center of mass of the Fourier (spatial) or Discrete
Revised: We characterized each neuron’s learned ﬁlters interms of their spectral centroid, deﬁned as the center of mass of the Fourier (spatial) or Discrete

Original: Cosine (temporal) transform.
Revised: Cosine (temporal) transform.

Original: Despite the fact that each model RGC was given its own spatialand temporal ﬁlter parameters (8, 6), the learned ﬁlter shapes strongly clustered, forming mosaicswith nearly uniform response properties (Figure 3A–C).
Revised: Despite the fact that each model neuron was given its own spatial andtemporal ﬁlter parameters (8, 6), neurons’ learned ﬁlter shapes strongly clustered, forming mosaicswith nearly uniform response properties (Figure 4A–C).

Original: Critically, the emergence of new cell typesshifted the spectral responses of previously established ones, with new cell types compressing thespectral windows of one another as they further specialized.
Revised: Critically, the emergence of new mosaicsshifted the spectral responses of previously established ones, with new mosaics compressing the [Figure] A B

Original: Moreover, mosaic density increased withincreasing RGC number, shifting the centroids of early mosaics toward increasingly higher spatialfrequencies.
Revised: C (1) All videos(2) (3)(4) [Figure] All videos Slow videos

Revised: [Figure]

Revised: produced only smoothing kernels, while training spectral windows of one another as they further specialized.

Revised: Moreover, mosaic density increased withincreasing neuron number, shifting the centroids of early mosaics toward increasingly higher spatialfrequencies.

Original: This is also apparent in the forms of the typical learned ﬁlters and their power spectra:new ﬁlters selected for increasingly high-frequency content in the temporal domain (Figure 3D).
Revised: This is also apparent in the forms of the typical learned ﬁlters and their power spectra:new ﬁlters selected for increasingly high-frequency content in the temporal domain (Figure 4D).

Original: We likewise analyzed the coverage factors of both individual mosaics and the entire collection, deﬁnedas the proportion of visual space covered by the learned RFs.
Revised: We likewise analyzed the coverage factors of both individual mosaics and the entire collection,deﬁned as the proportion of visual space covered by the learned receptive ﬁelds.

Original: More speciﬁcally, we deﬁned the spatialradius of an RF as the distance from its center at which intensity dropped to 20% of its peak andused this area to compute a coverage factor, the ratio of total RF area to total visual space ( ⇡ / 4 of thesquare’s area due to circular masking).
Revised: More speciﬁcally,we deﬁned the spatial radius of an RF as the distance from its center at which intensity dropped to20% of its peak and used this area to compute a coverage factor, the ratio of total RF area to totalvisual space ( ⇡ / 4 of the square’s area due to circular masking) to obtain the coverage factor.

Original: Since coverage factors depend not simply on RGC number but on RF density, they provide an alternative measure of the effective number of distinct cell typeslearned by the model.
Revised: Sincecoverage factors depend not simply on neuron number but on RF density, they provide an alternativemeasure of the effective number of distinct mosaics learned by the model.

Original: As Figure 3E shows, coverage increases nearly linearly with RGC number,while coverage for newly formed mosaics increases linearly before leveling off.
Revised: As Figure 4E shows,coverage increases nearly linearly with neuron number, while coverage for newly formed mosaicsincreases linearly before leveling off.

Original: In other words,new cell types initially increase coverage of visual space by adding new RFs, but marginal gains incoverage diminish as density increases.
Revised: In other words, new mosaics initially increase coverage ofvisual space by adding new RFs, but marginal gains in coverage diminish as density increases.

Original: In all cases, the model dynamically adjusts the number oflearned cell types and the proportion of RGCs assigned to them as channel capacity increases.
Revised: In allcases, the model dynamically adjusts the number of learned RF types and the proportion of neuronsassigned to them as channel capacity increases.

Original: 4.2 Phase changes in mosaic arrangement
Revised: 4.3 Phase changes in mosaic arrangement

Original: In addition to retinal organization at the level of mosaics, a pair of recent papers reported bothexperimental [15] and theoretical [14] evidence for an additional degree of freedom in optimizinginformation encoding: the relative arrangement of ON and OFF mosaics.
Revised: In addition to retinal organization at the level of mosaics, a pair of recent papers reported bothexperimental [12] and theoretical [11] evidence for an additional degree of freedom in optimizinginformation encoding: the relative arrangement of ON and OFF mosaics.

Original: Jun et al.
Revised: Jun et al.

Original: studied this for thecase of natural images in [14], demonstrating that the optimal conﬁguration of ON and OFF mosaicsis alignment (RFs co-located) at low output noise levels and anti-alignment (OFF RFs between ON
Revised: studied this for thecase of natural images in [11], demonstrating that the optimal conﬁguration of ON and OFF mosaicsis alignment (RFs co-located) at low output noise levels and anti-alignment (OFF RFs between ON

Original: RFs and vice-versa) under higher levels of retinal output noise.
Revised: RFs and vice-versa) under higher levels of retinal output noise.

Original: Moreover, this transition is abrupt,constituting phase change in optimal mosaic arrangement.
Revised: Moreover, this transition is abrupt,constituting phase change in optimal mosaic arrangement.

Original: We thus asked whether learned  mosaics  exhibited a similar phase transition  for natural video encoding.
Revised: We thus asked whether learned spatiotemporal mosaics in the space-time optimization exhibited asimilar phase transition and how the characteristics of this transition might vary for different mosaics.

Original: To do so, following [14], we repeatedly optimized a smallmodel ( J = 14 , 7 ON, 7 OFF)  for multiplelearned ﬁlter types while systematically varying levels of input and output noise.
Revised: To do so, following [11], we repeatedly optimized a small ( J = 14 , 7 ON, 7 OFF) model for multiplelearned ﬁlter types while systematically varying levels of input and output noise.

Original: In each case, one
Revised: In each case, one

Original: ON-OFF pair was ﬁxed at the center of the space, while the locations of the others were allowed tovary.
Revised: ON-OFF pair was ﬁxed at the center of the space, while the locations of the others were allowed tovary.

Original: We used RF size D = 8 2 pixels for Slow and D = 12 2 for FastA and FastB to allow the sizeof spatial kernels to be similar to those of the previous experiments, and we imposed the additionalconstraint that the shape parameters a j , b j , and c j in (8) are shared across RGCs.
Revised: We used receptive ﬁeld size D = 8 2 pixels for Slow and D = 12 2 for FastA and FastB to allowthe size of spatial kernels to be similar to those of the previous experiments, and we imposed theadditional constraint that the shape parameters a j , b j , and c j in (6) be same across neurons.

Original: Under these conditions, the six free pairs of RFs converged to either aligned (overlapping) oranti-aligned (alternating) positions along the edges of the circular visual space, allowing for astraightforward examination of the effect of input and output noises on mosaic arrangement.
Revised: Under these conditions, the six free pairs of RFs converged to either aligned (overlapping) oranti-aligned (alternating) positions along the edges of the circular visual space, allowing for a straight- forward examination of the effect of input and output noises on mosaic arrangement.

Original: Figure4A-C shows that the phase transition boundaries closely follow the pattern observed in [14]: increasingoutput noise shifts the optimal conﬁguration from alignment to anti-alignment.
Revised: Figure 5A–Cshows that the phase transition boundaries closely follow the pattern observed in [11]: increasingoutput noise shifts the optimal conﬁguration from alignment to anti-alignment.

Original: Moreover, for eachof the tested ﬁlters, increasing input noise discourages this transition.
Revised: Moreover, for eachof the tested ﬁlters, increasing input noise discourages this transition.

Original: This effect also follows fromthe analysis presented in [14], since higher input noise increases coactivation of nearby pairs of RFs,requiring larger thresholds to render ON-OFF pairs approximately indpendent (Appendix B).
Revised: This effect also follows fromthe analysis presented in [11], since higher input noise increases coactivation of nearby pairs of RFs,requiring larger thresholds to render ON-OFF pairs approximately indpendent (Appendix B).

Original:  Discussion
Revised:  Discussion

Original: Related work: As reviewed in the introduction, this study builds on a long line of work usingefﬁcient coding principles to understand retinal processing.

Original: In addition, it is related to work examiningencoding of natural videos [25, 22, 16] and prediction in space-time.

Original: The most closely related workto this one is that of [13], which also considered efﬁcient coding of natural videos and consideredthe tradeoffs involved in multiple cell types.

Original: Our treatment here differs from that work in severalkey ways: First, while [13] was concerned with demonstrating that multiple cell types could provebeneﬁcial for encoding (in a framework focused on reconstruction error), that study predeterminedthe number of cell types and mosaic structure and only optimized their relative spacing.

Original: By contrast,this work is focused on how the number of cell types is dynamically determined, and how theresulting mosaics arrange themselves, as a function of the number of units available for encoding(i.e.

Original: , the channel capacity).

Original: Speciﬁcally, we follow previous efﬁcient coding models [8, 9, 10, 11] inmaximizing mutual information and do not assume an a priori mosaic arrangement, a particular cellspacing, or a particular number of cell types— all of these emerge via optimization in our formulation .

Original: Second, while the computational model of [13] optimized strides for a pair of rectangular arrays of

Original: RGCs, we individually optimize RF locations and shapes, allowing us to study changes in optimal RFsize and density as new, partial mosaics begin to form.

Original: Third, while [13] used zero-padding of naturalvideos to bias learned temporal ﬁlters toward those of observed RGCs, welink the form of temporal RFs to biophysical limits on the ﬁltering properties of bipolar cells, producing temporal ﬁlters withthe delay properties observed in real data. Finally, while [13] only considered a single noise source intheir model, we consider noise in both photoreceptor responses (input noise) and RGC responses(output noise), allowing us to investigate transitions in the optimal relative arrangement of mosaics[14, 15]. We have shown that efﬁcient coding of natural videos produces multiple cell types with complementary RF properties.
Revised: Here, we have shown that efﬁcient coding of natural movies results in the formation of multiplereceptive ﬁeld mosaics with complementary ﬁltering properties [10].

Original: In addition, we have shown for the ﬁrst time that the number and characteristicsof these cell types depend crucially on the channel capacity: the number of available RGCs.
Revised: In addition, we have shown howthe number and characteristics of these mosaics depend crucially on the channel capacity of the retinaas dictated by the number of available neurons.

Original: As newsimulated RGCs become available, they are initially concentrated into mosaics with more denselypacked RFs, improving the spatialfrequency bandwidth over which information is encoded. However,as this strategy produces diminishing returns, new cell types encoding higher-frequency temporalfeatures emerge in the optimization process.
Revised: As new neurons become available, they are initiallyconcentrated into denser RFs, improving the spatial bandwidth over which information is encoded, butas this strategy becomes less effective, new cell types encoding higher-frequency temporal featuresform.

Original: These new cell types capture information over distinctspatiotemporal frequency bands, and their formation leads to upward shifts in the spatial frequencyresponses of previously formed cell types.
Revised: These new cell types capture information over distinct spatiotemporal frequency bands, andtheir formation leads to resultant shifts in the response properties of previous cell types (up in spatialfrequency, down in temporal frequency).

Original: Moreover, pairs of ON and OFF mosaics continue toexhibit the phase transition between alignment and anti-alignment revealed in a purely spatial optimization of efﬁcient coding [14], suggesting that mosaic coordination is a general strategy forincreasing coding efﬁciency.
Revised: Moreover, these receptive ﬁeld mosaics continue to exhibitthe phase transition between alignment and anti-alignment between pairs of ON and OFF pairs studiedin [11], suggesting that mosaic coordination may prove a general strategy for increasing codingefﬁciency in the retina.

Original: Furthermore, despite the  assumptions of this model—linear ﬁltering,separable ﬁlters, ﬁring rates instead of spikes—our results are consistent with observed retinal data.
Revised: Moreover, despite the strong assumptions of our model—linear ﬁltering,separable ﬁlters, ﬁring rates instead of spikes—our results are in strong agreement with observedretinal data.

Original: For example, RGCs with small spatial RFs exhibit more prolonged temporal integration: they are alsomore low-pass in their temporal frequency tuning.

Original: Second, there is greater variability in the size andshape of spatial RFs at a given retinal location, but temporal RFs exhibit remarkably little variabilityin our simulations and in data [19].

Original: Thus, these results further testify to the power of efﬁcient codingprinciples in providing a conceptual framework for understandingthe nervous system.
Revised: Thus, these results further testify to the power of efﬁcient coding principles in providinga principled framework for understanding nervous system organization.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .
Revised: [Yes] See Section ?? .•

Original: • Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.
Revised: [No] The code and the data areproprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We address this in the Discussion.
Revised: [Yes] We address this in the Discussion.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] The focus
Revised: [No] The focus

Original: of this work is information-theoretic analyses on a simple computational model for theretina.
Revised: of this work is information-theoretic analyses on a simple computational model for the retina.

Original: As such, we do not anticipate any direct, negative societal impacts.
Revised: As such, we do not anticipate any direct, negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] Detailedtreatments are Appendices A and B.
Revised: [Yes] Detailedtreatments are Appendices A and B.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] Detailed treatments are Appendices A and B.
Revised: [Yes] Detailed treatmentsare Appendices A and B.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?
Revised: perimental results (either in the supplemental material or as a URL)?

Original: [Yes] The code
Revised: [Yes] The code

Original: and instructions to run the complete implementation of this work are included in the
Revised: and instructions to run the complete implementation of this work are included in the

Original: supplementary materials, with the necessary links to download the datasets for full
Revised: supplementary materials, with the necessary links to download the datasets for full

Original: replication.
Revised: replication.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] (and partially N/A) our work did not include data splits, and other,
Revised: [Yes] (and partially N/A) our work did not include data splits, and other,

Original: empirically chosen hyperparameters are detailed in the main text and the supplement.
Revised: empirically chosen hyperparameters are detailed in the main text and the supplement.

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [No] Our ﬁgures were either conceptual or theoretical plots or
Revised: [No] Our ﬁgures were either conceptual or theoretical plots or

Original: visualizing exemplar data, where error bars are not suitable.
Revised: visualizing exemplar data, where error bars are not suitable.

Original: (d) Did you include the total amount of compute and the type of resources used (e g.,
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,

Original: type of GPUs, internal cluster, or cloud provider)?
Revised: type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We detail the compute usage
Revised: [Yes] We detail the compute usage

Original: together with other training details in the supplement.
Revised: together with other training details in the supplement.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We primarily use the
Revised: [Yes] We primarily use the

Original: Chicago Motion Database [22] and data derived from it.
Revised: Chicago Motion Database [14] and data derived from it.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] The license is not speciﬁed in the
Revised: [No] The license is not speciﬁed in the

Original: paper and the website https://cmd.rcc.uchicago.edu , but the website’s about
Revised: paper and the website https://cmd.rcc.uchicago.edu , but the website’s about

Original: page says it is “intended for use in visual neuroscience experiments, as training data
Revised: page says it is “intended for use in visual neuroscience experiments, as training data

Original: for machine vision applications, ...”.
Revised: for machine vision applications, ...”.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: [Table]

Original: We have included zipped source code in the supplement to preserve anonymity for
Revised: We have included zipped source code in the supplement to preserve anonymity for

Original: [Table]

Original: review but will include a link to the GitHub repository in a camera-ready version.
Revised: review but will include a link to the GitHub repository in a camera-ready version.

Original: [Table]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]
Revised: [Table]

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: We did not use human data.
Revised: We did not use human data.

Original: [Table]
Revised: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]
Revised: [Table]

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: Our data consisted of images of natural scenes.
Revised: Our data consisted of images of natural scenes.

Original: [Table]
Revised: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]
Revised: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]
Revised: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: We did not use crowdsourcing or human subjects
Revised: We did not use crowdsourcing or human subjects

Original: [Table]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]
Revised: [Table]

Original: Board (IRB) approvals, if applicable?
Revised: (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: We did not use crowdsourcing or human
Revised: We did not use crowdsourcing or human

Original: [Table]

Original: subjects

Original: [Table]
Revised: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]
Revised: [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: We did not use crowdsourcing or human
Revised: We did not use crowdsourcing or human

Original: [Table]
Revised: [Table]

Original: subjects

Original: [Table]