Original: Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorization
Revised: Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorization

Original: Abstract
Revised: Abstract

Original: Recurrent neural networks (RNNs) have been shown to perform better than feedforward architectures in visual object categorization tasks, especially in challenging conditions such as cluttered images.
Revised: Recurrent neural networks (RNNs) have been shown to perform better than feedforward architectures in visual object categorization tasks, especially in challenging conditions such as cluttered images.

Original: However, little is known about the exact computational role of recurrent information ﬂow in these conditions.
Revised: However, little is known about the exact computational role of recurrent information ﬂow in these conditions.

Original: Here we test
Revised: Here we test

Original: RNNs trained for object categorization on the hypothesis that recurrence iteratively aids object categorization via the communication of category-orthogonal auxiliary variables (the location, orientation, and scale of the object).
Revised: RNNs trained for object categorization on the hypothesis that recurrence iteratively aids object categorization via the communication of category-orthogonal auxiliary variables (the location, orientation, and scale of the object).

Original: Using diagnostic linear readouts, we ﬁnd that: (a) information about auxiliary variables increases across time in all network layers, (b) this information is indeed present in the recurrent information ﬂow, and (c) its manipulation signiﬁcantly affects task performance.
Revised: Using diagnostic linear readouts, we ﬁnd that: (a) information about auxiliary variables increases across time in all network layers, (b) this information is indeed present in the recurrent information ﬂow, and (c) its manipulation signiﬁcantly affects task performance.

Original: These observations conﬁrm the hypothesis that category-orthogonal auxiliary variable information is conveyed through recurrent connectivity and is used to optimize category inference in cluttered environments.
Revised: These observations conﬁrm the hypothesis that category-orthogonal auxiliary variable information is conveyed through recurrent connectivity and is used to optimize category inference in cluttered environments.

Original:  Introduction
Revised:  Introduction

Original: While feedforward neural networks (FNNs) have demonstrated far-reaching success in the task of visual object categorization, recurrent neural networks (RNNs), inspired by the abundance and usefulness of recurrent connectivity in the primate visual system (Kar et al., 2019; Kietzmann et al., 2019), have been shown to outperform them in some settings (Spoerer et al., 2020; Kubilius et al., 2018).
Revised: While feedforward neural networks (FNNs) have demonstrated far-reaching success in the task of visual object categorization, recurrent neural networks (RNNs), inspired by the abundance and usefulness of recurrent connectivity in the primate visual system (Kar et al., 2019; Kietzmann et al., 2019), have been shown to outperform them in some settings (Spoerer et al., 2020; Kubilius et al., 2018).

Original: This advantage manifests particularly under challenging conditions such as partial object occlusion and clutter (Spoerer et al., 2017; Ernst et al., 2019).
Revised: This advantage manifests particularly under challenging conditions such as partial object occlusion and clutter (Spoerer et al., 2017; Ernst et al., 2019).

Original: However, beyond the empirical ﬁnding that recurrence can help object categorization, it remains unclear what information is conveyed by recurrent connections and what its functional role is.
Revised: However, beyond the empirical ﬁnding that recurrence can help object categorization, it remains unclear what information is conveyed by recurrent connections and what its functional role is.

Original: In contrast to FNNs, unit activations in RNNs are a function of both their input and their prior activations.
Revised: In contrast to FNNs, unit activations in RNNs are a function of both their input and their prior activations.

Original: This enables these networks to process input in time-varying, context-dependent, and conditional manner.
Revised: This enables these networks to process input in time-varying, context-dependent, and conditional manner.

Original: The usefulness of these conditional computations has been observed, for example, in networks processing natural language, in which contextual words, such as negations, can ‘steer’ a network’s trajectory in state space to process subsequent words differently (Maheswaranathan &
Revised: The usefulness of these conditional computations has been observed, for example, in networks processing natural language, in which contextual words, such as negations, can ‘steer’ a network’s trajectory in state space to process subsequent words differently (Maheswaranathan &

Original: Sussillo, 2020).
Revised: Sussillo, 2020).

Original: In the primate visual cortex, recurrence is believed to underlie computations that can beneﬁt from contextual signals, such as assigning local features to a ﬁgure or the background based on global shape consistency (Roelfsema et al., 2007; van Bergen & Kriegeskorte, 2020).
Revised: In the primate visual cortex, recurrence is believed to underlie computations that can beneﬁt from contextual signals, such as assigning local features to a ﬁgure or the background based on global shape consistency (Roelfsema et al., 2007; van Bergen & Kriegeskorte, 2020).

Original: In object categorization, contextual modulations would likely take advantage of the structure of real-world data, similar to how recurrence can exploit the part-whole hierarchy of visual shapes in ﬁgure-ground segmentation or the branching structure of phrases in natural language.
Revised: In object categorization, contextual modulations would likely take advantage of the structure of real-world data, similar to how recurrence can exploit the part-whole hierarchy of visual shapes in ﬁgure-ground segmentation or the branching structure of phrases in natural language.

Original: In particular, natural images are a function of both the categories of objects therein and other category-orthogonal auxiliary variables, such as the objects’ location, orientation or scale.
Revised: In particular, natural images are a function of both the categories of objects therein and other category-orthogonal auxiliary variables, such as the objects’ location, orientation or scale.

Original: A potential role of recurrent connectivity could be to aid the selection of information that is most relevant for object categorization, by ﬁrst extracting auxiliary variables about the object and subsequently to condition information processing on that information.
Revised: A potential role of recurrent connectivity could be to aid the selection of information that is most relevant for object categorization, by ﬁrst extracting auxiliary variables about the object and subsequently to condition information processing on that information.

Original: That is, such a mechanism could iteratively focus on the location and features corresponding to the object in the image and ﬁlter out irrelevant noise such as clutter.
Revised: That is, such a mechanism could iteratively focus on the location and features corresponding to the object in the image and ﬁlter out irrelevant noise such as clutter.

Original: That is, auxiliary, category-orthogonal information would not be discarded due to it being non-diagnostic for the category identity, but rather recurrent connectivity would use this information to guide and improve performance of the main task of object categorization (Fig. 1).
Revised: That is, auxiliary, category-orthogonal information would not be discarded due to it being non-diagnostic for the category identity, but rather recurrent connectivity would use this information to guide and improve performance of the main task of object categorization (Fig. 1).

Original: To test this hypothesis, we trained and tested multiple instances of an RNN on an object categorization task while presenting target objects in cluttered environments.
Revised: To test this hypothesis, we trained and tested multiple instances of an RNN on an object categorization task while presenting target objects in cluttered environments.

Original: We used diagnostic readouts across layers and time to characterise the presence of information related to auxiliary variables, and performed in-silico causal experiments to further elucidate their computational role in object categorization.
Revised: We used diagnostic readouts across layers and time to characterise the presence of information related to auxiliary variables, and performed in-silico causal experiments to further elucidate their computational role in object categorization.

Original:  Methods
Revised:  Methods

Original: Primary details about the network architecture and the datasets are mentioned below.
Revised: Primary details about the network architecture and the datasets are mentioned below.

Original: Please refer to the Appendix for exhaustive details.
Revised: Please refer to the Appendix for exhaustive details.

Original: 2.1 Network architecture
Revised: 2.1 Network architecture

Original: The recurrent network architecture used for training and subsequent analyses consisted of two convolutional layers and one fully connected layer, as illustrated in Fig.
Revised: The recurrent network architecture used for training and subsequent analyses consisted of two convolutional layers and one fully connected layer, as illustrated in Fig.

Original: 2A.
Revised: 2A.

Original: The architecture contained both lateral and top-down recurrent connections.
Revised: The architecture contained both lateral and top-down recurrent connections.

Original: Top-down connections were sent from a given layer to all the previous layers, including the input.
Revised: Top-down connections were sent from a given layer to all the previous layers, including the input.

Original: The RNN was unrolled for 4 timesteps.
Revised: The RNN was unrolled for 4 timesteps.

Original: The activations of the fully-connected layers at the end of each timestep were concatenated and mapped to the classiﬁcation output with a fully-connected layer.
Revised: The activations of the fully-connected layers at the end of each timestep were concatenated and mapped to the classiﬁcation output with a fully-connected layer.

Original: 2.2 Combining the feedforward and recurrent information ﬂow
Revised: 2.2 Combining the feedforward and recurrent information ﬂow

Original: We operationalize the effect of recurrent connectivity in terms of gain modulation (multiplicative interaction).
Revised: We operationalize the effect of recurrent connectivity in terms of gain modulation (multiplicative interaction).

Original: For this, we summed the incoming lateral and top-down recurrent activations, and multiplied them, element-wise, with the feedforward activations.
Revised: For this, we summed the incoming lateral and top-down recurrent activations, and multiplied them, element-wise, with the feedforward activations.

Original: We refer to the summed recurrent activations fed to a layer as the recurrent information ﬂow to that layer.
Revised: We refer to the summed recurrent activations fed to a layer as the recurrent information ﬂow to that layer.

Original: For a layer l at timestep t the activation was given by:
Revised: For a layer l at timestep t the activation was given by:

Original: [Equation]
Revised: [Equation]

Original: Recurrent information ﬂow to layer l at timestep t
Revised: Recurrent information ﬂow to layer l at timestep t

Original: [Equation]
Revised: [Equation]

Original: where C ff , C lat and C td correspond to the feedforward, lateral and top-down transformations corresponding to layer l respectively
Revised: where C ff , C lat and C td correspond to the feedforward, lateral and top-down transformations corresponding to layer l respectively

Original: Activations at the Input layer were clipped between 0 and 1 , to ensure the modulated input images always remained in the same range across timesteps.
Revised: Activations at the Input layer were clipped between 0 and 1 , to ensure the modulated input images always remained in the same range across timesteps.

Original: 2.3 Dataset and task
Revised: 2.3 Dataset and task

Original: In order to study the effects of recurrent information ﬂow, we trained our RNNs under challenging visual conditions.
Revised: In order to study the effects of recurrent information ﬂow, we trained our RNNs under challenging visual conditions.

Original: Speciﬁcally, we generated a dataset in which target objects were manipulated according to a number of category-orthogonal variables.
Revised: Speciﬁcally, we generated a dataset in which target objects were manipulated according to a number of category-orthogonal variables.

Original: Object categories were taken from the
Revised: Object categories were taken from the

Original: MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017) datasets - corresponding tocategories in total.
Revised: MNIST (LeCun et al., 1998) and Fashion-MNIST (Xiao et al., 2017) datasets - corresponding tocategories in total.

Original: The objects were randomly varied in location, orientation and scale.
Revised: The objects were randomly varied in location, orientation and scale.

Original: In addition, structured clutter, i.e. randomly sampled fragments of other objects in the dataset, was added to the images.
Revised: In addition, structured clutter, i.e. randomly sampled fragments of other objects in the dataset, was added to the images.

Original: Each of the auxiliary variables (horizontal and vertical locations, orientation and scale;
Revised: Each of the auxiliary variables (horizontal and vertical locations, orientation and scale;

Original: Fig.
Revised: Fig.

Original: 2B) had two possible values.
Revised: 2B) had two possible values.

Original: In the results, the measures for the vertical and horizontal locations are averaged and summarized as one auxiliary variable ( location ).
Revised: In the results, the measures for the vertical and horizontal locations are averaged and summarized as one auxiliary variable ( location ).

Original: The RNN was trained for 20 -way classiﬁcation.
Revised: The RNN was trained for 20 -way classiﬁcation.

Original: 5 instances of the RNN were trained from random initializations to assess the robustness of our ﬁndings (Mehrer et al., 2020).
Revised: 5 instances of the RNN were trained from random initializations to assess the robustness of our ﬁndings (Mehrer et al., 2020).

Original:  Results
Revised:  Results

Original: The networks’ accuracy (averaged across 5 trained instances of the network) on a test set ( 10 ,images) was 81 .
Revised: The networks’ accuracy (averaged across 5 trained instances of the network) on a test set ( 10 ,images) was 81 .

Original: 2% (chance performance for 20 -way classiﬁcation is 5% ), implying that the network successfully learned to classify the images in the dataset.
Revised: 2% (chance performance for 20 -way classiﬁcation is 5% ), implying that the network successfully learned to classify the images in the dataset.

Original: Next, we analyzed the networks’ activations to assess whether auxiliary variables were extracted or suppressed, and how those variables affected information ﬂow and network performance.
Revised: Next, we analyzed the networks’ activations to assess whether auxiliary variables were extracted or suppressed, and how those variables affected information ﬂow and network performance.

Original: 3.1 Category-orthogonal information is expressed incrementally over time in the RNN
Revised: 3.1 Category-orthogonal information is expressed incrementally over time in the RNN

Original: To test for the presence of category-orthogonal information in the RNN activation patterns across layers and timesteps, we trained linear diagnostic readouts targeting auxiliary variables (i.e. predicting the location, scale, and orientation of the object), in addition to determining the presence of category information in the network activations.
Revised: To test for the presence of category-orthogonal information in the RNN activation patterns across layers and timesteps, we trained linear diagnostic readouts targeting auxiliary variables (i.e. predicting the location, scale, and orientation of the object), in addition to determining the presence of category information in the network activations.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: 3A.
Revised: 3A.

Original: First, our analyses revealed that category information increased with both layer depth (related to hierarchical processing) and time, indicating that the recurrent information ﬂow carried information relevant to categorization at each timestep.
Revised: First, our analyses revealed that category information increased with both layer depth (related to hierarchical processing) and time, indicating that the recurrent information ﬂow carried information relevant to categorization at each timestep.

Original: Importantly, the values of all auxiliary variables could be decoded in all layers and at all timesteps, with increasing performance with increasing timesteps (averaged across layers, network instances, and auxiliary variables: 65 . 2% , 73 . 0% , 82 .
Revised: Importantly, the values of all auxiliary variables could be decoded in all layers and at all timesteps, with increasing performance with increasing timesteps (averaged across layers, network instances, and auxiliary variables: 65 . 2% , 73 . 0% , 82 .

Original: 9% for timesteps 1-4).
Revised: 9% for timesteps 1-4).

Original: In summary, our analyses revealed that, instead of ﬁltering out the category-orthogonal information, the RNNs extracted auxiliary variable information incrementally across time.
Revised: In summary, our analyses revealed that, instead of ﬁltering out the category-orthogonal information, the RNNs extracted auxiliary variable information incrementally across time.

Original: Importantly, auxiliary variable information (and also category information) could also be decoded increasingly well from the input image.
Revised: Importantly, auxiliary variable information (and also category information) could also be decoded increasingly well from the input image.

Original: This is due to the fact that the RNNs were set up to feed back information all the way down to the input, and as a result, the corresponding feedback effects can be easily visualised.
Revised: This is due to the fact that the RNNs were set up to feed back information all the way down to the input, and as a result, the corresponding feedback effects can be easily visualised.

Original: As can be seen in Fig.
Revised: As can be seen in Fig.

Original: 3B, incremental decoding of auxiliary variables was accompanied by clutter-reduction over time.
Revised: 3B, incremental decoding of auxiliary variables was accompanied by clutter-reduction over time.

Original: 3.2 Recurrent information ﬂow includes category-orthogonal information that guides subsequent network inference
Revised: 3.2 Recurrent information ﬂow includes category-orthogonal information that guides subsequent network inference

Original: To determine whether the auxiliary variable information was encoded in the recurrent information ﬂow, we decoded all variables at each timestep from recurrent information only (starting from the second timepoint at which recurrence comes into effect).
Revised: To determine whether the auxiliary variable information was encoded in the recurrent information ﬂow, we decoded all variables at each timestep from recurrent information only (starting from the second timepoint at which recurrence comes into effect).

Original: We found that the information about both auxiliary variables and category increased over time in the recurrent information ﬂow to all layers, consistent with the information in the layers’ activations reported earlier (see Fig. S1).
Revised: We found that the information about both auxiliary variables and category increased over time in the recurrent information ﬂow to all layers, consistent with the information in the layers’ activations reported earlier (see Fig. S1).

Original: Does successful decoding of auxiliary variables from recurrent information ﬂow imply functional impact, or is it a side-effect of layer activations with no causal role in the networks’ categorization performance?
Revised: Does successful decoding of auxiliary variables from recurrent information ﬂow imply functional impact, or is it a side-effect of layer activations with no causal role in the networks’ categorization performance?

Original: To answer this question, we conducted a perturbation analysis in which we exchanged the feedback to a given layer and timepoint with feedback extracted from another, systematically perturbed, image from the dataset (Fig. 4A), i.e. feedback signalling the wrong value of the auxiliary variable.
Revised: To answer this question, we conducted a perturbation analysis in which we exchanged the feedback to a given layer and timepoint with feedback extracted from another, systematically perturbed, image from the dataset (Fig. 4A), i.e. feedback signalling the wrong value of the auxiliary variable.

Original: As any manipulation may alter network performance, we furthermore included a control condition in which we exchanged the original feedback signal with a randomly rotated version of the perturbation vector (Fig.
Revised: As any manipulation may alter network performance, we furthermore included a control condition in which we exchanged the original feedback signal with a randomly rotated version of the perturbation vector (Fig.

Original: 4B corresponding to the systematic perturbation, see Appendix for details).
Revised: 4B corresponding to the systematic perturbation, see Appendix for details).

Original: This control perturbation signals different, but not entirely misleading, information about the auxiliary variables (as it does not consistently align with changes in any of the auxiliary variables).
Revised: This control perturbation signals different, but not entirely misleading, information about the auxiliary variables (as it does not consistently align with changes in any of the auxiliary variables).

Original: The functional importance of a variable in the recurrent information ﬂow was quantiﬁed as follows:
Revised: The functional importance of a variable in the recurrent information ﬂow was quantiﬁed as follows:

Original: ﬁrst, we contrasted the networks’ task performance following two types of recurrent information
Revised: ﬁrst, we contrasted the networks’ task performance following two types of recurrent information

Original: ﬂow perturbation: systematic and control.
Revised: ﬂow perturbation: systematic and control.

Original: That is, we computed how much more a systematic perturbation affected task accuracy compared to a control perturbation.
Revised: That is, we computed how much more a systematic perturbation affected task accuracy compared to a control perturbation.

Original: This difference was then normalised based on the original network performance.
Revised: This difference was then normalised based on the original network performance.

Original: As a result, functional importance was computed as  Accuracy control − Accuracy systematic ) /Accuracy
Revised: As a result, functional importance was computed as ( Accuracy control − Accuracy systematic ) /Accuracy

Original: original (Fig. 4C).
Revised: original (Fig. 4C).

Original: This analysis demonstrated that location information in the recurrent information ﬂow to the Input was functionally important at timesteps 3 and 4 (Fig.
Revised: This analysis demonstrated that location information in the recurrent information ﬂow to the Input was functionally important at timesteps 3 and 4 (Fig.

Original: 4D), which corresponded to the major part of the clutter-reduction observed in Fig.
Revised: 4D), which corresponded to the major part of the clutter-reduction observed in Fig.

Original: 3B.
Revised: 3B.

Original: In addition to location, information about other auxiliary object features - orientation, scale, and category - was functionally relevant in the recurrent information ﬂow to the convolutional layers.
Revised: In addition to location, information about other auxiliary object features - orientation, scale, and category - was functionally relevant in the recurrent information ﬂow to the convolutional layers.

Original: Neither the auxiliary variables nor category information in the lateral information ﬂow to FC were functionally important (data not shown).
Revised: Neither the auxiliary variables nor category information in the lateral information ﬂow to FC were functionally important (data not shown).

Original: In line with this observation, ablating that lateral connection from the trained RNN led to no performance reduction.
Revised: In line with this observation, ablating that lateral connection from the trained RNN led to no performance reduction.

Original: In summary, auxiliary variable information in the recurrent information ﬂows seems to causally guide the information processing in the network to support object categorization.
Revised: In summary, auxiliary variable information in the recurrent information ﬂows seems to causally guide the information processing in the network to support object categorization.

Original:  Discussion
Revised:  Discussion

Original: We investigated the role of recurrent information ﬂow in RNNs trained for object categorization in cluttered environments.
Revised: We investigated the role of recurrent information ﬂow in RNNs trained for object categorization in cluttered environments.

Original: We hypothesized that, to improve categorization performance, categoryorthogonal variables are extracted, rather than ﬁltered out, and subsequently used by the RNN to constrain later information processing.
Revised: We hypothesized that, to improve categorization performance, categoryorthogonal variables are extracted, rather than ﬁltered out, and subsequently used by the RNN to constrain later information processing.

Original: Consistent with this hypothesis, we found that (i) information about all auxiliary variables was present at all network layers, (ii) this information became more prominent across time, and (iii) perturbing this information in the recurrent information ﬂow signiﬁcantly reduced network performance.
Revised: Consistent with this hypothesis, we found that (i) information about all auxiliary variables was present at all network layers, (ii) this information became more prominent across time, and (iii) perturbing this information in the recurrent information ﬂow signiﬁcantly reduced network performance.

Original: The task of object categorization has traditionally been cast in terms of extracting representations invariant to all category-orthogonal variables.
Revised: The task of object categorization has traditionally been cast in terms of extracting representations invariant to all category-orthogonal variables.

Original: However, extracting auxiliary variables from visual input might be important for natural organisms, who must also be able to keep track of the location and other properties of objects in order to survive (for instance, where a predator is and whether it is asleep or awake).
Revised: However, extracting auxiliary variables from visual input might be important for natural organisms, who must also be able to keep track of the location and other properties of objects in order to survive (for instance, where a predator is and whether it is asleep or awake).

Original: This can explain the ﬁnding that primate inferior temporal cortex does contain such auxiliary variable information (Hong et al., 2016).
Revised: This can explain the ﬁnding that primate inferior temporal cortex does contain such auxiliary variable information (Hong et al., 2016).

Original: In that study, surprisingly, information about the auxiliary variables was also found in a feedforward neural network trained exclusively for object categorization.
Revised: In that study, surprisingly, information about the auxiliary variables was also found in a feedforward neural network trained exclusively for object categorization.

Original: That ﬁnding, and our current results, echo several proposals that suggest that optimally separating object categories might in fact require explicitly extracting auxiliary variables that characterize the variation of the objects in their images (DiCarlo & Cox, 2007; Patel et al., 2016).
Revised: That ﬁnding, and our current results, echo several proposals that suggest that optimally separating object categories might in fact require explicitly extracting auxiliary variables that characterize the variation of the objects in their images (DiCarlo & Cox, 2007; Patel et al., 2016).

Original: As proposed in the introduction, once auxiliary variable information has been extracted, it can be used to improve categorization performance by conditioning category inference on the values of the auxiliary variables (Fig. 1).
Revised: As proposed in the introduction, once auxiliary variable information has been extracted, it can be used to improve categorization performance by conditioning category inference on the values of the auxiliary variables (Fig. 1).

Original: RNNs are particularly suited for this, since their architecture provides separate channels for inference (feedforward information ﬂow) and conditioning (recurrent information ﬂow) - an inductive bias that matches the proposed interaction between auxiliary variables and category inference.
Revised: RNNs are particularly suited for this, since their architecture provides separate channels for inference (feedforward information ﬂow) and conditioning (recurrent information ﬂow) - an inductive bias that matches the proposed interaction between auxiliary variables and category inference.

Original: This inductive bias might be the reason recurrent architectures outperform parameter-matched feedforward architectures, particularly when the category inference is ambiguous - such as in the presence of clutter - where iterative conditioning from the auxiliary variables might be beneﬁcial (Spoerer et al., 2017; Ernst et al., 2019).
Revised: This inductive bias might be the reason recurrent architectures outperform parameter-matched feedforward architectures, particularly when the category inference is ambiguous - such as in the presence of clutter - where iterative conditioning from the auxiliary variables might be beneﬁcial (Spoerer et al., 2017; Ernst et al., 2019).

Original: Additionally, top-down recurrent connections might be advantageous by allowing the networks to condition the inference at different hierarchical levels.
Revised: Additionally, top-down recurrent connections might be advantageous by allowing the networks to condition the inference at different hierarchical levels.

Original: This might be particularly important in the case of cluttered images since the lower spatial resolution of later layers might not allow the disentanglement of the target object from the clutter, and information selection might thus be more effective at earlier layers.
Revised: This might be particularly important in the case of cluttered images since the lower spatial resolution of later layers might not allow the disentanglement of the target object from the clutter, and information selection might thus be more effective at earlier layers.

Original: Relatedly, residual blocks, a popular architectural pattern in feedforward models, might provide an inductive bias similar to recurrence (Liao & Poggio, 2016), enabling an equivalent form of iterative processing, which could explain their increased efﬁcacy in categorization (compared to vanilla feedforward models), especially for ambiguous images (Jastrz˛ebski et al., 2017).
Revised: Relatedly, residual blocks, a popular architectural pattern in feedforward models, might provide an inductive bias similar to recurrence (Liao & Poggio, 2016), enabling an equivalent form of iterative processing, which could explain their increased efﬁcacy in categorization (compared to vanilla feedforward models), especially for ambiguous images (Jastrz˛ebski et al., 2017).

Original: Another relevant inductive bias in our networks is the way in which feedforward and recurrent information interact.
Revised: Another relevant inductive bias in our networks is the way in which feedforward and recurrent information interact.

Original: We focused our main analyses on an RNN architecture with multiplicative feedback interaction.
Revised: We focused our main analyses on an RNN architecture with multiplicative feedback interaction.

Original: In an RNN with additive interactions, we observed similar results in terms of the decodability of auxiliary variables across time and layers (see Appendix).
Revised: In an RNN with additive interactions, we observed similar results in terms of the decodability of auxiliary variables across time and layers (see Appendix).

Original: The usefulness of the extraction of auxiliary variables therefore seems to be independent of the mode of interaction.
Revised: The usefulness of the extraction of auxiliary variables therefore seems to be independent of the mode of interaction.

Original: However, with the additive interactions, we could not observe similarly strong clutter reduction in the input over time.
Revised: However, with the additive interactions, we could not observe similarly strong clutter reduction in the input over time.

Original: The solution found by the multiplicative interactions for combining recurrent and feedforward information was therefore different, aligning more closely with our intuitive notion of information selection.
Revised: The solution found by the multiplicative interactions for combining recurrent and feedforward information was therefore different, aligning more closely with our intuitive notion of information selection.

Original: An exact characterization of this distinction between the two types of interactions is beyond the scope of this study.
Revised: An exact characterization of this distinction between the two types of interactions is beyond the scope of this study.

Original: However, it is useful to note that multiplicative interactions have been proposed to be a useful inductive bias for conditional computations (Jayakumar et al., 2020), so characterizing their unique role in how information can be conditioned with auxiliary variables for object categorization, is a promising avenue for future research.
Revised: However, it is useful to note that multiplicative interactions have been proposed to be a useful inductive bias for conditional computations (Jayakumar et al., 2020), so characterizing their unique role in how information can be conditioned with auxiliary variables for object categorization, is a promising avenue for future research.

Original: A Appendix A.1 Dataset generation
Revised: A Appendix A.1 Dataset generation

Original: Each image in our dataset was 100 × 100 pixels in size.
Revised: Each image in our dataset was 100 × 100 pixels in size.

Original: Each image contained one intact target object (originally 28 × 28 px), rotated and scaled.
Revised: Each image contained one intact target object (originally 28 × 28 px), rotated and scaled.

Original: 7 other scrambled objects comprised the clutter, each of which were also rotated and scaled, then divided into 9 square blocks, and permuted randomly.
Revised: 7 other scrambled objects comprised the clutter, each of which were also rotated and scaled, then divided into 9 square blocks, and permuted randomly.

Original: The image was divided into four quadrants, corresponding to the two horizontal and two vertical locations.
Revised: The image was divided into four quadrants, corresponding to the two horizontal and two vertical locations.

Original: Each quadrant contained two objects - one overlaid on top of the other.
Revised: Each quadrant contained two objects - one overlaid on top of the other.

Original: One of the quadrants contained the intact object which was always overlaid on the scrambled object, to ensure it wouldn’t
Revised: One of the quadrants contained the intact object which was always overlaid on the scrambled object, to ensure it wouldn’t

Original: The horizontal and vertical location of the target object was chosen randomly for each image, meaning that the object would be located at the center of one of the four quadrants, jittered both horizontally and vertically between − 2 . 5 and 2 .
Revised: The horizontal and vertical location of the target object was chosen randomly for each image, meaning that the object would be located at the center of one of the four quadrants, jittered both horizontally and vertically between − 2 . 5 and 2 .

Original: 5 pixels (uniform distribution).
Revised: 5 pixels (uniform distribution).

Original: The orientation of the objects (both target and scrambled) was randomly chosen to be either 30 degrees clockwise or counterclockwise, with a uniformly distributed jitter of ± 5 pixels.
Revised: The orientation of the objects (both target and scrambled) was randomly chosen to be either 30 degrees clockwise or counterclockwise, with a uniformly distributed jitter of ± 5 pixels.

Original: Scaling was either 0 .
Revised: Scaling was either 0 .

Original: 9 or 1 .
Revised: 9 or 1 .

Original: 5 times the original object size, with a jitter of ± 0 .
Revised: 5 times the original object size, with a jitter of ± 0 .

Original: The split between train, validation and test sets was done by randomly drawing images only from the corresponding splits of the original datasets.
Revised: The split between train, validation and test sets was done by randomly drawing images only from the corresponding splits of the original datasets.

Original: The datasets provided the 110 , 000 possible source images for the training set (MNIST and Fashion-MNIST combined), 10 , 000 for the validation set and 20 , 000 for the test set.
Revised: The datasets provided the 110 , 000 possible source images for the training set (MNIST and Fashion-MNIST combined), 10 , 000 for the validation set and 20 , 000 for the test set.

Original: Training images were generated on the ﬂy for each training batch.
Revised: Training images were generated on the ﬂy for each training batch.

Original: The test set contained 10 , 000 images.
Revised: The test set contained 10 , 000 images.

Original: A.2 Architecture details
Revised: A.2 Architecture details

Original: The RNN contained two convolutional layers, both with 5 × 5 kernels, stride of 1 , no padding, and with 8 and 16 channels respectively, and one fully-connected layer with 128 units.
Revised: The RNN contained two convolutional layers, both with 5 × 5 kernels, stride of 1 , no padding, and with 8 and 16 channels respectively, and one fully-connected layer with 128 units.

Original: 3 × 3 max pooling was applied after each convolutional layer.
Revised: 3 × 3 max pooling was applied after each convolutional layer.

Original: The lateral recurrent connections from each convolutional layer to itself were convolutional layers with the same number of input and output channels ( 8 and 16 ), kernel size 5 × 5 , stride 1 and padding 2 .
Revised: The lateral recurrent connections from each convolutional layer to itself were convolutional layers with the same number of input and output channels ( 8 and 16 ), kernel size 5 × 5 , stride 1 and padding 2 .

Original: The lateral connection for the fully connected layer was a fully connected layer with 128 units.
Revised: The lateral connection for the fully connected layer was a fully connected layer with 128 units.

Original: The top-down connections from convolutional layers to the input and other convolutional layers were transposed convolutional layers, upsampling to the size of the target feature
Revised: The top-down connections from convolutional layers to the input and other convolutional layers were transposed convolutional layers, upsampling to the size of the target feature

Original: Conv kernel
Revised: Conv kernel

Original: 7 ×
Revised: 7 ×

Original: Conv
Revised: Conv

Original: Input had kernel size 20 × 20 stride 10 and no padding; Conv2 to Conv1 had kernel size 16 × 16 , stride 10 and no padding.
Revised: Input had kernel size 20 × 20 stride 10 and no padding; Conv2 to Conv1 had kernel size 16 × 16 , stride 10 and no padding.

Original: connections the fully-connected layer were fully-connected layers whose outputs were restructured to match the convolutional layers.
Revised: connections the fully-connected layer were fully-connected layers whose outputs were restructured to match the convolutional layers.

Original: A.3 Training the RNN
Revised: A.3 Training the RNN

Original: The network was trained for 20 -way classiﬁcation using a cross-entropy loss between the network’s output and the correct object category.
Revised: The network was trained for 20 -way classiﬁcation using a cross-entropy loss between the network’s output and the correct object category.

Original: The training images were generated on the ﬂy.
Revised: The training images were generated on the ﬂy.

Original: We used the
Revised: We used the

Original: Adam optimizer for training, with a batch size of 32 , and learning rate of 10 − 4 (after manual tuning to get the best validation accuracy).
Revised: Adam optimizer for training, with a batch size of 32 , and learning rate of 10 − 4 (after manual tuning to get the best validation accuracy).

Original: The network was trained for 300 , 000 iterations.
Revised: The network was trained for 300 , 000 iterations.

Original: Before training, the weights were initialized using Xavier initialization and the biases were initialized as zeros.
Revised: Before training, the weights were initialized using Xavier initialization and the biases were initialized as zeros.

Original: A.4 Decoding approach
Revised: A.4 Decoding approach

Original: To measure the amount of explicit information about a given variable present in the activations, at each layer and at each timestep, we trained and tested a linear classiﬁer (a support vector machine,
Revised: To measure the amount of explicit information about a given variable present in the activations, at each layer and at each timestep, we trained and tested a linear classiﬁer (a support vector machine,

Original: SVM) to classify the correct value of that variable.
Revised: SVM) to classify the correct value of that variable.

Original: We used the Python scikit-learn function svm.
Revised: We used the Python scikit-learn function svm.

Original: LinearSVC() , with default parameters, which is able to handle binary as well as multi-class classiﬁcation, which was needed to decode category.
Revised: LinearSVC() , with default parameters, which is able to handle binary as well as multi-class classiﬁcation, which was needed to decode category.

Original: Each classiﬁer was trained with 800 images and tested on 200 images (all randomly drawn from the test set).
Revised: Each classiﬁer was trained with 800 images and tested on 200 images (all randomly drawn from the test set).

Original: This procedure was repeated 10 times (accuracies averaged), for each of the 5 RNN instances.
Revised: This procedure was repeated 10 times (accuracies averaged), for each of the 5 RNN instances.

Original: A.5 Perturbation analysis
Revised: A.5 Perturbation analysis

Original: To generate the perturbed feedback to the network at a given timestep t and layer l , we ran two copies of the network.
Revised: To generate the perturbed feedback to the network at a given timestep t and layer l , we ran two copies of the network.

Original: One was fed the original input image, and the other the same image, altered in one particular variable (auxiliary or category).
Revised: One was fed the original input image, and the other the same image, altered in one particular variable (auxiliary or category).

Original: For example, vertical location could be changed from top to bottom (Fig. 4A).
Revised: For example, vertical location could be changed from top to bottom (Fig. 4A).

Original: From this network we extracted the perturbed recurrent information ﬂow to l at t , and then fed this recurrent information ﬂow, at timestep t , to layer l of the network which had received the original image, thus combining the activations for the original image with the recurrent information ﬂow resulting from the perturbed image.
Revised: From this network we extracted the perturbed recurrent information ﬂow to l at t , and then fed this recurrent information ﬂow, at timestep t , to layer l of the network which had received the original image, thus combining the activations for the original image with the recurrent information ﬂow resulting from the perturbed image.

Original: We then compared this network’s systematically perturbed classiﬁcation accuracy to the network’s accuracy for the same image without any perturbations.
Revised: We then compared this network’s systematically perturbed classiﬁcation accuracy to the network’s accuracy for the same image without any perturbations.

Original: To control for the unique role of the variables in determining network performance, we also measured the accuracy of a network that received the same perturbed recurrent information ﬂow, but with the elements of the recurrent information ﬂow vector randomly permuted (control recurrent information ﬂow).
Revised: To control for the unique role of the variables in determining network performance, we also measured the accuracy of a network that received the same perturbed recurrent information ﬂow, but with the elements of the recurrent information ﬂow vector randomly permuted (control recurrent information ﬂow).

Original: This corresponds to a recurrent information ﬂow that differs from the original one by the same amount (vector length) as the relevant perturbed recurrent information ﬂow, but along a direction that, on average, does not align with any of the relevant variables (Fig. 4B).
Revised: This corresponds to a recurrent information ﬂow that differs from the original one by the same amount (vector length) as the relevant perturbed recurrent information ﬂow, but along a direction that, on average, does not align with any of the relevant variables (Fig. 4B).

Original: The functional importance score ( control − systematic
Revised: The functional importance score ( control − systematic

Original: [Table]
Revised: [Table]

Original: , Fig.
Revised: , Fig.

Original: 4C) for each variable manipulation was computed as the average across 1000 images (randomly drawn from the test set).
Revised: 4C) for each variable manipulation was computed as the average across 1000 images (randomly drawn from the test set).

Original: This procedure was repeated 5 times (scores averaged), for each of the 5 RNN instances.
Revised: This procedure was repeated 5 times (scores averaged), for each of the 5 RNN instances.

Original: In the case of category, two types of perturbations were analyzed.
Revised: In the case of category, two types of perturbations were analyzed.

Original: Either the category of the intact object was changed within its dataset (within-domain perturbation, e.g. 3 changed to a 4 , or ’trouser’ changed to ’dress’) or the category was changed across datasets (between-domain perturbation, e.g.changed to a ’trouser’).
Revised: Either the category of the intact object was changed within its dataset (within-domain perturbation, e.g. 3 changed to a 4 , or ’trouser’ changed to ’dress’) or the category was changed across datasets (between-domain perturbation, e.g.changed to a ’trouser’).

Original: In the results shown in the main text and in Fig.
Revised: In the results shown in the main text and in Fig.

Original: S2C, the averaged functional importance scores of these two types of category perturbations were shown.
Revised: S2C, the averaged functional importance scores of these two types of category perturbations were shown.

Original: For both the additive and multiplicative interactions, the functional importance of the domain information of the category (indexed by the between-domain perturbation) was equivalent or higher than the within-domain information of the object category (which was functionally important at all the layers and timesteps where domain information was functionally relevant).
Revised: For both the additive and multiplicative interactions, the functional importance of the domain information of the category (indexed by the between-domain perturbation) was equivalent or higher than the within-domain information of the object category (which was functionally important at all the layers and timesteps where domain information was functionally relevant).

Original: A.6 Results for additive recurrent interactions
Revised: A.6 Results for additive recurrent interactions

Original: The additive interactions between the feedforward and recurrent information ﬂow were implemented as follows: for a layer l at timestep t the activation was given by:
Revised: The additive interactions between the feedforward and recurrent information ﬂow were implemented as follows: for a layer l at timestep t the activation was given by:

Original: [Equation]
Revised: [Equation]

Original: Recurrent information ﬂow to layer l at timestep t
Revised: Recurrent information ﬂow to layer l at timestep t

Original: [Equation]
Revised: [Equation]

Original: where C ff , C lat and C td correspond to the feedforward, lateral and top-down transformations corresponding to layer l respectively
Revised: where C ff , C lat and C td correspond to the feedforward, lateral and top-down transformations corresponding to layer l respectively

Original: All the other details of the network architecture and training were identical to the network with multiplicative interactions.
Revised: All the other details of the network architecture and training were identical to the network with multiplicative interactions.

Original: The proﬁle of results for the network with additive recurrent interactions had both similarities and differences to the multiplicative ones.
Revised: The proﬁle of results for the network with additive recurrent interactions had both similarities and differences to the multiplicative ones.

Original: The decoding of variables at all layers and timesteps showed a similar proﬁle (Fig.
Revised: The decoding of variables at all layers and timesteps showed a similar proﬁle (Fig.

Original: S2A), but in visualizing the modulation at the input image level, no clutter reduction comparable to that in the multiplicative network was visible (Fig. S2B).
Revised: S2A), but in visualizing the modulation at the input image level, no clutter reduction comparable to that in the multiplicative network was visible (Fig. S2B).

Original: Consistent with this observation, no variable was found to be functionally important in the recurrent information ﬂow to the input, while other layers showed a pattern of variable importance similar to that of the network with multiplicative recurrent interactions (Fig.
Revised: Consistent with this observation, no variable was found to be functionally important in the recurrent information ﬂow to the input, while other layers showed a pattern of variable importance similar to that of the network with multiplicative recurrent interactions (Fig.

Original: S2C).
Revised: S2C).