Original: Path Independent Equilibrium Networks Can Better Exploit Test-Time Computation
Revised: Path Independent Equilibrium Networks Can Better Exploit Test-Time Computation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Designing networks capable of attaining better performance with an increased inference budget is important to facilitate generalization to harder problem instances.
Revised: Designing networks capable of attaining better performance with an increased inference budget is important to facilitate generalization to harder problem instances.

Original: Recent efforts have shown promising results in this direction by making use ofdepth-wise recurrent networks.
Revised: Recent efforts have shown promising results in this direction by making use ofdepth-wise recurrent networks.

Original: In this work, we reproduce the performance of theprior art using a broader class of architectures called equilibrium models, and findthat stronger generalization performance on harder examples (which require moreiterations of inference to get correct) strongly correlates with the path independence of the system—its ability to converge to the same attractor (or limit cycle)regardless of initialization, given enough computation.
Revised: In this work, we reproduce the performance of theprior art using a broader class of architectures called equilibrium models, and findthat stronger generalization performance on harder examples (which require moreiterations of inference to get correct) strongly correlates with the path independence of the system—its ability to converge to the same attractor (or limit cycle)regardless of initialization, given enough computation.

Original: Experimental interventionsmade to promote path independence result in improved generalization on harder(and thus more compute-hungry) problem instances, while those that penalize itdegrade this ability.
Revised: Experimental interventionsmade to promote path independence result in improved generalization on harder(and thus more compute-hungry) problem instances, while those that penalize itdegrade this ability.

Original: Path independence analyses are also useful on a per-examplebasis: for equilibrium models that have good in-distribution performance, path independence on out-of-distribution samples strongly correlates with accuracy.
Revised: Path independence analyses are also useful on a per-examplebasis: for equilibrium models that have good in-distribution performance, path independence on out-of-distribution samples strongly correlates with accuracy.

Original: Thus,considering equilibrium models and path independence jointly leads to a valuablenew viewpoint under which we can study the generalization performance of thesenetworks on hard problem instances.
Revised: Thus,considering equilibrium models and path independence jointly leads to a valuablenew viewpoint under which we can study the generalization performance of thesenetworks on hard problem instances.

Original:  Introduction
Revised:  Introduction

Original: One of the main challenges limiting practical applicability of modern deep learning systems is theability to generalize outside the training distribution [Koh et al., 2021].
Revised: One of the main challenges limiting practical applicability of modern deep learning systems is theability to generalize outside the training distribution [Koh et al., 2021].

Original: One particularly importanttype of out-of-distribution (OOD) generalization is upwards generalization , or the ability to generalizeto more difficult problem instances than those encountered at training time [Selsam et al., 2018,
Revised: One particularly importanttype of out-of-distribution (OOD) generalization is upwards generalization , or the ability to generalizeto more difficult problem instances than those encountered at training time [Selsam et al., 2018,

Original: Bansal et al., 2022, Schwarzschild et al., 2021b, Nye et al., 2021].
Revised: Bansal et al., 2022, Schwarzschild et al., 2021b, Nye et al., 2021].

Original: Often, good performance onmore difficult instances will require a larger amount of test-time computation, so a natural questionarises: how can we design neural net architectures which can reliably exploit additional test-timecomputation to achieve better accuracy?
Revised: Often, good performance onmore difficult instances will require a larger amount of test-time computation, so a natural questionarises: how can we design neural net architectures which can reliably exploit additional test-timecomputation to achieve better accuracy?

Original: We argue that a key determiner of whether a learned model can exploit additional test-time computation is whether the dynamical system corresponding to the model is globally stable ; that is, whetherthe learned model converges to the same limit point (or limit cycle), regardless of the initialization ofthe system or the number of times the components of the model are applied.
Revised: We argue that a key determiner of whether a learned model can exploit additional test-time computation is whether the dynamical system corresponding to the model is globally stable ; that is, whetherthe learned model converges to the same limit point (or limit cycle), regardless of the initialization ofthe system or the number of times the components of the model are applied.

Original: For example, a simpleintegrator x t +1 = x t + 1 is clearly not globally stable, as it the final state depends on the initial stateand the number of iterations run; conversely, the system x t +1 =
Revised: For example, a simpleintegrator x t +1 = x t + 1 is clearly not globally stable, as it the final state depends on the initial stateand the number of iterations run; conversely, the system x t +1 =

Original: ( x t + 1) / 2 is globally stable, as itwill converge to the solution x T = 1 as T → ∞
Revised: ( x t + 1) / 2 is globally stable, as itwill converge to the solution x T = 1 as T → ∞

Original: regardless of the initial condition of x 0 .
Revised: regardless of the initial condition of x 0 .

Original: However,the term “global stability” is slightly overloaded, as it sometimes requires convergence to a singlepoint [Slotine et al., 1991], and sometimes just implies the system is convergent everywhere, even ifto different points[Wang et al., 2003, Sriperumbudur and Lanckriet, 2009].
Revised: However,the term “global stability” is slightly overloaded, as it sometimes requires convergence to a singlepoint [Slotine et al., 1991], and sometimes just implies the system is convergent everywhere, even ifto different points[Wang et al., 2003, Sriperumbudur and Lanckriet, 2009].

Original: We thus choose the term path independence to refer specifically to the fact that the system will converge to the same limitingbehavior (whatever that might be) regardless of the initial state of the system.
Revised: We thus choose the term path independence to refer specifically to the fact that the system will converge to the same limitingbehavior (whatever that might be) regardless of the initial state of the system.

Original: Intuitively, path independent algorithms can more easily take advantage of additional test-timeiterations than path dependent ones.
Revised: Intuitively, path independent algorithms can more easily take advantage of additional test-timeiterations than path dependent ones.

Original: For instance, gradient descent applied to a convex objective ispath independent, and correspondingly when confronted with a more ill-conditioned problem instance,one can compensate by increasing the number of iterations.
Revised: For instance, gradient descent applied to a convex objective ispath independent, and correspondingly when confronted with a more ill-conditioned problem instance,one can compensate by increasing the number of iterations.

Original: Conversely, a weather simulation ispath dependent, and extending the simulation won’t yield more accurate predictions of a given day’sweather.
Revised: Conversely, a weather simulation ispath dependent, and extending the simulation won’t yield more accurate predictions of a given day’sweather.

Original: Based on this intuition, we hypothesize that path independence of a learned model is a keydeterminer of whether it can take advantage of an increased test-time iteration budget.
Revised: Based on this intuition, we hypothesize that path independence of a learned model is a keydeterminer of whether it can take advantage of an increased test-time iteration budget.

Original: In principle, neural nets have the capacity to represent many different approaches to solving a problem;for instance, even vanilla RNNs are Turing-complete [Siegelmann and Sontag, 1995].
Revised: In principle, neural nets have the capacity to represent many different approaches to solving a problem;for instance, even vanilla RNNs are Turing-complete [Siegelmann and Sontag, 1995].

Original: However, anyparticular combination of architecture and training regime may exhibit a strong inductive bias towardsa particular approach.
Revised: However, anyparticular combination of architecture and training regime may exhibit a strong inductive bias towardsa particular approach.

Original: For instance, deep equilibrium models [Bai et al., 2019] use implicit weightupdates which depend only on the final hidden state (not the path), and hence ought to encouragepath independence, despite the ability to represent path-dependent functions.
Revised: For instance, deep equilibrium models [Bai et al., 2019] use implicit weightupdates which depend only on the final hidden state (not the path), and hence ought to encouragepath independence, despite the ability to represent path-dependent functions.

Original: We hypothesize that thedegree of path independence has a fundamental impact on generalization:
Revised: We hypothesize that thedegree of path independence has a fundamental impact on generalization:

Original: Path Independence Hypothesis
Revised: Path Independence Hypothesis

Original: Models which successfully fit the training distri-
Revised: Models which successfully fit the training distri-

Original: bution with a
Revised: bution with a

Original: path
Revised: path

Original: independent function are better able to
Revised: independent function are better able to

Original: exploit more
Revised: exploit more

Original: test
Revised: test

Original: time
Revised: time

Original: iterations
Revised: iterations

Original: to
Revised: to

Original: achieve higher accuracy
Revised: achieve higher accuracy

Original: , compared to
Revised: , compared to

Original: those which fit
Revised: those which fit

Original: the
Revised: the

Original: training
Revised: training

Original: dis
Revised: dis

Original: tribution with a path-dependent function
Revised: tribution with a path-dependent function

Original: We first introduce a metric for measuring the path independence of a learned model, the Fixed Point
Revised: We first introduce a metric for measuring the path independence of a learned model, the Fixed Point

Original: Alignment (FPA) score.
Revised: Alignment (FPA) score.

Original: On two algorithmic generalization benchmarks introduced by Schwarzschildet al.
Revised: On two algorithmic generalization benchmarks introduced by Schwarzschildet al.

Original: [2021a,b], we show that a model’s FPA score is strongly correlated with its accuracy when farmore iterations are used at test time than at training time.
Revised: [2021a,b], we show that a model’s FPA score is strongly correlated with its accuracy when farmore iterations are used at test time than at training time.

Original: In general, we find that path independentmodels increase their accuracy monotonically with the number of test-time iterations, whereas pathdependent models degrade when the number of test time iterations exceeds those at training time.
Revised: In general, we find that path independentmodels increase their accuracy monotonically with the number of test-time iterations, whereas pathdependent models degrade when the number of test time iterations exceeds those at training time.

Original: We find that previously introduced architectural features which improve generalization, includinginput injection and implicit updates which are explicitly captured by equilibrium fixed point models,improve both the path independence and the accuracy.
Revised: We find that previously introduced architectural features which improve generalization, includinginput injection and implicit updates which are explicitly captured by equilibrium fixed point models,improve both the path independence and the accuracy.

Original: Furthermore, we perform an experimentalmanipulation by introducing regularizers which directly promote or punish path independence.
Revised: Furthermore, we perform an experimentalmanipulation by introducing regularizers which directly promote or punish path independence.

Original: We find that these manipulations, while increasing or decreasing the FPA score, also have thecorresponding effect on accuracy.
Revised: We find that these manipulations, while increasing or decreasing the FPA score, also have thecorresponding effect on accuracy.

Original:  Background
Revised:  Background

Original: Equilibrium Models Equilibrium models treat computing internal representations as a fixed-pointfinding problem.
Revised: Equilibrium Models Equilibrium models treat computing internal representations as a fixed-pointfinding problem.

Original: [McClelland and Rumelhart, 1989, Liao et al., 2018, Bai et al., 2019]
Revised: [McClelland and Rumelhart, 1989, Liao et al., 2018, Bai et al., 2019]

Original: Concretely,letting x ∈ R n x and f w : R n x × n z (cid:55)→ R n z stand for an input and the equilibrium model function (or
Revised: Concretely,letting x ∈ R n x and f w : R n x × n z (cid:55)→ R n z stand for an input and the equilibrium model function (or

Original: “cell") parametrized by w ∈ R n w respectively, equilibrium models aim to solve for the fixed pointz ∗ ∈ R n z that satisfies z ∗ =
Revised: “cell") parametrized by w ∈ R n w respectively, equilibrium models aim to solve for the fixed pointz ∗ ∈ R n z that satisfies z ∗ =

Original: f w ( x , z ∗ ) .
Revised: f w ( x , z ∗ ) .

Original: The cell f w might represent anything from a fully connectedlayer to a transformer block [Vaswani et al., 2017].
Revised: The cell f w might represent anything from a fully connectedlayer to a transformer block [Vaswani et al., 2017].

Original: We emphasize that f w directly depends on theinput x – following existing literature, we refer to this as input injection .
Revised: We emphasize that f w directly depends on theinput x – following existing literature, we refer to this as input injection .

Original: The outcome of the fixedpoint finding process might depend on the initial fixed point guess.
Revised: The outcome of the fixedpoint finding process might depend on the initial fixed point guess.

Original: To make this dependency explicit,we define the function FIX : R n x × n z (cid:55)→ R n z that maps an input x and an initial guess for the fixedpoint z 0 to an output that satisfies the fixed point equation FIX f w ( x , z 0 ) :
Revised: To make this dependency explicit,we define the function FIX : R n x × n z (cid:55)→ R n z that maps an input x and an initial guess for the fixedpoint z 0 to an output that satisfies the fixed point equation FIX f w ( x , z 0 ) :

Original: = f w ( x , FIX f w ( x , z 0 )) .
Revised: = f w ( x , FIX f w ( x , z 0 )) .

Original: The behaviour of FIX depends on the solver that’s used to find fixed points.
Revised: The behaviour of FIX depends on the solver that’s used to find fixed points.

Original: The most straightforward approach to solve for fixed-points is the fixed point iteration method, whichrecursively applies the function f w on the internal representations z (i.e. z t +1 = f w ( x , z t ) ).
Revised: The most straightforward approach to solve for fixed-points is the fixed point iteration method, whichrecursively applies the function f w on the internal representations z (i.e. z t +1 = f w ( x , z t ) ).

Original: Ifcertain conditions are satisfied, (such as the fixed iterations being globally contractive – more generalconditions are discussed below), this procedure converges 2 to a fixed point: z ∗ = f w ( x , z ∗ ) .
Revised: Ifcertain conditions are satisfied, (such as the fixed iterations being globally contractive – more generalconditions are discussed below), this procedure converges 2 to a fixed point: z ∗ = f w ( x , z ∗ ) .

Original: Assolving for fixed points exactly is expensive, fixed point iterations are often terminated after a fixednumber of steps or when the norm of the difference between subsequent iterates falls below a preselected threshold.
Revised: Assolving for fixed points exactly is expensive, fixed point iterations are often terminated after a fixednumber of steps or when the norm of the difference between subsequent iterates falls below a preselected threshold.

Original: The model weights can be updated using gradients computed via backpropagatingthrough the full forward computational graph.
Revised: The model weights can be updated using gradients computed via backpropagatingthrough the full forward computational graph.

Original: If one commits to using fixed point iterations as the root solver, then the output z ∗ of equilibrium models can be interpreted as the infinite-depth limit of an input-injected, weight-tied modelf ∞ w ( x , z ∗ ) =
Revised: If one commits to using fixed point iterations as the root solver, then the output z ∗ of equilibrium models can be interpreted as the infinite-depth limit of an input-injected, weight-tied modelf ∞ w ( x , z ∗ ) =

Original: lim n →∞ f ( n ) w ( x , z 0 ) where the notation f ( n ) stands for n repeated applications of fon its own output, and z 0 stands for the fixed point initialization.
Revised: lim n →∞ f ( n ) w ( x , z 0 ) where the notation f ( n ) stands for n repeated applications of fon its own output, and z 0 stands for the fixed point initialization.

Original: Deep Equilibrium (DEQ)
Revised: Deep Equilibrium (DEQ)

Original: Models Deep equilibrium models [Bai et al., 2019] solve for fixed pointsexplicitly using black-box root finders, such as Broyden’s method [Broyden, 1965] or Andersonacceleration [Anderson, 1965].
Revised: Models Deep equilibrium models [Bai et al., 2019] solve for fixed pointsexplicitly using black-box root finders, such as Broyden’s method [Broyden, 1965] or Andersonacceleration [Anderson, 1965].

Original: In order to avoid explicitly differentiating through the root-findingprocedure, DEQ models utilize implicit differentiation to compute gradients.
Revised: In order to avoid explicitly differentiating through the root-findingprocedure, DEQ models utilize implicit differentiation to compute gradients.

Original: Concretely, lettingg w ( x , z ) =
Revised: Concretely, lettingg w ( x , z ) =

Original: f w ( x , z ) − z for a fixed point z ∗ , the Jacobian of z ∗ with respect to the DEQ modelweights can be given by:
Revised: f w ( x , z ) − z for a fixed point z ∗ , the Jacobian of z ∗ with respect to the DEQ modelweights can be given by:

Original: [Equation]
Revised: [Equation]

Original: Computing the inverse of a Jacobian can become computationally expensive.
Revised: Computing the inverse of a Jacobian can become computationally expensive.

Original: Recent works [Genget al., 2021a, Fung et al., 2021] have shown that the inverse-Jacobian term in Eq.
Revised: Recent works [Genget al., 2021a, Fung et al., 2021] have shown that the inverse-Jacobian term in Eq.

Original: (1) can be replacedwith an identity matrix i.e. Jacobian-free or an approximate inverse-Jacobian
Revised: (1) can be replacedwith an identity matrix i.e. Jacobian-free or an approximate inverse-Jacobian

Original: [Geng et al., 2021b]without affecting the final performance of DEQs.
Revised: [Geng et al., 2021b]without affecting the final performance of DEQs.

Original: This approximation makes the backward passof DEQs inexpensive and lightweight.
Revised: This approximation makes the backward passof DEQs inexpensive and lightweight.

Original: DEQ models have been shown to achieve state-of-the-artperformance on various tasks including language modelling [Bai et al., 2019], image recognition,semantic segmentation [Bai et al., 2020], object detection [Wang et al., 2020], and graph modeling[Gu et al., 2020, Liu et al., 2021a].
Revised: DEQ models have been shown to achieve state-of-the-artperformance on various tasks including language modelling [Bai et al., 2019], image recognition,semantic segmentation [Bai et al., 2020], object detection [Wang et al., 2020], and graph modeling[Gu et al., 2020, Liu et al., 2021a].

Original: DEQs vs. Depthwise Recurrent Models Both DEQs and depthwise recurrent (i.e. weight-tied,fixed-depth) networks leverage weight-tying i.e., they apply the same transformation at each layer,f
Revised: DEQs vs. Depthwise Recurrent Models Both DEQs and depthwise recurrent (i.e. weight-tied,fixed-depth) networks leverage weight-tying i.e., they apply the same transformation at each layer,f

Original: [ i ] ω = f ω ∀ i .
Revised: [ i ] ω = f ω ∀ i .

Original: However, the key difference between the two is that weight-tied fixed depth networksrequire backpropagation through an explicit stack of layers but DEQs directly solve for fixed pointsusing black-box solvers during the forward pass and use implicit differentiation to compute gradients.
Revised: However, the key difference between the two is that weight-tied fixed depth networksrequire backpropagation through an explicit stack of layers but DEQs directly solve for fixed pointsusing black-box solvers during the forward pass and use implicit differentiation to compute gradients.

Original: Convergence As alluded above, in order to guarantee convergence to a unique fixed point, it sufficesfor the cell of the equilibrium model to be contractive over its input domain 3 (i.e. the singularvalues of its Jacobian all lie below 1 ).
Revised: Convergence As alluded above, in order to guarantee convergence to a unique fixed point, it sufficesfor the cell of the equilibrium model to be contractive over its input domain 3 (i.e. the singularvalues of its Jacobian all lie below 1 ).

Original: Previous work have leveraged Lipschitz constrained cells toensure contractivity[Revay et al., 2020].
Revised: Previous work have leveraged Lipschitz constrained cells toensure contractivity[Revay et al., 2020].

Original: Other approaches for ensuring global convergence exist: themonotone DEQ architecture guarantees global convergence by utilizing a DEQ parametrization thatbears similarities to solutions to a particular form of monotone operator splitting problem [Winstonand Kolter, 2020].
Revised: Other approaches for ensuring global convergence exist: themonotone DEQ architecture guarantees global convergence by utilizing a DEQ parametrization thatbears similarities to solutions to a particular form of monotone operator splitting problem [Winstonand Kolter, 2020].

Original: Unrestricted DEQs aren’t constrained enough to guarantee convergence: theycan easily express globally divergent vector fields that prohibit the existence of fixed points.
Revised: Unrestricted DEQs aren’t constrained enough to guarantee convergence: theycan easily express globally divergent vector fields that prohibit the existence of fixed points.

Original: Itis, therefore, interesting that they can and often learn path independent solutions.
Revised: Itis, therefore, interesting that they can and often learn path independent solutions.

Original: Also note thatthe “infinite-depth weight tied network" interpretation of DEQs is less general than the implicitformulation presented above, as the latter admits unstable fixed points as well.
Revised: Also note thatthe “infinite-depth weight tied network" interpretation of DEQs is less general than the implicitformulation presented above, as the latter admits unstable fixed points as well.

Original: Banach
Revised: Banach

Original: Theorem.
Revised: Theorem.

Original: [Figure]
Revised: [Figure]

Original:  Upwards Generalization with Equilibrium Models
Revised:  Upwards Generalization with Equilibrium Models

Original: To study the effects of test time computation, it is useful to consider tasks with an explicit difficultyparameter, so that the learned models can be tested on more difficult instances which require a largenumber of iterations to solve correctly.
Revised: To study the effects of test time computation, it is useful to consider tasks with an explicit difficultyparameter, so that the learned models can be tested on more difficult instances which require a largenumber of iterations to solve correctly.

Original: We focus in particular on two algorithmic generalizationbenchmarks introduced by Schwarzschild et al.
Revised: We focus in particular on two algorithmic generalizationbenchmarks introduced by Schwarzschild et al.

Original: [2021a,b]: prefix sum and mazes .
Revised: [2021a,b]: prefix sum and mazes .

Original: In this section, wefirst replicate their findings on these problems, and then extend them to a wider variety of architecturesconsidered in this paper, most notably deep equilibrium models.
Revised: In this section, wefirst replicate their findings on these problems, and then extend them to a wider variety of architecturesconsidered in this paper, most notably deep equilibrium models.

Original: Tasks Prefix-sum is a sequence-to-sequence task whereby the network is given a sequence of 0-1 bits,and is trained to output, for each bit, the parity of all of the bits received since the beginning of thesequence until the current bit.
Revised: Tasks Prefix-sum is a sequence-to-sequence task whereby the network is given a sequence of 0-1 bits,and is trained to output, for each bit, the parity of all of the bits received since the beginning of thesequence until the current bit.

Original: We train on 10,000 unique 32-bit binary strings, and report results onbinary strings of other lengths.
Revised: We train on 10,000 unique 32-bit binary strings, and report results onbinary strings of other lengths.

Original: The mazes task is also an image-to-image task, where the input is athree-channel RGB image.
Revised: The mazes task is also an image-to-image task, where the input is athree-channel RGB image.

Original: The ‘start’ and ‘finish’ positions are marked by a red and a green squarerespectively; walls are marked in black.
Revised: The ‘start’ and ‘finish’ positions are marked by a red and a green squarerespectively; walls are marked in black.

Original: The output is the optimal path in the maze that connectsthese two points without passing through the walls.
Revised: The output is the optimal path in the maze that connectsthese two points without passing through the walls.

Original: We train on 50,000 small mazes of size 9 × 9 ,and report upward generalization results on larger mazes.
Revised: We train on 50,000 small mazes of size 9 × 9 ,and report upward generalization results on larger mazes.

Original: Instances of each of these problems areprovided in the Supplementary Material.
Revised: Instances of each of these problems, aswell as additional image classification and continuous optimization results can be found in SM.

Original: Strong Upward Generalization Figure 2a shows that equilibrium models demonstrate very strongupward generalization performance compared to non weight-tied fixed-depth models.
Revised: Strong Upward Generalization Figure 2a shows that equilibrium models demonstrate very strongupward generalization performance compared to non weight-tied fixed-depth models.

Original: Moreover,
Revised: Moreover,

Original: Figure 2b shows that increasing inference depth consistently improves performance—especially onharder problem instances.
Revised: Figure 2b shows that increasing inference depth consistently improves performance—especially onharder problem instances.

Original: Terminology and Abbreviations We use the term ‘solver’ to refer to the use of black-box root finderslike Anderson acceleration to find fixed points of a DEQ.
Revised: Terminology and Abbreviations We use the term ‘solver’ to refer to the use of black-box root finderslike Anderson acceleration to find fixed points of a DEQ.

Original: These networks use implicit gradients (IFT)for the backward pass.
Revised: These networks use implicit gradients (IFT)for the backward pass.

Original: The term ‘unroll’ refers to equilibrium models that use regular fixed-pointiterations to compute the equilibrium point.
Revised: The term ‘unroll’ refers to equilibrium models that use regular fixed-pointiterations to compute the equilibrium point.

Original: We use the abbreviation ‘bp’ to refer to backpropagationgradients, and ‘inj’ to refer to input injection.
Revised: We use the abbreviation ‘bp’ to refer to backpropagationgradients, and ‘inj’ to refer to input injection.

Original: The term ‘progressive net’ refers to the deep thinkingnetworks trained with progressive training as proposed by Bansal et al.
Revised: The term ‘progressive net’ refers to the deep thinkingnetworks trained with progressive training as proposed by Bansal et al.

Original: We use ‘PI’ and ‘nonPI’ to refer to path independent and non dependent networks, respectively.
Revised: We use ‘PI’ and ‘nonPI’ to refer to path independent and non dependent networks, respectively.

Original:  Path Independence
Revised:  Path Independence

Original: Having intuitively motivated the idea of path independence in Sec. 1, we now formulate a precisedefinition: we say that the computation performed by a recurrent operator computing function f won an input x is path independent if it converges to the same limiting behavior regardless of thecurrent state z t .
Revised: Having intuitively motivated the idea of path independence in Sec. 1, we now formulate a precisedefinition: we say that the computation performed by a recurrent operator computing function f won an input x is path independent if it converges to the same limiting behavior regardless of thecurrent state z t .

Original: As a special case, if the computation is convergent, this property is equivalent to theexistence of a unique fixed point z ∗ such that f ∞ w ( x , z 0 ) =
Revised: As a special case, if the computation is convergent, this property is equivalent to theexistence of a unique fixed point z ∗ such that f ∞ w ( x , z 0 ) =

Original: z ∗ for any z 0 .
Revised: z ∗ for any z 0 .

Original: However, our definitionallows for other behaviors such as limit cycles (see Section 7).
Revised: However, our definitionallows for other behaviors such as limit cycles (see Section 7).

Original: Some architectures guarantee the path independence property (see Section 2).
Revised: Some architectures guarantee the path independence property (see Section 2).

Original: However, most common
Revised: However, most common

Original: DEQ architectures—and the ones we will be using throughout this paper—have the expressive power to easily learn multiple fixed points per input.
Revised: DEQ architectures—and the ones we will be using throughout this paper—have the expressive power to easily learn multiple fixed points per input.

Original: Since it is unclear whether architectures enforcing thecontraction property lose expressiveness [Bai et al., 2021], we focus our investigation on architecturesnot guaranteed to be path independent.
Revised: Since it is unclear whether architectures enforcing thecontraction property lose expressiveness [Bai et al., 2021], we focus our investigation on architecturesnot guaranteed to be path independent.

Original: PI networks represent a different model for computation than standard feed-forward networks: insteadof learning an entire computational graph to map inputs to outputs, they only have to learn where tostop .
Revised: PI networks represent a different model for computation than standard feed-forward networks: insteadof learning an entire computational graph to map inputs to outputs, they only have to learn where tostop .

Original: We dedicate the rest of the paper on exploring the Path Independence Hypothesis —the idea thatpath dependent networks can better exploit test-time computation and hence show strong upwardgeneralization from easy to hard problem instances of algorithmic tasks.
Revised: We dedicate the rest of the paper on exploring the Path Independence Hypothesis —the idea thatpath dependent networks can better exploit test-time computation and hence show strong upwardgeneralization from easy to hard problem instances of algorithmic tasks.

Original: Before establishing a connection between path independence and out-of-distribution generalization,we first describe two architectural components that are necessary conditions for achieving pathindependence, then proceed to describe a metric to quantify degrees of path independence.
Revised: Before establishing a connection between path independence and out-of-distribution generalization,we first describe two architectural components that are necessary conditions for achieving pathindependence, then proceed to describe a metric to quantify degrees of path independence.

Original: 4.1 Architectural Components Necessary for Path Independence
Revised: 4.1 Architectural Components Necessary for Path Independence

Original: Past work has observed that weight tying and input injection are both crucial for upwards generalization [Bansal et al., 2022].
Revised: Past work has observed that weight tying and input injection are both crucial for upwards generalization [Bansal et al., 2022].

Original: We observe that both architectural components are also necessary for alearned model to be PI 4 .
Revised: We observe that both architectural components are also necessary for alearned model to be PI 4 .

Original: Without weight tying, the network is constrained to have a fixed forwarddepth, so it is meaningless to talk about the limiting behavior in large depth.
Revised: Without weight tying, the network is constrained to have a fixed forwarddepth, so it is meaningless to talk about the limiting behavior in large depth.

Original: Input injection ensuresthat the equilibrium point depends on the input despite having an “infinite depth".
Revised: Input injection ensuresthat the equilibrium point depends on the input despite having an “infinite depth".

Original: Without inputinjection, a PI network would necessarily forget the input; hence, any model which successfully fitsthe training distribution must be path dependent.
Revised: Without inputinjection, a PI network would necessarily forget the input; hence, any model which successfully fitsthe training distribution must be path dependent.

Original: Interestingly, both architectural motifs are also key components of deep equilibrium models [Bai et al.
Revised: Interestingly, both architectural motifs are also key components of deep equilibrium models [Bai et al.

Original: ,2019]; in that work, the motivation was to enable efficient gradient estimation via the implicit functiontheorem (IFT)— a concept closely related to path independence, since the premise of the IFT gradientestimator is that only the final hidden state matters, not the path taken to get there.
Revised: ,2019]; in that work, the motivation was to enable efficient gradient estimation via the implicit functiontheorem (IFT)— a concept closely related to path independence, since the premise of the IFT gradientestimator is that only the final hidden state matters, not the path taken to get there.

Original: It is strikingthat two separate lines of work would converge on the same architectural motifs, one motivated bygeneralization and the other by a variant of path independence.
Revised: It is strikingthat two separate lines of work would converge on the same architectural motifs, one motivated bygeneralization and the other by a variant of path independence.

Original: We believe this is not a coincidence.
Revised: We believe this is not a coincidence.

Original: Reproducing the results of Bansal et al.
Revised: Reproducing the results of Bansal et al.

Original: [2022], in Figure 2a we show upward generalization performance using both equilibrium models and progressive nets [Bansal et al., 2022] – and the lackthereof using non-input injected networks.
Revised: [2022], in Figure 2a we show upward generalization performance using both equilibrium models and progressive nets [Bansal et al., 2022] – and the lackthereof using non-input injected networks.

Original: For the remainder of this paper, we focus on architectureswith both input injection and weight tying.
Revised: For the remainder of this paper, we focus on architectureswith both input injection and weight tying.

Original: 4.2 Quantifying Path Independence
Revised: 4.2 Quantifying Path Independence

Original: We propose a simple metric to quantify path independence based on the directional alignment ofthe fixed points computed with the same input, but different initializations.
Revised: We propose a simple metric to quantify path independence based on the directional alignment ofthe fixed points computed with the same input, but different initializations.

Original: We name this metric the
Revised: We name this metric the

Original: Fixed Point Alignment (FPA) score .
Revised: Fixed Point Alignment (FPA) score .

Original: Pseudocode to compute the metric is given in Algorithm Box1.
Revised: Pseudocode to compute the metric is given in Algorithm Box1.

Original: The FPA score is the average cosine similarity between the fixed point obtained with the trainingtime initialization (often simply the zero vector) and the fixed points obtained when one initializesthe solver using the fixed points computed on different inputs .
Revised: The FPA score is the average cosine similarity between the fixed point obtained with the trainingtime initialization (often simply the zero vector) and the fixed points obtained when one initializesthe solver using the fixed points computed on different inputs .

Original: Higher FPA scores (with 1 being thehighest value) imply higher degrees of path independence.
Revised: Higher FPA scores (with 1 being thehighest value) imply higher degrees of path independence.

Original: In Section 5, we show a strong correlationbetween path independence and accuracy using the FPA score.
Revised: In Section 5, we show a strong correlationbetween path independence and accuracy using the FPA score.

Original: FPA score is cheap to compute, is a reliable indicator of path independence (see below) and is unitless,meaning networks obtained from different training runs can be compared on equal footing.
Revised: FPA score is cheap to compute, is a reliable indicator of path independence (see below) and is unitless,meaning networks obtained from different training runs can be compared on equal footing.

Original: See SM forother metrics we’ve considered for quantifying path independence and why we settled on FPA score.
Revised: See SM forother metrics we’ve considered for quantifying path independence and why we settled on FPA score.

Original: Stress-testing the FPA score To stress-test the extent to which the FPA score really measures pathindependence, we search for adversarial initializations that are optimized to result in distinct fixedpoints, hence low FPA values.
Revised: Stress-testing the FPA score To stress-test the extent to which the FPA score really measures pathindependence, we search for adversarial initializations that are optimized to result in distinct fixedpoints, hence low FPA values.

Original: (Unlike adversarial examples, this attack is not constrained to an ε -ball.)
Revised: (Unlike adversarial examples, this attack is not constrained to an ε -ball.)

Original: We use the L-BFGS
Revised: We use the L-BFGS

Original: [Liu and Nocedal, 1989] optimizer, and repeat the search multiple times startingfrom different fixed point initializations.
Revised: [Liu and Nocedal, 1989] optimizer, and repeat the search multiple times startingfrom different fixed point initializations.

Original: We include pseudocode in the Supplementary material.
Revised: We include pseudocode in the Supplementary material.

Original: The results of the adversarial stress test can be seen in Table 1.
Revised: The results of the adversarial stress test can be seen in Table 1.

Original: The results corroborate that the FPAscore is indeed a reliable measure of path independence; while it is not possible to find adversarialinitializations for high FPA networks (indicating high path independence), low FPA networks caneasily be adversarially initialized to be steered away from the original fixed point estimate.
Revised: The results corroborate that the FPAscore is indeed a reliable measure of path independence; while it is not possible to find adversarialinitializations for high FPA networks (indicating high path independence), low FPA networks caneasily be adversarially initialized to be steered away from the original fixed point estimate.

Original: Input: A batched input (cid:20) xx(cid:21) , an operator f w
Revised: Input: A batched input (cid:20) xx(cid:21) , an operator f w

Original: Initialize: (cid:20) zz(cid:21) =
Revised: Initialize: (cid:20) zz(cid:21) =

Original: Define: h ( y 1 , y 2 ) = y∥ y 1 ∥ 2 · y∥ y 2 ∥
Revised: Define: h ( y 1 , y 2 ) = y∥ y 1 ∥ 2 · y∥ y 2 ∥

Original: Compute (cid:20) z ′z ′(cid:21) =
Revised: Compute (cid:20) z ′z ′(cid:21) =

Original: FIX f w (cid:18)(cid:20) xx(cid:21) , (cid:20) zz(cid:21)(cid:19)
Revised: FIX f w (cid:18)(cid:20) xx(cid:21) , (cid:20) zz(cid:21)(cid:19)

Original: # Interchange and reinitialize fixed points
Revised: # Interchange and reinitialize fixed points

Original: Compute (cid:20) z ′′z ′′(cid:21) =
Revised: Compute (cid:20) z ′′z ′′(cid:21) =

Original: FIX f w (cid:18)(cid:20) xx(cid:21) , (cid:20) z ′z ′(cid:21)(cid:19) return average( h ( z ′′ 1 , z ′ 1 ) , h ( z ′′ 2 , z ′ 2 ))
Revised: FIX f w (cid:18)(cid:20) xx(cid:21) , (cid:20) z ′z ′(cid:21)(cid:19) return average( h ( z ′′ 1 , z ′ 1 ) , h ( z ′′ 2 , z ′ 2 ))

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: on 13 × 13 mazes and 64 bit prefix sum.
Revised: on 13 × 13 mazes and 64 bit prefix sum.

Original: Attacked FPA refers to the cosine similarity between the fixed point from zero initialization and an adversarial initialization.
Revised: Attacked FPA refers to the cosine similarity between the fixed point from zero initialization and an adversarial initialization.

Original:  Path Independence Correlates with Upward Generalization
Revised:  Path Independence Correlates with Upward Generalization

Original: Is path independence (as measured by the FPA score) is a strong predictor of upwards generalization?
Revised: Is path independence (as measured by the FPA score) is a strong predictor of upwards generalization?

Original: We took the trained networks from Section 3, computed their average FPA Scores on in- and out-ofdistribution splits and inspected whether the FPA scores are correlated with upward generalization.
Revised: We took the trained networks from Section 3, computed their average FPA Scores on in- and out-ofdistribution splits and inspected whether the FPA scores are correlated with upward generalization.

Original: On prefix sum experiments, we varied 1) network depth, 2) whether or not weight norm (wnorm)[Salimans and Kingma, 2016] was used or not 5 3) learning rate (one of [0.01, 0.001, 0.0001]) 4)forward solver (fixed point iterations or Anderson acceleration [Anderson, 1965], and 5) the gradientestimator (backprop or implicit gradients) 6 .
Revised: On prefix sum experiments, we varied 1) network depth, 2) whether or not weight norm (wnorm)[Salimans and Kingma, 2016] was used or not 5 3) learning rate (one of [0.01, 0.001, 0.0001]) 4)forward solver (fixed point iterations or Anderson acceleration [Anderson, 1965], and 5) the gradientestimator (backprop or implicit gradients) 6 .

Original: On the maze experiments, we varied 1) network depth, 2)use of weight norm 3) forward solver (fixed point iterations or Broyden solver [Broyden, 1965]), and5)
Revised: On the maze experiments, we varied 1) network depth, 2)use of weight norm 3) forward solver (fixed point iterations or Broyden solver [Broyden, 1965]), and5)

Original: the gradient estimator (backprop or implicit gradients)
Revised: the gradient estimator (backprop or implicit gradients)

Original: Figure 4 displays our findings.
Revised: Figure 4 displays our findings.

Original: We evaluated performance on a mixture of in and OOD validationdata - results on individual data splits can be found in the Appendix.
Revised: We evaluated performance on a mixture of in and OOD validationdata - results on individual data splits can be found in the Appendix.

Original: The results show a strongcorrelation between FPA score and accuracy when the inference depth is large enough.
Revised: The results show a strongcorrelation between FPA score and accuracy when the inference depth is large enough.

Original: This showsthat PI networks that allow for scaling test-time compute improve test-time accuracy (also see Figure2b).
Revised: This showsthat PI networks that allow for scaling test-time compute improve test-time accuracy (also see Figure2b).

Original: The in-distribution validation performance of non-PI networks degrade with deeper inferencedepths.
Revised: The in-distribution validation performance of non-PI networks degrade with deeper inferencedepths.

Original: Unsurprisingly, these networks generalize poorly on harder problem instances that provablyrequire deeper inference depths.
Revised: Unsurprisingly, these networks generalize poorly on harder problem instances that provablyrequire deeper inference depths.

Original: [Figure]
Revised: [Figure]

Original:  Experimental Manipulations of Path Independence
Revised:  Experimental Manipulations of Path Independence

Original: Previous sections demonstrated a strong correlation between path independence and the ability toexploit test-time iterations.
Revised: Previous sections demonstrated a strong correlation between path independence and the ability toexploit test-time iterations.

Original: Unfortunately, we can’t make a causal claim based on those studies:the observed effect could have been due to an unobserved confounder.
Revised: Unfortunately, we can’t make a causal claim based on those studies:the observed effect could have been due to an unobserved confounder.

Original: In this section, we intervenedirectly on path independence by imposing regularizers which directly encourage or penalize pathindependence.
Revised: In this section, we intervenedirectly on path independence by imposing regularizers which directly encourage or penalize pathindependence.

Original: We find that interventions designed to promote path independence also improvegeneralization, while interventions designed to reduce path independence also hurt generalization.
Revised: We find that interventions designed to promote path independence also improvegeneralization, while interventions designed to reduce path independence also hurt generalization.

Original: 6.1 Promoting Path Independence via Randomized Forward Passes
Revised: 6.1 Promoting Path Independence via Randomized Forward Passes

Original: A straightforward way to encourage path independence is simply to initialize the hidden states withrandom noise during training.
Revised: A straightforward way to encourage path independence is simply to initialize the hidden states withrandom noise during training.

Original: To this end, we experimented with initializing the hidden states withzeros on half of the examples in the batch, and with standard Gaussian noise on the rest of theexamples.
Revised: To this end, we experimented with initializing the hidden states withzeros on half of the examples in the batch, and with standard Gaussian noise on the rest of theexamples.

Original: The reason to include the zero-initializations at training time is that we initialize fromzeros at test time - not including this initialization during traning time causes a distribution shift.)
Revised: The reason to include the zero-initializations at training time is that we initialize fromzeros at test time - not including this initialization during traning time causes a distribution shift.)

Original: Another way to promote path independence is simply running the forward solver with randomizedcompute budgets/depths during training time.
Revised: Another way to promote path independence is simply running the forward solver with randomizedcompute budgets/depths during training time.

Original: While a path independent solution can be expected tobe robust against this intervention, a path dependent one will fail.
Revised: While a path independent solution can be expected tobe robust against this intervention, a path dependent one will fail.

Original: We took the training configurations of the 12 prefix-sum networks described in Section 5 that usefixed point iterations in their forward pass, and backpropagation gradient in their backward pass,and retrained them separately with the aforementioned mixed initialization and randomized depthstrategies without modifying any other experimental conditions.
Revised: We took the training configurations of the 12 prefix-sum networks described in Section 5 that usefixed point iterations in their forward pass, and backpropagation gradient in their backward pass,and retrained them separately with the aforementioned mixed initialization and randomized depthstrategies without modifying any other experimental conditions.

Original: As can be seen in the Figure 3, theinterventions lead to strong test-time path independent neural networks, while also reliably improvingin- and out-of-distribution validation accuracy.
Revised: As can be seen in the Figure 3, theinterventions lead to strong test-time path independent neural networks, while also reliably improvingin- and out-of-distribution validation accuracy.

Original: We especially emphasize that shallow networkstrained with mixed initialization actually remain far from having high FPA scores using the trainingtime forward pass conditions due to lack of convergence.
Revised: We especially emphasize that shallow networkstrained with mixed initialization actually remain far from having high FPA scores using the trainingtime forward pass conditions due to lack of convergence.

Original: However, since the mixed initializationstrategy results in path independent networks, scaling up test-time compute budget leads to high FPAscores, and therefore high upwards generalization.
Revised: However, since the mixed initializationstrategy results in path independent networks, scaling up test-time compute budget leads to high FPAscores, and therefore high upwards generalization.

Original: 6.2 Penalizing Path Independence via the Fixed Point Alignment Penalty
Revised: 6.2 Penalizing Path Independence via the Fixed Point Alignment Penalty

Original: Does an intervention that results in less path independence also result in poorer upwards generalization?
Revised: Does an intervention that results in less path independence also result in poorer upwards generalization?

Original: Like in the mixed initialization experiment, we retrained the 12 unroll + backpropagationnetworks with an additional auxiliary loss term that penalizes the dot product between the fixed pointscomputed from the same input, but different initializations sampled from standard Gaussian noise.
Revised: Like in the mixed initialization experiment, we retrained the 12 unroll + backpropagationnetworks with an additional auxiliary loss term that penalizes the dot product between the fixed pointscomputed from the same input, but different initializations sampled from standard Gaussian noise.

Original: Figure 3 shows that this intervention succeeded in pushing the FPA scores down, while also keepingthe accuracy on the same trend line.
Revised: Figure 3 shows that this intervention succeeded in pushing the FPA scores down, while also keepingthe accuracy on the same trend line.

Original: [Table]
Revised: [Table]

Original:  Disambiguating Convergence and Path Independence
Revised:  Disambiguating Convergence and Path Independence

Original: Is convergence necessary for path independence?
Revised: Is convergence necessary for path independence?

Original: We answer this statement in the negative, and showthat neither training-time convergence nor test-time convergence is required for path independence.
Revised: We answer this statement in the negative, and showthat neither training-time convergence nor test-time convergence is required for path independence.

Original: Instead convergence to the same limiting behavior regardless of initialization is important.
Revised: Instead convergence to the same limiting behavior regardless of initialization is important.

Original: Training Time Convergence We consider two deep equilibrium models (DEQs) trained on the
Revised: Training Time Convergence We consider two deep equilibrium models (DEQs) trained on the

Original: Mazes task—one trained with implicit gradients computed via implicit function theorem (IFT), andthe other trained with phantom gradients [Geng et al., 2021b].
Revised: Mazes task—one trained with implicit gradients computed via implicit function theorem (IFT), andthe other trained with phantom gradients [Geng et al., 2021b].

Original: We report the values of residuals( i.e., ∥ f w ( x , z ) − z ∥ 2 ), FPA scores and accuracies observed for in- and out-of-distribution data formazes in Table 2.
Revised: We report the values of residuals( i.e., ∥ f w ( x , z ) − z ∥ 2 ), FPA scores and accuracies observed for in- and out-of-distribution data formazes in Table 2.

Original: We observe that DEQs trained with phantom gradients have higher values of indistribution residuals but are path independent, as indicated by their high FPA scores, and show strongupward generalization as indicated by their good accuracy.
Revised: We observe that DEQs trained with phantom gradients have higher values of indistribution residuals but are path independent, as indicated by their high FPA scores, and show strongupward generalization as indicated by their good accuracy.

Original: The mixed-initialization intervention described in Section 6.1 also leads to a separation betweentraining-time convergence and path independence.
Revised: The mixed-initialization intervention described in Section 6.1 also leads to a separation betweentraining-time convergence and path independence.

Original: We found that it is possible to train very shallow( i.e., 5 layer) unrolled networks that, while being very far from converging during training andattaining poor in-distribution generalization, are able to achieve perfect performance when run formany more iterations during test time.
Revised: We found that it is possible to train very shallow( i.e., 5 layer) unrolled networks that, while being very far from converging during training andattaining poor in-distribution generalization, are able to achieve perfect performance when run formany more iterations during test time.

Original: Detailsare provided in the Supplementary material.
Revised: Detailsare provided in the Supplementary material.

Original: Test Time Convergence From Table 2, one might conclude that test time convergence is important forpath independence.
Revised: Test Time Convergence From Table 2, one might conclude that test time convergence is important forpath independence.

Original: However, we show that this connection is not necessary, and convergence to thesame fixed point is not a required condition for path independence.
Revised: However, we show that this connection is not necessary, and convergence to thesame fixed point is not a required condition for path independence.

Original: We study test time convergenceproperties of an unrolled weight-tied input-injected network trained with backpropagation underdifferent solvers.
Revised: We study test time convergenceproperties of an unrolled weight-tied input-injected network trained with backpropagation underdifferent solvers.

Original: This network is path independent, as indicated by its high FPA scores (0.99)
Revised: This network is path independent, as indicated by its high FPA scores (0.99)

Original: onboth in- and out-of-distribution data.
Revised: onboth in- and out-of-distribution data.

Original: We visualize the values of residuals observed at test time withfixed point iterations and Broyden’s method in Figure 5a.
Revised: We visualize the values of residuals observed at test time withfixed point iterations and Broyden’s method in Figure 5a.

Original: Both these solvers converge to differentlimit cycles but still show good upward generalization.
Revised: Both these solvers converge to differentlimit cycles but still show good upward generalization.

Original:  Path Independence on a Per-Example Level
Revised:  Path Independence on a Per-Example Level

Original: The connection between path independence and prediction correctness also largely holds on a perinstance basis.
Revised: The connection between path independence and prediction correctness also largely holds on a perinstance basis.

Original: Using the prefix-sum networks trained with the mixed-initialization strategy (themost performant group of networks in our intervention experiments), we plotted the distribution ofper-instance fixed point alignment scores, colored by whether the prediction on that instance was correct or not in Figure 5b.
Revised: Using the prefix-sum networks trained with the mixed-initialization strategy (themost performant group of networks in our intervention experiments), we plotted the distribution ofper-instance fixed point alignment scores, colored by whether the prediction on that instance was correct or not in Figure 5b.

Original: This suggests that path independence can be used as a valuable sanity-checkto determine whether a prediction is correct or not without the need for any label data, both in- andout-of-distribution.
Revised: This suggests that path independence can be used as a valuable sanity-checkto determine whether a prediction is correct or not without the need for any label data, both in- andout-of-distribution.

Original: We provide a more in-depth per-instance analysis in the Supplementary material.
Revised: We provide a more in-depth per-instance analysis in the Supplementary material.

Original:  Related Work
Revised:  Related Work

Original: There is a long line of research on neural networks that can adapt their computational budget based oncomplexity of task they are learning to solve—akin to the intrinsic mechanism in humans to reasonand solve problems.
Revised: There is a long line of research on neural networks that can adapt their computational budget based oncomplexity of task they are learning to solve—akin to the intrinsic mechanism in humans to reasonand solve problems.

Original: Schmidhuber [2012] introduced self-delimiting neural networks which are a typeof recurrent neural networks (RNNs) that adapt their compute based on the output of a special "halt"neuron.
Revised: Schmidhuber [2012] introduced self-delimiting neural networks which are a typeof recurrent neural networks (RNNs) that adapt their compute based on the output of a special "halt"neuron.

Original: Adaptive computation time (ACT)
Revised: Adaptive computation time (ACT)

Original: [Graves, 2016] also uses the output of a sigmoidal haltingunit to determine the termination condition of RNN, but it avoids long "thinking" time by explicitlypenalizing it.
Revised: [Graves, 2016] also uses the output of a sigmoidal haltingunit to determine the termination condition of RNN, but it avoids long "thinking" time by explicitlypenalizing it.

Original: Subsequent works have successfully applied variants of ACT in image classification andobject detection [Figurnov et al., 2017], visual reasoning [Eyzaguirre and Soto, 2020], Transformers[Vaswani et al., 2017] for language modelling [Dehghani et al., 2019, Elbayad et al., 2020a, Liu et al.
Revised: Subsequent works have successfully applied variants of ACT in image classification andobject detection [Figurnov et al., 2017], visual reasoning [Eyzaguirre and Soto, 2020], Transformers[Vaswani et al., 2017] for language modelling [Dehghani et al., 2019, Elbayad et al., 2020a, Liu et al.

Original: ,2021b], and recognizing textual entailment [Neumann et al., 2016].
Revised: ,2021b], and recognizing textual entailment [Neumann et al., 2016].

Original: PonderNet [Banino et al., 2021]reformulates the halting policy of ACT as a probabilistic model, and adds a regularization term in theloss objective to encourage exploration.
Revised: PonderNet [Banino et al., 2021]reformulates the halting policy of ACT as a probabilistic model, and adds a regularization term in theloss objective to encourage exploration.

Original: With these additions, PonderNet can extrapolate to more
Revised: With these additions, PonderNet can extrapolate to more

Original: [Figure]
Revised: [Figure]

Original: difficult examples on parity task.
Revised: difficult examples on parity task.

Original: In this work, we do not optimize or penalize the network for thenumber of computational steps.
Revised: In this work, we do not optimize or penalize the network for thenumber of computational steps.

Original: Our main goal is to understand the underlying mechanism that resultsin scalable generalization of equilibrium models on harder problem instances.
Revised: Our main goal is to understand the underlying mechanism that resultsin scalable generalization of equilibrium models on harder problem instances.

Original: Another family of models with the property of adaptive inference compute budget is early exitnetworks [Teerapittayanon et al., 2016, Laskaridis et al., 2021].
Revised: Another family of models with the property of adaptive inference compute budget is early exitnetworks [Teerapittayanon et al., 2016, Laskaridis et al., 2021].

Original: These networks have multipleadditional “exit” prediction heads along their depth.
Revised: These networks have multipleadditional “exit” prediction heads along their depth.

Original: At inference time, the result that satisfies an exitpolicy is selected as the prediction output.
Revised: At inference time, the result that satisfies an exitpolicy is selected as the prediction output.

Original: This approach of designing adaptive networks has beenadapted both in natural language processing [Schwartz et al., 2020, Soldaini and Moschitti, 2020,
Revised: This approach of designing adaptive networks has beenadapted both in natural language processing [Schwartz et al., 2020, Soldaini and Moschitti, 2020,

Original: Elbayad et al., 2020b, Zhou et al., 2020, Liu et al., 2020] and vision [Li et al., 2017, Wang et al., 2018,
Revised: Elbayad et al., 2020b, Zhou et al., 2020, Liu et al., 2020] and vision [Li et al., 2017, Wang et al., 2018,

Original: Xing et al., 2020, Kouris et al., 2021].
Revised: Xing et al., 2020, Kouris et al., 2021].

Original: Most of these architectures have complex sub-modules that aretrained in multiple stages, and require complex exit policies.
Revised: Most of these architectures have complex sub-modules that aretrained in multiple stages, and require complex exit policies.

Original: In contrast, equilibrium models have asimple architecture, and can use root solvers to efficiently solve for the fixed point at inference.
Revised: In contrast, equilibrium models have asimple architecture, and can use root solvers to efficiently solve for the fixed point at inference.

Original: More complex transformer-based language models like GPT-3 also struggle to generalize well onsimple algorithmic tasks like addition [Brown et al., 2020].
Revised: More complex transformer-based language models like GPT-3 also struggle to generalize well onsimple algorithmic tasks like addition [Brown et al., 2020].

Original: Recent work by Nye et al.
Revised: Recent work by Nye et al.

Original: [2021] showsthat transformers can be trained to perform well on algorithmic tasks and generalize on OOD databy emitting the intermediate steps of algorithm to a buffer called ‘scratchpad’.
Revised: [2021] showsthat transformers can be trained to perform well on algorithmic tasks and generalize on OOD databy emitting the intermediate steps of algorithm to a buffer called ‘scratchpad’.

Original: Using a scratchpadenables the model to revisit its errors and correct them.
Revised: Using a scratchpadenables the model to revisit its errors and correct them.

Original: Our current work is closely related to previous work by Schwarzschild et al.
Revised: Our current work is closely related to previous work by Schwarzschild et al.

Original: [2021b] on easy tohard generalization of RNN on algorithmic tasks.
Revised: [2021b] on easy tohard generalization of RNN on algorithmic tasks.

Original: A subsequent work [Bansal et al., 2022] proposesarchitectural choices and training mechanisms that enables weight tied networks to converge to a fixedpoint, and generalize on harder problem instances.
Revised: A subsequent work [Bansal et al., 2022] proposesarchitectural choices and training mechanisms that enables weight tied networks to converge to a fixedpoint, and generalize on harder problem instances.

Original: We identify that these networks belong a moregeneral class of networks called equilibrium models that learn solutions that are path independent,and can therefore generalize on harder problem instances.
Revised: We identify that these networks belong a moregeneral class of networks called equilibrium models that learn solutions that are path independent,and can therefore generalize on harder problem instances.

Original:  Conclusion
Revised:  Conclusion

Original: Being able to attain better levels of performance using a larger inference-time compute budget is afeat that eludes most standard deep learning architectures.
Revised: Being able to attain better levels of performance using a larger inference-time compute budget is afeat that eludes most standard deep learning architectures.

Original: This is especially relevant for tasks thatrequire upwards generalization , i.e., the ability to generalize from easy problem instances to hard ones.
Revised: This is especially relevant for tasks thatrequire upwards generalization , i.e., the ability to generalize from easy problem instances to hard ones.

Original: We show that equilibrium models are capable of displaying upwards generalization by exploitingscalable test-time compute.
Revised: We show that equilibrium models are capable of displaying upwards generalization by exploitingscalable test-time compute.

Original: We link this to a phenomenon we call path independence : the tendencyof an equilibrium network to converge to the same limiting behavior given an input, regardless of theinitial conditions.
Revised: We link this to a phenomenon we call path independence : the tendencyof an equilibrium network to converge to the same limiting behavior given an input, regardless of theinitial conditions.

Original: We investigate this phenomenon through careful experiments and verify that pathindependent networks indeed generalize well on harder problem instances by exploiting more testtime compute.
Revised: We investigate this phenomenon through careful experiments and verify that pathindependent networks indeed generalize well on harder problem instances by exploiting more testtime compute.

Original: Moreover, interventions on training conditions that promote path independence alsoimprove upwards generalization, while those that penalize it hurt this capability.
Revised: Moreover, interventions on training conditions that promote path independence alsoimprove upwards generalization, while those that penalize it hurt this capability.

Original: Our findings suggestthat path independent equilibrium models are a promising direction towards building general purposelearning systems whose test-time performance improves with larger compute.
Revised: Our findings suggestthat path independent equilibrium models are a promising direction towards building general purposelearning systems whose test-time performance improves with larger compute.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]
Revised: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes]
Revised: [Yes]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: N/A
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: A Alternative Approaches for Quantifying Path Independence
Revised: A Alternative Approaches for Quantifying Path Independence

Original: We review alternative methods for quantifying how path-independent a given equilibrium model is,and why the Fixed Point Alignment score is the most suitable one amongst them for our analyses.
Revised: We review alternative methods for quantifying how path-independent a given equilibrium model is,and why the Fixed Point Alignment score is the most suitable one amongst them for our analyses.

Original: We want a path independence metric to satisfy three criteria: 1) Dimensionless: Metrics computedfrom different training runs should be directly comparable with each other.
Revised: We want a path independence metric to satisfy three criteria: 1) Dimensionless: Metrics computedfrom different training runs should be directly comparable with each other.

Original: To ensure this, one has tomake sure that one uses dimensionless metrics (i.e. doesn’t have units).
Revised: To ensure this, one has tomake sure that one uses dimensionless metrics (i.e. doesn’t have units).

Original: 2) local and global pathindependence: The metric should test for path independence not just in a local neighborhood offixed points, but over a sufficiently large portion of the initialization domain.
Revised: 2) local and global pathindependence: The metric should test for path independence not just in a local neighborhood offixed points, but over a sufficiently large portion of the initialization domain.

Original: 3) efficiency: It shouldbe computationally tractable.
Revised: 3) efficiency: It shouldbe computationally tractable.

Original: With these in mind, we review three alternative ways of quantifying path independence.
Revised: With these in mind, we review three alternative ways of quantifying path independence.

Original: The summaryof the analysis is described in Table :
Revised: The summaryof the analysis is described in Table :

Original: • Jacobian Norm: The Frobenius norm of the Jacobian of the output of an equilibrium model
Revised: • Jacobian Norm: The Frobenius norm of the Jacobian of the output of an equilibrium model

Original: with respect to its fixed point initialization (i.e. || ∂ FIX( x , z 0 ) ∂ z|| 2 ) gives a robust measure of
Revised: with respect to its fixed point initialization (i.e. || ∂ FIX( x , z 0 ) ∂ z|| 2 ) gives a robust measure of

Original: how locally sensitive the network is to initializations.
Revised: how locally sensitive the network is to initializations.

Original: If this quantity is very small (andapproaching 0 with more root finding steps), one can conclude that the given equilibrium
Revised: If this quantity is very small (andapproaching 0 with more root finding steps), one can conclude that the given equilibrium

Original: model is path-independent on the given input.
Revised: model is path-independent on the given input.

Original: The main issues with this approach are: 1)
Revised: The main issues with this approach are: 1)

Original: It’s extremely computationally intensive to compute (since both the input and the output of
Revised: It’s extremely computationally intensive to compute (since both the input and the output of

Original: FIX are high-dimensional, neither forward nor backward differentiation can estimate this
Revised: FIX are high-dimensional, neither forward nor backward differentiation can estimate this

Original: Jacobian efficiently) 2)
Revised: Jacobian efficiently) 2)

Original: It has units, meaning it isn’t comparable across different training
Revised: It has units, meaning it isn’t comparable across different training

Original: runs, 3) It only measures local path independence.
Revised: runs, 3) It only measures local path independence.

Original: • Agreement with Adversarial Initializations: This method quantifies to what extent it is possible to directly optimize initializations such that they result in different fixed points.
Revised: • Agreement with Adversarial Initializations: This method quantifies to what extent it ispossible to directly optimize initializations such that they result in different fixed points.

Original: If
Revised: If

Original: the network is path independent, there shouldn’t exist an adversary that can find adversarial
Revised: the network is path independent, there shouldn’t exist an adversary that can find adversarial

Original: initializations.
Revised: initializations.

Original: The exact procedure for how adversarial fixed points can be found in SectionB.4.
Revised: The exact procedure for how adversarial fixed points can be found in SectionB.4.

Original: The downsides with this approach is that 1) it’s expensive - one has to solve
Revised: The downsides with this approach is that 1) it’s expensive - one has to solve

Original: an
Revised: an

Original: optimization problem for every problem instance, potentially using different optimizers and
Revised: optimization problem for every problem instance, potentially using different optimizers and

Original: initializations, and 2) it only checks for local path independence.
Revised: initializations, and 2) it only checks for local path independence.

Original: • Agreement with Swapped Initializations (Fixed Point Alignment): The main idea
Revised: • Agreement with Swapped Initializations (Fixed Point Alignment): The main idea

Original: behind this approach is to initialize the forward pass of an equilibrium model with fixed
Revised: behind this approach is to initialize the forward pass of an equilibrium model with fixed

Original: points obtained from other problem instances, and checking if the network still displays
Revised: points obtained from other problem instances, and checking if the network still displays

Original: the
Revised: the

Original: same asymptotic behaviour (i
Revised: same asymptotic behaviour (i

Original: e. finds the same fixed point).
Revised: e. finds the same fixed point).

Original: If one uses cosine similarity to measure consistency, then one recovers Fixed Point Alignment score proposed in Section4.2.
Revised: If one uses cosine similarity tomeasure consistency, then one recovers Fixed Point Alignment score proposed in Section4.2.

Original: Another option is to use the average Euclidean distance to check for consistency.
Revised: Another option is to use the average Euclidean distance to check for consistency.

Original: This
Revised: This

Original: is identical to computing the trace of the covariance matrix of the distance between thecomputed fixed points.
Revised: is identical to computing the trace of the covariance matrix of the distance between the computed fixed points.

Original: This approach – especially the one that utilizes cosine similarity to
Revised: This approach – especially the one that utilizes cosine similarity to

Original: quantify consistency – is appealing, as 1) because it’s unitless, it can be compared between
Revised: quantify consistency – is appealing, as 1) because it’s unitless, it can be compared between

Original: training runs on the same task, 2) it checks for non-local convergence by sampling from a
Revised: training runs on the same task, 2) it checks for non-local convergence by sampling from a

Original: distribution of relevant fixed points 3) it’s very efficient, requiring only two forward passes.
Revised: distribution of relevant fixed points 3) it’s very efficient, requiring only two forward passes.

Original: The version that utilizes Euclidean distance does have units, hence cannot be used for cross-
Revised: The version that utilizes Euclidean distance does have units, hence cannot be used for cross-

Original: model comparisons 7 ) To further make sure that this metric is correct, and does subsume
Revised: model comparisons 7 ) To further make sure that this metric is correct, and does subsume

Original: with local path independence, we ran stress tests described in Section 4.2 to make sure
Revised: with local path independence, we ran stress tests described in Section 4.2 to make sure

Original: this metric produces results consistent with the agreement with adversarial initializations
Revised: this metric produces results consistent with the agreement with adversarial initializations

Original: method described above.
Revised: method described above.

Original: Alternatives to Cosine Similarity in Measuring Directional Alignment While cosine similarityis a conceptually clean way of quantifying directional alignment, one can consider alternative kernelsas well.
Revised: Alternatives to Cosine Similarity in Measuring Directional Alignment While cosine similarityis a conceptually clean way of quantifying directional alignment, one can consider alternative kernelsas well.

Original: Ideally, our main results should not depend on the specific implementation details of pathindependence metrics (as long as they satisfy the criteria we set out above).
Revised: Ideally, our main results should not depend on the specific implementation details of pathindependence metrics (as long as they satisfy the criteria we set out above).

Original: To check how sensitive our results are on the precise functional form of the FPA score, we replaced itwith three other kernels and re-assessed whether the results pointed to the same high-level takeaways.
Revised: To check how sensitive our results are on the precise functional form of the FPA score, we replaced itwith three other kernels and re-assessed whether the results pointed to the same high-level takeaways.

Original: Concretely, we tried the Gaussian kernel ϕ g ( r ) =
Revised: Concretely, we tried the Gaussian kernel ϕ g ( r ) =

Original: exp( − ( ϵr ) 2 ) , Laplacian kernel ϕ l ( r ) =
Revised: exp( − ( ϵr ) 2 ) , Laplacian kernel ϕ l ( r ) =

Original: exp( −| ϵr | )and inverse multiquadratic kernel ϕ
Revised: exp( −| ϵr | )and inverse multiquadratic kernel ϕ

Original: i ( r ) = 1 √ 1+( ϵr ) 2 where the input to the kernel function is the
Revised: i ( r ) = 1 √ 1+( ϵr ) 2 where the input to the kernel function is the

Original: Euclidean distance of the normalized fixed points r = ||
Revised: Euclidean distance of the normalized fixed points r = ||

Original: z 1 ||
Revised: z 1 ||

Original: z
Revised: z

Original: || − z
Revised: || − z

Original: 2 || z
Revised: 2 || z

Original: 2 ||
Revised: 2 ||

Original: || .
Revised: || .

Original: We used ϵ = 5000 , whichgave a reasonable dynamic range in the resulting values.
Revised: We used ϵ = 5000 , whichgave a reasonable dynamic range in the resulting values.

Original: The results can be seen in Figure 6.
Revised: The results can be seen in Figure 6.

Original: The takeaways from the plots remain identical: **path independence is strongly correlated withgeneralization, regardless of the specific details of how path independence is quantified** (as long asit satisfies the criteria we set in Appendix A).
Revised: The takeaways from the plots remain identical: **path independence is strongly correlated withgeneralization, regardless of the specific details of how path independence is quantified** (as long asit satisfies the criteria we set in Appendix A).

Original: Importance of using dimensionless metrics:
Revised: Importance of using dimensionless metrics:

Original: Being unitless is necessary for a metric to becomparable across training runs, though obviously not sufficient.
Revised: Being unitless is necessary for a metric to becomparable across training runs, though obviously not sufficient.

Original: Consider two equilibrium models
Revised: Consider two equilibrium models

Original: M1 and M2, where the fixed points computed by M2 have the same direction as those computedby M1, but have twice the Euclidean norm.
Revised: M1 and M2, where the fixed points computed by M2 have the same direction as those computedby M1, but have twice the Euclidean norm.

Original: The behaviour of this M2 is qualitatively the same asthat of M1, but any metric that depends on the Euclidean metric (or Jacobian L2 norm, or any nonunitless metric) would report this network to be less path independent.
Revised: The behaviour of this M2 is qualitatively the same asthat of M1, but any metric that depends on the Euclidean metric (or Jacobian L2 norm, or any nonunitless metric) would report this network to be less path independent.

Original: Note that this is not a purelytheorical consideration: simply adding L2 regularization, or penalizing the magnitude of fixed pointsto encourage convergence will directly impact the scale of the fixed points (things that practitionersoften use), thereby rendering non-unitless metrics unreliable.
Revised: Note that this is not a purelytheorical consideration: simply adding L2 regularization, or penalizing the magnitude of fixed pointsto encourage convergence will directly impact the scale of the fixed points (things that practitionersoften use), thereby rendering non-unitless metrics unreliable.

Original: B Experimental Details
Revised: B Experimental Details

Original: This section contains the additional information needed to reproduce our experimental findings.
Revised: This section contains the additional information needed to reproduce our experimental findings.

Original: Thissection complements our code release.
Revised: Thissection complements our code release.

Original: Path Independence Metrics
Revised: Path Independence Metrics

Original: Dimensionless
Revised: Dimensionless

Original: Local & Global Coverage
Revised: Local & Global Coverage

Original: Efficient
Revised: Efficient

Original: [Table]
Revised: [Table]

Original:  ✗ ✗ ✗
Revised: Jacobian Norm ✗ ✗ ✗

Original: Using Adversarial Initializations ✓✓ ✓ ✗
Revised: Using Adversarial Initializations ✓✓ ✓ ✗

Original: [Table]
Revised: [Table]

Original: ✓✓ ✓ ✓✓
Revised: ✓✓ ✓ ✓✓

Original: B.1 Details for Experiments on Prefix Sum
Revised: B.1 Details for Experiments on Prefix Sum

Revised: [Table]

Original: Unless otherwise specified, all of the prefix-sum networks were trained under the following conditions:
Revised: Unless otherwise specified, all of the prefix-sum networks were trained under the following conditions:

Original: Training details: We used the Adam [Kingma and Ba, 2014] optimizer and trained for 30000gradient steps with a batch size of 150 .
Revised: Training details: We used the Adam [Kingma and Ba, 2014] optimizer and trained for 30000gradient steps with a batch size of 150 .

Original: We varied the initial learning rate on different runs ( 0 . 001 ,0 . 0001 and 0 . 00001 ).
Revised: We varied the initial learning rate on different runs ( 0 . 001 ,0 . 0001 and 0 . 00001 ).

Original: We applied a stepwise learning rate decay both with a factor of 0 .
Revised: We applied a stepwise learning rate decay both with a factor of 0 .

Original: 5 halfwayand three-quarters through the learning.
Revised: 5 halfwayand three-quarters through the learning.

Original: We also used gradient clipping at L2 norm equals 1.
Revised: We also used gradient clipping at L2 norm equals 1.

Original: Model selection: Since we didn’t observe overfitting in our experiments, we used the last modelsaved model checkpoint during evaluation.
Revised: Model selection: Since we didn’t observe overfitting in our experiments, we used the last modelsaved model checkpoint during evaluation.

Original: Training data: We used the length-32 training split of the prefix sum dataset described in
Revised: Training data: We used the length-32 training split of the prefix sum dataset described in

Original: Schwarzschild et al.
Revised: Schwarzschild et al.

Original: [2021a].
Revised: [2021a].

Original: Architecture: We used a 1D-convolutional depth-wise recurrent ResNet architecture roughly analogous to that used by Bansal et al.
Revised: Architecture: We used a 1D-convolutional depth-wise recurrent ResNet architecture roughly analogous to that used by Bansal et al.

Original: [2022] in their prefix sum experiments.
Revised: [2022] in their prefix sum experiments.

Original: The main differences include:(1)
Revised: The main differences include:(1)

Original: We used a single convolutional readout layer, as well as single convolutional layer to right beforethe depth-wise recurrent section of the network.
Revised: We used a single convolutional readout layer, as well as single convolutional layer to right beforethe depth-wise recurrent section of the network.

Original: (2) Unless otherwise specified, we initialized the fixedpoint at the 0 vector, unlike Bansal et al.
Revised: (2) Unless otherwise specified, we initialized the fixedpoint at the 0 vector, unlike Bansal et al.

Original: [2022] who initialized the fixed point vector directly with theinput.
Revised: [2022] who initialized the fixed point vector directly with theinput.

Original: (3) While Bansal et al.
Revised: (3) While Bansal et al.

Original: [2022] uses concatenation to do input injection, we directly added theinput (or a projection of it) to the hidden activations of the residual blocks, the same way the Kolteret al.
Revised: [2022] uses concatenation to do input injection, we directly added theinput (or a projection of it) to the hidden activations of the residual blocks, the same way the Kolteret al.

Original: [2020] does in their convolutional architecture.
Revised: [2020] does in their convolutional architecture.

Original: (4) We ablated using Weight Normalization [Salimans and Kingma, 2016], as this is something that has been reported to stabilize DEQ training.
Revised: (4) We ablated using Weight Normalization [Salimans and Kingma, 2016], as this is something that has been reported to stabilize DEQ training.

Original: Forward solvers: We used either fixed point iterations (of depth 6 or 32 ) or Anderson acceleration [Kolter et al., 2020] (with an iteration count of 6 or 32 and a memory of 3 ) depending on theexperimental purpose of the experiment.
Revised: Forward solvers: We used either fixed point iterations (of depth 6 or 32 ) or Anderson acceleration [Kolter et al., 2020] (with an iteration count of 6 or 32 and a memory of 3 ) depending on theexperimental purpose of the experiment.

Original: Backwards pass: We used either the standard backpropagation algorithm, the IFT gradients to trainthe networks or truncated backpropagation, depending on the purpose of the experiment.
Revised: Backwards pass: We used either the standard backpropagation algorithm, the IFT gradients to trainthe networks or truncated backpropagation, depending on the purpose of the experiment.

Original: We used
Revised: We used

Original: Anderson Acceleration to estimate the IFT gradients with an iteration budget of 10 and memory of3 .
Revised: Anderson Acceleration to estimate the IFT gradients with an iteration budget of 10 and memory of3 .

Original: We found that rescaling the Jacobian term that appears in Equation 1 with a constant less than 1stabilized training with the IFT gradients.
Revised: We found that rescaling the Jacobian term that appears in Equation 1 with a constant less than 1stabilized training with the IFT gradients.

Original: We used a rescaling factor of 0 . 8 .
Revised: We used a rescaling factor of 0 . 8 .

Original: While this is relativelynon-standard practice, we confirmed that using this stabilization strategy doesn’t yield qualitativelydifferent results than those reported in the paper.
Revised: While this is relativelynon-standard practice, we confirmed that using this stabilization strategy doesn’t yield qualitativelydifferent results than those reported in the paper.

Original: When we used truncated backpropagation, wedropped the gradients halfwah through the network (i.e. if the forward pass consisted of 10 iterationsof the recurrent cell, we only backpropagated through the last 5 iterations).
Revised: When we used truncated backpropagation, wedropped the gradients halfwah through the network (i.e. if the forward pass consisted of 10 iterationsof the recurrent cell, we only backpropagated through the last 5 iterations).

Original: Loss function: We used the cross entropy between the network predictions and the labels as the lossfunction.
Revised: Loss function: We used the cross entropy between the network predictions and the labels as the lossfunction.

Original: the prefix-sum task is a sequence-to-sequence problem, and each forward pass consists ofmultiple predictions.
Revised: the prefix-sum task is a sequence-to-sequence problem, and each forward pass consists ofmultiple predictions.

Original: We used the average of all of the cross entropy values computed in a singleforward pass.
Revised: We used the average of all of the cross entropy values computed in a singleforward pass.

Original: Compute resources: All of the prefix sum networks were trained using Intel Xeon Silver CPUSand NVIDIA GPUs (both P100, T4V1 and T4V2 models were used in the experiments).
Revised: Compute resources: All of the prefix sum networks were trained using Intel Xeon Silver CPUSand NVIDIA GPUs (both P100, T4V1 and T4V2 models were used in the experiments).

Original: Trainingone prefix-sum network took roughly 1 to 2 hours depending on how deep the forward pass is, andwhether one is using IFT or backpropagation gradients.
Revised: Trainingone prefix-sum network took roughly 1 to 2 hours depending on how deep the forward pass is, andwhether one is using IFT or backpropagation gradients.

Original: B.1.1 Reproducing Table 1 (Stress-Testing the FPA Score)
Revised: B.

Revised: .1 Reproducing Table Stress-Testing the FPA Score)

Original: The non-path independent network used in this table is a 15 -layer weight-tied input-injected networktrained with backpropagation gradients.
Revised: The non-path independent network used in this table is a 15 -layer weight-tied input-injected networktrained with backpropagation gradients.

Original: We set the maximum number of iterations for fixed pointcomputation to 11 as we found that this choice gives the highest test accuracy on 64 bit prefix sumstrings.
Revised: We set the maximum number of iterations for fixed pointcomputation to 11 as we found that this choice gives the highest test accuracy on 64 bit prefix sumstrings.

Original: While selecting the optimal value for maximum iterations, we increased the value up to 55iterations.
Revised: While selecting the optimal value for maximum iterations, we increased the value up to 55iterations.

Original: The path independent network is a 32 -layer weight-tied input inejcted network trained withbackpropagation gradients.
Revised: The path independent network is a 32 -layer weight-tied input inejcted network trained withbackpropagation gradients.

Original: The maximum iterations for this network are set to 500 .
Revised: The maximum iterations for this network are set to 500 .

Original: We used a batchsize of 1 and set the maximum number of L-BFGS [Liu and Nocedal, 1989] updates per batch to 50for both the networks.
Revised: We used a batchsize of 1 and set the maximum number of L-BFGS [Liu and Nocedal, 1989] updates per batch to 50for both the networks.

Original: We implemented this code in PyTorch and use these values of hyperparameters:lr=1 , tolerance_grad =1e-7, tolerance_change =1e-9, line_search_fn="strong_wolfe" .
Revised: We implemented this code in PyTorch and use these values of hyperparameters:lr=1 , tolerance_grad =1e-7, tolerance_change =1e-9, line_search_fn="strong_wolfe" .

Original: We report FPA scores and accuracy computed on 500 examples.
Revised: We report FPA scores and accuracy computed on 500 examples.

Original: We provide pseudocode in Algorithm
Revised: We provide pseudocode in Algorithm

Original: Box 

Revised: Box

Original: B.2 Details for Experiments on Mazes
Revised: B.2 Details for Experiments on Mazes

Original: Unless otherwise specified, all the networks for mazes were trained under the following conditions:
Revised: Unless otherwise specified, all the networks for mazes were trained under the following conditions:

Original: Training details: We used the Adam [Kingma and Ba, 2014] optimizer with an initial learning rateof 0 .
Revised: Training details: We used the Adam [Kingma and Ba, 2014] optimizer with an initial learning rateof 0 .

Original: 001 and batch size of 50 .
Revised: 001 and batch size of 50 .

Original: We use stepwise learning rate decay with a factor of 0 .
Revised: We use stepwise learning rate decay with a factor of 0 .

Original: 1 applied atepoch number 100 .
Revised: 1 applied atepoch number 100 .

Original: We trained all weight-tied input-injected networks for a total of 150 epochs.
Revised: We trained all weight-tied input-injected networks for a total of 150 epochs.

Original: Weobserved that DEQs needed a longer training time for their validation accuracy to improve.
Revised: Weobserved that DEQs needed a longer training time for their validation accuracy to improve.

Original: All DEQmodels were pretrained for 150 epochs and trained with IFT for another 100 epochs.
Revised: All DEQmodels were pretrained for 150 epochs and trained with IFT for another 100 epochs.

Original: Model selection: We used the checkpoint corresponding to the model with the best validationaccuracy during evaluation.
Revised: Model selection: We used the checkpoint corresponding to the model with the best validationaccuracy during evaluation.

Original: If multiple checkpoints had same validation accuracy, we selected themost recent checkpoint.
Revised: If multiple checkpoints had same validation accuracy, we selected themost recent checkpoint.

Original: Training data: We trained on the training split of 9 × 9 mazes from the dataset described in
Revised: Training data: We trained on the training split of 9 × 9 mazes from the dataset described in

Original: Schwarzschild et al.
Revised: Schwarzschild et al.

Original: [2021a].
Revised: [2021a].

Original: Architecture: We used a 2D-convolutional depth-wise recurrent ResNet architecture analogous tothat used by Bansal et al.
Revised: Architecture: We used a 2D-convolutional depth-wise recurrent ResNet architecture analogous tothat used by Bansal et al.

Original: [2022] in their mazes experiments.
Revised: [2022] in their mazes experiments.

Original: While pretraining DEQs, we do rolloutsfor 32 -iterations.
Revised: While pretraining DEQs, we do rolloutsfor 32 -iterations.

Original: For DEQ trained with phantom gradient in Table 2, we use a damping factor of0 .
Revised: For DEQ trained with phantom gradient in Table 2, we use a damping factor of0 .

Original: 75 and iterate for 2 iterations.
Revised: 75 and iterate for 2 iterations.

Original: Shallow weight-tied input-injected networks were trained with 8layers while the deeper weight-tied input-injected networks used 32 layers.
Revised: Shallow weight-tied input-injected networks were trained with 8layers while the deeper weight-tied input-injected networks used 32 layers.

Original: Both the feedforwardnetworks are 32 -layer deep non-weight tied networks.
Revised: Both the feedforwardnetworks are 32 -layer deep non-weight tied networks.

Original: We ablate on input-injection for these networks.
Revised: We ablate on input-injection for these networks.

Original: We used the original code released by [Bansal et al., 2022] to replicate the results for weighttied input-injected networks trained with progressive training.
Revised: We used the original code released by [Bansal et al., 2022] to replicate the results for weighttied input-injected networks trained with progressive training.

Original: In addition we ablate on weightnormalization [Salimans and Kingma, 2016] for all the architectures—both unrolled and DEQ.
Revised: In addition we ablate on weightnormalization [Salimans and Kingma, 2016] for all the architectures—both unrolled and DEQ.

Original: Wefound that weight normalization helps stabilize the training, especially DEQ models.
Revised: Wefound that weight normalization helps stabilize the training, especially DEQ models.

Original: We found thatusing normalization layers (like group norm [Wu and He, 2018]) hurts generalization in mazes unlikeprefix sum.
Revised: We found thatusing normalization layers (like group norm [Wu and He, 2018]) hurts generalization in mazes unlikeprefix sum.

Original: Therefore, none of our architectures for mazes use any normalization layers.
Revised: Therefore, none of our architectures for mazes use any normalization layers.

Original: Forward solvers: We used either fixed point iterations (of depth 8 or 32 ) or Broyden’s method [Broyden, 1965] (with an iteration count of 40 ) depending on the experimental purpose of the experiment.
Revised: Forward solvers: We used either fixed point iterations (of depth 8 or 32 ) or Broyden’s method [Broyden, 1965] (with an iteration count of 40 ) depending on the experimental purpose of the experiment.

Original: Backward pass: We used the standard backpropagation algorithm, the IFT gradients, truncatedbackpropagation or phantom gradients to train the networks, depending on the purpose of theexperiment.
Revised: Backward pass: We used the standard backpropagation algorithm, the IFT gradients, truncatedbackpropagation or phantom gradients to train the networks, depending on the purpose of theexperiment.

Original: We used Broyden’s method to estimate the IFT gradients with an iteration budget of 40 .
Revised: We used Broyden’s method to estimate the IFT gradients with an iteration budget of 40 .

Original: For experiments with truncated backpropagation, we follow the exact setting as specified in [Bansalet al., 2022].
Revised: For experiments with truncated backpropagation, we follow the exact setting as specified in [Bansalet al., 2022].

Original: Loss function: We used the cross entropy between the network predictions and the labels as the lossfunction.
Revised: Loss function: We used the cross entropy between the network predictions and the labels as the lossfunction.

Original: Compute resources: All networks for mazes were trained using Intel Xeon Silver CPUs and NVIDIA
Revised: Compute resources: All networks for mazes were trained using Intel Xeon Silver CPUs and NVIDIA

Original: GPUs (1080Ti, 2080Ti and A6000 models were used in the experiments).
Revised: GPUs (1080Ti, 2080Ti and A6000 models were used in the experiments).

Original: Training a weight-tiednetwork took roughly around 12 - 18 hours depending on how deep the forward pass is, whereastraining DEQ models took around 2 days of training time on a single GPU.
Revised: Training a weight-tiednetwork took roughly around 12 - 18 hours depending on how deep the forward pass is, whereastraining DEQ models took around 2 days of training time on a single GPU.

Original: B.3 Reproducing Figure
Revised: B.3 Reproducing Figure

Original: Both models were trained using 32 fixed point iterations and backpropagation.
Revised: Both models were trained using 32 fixed point iterations and backpropagation.

Original: The path independentmodel is input injected, the path dependent model is not input injected.
Revised: The path independentmodel is input injected, the path dependent model is not input injected.

Original: We run the network for 32fixed point iterations on five different validation examples of length 32 to obtain different hidden layerinitializations.
Revised: We run the network for 32fixed point iterations on five different validation examples of length 32 to obtain different hidden layerinitializations.

Original: We then run the models for 128 steps using these initializations to obtain five differenttrajectories.
Revised: We then run the models for 128 steps using these initializations to obtain five differenttrajectories.

Original: For the input injected network, we use the same input injection for all trajectories,obtained from a different example of length 32.
Revised: For the input injected network, we use the same input injection for all trajectories,obtained from a different example of length 32.

Original: We project the resulting hidden states onto tworandom orthogonal directions in the hidden layer state space.
Revised: We project the resulting hidden states onto tworandom orthogonal directions in the hidden layer state space.

Original: B.4 Experimental Details for Section 3 (Upwards Generalization with Equilibrium Models)
Revised: B.4 Experimental Details for Section 3 (Upwards Generalization with Equilibrium Models)

Original: B.4.1 Reproducing Figure 2a (OOD Generalization with Mazes)
Revised: B.4.1 Reproducing Figure 2a (OOD Generalization with Mazes)

Original: The training details for all the weight-tied input-injected networks used to produce this plot can befound in Section B.2.
Revised: The training details for all the weight-tied input-injected networks used to produce this plot can befound in Section B.2.

Original: The feedforward network used in this plot is a 32 -layer non-weight tied networkwithout any input injection that was trained with regular backpropagation.
Revised: The feedforward network used in this plot is a 32 -layer non-weight tied networkwithout any input injection that was trained with regular backpropagation.

Original: The maximum number oftest-time iterations (both unroll and Broyden solver) for mazes of size 9 , 13 , 25 , 31 and 59 were 50 ,100 , 400 , 600 , and 800 respectively.
Revised: The maximum number oftest-time iterations (both unroll and Broyden solver) for mazes of size 9 , 13 , 25 , 31 and 59 were 50 ,100 , 400 , 600 , and 800 respectively.

Original: The test accuracy for mazes of a given dimension was computedon the entire split of 10 , 000 mazes as provided in the original dataset [Schwarzschild et al., 2021b].
Revised: The test accuracy for mazes of a given dimension was computedon the entire split of 10 , 000 mazes as provided in the original dataset [Schwarzschild et al., 2021b].

Original: B.4.2 Reproducing Figure 2b (PI networks get better with more iterations)
Revised: B.4.2 Reproducing Figure 2b (PI networks get better with more iterations)

Original: The training details for the 12 networks used to produce this plot can be found in Section B.1.
Revised: The training details for the 12 networks used to produce this plot can be found in Section B.1.

Original: Wevaried (1) the depth ( 6 vs. 32 ), whether weight normalization[Salimans and Kingma, 2016] was usedor not, and the initial learning rate ( 0 . 001 , 0 . 0001 or 0 . 00001 ).
Revised: Wevaried (1) the depth ( 6 vs. 32 ), whether weight normalization[Salimans and Kingma, 2016] was usedor not, and the initial learning rate ( 0 . 001 , 0 . 0001 or 0 . 00001 ).

Original: We evaluated each model on 128 examples of length 16, 32, 64, and 128 each, using fixed pointiterations and measuring the average number of correct bits at each iteration.
Revised: We evaluated each model on 128 examples of length 16, 32, 64, and 128 each, using fixed pointiterations and measuring the average number of correct bits at each iteration.

Original: For each model anditeration step, we plot average accuracies over all splits and examples.
Revised: For each model anditeration step, we plot average accuracies over all splits and examples.

Original: The Fixed Point Alignment
Revised: The Fixed Point Alignment

Original: Scores for each model are calculated as in Figure 4a (see Appendix B.5.1).
Revised: Scores for each model are calculated as in Figure 4a (see Appendix B.5.1).

Original: B.4.3 Reproducing Table1 (Stress-Testing the FPA Score)
Revised: B.4.3 Reproducing Table

Revised: (Stress-Testing the FPA Score)

Original: The non-path independent network used in this table is a 8 -layer weight-tied input-injected networktrained with backpropagation gradients.
Revised: The non-path independent network used in this table is a 8 -layer weight-tied input-injected networktrained with backpropagation gradients.

Original: We set the maximum number of iterations for fixed pointcomputation to 12 as we found that this choice gives the highest test accuracy on 13 × 13 mazes.
Revised: We set the maximum number of iterations for fixed pointcomputation to 12 as we found that this choice gives the highest test accuracy on 13 × 13 mazes.

Original: While selecting the optimal value for maximum iterations, we increased the value up to 60 iterations.
Revised: While selecting the optimal value for maximum iterations, we increased the value up to 60 iterations.

Original: The path independent network is a 32 -layer weight-tied input inejcted network trained with backpropagation gradients.
Revised: The path independent network is a 32 -layer weight-tied input inejcted network trained with backpropagation gradients.

Original: The maximum iterations for this network are set to 500 .
Revised: The maximum iterations for this network are set to 500 .

Original: We used a batch size of 1and set the maximum number of L-BFGS
Revised: We used a batch size of 1and set the maximum number of L-BFGS

Original: [Liu and Nocedal, 1989] updates per batch to 50 for boththe networks.
Revised: [Liu and Nocedal, 1989] updates per batch to 50 for boththe networks.

Original: We implemented this code in PyTorch and use these values of hyperparameters: lr=1 ,tolerance_grad =1e-7, tolerance_change =1e-9, line_search_fn="strong_wolfe" .
Revised: We implemented this code in PyTorch and use these values of hyperparameters: lr=1 ,tolerance_grad =1e-7, tolerance_change =1e-9, line_search_fn="strong_wolfe" .

Original: We report FPA scores and accuracy computed on 500 examples.
Revised: We report FPA scores and accuracy computed on 500 examples.

Original: We provide pseudocode in Algorithm Box 

Revised: We provide pseudocode in Algorithm Box

Original: Algorithm
Revised: Algorithm

Original: Input: A trained network f w , an input x , N max number of LBFGS updates
Revised: Input: A trained network f w , an input x , N max number of LBFGS updates

Original: Define: h ( y 1 , y 2 ) :
Revised: Define: h ( y 1 , y 2 ) :

Original: =
Revised: =

Original: y∥ y 1 ∥ 2 · y∥ y 2 ∥random init( · ): A method that returns a random vector initialized from N ( 0 , I )
Revised: y∥ y 1 ∥ 2 · y∥ y 2 ∥random init( · ): A method that returns a random vector initialized from N ( 0 , I )

Original: Initialize: z = x or random init( x ) ▷ Disable gradients
Revised: Initialize: z = x or random init( x ) ▷ Disable gradients

Original: z 1 = FIX f w ( z , x ) ▷ Clone z 1 and Enable gradients on zfor trials from 1 to N do z 2 =
Revised: z 1 = FIX f w ( z , x ) ▷ Clone z 1 and Enable gradients on zfor trials from 1 to N do z 2 =

Original: FIX f w ( z 1 , x )
Revised: FIX f w ( z 1 , x )

Original: Perform L-BFGS 
Revised: Perform L-BFGS update on z 1 to minimize h ( z 1 , z 2 ) end for

Original: Output: z
Revised: Output: z

Original: B.5 Experimental Details for Section 5 (
Revised: B.5 Experimental Details for Section 5 (

Original: Correlates withUpwardGeneralization)B.5.1 Reproducing Figure 4a (Path Independence - Accuracy Correlation on Prefix Sum)
Revised: with

Revised:  Generalization)B.5.1 Reproducing Figure 4a (Path Independence - Accuracy Correlation on Prefix Sum)

Original: The training details for the 12 networks used to produce this plot can be found in Section B.1.
Revised: The training details for the 12 networks used to produce this plot can be found in Section B.1.

Original: Wetrained 12 networks under each category reported in the plot, where (1) the depth ( 6 vs. 32 ), whetherweight normalization[Salimans and Kingma, 2016] was used or not, and the initial learning rate( 0 . 001 , 0 . 0001 or 0 . 00001 ) were varied.
Revised: Wetrained 12 networks under each category reported in the plot, where (1) the depth ( 6 vs. 32 ), whetherweight normalization[Salimans and Kingma, 2016] was used or not, and the initial learning rate( 0 . 001 , 0 . 0001 or 0 . 00001 ) were varied.

Original: Each value on the plots were computed using 300 examples from the validation splits correspondingto lengths 16 , 32 , 64 , 128 and 256 (i.e. 5 ∗ 300 = 1500 samples per point).
Revised: Each value on the plots were computed using 300 examples from the validation splits correspondingto lengths 16 , 32 , 64 , 128 and 256 (i.e. 5 ∗ 300 = 1500 samples per point).

Original: The average cosinesimilarities involved in computing the Fixed Point Alignment scores were computed using 10 differentinitialization per example.
Revised: The average cosinesimilarities involved in computing the Fixed Point Alignment scores were computed using 10 differentinitialization per example.

Original: For the test-time forward pass, we used 512 iterations of Anderson
Revised: For the test-time forward pass, we used 512 iterations of Anderson

Original: Acceleration [Kolter et al., 2020] (note that this is using a more sophisticated solver using 16 timesmore forward iterations than the deepest of the trained networks).
Revised: Acceleration [Kolter et al., 2020] (note that this is using a more sophisticated solver using 16 timesmore forward iterations than the deepest of the trained networks).

Original: The trend lines on the plots werecomputed using the line that minimizes the least squares error.
Revised: The trend lines on the plots werecomputed using the line that minimizes the least squares error.

Original: B.5.2 Reproducing Figure 4b (Path Independence - Accuracy Correlation on Mazes)
Revised: B.5.2 Reproducing Figure 4b (Path Independence - Accuracy Correlation on Mazes)

Original: The architectural and training details for all the networks in this plot can be found in Section B.2.
Revised: The architectural and training details for all the networks in this plot can be found in Section B.2.

Original: Each value in this plot was computed using 500 examples from the validation splot correspondingto mazes of size 13 and 25.
Revised: Each value in this plot was computed using 500 examples from the validation splot correspondingto mazes of size 13 and 25.

Original: The average cosine similarities involved in computing the Fixed Point
Revised: The average cosine similarities involved in computing the Fixed Point

Original: Alignment scores were computed using 10 different initialization per example.
Revised: Alignment scores were computed using 10 different initialization per example.

Original: Unrolled + backpropnetworks use 100 test-time iterations for the forward pass for mazes of size 13 × 13 , and 400 testtime iterations for mazes of size 25 × 25 .
Revised: Unrolled + backpropnetworks use 100 test-time iterations for the forward pass for mazes of size 13 × 13 , and 400 testtime iterations for mazes of size 25 × 25 .

Original: We set the maximum number of permissible iterations forthe Broyden solver [Broyden, 1965], and for non-weight tied feedforward network to the same values.
Revised: We set the maximum number of permissible iterations forthe Broyden solver [Broyden, 1965], and for non-weight tied feedforward network to the same values.

Original: The shallow unrolled + backprop network was trained using 8 layers.
Revised: The shallow unrolled + backprop network was trained using 8 layers.

Original: Empirically, we obtained thehighest test accuracies on 11 th and 14 th iterations for mazes of size 13 × 13 and 25 × 25 respectively,for networks that use weight normalization.
Revised: Empirically, we obtained thehighest test accuracies on 11 th and 14 th iterations for mazes of size 13 × 13 and 25 × 25 respectively,for networks that use weight normalization.

Original: For networks that do not use weight normalization, thecorresponding numbers are reported on 12 th and 18 th iteration.
Revised: For networks that do not use weight normalization, thecorresponding numbers are reported on 12 th and 18 th iteration.

Original: The trend lines on the plots werecomputed using the line that minimizes the least squares error.
Revised: The trend lines on the plots werecomputed using the line that minimizes the least squares error.

Original: B.6 Experimental Details for Section 6 (Experimental Manipulations of Path Independence)B.6.1 Reproducing Figure 3 (Manipulations of PI)
Revised: B.6 Experimental Details for Section 6 (Experimental Manipulations of Path Independence)B.6.1 Reproducing Figure 3 (Manipulations of PI)

Original: The training details for the 12 networks that formed the baseline for the interventions are described in
Revised: The training details for the 12 networks that formed the baseline for the interventions are described in

Original: Section B.1.
Revised: Section B.1.

Original: We trained 12 networks under each category reported in the plot, where (1) the depth ( 6vs. 32 ), whether weight normalization[Salimans and Kingma, 2016] was used or not, and the initiallearning rate ( 0 . 001 , 0 . 0001 or 0 . 00001 ) were varied.
Revised: We trained 12 networks under each category reported in the plot, where (1) the depth ( 6vs. 32 ), whether weight normalization[Salimans and Kingma, 2016] was used or not, and the initiallearning rate ( 0 . 001 , 0 . 0001 or 0 . 00001 ) were varied.

Original: For each intervention, we kept all of the aforementioned training conditions except for the experimental manipulations that are intended to promote or hurt path independence.
Revised: For each intervention, we kept all of the aforementioned training conditions except for the experimental manipulations that are intended to promote or hurt path independence.

Original: • Mixed initialization: During each training forward pass, each sample was assigned witheither zero initialization (i
Revised: • Mixed initialization: During each training forward pass, each sample was assigned witheither zero initialization (i

Original: e. the fixed point was initialized with the 0 vector) or standardnormal distribution (i.e. ...) using a Bernoulli random variable of probability 0 .
Revised: e. the fixed point was initialized with the 0 vector) or standard normal distribution (i.e. ...) using a Bernoulli random variable of probability 0 .

Original: 5 (i
Revised: 5 (i

Original: e. theexamples that were run with zero vs. normal initializations were roughly half-half.• Randomized depth: Randomized training was done by first specifying the minimum andmaximum possible depths, then uniformly sampling a depth from that range during eachforward pass.
Revised: e. theexamples that were run with zero vs. normal initializations were roughly half-half.• Randomized depth: Randomized training was done by first specifying the minimum andmaximum possible depths, then uniformly sampling a depth from that range during eachforward pass.

Original: We used a range of 3 to 63 forward iterations for training 6 of the networks,and used a range of 32 to 64 for training the other 6 .
Revised: We used a range of 3 to 63 forward iterations for training 6 of the networks,and used a range of 32 to 64 for training the other 6 .

Original: • Alignment penalty:
Revised: • Alignment penalty:

Original: The alignment penalty, which was added to the original trainingloss in form of a regularized was computed as follows: (1) For each example in the batch,compute the fixed points k times using normal initialization, (
Revised: The alignment penalty, which was added to the original training loss in form of a regularized was computed as follows: (1) For each example in the batch,compute the fixed points k times using normal initialization, (

Original: ) Compute the dot productbetween each pair of fixed points (which yields k 2 − k dot products, excluding each fixedpoint’s dot product with itself), return the sum of the dot products, normalized by k 2 − k .
Revised: 2) Compute the dot productbetween each pair of fixed points (which yields k 2 − k dot products, excluding each fixedpoint’s dot product with itself), return the sum of the dot products, normalized by k 2 − k .

Original: Inour experiments, we picked k = 3 .
Revised: Inour experiments, we picked k = 3 .

Original: B.7 Experimental Details for Section 7 (Disambiguating Convergence and Path Independence)B.7.1 Reproducing Table 2 (Training Convergence vs. Accuracy)
Revised: B.7 Experimental Details for Section 7 (Disambiguating Convergence and Path Independence)B.7.1 Reproducing Table 2 (Training Convergence vs. Accuracy)

Original: The architectural and training details for DEQ trained with phantom gradients and with IFT havebeen provided in Section B.2.
Revised: The architectural and training details for DEQ trained with phantom gradients and with IFT havebeen provided in Section B.2.

Original: Both the networks use Broyden solver for the forward pass.
Revised: Both the networks use Broyden solver for the forward pass.

Original: We set thenumber of maximum permissible iterations to 40 for in-distribution examples ( i.e., 9 × 9 mazes) andto 400 for 25 × 25 mazes.
Revised: We set thenumber of maximum permissible iterations to 40 for in-distribution examples ( i.e., 9 × 9 mazes) andto 400 for 25 × 25 mazes.

Original: All the values reported in this table have been computed on 500 examplesfrom the respective splits for in-distribution and out-of-distribution data.
Revised: All the values reported in this table have been computed on 500 examplesfrom the respective splits for in-distribution and out-of-distribution data.

Original: B.7.2 Reproducing Figure 5a (
Revised: B.7.2 Reproducing Figure 5a (

Original:  Convergence vs
Revised: Time Convergence vs.

Revised: -Independence)

Original: The architectural and training details of weight-tied input-injected networks used in this figure havebeen provided in Section B.2.
Revised: The architectural and training details of weight-tied input-injected networks used in this figure havebeen provided in Section B.2.

Original: The reported numbers have been averaged over 10 different runs.
Revised: The reported numbers have been averaged over 10 different runs.

Original: Eachrun uses a batch size of 100.
Revised: Eachrun uses a batch size of 100.

Original: The maximum number of forward iterations is set to 200 for both fixedpoint and Broyden solvers.
Revised: The maximum number of forward iterations is set to 200 for both fixedpoint and Broyden solvers.

Original: B.7.3 Reproducing Figure 5b (Per Instance Path Independence vs. Accuracy)
Revised: B.7.3 Reproducing Figure 5b (Per Instance Path Independence vs. Accuracy)

Original: To compute the histogram, we took the 12 networks trained with the training condition that yieldedthe strongest OOD accuracy (mixed initialization) and computed both the accuracy and FPA scoreson 300 examples for the prefix-sum dataset validation splits of lengths 16 , 32 , 64 , 128 and 256 .
Revised: To compute the histogram, we took the 12 networks trained with the training condition that yieldedthe strongest OOD accuracy (mixed initialization) and computed both the accuracy and FPA scoreson 300 examples for the prefix-sum dataset validation splits of lengths 16 , 32 , 64 , 128 and 256 .

Original: Wegrouped each FPA score based on whether the prediction it’s paired with was correct or not, thenproduced the histogram.
Revised: Wegrouped each FPA score based on whether the prediction it’s paired with was correct or not, thenproduced the histogram.

Original: C Limitations
Revised: C Limitations

Original: Currently, the experimental results are obtained only using two tasks: prefix-sum and mazes.
Revised: Currently, the experimental results are obtained only using two tasks: prefix-sum and mazes.

Original: Testingthe validity of our findings on a more diverse set of tasks would increase our confidence in the factthat the phenomenon reported in this paper is universal.
Revised: Testingthe validity of our findings on a more diverse set of tasks would increase our confidence in the factthat the phenomenon reported in this paper is universal.

Original: Moreover, it is not clear from our analysiswhich tasks would benefit from more test-time compute.
Revised: Moreover, it is not clear from our analysiswhich tasks would benefit from more test-time compute.

Original: While the necessity of additional test-timecompute is obvious for prefix sum and mazes, it’s less clear for perception-like tasks like imageclassification.
Revised: While the necessity of additional test-timecompute is obvious for prefix sum and mazes, it’s less clear for perception-like tasks like imageclassification.

Original: Furthermore, we believe that a more systematic study of how the choice of solver (i.e.
Revised: Furthermore, we believe that a more systematic study of how the choice of solver (i.e.

Original: Anderson (first order) vs. Broyden (second order) solvers) impacts the empirical FPA scores wouldstrengthen our findings.
Revised: Anderson (first order) vs. Broyden (second order) solvers) impacts the empirical FPA scores wouldstrengthen our findings.

Original: We observe, for example, that networks trained with Broyden struggle tofind fixed points at test time when run with fixed-point iterations.
Revised: We observe, for example, that networks trained with Broyden struggle tofind fixed points at test time when run with fixed-point iterations.

Original: Expanding on these findings wouldimprove our analysis and let us make more concrete suggestions to practitioners aiming to use thepath-independence related concepts from this paper.
Revised: Expanding on these findings wouldimprove our analysis and let us make more concrete suggestions to practitioners aiming to use thepath-independence related concepts from this paper.

Original: D Path Independence vs. Accuracy Plots Different Difficulty Levels
Revised: D Path Independence vs. Accuracy Plots Different Difficulty Levels

Original: The path-independence vs. accuracy plots in Figure 4b uses averaged values over different problemdifficulties.
Revised: The path-independence vs. accuracy plots in Figure 4b uses averaged values over different problemdifficulties.

Original: In Figure 7, we show how these correlations look like at different lengths on the prefixsum task.
Revised: In Figure 7, we show how these correlations look like at different lengths on the prefixsum task.

Original: The same figure also shows how the accuracy differs if one does inference using thetraining time forward pass budget.
Revised: The same figure also shows how the accuracy differs if one does inference using thetraining time forward pass budget.

Original: The main takeaways from this plot are:
Revised: The main takeaways from this plot are:

Original: • Out-of-distribution problem lengths (right top of the Figure) do indeed require larger in-
Revised: • Out-of-distribution problem lengths (right top of the Figure) do indeed require larger in-

Original: ference depths - all models perform very poorly when inference is run with training-time
Revised: ference depths - all models perform very poorly when inference is run with training-time

Original: forward pass budget.
Revised: forward pass budget.

Original: • The correlation between path independence and accuracy - as the definition implies - ismore apparent in the very large forward pass limit.
Revised: • The correlation between path independence and accuracy - as the definition implies - ismore apparent in the very large forward pass limit.

Original: In this regime, the correlation persistsfor all lengths.
Revised: In this regime, the correlation persists for all lengths.

Original: • While networks with low PI values using the training-time budget occasionally perform
Revised: • While networks with low PI values using the training-time budget occasionally perform

Original: worse when the test-time compute is increased, that doesn’t happen with path-independentnetworks.
Revised: worse when the test-time compute is increased, that doesn’t happen with path-independentnetworks.

Original: E Intervention Results on Different Difficulty Levels
Revised: E Intervention Results on Different Difficulty Levels

Original: Mirroring Section D, we provide how the accuracy correlates with path independence on 1) differentproblem difficulties ( 32 is in-distribution) and different forward pass budgets in Figure 8.
Revised: Mirroring Section D, we provide how the accuracy correlates with path independence on 1) differentproblem difficulties ( 32 is in-distribution) and different forward pass budgets in Figure 8.

Original: We especially emphasize the mixed initialization results (mixed initialization is one of the interventions that promotes path independence).
Revised: We especially emphasize the mixed initialization results (mixed initialization is one of the interventions that promotes path independence).

Original: Note that some of these networks (half of which are as shallow as 6 layers) do poorly in and out-of-distribution when the forward pass uses the training forwardpass budget.
Revised: Note that some of these networks (half of which are as shallow as 6 layers) do poorly in and out-of-distribution when the forward pass uses the training forwardpass budget.

Original: Unsurprisingly, the FPA scores for these networks are low in this condition.
Revised: Unsurprisingly, the FPA scores for these networks are low in this condition.

Original: However,when additional test-time compute is provided, these networks reach perfect in-distribution accuracy,and near-perfect OOD accuracy (especially on length = 64 split).
Revised: However,when additional test-time compute is provided, these networks reach perfect in-distribution accuracy,and near-perfect OOD accuracy (especially on length = 64 split).

Original: This demonstrates that training timeconvergence is not required for path independence.
Revised: This demonstrates that training timeconvergence is not required for path independence.

Original: We find it surprising that with the help of a pathindependence-promoting regularization procedure, it’s possible to train very shallow (i.e. less than 6layers) networks such that they can exploit additional test-time compute to achieve very high accuracy.
Revised: We find it surprising that with the help of a pathindependence-promoting regularization procedure, it’s possible to train very shallow (i.e. less than 6layers) networks such that they can exploit additional test-time compute to achieve very high accuracy.

Original: F Per-Instance Path Independence Analyses - Convergence vs. PathIndependence
Revised: F Per-Instance Path Independence Analyses - Convergence vs. PathIndependence

Original: In addition to the analysis in Section 8, we also analyzed how test-time convergence correlates withpath independence on a per-instance basis.
Revised: In addition to the analysis in Section 8, we also analyzed how test-time convergence correlates withpath independence on a per-instance basis.

Original: The per-instance FPA score vs. convergence (measured interms of the L2 distance between the last two root solver iterates) can be seen in Figure 9.
Revised: The per-instance FPA score vs. convergence (measured interms of the L2 distance between the last two root solver iterates) can be seen in Figure 9.

Original: We usedthe 12 networks trained using the mixed initialization strategy (since these networks yielded the bestin and OOD performance), and used 300 samples from the length splits 16 , 32 , 64 , 128 and 256 .
Revised: We usedthe 12 networks trained using the mixed initialization strategy (since these networks yielded the bestin and OOD performance), and used 300 samples from the length splits 16 , 32 , 64 , 128 and 256 .

Original: This analysis reaffirms our past observation that test-time convergence is not needed for path independence.
Revised: This analysis reaffirms our past observation that test-time convergence is not needed for path independence.

Original: The plot contains data from three different regimes:
Revised: The plot contains data from three different regimes:

Original: • Full convergence to a fixed point: These datapoints, observed on the right bottom part ofFigure 9, corresponds to samples where the solver nearly converges to a fixed point.
Revised: • Full convergence to a fixed point: These datapoints, observed on the right bottom part ofFigure 9, corresponds to samples where the solver nearly converges to a fixed point.

Original: This isalmost always associated with high FPA scores, and good per-instance accuracy. • Limit cycles: These datapoints, corresponding to the right-top part of Figure 9, correspondto cases where the solver doesn’t converge to a fixed points, but enters an orbit aroundit.
Revised: This isalmost always associated with high FPA scores, and good per-instance accuracy.• Limit cycles: These datapoints, corresponding to the right-top part of Figure 9, correspondto cases where the solver doesn’t converge to a fixed points, but enters an orbit aroundit.

Original: FPA scores, as well as the per-instance accuracy values remain high in this regimeas well, indicating that test-time convergence (to a fixed point) is not necessary for pathindependence, and good accuracy.
Revised: FPA scores, as well as the per-instance accuracy values remain high in this regimeas well, indicating that test-time convergence (to a fixed point) is not necessary for pathindependence, and good accuracy.

Original: Note that the boundary between the full-convergenceregime and the limit cycle regime is gradual.• Divergence:
Revised: Note that the boundary between the full-convergenceregime and the limit cycle regime is gradual.• Divergence:

Original: On a number of the samples, the solver diverges.
Revised: On a number of the samples, the solver diverges.

Original: These are associated with highresiduals, low FPA scores and low per-instance accuracies.
Revised: These are associated with highresiduals, low FPA scores and low per-instance accuracies.

Original: This shows that the main sourceof per-instance lack of path independence on networks that are otherwise overwhelminglypath independent (on other samples) is almost always solver divergence.
Revised: This shows that the main sourceof per-instance lack of path independence on networks that are otherwise overwhelminglypath independent (on other samples) is almost always solver divergence.

Original: G Test Time Convergence and Path independence
Revised: G Test Time Convergence and Path independence

Original: We provide a per-instance analysis of test-time convergence in Figure 10.
Revised: We provide a per-instance analysis of test-time convergence in Figure 10.

Original: We display plots for perinstance residuals i.e. ∥ f ( x, z ) − z ∥ 2 for Broyden’s method and naive fixed point iterations oversolver steps.
Revised: We display plots for perinstance residuals i.e. ∥ f ( x, z ) − z ∥ 2 for Broyden’s method and naive fixed point iterations oversolver steps.

Original: In addition, we also plot L2 norm between the fixed points of these solvers at everystep.
Revised: In addition, we also plot L2 norm between the fixed points of these solvers at everystep.

Original: We provide plots for both in-distribution data i.e. 9 × 9 mazes and on more difficult probleminstances i.e. 13 × 13 and 25 × 25 mazes.
Revised: We provide plots for both in-distribution data i.e. 9 × 9 mazes and on more difficult probleminstances i.e. 13 × 13 and 25 × 25 mazes.

Original: Across all the mazes, we observe that there are probleminstances where both the solvers converge to the same limiting behavior (as indicated by low valuesof residuals and L2 norms).
Revised: Across all the mazes, we observe that there are probleminstances where both the solvers converge to the same limiting behavior (as indicated by low valuesof residuals and L2 norms).

Original: However, there are a considerable number of points where naive fixedpoint iterations converge to a limit cycle but still output correct predictions.
Revised: However, there are a considerable number of points where naive fixedpoint iterations converge to a limit cycle but still output correct predictions.

Original: This behavior can beseen on both in-distribution data and on harder problem instances.
Revised: This behavior can beseen on both in-distribution data and on harder problem instances.

Original: We find that the absolute residualsbetween the points in the limit cycles are a small percentage of the Euclidean norms of the points:0.9% for 9x9 mazes, 0.49% for 13x13 mazes and 1.5% for 25x25 mazes which indicates that theselimit cycles are localized.
Revised: We find that the absolute residualsbetween the points in the limit cycles are a small percentage of the Euclidean norms of the points:0.9% for 9x9 mazes, 0.49% for 13x13 mazes and 1.5% for 25x25 mazes which indicates that theselimit cycles are localized.

Original: These examples of problem instances where both the solvers convergeto different limiting behavior reaffirm our observation that convergence to the same fixed point attest-time is not a necessary condition for path independence.
Revised: These examples of problem instances where both the solvers convergeto different limiting behavior reaffirm our observation that convergence to the same fixed point attest-time is not a necessary condition for path independence.

Original: We used the 12 networks trained using the mixed initialization strategy (since these networks yielded the best in and OOD performance), and used 300 samples from the length splits 16 , 32 , 64 , 128 and 256 .
Revised: We used the 12 networks trained using the mixed initialization strategy (since these networks yielded the best in and OOD performance), and used 300 samples from the length splits 16 , 32 , 64 , 128 and 256 .

Original: While good convergence is almost always associated with high FPA values, the converse is not necessarily true: there are many samples on which convergence is poor, yet the FPA score is high.
Revised: While good convergence is almost always associated with high FPA values, the converse is not necessarily true: there are many samples on which convergence is poor, yet the FPA score is high.

Original: H Results on the Blurry MNIST task
Revised: H Results on the Blurry MNIST Task

Original: We tested our path-independence hypothesis on task we call BlurryMNIST [Liang et al.].
Revised: We tested our path-independence hypothesis on task we call BlurryMNIST [Liang et al.].

Original: This taskinvolves training an image classifier on MNIST digits corrupted with small degrees of Gaussian blur,and testing the performance on significantly more corrupted ones.
Revised: This taskinvolves training an image classifier on MNIST digits corrupted with small degrees of Gaussian blur,and testing the performance on significantly more corrupted ones.

Original: The blur corruption is implementedby convolving each image with Gaussian filters of differing standard deviations.
Revised: The blur corruption is implementedby convolving each image with Gaussian filters of differing standard deviations.

Original: We used standarddeviations from 2 to 5 .
Revised: We used standarddeviations from 2 to 5 .

Original: 5 (in increments of 0 . 5 ) to generate each split.
Revised: 5 (in increments of 0 . 5 ) to generate each split.

Original: This dataset allows for testingin and out-of-distribution performance by way of training on a subset of the splits (i.e. the four lowestblur splits) and testing performance on all splits.
Revised: This dataset allows for testingin and out-of-distribution performance by way of training on a subset of the splits (i.e. the four lowestblur splits) and testing performance on all splits.

Original: We trained a number of fully connected equilibrium models on the BlurryMNIST task and inspectedwhether path independence still correlates with accuracy.
Revised: We trained a number of fully connected equilibrium models on the BlurryMNIST task and inspectedwhether path independence still correlates with accuracy.

Original: The results can be found in Figure 11.
Revised: The results can be found in Figure 11.

Original: Thecorrelation we reported in the main body of the paper also holds in the BlurryMNIST dataset.
Revised: Thecorrelation we reported in the main body of the paper also holds in the BlurryMNIST dataset.

Revised: Notethat the in-distribution error rates of the trained models vary between 1 and 5 percent.

Revised: I Results on Matrix Inverse Task

Revised: We also tested the connection between path independence and generalization on the Matrix Inversiontask proposed by Du et al.

Revised: This task is concerned with learning to invert 20x20 matrices.

Revised: Success is defined by how well the trained model works on matrices with worse condition numbersthan those observed during training.

Original: [Figure]
Revised: [Figure]

Revised: We trained a number of equilibrium models where we varied 1) the forward pass (fixed point iterationsvs.

Revised: solver) 2) the backwards pass (backprop gradients or implicit gradients) 3) learning rate ( 0 . 001 ,0 . 0001 and 0 . 00001 ), 4) learning rate schedule (step decays of magnitude 0 .

Revised: 5 at different pointsduring training) and 5) whether layernorm Ba et al.

Revised: [2016] was used or not.

Revised: The results can be seen in

Revised: Figure 12.

Revised: We see a similar pattern that we saw on earlier tasks: lack of path independence correlateswith poor generalization.

Revised: Note that our best model matches the performance of the energy basedmodel approach proposed by Du et al.

Revised: [2022] in the matrix inversion task and significantly beats their baselines.