Original: Under review at the ICLR 2022 workshop on Objects, Structure and Causality

Original: O BJECT R EPRESENTATIONS AS F IXED P OINTS : T RAINING I TERATIVE I NFERENCE A LGORITHMS WITH I MPLICIT D IFFERENTIATION
Revised: O BJECT R EPRESENTATIONS AS F IXED P OINTS : T RAINING I TERATIVE I NFERENCE A LGORITHMS WITH I MPLICIT D IFFERENTIATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep generative models, particularly those that aim to factorize the observations into discrete entities (such as objects), must often use iterative inference procedures that break symmetries among equally plausible explanations for the data.
Revised: Deep generative models, particularly those that aim to factorize the observations into discrete entities (such as objects), must often use iterative inference procedures that break symmetries among equally plausible explanations for the data.

Original: Such inference procedures include variants of the expectation-maximization algorithm and structurally resemble clustering algorithms in a latent space.
Revised: Such inference procedures include variants of the expectation-maximization algorithm and structurally resemble clustering algorithms in a latent space.

Original: However, combining such methods with deep neural networks necessitates differentiating through the inference process, which can make optimization exceptionally challenging.
Revised: However, combining such methods with deep neural networks necessitates differentiating through the inference process, which can make optimization exceptionally challenging.

Original: We observe that such iterative amortized inference methods can be made differentiable by means of the implicit function theorem, and develop an implicit differentiation approach that improves the stability and tractability of training such models by decoupling the forward and backward passes.
Revised: We observe that such iterative amortized inference methods can be made differentiable by means of the implicit function theorem, and develop an implicit differentiation approach that improves the stability and tractability of training such models by decoupling the forward and backward passes.

Original: This connection enables us to apply recent advances in optimizing implicit layers to not only improve the stability and optimization of the slot attention module in SLATE, a state-of-the-art method for learning entity representations, but do so with constant space and time complexity in backpropagation and only one additional line of code.
Revised: This connection enables us to apply recent advances in optimizing implicit layers to not only improve the stability and optimization of the slot attention module in SLATE, a state-of-the-art method for learning entity representations, but do so with constant space and time complexity in backpropagation and only one additional line of code.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: A major goal of building human-level artificial intelligence is that of replicating how humans abstract experience into coherent entities that are used for high-level reasoning.
Revised: A major goal of building human-level artificial intelligence is that of replicating how humans abstract experience into coherent entities that are used for high-level reasoning.

Original: A concrete setting for studying this question has been in so-called object-centric learning , which seeks to decompose observations x into a set of independent representations of entities without supervision on how to decompose.
Revised: A concrete setting for studying this question has been in so-called object-centric learning , which seeks to decompose observations x into a set of independent representations of entities without supervision on how to decompose.

Original: Each datapoint x n (e.g. image) is modeled as a set of independent sensor measurements x n,m (e.g. pixels) which are generally posited as having been generated from a mixture model whose components represent the entities.
Revised: Each datapoint x n (e.g. image) is modeled as a set of independent sensor measurements x n,m (e.g. pixels) which are generally posited as having been generated from a mixture model whose components represent the entities.

Original: Under a clustering lens, the problem reduces to finding the K groups of cluster parameters θ n := { θ n,k } Kk =1 and cluster assignments ϕ n,m := { ϕ n,m,k } Kk =1 that were responsible for the measurements x n,m of the datapoint x n .
Revised: Under a clustering lens, the problem reduces to finding the K groups of cluster parameters θ n := { θ n,k } Kk =1 and cluster assignments ϕ n,m := { ϕ n,m,k } Kk =1 that were responsible for the measurements x n,m of the datapoint x n .

Original: Modeling entities as cluster components encodes the assumption that entities are independent and symmetric , thereby requiring a mechanism for breaking symmetry equally valid explanations of the same observation.
Revised: Modeling entities as cluster components encodes the assumption that entities are independent and symmetric , thereby requiring a mechanism for breaking symmetry equally valid explanations of the same observation.

Original: Thus inference is typically done by breaking symmetry via a set of random initial guesses θ n,k and then iteratively updating θ n and ϕ n,m during the course of execution.
Revised: Thus inference is typically done by breaking symmetry via a set of random initial guesses θ n,k and then iteratively updating θ n and ϕ n,m during the course of execution.

Original: Methods that learn the update rule as a deep network f w are instances of iterative amortized inference (Marino et al., 2018).
Revised: Methods that learn the update rule as a deep network f w are instances of iterative amortized inference (Marino et al., 2018).

Original: The state-of-the-art slot attention module (Locatello et al., 2020), e.g., computes θ nt +1 ← f ( θ nt , x n ) , where ϕ n,m is updated as an intermediate step inside f w .
Revised: The state-of-the-art slot attention module (Locatello et al., 2020), e.g., computes θ nt +1 ← f ( θ nt , x n ) , where ϕ n,m is updated as an intermediate step inside f w .

Original: The θ n , called slots , serve as input to a downstream objective, e.g. image reconstruction.
Revised: The θ n , called slots , serve as input to a downstream objective, e.g. image reconstruction.

Original: Other instances of this approach include (Greff et al., 2017; Van Steenkiste et al., 2018; Greff et al., 2019; Veerapaneni et al., 2020; Locatello et al., 2020; Kipf et al., 2021;
Revised: Other instances of this approach include (Greff et al., 2017; Van Steenkiste et al., 2018; Greff et al., 2019; Veerapaneni et al., 2020; Locatello et al., 2020; Kipf et al., 2021;

Original: Zoran et al., 2021; Singh et al., 2021).
Revised: Zoran et al., 2021; Singh et al., 2021).

Original: All differentiate through the unrolling of f w .
Revised: All differentiate through the unrolling of f w .

Original: Despite their conceptual elegance, it has been difficult to scale such iterative amortized inference methods beyond modeling simple static scenes or short video sequences because differentiating through the unrolled forward iteration makes training unstable.
Revised: Despite their conceptual elegance, it has been difficult to scale such iterative amortized inference methods beyond modeling simple static scenes or short video sequences because differentiating through the unrolled forward iteration makes training unstable.

Original: Fig.
Revised: Fig.

Original: 5 shows that the spectral norm of the Jacobian of f w gradually increases over the course of training, which has empirically been observed to cause training instabilities (Bai et al., 2021).
Revised: 5 shows that the spectral norm of the Jacobian of f w gradually increases over the course of training, which has empirically been observed to cause training instabilities (Bai et al., 2021).

Original: Such instabilities result in sensitivity to hyperparameter choices (e.g., number of inference iterations) and have motivated adding optimization tricks such as gradient clipping, learning rate warm-up, and learning rate decay, all of which make such models more complex and harder to use, restrict the model from optimizing its learning objective fully, and only temporarily delay instabilities that still emerge in later stages of training.
Revised: Such instabilities result in sensitivity to hyperparameter choices (e.g., number of inference iterations) and have motivated adding optimization tricks such as gradient clipping, learning rate warm-up, and learning rate decay, all of which make such models more complex and harder to use, restrict the model from optimizing its learning objective fully, and only temporarily delay instabilities that still emerge in later stages of training.

Original: To solve this problem, we observe that previous methods have not taken advantage of the fact that f w can be viewed as a fixed point operation.
Revised: To solve this problem, we observe that previous methods have not taken advantage of the fact that f w can be viewed as a fixed point operation.

Original: Thus, f w can be trained with implicit differentiation applied at the fixed point, without backpropagating gradients through the unrolled iterations.
Revised: Thus, f w can be trained with implicit differentiation applied at the fixed point, without backpropagating gradients through the unrolled iterations.

Original: Our primary contribution is to propose implicit differentiation for training the iterative amortized inference procedures of symmetric generative models, such as those used for learning object representations.
Revised: Our primary contribution is to propose implicit differentiation for training the iterative amortized inference procedures of symmetric generative models, such as those used for learning object representations.

Original: Specifically, we show across three datasets that, compared to the latest state-of-the-art of these methods, SLATE (Singh et al., 2021), our method for training SLATE achieves much lower validation loss in training, as well as lower Fr´echet inception distance (FID) (Heusel et al., 2017) and mean squared error (MSE) in image reconstruction.
Revised: Specifically, we show across three datasets that, compared to the latest state-of-the-art of these methods, SLATE (Singh et al., 2021), our method for training SLATE achieves much lower validation loss in training, as well as lower Fr ´ echet inception distance (FID) (Heusel et al., 2017) and mean squared error (MSE) in image reconstruction.

Original: Our method also removes the need for gradient clipping, learning rate decay, learning rate warmup, or tuning the number of iterations, while achieving lower space and time complexity in the backward pass, all with just one additional line of code.
Revised: Our method also removes the need for gradient clipping, learning rate decay, learning rate warmup, or tuning the number of iterations, while achieving lower space and time complexity in the backward pass, all with just one additional line of code.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Implicit differentiation is a technique for computing the gradients of a function defined in terms of satisfying a joint condition of the input and output.
Revised: Implicit differentiation is a technique for computing the gradients of a function defined in terms of satisfying a joint condition of the input and output.

Original: For example, a fixed point operation f is defined to satisfy “find λ such that λ
Revised: For example, a fixed point operation f is defined to satisfy “find λ such that λ

Original: = f ( x, λ ) ,” rather than through an explicit parameterization of f .
Revised: = f ( x, λ ) ,” rather than through an explicit parameterization of f .

Original: This fixed point λ ∗ can be computed by simply repeatedly applying f or by using a black-box root-finding solver.
Revised: This fixed point λ ∗ can be computed by simply repeatedly applying f or by using a black-box root-finding solver.

Original: Letting f w be parameterized by weights w , with input x and fixed point λ ∗ , the implicit function theorem (Cauchy, 1831) enables us to directly compute the gradient of the loss ℓ with respect to w , using only the output λ ∗ :[Equation]
Revised: Letting f w be parameterized by weights w , with input x and fixed point λ ∗ , the implicit function theorem (Cauchy, 1831) enables us to directly compute the gradient of the loss ℓ with respect to w , using only the output λ ∗ :

Original: [Equation]
Revised: [Equation]

Original: where J f w ( λ ∗ ) is the Jacobian matrix of f w evaluated at λ ∗ .
Revised: where J f w ( λ ∗ ) is the Jacobian matrix of f w evaluated at λ ∗ .

Original: Compared to backpropagating through the unrolled iteration of f , which is just one of many choices of the solver, implicit differentiation via
Revised: Compared to backpropagating through the unrolled iteration of f , which is just one of many choices of the solver, implicit differentiation via

Original: Eq.
Revised: Eq.

Original: 1 removes the memory cost of storing any intermediate results from the unrolled iteration.
Revised: 1 removes the memory cost of storing any intermediate results from the unrolled iteration.

Original: Much effort has been put into approximating the inverse-Jacobian term ( I − J f w ( λ ∗ )) − 1 which has
Revised: Much effort has been put into approximating the inverse-Jacobian term ( I − J f w ( λ ∗ )) − 1 which has

Original: O ( n 3 ) complexity to compute.
Revised: O ( n 3 ) complexity to compute.

Original: Geng et al.
Revised: Geng et al.

Original: ; Fung et al.
Revised: ; Fung et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: (2021); Shaban et al.
Revised: (2021); Shaban et al.

Original: (2019) propose instead to approximate ( I − J f w ( λ ∗ )) − 1 with its Neumann series expansion:
Revised: (2019) propose instead to approximate ( I − J [Equation] ) − 1 with its Neumann series expansion:

Original: [Equation]
Revised: [Equation]

Original: The first-order approximation ( T = 1 ) amounts to applying f once to the fixed point λ ∗ and differentiating through the resulting computation graph.
Revised: The first-order approximation ( T = 1 ) amounts to applying f once to the fixed point λ ∗ and differentiating through the resulting computation graph.

Original: This is not only cheap to compute and easy to implement, but has also been shown empirically (Geng et al., 2021) to have a regularizing effect on the spectral norm of J f w without sacrificing performance.
Revised: This is not only cheap to compute and easy to implement, but has also been shown empirically (Geng et al., 2021) to have a regularizing effect on the spectral norm of J f w without sacrificing performance.

Original:  I MPLICIT O BJECT -C ENTRIC L EARNING
Revised:  I MPLICIT O BJECT -C ENTRIC L EARNING

Original: [Figure]
Revised: [Figure]

Original: By recognizing that slot attention implements a fixed point operation
Revised: By recognizing that slot attention implements a fixed point operation

Original: [Figure]
Revised: [Figure]

Original: we propose implicit slot attention : a method for training the state-of-theart slot attention module (Locatello et al., 2020), with the simplest and most effective method that we have empirically found for approximating the implicit gradient, which is its first-order Neumann approximation, although in principle any black box solver for computing the fixed point and black box gradient estimator for computing the implicit gradient can be used.
Revised: we propose implicit slot attention : a method for training the state-of-theart slot attention module (Locatello et al., 2020), with the simplest and most effective method that we have empirically found for approximating the implicit gradient, which is its first-order Neumann approximation, although in principle any black box solver for computing the fixed point and black box gradient estimator for computing the implicit gradient can be used.

Original: It can be implemented by simply differentiating the computation graph of applying the slot attention update once to the fixed point θ n ∗ , where θ n ∗ is computed by simply iterating the slot attention module forward as usual, but without the gradient tape.
Revised: It can be implemented by simply differentiating the computation graph of applying the slot attention update once to the fixed point θ n ∗ , where θ n ∗ is computed by simply iterating the slot attention module forward as usual, but without the gradient tape.

Original: The time and space complexity of backpropagation for our method compared to vanilla slot attention as a function of the number of slot attention iterations n , is shown below:
Revised: The time and space complexity of backpropagation for our method compared to vanilla slot attention as a function of the number of slot attention iterations n , is shown below:

Original: [Equation]
Revised: [Equation]

Original: Our method is not only more efficient but also requires only one additional line of code (Fig. 2).
Revised: Our method is not only more efficient but also requires only one additional line of code (Fig. 2).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The main hypothesis behind this paper is that implicit differentiation can improve the training of iterative amortized inference methods for object-centric learning.
Revised: The main hypothesis behind this paper is that implicit differentiation can improve the training of iterative amortized inference methods for object-centric learning.

Original: We test this hypothesis by replacing the backward pass of the slot attention module in SLATE (Singh et al., 2021) with the first-order
Revised: We test this hypothesis by replacing the backward pass of the slot attention module in SLATE (Singh et al., 2021) with the first-order

Original: Neumann approximation of the implicit gradient, and measuring optimization performance.
Revised: Neumann approximation of the implicit gradient, and measuring optimization performance.

Original: For the task of image reconstruction, SLATE uses a discrete VAE (Ramesh et al., 2021) to compress an input image into a grid of discrete tokens.
Revised: For the task of image reconstruction, SLATE uses a discrete VAE (Ramesh et al., 2021) to compress an input image into a grid of discrete tokens.

Original: These tokens index into a codebook of latent code-vectors, which, after applying a learned position encoding, serve as the input to the slot attention module.
Revised: These tokens index into a codebook of latent code-vectors, which, after applying a learned position encoding, serve as the input to the slot attention module.

Original: An Image GPT decoder (Chen et al., 2020) is trained with a cross-entropy loss to autoregressively reconstruct the latent code-vectors, using the outputted slots from slot attention as queries and the latent code-vectors as keys/values.
Revised: An Image GPT decoder (Chen et al., 2020) is trained with a cross-entropy loss to autoregressively reconstruct the latent code-vectors, using the outputted slots from slot attention as queries and the latent code-vectors as keys/values.

Original: Gradients are blocked from flowing in and out of the discrete VAE to the rest of the network (i.e. the slot attention module and the Image GPT decoder), but the entire system is trained simultaneously.
Revised: Gradients are blocked from flowing in and out of the discrete VAE to the rest of the network (i.e. the slot attention module and the Image GPT decoder), but the entire system is trained simultaneously.

Original: We consider three datasets: CLEVR-Mirror (Singh et al., 2021), Shapestacks (Groth et al., 2018), and COCO-2017 (Lin et al., 2014).
Revised: We consider three datasets: CLEVR-Mirror (Singh et al., 2021), Shapestacks (Groth et al., 2018), and COCO-2017 (Lin et al., 2014).

Original: We obtained CLEVR-Mirror directly from the SLATE authors and used a 70-15-15 split for training, validation, and testing.
Revised: We obtained CLEVR-Mirror directly from the SLATE authors and used a 70-15-15 split for training, validation, and testing.

Original: We pooled all the data variants of
Revised: We pooled all the data variants of

Original: Shapestacks together as Singh et al.
Revised: Shapestacks together as Singh et al.

Original: (2021) did and used the original train-validation-test splits.
Revised: (2021) did and used the original train-validation-test splits.

Original: The
Revised: The

Original: COCO-2017 dataset was downloaded from FiftyOne and used the original train-validation-test splits.
Revised: COCO-2017 dataset was downloaded from FiftyOne and used the original train-validation-test splits.

Original: 4. D OES IMPLICIT DIFFERENTIATION STABILIZE THE TRAINING OF SLOT ATTENTION ?
Revised: 4. D OES IMPLICIT DIFFERENTIATION STABILIZE THE TRAINING OF SLOT ATTENTION ?

Original: Using the two primary metrics used in Singh et al.
Revised: Using the two primary metrics used in Singh et al.

Original: (2021), images generated by SLATE trained with implicit differentiation achieve both lower pixel-wise mean-squared error and FID score (Heusel et al., 2017).
Revised: (2021), images generated by SLATE trained with implicit differentiation achieve both lower pixel-wise mean-squared error and FID score (Heusel et al., 2017).

Original: The FID score was computed with the PyTorch-Ignite (Fomin et al., 2020) library using the inception network from the PyTorch port of the FID official implementation.
Revised: The FID score was computed with the PyTorch-Ignite (Fomin et al., 2020) library using the inception network from the PyTorch port of the FID official implementation.

Original: All methods were trained
Revised: All methods were trained

Original: [Figure]
Revised: [Figure]

Original: for 250k gradient steps.
Revised: for 250k gradient steps.

Original: Table 1 compares the FID and MSE scores of the images that result from compressing the SLATE encoder’s set of discrete tokens through the slot attention bottleneck, using
Revised: Table 1 compares the FID and MSE scores of the images that result from compressing the SLATE encoder’s set of discrete tokens through the slot attention bottleneck, using

Original: Image-GPT to autoregressively re-generate these image tokens one by one, and using the discrete
Revised: Image-GPT to autoregressively re-generate these image tokens one by one, and using the discrete

Original: VAE decoder to render the generated image tokens.
Revised: VAE decoder to render the generated image tokens.

Original: Implicit differentiation significantly improves the quantitative image reconstruction metrics of SLATE across the test sets of CLEVR-Mirrors,
Revised: Implicit differentiation significantly improves the quantitative image reconstruction metrics of SLATE across the test sets of CLEVR-Mirrors,

Original: Shapestacks, and COCO.
Revised: Shapestacks, and COCO.

Original: In the case of MSE for CLEVR, this is almost a 7x improvement.
Revised: In the case of MSE for CLEVR, this is almost a 7x improvement.

Original: [Table]
Revised: [Table]

Original: The higher quantitiatve metrics also translate into better quality reconstructions on the test set, as shown in Figure 3.
Revised: The higher quantitiatve metrics also translate into better quality reconstructions on the test set, as shown in Figure 3.

Original: For CLEVR-Mirrors, vanilla SLATE sometimes drops or changes the appearance of objects, even simple scenes with three objects.
Revised: For CLEVR-Mirrors, vanilla SLATE sometimes drops or changes the appearance of objects, even simple scenes with three objects.

Original: In contrast, the reconstructions produced from training with implicit differentiation match the ground truth very closely.
Revised: In contrast, the reconstructions produced from training with implicit differentiation match the ground truth very closely.

Original: For
Revised: For

Original: Shapestacks, our method consistently segements the scene into constituent objects.
Revised: Shapestacks, our method consistently segements the scene into constituent objects.

Original: This is sometimes the case with vanilla SLATE on the training and validation set as well, but we observed for both of the seeds we ran that vanilla SLATE produced degenerated attention maps where one slot captures the entire foreground, and the background is divided among the other slots.
Revised: This is sometimes the case with vanilla SLATE on the training and validation set as well, but we observed for both of the seeds we ran that vanilla SLATE produced degenerated attention maps where one slot captures the entire foreground, and the background is divided among the other slots.

Original: The visual complexity of the COCO dataset is much higher than either CLEVR-Mirrors and Shapestacks, and the reconstructions on the COCO dataset are quite poor, for both SLATE’s discrete VAE and consequently for the reconstruction through the slot bottleneck.
Revised: The visual complexity of the COCO dataset is much higher than either CLEVR-Mirrors and Shapestacks, and the reconstructions on the COCO dataset are quite poor, for both SLATE’s discrete VAE and consequently for the reconstruction through the slot bottleneck.

Original: This may be expected because we did not attempt to tune SLATE’s hyperparameters to COCO, but it does highlight the gap that still exists between using the state-of-the-art in object-centric learning out-of-the-box and what the community may want these methods to do.
Revised: This may be expected because we did not attempt to tune SLATE’s hyperparameters to COCO, but it does highlight the gap that still exists between using the state-of-the-art in object-centric learning out-of-the-box and what the community may want these methods to do.

Original: The attention masks for both the vanilla SLATE and our method furthermore do not appear to correspond consistently to coherent objects in COCO but rather patches on the image that do not immediately seem to match with our human intuition of what constitutes a visual entity.
Revised: The attention masks for both the vanilla SLATE and our method furthermore do not appear to correspond consistently to coherent objects in COCO but rather patches on the image that do not immediately seem to match with our human intuition of what constitutes a visual entity.

Original: 4. C AN WE SIMPLIFY THE NEED FOR OPTIMIZATION TRICKS ?
Revised: 4. C AN WE SIMPLIFY THE NEED FOR OPTIMIZATION TRICKS ?

Original: To further understand the benefits of implicit differentiation, we then ask whether it stabilizes the training of slot attention without the need for optimization tricks like learning rate decay, gradient clipping, and learning warmup.
Revised: To further understand the benefits of implicit differentiation, we then ask whether it stabilizes the training of slot attention without the need for optimization tricks like learning rate decay, gradient clipping, and learning warmup.

Original: Fig.
Revised: Fig.

Original: 4 shows that these tricks generally help regularize spectral norm of the Jacobian of vanilla slot attention but are not required by our method.
Revised: 4 shows that these tricks generally help regularize spectral norm of the Jacobian of vanilla slot attention but are not required by our method.

Original: Decaying the learning rate regularizes the Jacobian norm from exploding, but it also hurts optimization performance for both our method and vanilla SLATE, as expected.
Revised: Decaying the learning rate regularizes the Jacobian norm from exploding, but it also hurts optimization performance for both our method and vanilla SLATE, as expected.

Original: When we remove gradient clipping, the Jacobian norm
Revised: When we remove gradient clipping, the Jacobian norm

Original: [Figure]
Revised: [Figure]

Original: v a li d a t i o n c r o ss e n tr o p y
Revised: v a li d a t i o n c r o ss e n tr o p y

Original: [Figure]
Revised: [Figure]

Original: of vanilla SLATE explodes,
Revised: of vanilla SLATE explodes,

Original: as do its gradients (Fig. 5a), whereas both stay stable for our method.
Revised: as do its gradients (Fig. 5a), whereas both stay stable for our method.

Original: Lastly, removing learning rate warmup also consistently makes vanilla SLATE’s training unstable, whereas it only affects the stability of our method for one out of three seeds.
Revised: Lastly, removing learning rate warmup also consistently makes vanilla SLATE’s training unstable, whereas it only affects the stability of our method for one out of three seeds.

Original: Finally, Fig.
Revised: Finally, Fig.

Original: 5b shows that our method is not sensitive to the number of iterations with which to iterate the slot attention cell, whereas vanilla slot attention is, with more iterations being harder to train.
Revised: 5b shows that our method is not sensitive to the number of iterations with which to iterate the slot attention cell, whereas vanilla slot attention is, with more iterations being harder to train.

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION AND L IMITATIONS
Revised:  D ISCUSSION AND L IMITATIONS

Original: We have proposed implicit differentiation for training the iterative amortized inference procedures of symmetric generative models and demonstrated this technique on a state-of-the-art object-centric learning method.
Revised: We have proposed implicit differentiation for training the iterative amortized inference procedures of symmetric generative models and demonstrated this technique on a state-of-the-art object-centric learning method.

Original: Our results show clear signal that implicit differentiation can offer a significant optimization improvement over backpropagating through the unrolled iteration of slot attention, and potentially any iterative inference algorithm, with lower space and time complexity and only one additional line of code.
Revised: Our results show clear signal that implicit differentiation can offer a significant optimization improvement over backpropagating through the unrolled iteration of slot attention, and potentially any iterative inference algorithm, with lower space and time complexity and only one additional line of code.

Original: Despite our work pushing the optimization performance for a state-ofthe-art model in object-centric learning, the discrepancy between the quantitative improvement in optimization and evaluation metrics on the one hand and the less intuitive qualitative attention masks on real world observations like COCO (Fig. 3) on the other hand still suggests a gap between what we optimize these methods to do and what we actually want them to do.
Revised: Despite our work pushing the optimization performance for a state-ofthe-art model in object-centric learning, the discrepancy between the quantitative improvement in optimization and evaluation metrics on the one hand and the less intuitive qualitative attention masks on real world observations like COCO (Fig. 3) on the other hand still suggests a gap between what we optimize these methods to do and what we actually want them to do.

Original: This paper proposes a novel conceptualization of object representations as fast weights that converge towards a set of fixed points during execution.
Revised: This paper proposes a novel conceptualization of object representations as fast weights that converge towards a set of fixed points during execution.

Original: Because it is so simple to apply implicit differentiation to any fixed point algorithm, we hope this work inspires future work to leverage tools developed for implicit differentiation for improving object-centric learning and methods for learning latent structure more broadly.
Revised: Because it is so simple to apply implicit differentiation to any fixed point algorithm, we hope this work inspires future work to leverage tools developed for implicit differentiation for improving object-centric learning and methods for learning latent structure more broadly A CKNOWLEDGEMENTS This work was supported by ARL, W911NF2110097, with computing support from Google Cloud Platform.