Original: H OPFIELD N ETWORKS IS A LL Y OU N EED
Revised: H OPFIELD N ETWORKS IS A LL Y OU N EED

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce a modern Hopﬁeld network with continuous states and a corresponding update rule.
Revised: We introduce a modern Hopﬁeld network with continuous states and a corresponding update rule

Original: The new Hopﬁeld network can store exponentially (with the dimension of the associative space)
Revised: The new Hopﬁeld network can store exponentially (with the dimension of the associative space)

Original: many patterns, retrieves the pattern with one update, and has exponentially small retrieval errors.
Revised: many patterns, retrieves the pattern with one update and has exponentially small retrieval errors.

Original: It has three types of energy minima (ﬁxed points of the update): (1) global ﬁxed point averaging over all patterns, (2) metastable states averaging over a subset of patterns, and (3) ﬁxed points which store a single pattern.
Revised: It has three types of energy minima (ﬁxed points of the update): (1) global ﬁxed point averaging over all patterns, (2) metastable states averaging over a subset of patterns and (3) ﬁxed points which store a single pattern.

Original: The new update rule is equivalent to the attention mechanism used in transformers.
Revised: The new update rule is equivalent to the attention mechanism used in transformers.

Original: This equivalence enables a characterization of the heads of transformer models.
Revised: This equivalence enables a characterization of the heads of transformer models.

Original: These heads perform in the ﬁrst layers preferably global averaging and in higher layers partial averaging via metastable states.
Revised: These heads perform in the ﬁrst layers preferably global averaging and in higher layers partial averaging via metastable states.

Original: The new modern Hopﬁeld network can be integrated into deep learning architectures as layers to allow the storage of and access to raw input data, intermediate results, or learned prototypes.
Revised: The new modern Hopﬁeld network can be integrated into deep learning architectures as layers to allow the storage of and access to raw input data, intermediate results, or learned prototypes.

Original: These Hopﬁeld layers enable new ways of deep learning, beyond fully-connected, convolutional, or recurrent networks, and provide pooling, memory, association, and attention mechanisms.
Revised: These Hopﬁeld layers enable new ways of deep learning, beyond fully-connected, convolutional or recurrent networks and provide pooling, memory, association and attention mechanisms.

Original: We demonstrate the broad applicability of the Hopﬁeld layers across various domains.
Revised: We demonstrate the broad applicability of the Hopﬁeld layers across various domains.

Original: Hopﬁeld layers improved state-of-the-art on three out of four considered multiple instance learning problems as well as on immune repertoire classiﬁcation with several hundreds of thousands of instances.
Revised: Hopﬁeld layers improved state-of-the-art on three out of four considered multiple instance learning problems as well as on immune repertoire classiﬁcation with several hundreds of thousands of instances.

Original: On the UCI benchmark collections of small classiﬁcation tasks, where deep learning methods typically struggle, Hopﬁeld layers yielded the best results among different machine learning methods.
Revised: On the UCI benchmark collections of small classiﬁcation tasks where deep learning methods typically struggle , Hopﬁeld layers yielded a new state -of- the-art when compared to different machine learning methods.

Original: Finally, Hopﬁeld layers achieved state-of-the-art on two drug design datasets.
Revised: Finally, Hopﬁeld layers achieved state-of-the-art on two drug design datasets.

Revised: The implementation is available at: https://github.com/ml-jku/hopfield-layers

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The deep learning community has been looking for alternatives to recurrent neural networks (RNNs) for storing information.
Revised: The deep learning community has been looking for alternatives to recurrent neural networks (RNNs) for storing information.

Original: For example, linear memory networks use a linear autoencoder for sequences as a memory (Carta et al., 2020).
Revised: For example, linear memory networks use a linear autoencoder for sequences as a memory (Carta et al., 2020).

Original: Additional memories for RNNs like holographic reduced representations (Danihelka et al., 2016), tensor product representations (Schlag & Schmidhuber, 2018; Schlag et al., 2019) and classical associative memories (Ba et al., 2016a;b; Zhang & Zhou, 2017) have been suggested.
Revised: Additional memories for RNNs like holographic reduced representations (Danihelka et al., 2016), tensor product representations (Schlag & Schmidhuber, 2018; Schlag et al., 2019) and classical associative memories (extended to fast weight approaches) (Schmidhuber, 1992; Ba et al., 2016a;b; Zhang & Zhou, 2017; Schlag et al., 2021) have been suggested.

Original: Most approaches to new memories are based on attention.
Revised: Most approaches to new memories are based on attention.

Original: The neural Turing machine (NTM) is equipped with an external memory and an attention process (Graves et al., 2014).
Revised: The neural Turing machine (NTM) is equipped with an external memory and an attention process (Graves et al., 2014).

Original: Memory networks (Weston et al., 2014) use an arg max attention by ﬁrst mapping a query and patterns into a space and then retrieving the pattern with the largest dot product.
Revised: Memory networks (Weston et al., 2014) use an arg max attention by ﬁrst mapping a query and patterns into a space and then retrieving the pattern with the largest dot product.

Original: End to end memory networks (EMN) make this attention scheme differentiable by replacing arg max through a softmax (Sukhbaatar et al., 2015a;b).
Revised: End to end memory networks (EMN) make this attention scheme differentiable by replacing arg max through a softmax (Sukhbaatar et al., 2015a;b).

Original: EMN with dot products became very popular and implement a key-value attention (Daniluk et al., 2017) for self-attention.
Revised: EMN with dot products became very popular and implement a key-value attention (Daniluk et al., 2017) for self-attention.

Original: An enhancement of EMN is the transformer (Vaswani et al., 2017a;b) and its extensions (Dehghani et al., 2018).
Revised: An enhancement of EMN is the transformer (Vaswani et al., 2017a;b) and its extensions (Dehghani et al., 2018).

Original: The transformer has had a great impact on the natural language processing (NLP) community, in particular via the BERT models (Devlin et al., 2018; 2019).
Revised: The transformer has had a great impact on the natural language processing (NLP) community, in particular via the BERT models (Devlin et al., 2018; 2019).

Original: Contribution of this work: (i) introducing novel deep learning layers that are equipped with a memory via modern Hopﬁeld networks, (ii) introducing a novel energy function and a novel update rule for continuous modern Hopﬁeld networks that are differentiable and typically retrieve patterns after one update.
Revised: Contribution of this work: (i) introducing novel deep learning layers that are equipped with a memory via modern Hopﬁeld networks, (ii) introducing a novel energy function and a novel update rule for continuous modern Hopﬁeld networks that are differentiable and typically retrieve patterns after one update.

Original: Differentiability is required for gradient descent parameter updates and retrieval with one update is compatible with activating the layers of deep networks.
Revised: Differentiability is required for gradient descent parameter updates and retrieval with one update is compatible with activating the layers of deep networks.

Original: We suggest using modern Hopﬁeld networks to store information or learned prototypes in different layers of neural networks.
Revised: We suggest using modern Hopﬁeld networks to store information or learned prototypes in different layers of neural networks.

Original: Binary Hopﬁeld networks were introduced as associative memories that can store and retrieve patterns (Hopﬁeld, 1982).
Revised: Binary Hopﬁeld networks were introduced as associative memories that can store and retrieve patterns (Hopﬁeld, 1982).

Original: A query pattern can retrieve the pattern to which it is most similar or an average over similar patterns.
Revised: A query pattern can retrieve the pattern to which it is most similar or an average over similar patterns.

Original: Hopﬁeld networks seem to be an ancient technique, however, new energy functions improved their properties.
Revised: Hopﬁeld networks seem to be an ancient technique, however, new energy functions improved their properties.

Original: The stability of spurious states or metastable states was sensibly reduced (Barra et al., 2018).
Revised: The stability of spurious states or metastable states was sensibly reduced (Barra et al., 2018).

Original: The largest and most impactful successes are reported on increasing the storage capacity of Hopﬁeld networks.
Revised: The largest and most impactful successes are reported on increasing the storage capacity of Hopﬁeld networks.

Original: In a d -dimensional space, the standard Hopﬁeld model can store d uncorrelated patterns without errors but only Cd/ log( d ) random patterns with C < 1 / 2 for a ﬁxed stable pattern or C < 1 / 4 if all patterns are stable (McEliece et al., 1987).
Revised: In a d -dimensional space, the standard Hopﬁeld model can store d uncorrelated patterns without errors but only Cd/ log( d ) random patterns with C < 1 / 2 for a ﬁxed stable pattern or C < 1 / 4 if all patterns are stable (McEliece et al., 1987).

Original: The same bound holds for nonlinear learning rules (Mazza, 1997).
Revised: The same bound holds for nonlinear learning rules (Mazza, 1997).

Original: Using tricks-of-trade and allowing small retrieval errors, the storage capacity is about 0 .
Revised: Using tricks-of-trade and allowing small retrieval errors, the storage capacity is about 0 .

Original: 138 d (Crisanti et al., 1986; Hertz et al., 1991; Torres et al., 2002).
Revised: 138 d (Crisanti et al., 1986; Hertz et al., 1991; Torres et al., 2002).

Original: If the learning rule is not related to the Hebb rule, then up to d patterns can be stored (Abu-Mostafa & StJacques, 1985).
Revised: If the learning rule is not related to the Hebb rule, then up to d patterns can be stored (Abu-Mostafa & StJacques, 1985).

Original: For Hopﬁeld networks with non-zero diagonal matrices, the storage can be increased to Cd log( d ) (Folli et al., 2017).
Revised: For Hopﬁeld networks with non-zero diagonal matrices, the storage can be increased to Cd log( d ) (Folli et al., 2017).

Original: In contrast to the storage capacity, the number of energy minima (spurious states, stable states) of Hopﬁeld networks is exponential in d (Tanaka & Edwards, 1980; Bruck & Roychowdhury, 1990; Wainrib & Touboul, 2013).
Revised: In contrast to the storage capacity, the number of energy minima (spurious states, stable states) of Hopﬁeld networks is exponential in d (Tanaka & Edwards, 1980; Bruck & Roychowdhury, 1990; Wainrib & Touboul, 2013).

Original: The standard binary Hopﬁeld network has an energy function that can be expressed as the sum of interaction functions F with F ( x ) = x 2 .
Revised: The standard binary Hopﬁeld network has an energy function that can be expressed as the sum of interaction functions F with F ( x ) = x 2 .

Original: Modern Hopﬁeld networks, also called “dense associative memory” (DAM) models, use an energy function with interaction functions of the form F ( x ) = x n and, thereby, achieve a storage capacity proportional to d n − 1 (Krotov & Hopﬁeld, 2016; 2018).
Revised: Modern Hopﬁeld networks, also called “dense associative memory” (DAM) models, use an energy function with interaction functions of the form F ( x ) = x n and, thereby, achieve a storage capacity proportional to d n − 1 (Krotov & Hopﬁeld, 2016; 2018).

Original: The energy function of modern Hopﬁeld networks makes them robust against adversarial attacks (Krotov & Hopﬁeld, 2018).
Revised: The energy function of modern Hopﬁeld networks makes them robust against adversarial attacks (Krotov & Hopﬁeld, 2018).

Original: Modern binary Hopﬁeld networks with energy functions based on interaction functions of the form F ( x ) = exp( x ) even lead to storage capacity of 2 d/ 2 , where all stored binary patterns are ﬁxed points but the radius of attraction vanishes (Demircigil et al., 2017).
Revised: Modern binary Hopﬁeld networks with energy functions based on interaction functions of the form F ( x ) = exp( x ) even lead to storage capacity of 2 d/ 2 , where all stored binary patterns are ﬁxed points but the radius of attraction vanishes (Demircigil et al., 2017).

Original: However, in order to integrate Hopﬁeld networks into deep learning architectures, it is necessary to make them differentiable, that is, we require continuous Hopﬁeld networks (Hopﬁeld, 1984; Koiran, 1994).
Revised: However, in order to integrate Hopﬁeld networks into deep learning architectures, it is necessary to make them differentiable, that is, we require continuous Hopﬁeld networks (Hopﬁeld, 1984; Koiran, 1994).

Original: Therefore, we generalize the energy function of Demircigil et al.
Revised: Therefore, we generalize the energy function of Demircigil et al.

Original: (2017) that builds on exponential interaction functions to continuous patterns and states and obtain a new modern Hopﬁeld network.
Revised: (2017) that builds on exponential interaction functions to continuous patterns and states and obtain a new modern Hopﬁeld network.

Original: We also propose a new update rule which ensures global convergence to stationary points of the energy (local minima or saddle points).
Revised: We also propose a new update rule which ensures global convergence to stationary points of the energy (local minima or saddle points).

Original: We prove that our new modern Hopﬁeld network typically retrieves patterns in one update step ( (cid:15) -close to the ﬁxed point) with an exponentially low error and has a storage capacity proportional to c d − 14 (reasonable settings for c = 1 . 37 and c = 3 . 15 are given in Theorem 3).
Revised: We prove that our new modern Hopﬁeld network typically retrieves patterns in one update step ( (cid:15) -close to the ﬁxed point) with an exponentially low error and has a storage capacity proportional to c d − 14 (reasonable settings for c = 1 . 37 and c = 3 . 15 are given in Theorem 3).

Original: The retrieval of patterns with one update is important to integrate Hopﬁeld networks in deep learning architectures, where layers are activated only once.
Revised: The retrieval of patterns with one update is important to integrate Hopﬁeld networks in deep learning architectures, where layers are activated only once.

Original: Surprisingly, our new update rule is also the key-value attention as used in transformer and BERT models (see Fig. 1).
Revised: Surprisingly, our new update rule is also the key-value attention as used in transformer and BERT models (see Fig. 1).

Original: Our modern Hopﬁeld networks can be integrated as a new layer in deep learning architectures for pooling, memory, prototype learning, and attention.
Revised: Our modern Hopﬁeld networks can be integrated as a new layer in deep learning architectures for pooling, memory, prototype learning, and attention.

Original: We test these new layers on different benchmark datasets and tasks like immune repertoire classiﬁcation.
Revised: We test these new layers on different benchmark datasets and tasks like immune repertoire classiﬁcation.

Original:  M ODERN H OPFIELD N ETS WITH C ONTINUOUS S TATES
Revised:  M ODERN H OPFIELD N ETS WITH C ONTINUOUS S TATES

Original: New energy function for continuous state Hopﬁeld networks.
Revised: New energy function for continuous state Hopﬁeld networks.

Original: In order to integrate modern
Revised: In order to integrate modern

Original: Hopﬁeld networks into deep learning architectures, we have to make them continuous.
Revised: Hopﬁeld networks into deep learning architectures, we have to make them continuous.

Original: To allow for continuous states, we propose a new energy function that is a modiﬁcation of the energy of modern
Revised: To allow for continuous states, we propose a new energy function that is a modiﬁcation of the energy of modern

Original: Hopﬁeld networks (Demircigil et al., 2017).
Revised: Hopﬁeld networks (Demircigil et al., 2017).

Original: We also propose a new update rule which can be proven to converge to stationary points of the energy (local minima or saddle points).
Revised: We also propose a new update rule which can be proven to converge to stationary points of the energy (local minima or saddle points).

Original: We have N stored (key) patterns x
Revised: We have N stored (key) patterns x

Original: i ∈ R d represented by the matrix X = ( x 1 , . . .
Revised: i ∈ R d represented by the matrix X = ( x 1 , . . .

Original: , x N ) with the largest pattern M = max i (cid:107) x
Revised: , x N ) with the largest pattern M = max i (cid:107) x

Original: i (cid:107) .
Revised: i (cid:107) .

Original: The state (query) pattern is ξ ∈ R d .
Revised: The state (query) pattern is ξ ∈ R d .

Original: For exponential interaction functions, we need the log-sum-exp function ( lse ) for 0 < β
Revised: For exponential interaction functions, we need the log-sum-exp function ( lse ) for 0 < β

Original: [Equation]
Revised: [Equation]

Original:  which is convex (see appendix Eq.
Revised: (1) which is convex (see appendix Eq.

Original: (460), and Lemma A22).
Revised: (461), and Lemma A22).

Original: The energy function E of the modern
Revised: The energy function E of the modern

Original: Hopﬁeld networks for binary patterns x i and a binary state pattern ξ is E = − (cid:80) Ni =
Revised: Hopﬁeld networks for binary patterns x i and a binary state pattern ξ is E = − (cid:80) Ni =

Original: [Equation]

Original:  F (cid:0) [Equation] T x
Revised: 1 F (cid:0) ξ T x

Original: [Equation] (cid:1) (Krotov & Hopﬁeld, 2016).
Revised: i (cid:1) (Krotov & Hopﬁeld, 2016).

Original: Here, F ( x ) = x n is the interaction function, where n = 2 gives the classical Hopﬁeld network.
Revised: Here, F ( x ) = x n is the interaction function, where n = 2 gives the classical Hopﬁeld network.

Original: The storage capacity is proportional to d n − 1 (Krotov & Hopﬁeld, 2016).
Revised: The storage capacity is proportional to d n − 1 (Krotov & Hopﬁeld, 2016).

Original: This model was generalized by Demircigil et al.
Revised: This model was generalized by Demircigil et al.

Original: to exponential interaction functions F ( x ) = exp( x ) which gives the energy E = − exp(lse(1 , X T ξ )) .
Revised: to exponential interaction functions F ( x ) = exp( x ) which gives the energy E = − exp(lse(1 , X T ξ )) .

Original: This energy leads to an exponential storage capacity of N = 2 d/ 2 for binary patterns.
Revised: This energy leads to an exponential storage capacity of N = 2 d/ 2 for binary patterns.

Original: Furthermore, with a single update, the ﬁxed point is recovered with high probability for random patterns.
Revised: Furthermore, with a single update, the ﬁxed point is recovered with high probability for random patterns.

Original: However, still this modern Hopﬁeld network has binary states.
Revised: However, still this modern Hopﬁeld network has binary states.

Original: We generalize this energy function to continuous-valued patterns while keeping the properties of the modern Hopﬁeld networks like the exponential storage capacity and the extremely fast convergence (see Fig. 1).
Revised: We generalize this energy function to continuous-valued patterns while keeping the properties of the modern Hopﬁeld networks like the exponential storage capacity and the extremely fast convergence (see Fig. 1).

Original: For the new energy we take the logarithm of the negative energy of modern Hopﬁeld networks and add a quadratic term of the current state.
Revised: For the new energy we take the logarithm of the negative energy of modern Hopﬁeld networks and add a quadratic term of the current state.

Original: The quadratic term ensures that the norm of the state vector ξ remains ﬁnite and the energy is bounded.
Revised: The quadratic term ensures that the norm of the state vector ξ remains ﬁnite and the energy is bounded.

Original: Classical Hopﬁeld networks do not require to bound the norm of their state vector, since it is binary and has ﬁxed length.
Revised: Classical Hopﬁeld networks do not require to bound the norm of their state vector, since it is binary and has ﬁxed length.

Original: We deﬁne the novel energy function E as
Revised: We deﬁne the novel energy function E as

Original: [Equation]
Revised: [Equation]

Original: We have 0 (cid:54) E (cid:54)
Revised: We have 0 (cid:54) E (cid:54)

Original: 2 M 2 (see appendix Lemma A1).
Revised: 2 M 2 (see appendix Lemma A1).

Original: Using p = softmax( β X T ξ ) , we deﬁne a novel update rule (see Fig. 1):
Revised: Using p = softmax( β X T ξ ) , we deﬁne a novel update rule (see Fig. 1):

Original: [Equation]
Revised: [Equation]

Original: The next theorem states that the update rule Eq.
Revised: The next theorem states that the update rule Eq.

Original: (3) converges globally.
Revised: (3) converges globally.

Original: The proof uses the ConcaveConvex Procedure (CCCP) (Yuille & Rangarajan, 2002; 2003), which is equivalent to Legendre minimization (Rangarajan et al., 1996; 1999) algorithms (Yuille & Rangarajan, 2003).
Revised: The proof uses the ConcaveConvex Procedure (CCCP) (Yuille & Rangarajan, 2002; 2003), which is equivalent to Legendre minimization (Rangarajan et al., 1996; 1999) algorithms (Yuille & Rangarajan, 2003).

Original: Theorem 1.
Revised: Theorem 1.

Original: The update rule Eq.
Revised: The update rule Eq.

Original: (3) converges globally: For ξ t +1 = f ( ξ t ) , the energy E( ξ t ) →
Revised: (3) converges globally: For ξ t +1 = f ( ξ t ) , the energy E( ξ t ) →

Original: E( ξ ∗ ) for t → ∞ and a ﬁxed point ξ ∗ .
Revised: E( ξ ∗ ) for t → ∞ and a ﬁxed point ξ ∗ .

Original: Proof.
Revised: Proof.

Original: The update rule in Eq.
Revised: The update rule in Eq.

Original: (3) is the CCCP for minimizing the energy E , which is the sum of the convex 1 / 2 ξ T ξ and concave − lse (see details in appendix Theorem 1).
Revised: (3) is the CCCP for minimizing the energy E , which is the sum of the convex 1 / 2 ξ T ξ and concave − lse (see details in appendix Theorem 1).

Original: Theorem 2 in Yuille &
Revised: Theorem 2 in Yuille &

Original: Rangarajan (2002) yields the global convergence property.
Revised: Rangarajan (2002) yields the global convergence property.

Original: Also, in Theorem 2 in Sriperumbudur &
Revised: Also, in Theorem 2 in Sriperumbudur &

Original: Lanckriet (2009) the global convergence of CCCP is proven via a rigorous analysis using Zangwill’s global convergence theory of iterative algorithms.
Revised: Lanckriet (2009) the global convergence of CCCP is proven via a rigorous analysis using Zangwill’s global convergence theory of iterative algorithms.

Original: The global convergence theorem only assures that for the energy E( ξ t ) → E( ξ ∗ ) for t → ∞ but not ξ t → ξ ∗ .
Revised: The global convergence theorem only assures that for the energy E( ξ t ) → E( ξ ∗ ) for t → ∞ but not ξ t → ξ ∗ .

Original: The next theorem strengthens Zangwill’s global convergence theorem (Meyer, 1976) and gives convergence results similar to those known for expectation maximization (Wu, 1983).
Revised: The next theorem strengthens Zangwill’s global convergence theorem (Meyer, 1976) and gives convergence results similar to those known for expectation maximization (Wu, 1983).

Original: Theorem 2.
Revised: Theorem 2.

Original: For the iteration Eq.
Revised: For the iteration Eq.

Original: (3) we have E ( ξ t ) → E ( ξ ∗ ) =
Revised: (3) we have E ( ξ t ) → E ( ξ ∗ ) =

Original: E ∗ as t → ∞ , for some stationary point ξ ∗ .
Revised: E ∗ as t → ∞ , for some stationary point ξ ∗ .

Original: Furthermore, (cid:13)(cid:13)
Revised: Furthermore, (cid:13)(cid:13)

Original: ξ t +1 − ξ t (cid:13)(cid:13) → 0 and either { ξ t } ∞ t =0 converges or, in the other case, the set of limit points of { ξ t } ∞ t =0 is a connected and compact subset of L (E ∗ ) , where
Revised: ξ t +1 − ξ t (cid:13)(cid:13) → 0 and either { ξ t } ∞ t =0 converges or, in the other case, the set of limit points of { ξ t } ∞ t =0 is a connected and compact subset of L (E ∗ ) , where

Original: L ( a ) = { ξ ∈ L | E ( ξ ) = a } and L is the set of stationary points of the iteration Eq.
Revised: L ( a ) = { ξ ∈ L | E ( ξ ) = a } and L is the set of stationary points of the iteration Eq.

Original: If L (E ∗ ) is ﬁnite, then any sequence { ξ t } ∞ t =0 generated by the iteration Eq.
Revised: If L (E ∗ ) is ﬁnite, then any sequence { ξ t } ∞ t =0 generated by the iteration Eq.

Original: (3) converges to some ξ ∗ ∈ L (E ∗ ) .
Revised: (3) converges to some ξ ∗ ∈ L (E ∗ ) .

Original: For a proof, see appendix Theorem 2.
Revised: For a proof, see appendix Theorem 2.

Original: Therefore, all the limit points of any sequence generated by the iteration Eq.
Revised: Therefore, all the limit points of any sequence generated by the iteration Eq.

Original: (3) are stationary points (local minima or saddle points) of the energy function E .
Revised: (3) are stationary points (local minima or saddle points) of the energy function E .

Original: Either the iteration converges or, otherwise, the set of limit points is a connected and compact set.
Revised: Either the iteration converges or, otherwise, the set of limit points is a connected and compact set.

Original: The next theorem gives the results on the storage capacity of our new continuous state modern
Revised: The next theorem gives the results on the storage capacity of our new continuous state modern

Original: Hopﬁeld network.
Revised: Hopﬁeld network.

Original: We ﬁrst deﬁne what we mean by storing and retrieving patterns using a modern
Revised: We ﬁrst deﬁne what we mean by storing and retrieving patterns using a modern

Original: Hopﬁeld network with continuous states.
Revised: Hopﬁeld network with continuous states.

Original: Deﬁnition 1 (Pattern Stored and Retrieved) .
Revised: Deﬁnition 1 (Pattern Stored and Retrieved) .

Original: We assume that around every pattern x i a sphere S i is given.
Revised: We assume that around every pattern x i a sphere S i is given.

Original: We say x i is stored if there is a single ﬁxed point x ∗ i ∈ S
Revised: We say x i is stored if there is a single ﬁxed point x ∗ i ∈ S

Original: i to which all points ξ ∈ S i converge, and S i ∩ S j = ∅ for i (cid:54)
Revised: i to which all points ξ ∈ S i converge, and S i ∩ S j = ∅ for i (cid:54)

Original: =
Revised: =

Original: j .
Revised: j .

Original: We say x
Revised: We say x

Original: i is retrieved for a given (cid:15) if iteration (update rule) Eq.
Revised: i is retrieved for a given (cid:15) if iteration (update rule) Eq.

Original: (3) gives a point ˜
Revised: (3) gives a point ˜

Original: x i that is at least (cid:15) -close to the single ﬁxed point x ∗ i ∈ S i .
Revised: x i that is at least (cid:15) -close to the single ﬁxed point x ∗ i ∈ S i .

Original: The retrieval error is (cid:107) ˜
Revised: The retrieval error is (cid:107) ˜

Original: x
Revised: x

Original: i − x i (cid:107) .
Revised: i − x i (cid:107) .

Original: As with classical Hopﬁeld networks, we consider patterns on the sphere, i.e. patterns with a ﬁxed norm.
Revised: As with classical Hopﬁeld networks, we consider patterns on the sphere, i.e. patterns with a ﬁxed norm.

Original: For randomly chosen patterns, the number of patterns that can be stored is exponential in the dimension d of the space of the patterns ( x i ∈ R d ).
Revised: For randomly chosen patterns, the number of patterns that can be stored is exponential in the dimension d of the space of the patterns ( x i ∈ R d ).

Original: Theorem 3.
Revised: Theorem 3.

Original: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M :=
Revised: We assume a failure probability 0 < p (cid:54) 1 and randomly chosen patterns on the sphere with radius M :=

Original: K √ d − 1 .
Revised: K √ d − 1 .

Original: We deﬁne a := 2 d − 1 (1 + ln(2 βK 2 p ( d − 1))) , b := 2 K 2 β 5 , and c :=
Revised: We deﬁne a := 2 d − 1 (1 + ln(2 βK 2 p ( d − 1))) , b := 2 K 2 β 5 , and c :=

Original: b W 0 (exp( a +ln( b )) , where W 0 is the upper branch of the Lambert W function (Olver et al., 2010, (4.13)), and ensure c ≥ (cid:16) 2 √ p (cid:17) 4 d − 1 .
Revised: b W 0 (exp( a +ln( b )) , where W 0 is the upper branch of the Lambert W function (Olver et al., 2010, (4.13)), and ensure c ≥ (cid:16) 2 √ p (cid:17) 4 d − 1 .

Original: Then with probability 1 − p , the number of random patterns that can be stored is
Revised: Then with probability 1 − p , the number of random patterns that can be stored is

Original: [Equation]
Revised: [Equation]

Original: Therefore it is proven for c ≥ 3 .
Revised: Therefore it is proven for c ≥ 3 .

Original: 1546 with β = 1 , K = 3 , d = 20 and p = 0 .
Revised: 1546 with β = 1 , K = 3 , d = 20 and p = 0 .

Original: 001 ( a + ln( b ) > 1 . 27 ) and proven for c ≥ 1 . 3718 with β = 1 , K = 1 , d = 75 , and p = 0 .
Revised: 001 ( a + ln( b ) > 1 . 27 ) and proven for c ≥ 1 . 3718 with β = 1 , K = 1 , d = 75 , and p = 0 .

Original: 001 ( a + ln( b ) < − 0 . 94 ).
Revised: 001 ( a + ln( b ) < − 0 . 94 ).

Original: For a proof, see appendix Theorem A5.
Revised: For a proof, see appendix Theorem A5.

Original: The next theorem states that the update rule typically retrieves patterns after one update.
Revised: The next theorem states that the update rule typically retrieves patterns after one update.

Original: Retrieval of a pattern x i for ﬁxed point x ∗ i and query ξ is deﬁned via an (cid:15) by (cid:107) f ( ξ )
Revised: Retrieval of a pattern x i for ﬁxed point x ∗ i and query ξ is deﬁned via an (cid:15) by (cid:107) f ( ξ )

Original: − x ∗ i (cid:107) < (cid:15) , that is, the update is (cid:15) -close to the ﬁxed point.
Revised: − x ∗ i (cid:107) < (cid:15) , that is, the update is (cid:15) -close to the ﬁxed point.

Original: Retrieval with one update is crucial to integrate modern Hopﬁeld networks into deep learning architectures, where layers are activated only once.
Revised: Retrieval with one update is crucial to integrate modern Hopﬁeld networks into deep learning architectures, where layers are activated only once.

Original: First we need the concept of separation of a pattern.
Revised: First we need the concept of separation of a pattern.

Original: For pattern x i we deﬁne its separation ∆ i to other patterns by:
Revised: For pattern x i we deﬁne its separation ∆ i to other patterns by:

Original: [Equation]
Revised: [Equation]

Original: The update rule retrieves patterns with one update for well separated patterns, that is, patterns with large ∆ i .
Revised: The update rule retrieves patterns with one update for well separated patterns, that is, patterns with large ∆ i .

Original: Theorem 4.
Revised: Theorem 4.

Original: With query ξ , after one update the distance of the new point f ( ξ ) to the ﬁxed point x ∗ i is exponentially small in the separation ∆ i .
Revised: With query ξ , after one update the distance of the new point f ( ξ ) to the ﬁxed point x ∗ i is exponentially small in the separation ∆ i .

Original: The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ ξ and its value J m in the mean value theorem are: 
Revised: The precise bounds using the Jacobian J = ∂f ( ξ ) ∂ ξ and its value J m in the mean value theorem are: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − x ∗
Revised: For given (cid:15) and sufﬁcient large ∆ i , we have (cid:107) f ( ξ ) − x ∗

Original: i (cid:107) < (cid:15) , that is, retrieval with one update.
Revised: i (cid:107) < (cid:15) , that is, retrieval with one update.

Original: See proof in appendix Theorem A8.
Revised: See proof in appendix Theorem A8.

Original: At the same time, the retrieval error decreases exponentially with the separation ∆ i .
Revised: At the same time, the retrieval error decreases exponentially with the separation ∆ i .

Original: Theorem 5 (Exponentially Small Retrieval Error) .
Revised: Theorem 5 (Exponentially Small Retrieval Error) .

Original: The retrieval error (cid:107) f ( ξ ) − x i (cid:107) of pattern x
Revised: The retrieval error (cid:107) f ( ξ ) − x i (cid:107) of pattern x

Original: i is bounded by
Revised: i is bounded by

Original: [Equation]
Revised: [Equation]

Original: and for (cid:107) x i − x ∗ i (cid:107) (cid:54) 1 2 β M together with (cid:107) x i − ξ (cid:107) (cid:54) 1 2 β M by 
Revised: and for (cid:107) x i − x ∗ i (cid:107) (cid:54) 1 2 β M together with (cid:107) x i − ξ (cid:107) (cid:54) 1 2 β M by [Equation] x i [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: See proof in appendix Theorem A9.
Revised: See proof in appendix Theorem A9.

Original: Metastable states and one global ﬁxed point.
Revised: Metastable states and one global ﬁxed point.

Original: So far, we considered patterns x i that are well separated and the iteration converges to a ﬁxed point which is near a pattern x i .
Revised: So far, we considered patterns x i that are well separated and the iteration converges to a ﬁxed point which is near a pattern x i .

Original: If no pattern x i is well separated from the others, then the iteration converges to a global ﬁxed point close to the arithmetic mean of the vectors.
Revised: If no pattern x i is well separated from the others, then the iteration converges to a global ﬁxed point close to the arithmetic mean of the vectors.

Original: In this case the softmax vector p is close to uniform, that is, p i = 1 /N .
Revised: In this case the softmax vector p is close to uniform, that is, p i = 1 /N .

Original: If some vectors are similar to each other and well separated from all other vectors, then a metastable state near the similar vectors exists.
Revised: If some vectors are similar to each other and well separated from all other vectors, then a metastable state near the similar vectors exists.

Original: Iterations that start near the metastable state converge to this metastable state, also if initialized by one of the similar patterns.
Revised: Iterations that start near the metastable state converge to this metastable state, also if initialized by one of the similar patterns.

Original: For convergence proofs to one global ﬁxed point and to metastable states see appendix Lemma A7 and Lemma A12, respectively.
Revised: For convergence proofs to one global ﬁxed point and to metastable states see appendix Lemma A7 and Lemma A12, respectively.

Original: Hopﬁeld update rule is attention of the transformer.
Revised: Hopﬁeld update rule is attention of the transformer.

Original: The Hopﬁeld network update rule is the attention mechanism used in transformer and BERT models (see Fig. 1).
Revised: The Hopﬁeld network update rule is the attention mechanism used in transformer and BERT models (see Fig. 1).

Original: To see this, we assume
Revised: To see this, we assume

Original: N stored (key) patterns y i and S state (query) patterns
Revised: N stored (key) patterns y i and S state (query) patterns

Original: r i that are mapped to the Hopﬁeld space of dimension d k .
Revised: r i that are mapped to the Hopﬁeld space of dimension d k .

Original: We set x i =
Revised: We set x i =

Original: W TK
Revised: W TK

Original: y i , ξ i =
Revised: y i , ξ i =

Original: W TQ r i , and multiply the result of our update rule with W V .
Revised: W TQ r i , and multiply the result of our update rule with W V .

Original: The matrices Y =
Revised: The matrices Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y N ) T and R =
Revised: , y N ) T and R =

Original: ( r 1 , . .
Revised: ( r 1 , . .

Original: , r S ) T combine the y i and r i as row vectors.
Revised: , r S ) T combine the y i and r i as row vectors.

Original: We deﬁne the matrices X T
Revised: We deﬁne the matrices X T

Original: = K = Y W K , Ξ T = Q = RW Q , and V =
Revised: = K = Y W K , Ξ T = Q = RW Q , and V =

Original: Y W K W V =
Revised: Y W K W V =

Original: X T W V , where
Revised: X T W V , where

Original: W K ∈ R d y × d k , W Q ∈ R
Revised: W K ∈ R d y × d k , W Q ∈ R

Original: d r × d k , W V ∈ R
Revised: d r × d k , W V ∈ R

Original: d k × d v .
Revised: d k × d v .

Original: If β = 1 / √ d k and softmax ∈ R N is changed to a row vector, we obtain for the update rule Eq.
Revised: If β = 1 / √ d k and softmax ∈ R N is changed to a row vector, we obtain for the update rule Eq.

Original: (3) multiplied by W V :
Revised: (3) multiplied by W V :

Original: [Equation]
Revised: [Equation]

Original: The left part of Eq.
Revised: The left part of Eq.

Original: (10) is the transformer attention.
Revised: (10) is the transformer attention.

Original: In the transformer self-attention R = Y , and
Revised: In the transformer self-attention R = Y , and

Original: W K W V replaced by just W V .
Revised: W K W V replaced by just W V .

Original: Besides the attention mechanism, Hopﬁeld networks allow for other functionalities in deep network architectures, which we introduce via speciﬁc layers in the next section.
Revised: Besides the attention mechanism, Hopﬁeld networks allow for other functionalities in deep network architectures, which we introduce via speciﬁc layers in the next section.

Original: The right part of Eq.
Revised: The right part of Eq.

Original: (10) serves to explain these speciﬁc layers.
Revised: (10) serves to explain these speciﬁc layers.

Original:  N EW H OPFIELD L AYERS FOR D EEP L EARNING
Revised:  N EW H OPFIELD L AYERS FOR D EEP L EARNING

Original: Modern Hopﬁeld networks with continuous states can be integrated into deep learning architectures, because they are continuous and differentiable with respect to their parameters.
Revised: Modern Hopﬁeld networks with continuous states can be integrated into deep learning architectures, because they are continuous and differentiable with respect to their parameters.

Original: Furthermore, they typically retrieve patterns with one update, which is conform to deep learning layers that are activated only once.
Revised: Furthermore, they typically retrieve patterns with one update, which is conform to deep learning layers that are activated only once.

Original: For these two reasons, modern Hopﬁeld networks can serve as specialized layers in deep networks to equip them with memories.
Revised: For these two reasons, modern Hopﬁeld networks can serve as specialized layers in deep networks to equip them with memories.

Revised: Below, we introduce three types of Hopﬁeld layers:

Revised: Hopfield , HopfieldPooling , and HopfieldLayer .

Revised: Possible applications of Hopﬁeld layers in deep network architectures comprise:

Original: These layers enable transformer-like self-attention, sequence-to-sequence learning, decoder-encoder attention, time series prediction, sequence analysis, multiple instancelearning, learning  of  point sets, learning by associations, and averaging & pooling operations.
Revised: • multiple instance learning (MIL) (Dietterich et al., 1997), • processing of and learning with point sets (Qi et al., 2017a;b; Xu et al., 2018), • set-based and permutation invariant learning (Guttenberg et al., 2016; Ravanbakhsh et al., 2016; Zaheer et al., 2017; Korshunova et al., 2018; Ilse et al., 2018; Zhai et al., 2020), • attention-based learning (Vaswani et al., 2017a), • deep learning with associative memories (Graves et al., 2014; Weston et al., 2014; Ba et al., 2016a;b; Schlag & Schmidhuber, 2018; Schlag et al., 2019), • natural language processing (Devlin et al., 2018; 2019), • sequence analysis and time series prediction (Hochreiter, 1991; Hochreiter & Schmidhuber, 1997; Cho et al., 2014), and • storing and retrieving reference data, e

Revised: g. the training data, outliers, high error data points

Revised: , prototypes or cluster centers, support vectors & border cases.

Original: Hopﬁeld network layers can substitute existing layers like pooling layers, permutation equivariant layers (Guttenberg et al., 2016; Ravanbakhsh et al., 2016), GRU (Cho et al., 2014) & LSTM (Hochreiter, 1991; Hochreiter & Schmidhuber, 1997) layers, and attention layers (Vaswani et al., 2017a;b; Bahdanau et al., 2014).
Revised: Hopﬁeld network layers can substitute existing layers like pooling layers, permutation equivariant layers (Guttenberg et al., 2016; Ravanbakhsh et al., 2016), GRU (Cho et al., 2014) & LSTM (Hochreiter, 1991; Hochreiter & Schmidhuber, 1997) layers, and attention layers (Vaswani et al., 2017a;b; Bahdanau et al., 2014).

Original: Types of neural networks.
Revised: Types of neural networks.

Original: We consider two types of feedforward neural networks: (I) Neural networks that propagate an activation vector from the input layer to the output layer.
Revised: We consider two types of feedforward neural networks: (I) Neural networks that propagate an activation vector from the input layer to the output layer.

Original: Examples are fully-connected or convolutional neural networks.
Revised: Examples are fully-connected or convolutional neural networks.

Original: (II)
Revised: (II)

Original: Neural networks that propagate a set of vectors from the input layer to the output layer, where each layer applies the same operation to each element of the set and the output layer may summarize the set via a vector.
Revised: Neural networks that propagate a set of vectors from the input layer to the output layer, where each layer applies the same operation to each element of the set and the output layer may summarize the set via a vector.

Original: An example is the transformer.
Revised: An example is the transformer.

Original: Recurrent neural networks are networks of type (I), which are iteratively applied to a set or a sequence, where intermediate results are stored in a memory and can be reused.
Revised: Recurrent neural networks are networks of type (I), which are iteratively applied to a set or a sequence, where intermediate results are stored in a memory and can be reused.

Original: Modern
Revised: Modern

Original: Hopﬁeld networks can be integrated into both types of neural network architectures and enable to equip each of their layers with associative memories.
Revised: Hopﬁeld networks can be integrated into both types of neural network architectures and enable to equip each of their layers with associative memories.

Original: See Fig.
Revised: See Fig.

Original: Types of new Hopﬁeld layers.
Revised: Types of new Hopﬁeld layers.

Revised: We introduce three types of Hopﬁeld layers: Hopfield , HopfieldPooling , and

Original:  The continuous modern Hopﬁeld network results in a plethora of new deep learning architectures, since we can (a) propagate sets or single vectors, (b) propagate queries, stored patterns, or both, (c) learn static queries or stored patterns, (d) ﬁll the memory by training sets, prototypes, or external data.
Revised: HopfieldLayer . The continuous modern Hopﬁeld network results in a plethora of new deep learning architectures, since we can (a) propagate sets or single vectors, (b) propagate queries, stored patterns, or both, (c) learn static queries or stored patterns, (d) ﬁll the memory by training sets, prototypes, or external data.

Original: Next, we provide three useful types of Hopﬁeld layers.
Revised: Next, we provide three useful types of Hopﬁeld layers.

Original:  (1) Layer Hopfield for networks that propagate sets of vectors via state (query) patterns R and stored (key) patterns Y .
Revised: The implementation is available at: https://github.com/ml-jku/hopfield-layers (1) Layer Hopfield for networks that propagate sets of vectors via state (query) patterns R and stored (key) patterns Y .

Original: The layer Hopfield is the realization of formula (10).
Revised: The layer Hopfield is the realization of formula (10).

Original: The memory of the Hopfield layer can be ﬁlled with sets from the input or previous layers , see Fig.
Revised: The memory of the Hopfield layer can be ﬁlled with sets from the input or previous layers , see Fig.

Original: The memory may be ﬁlled with a reference set, which is covered by providing the reference set as additional input.
Revised: The memory may be ﬁlled with a reference set, which is covered by providing the reference set as additional input.

Original: Thus, the layer Hopfield allows the association of two sets.
Revised: Thus, the layer Hopfield allows the association of two sets.

Original: A prominent example of a layer that performs such association is the transformer attention mechanism, which associates keys and queries, e.g. two point sets that have to be compared.
Revised: A prominent example of a layer that performs such association is the transformer attention mechanism, which associates keys and queries, e.g. two point sets that have to be compared.

Original: This layer allows for different kinds of sequence-to-sequence learning, point set operations, and retrieval-based methods.
Revised: This layer allows for different kinds of sequence-to-sequence learning, point set operations, and retrieval-based methods.

Original: The layer
Revised: The layer

Original: Hopfield with skip connections in a ResNet architecture is identical to the popular transformer and
Revised: Hopfield with skip connections in a ResNet architecture is identical to the popular transformer and

Original: BERT models.
Revised: BERT models.

Original: In the experiments, we analyzed these Hopﬁeld layers in transformer architectures.
Revised: In the experiments, we analyzed these Hopﬁeld layers in transformer architectures.

Original: In our experiments in which we compare machine learning methods on small datasets of the UCI benchmark collection the layer Hopfield is also used.
Revised: In our experiments in which we compare machine learning methods on small datasets of the UCI benchmark collection the layer Hopfield is also used.

Original: [Equation]
Revised: [Equation]

Original: performs A single HopfieldPooling layer allows for multiple instance learning.
Revised: A single HopfieldPooling layer allows for multiple instance learning.

Original: Static state (query) patterns together with position encoding in the keys allows for performing pooling operations.
Revised: Static state (query) patterns together with position encoding in the keys allows for performing pooling operations.

Original: The position encoding can be two-dimensional, where standard convolutional ﬁlters can be constructed as in convolutional neural networks (CNNs).
Revised: The position encoding can be two-dimensional, where standard convolutional ﬁlters can be constructed as in convolutional neural networks (CNNs).

Original: The HopfieldPooling layer can substitute pooling, averaging, LSTM, and permutation equivariant layers.
Revised: The HopfieldPooling layer can substitute pooling, averaging, LSTM, and permutation equivariant layers.

Original: See Fig.
Revised: See Fig.

Original: The layer HopfieldPooling is used for experiments with multiple instance learning tasks, e.g. for immune repertoire classiﬁcation in the experiments.
Revised: The layer HopfieldPooling is used for experiments with multiple instance learning tasks, e.g. for immune repertoire classiﬁcation in the experiments.

Original: [Equation]
Revised: [Equation]

Revised: patterns of each are static can learned set vectors Z where the number of vectors equals the number of query patterns can realize multiple instance learning.

Original: (3) Layer HopfieldLayer for networks that propagate a vector or a set of vectors via state (query) patterns R .
Revised: (3) Layer HopfieldLayer for networks that propagate a vector or a set of vectors via state (query) patterns R .

Original: The queries R can be input vectors or queries that are computed from the output of previous layers.
Revised: The queries R can be input vectors or queries that are computed from the output of previous layers.

Original: The memory of the HopfieldLayer layer is ﬁlled with a ﬁxed set , which can be the training set, a reference set, prototype set, or a learned set (a learned matrix).
Revised: The memory of the HopfieldLayer layer is ﬁlled with a ﬁxed set , which can be the training set, a reference set, prototype set, or a learned set (a learned matrix).

Original: The stored (key) patterns are static and can be learned.
Revised: The stored (key) patterns are static and can be learned.

Original: If the training set is stored in the memory, then each layer constructs a new set of queries based on the query results of previous layers.
Revised: If the training set is stored in the memory, then each layer constructs a new set of queries based on the query results of previous layers.

Original: The stored patterns can be initialized by the training set or a reference set and then learned, in which case they deviate from the training set.
Revised: The stored patterns can be initialized by the training set or a reference set and then learned, in which case they deviate from the training set.

Original: The stored patterns can be interpreted as weights from the state (query) to hidden neurons that have a softmax activation function (Krotov & Hopﬁeld, 2020).
Revised: The stored patterns can be interpreted as weights from the state (query) to hidden neurons that have a softmax activation function (Krotov & Hopﬁeld, 2020).

Original: The layer
Revised: The layer

Original: HopfieldLayer can substitute a fully connected layer, see Fig.
Revised: HopfieldLayer can substitute a fully connected layer, see Fig.

Original: A single HopfieldLayer layer also allows for approaches similar to support vector machines (SVMs), approaches similar to k -nearest neighbor, approaches similar to learning vector quantization, and pattern search.
Revised: A single HopfieldLayer layer also allows for approaches similar to support vector machines (SVMs), approaches similar to k -nearest neighbor, approaches similar to learning vector quantization, and pattern search.

Original: For classiﬁcation, the raw data
Revised: For classiﬁcation, the raw data

Original: y i =
Revised: y i =

Original: ( z i , t i ) can be the concatenation of input z
Revised: ( z i , t i ) can be the concatenation of input z

Original: i and target t i .
Revised: i and target t i .

Original: In this case, the matrices W K and W V can be designed such that inside the softmax the input z
Revised: In this case, the matrices W K and W V can be designed such that inside the softmax the input z

Original: i is used and outside the softmax the target t i .
Revised: i is used and outside the softmax the target t i .

Original: Thus, the softmax provides a weighted average of the target vectors based on the similarity between the query and the inputs.
Revised: Thus, the softmax provides a weighted average of the target vectors based on the similarity between the query and the inputs.

Original: Also SVM models, k -nearest neighbor, and learning vector quantization can be considered as weighted averages of the targets.
Revised: Also SVM models, k -nearest neighbor, and learning vector quantization can be considered as weighted averages of the targets.

Original: The encoder-decoder attention layer of the transformers are a HopfieldLayer layer, where the memory is ﬁlled with the encoder output set.
Revised: The encoder-decoder attention layer of the transformers are a HopfieldLayer layer, where the memory is ﬁlled with the encoder output set.

Original: In our experiments with the drug design benchmark datasets, the layer HopfieldLayer has been applied and compared to other machine learning methods.
Revised: In our experiments with the drug design benchmark datasets, the layer HopfieldLayer has been applied and compared to other machine learning methods.

Original: [Equation]
Revised: [Equation]

Original: Additional functionality of new Hopﬁeld layers.
Revised: Additional functionality of new Hopﬁeld layers.

Original: The insights about energy, convergence, and storage properties provide all new Hopﬁeld layers with additional functionalities: i) multiple updates to control how precise ﬁxed points are found without additional parameters needed.
Revised: The insights about energy, convergence, and storage properties provide all new Hopﬁeld layers with additional functionalities: i) multiple updates to control how precise ﬁxed points are found without additional parameters needed.

Original: ii) variable β to determine the kind of ﬁxed points such as the size of metastable states.
Revised: ii) variable β to determine the kind of ﬁxed points such as the size of metastable states.

Original: The variable β controls over how many patterns is averaged.
Revised: The variable β controls over how many patterns is averaged.

Original: As observed in the experiments, the variable is relevant in combination with the learning rate to steer the learning dynamics.
Revised: As observed in the experiments, the variable is relevant in combination with the learning rate to steer the learning dynamics.

Original: The parameter β governs the ﬁxed point dynamics and can be learned, too.
Revised: The parameter β governs the ﬁxed point dynamics and can be learned, too.

Original: iii) controlling the storage capacity via the dimension of the associative space.
Revised: iii) controlling the storage capacity via the dimension of the associative space.

Original: The storage capacity can be relevant for tasks with a huge number of instances as in the immune repertoire classiﬁcation experiment.
Revised: The storage capacity can be relevant for tasks with a huge number of instances as in the immune repertoire classiﬁcation experiment.

Original: iv) pattern normalization controls, like the layernorm, the ﬁxed point dynamics by the norm and shift of the patterns.
Revised: iv) pattern normalization controls, like the layernorm, the ﬁxed point dynamics by the norm and shift of the patterns.

Original: For more details see appendix, Section A.6.
Revised: For more details see appendix, Section A.6.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We show that our proposed Hopﬁeld layers can be applied successfully to a wide range of tasks.
Revised: We show that our proposed Hopﬁeld layers can be applied successfully to a wide range of tasks.

Original: The tasks are from natural language processing, contain multiple instance learning problems, a collection of small classiﬁcation tasks, and drug design problems.
Revised: The tasks are from natural language processing, contain multiple instance learning problems, a collection of small classiﬁcation tasks, and drug design problems.

Original: Analysis of transformer and BERT models.
Revised: Analysis of transformer and BERT models.

Original: Transformer and BERT models can be implemented by the layer Hopfield .
Revised: Transformer and BERT models can be implemented by the layer Hopfield .

Original: The kind of ﬁxed point of the Hopﬁeld net is determined by how the pattern x i is separated from others patterns.
Revised: The kind of ﬁxed point of the Hopﬁeld net is determined by how the pattern x i is separated from others patterns.

Original: (a) a global ﬁxed point : no separation of a pattern from the others, (b) a ﬁxed point close to a single pattern : pattern is separated from other patterns, (c) metastable state : some patterns are similar to each other and well separated from all other vectors.
Revised: (a) a global ﬁxed point : no separation of a pattern from the others, (b) a ﬁxed point close to a single pattern : pattern is separated from other patterns, (c) metastable state : some patterns are similar to each other and well separated from all other vectors.

Original: We observed that the attention heads of transformer and BERT models are predominantly in metastable states, which are categorized into four classes: (I) averaging over a very large number of patterns (very large metastable state or ﬁxed point (a)), (II) averaging over a large number of patterns (large metastable state), (III) averaging over a medium number of patterns (medium metastable state), (IV) averaging over a small number of patterns (small metastable state or ﬁxed point (c)).
Revised: We observed that the attention heads of transformer and BERT models are predominantly in metastable states, which are categorized into four classes: (I) averaging over a very large number of patterns (very large metastable state or ﬁxed point (a)), (II) averaging over a large number of patterns (large metastable state), (III) averaging over a medium number of patterns (medium metastable state), (IV) averaging over a small number of patterns (small metastable state or ﬁxed point (c)).

Original: For analyzing the metastable states, we calculated the minimal number k of softmax values required to sum up to 0 .
Revised: For analyzing the metastable states, we calculated the minimal number k of softmax values required to sum up to 0 .

Original: Hence, k indicates the size of a metastable state.
Revised: Hence, k indicates the size of a metastable state.

Original: To determine in which of the four classes a head is mainly operating, we computed the distribution of k across sequences.
Revised: To determine in which of the four classes a head is mainly operating, we computed the distribution of k across sequences.

Original: Concretely, for N tokens and for ¯ k as the median of the distribution, a head is classiﬁed as operating in class (I) if 1 / 2 N < ¯ k , as operating in class (II) if 1 / 8 N < ¯ k (cid:54) 1 / 2 N , as operating in class (III) if 1 / 32 N < ¯ k (cid:54) 1 / 8 N , and as operating in class (IV) if ¯ k (cid:54) 1 / 32 N .
Revised: Concretely, for N tokens and for ¯ k as the median of the distribution, a head is classiﬁed as operating in class (I) if 1 / 2 N < ¯ k , as operating in class (II) if 1 / 8 N < ¯ k (cid:54) 1 / 2 N , as operating in class (III) if 1 / 32 N < ¯ k (cid:54) 1 / 8 N , and as operating in class (IV) if ¯ k (cid:54) 1 / 32 N .

Original: We analyzed pre-trained BERT models from Hugging
Revised: We analyzed pre-trained BERT models from Hugging

Original: Face Inc. (Wolf et al., 2019) according to these operating classes.
Revised: Face Inc. (Wolf et al., 2019) according to these operating classes.

Original: In Fig.
Revised: In Fig.

Original: A.3 in the appendix the distribution of the pre-trained bert-base-cased model is depicted (for other models see appendix
Revised: A.3 in the appendix the distribution of the pre-trained bert-base-cased model is depicted (for other models see appendix

Original: Section A.5.1.4).
Revised: Section A.5.1.4).

Original: Operating classes (II) (large metastable states) and (IV) (small metastable states) are often observed in the middle layers.
Revised: Operating classes (II) (large metastable states) and (IV) (small metastable states) are often observed in the middle layers.

Original: Operating class (I) (averaging over a very large number of patterns) is abundant in lower layers.
Revised: Operating class (I) (averaging over a very large number of patterns) is abundant in lower layers.

Original: Similar observations have been reported in other studies (Toneva & Wehbe, 2019a;b; Tay et al., 2020).
Revised: Similar observations have been reported in other studies (Toneva & Wehbe, 2019a;b; Tay et al., 2020).

Original: Operating class ( III ) (medium metastable states) is predominant in the last layers.
Revised: Operating class (III) (medium metastable states) is predominant in the last layers.

Original: Multiple Instance Learning Datasets.
Revised: Multiple Instance Learning Datasets.

Original: For multiple instance learning (MIL) (Dietterich et al., 1997), we integrate our new Hopﬁeld network via the layer HopfieldPooling into deep learning architectures.
Revised: For multiple instance learning (MIL) (Dietterich et al., 1997), we integrate our new Hopﬁeld network via the layer HopfieldPooling into deep learning architectures.

Original: Recently, deep learning methods have been applied to MIL problems (Ilse et al., 2018), but still the performance on many datasets lacks improvement.
Revised: Recently, deep learning methods have been applied to MIL problems (Ilse et al., 2018), but still the performance on many datasets lacks improvement.

Original: Thus, MIL datasets still pose an interesting challenge, in which Hopﬁeld layers equipped with memory are a promising approach.
Revised: Thus, MIL datasets still pose an interesting challenge, in which Hopﬁeld layers equipped with memory are a promising approach.

Original: • Immune Repertoire Classiﬁcation.
Revised: • Immune Repertoire Classiﬁcation.

Original: The ﬁrst MIL task is immune repertoire classiﬁcation, where a deep learning architecture with HopfieldPooling (DeepRC) was used (Widrich et al., 2020a;b).
Revised: The ﬁrst MIL task is immune repertoire classiﬁcation, where a deep learning architecture with HopfieldPooling (DeepRC) was used (Widrich et al., 2020a;b).

Original: Immune repertoire classiﬁcation (Emerson et al., 2017) typically requires to extract few patterns from a large set of sequences, the repertoire, that are indicative for the respective immune status.
Revised: Immune repertoire classiﬁcation (Emerson et al., 2017) typically requires to extract few patterns from a large set of sequences, the repertoire, that are indicative for the respective immune status.

Original: The datasets contain ≈ 300,000 instances per immune repertoire, which represents one of the largest multiple instance learning experiments ever conducted (Carbonneau et al., 2018).
Revised: The datasets contain ≈ 300,000 instances per immune repertoire, which represents one of the largest multiple instance learning experiments ever conducted (Carbonneau et al., 2018).

Original: Most MIL methods fail due the large number of instances.
Revised: Most MIL methods fail due the large number of instances.

Original: This experiment comprises real-world and simulated datasets.
Revised: This experiment comprises real-world and simulated datasets.

Original: Simulated datasets are generated by implanting sequence motifs (Akbar et al., 2019;
Revised: Simulated datasets are generated by implanting sequence motifs (Akbar et al., 2019;

Original: Weber et al., 2020) with low frequency into simulated or experimentally-observed immune receptor sequences.
Revised: Weber et al., 2020) with low frequency into simulated or experimentally-observed immune receptor sequences.

Original: The performance of DeepRC was compared with other machine learning methods: (i) known motif, (ii) SVM using k -mers and MinMax or Jaccard kernel, (iii) K -Nearest Neighbor (KNN) with kmers, (iv) logistic regression with k -mers, (v) burden test with k -mers, and (vi) logistic multiple  instance learning (lMIL).
Revised: The performance of DeepRC was compared with other machine learning methods: (i) known motif, (ii) SVM using k -mers and MinMax or Jaccard kernel, (iii) K -Nearest Neighbor (KNN) with kmers, (iv) logistic regression with k -mers, (v) burden test with k -mers, and (vi) logistic multiple [Table] instance learning (lMIL).

Original: On the real-world dataset DeepRC achieved an AUC of 0 .
Revised: On the real-world dataset DeepRC achieved an AUC of 0 .

Original: 832 ± 0 . 022 , followed by the SVM with MinMax kernel (AUC 0 .
Revised: 832 ± 0 . 022 , followed by the SVM with MinMax kernel (AUC 0 .

Original: 825 ± 0 . 022 ) and the burden test with an AUC of 0 .
Revised: 825 ± 0 . 022 ) and the burden test with an AUC of 0 .

Original: 699 ± 0 .
Revised: 699 ± 0 .

Original: Across datasets, DeepRC outperformed all competing methods with respect to average AUC (Widrich et al., 2020a;b).
Revised: Across datasets, DeepRC outperformed all competing methods with respect to average AUC (Widrich et al., 2020a;b).

Original: • MIL benchmark datasets.
Revised: • MIL benchmark datasets

Revised: We apply

Revised: Hopﬁeld layers

Original: We apply Hopﬁeld layers to further MIL datasets (Ilse et al., 2018; Küçüka¸scı & Baydo ˘ gan, 2018; Cheplygina et al., 2016):
Revised: to further MIL datasets (Ilse et al., 2018; Küçüka¸scı & Baydo ˘ gan, 2018; Cheplygina et al., 2016):

Original: Elephant, Fox and Tiger for image annotation (Andrews et al., 2003).
Revised: Elephant, Fox and Tiger for image annotation (Andrews et al., 2003).

Original: These datasets consist of color images from the Corel dataset that have been preprocessed and segmented.
Revised: These datasets consist of color images from the Corel dataset that have been preprocessed and segmented.

Original: An image consists of a set of segments (or blobs), each characterized by color, texture and shape descriptors.
Revised: An image consists of a set of segments (or blobs), each characterized by color, texture and shape descriptors.

Original: The datasets have 100 positive and 100 negative example images.
Revised: The datasets have 100 positive and 100 negative example images.

Original: The latter have been randomly drawn from a pool of photos of other animals.
Revised: The latter have been randomly drawn from a pool of photos of other animals.

Original: Elephant comprises 1,391 instances and 230 features, Fox 1,320 instances and 230 features, and Tiger has 1,220 instances and 230 features.
Revised: Elephant comprises 1,391 instances and 230 features, Fox 1,320 instances and 230 features, and Tiger has 1,220 instances and 230 features.

Original: Furthermore, we use the UCSB breast cancer classiﬁcation (Kandemir et al., 2014) dataset, which consists of 2,002 instances across  input objects.
Revised: Furthermore, we use the UCSB breast cancer classiﬁcation (Kandemir et al., 2014) dataset, which consists of 2,002 instances across 58 input objects.

Original: An instance represents a patch of a histopathological image of cancerous or normal tissue.
Revised: An instance represents a patch of a histopathological image of cancerous or normal tissue.

Original: The layer HopfieldPooling is used, which allows for computing a per-input-object representation by extracting an average of instances that are indicative for one of the two classes.
Revised: The layer HopfieldPooling is used, which allows for computing a per-input-object representation by extracting an average of instances that are indicative for one of the two classes.

Original: The input to the layer HopfieldPooling is a set of embedded instances Y .
Revised: The input to the layer HopfieldPooling is a set of embedded instances Y .

Original: A trainable but ﬁxed state (query) pattern Q is used for averaging over class-indicative instances.
Revised: A trainable but ﬁxed state (query) pattern Q is used for averaging over class-indicative instances.

Original: This averaging enables a compression of variable-sized bags to a ﬁxedsized representation to discriminate the bags.
Revised: This averaging enables a compression of variable-sized bags to a ﬁxedsized representation to discriminate the bags.

Original: More details in appendix Sec. A.5.2.
Revised: More details in appendix Sec. A.5.2.

Original: Our approach has set a new state-of-the-art and has outperformed other methods (Küçüka¸scı & Baydo˘gan, 2018;
Revised: Our approach has set a new state-of-the-art and has outperformed other methods (Küçüka¸scı & Baydo˘gan, 2018;

Original: Carbonneau et al., 2016) on the datasets Tiger, Elephant and UCSB Breast Cancer (see Table 1).
Revised: Carbonneau et al., 2016) on the datasets Tiger, Elephant and UCSB Breast Cancer (see Table 1).

Original: UCI Benchmark Collection.
Revised: UCI Benchmark Collection.

Original: So far deep learning struggled with small datasets.
Revised: So far deep learning struggled with small datasets.

Original: However, Hopﬁeld networks are promising for handling small datasets, since they can store the training data points or their representations to perform similarity-based, nearest neighbor, or learning vector quantization methods.
Revised: However, Hopﬁeld networks are promising for handling small datasets, since they can store the training data points or their representations to perform similarity-based, nearest neighbor, or learning vector quantization methods.

Original: Therefore, we test the Hopﬁeld layer Hopfield on the small datasets of the UC Irvine (UCI) Machine Learning Repository that have been used to benchmark supervised learning methods (Fernández-Delgado et al., 2014; Wainberg et al., 2016;
Revised: Therefore, we test the Hopﬁeld layer Hopfield on the small datasets of the UC Irvine (UCI) Machine Learning Repository that have been used to benchmark supervised learning methods (Fernández-Delgado et al., 2014; Wainberg et al., 2016;

Original: Khan et al., 2018) and also feed-forward neural networks (Klambauer et al., 2017a; Wu et al., 2018),
Revised: Khan et al., 2018) and also feed-forward neural networks (Klambauer et al., 2017a; Wu et al., 2018),

Original: where our
Revised: where our

Original: [Table]

Original: Hopﬁeld networks could exploit their memory.
Revised: Hopﬁeld networks could exploit their memory.

Original: The whole 121 datasets in the collection vary strongly with respect to their size, number of features [Table] and difﬁculties (Fernández-Delgado et al., 2014), such that they have been divided into 75 “small datasets” with less than 1,000 samples and 45 “large datasets” with more than or equal to 1,000 samples in Klambauer et al. (2017a).
Revised: The whole 121 datasets in the collection vary strongly with respect to their size, number of features, and difﬁculties (Fernández-Delgado et al., 2014), such that they have been divided into 75 “small datasets” with less than 1,000 samples and 45 “large datasets” with more than or equal to 1,000 samples in Klambauer et al. (2017a).

Original: [Table]
Revised: [Table]

Original: On the 75 small datasets, Random Forests (

Original: and Support Vector Machines (SVM) are highly accurate, whereas on the large datasets, deep learning methods and neural networks are in the lead (Klambauer et al., 2017a;b; Wu et al., 2018).
Revised: and Support Vector Machines (SVM) are highly accurate, whereas on the large datasets, deep learning methods and neural networks are in the lead (Klambauer et al., 2017a;b; Wu et al., 2018).

Original: We applied a modern Hopﬁeld network via the layer Hopfield , where a self-normalizing net (SNN) maps the input vector to Y and R .
Revised: We applied a modern Hopﬁeld network via the layer HopfieldLayer , where a selfnormalizing net (SNN) maps the input vector to Y and R .

Original: The output Z of Hopfield enters a softmax output.
Revised: The output Z of HopfieldLayer enters a softmax output.

Original: We compared our modern Hopﬁeld networks against deep learning methods (e.g. SNNs, resnet), RFs, SVMs, boosting, bagging, and many other machine learning methods of (Fernández-Delgado et al., 2014).
Revised: We compared our modern Hopﬁeld networks against deep learning methods (e.g. SNNs, resnet), RFs, SVMs, boosting, bagging, and many other machine learning methods of Fernández-Delgado et al.

Original: Since for each method, multiple variants and implementations had been included, we used method groups and representatives as deﬁned by Klambauer et al. (2017a).
Revised: Since for each method, multiple variants and implementations had been included, we used method groups and representatives as deﬁned by Klambauer et al. (2017a).

Original: For each dataset, a ranking of the methods was calculated which is presented in Table 2.
Revised: For each dataset, a ranking of the methods was calculated which is presented in Table 2.

Original: We found that Hopﬁeld networks outperform all other methods on the small datasets, setting a new state-of-the-art for 10 datasets.
Revised: We found that Hopﬁeld networks outperform all other methods on the small datasets, setting a new state-of-the-art for 10 datasets.

Original: The difference is signiﬁcant with a p -value of 0 . (Wilcoxon signed rank test).
Revised: The difference is signiﬁcant except for the ﬁrst three runner-up methods (Wilcoxon signed rank test).

Original: See appendix Section A.5.3 for details.
Revised: See appendix Section A.5.3 for details.

Original: Drug Design Benchmark Datasets.
Revised: Drug Design Benchmark Datasets.

Original: We test the Hopﬁeld layer HopfieldLayer , on four drug design datasets.
Revised: We test the Hopﬁeld layer HopfieldLayer , on four drug design datasets.

Original: These datasets represent four main areas of modeling tasks in drug design, concretely to develop accurate models for predicting a) new anti-virals (HIV) by the Drug Therapeutics Program (DTP)
Revised: These datasets represent four main areas of modeling tasks in drug design, concretely to develop accurate models for predicting a) new anti-virals (HIV) by the Drug Therapeutics Program (DTP)

Original: AIDS Antiviral Screen, b) new protein inhibitors, concretely human β -secretase (BACE) inhibitors by Subramanian et al.
Revised: AIDS Antiviral Screen, b) new protein inhibitors, concretely human β -secretase (BACE) inhibitors by Subramanian et al.

Original: (2016), c) metabolic effects as blood-brain barrier permeability (BBBP) (Martins et al., 2012) and d) side effects of a chemical compound from the Side Effect Resource (SIDER)
Revised: (2016), c) metabolic effects as blood-brain barrier permeability (BBBP) (Martins et al., 2012) and d) side effects of a chemical compound from the Side Effect Resource (SIDER)

Original: Kuhn et al.
Revised: Kuhn et al.

Original: We applied the Hopﬁeld layer HopfieldLayer , where the training data is used as stored patterns Y , the input vector as state pattern R , and the corresponding training label to project the output of the Hopﬁeld layer Y W V .
Revised: We applied the Hopﬁeld layer HopfieldLayer , where the training data is used as stored patterns Y , the input vector as state pattern R , and the corresponding training label to project the output of the Hopﬁeld layer Y W V .

Original: Our architecture with HopfieldLayer has reached state-of-the-art for predicting side effects on SIDER 0 .
Revised: Our architecture with HopfieldLayer has reached state-of-the-art for predicting side effects on SIDER 0 .

Original: 672 ± 0 .
Revised: 672 ± 0 .

Original: 019 as well as for predicting β -secretase BACE 0 .
Revised: 019 as well as for predicting β -secretase BACE 0 .

Original: 902 ± 0 .
Revised: 902 ± 0 .

Original: For details, see Table A.5 in the appendix.
Revised: For details, see Table A.5 in the appendix.

Original: Conclusion.
Revised: Conclusion.

Original: We have introduced a modern Hopﬁeld network with continuous states and the corresponding new update rule.
Revised: We have introduced a modern Hopﬁeld network with continuous states and the corresponding new update rule.

Original: This network can store exponentially many patterns, retrieves patterns with one update, and has exponentially small retrieval errors.
Revised: This network can store exponentially many patterns, retrieves patterns with one update, and has exponentially small retrieval errors.

Original: We analyzed the attention heads of BERT models.
Revised: We analyzed the attention heads of BERT models.

Original: The new modern Hopﬁeld networks have been integrated into deep learning architectures as layers to allow the storage of and access to raw input data, intermediate results, or learned prototypes.
Revised: The new modern Hopﬁeld networks have been integrated into deep learning architectures as layers to allow the storage of and access to raw input data, intermediate results, or learned prototypes.

Original: These Hopﬁeld layers enable new ways of deep learning, beyond fully-connected, convolutional, or recurrent networks, and provide pooling, memory, association, and attention mechanisms.
Revised: These Hopﬁeld layers enable new ways of deep learning, beyond fully-connected, convolutional, or recurrent networks, and provide pooling, memory, association, and attention mechanisms.

Original: Hopﬁeld layers that equip neural network layers with memories improved state-of-the-art in three out of four considered multiple instance learning problems and on immune repertoire classiﬁcation, and on two drug design dataset.
Revised: Hopﬁeld layers that equip neural network layers with memories improved state-of-the-art in three out of four considered multiple instance learning problems and on immune repertoire classiﬁcation, and on two drug design dataset.

Original: They yielded the best results among different machine learning methods on the
Revised: They yielded the best results among different machine learning methods on the

Original: UCI benchmark collections of small classiﬁcation tasks.
Revised: UCI benchmark collections of small classiﬁcation tasks.

Original: A A PPENDIX
Revised: A CKNOWLEDGMENTS

Revised: The ELLIS Unit Linz, the LIT AI Lab and the Institute for Machine Learning are supported by the Land Oberösterreich, LIT grants DeepToxGen (LIT-2017-3-YOU-003), and AI-SNN (LIT2018-6-YOU-214), the Medical Cognitive Computing Center (MC3), Janssen Pharmaceutica, UCB

Revised: Biopharma, Merck Group, Audi.

Revised: JKU Deep Learning Center, Audi Electronic Venture GmbH,

Revised: TGW, Primal, S3AI (FFG-872172), Silicon Austria Labs (SAL), FILL, EnliteAI, Google Brain,

Revised: ZF Friedrichshafen AG, Robert Bosch GmbH, TÜV Austria, DCS, and the NVIDIA Corporation.

Original:  This appendixto the paper “Hopﬁeld networks is all you need” consists of Section A.1 to Section A.6.
Revised: IARAI is supported by Here Technologies. A A PPENDIX This appendix consists of six sections (A.1–A.6).

Original: Section A.1 introduces the new modern Hopﬁeld network with continuous states and its update rule.
Revised: Section A.1 introduces the new modern Hopﬁeld network with continuous states and its update rule.

Original: Furthermore, Section A.1 provides a thorough and profound theoretical analysis of this new Hopﬁeld network.
Revised: Furthermore, Section A.1 provides a thorough and profound theoretical analysis of this new Hopﬁeld network.

Original: Section A.2 provides the mathematical background required in Section A.1.
Revised: Section A.2 provides the mathematical background for Section A.1.

Original: Section A. reviews binary Modern Hopﬁeld Networks of Krotov & Hopﬁeld.
Revised: Section A.3 reviews binary Modern Hopﬁeld Networks of Krotov & Hopﬁeld.

Original: Section A.4 shows that the Hopﬁeld update rule is the attention mechanism of the transformer.
Revised: Section A.4 shows that the Hopﬁeld update rule is the attention mechanism of the transformer.

Original: Section A.5 gives details on the experiments related to the new Hopﬁeld network.
Revised: Section A.5 gives details on the experiments.

Original: Section A.6 describes the PyTorch implementation of a Hopﬁeld layer and a Hopﬁeld pooling layer and how to use these layers.
Revised: Section A.6 describes the PyTorch implementation of layers based on the new Hopﬁeld networks and how to use them.

Original: C ONTENTS OF THE APPENDIX 
Revised: C ONTENTS OF THE APPENDIX A.1 Continuous State Modern Hopﬁeld Networks (A New Concept)

Original: A .1 Continuous State Modern Hopﬁeld Networks A New

Original: Concept) . . . . . . . . . . .

Original: A.1. Introduction A.
Revised: A.1.1 Introduction . .

Original:  New Energy Function A. [Table]
Revised: A.1.2 New Energy Function . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original:  New UpdateRule
Revised: A.1.3 New Update

Revised: Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A.1.

Original: Global Convergence of the Update Rule
Revised: Global Convergence of the Update Rule

Original: A
Revised: A.1.

Original: Local Convergenceof the Update Rule: FixedPoint Iteration
Revised: Local Convergence the Update Rule Fixed Iteration

Original: A.
Revised: A.

Original: 6 Properties of Fixed Points Near Stored Pattern
Revised: 6 Properties of Fixed Points Near Stored Pattern

Original: A 7 Learning Associations A.1.8 Inﬁnite Many Patterns and Forgetting Patterns . .
Revised: A.1.7 Learning Associations . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.1.
Revised: A.

Original: States . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Properties of Softmax, Log Sum- , Legendre Transform, Lambert W Function
Revised: Inﬁnite Many Patterns and Forgetting Patterns

Original: A.
Revised: A

Original: Hopﬁeld States ( and Hopﬁeld) . . . . . . . . . . 
Revised: Number of Spurious States Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A

Original: A.3.
Revised: A.3.

Original: Hopﬁeld Networks: Introduction . . . . . . . . . . . . . . . . . ..
Revised: Hopﬁeld : Introduction . . . . . . . . . . . . . . . . . .

Original: A.3.2 Energy and Update Rule for Binary Modern Hopﬁeld Networks . .
Revised: A.3.2 Energy and Update Rule for Binary Modern Hopﬁeld Networks . .

Original: A.4 Hopﬁeld Update Rule is Attention of The Transformer . . . . . . . . . . . . . . ..
Revised: A.4 Hopﬁeld Update Rule is Attention of The Transformer . . . . . . . . . . . . . . .

Original: A.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A.5.
Revised: A.5.

Original: in Transformersdescribed by Hopﬁeld dynamics . .
Revised: Transformers by Hopﬁeld dynamics . A Multiple

Original: A.5.2 Experiment
Revised: A Classiﬁcation

Original: Datasets. . . . . . . . . . . . . . .
Revised: Benchmark Datasets . . . . . .

Original: A.5.
Revised: A.5.

Original: Datasets . . . . . . . 
Revised: Datasets . . . . . . . . . . . . . .

Original: A.5.4 Experiment 4: Drug
Revised: A.6 PyTorch Implementation

Original: Benchmark Datasets . . . . . . . . . . . . . . .A.6 PyTorch of a Hopﬁeld
Revised: of Hopﬁeld Layers . . . . . . . . . . . . . . . . . . . .

Original: A.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.

Original: A.6.2 Functionality . .A.6.3 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: 2 Functionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A .6.3 Usage

Original: L IST OF THEOREMS
Revised: L IST OF THEOREMS

Original: A
Revised: A

Original: Theorem (Global Convergence (Zangwill): Energy) . . . . . . . . . . . . . . . . ..
Revised: Theorem (Global Convergence (Zangwill): Energy) . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Global Convergence: Stationary Points) . . . . . . . . . . . . . . . . . .
Revised: Theorem (Global Convergence: Stationary Points) . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Storage Capacity (M=2): Placed Patterns) . . . . . . . . . . . . . . . .
Revised: Theorem (Storage Capacity (M=2): Placed Patterns) . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Storage Capacity (M=5): Placed Patterns) . . . . . . . . . . . . . . . .
Revised: Theorem (Storage Capacity (M=5): Placed Patterns) . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Storage Capacity (Main): Random Patterns) . . . . . . . . . . . . . . .
Revised: Theorem (Storage Capacity (Main): Random Patterns) . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Storage Capacity (d computed): Random Patterns) . . . . . . . . . . . .
Revised: Theorem (Storage Capacity (d computed): Random Patterns) . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Storage Capacity (expected separation): Random Patterns) . . . . . . . ..
Revised: Theorem (Storage Capacity (expected separation): Random Patterns) . . . . . . . .

Original: A
Revised: A

Original: Theorem (Pattern Retrieval with One Update) . . . . . . . . . . . . . . . . . . . .
Revised: Theorem (Pattern Retrieval with One Update) . . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Theorem (Exponentially Small Retrieval Error) . . . . . . . . . . . . . . . . . . .
Revised: Theorem (Exponentially Small Retrieval Error) . . . . . . . . . . . . . . . . . . .

Original: A10 Theorem (Storage Capacity for Binary Modern Hopﬁeld Nets (Demircigil et al. 2017))
Revised: A10 Theorem (Storage Capacity for Binary Modern Hopﬁeld Nets (Demircigil et al. 2017))

Original: L IST OF DEFINITIONS
Revised: L IST OF DEFINITIONS

Original: A
Revised: A

Original: Deﬁnition (Softmax) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Deﬁnition (Softmax) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Deﬁnition (Log-Sum-Exp Function) . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Deﬁnition (Log-Sum-Exp Function) . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Deﬁnition (Convex Conjugate) . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Deﬁnition (Convex Conjugate) . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Original: Deﬁnition (Legendre Transform) . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Deﬁnition (Legendre Transform) . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: A
Revised: A

Revised: Deﬁnition (Epi-Sum) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Revised: A

Original: Deﬁnition (Lambert Function) . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Deﬁnition (Lambert Function) . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: L IST OF FIGURES
Revised: L IST OF FIGURES

Revised: A.1 The three cases of ﬁxed points .

Revised: A

Revised: A.

Revised: [Table]

Original: L IST OF TABLES
Revised: L IST OF TABLES

Original:  A.3 Hyperparameter search-space forgrid-search on small UCI benchmark datasets. All models were trained for 100 epochs using stochastic gradient descent with early stopping based on the validation set accuracy and a minibatch size of 4 samples. . .
Revised: A.1 Results of immune repertoire classiﬁcation across all datasets . . . . . . . . . . . . A.2 Hyperparameter selection for MIL datasets . . . . . . . . . . . . . . . . . . . . . A.3 Hyperparameter selection for small UCI benchmark datasets . . . . . . . . . . . .

Original: A.4 Hyperparameter search-space for grid-search on HIV, BACE, BBBP and SIDER. All models were trained if applicable for 4 epochs using Adam and a batch size of 1 sample.
Revised: A.4 Hyperparameter selection for drug design datasets . . . . . . . . . . . . . . . . . .

Original: A.5 Predictive performance (ROCAUC) on test set as reported by Jiang et al.
Revised: A.5 Results on benchmark datasets . . . . . . . . . . . . . . . . . . . . .

Original: (2020) for 50 random splits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: A.

Original: A. C ONTINUOUS S TATE M ODERN H OPFIELD N ETWORKS (A N EW C ONCEPT ) A.1. I NTRODUCTION
Revised: C ONTINUOUS S TATE M ODERN H OPFIELD N ETWORKS (A N EW C ONCEPT ) A.1. I NTRODUCTION

Original: In Section A.1 our new modern Hopﬁeld network is introduced.
Revised: In Section A.1 our new modern Hopﬁeld network is introduced.

Original: In Subsection A.1.2 we present the new energy function.
Revised: In Subsection A.1.2 we present the new energy function.

Original: Then in Subsection A.1.3, our new update rule is introduced.
Revised: Then in Subsection A.1.3, our new update rule is introduced.

Original: In Subsection A.1.4, we show that this update rule ensures global convergence.
Revised: In Subsection A.1.4, we show that this update rule ensures global convergence.

Original: We show that all the limit points of any sequence generated by the update rule are the stationary points (local minima or saddle points) of the energy function.
Revised: We show that all the limit points of any sequence generated by the update rule are the stationary points (local minima or saddle points) of the energy function.

Original: In Section A.1.5, we consider the local convergence of the update rule and see that patterns are retrieved with one update.
Revised: In Section A.1.5, we consider the local convergence of the update rule and see that patterns are retrieved with one update.

Original: In Subsection A.1.6, we consider the properties of the ﬁxed points that are associated with the stored patterns.
Revised: In Subsection A.1.6, we consider the properties of the ﬁxed points that are associated with the stored patterns.

Original: In Subsection A.1.6.1, we show that exponentially many patterns can be stored.
Revised: In Subsection A.1.6.1, we show that exponentially many patterns can be stored.

Original: The main result is given in Theorem A5: For random patterns on a sphere we can store and retrieve exponentially (in the dimension of the Hopﬁeld space) many patterns.
Revised: The main result is given in Theorem A5: For random patterns on a sphere we can store and retrieve exponentially (in the dimension of the Hopﬁeld space) many patterns.

Original: Subsection A.1.6.2 reports that patterns are typically retrieved with one update step and that the retrieval error is exponentially small.
Revised: Subsection A.1.6.2 reports that patterns are typically retrieved with one update step and that the retrieval error is exponentially small.

Original: In Subsection A.1.7, we consider how associations for the new Hopﬁeld networks can be learned.
Revised: In Subsection A.1.7, we consider how associations for the new Hopﬁeld networks can be learned.

Original: In Subsection A.1.7.2, we analyze if the association is learned directly by a bilinear form.
Revised: In Subsection A.1.7.2, we analyze if the association is learned directly by a bilinear form.

Original: In
Revised: In

Original: Subsection A.1.7.3, we analyze if stored patterns and query patterns are mapped to the space of the Hopﬁeld network.
Revised: Subsection A.1.7.3, we analyze if stored patterns and query patterns are mapped to the space of the Hopﬁeld network.

Original: Therefore, we treat the architecture of the transformer and BERT.
Revised: Therefore, we treat the architecture of the transformer and BERT.

Original: In
Revised: In

Original: Subsection A.1.8, we introduce a temporal component into the new Hopﬁeld network that leads to a forgetting behavior.
Revised: Subsection A.1.8, we introduce a temporal component into the new Hopﬁeld network that leads to a forgetting behavior.

Original: The forgetting allows us to treat inﬁnite memory capacity in Subsection A.1.8.1.
Revised: The forgetting allows us to treat inﬁnite memory capacity in Subsection A.1.8.1.

Original: In Subsection A.1.8.2, we consider the controlled forgetting behavior.
Revised: In Subsection A.1.8.2, we consider the controlled forgetting behavior.

Original: In Section A.2, we provide the mathematical background that is needed for our proofs.
Revised: In Section A.2, we provide the mathematical background that is needed for our proofs.

Original: In particular we give lemmas on properties of the softmax, the log-sum-exponential, the Legendre transform, and the Lambert W function.
Revised: In particular we give lemmas on properties of the softmax, the log-sum-exponential, the Legendre transform, and the Lambert W function.

Original: In Section A.3, we review the new Hopﬁeld network as introduced by Krotov and Hopﬁeld in 2016.
Revised: In Section A.3, we review the new Hopﬁeld network as introduced by Krotov and Hopﬁeld in 2016.

Original: However in contrast to our new Hopﬁeld network, the Hopﬁeld network of Krotov and Hopﬁeld is binary, that is, a network with binary states.
Revised: However in contrast to our new Hopﬁeld network, the Hopﬁeld network of Krotov and Hopﬁeld is binary, that is, a network with binary states.

Original: In Subsection A.3.1, we give an introduction to neural networks equipped with associative memories and new Hopﬁeld networks.
Revised: In Subsection A.3.1, we give an introduction to neural networks equipped with associative memories and new Hopﬁeld networks.

Original: In Subsection A.3.1.1, we discuss neural networks that are enhanced by an additional external memory and by attention mechanisms.
Revised: In Subsection A.3.1.1, we discuss neural networks that are enhanced by an additional external memory and by attention mechanisms.

Original: In Subsection A.3.1.2, we give an overview over the modern Hopﬁeld networks.
Revised: In Subsection A.3.1.2, we give an overview over the modern Hopﬁeld networks.

Original: Finally, in Subsection A.3.2, we present the energy function and the update rule for the modern, binary
Revised: Finally, in Subsection A.3.2, we present the energy function and the update rule for the modern, binary

Original: Hopﬁeld networks.
Revised: Hopﬁeld networks.

Original: A.1. N EW E NERGY F UNCTION
Revised: A.1. N EW E NERGY F UNCTION

Original: We have patterns x 1 , . . .
Revised: We have patterns x 1 , . . .

Original: , x N that are represented by the matrix
Revised: , x N that are represented by the matrix

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The largest norm of a pattern is
Revised: The largest norm of a pattern is

Original: [Equation]
Revised: [Equation]

Original: The query or state of the Hopﬁeld network is ξ .
Revised: The query or state of the Hopﬁeld network is ξ .

Original: The energy function E in the new type of Hopﬁeld models of Krotov and Hopﬁeld is E =
Revised: The energy function E in the new type of Hopﬁeld models of Krotov and Hopﬁeld is E =

Original: −
Revised: −

Original: (cid:80) Ni =
Revised: (cid:80) Ni =

Original: 1 F (cid:0) ξ T x
Revised: 1 F (cid:0) ξ T x

Original: i (cid:1) for binary patterns x i and binary state ξ with interaction function F ( x ) = x n , where n = 2 gives classical Hopﬁeld model (Krotov & Hopﬁeld, 2016).
Revised: i (cid:1) for binary patterns x i and binary state ξ with interaction function F ( x ) = x n , where n = 2 gives classical Hopﬁeld model (Krotov & Hopﬁeld, 2016).

Original: The storage capacity is proportional to d n − 1 (Krotov & Hopﬁeld, 2016).
Revised: The storage capacity is proportional to d n − 1 (Krotov & Hopﬁeld, 2016).

Original: This model was generalized by Demircigil et al.
Revised: This model was generalized by Demircigil et al.

Original: (Demircigil et al., 2017) to exponential interaction functions F ( x ) = exp( x ) , which gives the energy E = − exp(lse(1 , X T ξ )) .
Revised: (Demircigil et al., 2017) to exponential interaction functions F ( x ) = exp( x ) , which gives the energy E = − exp(lse(1 , X T ξ )) .

Original: This energy leads to an exponential storage capacity of N = 2 d/ 2 for binary patterns.
Revised: This energy leads to an exponential storage capacity of N = 2 d/ 2 for binary patterns.

Original: Furthermore, with a single update the ﬁxed point is recovered with high probability.
Revised: Furthermore, with a single update the ﬁxed point is recovered with high probability.

Original: See more details in Section A.3.
Revised: See more details in Section A.3.

Original: In contrast to the these binary modern Hopﬁeld networks, we focus on modern Hopﬁeld networks with continuous state s that can store continuous patterns .
Revised: In contrast to the these binary modern Hopﬁeld networks, we focus on modern Hopﬁeld networks with continuous state s that can store continuous patterns .

Original: We generalize the energy of Demircigil et al.
Revised: We generalize the energy of Demircigil et al.

Original: (Demircigil et al., 2017) to continuous states while keeping the lse properties which ensure high storage capacity and fast convergence.
Revised: (Demircigil et al., 2017) to continuous states while keeping the lse properties which ensure high storage capacity and fast convergence.

Original: Our new energy E for a continuous query or state ξ is deﬁned as
Revised: Our new energy E for a continuous query or state ξ is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: First let us collect and prove some properties of E .
Revised: First let us collect and prove some properties of E .

Original: The next lemma gives bounds on the energy E .
Revised: The next lemma gives bounds on the energy E .

Original: Lemma A1.
Revised: Lemma A1.

Original: The energy E is larger than zero:
Revised: The energy E is larger than zero:

Original: [Equation]
Revised: [Equation]

Revised: For ξ in the simplex deﬁned by the patterns

Original: [Equation]
Revised: [Equation]

Original: For ξ in the simplex deﬁned by the patterns, the energy E is upper bounded by:
Revised: the energy E is upper bounded by:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: We start by deriving the lower bound of zero.
Revised: We start by deriving the lower bound of zero.

Original: The pattern most similar to query or state ξ is x ξ :
Revised: The pattern most similar to query or state ξ is x ξ :

Original: [Equation]
Revised: [Equation]

Original: We obtain
Revised: We obtain

Original: [Equation]
Revised: [Equation]

Original: The energy is zero and, therefore, the bound attained, if all x i are equal, that is, x i = x
Revised: The energy is zero and, therefore, the bound attained, if all x i are equal, that is, x i = x

Original: for all i and ξ = x .
Revised: for all i and ξ = x .

Original: For deriving upper bounds on the energy E
Revised: For deriving upper bounds on the energy E

Original: [Equation]
Revised: [Equation]

Original: we require the the query ξ to be in the simplex deﬁned by the patterns
Revised: we require the the query ξ to be in the simplex deﬁned by the patterns

Original: [Equation]
Revised: [Equation]

Original: that is,
Revised: that is,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst upper bound is.
Revised: The ﬁrst upper bound is.

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst inequality we applied Lemma A19 to − lse( β, X T ξ ) with z =
Revised: For the ﬁrst inequality we applied Lemma A19 to − lse( β, X T ξ ) with z =

Original: p giving
Revised: p giving

Original: [Equation]
Revised: [Equation]

Original: as the term involving the logarithm is non-positive
Revised: as the term involving the logarithm is non-positive.

Original: [Equation] Next we derive the second upper bound
Revised: Next we derive the second upper bound, for which we need the mean m x of the patterns

Original: [Equation]
Revised: [Equation]

Original: for which we need the mean m x of the patterns

Original: [Equation]
Revised: [Equation]

Original: We obtain
Revised: We obtain

Original: [Equation]
Revised: [Equation]

Original: where for the ﬁrst inequality we again applied Lemma A19 with z = (1 /N, . .
Revised: where for the ﬁrst inequality we again applied Lemma A19 with z = (1 /N, . .

Original: , 1 /N )
Revised: , 1 /N )

Original: and β − 1 (cid:80) i 1 /N ln(1 /N )
Revised: and β − 1 (cid:80) i 1 /N ln(1 /N )

Original: = − β − 1 ln( N ) .
Revised: = − β − 1 ln( N ) .

Original: This inequality also follows from Jensen’s inequality.
Revised: This inequality also follows from Jensen’s inequality.

Original: The second inequality uses the Cauchy-Schwarz inequality.
Revised: The second inequality uses the Cauchy-Schwarz inequality.

Original: The last inequality uses
Revised: The last inequality uses

Original: [Equation]
Revised: [Equation]

Original: and
Revised: i

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: 26) and

Revised: [Equation]

Revised: [Equation]

Original: A.1. N EW U PDATE R ULE
Revised: A.1. N EW U PDATE R ULE

Original: We now introduce an update rule for minimizing the energy function E .
Revised: We now introduce an update rule for minimizing the energy function E .

Original: The new update rule is
Revised: The new update rule is

Original: [Equation]
Revised: [Equation]

Original: (27) where we used
Revised: where we used

Original: [Equation]
Revised: [Equation]

Original: The new state ξ new is in the simplex deﬁned by the patterns, no matter what the previous state ξ was.
Revised: The new state ξ new is in the simplex deﬁned by the patterns, no matter what the previous state ξ was.

Original: For comparison, the synchronous update rule for the classical Hopﬁeld network with threshold zero is [Equation]
Revised: For comparison, the synchronous update rule for the classical Hopﬁeld network with threshold zero is ξ new = sgn ( XX T ξ ) .

Original: Therefore, instead of using the vector X T ξ as in the classical Hopﬁeld network, its softmax version softmax( β X T ξ ) is used.
Revised: Therefore, instead of using the vector X T ξ as in the classical Hopﬁeld network, its softmax version softmax( β X T ξ ) is used.

Original: In the next section (Section A.1.4) we show that the update rule Eq.
Revised: In the next section (Section A.1.4) we show that the update rule Eq.

Original: (27) ensures global convergence.
Revised: (28) ensures global convergence.

Original: We show that all the limit points of any sequence generated by the update rule are the stationary points (local minima or saddle points) of the energy function E .
Revised: We show that all the limit points of any sequence generated by the update rule are the stationary points (local minima or saddle points) of the energy function E .

Original: In Section A.1.5 we consider the local convergence of the update rule Eq.
Revised: In Section A.1.5 we consider the local convergence of the update rule Eq.

Original: (27) and see that patterns are retrieved with one update.
Revised: (28) and see that patterns are retrieved with one update.

Original: A.1. G LOBAL C ONVERGENCE OF THE U PDATE R ULE
Revised: A.1. G LOBAL C ONVERGENCE OF THE U PDATE R ULE

Original: We are interested in the global convergence , that is, convergence from each initial point, of the iteration
Revised: We are interested in the global convergence , that is, convergence from each initial point, of the iteration

Original: [Equation]
Revised: [Equation]

Original: where we used
Revised: where we used

Original: [Equation]
Revised: [Equation]

Original: We deﬁned the energy function
Revised: We deﬁned the energy function

Original: [Equation]
Revised: [Equation]

Original: We will show that the update rule in Eq.
Revised: We will show that the update rule in Eq.

Original: (30) is the Concave-Convex Procedure (CCCP) for minimizing the energy E .
Revised: (31) is the Concave-Convex Procedure (CCCP) for minimizing the energy E .

Original: The CCCP is proven to converge globally.
Revised: The CCCP is proven to converge globally.

Original: Theorem A1 (Global Convergence (Zangwill): Energy) .
Revised: Theorem A1 (Global Convergence (Zangwill): Energy) .

Original: The update rule Eq.
Revised: The update rule Eq.

Original: (30) converges globally:
Revised: (31) converges globally:

Original: For ξ t +1 = f ( ξ t ) , the energy E( ξ t ) → E( ξ ∗ ) for t → ∞ and a ﬁxed point ξ ∗ .
Revised: For ξ t +1 = f ( ξ t ) , the energy E( ξ t ) → E( ξ ∗ ) for t → ∞ and a ﬁxed point ξ ∗ .

Original: Proof.
Revised: Proof.

Original: The Concave-Convex Procedure (CCCP) (Yuille & Rangarajan, 2002; 2003) minimizes a function that is the sum of a concave function and a convex function.
Revised: The Concave-Convex Procedure (CCCP) (Yuille & Rangarajan, 2002; 2003) minimizes a function that is the sum of a concave function and a convex function.

Original: CCCP is equivalent to Legendre minimization (Rangarajan et al., 1996; 1999) algorithms (Yuille & Rangarajan, 2003).
Revised: CCCP is equivalent to Legendre minimization (Rangarajan et al., 1996; 1999) algorithms (Yuille & Rangarajan, 2003).

Original: The Jacobian of the softmax is positive semi-deﬁnite according to Lemma A22.
Revised: The Jacobian of the softmax is positive semi-deﬁnite according to Lemma A22.

Original: The Jacobian of the softmax is the Hessian of the lse , therefore lse is a convex and − lse a concave function.
Revised: The Jacobian of the softmax is the Hessian of the lse , therefore lse is a convex and − lse a concave function.

Original: Therefore, the energy function E( ξ ) is the sum of the convex function E 1 ( ξ )
Revised: Therefore, the energy function E( ξ ) is the sum of the convex function E 1 ( ξ )

Original: = 1 / 2 ξ T ξ + C 1 and the concave function
Revised: = 1 / 2 ξ T ξ + C 1 and the concave function

Revised: E 2 (

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: where C 1 does not depend on ξ .
Revised: where C 1 does not depend on ξ .

Original: The Concave-Convex Procedure (CCCP) (Yuille & Rangarajan, 2002; 2003) applied to E is
Revised: The Concave-Convex Procedure (CCCP) (Yuille & Rangarajan, 2002; 2003) applied to E is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which is
Revised: which is

Original: [Equation]
Revised: [Equation]

Original: The resulting update rule is
Revised: The resulting update rule is

Original: [Equation]
Revised: [Equation]

Original: 39) usingp
Revised: using

Original: [Equation]
Revised: [Equation]

Original: This is the update rule in Eq.
Revised: This is the update rule in Eq.

Original: Theorem 2 in Yuille & Rangarajan (2002) and Theorem 2 in Yuille & Rangarajan (2003) state that the update rule Eq.
Revised: Theorem 2 in Yuille & Rangarajan (2002) and Theorem 2 in Yuille & Rangarajan (2003) state that the update rule Eq.

Original: (30) is guaranteed to monotonically decrease the energy E as a function of time.
Revised: (31) is guaranteed to monotonically decrease the energy E as a function of time.

Original: See also Theorem 2 in Sriperumbudur & Lanckriet (2009).
Revised: See also Theorem 2 in Sriperumbudur & Lanckriet (2009).

Original: Although the objective converges in all cases, it does not necessarily converge to a local minimum (Lipp & Boyd, 2016).
Revised: Although the objective converges in all cases, it does not necessarily converge to a local minimum (Lipp & Boyd, 2016).

Original: However the convergence proof of CCCP in Yuille & Rangarajan (2002; 2003) was not as rigorous as required.
Revised: However the convergence proof of CCCP in Yuille & Rangarajan (2002; 2003) was not as rigorous as required.

Original: In Sriperumbudur & Lanckriet (2009) a rigorous analysis of the convergence of CCCP is performed using Zangwill’s global convergence theory of iterative algorithms.
Revised: In Sriperumbudur & Lanckriet (2009) a rigorous analysis of the convergence of CCCP is performed using Zangwill’s global convergence theory of iterative algorithms.

Original: In Sriperumbudur & Lanckriet (2009) the minimization problem
Revised: In Sriperumbudur & Lanckriet (2009) the minimization problem

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: is considered with E 1 convex, − E 2 convex, c component-wise convex function, and d an afﬁne function.
Revised: is considered with E 1 convex, − E 2 convex, c component-wise convex function, and d an afﬁne function.

Original: The CCCP algorithm solves this minimization problem by linearization of the concave part and is deﬁned in Sriperumbudur & Lanckriet (2009) as
Revised: The CCCP algorithm solves this minimization problem by linearization of the concave part and is deﬁned in Sriperumbudur & Lanckriet (2009) as

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the upper bound E C on the energy:
Revised: We deﬁne the upper bound E C on the energy:

Original: [Equation]
Revised: [Equation]

Original: E C is equal to the energy E ( ξ t ) for ξ = ξ t :
Revised: E C is equal to the energy E ( ξ t ) for ξ = ξ t :

Original: [Equation]
Revised: [Equation]

Original: Since − E 2 is convex, the ﬁrst order characterization of convexity holds (Eq. 3.2 in Boyd & Vandenberghe (2009)): 
Revised: Since − E 2 is convex, the ﬁrst order characterization of convexity holds (Eq. 3.2 in Boyd & Vandenberghe (2009)): [Equation]

Original: [Equation]
Revised: [Equation]

Original: that is
Revised: that is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, for ξ (cid:54) =
Revised: Therefore, for ξ (cid:54) =

Original: ξ t the function E C is an upper bound on the energy:
Revised: ξ t the function E C is an upper bound on the energy:

Original: [Equation]
Revised: [Equation]

Original: where C 2 does not depend on ξ .
Revised: where C 2 does not depend on ξ .

Original: Since we do not have constraints, ξ t +1 is deﬁned as
Revised: Since we do not have constraints, ξ t +1 is deﬁned as

Original: [Equation]
Revised: [Equation]

Original:  hence E C (cid:0) ξ t +1 , ξ t [Equation] (cid:54) E C ( ξ t , ξ t ) .
Revised: 49) hence E C (cid:0) ξ t +1 , ξ t (cid:1) (cid:54) E C ( ξ t , ξ t ) .

Original: Combining the inequalities gives:
Revised: Combining the inequalities gives:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since we do not have constraints, ξ t +1 is the minimum of
Revised: Since we do not have constraints, ξ t +1 is the minimum of

Original: [Equation]
Revised: [Equation]

Original:  as a function of ξ .
Revised: 51) as a function of ξ .

Original: For a minimum not at the border, the derivative has to be the zero vector
Revised: For a minimum not at the border, the derivative has to be the zero vector

Original: [Equation]
Revised: [Equation]

Original: and the Hessian must be positive semi-deﬁnite
Revised: and the Hessian must be positive semi-deﬁnite

Original: [Equation]
Revised: [Equation]

Original: The Hessian is strict positive deﬁnite everywhere, therefore the optimization problem is strict convex (if the domain is convex) and there exist only one minimum, which is a global minimum.
Revised: The Hessian is strict positive deﬁnite everywhere, therefore the optimization problem is strict convex (if the domain is convex) and there exist only one minimum, which is a global minimum.

Original: E C can even be written as a quadratic form:
Revised: E C can even be written as a quadratic form:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where C 3 does not depend on ξ .
Revised: where C 3 does not depend on ξ .

Original: Therefore, the minimum is
Revised: Therefore, the minimum is

Original: [Equation]
Revised: [Equation]

Original: 54) if it is in the domain as we assume.
Revised: 55) if it is in the domain as we assume.

Original: Using M = max i (cid:107) x i (cid:107) , ξ t +1 is in the sphere S
Revised: Using M = max i (cid:107) x i (cid:107) , ξ t +1 is in the sphere S

Original: = { x | (cid:107) x (cid:107) (cid:54) M } which is a convex and compact set.
Revised: = { x | (cid:107) x (cid:107) (cid:54) M } which is a convex and compact set.

Original: Hence, if ξ 0 ∈ S , then the iteration is a mapping from S to S .
Revised: Hence, if ξ 0 ∈ S , then the iteration is a mapping from S to S .

Original: Therefore, the point-set-map deﬁned by the iteration Eq.
Revised: Therefore, the point-set-map deﬁned by the iteration Eq.

Original: (54) is uniformly compact on S according to Remark 7 in Sriperumbudur & Lanckriet (2009).
Revised: (55) is uniformly compact on S according to Remark 7 in Sriperumbudur & Lanckriet (2009).

Original: Theorem 2 and Theorem 4 in (Sriperumbudur & Lanckriet, 2009) states that all the limit points of the iteration Eq.
Revised: Theorem 2 and Theorem 4 in (Sriperumbudur & Lanckriet, 2009) states that all the limit points of the iteration Eq.

Original: (54) are stationary points.
Revised: (55) are stationary points.

Original: These theorems follow from Zangwill’s global convergence theorem: Convergence Theorem A, page 91 in Zangwill (1969) and page 3 in Wu (1983).
Revised: These theorems follow from Zangwill’s global convergence theorem: Convergence Theorem A, page 91 in Zangwill (1969) and page 3 in Wu (1983).

Original: The global convergence theorem only assures that for the sequence ξ t +1 = f
Revised: The global convergence theorem only assures that for the sequence ξ t +1 = f

Original: ( ξ t ) and a function Φ we have Φ( ξ t ) → Φ( ξ ∗ ) for t → ∞ but not ξ t → ξ ∗ .
Revised: ( ξ t ) and a function Φ we have Φ( ξ t ) → Φ( ξ ∗ ) for t → ∞ but not ξ t → ξ ∗ .

Original: However, if f is strictly monotone with respect to Φ , then we can strengthen Zangwill’s global convergence theorem (Meyer, 1976).
Revised: However, if f is strictly monotone with respect to Φ , then we can strengthen Zangwill’s global convergence theorem (Meyer, 1976).

Original: We set Φ =
Revised: We set Φ =

Original: E and show E( ξ t +1 )
Revised: E and show E( ξ t +1 )

Original: <
Revised: <

Original: E( ξ t ) if ξ t is not a stationary point of E , that is, f is strictly monotone with respect to E .
Revised: E( ξ t ) if ξ t is not a stationary point of E , that is, f is strictly monotone with respect to E .

Original: The following theorem is similar to the convergence results for the expectation maximization (EM) algorithm in Wu (1983) which are given in theorems 1 to 6 in Wu (1983).
Revised: The following theorem is similar to the convergence results for the expectation maximization (EM) algorithm in Wu (1983) which are given in theorems 1 to 6 in Wu (1983).

Original: The following theorem is also very similar to Theorem 8 in Sriperumbudur & Lanckriet (2009).
Revised: The following theorem is also very similar to Theorem 8 in Sriperumbudur & Lanckriet (2009).

Original: Theorem A2 (Global Convergence: Stationary Points) .
Revised: Theorem A2 (Global Convergence: Stationary Points) .

Original: For the iteration Eq.
Revised: For the iteration Eq.

Original: (54) we have E ( ξ t ) →
Revised: (55) we have E ( ξ t ) →

Original: E ( ξ ∗ ) =
Revised: E ( ξ ∗ ) =

Original: E ∗ as t → ∞ , for some stationary point ξ ∗ .
Revised: E ∗ as t → ∞ , for some stationary point ξ ∗ .

Original: Furthermore (cid:13)(cid:13)
Revised: Furthermore (cid:13)(cid:13)

Original: ξ t +1 − ξ t (cid:13)(cid:13) → 0 and either { ξ t } ∞ t =0 converges or, in the other case, the set of limit points of { ξ t } ∞ t =0 is a connected and compact subset of L (E ∗ ) , where L ( a ) =
Revised: ξ t +1 − ξ t (cid:13)(cid:13) → 0 and either { ξ t } ∞ t =0 converges or, in the other case, the set of limit points of { ξ t } ∞ t =0 is a connected and compact subset of L (E ∗ ) , where L ( a ) =

Original: { ξ ∈ L | E ( ξ ) = a } and L is the set of stationary points of the iteration Eq.
Revised: { ξ ∈ L | E ( ξ ) = a } and L is the set of stationary points of the iteration Eq.

Original: If L (E ∗ ) is ﬁnite, then any sequence { ξ t } ∞ t =0 generated by the iteration Eq.
Revised: If L (E ∗ ) is ﬁnite, then any sequence { ξ t } ∞ t =0 generated by the iteration Eq.

Original: (54) converges to some ξ ∗ ∈ L (E ∗ ) .
Revised: (55) converges to some ξ ∗ ∈ L (E ∗ ) .

Original: Proof.
Revised: Proof.

Original: We have E ( ξ t ) =
Revised: We have E ( ξ t ) =

Original: E 1 ( ξ t ) + E 2 ( ξ t ) .
Revised: E 1 ( ξ t ) + E 2 ( ξ t ) .

Original: The gradient ∇ ξ E 2 ( ξ t ) =
Revised: The gradient ∇ ξ E 2 ( ξ t ) =

Original: −∇ ξ lse( β, X T ξ ) is continuous.
Revised: −∇ ξ lse( β, X T ξ ) is continuous.

Original: Therefore, Eq.
Revised: Therefore, Eq.

Original: (50) has minimum in the sphere S , which is a convex and compact set.
Revised: (51) has minimum in the sphere S , which is a convex and compact set.

Original: If ξ t +1 (cid:54) =
Revised: If ξ t +1 (cid:54) =

Original: ξ t , then ξ t was not the minimum of Eq.
Revised: ξ t , then ξ t was not the minimum of Eq.

Original: (47) as the derivative at ξ t is not equal to zero.
Revised: (48) as the derivative at ξ t is not equal to zero.

Original: Eq.
Revised: Eq.

Original: (52) shows that the optimization problem Eq.
Revised: (53) shows that the optimization problem Eq.

Original: (47) is strict convex, hence it has only one minimum, which is a global minimum.
Revised: (48) is strict convex, hence it has only one minimum, which is a global minimum.

Original: Eq.
Revised: Eq.

Original: (53) shows that the optimization problem Eq.
Revised: (54) shows that the optimization problem Eq.

Original: (47) is even a quadratic form.
Revised: (48) is even a quadratic form.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, the point-set-map deﬁned by the iteration Eq.
Revised: Therefore, the point-set-map deﬁned by the iteration Eq.

Original: (54) (for deﬁnitions see (Sriperumbudur & Lanckriet, 2009)) is strictly monotonic with respect to E .
Revised: (55) (for deﬁnitions see (Sriperumbudur & Lanckriet, 2009)) is strictly monotonic with respect to E .

Original: Therefore, we can apply Theorem 3 in
Revised: Therefore, we can apply Theorem 3 in

Original: Sriperumbudur & Lanckriet (2009) or Theorem 3.1 and Corollary 3.2 in Meyer (1976), which give the statements of the theorem.
Revised: Sriperumbudur & Lanckriet (2009) or Theorem 3.1 and Corollary 3.2 in Meyer (1976), which give the statements of the theorem.

Original: We showed global convergence of the iteration Eq.
Revised: We showed global convergence of the iteration Eq.

Original: We have shown that all the limit points of any sequence generated by the iteration Eq.
Revised: We have shown that all the limit points of any sequence generated by the iteration Eq.

Original: (30) are the stationary points (critical points; local minima or saddle points) of the energy function E .
Revised: (31) are the stationary points (critical points; local minima or saddle points) of the energy function E .

Original: Local maxima as stationary points are only possible if the iterations exactly hits a local maximum.
Revised: Local maxima as stationary points are only possible if the iterations exactly hits a local maximum.

Original: However, convergence to a local maximum without being there is not possible because Eq.
Revised: However, convergence to a local maximum without being there is not possible because Eq.

Original: (55) ensures a strict decrease of the energy E .
Revised: (56) ensures a strict decrease of the energy E .

Original: Therefore, almost sure local maxima are not obtained as stationary points.
Revised: Therefore, almost sure local maxima are not obtained as stationary points.

Original: Either the iteration converges or, in the second case, the set of limit points is a connected and compact set.
Revised: Either the iteration converges or, in the second case, the set of limit points is a connected and compact set.

Original: But what happens if ξ 0 is in an (cid:15) -neighborhood around a local minimum ξ ∗ ?
Revised: But what happens if ξ 0 is in an (cid:15) -neighborhood around a local minimum ξ ∗ ?

Original: Will the iteration Eq.
Revised: Will the iteration Eq.

Original: (30) converge to ξ ∗ ?
Revised: (31) converge to ξ ∗ ?

Original: What is the rate of convergence?
Revised: What is the rate of convergence?

Original: These questions are about local convergence which will be treated in detail in next section. 
Revised: These questions are about local convergence which will be treated in detail in next section. A. L OCAL

Original: A.1. L OCAL CONVERGENCE OF THE U PDATE R ULE : F IXED P OINT I TERATION
Revised: C

Revised: ONVERGENCE OF THE U PDATE R ULE : F IXED P OINT I TERATION

Original: For the proof of local convergence to a ﬁxed point we will apply Banach ﬁxed point theorem.
Revised: For the proof of local convergence to a ﬁxed point we will apply Banach ﬁxed point theorem.

Original: For the rate of convergence we will rely on properties of a contraction mapping.
Revised: For the rate of convergence we will rely on properties of a contraction mapping.

Original: A.1.5.1 General Bound on the Jacobian of the Iteration.
Revised: A.1.5.1 General Bound on the Jacobian of the Iteration.

Original: We consider the iteration
Revised: We consider the iteration

Original: [Equation]
Revised: [Equation]

Original: 56) using
Revised: 57) using

Original: [Equation]
Revised: [Equation]

Original: The Jacobian J is symmetric and has the following form:
Revised: The Jacobian J is symmetric and has the following form:

Original: [Equation]
Revised: [Equation]

Original: 58) where J s is Jacobian of the softmax.
Revised: 59) where J s is Jacobian of the softmax.

Original: To analyze the local convergence of the iteration, we distinguish between the following three cases (see also Fig.
Revised: To analyze the local convergence of the iteration, we distinguish between the following three cases (see also Fig.

Original: A.1).
Revised: A.1).

Original: Here we only provide an informal discussion to give the reader some intuition.
Revised: Here we only provide an informal discussion to give the reader some intuition.

Original: A rigorous formulation of the results can be found in the corresponding subsections.
Revised: A rigorous formulation of the results can be found in the corresponding subsections.

Original: a) If the patterns x i are not well separated, the iteration goes to a ﬁxed point close to the arithmetic mean of the vectors.
Revised: a) If the patterns x i are not well separated, the iteration goes to a ﬁxed point close to the arithmetic mean of the vectors.

Original: In this case p is close to p i = 1 /N .
Revised: In this case p is close to p i = 1 /N .

Original: b) If the patterns x i are well separated, then the iteration goes to the pattern to which the initial ξ is similar.
Revised: b) If the patterns x i are well separated, then the iteration goes to the pattern to which the initial ξ is similar.

Original: If the initial ξ is similar to a vector x i then it will converge to a vector close to x i and p will converge to a vector close to e i .
Revised: If the initial ξ is similar to a vector x i then it will converge to a vector close to x i and p will converge to a vector close to e i .

Original: c)
Revised: c)

Original: If some vectors are similar to each other but well separated from all other vectors, then a so called metastable state between the similar vectors exists.
Revised: If some vectors are similar to each other but well separated from all other vectors, then a so called metastable state between the similar vectors exists.

Original: Iterations that start near the metastable state converge to this metastable state.
Revised: Iterations that start near the metastable state converge to this metastable state.

Original: [Figure]
Revised: [Figure]

Original: m is the ﬁxed point (ﬁxed point is average of all patterns) no pattern is well separated from the others
Revised: ) Metastable state (ﬁxed point is average of similar patterns) : x i and x j are similar to each other and not well separated.

Original:  A  m  is 
Revised: The ﬁxed point m ∗ x is a metastable state that is close to the mean m x of the similar patterns. c) Global ﬁxed point (ﬁxed point is average of all patterns) : no pattern is well separated from the others. A single global ﬁxed point m ∗ x exists that is close to the arithmetic mean m x of all patterns.

Original: We begin with a bound on the Jacobian of the iteration, thereby heavily relying on the Jacobian of the softmax from Lemma A24.
Revised: We begin with a bound on the Jacobian of the iteration, thereby heavily relying on the Jacobian of the softmax from Lemma A24.

Original: Lemma A2.
Revised: Lemma A2.

Original: For N patterns X
Revised: For N patterns X

Original: =
Revised: =

Original: ( x 1 , . .
Revised: ( x 1 , . .

Original: , x N ) , p = softmax( β X T ξ ) , M = max i (cid:107) x
Revised: , x N ) , p = softmax( β X T ξ ) , M = max i (cid:107) x

Original: i (cid:107) , and m = max i p
Revised: i (cid:107) , and m = max i p

Original: i (1 − p i ) , the spectral norm of the Jacobian J of the ﬁxed point iteration is bounded:[Equation] 22 m [Equation]
Revised: i (1 − p i ) , the spectral norm of the Jacobian J of the ﬁxed point iteration is bounded:

Original: β N

Original: [Equation]
Revised: [Equation]

Original: If p max = max i p i ≥ 1 − (cid:15) , then for the spectral norm of the Jacobian holds
Revised: If p max = max i p i ≥ 1 − (cid:15) , then for the spectral norm of the Jacobian holds

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Withp
Revised: With

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  the symmetric Jacobian J is
Revised: 62) the symmetric Jacobian J is

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where J s is Jacobian of the softmax.
Revised: where J s is Jacobian of the softmax.

Original: With m = max i p i (1 − p i ) , Eq.
Revised: With m = max i p i (1 − p i ) [Equation] Eq.

Original: (475) from Lemma A24 is
Revised: (476) from Lemma A24 is

Original: [Equation]
Revised: [Equation]

Original: Using this bound on (cid:107) J s (cid:107) 2 , we obtain
Revised: Using this bound on (cid:107) J s (cid:107) [Equation] we obtain

Original: [Equation]
Revised: [Equation]

Original: The spectral norm (cid:107) .
Revised: The spectral norm (cid:107) .

Original: (cid:107) 2 is bounded by the Frobenius norm (cid:107) .
Revised: (cid:107) 2 is bounded by the Frobenius norm (cid:107) .

Original: (cid:107) F which can be expressed by the norm squared of its column vectors:
Revised: (cid:107) F which can be expressed by the norm squared of its column vectors:

Original: [Equation]
Revised: [Equation]

Original: Therefore, we obtain the ﬁrst statement of the lemma:
Revised: Therefore [Equation] we obtain the ﬁrst statement of the lemma:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: With p max = max i p i ≥ 1 − (cid:15) Eq.
Revised: With p max = max i p i ≥ 1 − (cid:15) Eq.

Original: (479) in Lemma A24 is
Revised: (480) in Lemma A24 is

Original: [Equation]
Revised: [Equation]

Original: Using this inequality [Equation] we obtain the second statement of the lemma:
Revised: Using this inequality, we obtain the second statement of the lemma:

Original: [Equation]
Revised: [Equation]

Original: We now deﬁne the “separation” ∆ i of a pattern x
Revised: We now deﬁne the “separation” ∆ i of a pattern x

Original: i from data X = ( x 1 , . .
Revised: i from data X = ( x 1 , . .

Original: , x N ) here, since it has an important role for the convergence properties of the iteration.
Revised: , x N ) here, since it has an important role for the convergence properties of the iteration.

Original: Deﬁnition 2 (Separation of Patterns) .
Revised: Deﬁnition 2 (Separation of Patterns) .

Original: We deﬁne ∆ i , i [Equation] e. the separation of pattern x i from data
Revised: We deﬁne ∆ i , i.e. the separation of pattern x i from data

Original: X = ( x 1 , . . .
Revised: X = ( x 1 , . . .

Original: , x N ) as:
Revised: , x N ) as:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The pattern is separated from the other data if 0 < ∆ i .
Revised: The pattern is separated from the other data if 0 < ∆ i .

Original: Using the parallelogram identity, ∆ i can also be expressed as
Revised: Using the parallelogram identity, ∆ i can also be expressed as

Original: [Equation]
Revised: [Equation]

Original: j (cid:54)

Original: [Equation]

Original: (cid:16) (cid:107)

Original: [Equation]

Original:  (cid:107) [Equation] (cid:107) 
Revised: For (cid:107) x i (cid:107) =

Original: [Equation]

Original: (cid:107)

Original: [Equation] (cid:17) . For (cid:107) x i (cid:107) = (cid:107) x j (cid:107) we have ∆ i = 1 / 2 min j,j (cid:54) =
Revised: (cid:107) x j (cid:107) we have ∆ i = 1 / 2 min j,j (cid:54) =

Original: i (cid:107) x
Revised: i (cid:107) x

Original: i − x j (cid:107) 2 .
Revised: i − x j (cid:107) 2 .

Original: Analog we say for a query ξ and data X =
Revised: Analog we say for a query ξ and data X =

Original: ( x 1 , . .
Revised: ( x 1 , . .

Original: , x N ) , that x i is least separated from ξ while being separated from other x j with j (cid:54)
Revised: , x N ) , that x i is least separated from ξ while being separated from other x j with j (cid:54)

Original: =
Revised: =

Original: i if
Revised: i if

Original: [Equation]
Revised: [Equation]

Original: Next we consider the case where the iteration has only one stable ﬁxed point.
Revised: Next we consider the case where the iteration has only one stable ﬁxed point.

Original: A.1.5.2 One Stable State: Fixed Point Near the Mean of the Patterns.
Revised: A.1.5. One Stable State: Fixed Point Near the Mean of the Patterns.

Original: We start with the case where no pattern is well separated from the others.
Revised: We start with the case where no pattern is well separated from the others

Original: • Global ﬁxed point near the global mean: Analysis using the data center.
Revised: • Global ﬁxed point near the global mean: Analysis using the data center.

Original: We revisit the bound on the Jacobian of the iteration by utilizing properties of pattern distributions.
Revised: We revisit the bound on the Jacobian of the iteration by utilizing properties of pattern distributions.

Original: We begin with a probabilistic interpretation where we consider p i as the probability of selecting the vector x i .
Revised: We begin with a probabilistic interpretation where we consider p i as the probability of selecting the vector x i .

Original: Consequently, we deﬁne expectations as E p [ f ( x )]
Revised: Consequently, we deﬁne expectations as E p [ f ( x )]

Original: = (cid:80) Ni =1 p i f ( x i ) .
Revised: = (cid:80) Ni =1 p i f ( x i ) .

Original: In this setting the matrix
Revised: In this setting the matrix

Original: [Equation]
Revised: [Equation]

Original: 73) is the covariance matrix of data X when its vectors are selected according to the probability p :
Revised: 74) is the covariance matrix of data X when its vectors are selected according to the probability p :

Original: [Equation]
Revised: [Equation]

Original: therefore we have
Revised: therefore we have

Original: [Equation]
Revised: [Equation]

Original: The largest eigenvalue of the covariance matrix (equal to the largest singular value) is the variance in the direction of the eigenvector associated with the largest eigenvalue.
Revised: The largest eigenvalue of the covariance matrix (equal to the largest singular value) is the variance in the direction of the eigenvector associated with the largest eigenvalue.

Original: We deﬁne:
Revised: We deﬁne:

Original: [Equation]
Revised: [Equation]

Original: m x is the arithmetic mean (the center) of the patterns.
Revised: m x is the arithmetic mean (the center) of the patterns.

Original: m max is the maximal distance of the patterns to the center m x .
Revised: m max is the maximal distance of the patterns to the center m x .

Original: The variance of the patterns is
Revised: The variance of the patterns is

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The maximal distance to the center m max allows the derivation of a bound on the norm of the
Revised: The maximal distance to the center m max allows the derivation of a bound on the norm of the

Original: Jacobian.
Revised: Jacobian.

Original: Next lemma gives a condition for a global ﬁxed point.
Revised: Next lemma gives a condition for a global ﬁxed point.

Original: Lemma A3.
Revised: Lemma A3.

Original: The following bound on the norm (cid:107) J (cid:107) 2 of the Jacobian of the ﬁxed point iteration f holds independent of p or the query ξ .[Equation]
Revised: The following bound on the norm (cid:107) J (cid:107) 2 of the Jacobian of the ﬁxed point iteration f holds independent of p or the query ξ .

Original: [Equation]
Revised: [Equation]

Original: For β m 2max < 1 there exists a unique ﬁxed point (global ﬁxed point) of iteration f in each compact set.
Revised: For β m 2max < 1 there exists a unique ﬁxed point (global ﬁxed point) of iteration f in each compact set.

Original: Proof.
Revised: Proof.

Original: In order to bound the variance we compute the vector a that minimizes 
Revised: In order to bound the variance we compute the vector a that minimizes f ( a ) =

Revised: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: The solution to

Original: [Equation]
Revised: [Equation]

Original: is
Revised: is

Original: [Equation]
Revised: [Equation]

Original: The Hessian of f is positive deﬁnite since
Revised: The Hessian of f is positive deﬁnite since

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: and f is a convex function.
Revised: and f is a convex function.

Original: Hence, the mean 
Revised: Hence, the mean ¯ x

Original: [Equation]
Revised: [Equation]

Original: minimizes [Equation] = [Equation]
Revised: minimizes (cid:80) Ni =1 p

Original: p [Equation] x i − a (cid:107) 2 .
Revised: i (cid:107) x i − a (cid:107) 2 .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Let us quickly recall that the spectral norm of an outer product of two vectors is the product of the
Revised: Let us quickly recall that the spectral norm of an outer product of two vectors is the product of the

Original: Euclidean norms of the vectors:
Revised: Euclidean norms of the vectors:

Original: [Equation]
Revised: [Equation]

Original: since bb T has eigenvector b / (cid:107) b (cid:107) with eigenvalue (cid:107) b (cid:107) 2 and otherwise zero eigenvalues.
Revised: since bb T has eigenvector b / (cid:107) b (cid:107) with eigenvalue (cid:107) b (cid:107) 2 and otherwise zero eigenvalues.

Original: We now bound the variance of the patterns:
Revised: We now bound the variance of the patterns:

Original: [Equation]
Revised: [Equation]

Original: The bound of the lemma on (cid:107) J (cid:107) 2 follows from Eq.For (cid:107) J (cid:107)
Revised: The bound of the lemma on (cid:107) J (cid:107) 2 follows from Eq.

Original: [Equation] β m [Equation] 1 we have a contraction mapping on each compact set.
Revised: For (cid:107) J (cid:107) 2 (cid:54) β m 2 max < 1 we have a contraction mapping on each compact set.

Original: Banach ﬁxed point theorem says there is a unique ﬁxed point in the compact set.
Revised: Banach ﬁxed point theorem says there is a unique ﬁxed point in the compact set.

Original: Now let us further investigate the tightness of the bound on (cid:107)
Revised: Now let us further investigate the tightness of the bound on (cid:107)

Original: Var p [ x ] (cid:107) 2 via (cid:107) x
Revised: Var p [ x ] (cid:107) 2 via (cid:107) x

Original: i − ¯ x (cid:107) 2 : we consider the trace, which is the sum (cid:80) dk
Revised: i − ¯ x (cid:107) 2 : we consider the trace, which is the sum (cid:80) dk

Original: =1 e k of the w.l.o.g. ordered nonnegative eigenvalues
Revised: =1 e k of the w.l.o.g. ordered nonnegative eigenvalues

Original: e k of Var p
Revised: e k of Var p

Original: [ x ]
Revised: [ x ]

Original: The spectral norm is equal to the largest eigenvalue e 1 , which is equal to the largest singular value, as we have positive semideﬁnite matrices.
Revised: The spectral norm is equal to the largest eigenvalue e 1 , which is equal to the largest singular value, as we have positive semideﬁnite matrices.

Original: We obtain:
Revised: We obtain:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the tightness of the bound depends on eigenvalues which are not the largest.
Revised: Therefore, the tightness of the bound depends on eigenvalues which are not the largest.

Original: Hence variations which are not along the largest variation weaken the bound.
Revised: Hence variations which are not along the largest variation weaken the bound.

Original: Next we investigate the location of ﬁxed points which existence is ensured by the global convergence stated in Theorem A2.
Revised: Next we investigate the location of ﬁxed points which existence is ensured by the global convergence stated in Theorem A2.

Original: For N patterns X
Revised: For N patterns X

Original: =
Revised: =

Original: ( x 1 , . .
Revised: ( x 1 , . .

Original: , x N ) , we consider the iteration
Revised: , x N ) , we consider the iteration

Original: [Equation]
Revised: [Equation]

Original: (91) usingp = softmax( β X T ξ ) .
Revised: using

Original: (92) ξ new is in the simplex of the patterns, that is, ξ new = (cid:80) i p i
Revised: [Equation] ξ new is in the simplex of the patterns, that is, ξ new = (cid:80) i p i

Original: x i with (cid:80) i p i
Revised: x i with (cid:80) i p i

Original: = 1 and 0 (cid:54) p i .
Revised: = 1 and 0 (cid:54) p i .

Original: Hence, after one update ξ is in the simplex of the pattern and stays there.
Revised: Hence, after one update ξ is in the simplex of the pattern and stays there.

Original: If the center m x is the zero vector m x = 0 , that is, the data is centered, then the mean is a ﬁxed point of the iteration.
Revised: If the center m x is the zero vector m x = 0 , that is, the data is centered, then the mean is a ﬁxed point of the iteration.

Original: For ξ = m x =we have
Revised: For ξ = m x =we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: In particular normalization methods like batch normalization would promote the mean as a ﬁxed point.
Revised: In particular normalization methods like batch normalization would promote the mean as a ﬁxed point.

Original: We consider the differences of dot products for x i : x Ti x i − x
Revised: We consider the differences of dot products for x i : x Ti x i − x

Original: Ti x j = x Ti ( x i − x j ) , for ﬁxed point m ∗ x : ( m ∗ x ) T x i − ( m ∗ x ) T x j = ( m ∗ x ) T ( x i − x j ) [Equation] and for the center m x : m T x x i − m T x x j = m T x ( x i − x j ) .
Revised: Ti x j = x Ti ( x i − x j ) , for ﬁxed point m ∗ x : ( m ∗ x ) T x i − ( m ∗ x ) T x j = ( m ∗ x ) T ( x i − x j ) , and for the center m x : m T x x i − m T x x j = m T x ( x i − x j ) .

Original: Using the Cauchy-Schwarz inequality
Revised: Using the Cauchy-Schwarz inequality, we get

Original: [Equation]
Revised: [Equation]

Original: we get

Original: [Equation]
Revised: [Equation]

Original: This inequality gives:
Revised: This inequality gives:

Original: [Equation]
Revised: [Equation]

Original: where we used (cid:107) [Equation]
Revised: where we used (cid:107) ξ − 0 (cid:107) (cid:54) (cid:107) ξ − m x (cid:107) + (cid:107) m x − 0 (cid:107) , (cid:107) ξ − m x (cid:107) = (cid:107) (cid:80) i p i x i − m x (cid:107) (cid:54) (cid:80)

Original:  p i (cid:107) x
Revised: i p i (cid:107) x

Original: i − m x (cid:107) (cid:54) m max , and M = max i (cid:107) x
Revised: i − m x (cid:107) (cid:54) m max , and M = max i (cid:107) x

Original: i (cid:107) .
Revised: i (cid:107) .

Original: In particular 
Revised: In particular β

Original: [Equation]
Revised: [Equation]

Revised: (98) β

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: Let i = arg max
Revised: Let i = arg max

Original: j ξ T x j , therefore the maximal softmax component is i .
Revised: j ξ T x j , therefore the maximal softmax component is i .

Original: For the maximal softmax component i we have:
Revised: For the maximal softmax component i we have:

Original: [Equation]
Revised: [Equation]

Original: Analogously we obtain for i = arg max j m T x x j , a bound on the maximal softmax component i if the center is put into the iteration:
Revised: Analogously we obtain for i = arg max j m T x x j , a bound on the maximal softmax component i if the center is put into the iteration:

Original: [Equation]
Revised: [Equation]

Original: Analog we obtain a bound for i
Revised: Analog we obtain a bound for i

Original: = arg max j ( m ∗ x ) T x j on the maximal softmax component i of the ﬁxed point:
Revised: = arg max j ( m ∗ x ) T x j on the maximal softmax component i of the ﬁxed point:

Original: [Equation]
Revised: [Equation]

Original: The two important terms are m max , the variance or spread of the data and (cid:107) m x (cid:107) , which tells how well the data is centered.
Revised: The two important terms are m max , the variance or spread of the data and (cid:107) m x (cid:107) , which tells how well the data is centered.

Original: For a contraction mapping we already required βm 2max < 1 , therefore the ﬁrst term in the exponent is 2 βm 2max < 2 .
Revised: For a contraction mapping we already required βm 2max < 1 , therefore the ﬁrst term in the exponent is 2 βm 2max < 2 .

Original: The second term 2 βm max (cid:107) m x (cid:107) is small if the data is centered.
Revised: The second term 2 βm max (cid:107) m x (cid:107) is small if the data is centered.

Original: • Global ﬁxed point near the global mean: Analysis using softmax values.
Revised: • Global ﬁxed point near the global mean: Analysis using softmax values.

Original: If ξ T x i ≈ ξ T x j for all i and j , then p i ≈ 1 /N and we have m = max i p i (1 − p i ) < 1 /N .
Revised: If ξ T x i ≈ ξ T x j for all i and j , then p i ≈ 1 /N and we have m = max i p i (1 − p i ) < 1 /N .

Original: For
Revised: For

Original: M (cid:54) 1 / √ 2 β we obtain from Lemma A2:
Revised: M (cid:54) 1 / √ 2 β we obtain from Lemma A2:

Original: [Equation]
Revised: [Equation]

Original: The local ﬁxed point is m ∗ x
Revised: The local ﬁxed point is m ∗ x

Original: ≈ m x = [Equation] 1 /N 
Revised: ≈ m x = (1 /N )

Original: [Equation]
Revised: (cid:80) Ni

Original: =1 x i with p i ≈ 1 /N 
Revised: =1 x i with p i ≈ 1 /N .

Original: [Equation]

Original: We now treat this case more formally.
Revised: We now treat this case more formally.

Original: First we discuss conditions that ensure that the iteration is a contraction mapping.
Revised: First we discuss conditions that ensure that the iteration is a contraction mapping.

Original: We consider the iteration Eq.
Revised: We consider the iteration Eq.

Original: (56) in the variable p :
Revised: (57) in the variable p :

Original: [Equation]
Revised: [Equation]

Original: The Jacobian is
Revised: The Jacobian is

Original: [Equation]
Revised: [Equation]

Original: with
Revised: with

Original: [Equation]
Revised: [Equation]

Original: The version of the mean value theorem in Lemma A32 states for J m = (cid:82) {+
+}
Revised: The version of the mean value theorem in Lemma A32 states for J m = (cid:82) {+
+}

Original: [Equation]
Revised: J( λ p ) d λ = X

Original:  ms with the symmetric matrix J ms = (cid:82) 1 

Revised: T X J ms with the symmetric matrix J ms = (cid:82) 1

Original: J s ( λ p ) d λ : pnew = [Equation] T
Revised: J s ( λ p ) d λ : p

Original: [Equation]
Revised: [Equation]

Original: With m = max i p i (1 − p i ) , Eq.
Revised: With m = max i p i (1 − p i ) , Eq.

Original: (475) from Lemma A24 is
Revised: (476) from Lemma A24 is

Original: [Equation]
Revised: [Equation]

Original: First observe that λp i (1 − λp i ) (cid:54) p i (
Revised: First observe that λp i (1 − λp i ) (cid:54) p i (

Original: 1 − p i ) for p
Revised: 1 − p i ) for p

Original: i (cid:54) 0 .
Revised: i (cid:54) 0 .

Original: 5 and λ ∈ [0 , 1] , since p i (1 − p i ) −
Revised: 5 and λ ∈ [0 , 1] , since p i (1 − p i ) −

Original: λp i (1 − λp i ) =
Revised: λp i (1 − λp i ) =

Original: (1 − λ )
Revised: (1 − λ )

Original: p i (1 − (1 + λ ) p i ) ≥ 0 .
Revised: p i (1 − (1 + λ ) p i ) ≥ 0 .

Original: For max i p i (cid:54) 0 .
Revised: For max i p i (cid:54) 0 .

Original: 5 this observation leads to the following bound for J ms :
Revised: 5 this observation leads to the following bound for J ms :

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (478) in Lemma A24 states that every J s is bounded by 1 / 2 β , therefore also the mean:
Revised: (479) in Lemma A24 states that every J s is bounded by 1 / 2 β , therefore also the mean:

Original: [Equation]
Revised: [Equation]

Original: Since m = max i p i (
Revised: Since m = max i p i (

Original: 1 − p i )
Revised: 1 − p i )

Original: < max i p i
Revised: < max i p i

Original: = p max , the previous bounds can be combined as follows:
Revised: = p max , the previous bounds can be combined as follows:

Original: [Equation]
Revised: [Equation]

Revised: 2 min { 0 . 25 , p max } β .

Original: Consequently,
Revised: Consequently,

Original: [Equation]
Revised: [Equation]

Original: 112) where we used Eq.
Revised: where we used Eq.

Original: (cid:13)(cid:13)
Revised: (cid:13)(cid:13)

Original: X T X (cid:13)(cid:13) 

Revised: X T X (cid:13)(cid:13)

Original: = (cid:13)(cid:13)
Revised: = (cid:13)(cid:13)

Original: XX T (cid:13)(cid:13) 2 , therefore (cid:13)(cid:13)
Revised: XX T (cid:13)(cid:13) 2 , therefore (cid:13)(cid:13)

Original: X T X (cid:13)(cid:13) 2 is N times the maximal second moment of the data squared.
Revised: X T X (cid:13)(cid:13) 2 is N times the maximal second moment of the data squared.

Original: Obviously, g ( p ) is a contraction mapping in compact sets, where
Revised: Obviously, g ( p ) is a contraction mapping in compact sets, where

Original: [Equation]
Revised: [Equation]

Original: S is the sphere around the origin 0 with radius one.
Revised: S is the sphere around the origin 0 with radius one.

Original: For
Revised: For

Original: [Equation]
Revised: [Equation]

Original: new [Equation] 114) we have (cid:107) p (cid:107) (cid:54) (cid:107) p (cid:107) 1 = 1 and (cid:107) p new (cid:107) (cid:54) (cid:107) p new (cid:107) 1 = 1 .
Revised: we have (cid:107) p (cid:107) (cid:54) (cid:107) p (cid:107) 1 = 1 and (cid:107) p new (cid:107) (cid:54) (cid:107) p new (cid:107) 1 = 1 .

Original: Therefore, g maps points from S into S .
Revised: Therefore, g maps points from S into S .

Original: g is a contraction mapping for
Revised: g is a contraction mapping for

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: According to Banach ﬁxed point theorem g has a ﬁxed point in the sphere S .
Revised: According to Banach ﬁxed point theorem g has a ﬁxed point in the sphere S .

Original: Hölder’s inequality gives:
Revised: Hölder’s inequality gives:

Original: [Equation]
Revised: [Equation]

Original: Alternatively:
Revised: Alternatively:

Original: [Equation]
Revised: [Equation]

Original: Let now S be the sphere around the origin 0 with radius 1 / √ N + √ p max and let (cid:107)
Revised: Let now S be the sphere around the origin 0 with radius 1 / √ N + √ p max and let (cid:107)

Original: J m ( p ) (cid:107) 

Revised: J m ( p ) (cid:107)

Original: (cid:54) c <for p ∈ S .
Revised: (cid:54) c <for p ∈ S .

Original: The old p is in the sphere S ( p ∈ S ) since p max < √ p max for p max < 1 .
Revised: The old p is in the sphere S ( p ∈ S ) since p max < √ p max for p max < 1 .

Original: We have
Revised: We have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, g is a mapping from S into S and a contraction mapping.
Revised: Therefore, g is a mapping from S into S and a contraction mapping.

Original: According to Banach ﬁxed point theorem, a ﬁxed point exists in S .
Revised: According to Banach ﬁxed point theorem, a ﬁxed point exists in S .

Original: For the 1-norm, we use Lemma A24 and (cid:107) p (cid:107) 1 = 1 to obtain from Eq.
Revised: For the 1-norm, we use Lemma A24 and (cid:107) p (cid:107) 1 = 1 to obtain from Eq.

Original: [Equation]
Revised: [Equation]

Original:  where m = max i p i (1 − p i ) , M 1 = (cid:107) X (cid:107) 1 = max i (cid:107) 
Revised: [Equation] 122) where m = max i p i (1 − p i ) , M 1 = (cid:107) X (cid:107) 1 = max i (cid:107) x

Original: [Equation]

Original: i (cid:107) 1 , M = max i (cid:107) x i (cid:107) , (cid:107) X (cid:107) ∞ =
Revised: i (cid:107) 1 , M = max i (cid:107) x i (cid:107) , (cid:107) X (cid:107) ∞ =

Original: (cid:13)(cid:13) X T (cid:13)(cid:13) 1 = max i (cid:13)(cid:13)
Revised: (cid:13) (cid:13) X T (cid:13) (cid:13) 1 = max i (cid:13)(cid:13)

Original: [ X T ] i (cid:13)(cid:13) 1 (maximal absolute row sum norm), and M ∞ =
Revised: [ X T ] i (cid:13)(cid:13) 1 (maximal absolute row sum norm), and M ∞ =

Original: max i (cid:107) x i (cid:107) ∞ .
Revised: max i (cid:107) x i (cid:107) ∞ .

Original: Let us quickly mention some auxiliary estimates related to X T X :
Revised: Let us quickly mention some auxiliary estimates related to X T X :

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequaltiy is from Hölder’s inequality.
Revised: where the ﬁrst inequaltiy is from Hölder’s inequality.

Original: We used
Revised: We used

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is from Hölder’s inequality (here the same as the Cauchy-Schwarz inequality).
Revised: where the ﬁrst inequality is from Hölder’s inequality (here the same as the Cauchy-Schwarz inequality).

Original: See proof of Lemma A24 for the 1-norm bound on J s .
Revised: See proof of Lemma A24 for the 1-norm bound on J s .

Original: Everything else follows from the fact that the 1-norm is sub-multiplicative as induced matrix norm.
Revised: Everything else follows from the fact that the 1-norm is sub-multiplicative as induced matrix norm.

Original: We consider the minimal (cid:107) p (cid:107) .
Revised: We consider the minimal (cid:107) p (cid:107) .

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The solution to this minimization problem is p = (1 /N ) 1 .
Revised: The solution to this minimization problem is p = (1 /N ) 1 .

Original: Therefore, we have 1 / √ N (cid:54) (cid:107) p (cid:107) and 1 /N (cid:54) (cid:107) p (cid:107) 2 Using Eq.
Revised: Therefore, we have 1 / √ N (cid:54) (cid:107) p (cid:107) and 1 /N (cid:54) (cid:107) p (cid:107) 2 Using Eq.

Original: (118) we obtain
Revised: (119) we obtain

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: since p new ∈ S and p ∈ S .
Revised: since p new ∈ S and p ∈ S .

Original: For the ﬁxed point, we have
Revised: For the ﬁxed point, we have

Original: [Equation]
Revised: [Equation]

Original:  and hence 
Revised: 128) and hence [Equation] 1 − (cid:107) [Equation]

Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Therefore, for small (cid:107) J m (cid:107) 2 we have p ∗ ≈ (1 /N ) 1 .
Revised: Therefore, for small (cid:107) J m (cid:107) 2 we have p ∗ ≈ (1 /N ) 1 .

Original: A.1.5.3 Many Stable States: Fixed Points
Revised: A.1.5.3 Many Stable States: Fixed Points

Original: Near Stored Patterns.
Revised: Near Stored Patterns.

Original: We move on to the next case, where the patterns x i are well separated.
Revised: We move on to the next case, where the patterns x i are well separated.

Original: In this case the iteration goes to the pattern to which the initial ξ is most similar.
Revised: In this case the iteration goes to the pattern to which the initial ξ is most similar.

Original: If the initial ξ is similar to a vector x i then it will converge to x i and p will be e i .
Revised: If the initial ξ is similar to a vector x i then it will converge to x i and p will be e i .

Original: The main ingredients are again Banach’s Theorem and estimates on the Jacobian norm.
Revised: The main ingredients are again Banach’s Theorem and estimates on the Jacobian norm.

Original: • Proof of a ﬁxed point by Banach Fixed Point Theorem.
Revised: • Proof of a ﬁxed point by Banach Fixed Point Theorem.

Original: → Mapped Vectors Stay in a Compact Environment.
Revised: → Mapped Vectors Stay in a Compact Environment.

Original: We show that if x i is sufﬁcient dissimilar to other x j then there is an compact environment of x i (a sphere) where the ﬁxed point iteration maps this environment into itself.
Revised: We show that if x i is sufﬁcient dissimilar to other x j then there is an compact environment of x i (a sphere) where the ﬁxed point iteration maps this environment into itself.

Original: The idea of the proof is to deﬁne a sphere around x i for which points from the sphere are mapped by f into the sphere.
Revised: The idea of the proof is to deﬁne a sphere around x i for which points from the sphere are mapped by f into the sphere.

Original: We ﬁrst need following lemma which bounds the distance (cid:107) x
Revised: We ﬁrst need following lemma which bounds the distance (cid:107) x

Original: i − f ( ξ ) (cid:107) , where x i is the pattern that is least separated from ξ but separated from other patterns.
Revised: i − f ( ξ ) (cid:107) , where x i is the pattern that is least separated from ξ but separated from other patterns.

Original: Lemma A4.
Revised: Lemma A4.

Original: For a query ξ and data X = ( x 1 , . . .
Revised: For a query ξ and data X = ( x 1 , . . .

Original: , x N ) , there exists a x
Revised: , x N ) , there exists a x

Original: i that is least separated from ξ while being separated from other x j with j (cid:54) = i : 
Revised: i that is least separated from ξ while being separated from other x j with j (cid:54) = i : [Equation]

Original: [Equation]
Revised: [Equation]

Original: For x i , the following holds:
Revised: For x i , the following holds:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For the softmax component i we have:
Revised: For the softmax component i we have:

Original: [Equation]
Revised: [Equation]

Original: For softmax components k (cid:54) =
Revised: For softmax components k (cid:54) =

Original: i we have
Revised: i we have

Original: [Equation]
Revised: [Equation]

Original: The iteration f can be written as
Revised: The iteration f can be written as

Original: [Equation]
Revised: [Equation]

Original: We now can bound (cid:107) 
Revised: We now can bound (cid:107) x

Revised: i −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: We deﬁne ∆ i , i.e. the separation of pattern x i from data X = ( x 1 , . . .
Revised: We deﬁne ∆ i , i.e. the separation of pattern x i from data X = ( x 1 , . . .

Original: , x N ) as:
Revised: , x N ) as:

Original: [Equation]
Revised: [Equation]

Original: The pattern is separated from the other data if 0 < ∆ i .
Revised: The pattern is separated from the other data if 0 < ∆ i .

Original: Using the parallelogram identity, ∆ i can also be expressed as
Revised: Using the parallelogram identity, ∆ i can also be expressed as

Original: [Equation]
Revised: [Equation]

Original: For (cid:107) x i (cid:107) =
Revised: For (cid:107) x i (cid:107) =

Original: (cid:107) x j (cid:107) we have ∆ i = 1 / 2 min j,j (cid:54) =
Revised: (cid:107) x j (cid:107) we have ∆ i = 1 / 2 min j,j (cid:54) =

Original: i (cid:107)x
Revised: i (cid:107)

Original: i − x j (cid:107) 2 .
Revised: [Equation]

Original: Next we deﬁne the sphere where we want to apply Banach ﬁxed point theorem.
Revised: Next we deﬁne the sphere where we want to apply Banach ﬁxed point theorem.

Original: Deﬁnition 3 (Sphere S i ) .
Revised: Deﬁnition 3 (Sphere S i ) .

Original: The sphere S i is deﬁned as
Revised: The sphere S i is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Lemma A5.
Revised: Lemma A5.

Original: With ξ given, if the assumptions
Revised: With ξ given, if the assumptions

Original: A1:ξ is inside sphere: ξ ∈ Si , A2:data point x i is well separatedfrom the otherdata:
Revised: A1: is inside sphere S A2: separated other

Original: [Equation]
Revised: [Equation]

Original: hold, then f ( ξ ) is inside the sphere: f ( ξ ) ∈ S i .
Revised: hold, then f ( ξ ) is inside the sphere: f ( ξ ) ∈ S i .

Original: Therefore, with assumption (A2), f is a mapping from S i into S i .
Revised: Therefore, with assumption (A2), f is a mapping from S i into S i .

Original: Proof.
Revised: Proof.

Original: We need the separation ˜∆ i of ξ from the data.
Revised: We need the separation ˜∆ i of ξ from the data.

Original: [Equation]
Revised: [Equation]

Original: ξ T x i − ξ T x j (cid:1) . Using the Cauchy-Schwarz inequality, we obtain for 1 (cid:54) j (cid:54) N :
Revised: Using the Cauchy-Schwarz inequality [Equation] we obtain for 1 (cid:54) j (cid:54) N :

Original: [Equation]
Revised: [Equation]

Original: We have the lower bound
Revised: We have the lower bound

Original: [Equation]
Revised: [Equation]

Original: where we used the assumption (A1) of the lemma.
Revised: where we used the assumption (A1) of the lemma.

Original: From the proof in Lemma A4 we have
Revised: From the proof in Lemma A4 we have

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: Lemma A4 states that
Revised: Lemma A4 states that

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where we used assumption (A2) of the lemma.
Revised: where we used assumption (A2) of the lemma.

Original: Therefore, f ( ξ ) is a mapping from the sphere S i into the sphere S i :
Revised: Therefore, f ( ξ ) is a mapping from the sphere S i into the sphere S i :

Original: If ξ ∈ S i then f ( ξ ) ∈ S i .
Revised: If ξ ∈ S i then f ( ξ ) ∈ S i .

Original: • Contraction mapping
Revised: • Contraction mapping.

Original: For applying Banach ﬁxed point theorem we need to show that f is contraction in the compact environment S i .
Revised: For applying Banach ﬁxed point theorem we need to show that f is contraction in the compact environment S i .

Original: Lemma A6.
Revised: Lemma A6.

Original: Assume that
Revised: Assume that

Original: [Equation]
Revised: [Equation]

Original: then f is a contraction mapping in S i .
Revised: then f is a contraction mapping in S i .

Original: Proof.
Revised: Proof.

Original: The version of the mean value theorem Lemma A32 states for J m =(cid:82) 1 0 J( λ ξ +(1 − λ ) x i [Equation] λ
Revised: The version of the mean value theorem Lemma A32 states for J m =

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: We deﬁne ˜ ξ = λ ξ + (1 − λ )
Revised: We deﬁne ˜ ξ = λ ξ + (1 − λ )

Original: x i for some λ ∈ [0 , 1] .
Revised: x i for some λ ∈ [0 , 1] .

Original: From the proof in Lemma A4 we havep
Revised: From the proof in Lemma A4 we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: First we compute an upper bound on ˜ (cid:15) .
Revised: First we compute an upper bound on ˜ (cid:15) .

Original: We need the separation ˜∆ i of ξ from the data.
Revised: We need the separation ˜∆ i of ξ from the data.

Original: Using the
Revised: Using the

Original: Cauchy-Schwarz inequality, we obtain for 1 (cid:54) j (cid:54) N :
Revised: Cauchy-Schwarz inequality, we obtain for 1 (cid:54) j (cid:54) N :

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We have the lower bound on ˜∆ i : 
Revised: We have the lower bound on ˜∆ i : ˜∆

Revised: [Equation]

Revised: =

Revised: i

Original: [Equation]
Revised: [Equation]

Revised: x

Original: where we used (cid:13)(cid:13)(cid:13) 
Revised: [Equation] (cid:13)(cid:13)(cid:13) [Equation] (cid:17)

Revised: [Equation]

Revised: (cid:16)

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original:  = 
Revised: i (cid:13)(cid:13)(cid:13) M (cid:17)(cid:17) (156) = [Equation] (cid:13)(cid:13)(cid:13) [Equation]

Revised: i (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Revised: min

Revised: [Equation]

Revised: (cid:54) =

Revised: [Equation]

Revised: (cid:0)

Revised: [Equation]

Revised: (cid:1)

Original:  x i 
Revised: [Equation] (cid:13)(cid:13)(cid:13) [Equation] x i (cid:13)(cid:13)(cid:13) M ≥ ∆ i − 2 (cid:107)

Original: [Equation]
Revised: [Equation]

Original:  − x i (cid:107) .
Revised: (cid:107) M , where we used (cid:13)(cid:13)(cid:13) ˜ ξ − x i (cid:13)(cid:13)(cid:13) = λ (cid:107) ξ − x i (cid:107) (cid:54) (cid:107) ξ − x i (cid:107) .

Original: From the deﬁnition of ˜ (cid:15) in Eq.
Revised: From the deﬁnition of ˜ (cid:15) in Eq.

Original: (151) we have 
Revised: (152) we have ˜

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: [Equation]

Original:  where we used ξ ∈ S i , therefore (cid:107)
Revised: [Equation] [Equation] [Equation] , where we used ξ ∈ S i , therefore (cid:107)

Original: ξ − x i (cid:107) (cid:54) 1 β N M .
Revised: ξ − x i (cid:107) (cid:54) 1 β N M .

Original: Next we compute an lower bound on ˜ (cid:15) .
Revised: Next we compute an lower bound on ˜ (cid:15) .

Original: We start with an upper on ˜∆ i : 
Revised: We start with an upper on ˜∆ i : ˜ ∆ i (cid:54) min j,j (cid:54) = i (cid:16)(cid:16) x [Equation] + [Equation]

Revised: (cid:13)

Original: [Equation]
Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: (cid:13)(cid:13) (cid:13)

Revised: M

Original: where we used (cid:13)(cid:13)(cid:13) ˜ ξ −  x i (cid:13)(cid:13)(cid:13) =
Revised: [Equation] − (cid:16) x [Equation] j − (cid:13)(cid:13) (cid:13) ˜

Revised: [Equation]

Revised: x i (cid:13)(cid:13) (cid:13) M (cid:17)(cid:17) (158) =

Revised: [Equation]

Revised: M (cid:54) ∆ i + 2 (cid:107) ξ − x i (cid:107) M , where we used (cid:13)(cid:13)(cid:13) ˜

Revised: ξ − x i (cid:13)(cid:13)(cid:13) =

Original: λ (cid:107)
Revised: λ (cid:107)

Original: ξ − x i (cid:107) (cid:54) (cid:107) ξ − x i (cid:107) .
Revised: ξ − x i (cid:107) (cid:54) (cid:107) ξ − x i (cid:107) .

Original: From the deﬁnition of ˜ (cid:15) in Eq.
Revised: From the deﬁnition of ˜ (cid:15) in Eq.

Original: (151) we have [Equation]
Revised: (152) we have ˜ (cid:15) =

Original:  where we used ξ ∈ S i , therefore (cid:107)
Revised: ( N − 1) exp( − β ˜∆ i ) (159) ≥ ( N − 1) exp ( − β (∆ i + 2 (cid:107) ξ − x i (cid:107) M )) ≥ ( N − 1) exp (cid:18) − β (cid:18) ∆ i + β N (cid:19)(cid:19) , where we used ξ ∈ S i , therefore (cid:107)

Original: ξ − x i (cid:107) (cid:54) 1 β N M .
Revised: ξ − x i (cid:107) (cid:54) 1 β N M .

Original: Now we bound the Jacobian.
Revised: Now we bound the Jacobian.

Original: We can assume ˜ (cid:15) (cid:54) 0 .
Revised: We can assume ˜ (cid:15) (cid:54) 0 .

Original: 5 otherwise (1 − ˜ (cid:15) )
Revised: 5 otherwise (1 − ˜ (cid:15) )

Original: (cid:54) 0 .
Revised: (cid:54) 0 .

Original: 5 in the following.
Revised: 5 in the following.

Original: From the proof of Lemma A24 we know for p max ( ˜ ξ ) ≥ 1 − ˜ (cid:15) , then p i ( ˜ ξ ) (cid:54) ˜ (cid:15) for p
Revised: From the proof of Lemma A24 we know for p max ( ˜ ξ ) ≥ 1 − ˜ (cid:15) , then p i ( ˜ ξ ) (cid:54) ˜ (cid:15) for p

Original: i ( ˜ ξ ) (cid:54) =
Revised: i ( ˜ ξ ) (cid:54) =

Original: p max ( ˜ ξ ) .
Revised: p max ( ˜ ξ ) .

Original: Therefore, p i (˜ ξ )(1 − p
Revised: Therefore, p i (˜ ξ )(1 − p

Original: i
Revised: i

Original: ˜ ξ )) (cid:54) m
Revised: ˜ ξ )) (cid:54) m

Original: (cid:54) ˜ (cid:15) (1 − ˜ (cid:15) ) for all i .
Revised: (cid:54) ˜ (cid:15) (1 − ˜ (cid:15) ) for all i .

Original: Next we use the derived upper and lower bound on ˜ (cid:15) in previous Eq.
Revised: Next we use the derived upper and lower bound on ˜ (cid:15) in previous Eq.

Original: (60) in Lemma A2:
Revised: (61) in Lemma A2:

Original: [Equation]
Revised: [Equation]

Original: The bound Eq.
Revised: The bound Eq.

Original: (159) holds for the mean J m , too, since it averages over J( ˜ ξ ) :
Revised: (160) holds for the mean J m , too, since it averages over J(˜ ξ ) :

Original: [Equation]
Revised: [Equation]

Original: The assumption of the lemma is
Revised: The assumption of the lemma is

Original: [Equation]
Revised: [Equation]

Original: This is
Revised: This is

Original: [Equation]
Revised: [Equation]

Original: Therefore, the spectral norm (cid:107) J (cid:107) 2 can be bounded by:
Revised: Therefore, the spectral norm (cid:107) J (cid:107) 2 can be bounded by:

Original: [Equation]
Revised: [Equation]

Original: Therefore, f is a contraction mapping in S i .
Revised: Therefore, f is a contraction mapping in S i .

Original: • Banach Fixed Point Theorem.
Revised: • Banach Fixed Point Theorem.

Original: Now we have all ingredients to apply Banach ﬁxed point theorem.
Revised: Now we have all ingredients to apply Banach ﬁxed point theorem.

Original: Lemma A7.
Revised: Lemma A7.

Original: Assume that
Revised: Assume that

Original: [Equation]
Revised: [Equation]

Original: then f has a ﬁxed point in S i .
Revised: then f has a ﬁxed point in S i .

Original: Proof.
Revised: Proof.

Original: We use Banach ﬁxed point theorem: Lemma A5 says that f maps from S i into S i .
Revised: We use Banach ﬁxed point theorem: Lemma A5 says that f maps from S i into S i .

Original: Lemma A says that f is a contraction mapping in S i .
Revised: Lemma A says that f is a contraction mapping in S i .

Original: • Contraction mapping with a ﬁxed point.
Revised: • Contraction mapping with a ﬁxed point.

Original: We have shown that a ﬁxed point exists.
Revised: We have shown that a ﬁxed point exists.

Original: We want to know how fast the iteration converges to the ﬁxed point.
Revised: We want to know how fast the iteration converges to the ﬁxed point.

Original: Let x
Revised: Let x

Original: ∗
Revised: ∗

Original: i be the ﬁxed point of the iteration f in the sphere S i .
Revised: i be the ﬁxed point of the iteration f in the sphere S i .

Original: Using the mean value theorem
Revised: Using the mean value theorem

Original: Lemma A32, we have with J m = (cid:82) 1 0 J( λ ξ + (1 − λ )
Revised: Lemma A32, we have with J m = (cid:82) [Equation] J [Equation]

Original: x ∗ i )
Revised: + [Equation] [Equation]

Original: d λ:
Revised: λ

Original: [Equation]
Revised: [Equation]

Revised: ∗ i

Original: [Equation]
Revised: [Equation]

Original: According to Lemma A24, if p max = max i p i ≥ 1 − (cid:15) for all ˜ x = λ ξ +(1 − λ ) x ∗ i , then the spectral norm of the Jacobian is bounded by
Revised: According to Lemma A24, if p max = max i p i [Equation] 1 − (cid:15) for all ˜ x = λ ξ +

Original: [Equation]
Revised: [Equation]

Revised: 1 − λ )

Original: The norm of  Jacobianat ˜ x is bounded 
Revised: x ∗ i , then the spectral norm of the Jacobian is bounded by

Original: [Equation]
Revised: [Equation]

Revised: The norm of Jacobian at ˜ x is bounded

Original: [Equation]
Revised: [Equation]

Original: We used that the spectral norm (cid:107) .
Revised: We used that the spectral norm (cid:107) .

Original: (cid:107) 2 is bounded by the Frobenius norm (cid:107) .
Revised: (cid:107) 2 is bounded by the Frobenius norm (cid:107) .

Original: (cid:107) F which can be expressed by the norm squared of its column vectors:
Revised: (cid:107) F which can be expressed by the norm squared of its column vectors:

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: The norm of Jacobian of the ﬁxed point iteration is bounded
Revised: The norm of Jacobian of the ﬁxed point iteration is bounded

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The separation of pattern x i from data X = ( x 1 , . . .
Revised: The separation of pattern x i from data X = ( x 1 , . . .

Original: , x N ) is
Revised: , x N ) is

Original: [Equation]
Revised: [Equation]

Original: We need the separation ˜∆ i of ˜ x = λ ξ + (1 − λ )
Revised: We need the separation ˜∆ i of ˜ x = λ ξ + (1 − λ )

Original: x ∗ i from the data:
Revised: x ∗ i from the data:

Original: [Equation]
Revised: [Equation]

Revised: (cid:0)

Revised: [Equation]

Revised: (cid:1) .

Original: We compute a lower bound on ˜∆ i .
Revised: We compute a lower bound on ˜∆ i .

Original: Using the Cauchy-Schwarz inequality, we obtain for 1 (cid:54) j (cid:54) N :[Equation] M .
Revised: Using the Cauchy-Schwarz inequality, we obtain for 1 (cid:54) j (cid:54) N :

Original: [Equation]
Revised: [Equation]

Original: We have the lower bound
Revised: We have the lower bound

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: For the softmax component i we have:
Revised: For the softmax component i we have:

Original: [Equation]
Revised: [Equation]

Original: ≥

Original: [Equation]

Original: = 1 − (cid:15) .
Revised: = 1 − (cid:15) .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: We can bound the spectral norm of the Jacobian [Equation] which upper bounds the Lipschitz constant:
Revised: We can bound the spectral norm of the Jacobian, which upper bounds the Lipschitz constant:

Original: [Equation]
Revised: [Equation]

Original: For a contraction mapping we require
Revised: For a contraction mapping we require

Original: [Equation]
Revised: [Equation]

Original: which can be ensured by
Revised: which can be ensured by

Original: [Equation]
Revised: [Equation]

Original: Solving this inequality for ∆ i gives
Revised: Solving this inequality for ∆ i gives

Original: [Equation]
Revised: [Equation]

Original: In an environment around x ∗ i in which Eq.
Revised: In an environment around x ∗ i in which Eq.

Original: (182) holds
Revised: (183) holds

Original: [Equation]
Revised: [Equation]

Original: f is a contraction mapping and every point converges under the iteration f to x ∗ i when the iteration stays in the environment.
Revised: f is a contraction mapping and every point converges under the iteration f to x ∗ i when the iteration stays in the environment.

Original: After every iteration the mapped point f ( ξ ) is closer to the ﬁxed point x ∗ i than the original point x i :
Revised: After every iteration the mapped point f ( ξ ) is closer to the ﬁxed point x ∗ i than the original point x i :

Original: [Equation]
Revised: [Equation]

Original: Using
Revised: Using

Original: [Equation]
Revised: [Equation]

Original:  we obtain
Revised: 185) we obtain

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For large ∆ i the iteration is close to the ﬁxed point even after one update.
Revised: For large ∆ i the iteration is close to the ﬁxed point even after one update.

Original: This has been conﬁrmed in several experiments.
Revised: This has been conﬁrmed in several experiments.

Original: A.1.5.4 Metastable States: Fixed Points Near Mean of Similar Patterns.
Revised: A. 4 Metastable States: Fixed Points Near Mean of Similar Patterns.

Original: The proof concept is the same as for a single pattern but now for the arithmetic mean of similar patterns.
Revised: The proof concept is the same as for a single pattern but now for the arithmetic mean of similar patterns.

Original: • Bound on the Jacobian
Revised: • Bound on the Jacobian.

Original: The Jacobian of the ﬁxed point iteration is
Revised: The Jacobian of the ﬁxed point iteration is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: If we consider p i as the probability of selecting the vector x i , then we can deﬁne expectations as
Revised: If we consider p i as the probability of selecting the vector x i , then we can deﬁne expectations as

Original: E p [ f ( x )]
Revised: E p [ f ( x )]

Original: = (cid:80) Ni =1 p i f ( x i ) .
Revised: = (cid:80) Ni =1 p i f ( x i ) .

Original: In this setting the matrix
Revised: In this setting the matrix

Original: [Equation]
Revised: [Equation]

Original: 187) is the covariance matrix of data X when its vectors are selected according to the probability p : 
Revised: (188) is the covariance matrix of data X when its vectors are selected according to the probability p : [Equation] (189) =

Original: [Equation]
Revised: [Equation]

Revised: p

Original: [Equation]
Revised: [Equation]

Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: therefore we have
Revised: therefore we have

Original: [Equation]
Revised: [Equation]

Original: We now elaborate more on this interpretation as variance.
Revised: We now elaborate more on this interpretation as variance.

Original: Speciﬁcally the singular values of J (or in other words: the covariance) should be reasonably small.
Revised: Speciﬁcally the singular values of J (or in other words: the covariance) should be reasonably small.

Original: The singular values are the key to ensure convergence of the iteration Eq.
Revised: The singular values are the key to ensure convergence of the iteration Eq.

Original: Next we present some thoughts.
Revised: Next we present some thoughts.

Original: It’s clear that the largest eigenvalue of the covariance matrix (equal to the largest singular value) is the variance in the direction of the eigenvector associated with the largest eigenvalue.
Revised: It’s clear that the largest eigenvalue of the covariance matrix (equal to the largest singular value) is the variance in the direction of the eigenvector associated with the largest eigenvalue.

Original: Furthermore the variance goes to zero as one p i goes to one, since only one pattern is chosen and there is no variance.
Revised: Furthermore the variance goes to zero as one p i goes to one, since only one pattern is chosen and there is no variance.

Original: The variance is reasonable small if all patterns are chosen with equal probability.
Revised: The variance is reasonable small if all patterns are chosen with equal probability.

Original: The variance is small if few similar patterns are chosen with high probability.
Revised: The variance is small if few similar patterns are chosen with high probability.

Original: If the patterns are sufﬁcient similar, then the spectral norm of the covariance matrix is smaller than one.
Revised: If the patterns are sufﬁcient similar, then the spectral norm of the covariance matrix is smaller than one.

Original: The ﬁrst three issues have already been adressed.
Revised: The ﬁrst three issues have already been adressed.

Original: Now we focus on the last one in greater detail.
Revised: Now we focus on the last one in greater detail.

Original: We assume that the ﬁrst l patterns are much more probable (and similar to one another) than the other patterns.
Revised: We assume that the ﬁrst l patterns are much more probable (and similar to one another) than the other patterns.

Original: Therefore, we deﬁne:
Revised: Therefore, we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: M is an upper bound on the Euclidean norm of the patterns, which are vectors.
Revised: M is an upper bound on the Euclidean norm of the patterns, which are vectors.

Original: (cid:15) is an upper bound on the probability γ of not choosing one of the ﬁrst l patterns, while 1 − (cid:15) is a lower bound the probability (1 − γ ) of choosing one of the ﬁrst l patterns.
Revised: (cid:15) is an upper bound on the probability γ of not choosing one of the ﬁrst l patterns, while 1 − (cid:15) is a lower bound the probability (1 − γ ) of choosing one of the ﬁrst l patterns.

Original: m
Revised: m

Original: x is the arithmetic mean (the center) of the ﬁrst l patterns.
Revised: x is the arithmetic mean (the center) of the ﬁrst l patterns.

Original: m max is the maximal distance of the patterns to the center m x .
Revised: m max is the maximal distance of the patterns to the center m x .

Original: ˜ p is the probability p normalized for the ﬁrst l patterns.
Revised: ˜ p is the probability p normalized for the ﬁrst l patterns.

Original: The variance of the ﬁrst l patterns is
Revised: The variance of the ﬁrst l patterns is

Original: [Equation]
Revised: [Equation]

Original: Lemma A8.
Revised: Lemma A8.

Original: With the deﬁnitions in Eq.
Revised: With the deﬁnitions in Eq.

Original: (192) to Eq.
Revised: (193) to Eq.

Original: (199) , the following bounds on the norm (cid:107) J (cid:107)of the Jacobian of the ﬁxed point iteration hold.
Revised: (200) , the following bounds on the norm (cid:107) J (cid:107)of the Jacobian of the ﬁxed point iteration hold.

Original: The γ -bound for (cid:107) J (cid:107) 2 is
Revised: The γ -bound for (cid:107) J (cid:107) 2 is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: (201) and the (cid:15) -bound for (cid:107) J (cid:107) 2 is:

Original: [Equation]
Revised: [Equation]

Original: and the (cid:15)-bound for (cid:107) J (cid:107) 2 is:
Revised: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The variance Var ˜ p [ x 1: l ] can be expressed as:
Revised: The variance Var ˜ p [ x 1: l ] can be expressed as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: ] =
Revised: (cid:33) [Equation] [Equation] γ l (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: γ l (cid:88) i
Revised: (203) =

Original: = p
Revised: [Equation]

Original: i x i (cid:33) (cid:32) x
Revised: (cid:32) [Equation]

Original: i − 1 − γ l (cid:88) i 
Revised: (cid:88) i [Equation]

Original: = p
Revised: p

Original: i x i (cid:33) T (202) =
Revised: [Equation] (cid:33) [Equation] (cid:32)

Revised: [Equation]

Original: l (cid:88) i
Revised: (cid:88) i

Original: = p
Revised: = p

Original: i x i x Ti −  (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) T − 1 − γ (cid:32) l (cid:88) i = p

Original: i x i (cid:33)1 − γ (cid:32) l (cid:88) i 
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p i x i (cid:33) T

Original: = p
Revised: [Equation]

Original: i x i (cid:33) T − 1 − γ (cid:32) l (cid:88) i = p
Revised: (cid:32) l (cid:88) i = [Equation]

Original: i x i (cid:33) (cid:32) l (cid:88) i = p 
Revised: i [Equation] (cid:33) (cid:32) l (cid:88) i = p i

Original: i x i (cid:33) T + (cid:80) li =1 p i (1 − γ )(cid:32) l (cid:88) i
Revised: [Equation] i (cid:33) T = l (cid:88) i

Original: = p
Revised: = p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T = l (cid:88) i = p

Original: i x i x Ti − 1 − γ (cid:32) l (cid:88) i
Revised: i x i x Ti − 1 − γ (cid:32) l (cid:88) i

Original: = p
Revised: = p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T = l (cid:88) i =p
Revised: i x i (cid:33) T = [Equation] i =

Original: i x i x Ti − (cid:32) l (cid:88) i
Revised: [Equation]

Original: = p
Revised: p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: [Equation] (cid:33) [Equation] i = p

Original: i x i (cid:33) T + (cid:18) 1 − 1 − γ (cid:19) (cid:32) l (cid:88) i
Revised: i [Equation] i (cid:33) T + (cid:18) 1 − 1 − γ (cid:19) (cid:32) l (cid:88) i

Original: = p
Revised: = p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T = l (cid:88) i = p
Revised: i x i (cid:33) T = l (cid:88) i = p

Original: i x i x Ti − (cid:32) l (cid:88) i
Revised: i x i x Ti − (cid:32) l (cid:88) i

Original: = p
Revised: = p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T − γ 1 − γ (cid:32) l (cid:88) i
Revised: i x i (cid:33) T − γ 1 − γ (cid:32) l (cid:88) i

Original: = p
Revised: = p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T .
Revised: i x i (cid:33) T .

Original: Therefore, we have l (cid:88) i
Revised: Therefore, we have l (cid:88) i

Original: = p
Revised: = p

Original: i x i x Ti − (cid:32) l (cid:88) i
Revised: i x i x Ti − (cid:32) l (cid:88) i

Original: = p
Revised: = p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T (203) =
Revised: i x i (cid:33) T (204) =

Original: (1 − γ )
Revised: (1 − γ )

Original: Var ˜ p
Revised: Var ˜ p

Original: [ x 1: l ] + γ 1 − γ (cid:32) l (cid:88)
Revised: [ x 1: l ] + γ 1 − γ (cid:32) l (cid:88)

Original: i = p
Revised: i = p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T .
Revised: i x i (cid:33) T .

Original: We now can reformulate the Jacobian J :
Revised: We now can reformulate the Jacobian J :

Original: [Equation]

Original: p
Revised: J

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i
Revised: (cid:32)

Original: = l

Original: [Equation]
Revised: [Equation]

Original:  p
Revised: = p

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: +

Revised: N (cid:88) i

Original: i =  p 
Revised: = l + p i x

Original: [Equation]
Revised: [Equation]

Revised: Ti (205) − (cid:32) l (cid:88) i = p

Original:  x [Equation] +
Revised: i x i +

Original: N (cid:88) i
Revised: N (cid:88) i

Original: = l + p
Revised: = l + p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i +
Revised: i x i +

Original: N (cid:88) i
Revised: N (cid:88) i

Original: = l + p
Revised: = l + p

Original: i x i (cid:33) T 
Revised: i x i (cid:33) T 

Original:  =
Revised:  =

Original: β   l (cid:88) i
Revised: β   l (cid:88) i

Original: = p
Revised: = p

Original: i x i x Ti − (cid:32) l (cid:88) i
Revised: i x i x Ti − (cid:32) l (cid:88) i

Original: = p
Revised: = p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T +
Revised: i x i (cid:33) T +

Original: N (cid:88) i
Revised: N (cid:88) i

Original: = l + p
Revised: = l + p

Original: i
Revised: i

Original: x
Revised: x

Original: i x Ti − (cid:32) N (cid:88) i
Revised: i x Ti − (cid:32) N (cid:88) i

Original: = l + p
Revised: = l + p

Original: i x i (cid:33) (cid:32) N (cid:88) i
Revised: i x i (cid:33) (cid:32) N (cid:88) i

Original: = l + p
Revised: = l + p

Original: i x i (cid:33) T − (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) T − (cid:32) l (cid:88) i = p

Original: i x i (cid:33) (cid:32) N (cid:88) i
Revised: i x i (cid:33) (cid:32) N (cid:88) i

Original: =
Revised: =

Original: l + p
Revised: l + p

Original: i x i (cid:33) T − (cid:32) N (cid:88)
Revised: i x i (cid:33) T − (cid:32) N (cid:88)

Original: i =
Revised: i =

Original: l + p
Revised: l + p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T 
Revised: i x i (cid:33) T 

Original:  =
Revised:  =

Original: β   (1 − γ )
Revised: β   (1 − γ )

Original: Var ˜ p
Revised: Var ˜ p

Original: [ x 1: l ] + γ 1 − γ (cid:32) l (cid:88)
Revised: [ x 1: l ] + γ 1 − γ (cid:32) l (cid:88)

Original: i = p
Revised: i = p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T +
Revised: i x i (cid:33) T +

Original: N (cid:88) i
Revised: N (cid:88) i

Original: = l + p
Revised: = l + p

Original: i
Revised: i

Original: x
Revised: x

Original: i x Ti − (cid:32) N (cid:88) i
Revised: i x Ti − (cid:32) N (cid:88) i

Original: = l + p
Revised: = l + p

Original: i x i (cid:33) (cid:32) N (cid:88) i
Revised: i x i (cid:33) (cid:32) N (cid:88) i

Original: = l + p
Revised: = l + p

Original: i x i (cid:33) T − (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) T − (cid:32) l (cid:88) i = p

Original: i x i (cid:33) (cid:32) N (cid:88) i
Revised: i x i (cid:33) (cid:32) N (cid:88) i

Original: =
Revised: =

Original: l + p
Revised: l + p

Original: i x i (cid:33) T − (cid:32) N (cid:88)
Revised: i x i (cid:33) T − (cid:32) N (cid:88)

Original: i =
Revised: i =

Original: l + p
Revised: l + p

Original: i x i (cid:33) (cid:32) l (cid:88) i = p
Revised: i x i (cid:33) (cid:32) l (cid:88) i = p

Original: i x i (cid:33) T 
Revised: i x i (cid:33) T 

Original:  .
Revised:  .

Original: The spectral norm of an outer product of two vectors is the product of the Euclidean norms of the vectors:
Revised: The spectral norm of an outer product of two vectors is the product of the Euclidean norms of the vectors:

Original: [Equation]
Revised: [Equation]

Original: since bb T has eigenvector b / (cid:107) b (cid:107) with eigenvalue (cid:107) b (cid:107) 2 and otherwise zero eigenvalues.
Revised: since bb T has eigenvector b / (cid:107) b (cid:107) with eigenvalue (cid:107) b (cid:107) 2 and otherwise zero eigenvalues.

Original: We now bound the norms of some matrices and vectors:
Revised: We now bound the norms of some matrices and vectors:

Original: [Equation]
Revised: [Equation]

Revised: In order to bound the variance of the ﬁrst l patterns

Original: [Equation]
Revised: [Equation]

Original: In order to bound the variance of the ﬁrst l patterns, we compute the vector a that minimizes
Revised: we compute the vector a that minimizes

Original: [Equation]
Revised: [Equation]

Original: The solution to
Revised: The solution to

Original: [Equation]
Revised: [Equation]

Original: a

Original: [Equation]

Original: is
Revised: is

Original: [Equation]
Revised: [Equation]

Original: The Hessian of f is positive deﬁnite since
Revised: The Hessian of f is positive deﬁnite since

Original: [Equation]
Revised: [Equation]

Original: a

Original: [Equation]

Original: and f is a convex function.
Revised: and f is a convex function.

Original: Hence, the mean 
Revised: Hence, the mean ¯ x

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: minimizes (cid:80) Ni =1 p
Revised: minimizes (cid:80) Ni =1 p

Original: i (cid:107) x
Revised: i (cid:107) x

Original: i − a (cid:107) 2 .
Revised: i − a (cid:107) 2 .

Original: Therefore, we have 
Revised: Therefore, we have l

Original: [Equation]
Revised: [Equation]

Original: We now bound the variance on the ﬁrst l patterns:
Revised: We now bound the variance on the ﬁrst l patterns:

Original: [Equation]
Revised: [Equation]

Original: We obtain for the spectral norm of J :
Revised: We obtain for the spectral norm of J :

Original: [Equation]
Revised: [Equation]

Original: Combining the previous two estimates immediately leads to Eq.
Revised: Combining the previous two estimates immediately leads to Eq.

Original: The function h ( x ) = x
Revised: The function h ( x ) = x

Original: 2(2 − x ) has the derivative h (cid:48)
Revised: 2(2 − x ) has the derivative h (cid:48)

Original: ( x ) = 4(1 − x ) .
Revised: ( x ) = 4(1 − x ) .

Original: Therefore, h ( x ) is monotone increasing for x < 1 .
Revised: Therefore, h ( x ) is monotone increasing for x < 1 .

Original: For 0 (cid:54) γ (cid:54) (cid:15) < 1 , we can immediately deduce that γ 2(2 − γ ) (cid:54) (cid:15) 2(2 − (cid:15) ) .
Revised: For 0 (cid:54) γ (cid:54) (cid:15) < 1 , we can immediately deduce that γ 2(2 − γ ) (cid:54) (cid:15) 2(2 − (cid:15) ) .

Original: Since (cid:15) is larger than γ , we obtain the following (cid:15) -bound for (cid:107) J (cid:107) 2 :
Revised: Since (cid:15) is larger than γ , we obtain the following (cid:15) -bound for (cid:107) J (cid:107) 2 :

Original: [Equation]
Revised: [Equation]

Original: We revisit the bound on (1 − γ )
Revised: We revisit the bound on (1 − γ )

Original: Var ˜ p
Revised: Var ˜ p

Original: [ x 1: l ] .
Revised: [ x 1: l ] .

Original: The trace (cid:80) dk
Revised: The trace (cid:80) dk

Original: =

Revised: =

Original: e k is the sum of the eigenvalues e k .
Revised: e k is the sum of the eigenvalues e k .

Original: The spectral norm is equal to the largest eigenvalue e 1 , that is, the largest singular value.
Revised: The spectral norm is equal to the largest eigenvalue e 1 , that is, the largest singular value.

Original: We obtain:
Revised: We obtain:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the tightness of the bound depends on eigenvalues which are not the largest.
Revised: Therefore, the tightness of the bound depends on eigenvalues which are not the largest.

Original: That is variations which are not along the strongest variation weaken the bound.
Revised: That is variations which are not along the strongest variation weaken the bound.

Original: • Proof of a ﬁxed point by Banach Fixed Point Theorem.
Revised: • Proof of a ﬁxed point by Banach Fixed Point Theorem.

Original: Without restricting the generality, we assume that the ﬁrst l patterns are much more probable (and similar to one another) than the other patterns.
Revised: Without restricting the generality, we assume that the ﬁrst l patterns are much more probable (and similar to one another) than the other patterns.

Original: Therefore, we deﬁne:
Revised: Therefore, we deﬁne:

Original: [Equation]
Revised: [Equation]

Original: M is an upper bound on the Euclidean norm of the patterns, which are vectors.
Revised: M is an upper bound on the Euclidean norm of the patterns, which are vectors.

Original: (cid:15) is an upper bound on the probability γ of not choosing one of the ﬁrst l patterns, while 1 − (cid:15) is a lower bound the probability (1 − γ ) of choosing one of the ﬁrst l patterns.
Revised: (cid:15) is an upper bound on the probability γ of not choosing one of the ﬁrst l patterns, while 1 − (cid:15) is a lower bound the probability (1 − γ ) of choosing one of the ﬁrst l patterns.

Original: m x is the arithmetic mean (the center) of the ﬁrst l patterns
Revised: m x is the arithmetic mean (the center) of the ﬁrst l patterns.

Original: m max is the maximal distance of the patterns to the center m x 
Revised: m max is the maximal distance of the patterns to the center m x .

Original: ˜ p is the probability p normalized for the ﬁrst l patterns.
Revised: ˜ p is the probability p normalized for the ﬁrst l patterns.

Original: • Mapped vectors stay in a compact environment.
Revised: • Mapped vectors stay in a compact environment.

Original: We show that if m x is sufﬁcient dissimilar to other x j with l <
Revised: We show that if m x is sufﬁcient dissimilar to other x j with l <

Original: j then there is an compact environment of m x (a sphere) where the ﬁxed point iteration maps this environment into itself.
Revised: j then there is an compact environment of m x (a sphere) where the ﬁxed point iteration maps this environment into itself.

Original: The idea of the proof is to deﬁne a sphere around m x for which the points from the sphere are mapped by f into the sphere.
Revised: The idea of the proof is to deﬁne a sphere around m x for which the points from the sphere are mapped by f into the sphere.

Original: We ﬁrst need following lemma which bounds the distance (cid:107) m x − f ( ξ ) (cid:107) of a ξ which is close to m x.
Revised: We ﬁrst need following lemma which bounds the distance (cid:107) m x − f ( ξ ) (cid:107) of a ξ which is close to m x

Original: Lemma A9.
Revised: Lemma A9.

Original: For a query ξ and data X = ( x 1 , . . .
Revised: For a query ξ and data X = ( x 1 , . . .

Original: , x N ) , we deﬁne
Revised: , x N ) , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: The following holds:
Revised: The following holds:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let s = arg max j,j (cid:54) l ξ T x j , therefore ξ T m x = 1 l (cid:80) li =1 ξ T x 
Revised: Let s = arg max j,j (cid:54) l ξ T x j , therefore ξ T m x = 1 l (cid:80) li =1 ξ T x i

Original: i (cid:54) 1 l (cid:80) li =1 ξ T x s = ξ T x s .
Revised: [Equation] 1 l [Equation] li =1 ξ T x s = ξ T x s .

Original: For softmax components j with l < j we have [softmax
Revised: For softmax components j with l < j we have [softmax( β X T ξ )] j

Original: [Equation]
Revised: [Equation]

Original:  since ξ T x s − ξ T x j ≥ ξ T m x − ξ T x j for each j with l < j , therefore ξ T x s − ξ T x j ≥ c
Revised: ) since ξ T x s − ξ T x j ≥ ξ T m x − ξ T x j for each j with l < j , therefore ξ T x s − ξ T x j ≥ c

Original: The iteration f can be written as
Revised: The iteration f can be written as

Original: [Equation]
Revised: [Equation]

Original: We set p i =
Revised: We set p i =

Original: [softmax( β X T ξ )] i , therefore (cid:80) li =1 p i = 1 − γ ≥ 1 − (cid:15) and (cid:80) Ni
Revised: [softmax( β X T ξ )] i , therefore (cid:80) li =1 p i = 1 − γ ≥ 1 − (cid:15) and (cid:80) Ni

Original: = l +1 p
Revised: = l +1 p

Original: i
Revised: i

Original: = γ (cid:54) (cid:15) .
Revised: = γ (cid:54) (cid:15) .

Original: Therefore
Revised: Therefore

Revised: [Equation]

Revised: It follows that

Original: [Equation]
Revised: [Equation]

Revised: We now can bound (cid:107) m

Original: [Equation]
Revised: [Equation]

Revised: (cid:107) :

Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (235) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m x −

Original: [Equation]
Revised: [Equation]

Original: (cid:54)
Revised: (cid:88)

Revised: j = p

Original: [Equation]
Revised: [Equation]

Revised: x

Original: It follows that (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m
Revised: [Equation] − [Equation] (cid:88)

Revised: j = l +

Original: [Equation]
Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original:  − l (cid:88)
Revised: m x − l (cid:88)

Original: j = p
Revised: j = p

Original: [Equation]
Revised: [Equation]

Original:  − γ
Revised: 1 − γ

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:54) m max (233)
Revised: γ

Original: We now can bound (cid:107) m x − f ( ξ ) (cid:107) : (cid:107) m x − f ( ξ ) (cid:107) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m x − 
Revised: [Equation] − γ [Equation]

Original: N (cid:88)
Revised: (cid:88)

Revised: [Equation]

Original: j = p
Revised: = p

Original: j x j (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (234) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m x − l (cid:88) j = p
Revised: [Equation]

Original: j x j −
Revised: x j −

Original: N (cid:88)
Revised: N (cid:88)

Original: j = l + p j xj
Revised: j = l + p [Equation] x

Original:  (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) m x − l (cid:88)
Revised: [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:54) (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13) m x − l (cid:88)

Original: j =p
Revised: j =

Original: j 1 − γ x j +γ 1 − γ l (cid:88)
Revised: [Equation] (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) (cid:13) +

Original: j = p j x j −
Revised: [Equation]

Original: N (cid:88) j =l + pj x j (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:54) (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13) m x −
Revised: (cid:13)(cid:13)(cid:13) (cid:13)(cid:13) (cid:13) l (cid:88) j = p

Original: [Equation]
Revised: [Equation]

Original: = [Equation] p [Equation]
Revised: x j (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)

Original: [Table] + [Equation] [Table]
Revised: + (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)(cid:13)

Original: = [Table] p [Table]
Revised: N (cid:88)

Original: x [Table] + [Table]
Revised: j = l + p j x j

Original: N

Original: [Table]
Revised: [Table]

Original: p
Revised: (cid:13)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where we applied Eq.
Revised: where we applied Eq.

Original: (232) in the penultimate inequality.
Revised: (233) in the penultimate inequality.

Original: This is the statement of the lemma.
Revised: This is the statement of the lemma.

Original: The separation of the center (the arithmetic mean) m x of the ﬁrst l from data X
Revised: The separation of the center (the arithmetic mean) m x of the ﬁrst l from data X

Original: = ( x  +1 , . . .
Revised: = ( x l +1 , . . .

Original: , x N ) is ∆ m , deﬁned as
Revised: , x N ) is ∆ m , deﬁned as

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The center is separated from the other data x j with l < j if 0 < ∆ m .
Revised: The center is separated from the other data x j with l < j if 0 < ∆ m .

Original: By the same arguments as in
Revised: By the same arguments as in

Original: Eq.
Revised: Eq.

Original: (139), ∆ m can also be expressed as
Revised: (140), ∆ m can also be expressed as

Original: [Equation]
Revised: [Equation]

Original:  we have ∆ [Equation]
Revised: For (cid:107) m x (cid:107) = (cid:107) x j (cid:107) we have ∆ m = 1 / 2 min j,l<j (cid:107) m x − x j (cid:107) 2 .

Original: Next we deﬁne the sphere where we want to apply Banach ﬁxed point theorem.
Revised: Next we deﬁne the sphere where we want to apply Banach ﬁxed point theorem.

Original: Deﬁnition 4 (Sphere S m ) .
Revised: Deﬁnition 4 (Sphere S m ) .

Original: The sphere S m is deﬁned as
Revised: The sphere S m is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Lemma A10.
Revised: Lemma A10.

Original: With ξ given, if the assumptions
Revised: With ξ given, if the assumptions

Original: A1: ξ is inside sphere: ξ ∈ S 
Revised: A1: ξ is inside sphere: ξ ∈ S m

Original: m , A2: the center m x is well separated from other data x j with l < j :
Revised: A2: the center m x is well separated from other data x j with l < j :

Original: [Equation]
Revised: [Equation]

Original: A : the distance m max of similar patterns to 
Revised: A3: the distance m max of similar patterns to the center is sufﬁcient small:

Original: the

Original: center

Original: is sufﬁcient small:

Original: [Equation]
Revised: [Equation]

Original: hold, then f ( ξ ) ∈ S m .
Revised: hold, then f ( ξ ) ∈ S m .

Original: Therefore, under conditions (A2) and (A3), f is a mapping from S m into S m .
Revised: Therefore, under conditions (A2) and (A3), f is a mapping from S m into S m .

Original: Proof.
Revised: Proof.

Original: We need the separation ˜∆ m of ξ from the rest of the data, which is the last N − l data points
Revised: We need the separation ˜∆ m of ξ from the rest of the data, which is the last N − l data points