Original: S IMPLICIAL E MBEDDINGS IN S ELF -S UPERVISED L EARNING AND D OWNSTREAM C LASSIFICATION
Revised: S IMPLICIAL E MBEDDINGS IN S ELF -S UPERVISED L EARNING AND D OWNSTREAM C LASSIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into L simplices of V dimensions each using a softmax operation.
Revised: Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into L simplices of V dimensions each using a softmax operation.

Original: This procedure conditions the representation onto a constrained space during pre-training and imparts an inductive bias for group sparsity.
Revised: This procedure conditions the representation onto a constrained space during pre-training and imparts an inductive bias for group sparsity.

Original: For downstream classiﬁcation, we formally prove that the SEMrepresentation leads to  better generalization than an unnormalized representation.
Revised: For downstream classiﬁcation, we provide an upper bound and argue that using SEM leads to a better expected error than the unnormalized representation.

Original: Furthermore, we empirically demonstrate that SSL methods trained with SEMs have improved generalization on natural image datasets such as CIFAR-100 and
Revised: Furthermore, we empirically demonstrate that SSL methods trained with SEMs have improved generalization on natural image datasets such as CIFAR-100 and

Original: ImageNet.
Revised: ImageNet.

Original: Finally, when used in a downstream classiﬁcation task, we show that
Revised: Finally, when used in a downstream classiﬁcation task, we show that

Original: SEM features exhibit emergent semantic coherence where small groups of learned features are distinctly predictive of semantically-relevant classes.
Revised: SEM features exhibit emergent semantic coherence where small groups of learned features are distinctly predictive of semantically-relevant classes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: [Figure]
Revised: [Figure]

Original: Self-supervised learning (SSL) is an emerging family of methods that aim to learn representations of data without manual supervision, such as class labels.
Revised: Self-supervised learning (SSL) is an emerging family of methods that aim to learn representations of data without manual supervision, such as class labels.

Original: Recent works (Hjelm et al., 2019; Grill et al., 2020; Saeed et al., 2020; You et al., 2020) learn dense representations that can solve complex tasks by simply ﬁtting a linear model on top of the learned representation.
Revised: Recent works (Hjelm et al., 2019; Grill et al., 2020; Saeed et al., 2020; You et al., 2020) learn dense representations that can solve complex tasks by simply ﬁtting a linear model on top of the learned representation.

Original: While
Revised: While

Original: SSL is already highly effective, we show that changing the type of representation learned can improve both the performance and interpretability of these methods.
Revised: SSL is already highly effective, we show that changing the type of representation learned can improve both the performance and interpretability of these methods.

Original: For this we draw inspiration from overcomplete representations: representations of an input that are nonunique combinations of a number of basis vectors greater than the input’s dimensionality (Lewicki & Sejnowski, 2000).
Revised: For this we draw inspiration from overcomplete representations: representations of an input that are nonunique combinations of a number of basis vectors greater than the input’s dimensionality (Lewicki & Sejnowski, 2000).

Original: Mostly studied in the context of the sparse coding literature (Gregor & LeCun, 2010; Goodfellow et al., 2012; Olshausen, 2013), sparse overcomplete representations have been shown to increase stability in the presence of noise (Donoho et al., 2006), have applications in neuroscience (Olshausen & Field, 1996; Lee et al., 2007), and lead to more interpretable representations (Murphy et al., 2012; Fyshe et al., 2015; Faruqui et al., 2015).
Revised: Mostly studied in the context of the sparse coding literature (Gregor & LeCun, 2010; Goodfellow et al., 2012; Olshausen, 2013), sparse overcomplete representations have been shown to increase stability in the presence of noise (Donoho et al., 2006), have applications in neuroscience (Olshausen & Field, 1996; Lee et al., 2007), and lead to more interpretable representations (Murphy et al., 2012; Fyshe et al., 2015; Faruqui et al., 2015).

Original: However, the choice of basis vectors is generally assumed to be learned using traditional methods such as ICA (Teh et al., 2003) or ﬁtting linear models (Lewicki & Sejnowski, 2000), limiting the expressive power of the encoding function.
Revised: However, the choice of basis vectors is generally assumed to be learned using traditional methods such as ICA (Teh et al., 2003) or ﬁtting linear models (Lewicki & Sejnowski, 2000), limiting the expressive power of the encoding function.

Original: In this work, we show that SSL may be used to learn sparse and overcomplete representations.
Revised: In this work, we show that SSL may be used to learn sparse and overcomplete representations.

Original: Prior work has considered sparse representation but not sparse and overcomplete representation learning with SSL; for example, Dessì et al.
Revised: Prior work has considered sparse representation but not sparse and overcomplete representation learning with SSL; for example, Dessì et al.

Original: (2021) propose to discretize the output of the encoder in a SSL model using Gumbel-Softmax (Jang et al., 2017).
Revised: (2021) propose to discretize the output of the encoder in a SSL model using Gumbel-Softmax (Jang et al., 2017).

Original: However, we show that discretization during pre-training is not necessary to achieve a sparse representation.
Revised: However, we show that discretization during pre-training is not necessary to achieve a sparse representation.

Original: Instead, we propose to project the encoder’s output into L vectors of V dimensions onto which we apply a softmax function to impart an inductive bias toward sparse vectors (Correia et al., 2019;
Revised: Instead, we propose to project the encoder’s output into L vectors of V dimensions onto which we apply a softmax function to impart an inductive bias toward sparse vectors (Correia et al., 2019;

Original: Goyal et al., 2022), also alleviating the need to usebiased or high-variance gradient estimators to train the encoder.
Revised: Goyal et al., 2022), also alleviating the need to use high-variance gradient estimators to train the encoder.

Original: We refer to this embedding as
Revised: We refer to this embedding as

Original: Simplicial Embeddings (SEM), as the softmax functions map the unnormalized representations onto
Revised: Simplicial Embeddings (SEM), as the softmax functions map the unnormalized representations onto

Original: L simplices.
Revised: L simplices.

Original: The procedure to induce SEM is simple, efﬁcient, and generally applicable.
Revised: The procedure to induce SEM is simple, efﬁcient, and generally applicable.

Original: The SSL pre-training phase, used with SEM, learns a set of L approximately -sparse vectors.
Revised: The SSL pre-training phase, used with SEM, learns a set of L approximately -sparse vectors.

Original: Key to controlling the inductive bias of SEM during pre-training is the softmax temperature parameter: the lower the temperature, the stronger the bias toward sparsity.
Revised: Key to controlling the inductive bias of SEM during pre-training is the softmax temperature parameter: the lower the temperature, the stronger the bias toward sparsity.

Original: Consistent with earlier attempts at sparse representation learning (Coates & Ng, 2011), we ﬁnd that the optimal sparsity for pre-training need not match the optimal level for downstream learning.
Revised: Consistent with earlier attempts at sparse representation learning (Coates & Ng, 2011), we ﬁnd that the optimal sparsity for pre-training need not match the optimal level for downstream learning.

Original: For downstream classiﬁcation, we may discretize the learned representation by, for example, taking the argmax for each simplex.
Revised: For downstream classiﬁcation, we may discretize the learned representation by, for example, taking the argmax for each simplex.

Original: But, we can also use SEM to control the representation’s expressivity via the softmax’s temperature.
Revised: But, we can also use SEM to control the representation’s expressivity via the softmax’s temperature.

Original: We provide a theoretical bound showing that the expected error follows a trade-off between the training error and the representations’ expressivity, controlled by the softmax’s temperature used to normalize the representation for downstream classiﬁcation.
Revised: We provide a theoretical bound showing that the expected error follows a trade-off between the training error and the representations’ expressivity that can be controlled by the softmax’s temperature used to normalize the representation for downstream classiﬁcation.

Original: Our bound also shows improved downstream generalization as we increase L and V for SEM.
Revised: Our bound also shows improved expected error as we increase L and V for SEM.

Original: SEM is generally applicable to recent SSL methods.
Revised: SEM is generally applicable to recent SSL methods.

Original: Applying it to seven different SSL methods (Chen et al., 2020b;
Revised: Applying it to seven different SSL methods (Chen et al., 2020b;

Original: He et al., 2020; Grill et al., 2020; Caron et al., 2020; 2021; Zbontar et al., 2021; Bardes et al., 2022), we ﬁnd accuracy increases of 2% to 4% on CIFAR-100.
Revised: He et al., 2020; Grill et al., 2020; Caron et al., 2020; 2021; Zbontar et al., 2021; Bardes et al., 2022), we ﬁnd accuracy increases of 2% to 4% on CIFAR-100.

Original: We observe monotonic improvement as we increase the number of vectors L , showing the beneﬁt of the overcomplete representations learned by SEM, while this improvement is absent when we do not use softmax normalization.
Revised: We observe monotonic improvement as we increase the number of vectors L , showing the beneﬁt of the overcomplete representations learned by SEM, while this improvement is absent when we do not use softmax normalization.

Original: When training a SSL method with SEM on ImageNet we also observe improvements on in-distribution compared to the baseline (Figure 1).
Revised: When training a SSL method with SEM on ImageNet we also observe improvements on in-distribution compared to the baseline (Figure 1).

Original: We also observe improvement on out-of-distribution test sets, semi-supervised learning benchmark and transfer learning datasets, demonstrating the potential of
Revised: We also observe improvement on out-of-distribution test sets, semi-supervised learning benchmark and transfer learning datasets, demonstrating the potential of

Original: SEM for large scale applications.
Revised: SEM for large scale applications.

Original: Finally, we ﬁnd that SEM learns features that are closely aligned to the semantic categories in the data.
Revised: Finally, we ﬁnd that SEM learns features that are closely aligned to the semantic categories in the data.

Original: This demonstrates that SEM learns disentangled and interpretable representations, as previously observed in overcomplete representations (Faruqui et al., 2015).
Revised: This demonstrates that SEM learns disentangled and interpretable representations, as previously observed in overcomplete representations (Faruqui et al., 2015).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The softmax operation has been used in other contexts, notably as an architectural component for models to attend to context-dependent queries via, for example, an attention mechanism (Bahdanau et al., 2016; Vaswani et al., 2017; Correia et al., 2019; Goyal et al., 2022), a mixture of experts (Jordan & Jacobs, 1993) or memory augmented networks (Graves et al., 2014).
Revised: The softmax operation has been used in other contexts, notably as an architectural component for models to attend to context-dependent queries via, for example, an attention mechanism (Bahdanau et al., 2016; Vaswani et al., 2017; Correia et al., 2019; Goyal et al., 2022), a mixture of experts (Jordan & Jacobs, 1993) or memory augmented networks (Graves et al., 2014).

Original: This operation is also used for the computation of several SSL objectives such as InfoNCE (van den Oord et al., 2018; Hjelm et al., 2019), and as a normalization of the output to compute the objective in DINO and SWaV (Caron et al., 2020; 2021).
Revised: This operation is also used for the computation of several SSL objectives such as InfoNCE (van den Oord et al., 2018; Hjelm et al., 2019), and as a normalization of the output to compute the objective in DINO and SWaV (Caron et al., 2020; 2021).

Original: Different from these, our method places the softmax at the output of an encoder to constrain the representation into a set of L sparse vectors.
Revised: Different from these, our method places the softmax at the output of an encoder to constrain the representation into a set of L sparse vectors.

Original: Similar to our approach, other architectural constraints such as Dropout (Srivastava et al., 2014),
Revised: Similar to our approach, other architectural constraints such as Dropout (Srivastava et al., 2014),

Original: BatchNorm (Ioffe & Szegedy, 2015) and LayerNorm (Ba et al., 2016) also improve the training of large neural networks.
Revised: BatchNorm (Ioffe & Szegedy, 2015) and LayerNorm (Ba et al., 2016) also improve the training of large neural networks.

Original: However, contrary to SEMs, they are not used to induce sparsity on the representation or control its expressivity for downstream tasks.
Revised: However, contrary to SEMs, they are not used to induce sparsity on the representation or control its expressivity for downstream tasks.

Original: Closer to our work, Liu et al.
Revised: Closer to our work, Liu et al.

Original: (2021) propose to constrain the expressivity of the representation of a neural network with a set of discrete-valued symbols obtained using a set of Vector Quantized (Oord et al., 2018) bottlenecks.
Revised: (2021) propose to constrain the expressivity of the representation of a neural network with a set of discrete-valued symbols obtained using a set of Vector Quantized (Oord et al., 2018) bottlenecks.

Original: Similarly, Dessi et al.
Revised: Similarly, Dessi et al.

Original: (2021) propose a communication game with a discrete bottleneck.
Revised: (2021) propose a communication game with a discrete bottleneck.

Original: The idea of discretizing the encoder’s output is similar to using SEM vectors that are one-hot (e.g. temperature = 0 ) and only one symbol (e.g. L = 1 , V = 2048 ).
Revised: The idea of discretizing the encoder’s output is similar to using SEM vectors that are one-hot (e.g. temperature = 0 ) and only one symbol (e.g. L = 1 , V = 2048 ).

Original: In our work, we ﬁnd success in removing the hard-discretization and having L > 1 , which can be interepreted as combining several symbols.
Revised: In our work, we ﬁnd success in removing the hard-discretization and having L > 1 , which can be interepreted as combining several symbols.

Original:  S IMPLICIAL E MBEDDINGS
Revised:  S IMPLICIAL E MBEDDINGS

Original: Simplicial Embeddings (SEM) are representations that can be integrated easily into a contrastive learning model (Hjelm et al., 2019;
Revised: Simplicial Embeddings (SEM) are representations that can be integrated easily into a contrastive learning model (Hjelm et al., 2019;

Original: Chen et al., 2020b), the BYOL method (Grill et al., 2020), and other SSL methods (Caron et al., 2020; 2021; Zbontar et al., 2021).
Revised: Chen et al., 2020b), the BYOL method (Grill et al., 2020), and other SSL methods (Caron et al., 2020; 2021; Zbontar et al., 2021).

Original: For example, in BYOL, we insert the SEM after the encoder and before the projector and the rest is unchanged as shown in Figure 2c.
Revised: For example, in BYOL, we insert the SEM after the encoder and before the projector and the rest is unchanged as shown in Figure 2c.

Original: In this ﬁgure, t and t (cid:48) are augmentations deﬁned by the practitioner, ξ are parameters of the target network that are updated as moving average of the parameters θ of the online networks trained with
Revised: In this ﬁgure, t and t (cid:48) are augmentations deﬁned by the practitioner, ξ are parameters of the target network that are updated as moving average of the parameters θ of the online networks trained with

Original: SGD.
Revised: SGD.

Original: So, ξ are updated as follow: ξ ← αξ + (1 − α ) θ , with α ∈ [0 , 1] .
Revised: So, ξ are updated as follow: ξ ← αξ + (1 − α ) θ , with α ∈ [0 , 1] .

Original: [Figure]
Revised: [Figure]

Original: a) 0.
Revised: a) 0.

Original: 2.5 Entropy H ( z i )     : 10.
Revised: 2.5 Entropy H ( z i )     : 10.

Original: [Figure]
Revised: [Figure]

Original: τ
Revised: τ

Original: = large number of simplices two elements close
Revised: = large number of simplices two elements close

Original: Grill
Revised: Grill

Original: To produce SEM representation, the encoder’s output e is embedded into L vectors z i ∈ R V .
Revised: To produce SEM representation, the encoder’s output e is embedded into L vectors z i ∈ R V .

Original: A temperature parameter τ scales z i , and then a softmax re-normalizes each vector z
Revised: A temperature parameter τ scales z i , and then a softmax re-normalizes each vector z

Original: i to produce ¯ z i .
Revised: i to produce ¯ z i .

Original: Finally, the normalized vectors ¯ z i are concatenated to produce the vector ˆ z of length L · V .
Revised: Finally, the normalized vectors ¯ z i are concatenated to produce the vector ˆ z of length L · V .

Original: We illustrate SEM in Figure 2a.
Revised: We illustrate SEM in Figure 2a.

Original: Formally, the re-normalization is as follows:
Revised: Formally, the re-normalization is as follows:

Original: [Equation]
Revised: [Equation]

Original: I
Revised: I

Original: NDUCTIVE BIAS TOWARDS SPARSITY DURING PRE - TRAINING
Revised: NDUCTIVE BIAS TOWARDS SPARSITY DURING PRE - TRAINING

Original: In SEM, L controls the numbers of simplices and V controls the dimensionality of each simplex.
Revised: In SEM, L controls the numbers of simplices and V controls the dimensionality of each simplex.

Original: As such, the higher V is, the sparser the representation can be.
Revised: As such, the higher V is, the sparser the representation can be.

Original: During pre-training, the constraint induced by embedding the representation into a simplex biases each vector towards sparse vectors by creating a zero-sum competition between the components of the vector.
Revised: During pre-training, the constraint induced by embedding the representation into a simplex biases each vector towards sparse vectors by creating a zero-sum competition between the components of the vector.

Original: In order for a component to increase by α , then the other elements must decrease by α , and all elements are bounded by 0 .
Revised: In order for a component to increase by α , then the other elements must decrease by α , and all elements are bounded by 0 .

Original: For networks to learn useful features and minimize their objective, they must prioritize some components at the expense of others.
Revised: For networks to learn useful features and minimize their objective, they must prioritize some components at the expense of others.

Original: The strength of this bias is controlled via the pretraining temperature τ p of the softmax, and the size of the vectors V as it was noted in the context of attention (Vaswani et al., 2017; Wang et al., 2021b).
Revised: The strength of this bias is controlled via the pretraining temperature τ p of the softmax, and the size of the vectors V as it was noted in the context of attention (Vaswani et al., 2017; Wang et al., 2021b).

Original: For SSL methods with a target network, the temperature for the target network can be different to the online network’s as no gradient is back-propagated through it.
Revised: For SSL methods with a target network, the temperature for the target network can be different to the online network’s as no gradient is back-propagated through it.

Original: To visualize the effect of the temperature on SEM after pre-training, we interpret each simplex as a probability mass function p (¯ z ij ) where, for all i ∈
Revised: To visualize the effect of the temperature on SEM after pre-training, we interpret each simplex as a probability mass function p (¯ z ij ) where, for all i ∈

Original: [ L ] ,
Revised: [ L ] ,

Original: (cid:80) Vj
Revised: (cid:80) Vj

Original: =1 p (¯ z ij ) = 1 and p (¯ z ij ) ≥ 0 ∀ j .
Revised: =1 p (¯ z ij ) = 1 and p (¯ z ij ) ≥ 0 ∀ j .

Original: The entropy of a simplex ¯ z i , deﬁned as H (¯ z i ) :
Revised: The entropy of a simplex ¯ z i , deﬁned as H (¯ z i ) :

Original: = − (cid:80) V j =1 p (¯ z ij ) log p (¯ z ij ) , informs whether the simplex is a sparse or a dense vector.
Revised: = − (cid:80) V j =1 p (¯ z ij ) log p (¯ z ij ) , informs whether the simplex is a sparse or a dense vector.

Original: That is, if H (¯ z ( x ) i ) = 

Revised: That is, if H (¯ z ( x ) i ) =

Original: then the vector is one-hot.
Revised: then the vector is one-hot.

Original: On the other hand, if
Revised: On the other hand, if

Original: H (¯ z ( x ) i ) =
Revised: H (¯ z ( x ) i ) =

Original: ln( V ) then the vector is dense and uniform.
Revised: ln( V ) then the vector is dense and uniform.

Original: While the temperature τ p is merely a scaling of the logits, it has an important control over the learned representation’s entropy and resulting SEM sparsity.
Revised: While the temperature τ p is merely a scaling of the logits, it has an important control over the learned representation’s entropy and resulting SEM sparsity.

Original: We demonstrate this by learning a representation on CIFAR-100 using BYOL, and analyze the entropies of the resulting simplices.
Revised: We demonstrate this by learning a representation on CIFAR-100 using BYOL, and analyze the entropies of the resulting simplices.

Original: In Figure 2b, we plot the histogram of the entropies H (¯ z i ) , for a given τ p , of each simplex for each sample in the training set of CIFAR-100.
Revised: In Figure 2b, we plot the histogram of the entropies H (¯ z i ) , for a given τ p , of each simplex for each sample in the training set of CIFAR-100.

Original: We observe that even after pre-training, small temperatures ( τ p = 0 . 01 )
Revised: We observe that even after pre-training, small temperatures ( τ p = 0 . 01 )

Original: yields representations that are close to one-hot vectors while high temperatures yields vectors that are close to uniform vectors.
Revised: yields representations that are close to one-hot vectors while high temperatures yields vectors that are close to uniform vectors.

Original: By pre-training using a softmax, SEMs create representations that are conditioned to ﬁt onto simplices.
Revised: By pre-training using a softmax, SEMs create representations that are conditioned to ﬁt onto simplices.

Original: In pre-training, we select τ p for optimal inductive bias: τ p too small yields vanishing gradients (Wang et al., 2021b) and τ p too large yields a bias that is too weak.
Revised: In pre-training, we select τ p for optimal inductive bias: τ p too small yields vanishing gradients (Wang et al., 2021b) and τ p too large yields a bias that is too weak.

Original: We may select a different optimal τ d for downstream performance as discussed formally in the next subsection.
Revised: We may select a different optimal τ d for downstream performance as discussed formally in the next subsection.

Original: 3. SEM IMPROVEMENT ON THE GENERALIZATION OF THE DOWNSTREAM CLASSIFIER
Revised: 3. SEM IMPROVEMENT ON THE GENERALIZATION OF THE DOWNSTREAM CLASSIFIER

Original: In this subsection, we theoretically demonstrate the beneﬁt of training a downstream classiﬁer with
Revised: In this subsection, we theoretically demonstrate the beneﬁt of training a downstream classiﬁer with

Original: SEM normalized input compared to a baseline classiﬁer with unnormalized input.
Revised: SEM normalized input compared to a baseline classiﬁer with unnormalized input.

Original: We show that: (1) there is a trade-off between the training loss and the generalization gap, which is controlled by the value of τ d (denoted τ :=
Revised: We show that: (1) there is a trade-off between the training loss and the generalization gap, which is controlled by the value of τ d (denoted τ :=

Original: τ d in this subsection), (2) SEM can improve the base model performance when we attain good balance in this trade-off, and (3) the improvement due to SEM is expected to increase or stay constant as L and V increase.
Revised: τ d in this subsection), (2) SEM can improve the base model performance when we attain good balance in this trade-off, and (3) the improvement due to SEM is expected to increase or stay constant as L and V increase.

Original: In the remainder of this subsection, we introduce the notation and assumptions needed to understand and derive the result, then present our theoretical claim and discuss its implications.
Revised: In the remainder of this subsection, we introduce the notation and assumptions needed to understand and derive the result, then present our theoretical claim and discuss its implications.

Original: Notation.
Revised: Notation.

Original: We use a training dataset S =
Revised: We use a training dataset S =

Original: ( z ( i ) , y ( i ) )
Revised: ( z ( i ) , y ( i ) )

Original: ni
Revised: ni

Original: =1 of n samples for supervised training of a classiﬁer, using the representation z extracted from the pre-trained model 1 and the corresponding label y ∈ Y where Y is the space of possible labels.
Revised: =1 of n samples for supervised training of a classiﬁer, using the representation z extracted from the pre-trained model 1 and the corresponding label y ∈ Y where Y is the space of possible labels.

Original: Assume that z ∈ Z = [ − 1 , +1]
Revised: Assume that z ∈ Z = [ − 1 , +1]

Original: L × V , which means that z is a matrix with L rows and V columns.
Revised: L × V , which means that z is a matrix with L rows and V columns.

Original: We denote the element of z at row i and column j as z ij .
Revised: We denote the element of z at row i and column j as z ij .

Original: Let g represent the downstream classiﬁer.
Revised: Let g represent the downstream classiﬁer.

Original: We refer to the baseline downstream model with unnormalized input as f base , and f base ( z ) =
Revised: We refer to the baseline downstream model with unnormalized input as f base , and f base ( z ) =

Original: g ( z ) .
Revised: g ( z ) .

Original: The corresponding downstream model trained with the SEM normalization is f SEM( τ ) ( z ) =
Revised: The corresponding downstream model trained with the SEM normalization is f SEM( τ ) ( z ) =

Original: ( g ◦ σ τ )( z ) , where σ τ is applied element-wise along each row of z such that σ τ ( z ij ) =
Revised: ( g ◦ σ τ )( z ) , where σ τ is applied element-wise along each row of z such that σ τ ( z ij ) =

Original: e zij/τ (cid:80) Vt =1 e zit/τ for j = 1 , . .
Revised: e zij/τ (cid:80) Vt =1 e zit/τ for j = 1 , . .

Original: , V .
Revised: , V .

Original: Moreover, we deﬁne f S base and f S SEM( τ ) the base and the SEM normalized models obtained by ﬁtting the dataset S .
Revised: Moreover, we deﬁne f S base and f S SEM( τ ) the base and the SEM normalized models obtained by ﬁtting the dataset S .

Original: Finally, let H be the union of the hypothesis spaces of f SEM( τ ) and f base .
Revised: Finally, let H be the union of the hypothesis spaces of f SEM( τ ) and f base .

Original: To compare the quality of the base model and the model with SEM normalization, we analyze the generalization gap E z,y [ l ( f S ( z ) , y )]
Revised: To compare the quality of the base model and the model with SEM normalization, we analyze the generalization gap E z,y [ l ( f S ( z ) , y )]

Original: − 1 n (cid:80) ni
Revised: − 1 n (cid:80) ni

Original: =1 l ( f S ( z ( i ) ) , y ( i ) ) for each f S ∈ { f S SEM( τ ) , f S base } , where l : R × Y → R ≥ 0 is the per-sample loss.
Revised: =1 l ( f S ( z ( i ) ) , y ( i ) ) for each f S ∈ { f S SEM( τ ) , f S base } , where l : R × Y → R ≥ 0 is the per-sample loss.

Original: The key insight that we exploit for the theorem is that the softmax operation σ τ controls the expressivity of the input’s representation to g via the temperature τ .
Revised: The key insight that we exploit for the theorem is that the softmax operation σ τ controls the expressivity of the input’s representation to g via the temperature τ .

Original: We denote ϕ f base as an upper bound on the expressivity of z i for the baseline model f base , and ϕ f SEM( τ ) as the upper bound on the expressivity of σ τ ( z i ) for the model with SEM normalization f SEM( τ ) .
Revised: We denote ϕ f base as an upper bound on the expressivity of z i for the baseline model f base , and ϕ f SEM( τ ) as the upper bound on the expressivity of σ τ ( z i ) for the model with SEM normalization f SEM( τ ) .

Original: The formal deﬁnition of ϕ f base and ϕ f SEM( τ ) requires proof devices that will hinder the readability of this section, so we refer the reader to Appendix A for a detailed deﬁnition.
Revised: The formal deﬁnition of ϕ f base and ϕ f SEM( τ ) requires proof devices that will hinder the readability of this section, so we refer the reader to Appendix A for a detailed deﬁnition.

Original: Let ϕ f ∈ { ϕ f base , ϕ f SEM( τ ) } .
Revised: Let ϕ f ∈ { ϕ f base , ϕ f SEM( τ ) } .

Original: Intuitively, ϕ f S measures the largest possible distance that two embeddings can have such that the largest component remains the same for both embeddings.
Revised: Intuitively, ϕ f S measures the largest possible distance that two embeddings can have such that the largest component remains the same for both embeddings.

Original: We note that this measure depends only on V for f base , and on both V and τ for f SEM( τ ) .
Revised: We note that this measure depends only on V for f base , and on both V and τ for f SEM( τ ) .

Original: We use ϕ f S ( V, τ ) to denote the measure given by either model and note that τ has no effect for f base .
Revised: We use ϕ f S ( V, τ ) to denote the measure given by either model and note that τ has no effect for f base .

Original: Assumptions.
Revised: Assumptions.

Original: We assume that the per-sample loss is bounded such that l ( f ( z ) , y ) ≤ B for all f ∈ H and for all
Revised: We assume that the per-sample loss is bounded such that l ( f ( z ) , y ) ≤ B for all f ∈ H and for all

Original: ( z, y ) ∈
Revised: ( z, y ) ∈

Original: Z × Y .
Revised: Z × Y .

Original: For example, B = 1 for the 0-1 loss.
Revised: For example, B = 1 for the 0-1 loss.

Original: Next, let l y be the per-sample loss given y .
Revised: Next, let l y be the per-sample loss given y .

Original: We assume that l y ◦ g are uniformly Lipschitz functions for all y ∈ Y and g ∈ G S , where G S is the set of classiﬁers g returned by the training algorithm using the dataset S .
Revised: We assume that l y ◦ g are uniformly Lipschitz functions for all y ∈ Y and g ∈ G S , where G S is the set of classiﬁers g returned by the training algorithm using the dataset S .

Original: Let R be such a uniform
Revised: Let R be such a uniform

Original: Lipschitz constant.
Revised: Lipschitz constant.

Original: This means that | ( l y ◦ g )( σ f ( z )) − ( l y ◦ g )( σ f ( z (cid:48) )) | ≤ R (cid:107) σ f ( z ) − σ f ( z (cid:48) )
Revised: This means that | ( l y ◦ g )( σ f ( z )) − ( l y ◦ g )( σ f ( z (cid:48) )) | ≤ R (cid:107) σ f ( z ) − σ f ( z (cid:48) )

Original: (cid:107) F , where l y ( g ◦ σ f ( z ))
Revised: (cid:107) F , where l y ( g ◦ σ f ( z ))

Original: =
Revised: =

Original: l ( g ◦ σ f
Revised: l ( g ◦ σ f

Original: ( z ) , y ) , and σ f = σ τ when f = f
Revised: ( z ) , y ) , and σ f = σ τ when f = f

Original: SEM( τ ) and σ f is identity when f = f base .
Revised: SEM( τ ) and σ f is identity when f = f base .

Original: Finally, we assume that there exists ∆ > 0 such that for all representations z of the underlying distribution we have that for any i ∈
Revised: Finally, we assume that there exists ∆ > 0 such that for all representations z of the underlying distribution we have that for any i ∈

Original: [ L ] ,
Revised: [ L ] ,

Original: if k = arg max j ∈ [ V ] z ij , then z ik ≥ z ij + ∆ for any j (cid:54) =
Revised: if k = arg max j ∈ [ V ] z ij , then z ik ≥ z ij + ∆ for any j (cid:54) =

Original: k .
Revised: k .

Original: Since ∆ can be arbitrarily small (e.g. as small as machine precision), this assumption typically holds in practice.
Revised: Since ∆ can be arbitrarily small (e.g. as small as machine precision), this assumption typically holds in practice.

Original: We are now ready to state our theoretical claim.
Revised: We are now ready to state our theoretical claim.

Original: Theorem 1 illuminates the advantage of SEM and the effect of the hyper-parameter τ on the performance of the downstream classiﬁer.
Revised: Theorem 1 illuminates the advantage of SEM and the effect of the hyper-parameter τ on the performance of the downstream classiﬁer.

Original: We present the proof in Appendix A and we present empirical evidence of the theorem’s prediction in Figure 5.
Revised: We present the proof in Appendix A.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let V ≥ 2 .
Revised: Let V ≥ 2 .

Original: For any 1 ≥ δ > 0 , with probability at least 1 − δ , the following holds for any f S ∈ { f S SEM( τ ) , f S base } :
Revised: For any 1 ≥ δ > 0 , with probability at least 1 − δ , the following holds for any f S ∈ { f S SEM( τ ) , f S base } :

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is a constant in ( n, f, H , δ, τ, S ) .
Revised: where c > 0 is a constant in ( n, f, H , δ, τ, S ) .

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst statement of Theorem 1 shows that the expected loss is bounded by the three terms: the training loss 1 n (cid:80) ni =1 l ( f S ( z ( i ) ) , y ( i ) ) , the second term R (cid:112)
Revised: The ﬁrst statement of Theorem 1 shows that the expected loss is bounded by the three terms: the training loss 1 n (cid:80) ni =1 l ( f S ( z ( i ) ) , y ( i ) ) , the second term R (cid:112)

Original: Lϕ f S , and the third term c (cid:113) ln(2 /δ ) n .
Revised: Lϕ f S , and the third term c (cid:113) ln(2 /δ ) n .

Original: Since c is a constant in ( n, f, H , δ, τ, S ) , the third term goes to zero as n → ∞ and is the same with and without SEM.
Revised: Since c is a constant in ( n, f, H , δ, τ, S ) , the third term goes to zero as n → ∞ and is the same with and without SEM.

Original: Thus, for the purpose of assessing the impact of SEM, we can focus on the second term, where a difference arises.
Revised: Thus, for the purpose of assessing the impact of SEM, we can focus on the second term, where a difference arises.

Original: Theorem 1 shows that R (cid:112)
Revised: Theorem 1 shows that R (cid:112)

Original: Lϕ f S goes to zero with SEM; i.e., ϕ ( f S SEM( τ ) )
Revised: Lϕ f S goes to zero with SEM; i.e., ϕ ( f S SEM( τ ) )

Original: → 0 as τ → 0 .
Revised: → 0 as τ → 0 .

Original: Also, for any τ > 0 , the second term with SEM is strictly smaller than that without SEM as ϕ [Equation] ) < [-
-] and demonstrates that the improvement due to SEM is expected to asymptotically increase as V increases.
Revised: Also, for any τ > 0 , the second term with SEM is strictly smaller than that without SEM as ϕ f S SEM( τ ) − ϕ f S base ≤ 34 (1 − V ) < [-
-]0 and demonstrates that the improvement due to SEM is expected to asymptotically increase as V increases.

Original: Moreover, L is a multiplicative constant of ϕ which shows that, as L increases, the improvement due to SEM is also expected to be higher.
Revised: Moreover, L is a multiplicative constant of ϕ which shows that, as L increases, the improvement due to SEM is also expected to be higher.

Original: Overall, Theorem 1 shows the beneﬁt of SEM as well as the trade-off with τ .
Revised: Overall, Theorem 1 shows the beneﬁt of SEM as well as the trade-off with τ .

Original: When τ → 0 , the second term goes to zero, but the training loss (the ﬁrst term) can increase due to underﬁtting resulting from the reduction in representation expressivity.
Revised: When τ → 0 , the second term goes to zero, but the training loss (the ﬁrst term) can increase due to underﬁtting resulting from the reduction in representation expressivity.

Original: Thus, τ should be chosen to optimally balance this trade-off.
Revised: Thus, τ should be chosen to optimally balance this trade-off.

Original:  E MPIRICAL ANALYSIS
Revised:  E MPIRICAL ANALYSIS

Original: We empirically study the effect of SEM on the representation of SSL methods and demonstrate that
Revised: We empirically study the effect of SEM on the representation of SSL methods and demonstrate that

Original: SEM improves the test set accuracy on CIFAR-100 (Krizhevsky, 2009).
Revised: SEM improves the test set accuracy on CIFAR-100 (Krizhevsky, 2009).

Original: We compare SEM with other methods for inducing sparse representations during pretraining and demonstrate that SEM lead to better downstream accuracy.
Revised: We compare SEM with other methods for inducing sparse representations during pretraining and demonstrate that SEM lead to better downstream accuracy.

Original: On I MAGE N ET (Deng et al., 2009), we study the effect of SEM on robustness, semi-supervised learning and transfer learning datasets, demonstrating consistent improvement attributed to SEM.
Revised: On I MAGE N ET (Deng et al., 2009), we study the effect of SEM on robustness, semi-supervised learning and transfer learning datasets, demonstrating consistent improvement attributed to SEM.

Original: Finally, we present evidences that features produced by SEMs are more naturally aligned with the semantic categories of the data.
Revised: Finally, we present evidences that features produced by SEMs are more naturally aligned with the semantic categories of the data.

Original: Training setup.
Revised: Training setup.

Original: For all experiments, we build off the implementation of the baseline models from the
Revised: For all experiments, we build off the implementation of the baseline models from the

Original: Solo-Learn library (da Costa et al., 2021).
Revised: Solo-Learn library (da Costa et al., 2021).

Original: We probe the encoder’s output for the baseline methods, as typically done in the literature.
Revised: We probe the encoder’s output for the baseline methods, as typically done in the literature.

Original: For models with SEM, we probe the SEM normalized representation (i.e. ˆ z ).
Revised: For models with SEM, we probe the SEM normalized representation (i.e. ˆ z ).

Original: In our experiments, the embedder is a linear layer followed by BatchNorm (Ioffe & Szegedy, 2015).
Revised: In our experiments, the embedder is a linear layer followed by BatchNorm (Ioffe & Szegedy, 2015).

Original: Unless mentioned otherwise, we use L = 5000 and V = 13 for the SEM representation.
Revised: Unless mentioned otherwise, we use L = 5000 and V = 13 for the SEM representation.

Original: We do not perform any search for the non-SEM hyper-parameters.
Revised: We do not perform any search for the non-SEM hyper-parameters.

Original: The SEM hyper-parameters are selected by using a validation set of 10% of the training set of CIFAR-100 and 10 samples per class on the in distribution dataset for I MAGE N ET .
Revised: The SEM hyper-parameters are selected by using a validation set of 10% of the training set of CIFAR-100 and 10 samples per class on the in distribution dataset for I MAGE N ET .

Original: The test accuracy is obtained by retraining the model with all of the training data using the parameters found with the validation set.
Revised: The test accuracy is obtained by retraining the model with all of the training data using the parameters found with the validation set.

Original: We pre-train the 
Revised: We pre-train the SSL models for 200 epochs on I MAGE N ET and 1000 epochs on CIFAR-100.

Original: [Table]
Revised: [Table]

Original: 4. SEM IMPROVES ON DOWNSTREAM CLASSIFICATION
Revised: 4. SEM IMPROVES ON DOWNSTREAM CLASSIFICATION

Original: Baseline comparison.
Revised: Baseline comparison.

Original: We evaluate the effect of adding SEMs in seven modern SSL approaches.
Revised: We evaluate the effect of adding SEMs in seven modern SSL approaches.

Original: We take standard SimCLR (Chen et al., 2020b), MoCo-v2 (He et al., 2020), BYOL (Grill et al., 2020)
Revised: We take standard SimCLR (Chen et al., 2020b), MoCo-v2 (He et al., 2020), BYOL (Grill et al., 2020)

Original: Barlow-Twins (Zbontar et al., 2021), SwAV (Caron et al., 2020), DINO (Caron et al., 2021) and VicReg (Bardes et al., 2022) models and implement SEM after the encoder.
Revised: Barlow-Twins (Zbontar et al., 2021), SwAV (Caron et al., 2020), DINO (Caron et al., 2021) and VicReg (Bardes et al., 2022) models and implement SEM after the encoder.

Original: We compare our approach on CIFAR-100 with a ResNet-18  in Table 1.
Revised: We compare our approach on CIFAR-100 with a ResNet-18 and ResNet-50 in Table 1.

Revised: We found SWaV and DINO to be unstable with ResNet-50 thus have decided not to compare them with SEM.

Original: For every SSL methods, using SEMs improves the baseline methods by 2% to 4% demonstrating that SEM is a general approach that improves in-distribution generalization for SSL methods.
Revised: For every SSL methods, using SEMs improves the baseline methods by 2% to 4% demonstrating that SEM is a general approach that improves in-distribution generalization for SSL methods.

Original: [Figure]
Revised: [Figure]

Original: Accuracy
Revised: (V Q

Original: BYOL  70 .
Revised: BYOL ( e θ ) 70 .

Original: BYOL+Gumbel S.-T.
Revised: BYOL+G.S. ( e θ ) 63 . BYOL+G.S. ( ˆ z θ ) 54 . BYOL+V.Q. ( e θ ) 65 . BYOL+V

Original: BYOL+V.Q.
Revised: [Equation] Q. ( ˆ z θ ) 60 . BYOL+SEM( ˆ [Equation]

Original: BYOL+SEM ( τ d = 0) 73 .
Revised: θ [Equation] d = 0 ) 73 .

Original: BYOL+SEM ( τ d = 0 . 1) 73 .
Revised: BYOL+SEM( ˆ z θ , τ d = 0 . 1) 73 .

Original: Increasing the representation’s size of SEM increases the performance.
Revised: Increasing the representation’s size of SEM increases the performance.

Original: We ﬁnd that increasing L (the number of simplices of SEM) beyond the over-complete regime increases the downstream accuracy.
Revised: We ﬁnd that increasing L (the number of simplices of SEM) beyond the over-complete regime increases the downstream accuracy.

Original: This increased performance is not observed when we abstain from using the softmax normalization of SEM.
Revised: This increased performance is not observed when we abstain from using the softmax normalization of SEM.

Original: In Figure 3, using a ResNet-50 encoder, we compareBYOL + SEM, with an identical model without the
Revised: In Figure 3, using a ResNet-50 encoder, we compare

Original:  Softmax normalization which we call BYOL + Embed. As this is a control experiment, the extracted representation of BYOL + Embed is the embedder’s output z θ  .
Revised: BYOL + SEM, with an identical model without the Softmax normalization which we call BYOL + Embed and BYOL to which we increase the representation’s size before the meanpooling using the method proposed in (Dubois et al., 2022) and described in their Appendix F. To be clear, the extracted representation of BYOL + Embed is the embedder’s output z θ and the extracted representation for BYOL is the encoder’s output e θ .

Original: We ﬁx V =and scale L ∈ [10 , 10000] to get a range of representation sizes.
Revised: We ﬁx V = 13 and scale L ∈ [10 , 10000] to get a range of representation sizes.

Original: The mean and standard deviation over 5 seeds is plotted.

Original: This experiment demonstrates that SEM offers a simple way to scale up the capacity of the model and that the softmax normalization is necessary to attein increase performance.

Original: Comparison of SEM with hard discretization approaches.
Revised: Comparison of SEM with hard discretization approaches.

Original: Several other methods can be used to induce a sparse and overcomplete representation during pre-training and downstream classiﬁcation.
Revised: Several other methods can be used to induce a sparse and overcomplete representation during pre-training and downstream classiﬁcation.

Original: For example, we may sample L discrete one-hot codes of V dimensions using Gumbel Softmax (Jang et al., 2017) as done in Dessì et al.
Revised: For example, we may sample L discrete one-hot codes of V dimensions using Gumbel Softmax (Jang et al., 2017) as done in Dessì et al.

Original: We can also use Vector
Revised: We can also use Vector

Original: Quantization (VQ) (Oord et al., 2018) and consider L latent embedding spaces with V embedding vectors each, wherein the vectors are in R d .
Revised: Quantization (VQ) (Oord et al., 2018) and consider L latent embedding spaces with V embedding vectors each, wherein the vectors are in R d .

Original: In contrast to SEM, it is not possible to propagate the gradient through the bottleneck trivially and VQ uses straight-through estimation in the embedding space to backpropagate the gradient to the encoder.
Revised: In contrast to SEM, it is not possible to propagate the gradient through the bottleneck trivially and VQ uses straight-through estimation in the embedding space to back-propagate the gradient to the encoder.

Original: Here, we observe that these alternative approaches exhibit a considerable decrease in performance in comparison to the baseline as demonstrated in Table 2.
Revised: Here, we observe that these alternative approaches exhibit a considerable decrease in performance in comparison to the baseline as demonstrated in Table 2.

Original: In this table, we reproduce the same setup as SEM
Revised: In this table, we reproduce the same setup as SEM

Original: but we replace the Softmax with hard discretization baselines methods.
Revised: but we replace the Softmax with hard discretization baselines methods.

Original: For discretization with Gumbel Straight-Through estimation, we use the same setup as SEM with L = 5000 and V = 13 , that is 5000 one-hot vectors of 13 dimensions and τ = 2 2 .
Revised: For discretization with Gumbel Straight-Through estimation, we use the same setup as SEM with L = 5000 and V = 13 , that is 5000 one-hot vectors of 13 dimensions and τ = 2 2 .

Original: For VQ, we found that L = [-
-]
Revised: For VQ, we found that L = [-
-]512 and

Original: and V = 128 led to the best performance.
Revised: V = 128 led to the best performance.

Original: That is, we havelatent embedding spaces, each with 128 possible embedding vectors that are in R 32 .
Revised: That is, we have 512 latent embedding spaces, each withpossible embedding vectors that are in R 32 .

Original: We note that while we have not found hard-discretization to be successful during pre-training, we may hard-discretize a SEM representation for downstream task.
Revised: We note that while we have not found hard-discretization to be successful during pre-training, we may hard-discretize a SEM representation for downstream task.

Original: In Table 2, we also present SEM with τ DS = 0 , which correspond to using the discretized representation for downstream classiﬁcation.
Revised: In Table 2, we also present SEM with τ DS = 0 , which correspond to using the discretized representation for downstream classiﬁcation.

Original: We obtain the discrete representation by taking the argmax for each simplex.
Revised: We obtain the discrete representation by taking the argmax for each simplex.

Original: This result demonstrating that SEM with pre-training can be used to learn meaningful discrete codes for downstream applications and yields better performance than the baselines, implying that pre-training with SEM could be be used in applications that require discretization.
Revised: This result demonstrating that SEM with pre-training can be used to learn meaningful discrete codes for downstream applications and yields better performance than the baselines, implying that pre-training with SEM could be be used in applications that require discretization.

Original: Memory and computational efﬁciency of SEM.
Revised: Memory and computational efﬁciency of SEM.

Original: SEM’s performance improvements come at a cost of increased memory allocation (VRAM) due to additional parameters needed to perform the matrix multiplication, and slightly more computation (FLOPs/sample).
Revised: SEM’s performance improvements come at a cost of increased memory allocation (VRAM) due to additional parameters needed to perform the matrix multiplication, and slightly more computation (FLOPs/sample).

Original: For very large over-complete representation the increased memory requirement can impede practical application.
Revised: For very large over-complete representation the increased memory requirement can impede practical application.

Original: We propose a more efﬁcient version of SEM by sparsifying the matrix multiplication of the embedder and of the projector and detail this procedure in Appendix D.1.
Revised: We propose a more efﬁcient version of SEM by sparsifying the matrix multiplication of the embedder and of the projector and detail this procedure in Appendix D.1.

Original: As shown in Table 17, SEM with sparse matrix multiplication use only slightly more memory and compute but outperforms the BYOL baseline on CIFAR-100 though underperforming the regular SEM.
Revised: As shown in Table 17, SEM with sparse matrix multiplication use only slightly more memory and compute but outperforms the BYOL baseline on CIFAR-100 though underperforming the regular SEM.

Original: We also note that SEM’s memory cost becomes relatively minor as we scale up the encoder.
Revised: We also note that SEM’s memory cost becomes relatively minor as we scale up the encoder.

Original: As well, the computational cost of SEM is small compared to the total cost of pre-training and achieves higher accuracy using fewer FLOPs compared to scaling the encoder as shown in Figure 1.
Revised: As well, the computational cost of SEM is small compared to the total cost of pre-training and achieves higher accuracy using fewer FLOPs compared to scaling the encoder as shown in Figure 1.

Original: 4. A NALYZING THE PARAMETERS OF SEM
Revised: 4. A NALYZING THE PARAMETERS OF SEM

Original: We present two ﬁgures in this section to better understand the effect of the parameters of SEM on the downstream accuracy.
Revised: We present two ﬁgures in this section to better understand the effect of the parameters of SEM on the downstream accuracy.

Original: In Figure 4, we evaluate the effect of changing τ p and τ d on the downstream accuracy.
Revised: In Figure 4, we evaluate the effect of changing τ p and τ d on the downstream accuracy.

Original: In Figure 5, we evaluate the effect of L and V on the downstream accuracy and also contrast f base and
Revised: In Figure 5, we evaluate the effect of L and V on the downstream accuracy and also contrast f base and

Original: f SEM ( τ = 1) , allowing us to conﬁrm two predictions made in Section 3.2: the expected generalization improvement from SEM increases as we increase L and as we increase V .
Revised: f SEM ( τ = 1) by using the same encoder pre-trained with SEM. This allows us to relate some observations to the theorey presented in Section 3.2.

Original: Now, we discuss the effect of each of SEM’s parameter on the resulting downstream classiﬁcation.
Revised: Now, we discuss the effect of each of SEM’s parameter on the resulting downstream classiﬁcation.

Original: Increasing V yields a steep performance increase for small V but quickly plateau.
Revised: Increasing V yields a steep performance increase for small V but quickly plateau.

Original: In Figure 5b, we observe a steep increase of the accuracy for V < 13 followed by a plateau for V > 13 .
Revised: In Figure 5b, we observe a steep increase of the accuracy for V < 13 followed by a plateau for V > 13 .

Original: In
Revised: In

Original: Figure 4a, we observe that the optimal accuracy obtained for V = 1024 and L = 64 is similar to the one obtained for L = 50 (Embedding size=650) in Figure 3.
Revised: Figure 4a, we observe that the optimal accuracy obtained for V = 1024 and L = 64 is similar to the one obtained for L = 50 (Embedding size=650) in Figure 3.

Original: Increasing L yields monotonical improvement for downstream classiﬁcation.
Revised: Increasing L yields monotonical improvement for downstream classiﬁcation.

Original: In the regime that we can test it, increasing L lead to consistent improvement on the downstream accuracy as observed in
Revised: In the regime that we can test it, increasing L lead to consistent improvement on the downstream accuracy as observed in

Original: Figure 3 and Figure 5a.
Revised: Figure 3 and Figure 5a.

Original: Using SEM in pre-training only is not enough and using it in the downstream classiﬁer is necessary for the improved performance as demonstrated in Figure 5a.
Revised: Using SEM in pre-training only is not enough and using it in the downstream classiﬁer is necessary for the improved performance as demonstrated in Figure 5a.

Original: The optimal τ p depends on V .
Revised: The optimal τ p depends on V .

Original: As previously noted in the context of Attention (Vaswani et al., 2017;
Revised: As previously noted in the context of Attention (Vaswani et al., 2017;

Original: Wang et al., 2021a), the optimal attention’s temperature is proportional to attention’s vector size.
Revised: Wang et al., 2021a), the optimal attention’s temperature is proportional to attention’s vector size.

Original: We also observe this in SEM.
Revised: We also observe this in SEM.

Original: As presented in Figure 4a, the optimal τ p for larger V is higher.
Revised: As presented in Figure 4a, the optimal τ p for larger V is higher.

Original: Models with larger L are more robust to smaller τ d .
Revised: Models with larger L are more robust to smaller τ d .

Original: In Figure 4, we observe that SSL models are more robust to smaller τ d as L increase.
Revised: In Figure 4, we observe that SSL models are more robust to smaller τ d as L increase.

Original: We speculate that the information can be scattered across the simplices for large L , allowing to reduce the expressivity of each vector with minimal impact on the downstream accuracy
Revised: We speculate that the information can be scattered across the simplices for large L , allowing to reduce the expressivity of each vector with minimal impact on the downstream accuracy.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4. SEM IMPROVEMENT ON LARGE - SCALE DATASETS WITH I MAGE N ET
Revised: 4. SEM IMPROVEMENT ON LARGE - SCALE DATASETS WITH I MAGE N ET

Original: Figure 1 in the introduction demonstrates that using SEM leads to better in distribution generalization for I MAGE N ET and is a more efﬁcient method of scaling up the model as compared to scaling up the width of the ResNet-50 encoder.
Revised: Figure 1 in the introduction demonstrates that using SEM leads to better in distribution generalization for I MAGE N ET and is a more efﬁcient method of scaling up the model as compared to scaling up the width of the ResNet-50 encoder.

Original: Here, we demonstrate that SEM generally improves the accuracy on several robustness test sets, a semi-supervised learning benchmark and transfer learning datasets.
Revised: Here, we demonstrate that SEM generally improves the accuracy on several robustness test sets, a semi-supervised learning benchmark and transfer learning datasets.

Original: We use BYOL+SEM with an embedding size of 105 000 features ( L = 5000 and V = 21 ) for these experiments.
Revised: We use BYOL+SEM with an embedding size of 105 000 features ( L = 5000 and V = 21 ) for these experiments.

Original: The embedding is pre-trained for 200 epochs using the BYOL SSL procedure.
Revised: The embedding is pre-trained for 200 epochs using the BYOL SSL procedure.

Original: Robustness to out-of-distribution test sets.
Revised: Robustness to out-of-distribution test sets.

Original: We perform a comparative study using several test sets: ( IN ) the in-distribution test set provided in I MAGE N ET ; ( IN-C ) I MAGE N ET -C, which exhibits a set of common image corruptions (Hendrycks & Dietterich, 2019); ( IN-R ) I MAGE N ET -R
Revised: We perform a comparative study using several test sets: ( IN ) the in-distribution test set provided in I MAGE N ET ; ( IN-C ) I MAGE N ET -C, which exhibits a set of common image corruptions (Hendrycks & Dietterich, 2019); ( IN-R ) I MAGE N ET -R

Original: (Hendrycks et al., 2021) which consists of different renderings for several I MAGE N ET classes; and ( IN-V2 )
Revised: (Hendrycks et al., 2021) which consists of different renderings for several I MAGE N ET classes; and ( IN-V2 )

Original: I MAGE N ETV2 (Recht et al., 2019), a distinct test set for I MAGE N ET collected using the same process; ( IN-A )
Revised: I MAGE N ETV2 (Recht et al., 2019), a distinct test set for I MAGE N ET collected using the same process; ( IN-A )

Original: I MAGENET -A
Revised: I MAGENET -A

Original: (Chen et al., 2020a) contains a set of samples that are miclassiﬁer by a I MAGE N ET 
Revised: (Chen et al., 2020a) contains a set of samples that are miclassiﬁer by a I MAGE N ET [Table] [Table]

Original: ResNet-50 classiﬁer.
Revised: ResNet-50 classiﬁer.

Original: We use the methodology and software proposed in Djolonga et al.
Revised: We use the methodology and software proposed in Djolonga et al.

Original: [Table] [Table] to perform our experiments.
Revised: (2020; 2021) to perform our experiments.

Original: We observe that BYOL + SEM outperforms BYOL on every robustness datasets probed, demonstrating that SEM also improves generalization to out-of-distribution test sets.
Revised: We observe that BYOL + SEM outperforms BYOL on every robustness datasets probed, demonstrating that SEM also improves generalization to out-of-distribution test sets.

Original: Transfer learning.
Revised: Transfer learning.

Original: We probe the effectiveness of SEM in BYOL and MoCo when transferring representations trained on I MAGE N ET to other classiﬁcation tasks.
Revised: We probe the effectiveness of SEM in BYOL and MoCo when transferring representations trained on I MAGE N ET to other classiﬁcation tasks.

Original: We follow the linear evaluation and ﬁne-tuning methodologies described in previous works (Grill et al., 2020; Lee et al., 2021)
Revised: We follow the linear evaluation and ﬁne-tuning methodologies described in previous works (Grill et al., 2020; Lee et al., 2021)

Original: [Table]
Revised: [Table]

Original: which entails training a linear classiﬁer with logistic regression using sklearn (Pedregosa et al., 2011) on the embeddings of the samples and ﬁne-tuning the encoder respectively.
Revised: which entails training a linear classiﬁer with logistic regression using sklearn (Pedregosa et al., 2011) on the embeddings of the samples and ﬁne-tuning the encoder respectively.

Original: To avoid out-of-memory issues that may occur in the linear probe experiment with the sklearn solver when the number of features, we discretize our features and use sparse matrix to ﬁt the logistic regression.
Revised: To avoid out-of-memory issues that may occur in the linear probe experiment with the sklearn solver when the number of features, we discretize our features and use sparse matrix to ﬁt the logistic regression.

Original: This is equivalent to forcing τ d = 0 for all the experiments.
Revised: This is equivalent to forcing τ d = 0 for all the experiments.

Original: For the ﬁne-tuning experiments, we ﬁx τ d = 1 since the evaluation method allows for mini-batch gradient descent.
Revised: For the ﬁne-tuning experiments, we ﬁx τ d = 1 since the evaluation method allows for mini-batch gradient descent.

Original: We perform our transfer learning experiments on the following datasets: F OOD (Bossard et al., 2014), CIFAR-10 (C-10) (Krizhevsky, 2009), CIFAR-100 (C-100) (Krizhevsky, 2009), SUN (Xiao et al., 2010), DTD (Cimpoi et al., 2014) and F LOWER (Nilsback & Zisserman, 2008).
Revised: We perform our transfer learning experiments on the following datasets: F OOD (Bossard et al., 2014), CIFAR-10 (C-10) (Krizhevsky, 2009), CIFAR-100 (C-100) (Krizhevsky, 2009), SUN (Xiao et al., 2010), DTD (Cimpoi et al., 2014) and F LOWER (Nilsback & Zisserman, 2008).

Original: This task evaluates the generality of the encoder as it has to encode samples from various out-ofdistribution domains with categories that it may not have seen during training.
Revised: This task evaluates the generality of the encoder as it has to encode samples from various out-ofdistribution domains with categories that it may not have seen during training.

Original: We present our results in Table 4 and observe that SEM improves the transfer accuracy over the baseline for every datasets but DTD for the linear probe experiment.
Revised: We present our results in Table 4 and observe that SEM improves the transfer accuracy over the baseline for every datasets but DTD for the linear probe experiment.

Original: For DTD, we hypothesize that the drop in performance is due to the fact that we use a temperature that is too small.
Revised: For DTD, we hypothesize that the drop in performance is due to the fact that we use a temperature that is too small.

Original: Since this is a texture dataset with higher frequency, it might be the case that we need more expressivity to correctly ﬁt the data.
Revised: Since this is a texture dataset with higher frequency, it might be the case that we need more expressivity to correctly ﬁt the data.

Original: We support the conjecture with the ﬁne-tuning experiment where BYOL + SEM out-performs the baseline.
Revised: We support the conjecture with the ﬁne-tuning experiment where BYOL + SEM out-performs the baseline.

Original: [Table]
Revised: [Table]

Original: Semi-supervised learning.
Revised: Semi-supervised learning.

Original: We evaluate the effect of using SEM when ﬁne-tuning on a classiﬁcation task with a small subset of I MAGE N ET ’s training set.
Revised: We evaluate the effect of using SEM when ﬁne-tuning on a classiﬁcation task with a small subset of I MAGE N ET ’s training set.

Original: We follow the semi-supervised learning procedure of Chen et al.
Revised: We follow the semi-supervised learning procedure of Chen et al.

Original: (2020b);
Revised: (2020b);

Original: Grill et al.
Revised: Grill et al.

Original: and use the same ﬁxed splits of 1% and 10% of ImageNet labelled training set.
Revised: and use the same ﬁxed splits of 1% and 10% of ImageNet labelled training set.

Original: In Table 5, we demonstrate that using SEM lead to an important increased performance, especially in the low supervised data regime.
Revised: In Table 5, we demonstrate that using SEM lead to an important increased performance, especially in the low supervised data regime.

Original: 4. S EMANTIC COHERENCE OF SEM FEATURES
Revised: 4. S EMANTIC COHERENCE OF SEM FEATURES

Original: Here we demonstrate that SEM features are coherently aligned with the semantics present in the training data.
Revised: Here we demonstrate that SEM features are coherently aligned with the semantics present in the training data.

Original: Qualitatively, we visualize the most predictive features of a downstream linear classiﬁer trained on CIFAR-100 and see that the classes with similar predictive features are semantically related.
Revised: Qualitatively, we visualize the most predictive features of a downstream linear classiﬁer trained on CIFAR-100 and see that the classes with similar predictive features are semantically related.

Original: Quantitatively we propose a metric that returns the ratio of features mostly predictive for a classes that are in the same super class to total number of class predictive for this feature.
Revised: Quantitatively we propose a metric that returns the ratio of features mostly predictive for a classes that are in the same super class to total number of class predictive for this feature.

Original: For both our analysis, we use a linear classiﬁer trained on the features extracted from BYOL with and without SEM.
Revised: For both our analysis, we use a linear classiﬁer trained on the features extracted from BYOL with and without SEM.

Original: Consider the trained linear classiﬁer with a weight matrix W ∈ R N × C , with N features, and C classes.
Revised: Consider the trained linear classiﬁer with a weight matrix W ∈ R N × C , with N features, and C classes.

Original: By preserving the top K parameters of the weight matrix W for each class and pruning the features predictive for only one class, we create a bipartite graph between two set of nodes: the CIFAR-100 classes and the features of the representation.
Revised: By preserving the top K parameters of the weight matrix W for each class and pruning the features predictive for only one class, we create a bipartite graph between two set of nodes: the CIFAR-100 classes and the features of the representation.

Original: We denote this graph W K .
Revised: We denote this graph W K .

Original: The qualitative analysis is given by plotting the subset W 5 , obtained by taking the top 5 features for each class.
Revised: The qualitative analysis is given by plotting the subset W 5 , obtained by taking the top 5 features for each class.

Original: We present a subset of the graph for BYOL+SEM in Figure 6a and for BYOL in Figure 6b.
Revised: We present a subset of the graph for BYOL+SEM in Figure 6a and for BYOL in Figure 6b.

Original: The full graphs are presented in the Appendix.
Revised: The full graphs are presented in the Appendix.

Original: In the SEM plot, a set of connected componentsemerge, and the connected components of the graph are semantically related. For example, the
Revised: In the SEM plot, a set of connected components

Original: [Figure]
Revised: [Figure]

Original: the total number of pairwise categories.
Revised: the total number of pairwise categories.

Revised: emerge, and the connected components of the graph are semantically related.

Original:  ﬁrst set of connected components are ﬂowers, and the last set of connected components are aquatic mammals.
Revised: For example, the ﬁrst set of connected components are ﬂowers, and the last set of connected components are aquatic mammals.

Original: The same class coherence is not observed with either the BYOL baseline or with BYOL augmented with a large representation.
Revised: The same class coherence is not observed with either the BYOL baseline or with BYOL augmented with a large representation.

Original: In particular, we do not see a small number of semantically related connected components.
Revised: In particular, we do not see a small number of semantically related connected components.

Original: Instead, we see a large fully connected graphs.
Revised: Instead, we see a large fully connected graphs.

Original: Next, we describe how we quantitatively measure the semantic coherence of the features.
Revised: Next, we describe how we quantitatively measure the semantic coherence of the features.

Original: Notice that two classes share a common predictive feature on W K if they are 2-neighbour.
Revised: Notice that two classes share a common predictive feature on W K if they are 2-neighbour.

Original: Let N ( c i ) returns all pairs ( c i , c j ) for all j 2-neighbour of c i .
Revised: Let N ( c i ) returns all pairs ( c i , c j ) for all j 2-neighbour of c i .

Original: Moreover, deﬁne the operation is_super ( c i , c j ) which returns 1 if c i and c j are from the same CIFAR-100 superclass and 0 otherwise.
Revised: Moreover, deﬁne the operation is_super ( c i , c j ) which returns 1 if c i and c j are from the same CIFAR-100 superclass and 0 otherwise.

Original: We reproduce the superclass of CIFAR-100 in Table 21 in the Appendix.
Revised: We reproduce the superclass of CIFAR-100 in Table 22 in the Appendix.

Original: We measure semantic coherence as follows:
Revised: We measure semantic coherence as follows:

Original: [Equation]
Revised: [Equation]

Original: where C = 100 for CIFAR-100 and | · | is the cardinality of a set.
Revised: where C = 100 for CIFAR-100 and | · | is the cardinality of a set.

Original: We compare the semantic coherence of BYOL+SEM with the control experiments on BYOL: regular
Revised: We compare the semantic coherence of BYOL+SEM with the control experiments on BYOL: regular

Original: BYOL, BYOL with an embedding of the same size as BYOL+SEM but without the normalization and BYOL to which we applied linear ICA (Hyvärinen & Oja, 2000) in an attempt to disentangle the features.
Revised: BYOL, BYOL with an embedding of the same size as BYOL+SEM but without the normalization and BYOL to which we applied linear ICA (Hyvärinen & Oja, 2000) in an attempt to disentangle the features.

Original: In Figure 9, we plot the full graph W 5 for BYOL+SEM and the baselines.
Revised: In Figure 10, we plot the full graph W 5 for BYOL+SEM and the baselines.

Original: We observe that using the SEM yields semantically coherent features for all the classes of CIFAR-100.
Revised: We observe that using the SEM yields semantically coherent features for all the classes of CIFAR-100.

Original: This observation is consistent with the qualitative and quantitative experiments presented earlier and demonstrates that SEM’s inductive bias during pre-training leads to features that are semantically coherent with the semantic categories extant in the data.
Revised: This observation is consistent with the qualitative and quantitative experiments presented earlier and demonstrates that SEM’s inductive bias during pre-training leads to features that are semantically coherent with the semantic categories extant in the data.

Original: This arguably have important implications for improving the interpretability of SSL representations.
Revised: This arguably have important implications for improving the interpretability of SSL representations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: SEM is a simple, drop-in module that creates sparse overcomplete representations for standard SSL methods using a softmax operation.
Revised: SEM is a simple, drop-in module that creates sparse overcomplete representations for standard SSL methods using a softmax operation.

Original: This simple modiﬁcation leads to improved generalization on downstream classiﬁcation across several state-of-the-art SSL methods.
Revised: This simple modiﬁcation leads to improved generalization on downstream classiﬁcation across several state-of-the-art SSL methods.

Original: Furthermore, SEM improves performance on out-of-distribution, semi-supervised, and transfer learning tasks across the board and also scales with encoder size.
Revised: Furthermore, SEM improves performance on out-of-distribution, semi-supervised, and transfer learning tasks across the board and also scales with encoder size.

Original: By analyzing semantic coherence, we ﬁnd that SEMs naturally disentangle data into semantic categories without any explicit training objectives.
Revised: By analyzing semantic coherence, we ﬁnd that SEMs naturally disentangle data into semantic categories without any explicit training objectives.

Original: We hope this work motivates the investigation of representational inductive biases for SSL, in addition to models or different training procedures.
Revised: We hope this work motivates the investigation of representational inductive biases for SSL, in addition to models or different training procedures.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: Let us introduce additional notations used in the proofs.
Revised: Let us introduce additional notations used in the proofs.

Original: Deﬁne r
Revised: Deﬁne r

Original: =
Revised: =

Original: ( z, y ) ∈ R , (cid:96) ( f, r ) =
Revised: ( z, y ) ∈ R , (cid:96) ( f, r ) =

Original: l ( f ( z ) , y ) , ˜ C y,k 1 ,...,k L = { ( z, ˆ y )
Revised: l ( f ( z ) , y ) , ˜ C y,k 1 ,...,k L = { ( z, ˆ y )

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: Z

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: y
Revised: y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , and
Revised: , and

Original: [Equation]
Revised: [Equation]

Original: We then deﬁne C k to be the ﬂatten version of
Revised: We then deﬁne C k to be the ﬂatten version of

Original: [Equation]
Revised: [Equation]

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: ,k 1 ,...,k L ; i.e., {C k } Kk = =
Revised: ,k 1 ,...,k L ; i.e., {C k } Kk = =

Original: { ˜ C y,k 1 ,...,k L ,y } y ∈Y ,k 1 ,...,k L ∈ [ V ] with C 1 = ˜ C 1 , 1 ,..., 1 , C 2 = ˜ C 2 , 1 ,..., 1 , C |Y| =
Revised: { ˜ C y,k 1 ,...,k L ,y } y ∈Y ,k 1 ,...,k L ∈ [ V ] with C 1 = ˜ C 1 , 1 ,..., 1 , C 2 = ˜ C 2 , 1 ,..., 1 , C |Y| =

Original: ˜ C |Y| , 1 ,..., 1 , C |Y| +1 = ˜
Revised: ˜ C |Y| , 1 ,..., 1 , C |Y| +1 = ˜

Original: C 1 , 2 , 1 ,..., 1 , C 2 |Y| =
Revised: C 1 , 2 , 1 ,..., 1 , C 2 |Y| =

Original: ˜ C |Y| , 2 , 1 ,..., 1 , and so on.
Revised: ˜ C |Y| , 2 , 1 ,..., 1 , and so on.

Original: Similarly, deﬁne Z k to be the ﬂatten version of ˜
Revised: Similarly, deﬁne Z k to be the ﬂatten version of ˜

Original: Z k 1 ,...,k L .
Revised: Z k 1 ,...,k L .

Original: We also use Q i = { q ∈ [ − 1 , +1]
Revised: We also use Q i = { q ∈ [ − 1 , +1]

Original: V : i = arg
Revised: V : i = arg

Original: max j ∈
Revised: max j ∈

Original: [ V ] q j } , I k :
Revised: [ V ] q j } , I k :

Original: = I Sk :=
Revised: = I Sk :=

Original: { i ∈ [ n ] : r i ∈ C k } , and α k ( h ) :=
Revised: { i ∈ [ n ] : r i ∈ C k } , and α k ( h ) :=

Original: E r
Revised: E r

Original: [ (cid:96) ( h, r ) | r ∈ C k ] .
Revised: [ (cid:96) ( h, r ) | r ∈ C k ] .

Original: Moreover, we deﬁne ϕ
Revised: Moreover, we deﬁne ϕ

Original: ( f S base
Revised: ( f S base

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: 22 , and ϕ ( f S SEM( τ ) )
Revised: 22 , and ϕ ( f S SEM( τ ) )

Original: = sup i ∈
Revised: = sup i ∈

Original: [ V ] sup q,q (cid:48) ∈ Q i (cid:107) σ τ ( q ) − σ τ ( q (cid:48) )
Revised: [ V ] sup q,q (cid:48) ∈ Q i (cid:107) σ τ ( q ) − σ τ ( q (cid:48) )

Original: (cid:107) 

Revised: (cid:107)

Original: where σ τ ( q ) j = e qj/τ (cid:80) Vt =1 e qt/τ for j = 1 , . .
Revised: where σ τ ( q ) j = e qj/τ (cid:80) Vt =1 e qt/τ for j = 1 , . .

Original: , V .
Revised: , V .

Original: We ﬁrst decompose the generalization gap into two terms using the following lemma:
Revised: We ﬁrst decompose the generalization gap into two terms using the following lemma:

Original: Lemma 1.
Revised: Lemma 1.

Original: For any δ > 0 , with probability at least 1 − δ ,the following holds for all h ∈ H :
Revised: For any δ > 0 , with probability at least 1 − δ ,the following holds for all h ∈ H :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst write the expected error as the sum of the conditional expected error:
Revised: We ﬁrst write the expected error as the sum of the conditional expected error:

Original: [Equation]
Revised: [Equation]

Original: where r k is the random variable for the conditional with r ∈ C k .
Revised: where r k is the random variable for the conditional with r ∈ C k .

Original: Using this, we decompose the generalization error into two terms:
Revised: Using this, we decompose the generalization error into two terms:

Original: [Equation]
Revised: [Equation]

Original: The second term in the right-hand side of (3) is further simpliﬁed by using
Revised: The second term in the right-hand side of (3) is further simpliﬁed by using

Original: [Equation]
Revised: [Equation]

Original: as
Revised: as

Original: [Equation]
Revised: [Equation]

Original: Substituting these into equation (3) yields
Revised: Substituting these into equation (3) yields

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ≤ B
Revised: ≤ B

Original: K (cid:88) k
Revised: K (cid:88) k

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By using the Bretagnolle-Huber-Carol inequality (van der Vaart & Wellner, 1996, A6.6 Proposition), we have that for any δ > 0 , with probability at least 1 − δ ,
Revised: By using the Bretagnolle-Huber-Carol inequality (van der Vaart & Wellner, 1996, A6.6 Proposition), we have that for any δ > 0 , with probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: Here, notice that the term of (cid:80) Kk =1 (cid:12)(cid:12)(cid:12) Pr( r ∈ C k ) − |I
Revised: Here, notice that the term of (cid:80) Kk =1 (cid:12)(cid:12)(cid:12) Pr( r ∈ C k ) − |I

Original: k | n (cid:12)(cid:12)(cid:12) does not depend on h ∈ H .
Revised: k | n (cid:12)(cid:12)(cid:12) does not depend on h ∈ H .

Original: Moreover, note that for any ( f, h, M ) such that M > 0 and B ≥ 0 for all X , we have that P ( f ( X ) ≥ M ) ≥ P ( f ( X ) > M ) ≥ P ( Bf ( X )
Revised: Moreover, note that for any ( f, h, M ) such that M > 0 and B ≥ 0 for all X , we have that P ( f ( X ) ≥ M ) ≥ P ( f ( X ) > M ) ≥ P ( Bf ( X )

Original: + h ( X ) > BM + h ( X )) , where the probability is with respect to the randomness of X .
Revised: + h ( X ) > BM + h ( X )) , where the probability is with respect to the randomness of X .

Original: Thus, by combining (4) and (5), we have that for any h ∈ H , for any δ > 0 , with probability at least 1 − δ , the following holds for all h ∈ H ,
Revised: Thus, by combining (4) and (5), we have that for any h ∈ H , for any δ > 0 , with probability at least 1 − δ , the following holds for all h ∈ H ,

Original: [Equation]
Revised: [Equation]

Original: In particular, the ﬁrst term from the previous lemma will be bounded with the following lemma:
Revised: In particular, the ﬁrst term from the previous lemma will be bounded with the following lemma:

Original: Lemma 2.
Revised: Lemma 2.

Original: For any f ∈ { f S SEM( τ ) , f
Revised: For any f ∈ { f S SEM( τ ) , f

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By using the triangle inequality,
Revised: By using the triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: Furthermore, by using the triangle inequality,
Revised: Furthermore, by using the triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: If f = f S SEM( τ ) = g S SEM( τ ) ◦ σ τ , since g S SEM( τ ) ∈ G S , by using the Lipschitz continuity, boundedness, and non-negativity,
Revised: If f = f S SEM( τ ) = g S SEM( τ ) ◦ σ τ , since g S SEM( τ ) ∈ G S , by using the Lipschitz continuity, boundedness, and non-negativity,

Original: [Equation]
Revised: [Equation]

Original: Similarly, if f = f S base = g S base , since g S base ∈ G S , by using the Lipschitz continuity, boundedness, and non-negativity,
Revised: Similarly, if f = f S base = g S base , since g S base ∈ G S , by using the Lipschitz continuity, boundedness, and non-negativity,

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any f ∈ { f S SEM( τ ) , f S base } ,
Revised: Therefore, for any f ∈ { f S SEM( τ ) , f S base } ,

Original: [Equation]
Revised: [Equation]

Original: Combining Lemma 1 and Lemma 2, we obtain the following upper bound on the gap:
Revised: Combining Lemma 1 and Lemma 2, we obtain the following upper bound on the gap:

Original: Lemma 3.
Revised: Lemma 3.

Original: For any δ > 0 , with probability at least 1 − δ , the following holds for any f ∈ { f S SEM( τ ) , f S base } :
Revised: For any δ > 0 , with probability at least 1 − δ , the following holds for any f ∈ { f S SEM( τ ) , f S base } :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This follows directly from combining Lemma 1 and Lemma 2.
Revised: This follows directly from combining Lemma 1 and Lemma 2.

Original: We now provide an upper bound on ϕ ( f S SEM( τ ) ) in the following lemma:
Revised: We now provide an upper bound on ϕ ( f S SEM( τ ) ) in the following lemma:

Original: Lemma 4.
Revised: Lemma 4.

Original: For any τ > 0 ,
Revised: For any τ > 0 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Recall the deﬁnition:
Revised: Recall the deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: for j = 1 , . . .
Revised: for j = 1 , . . .

Original: , V .
Revised: , V .

Original: By the symmetry and independence over i ∈ [ V ] inside of the ﬁrst supremum, we have
Revised: By the symmetry and independence over i ∈ [ V ] inside of the ﬁrst supremum, we have

Original: [Equation]
Revised: [Equation]

Original: For any q, q (cid:48) ∈ Q 1 and i ∈ { 2 , . . .
Revised: For any q, q (cid:48) ∈ Q 1 and i ∈ { 2 , . . .

Original: , V } (with q = ( q 1 , . .
Revised: , V } (with q = ( q 1 , . .

Original: , q V ) and q (cid:48) =
Revised: , q V ) and q (cid:48) =

Original: ( q (cid:48) 1 , . . .
Revised: ( q (cid:48) 1 , . . .

Original: , q (cid:48) V ) ), there exists δ
Revised: , q (cid:48) V ) ), there exists δ

Original: i , δ (cid:48)
Revised: i , δ (cid:48)

Original: i > 0 such that
Revised: i > 0 such that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: i .
Revised: i .

Original: Here, since z ik − ∆ ≥ z ij from the assumption, we have that for all i ∈ { 2 , . . .
Revised: Here, since z ik − ∆ ≥ z ij from the assumption, we have that for all i ∈ { 2 , . . .

Original: , V } , δ i , δ (cid:48)
Revised: , V } , δ i , δ (cid:48)

Original: i ≥ ∆ > 0 .
Revised: i ≥ ∆ > 0 .

Original: Thus, we can rewrite
Revised: Thus, we can rewrite

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Using these,
Revised: Using these,

Original: [Equation]
Revised: [Equation]

Original: and for all j ∈ { 2 , . . .
Revised: and for all j ∈ { 2 , . . .

Original: , V } , σ
Revised: , V } , σ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: where I j := { 2 , . .
Revised: where I j := { 2 , . .

Original: . , V } \ { j } .
Revised: . , V } \ { j } .

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: and for all j ∈ { 2 , . . .
Revised: and for all j ∈ { 2 , . . .

Original: , V } ,
Revised: , V } ,

Original: [Equation]
Revised: [Equation]

Original: Using these
Revised: Using these

Original: [Equation]
Revised: [Equation]

Original: for any q
Revised: for any q

Original: [Equation]
Revised: [Equation]

Original: q (cid:48) ∈ Q 1 ,
Revised: q (cid:48) ∈ Q 1 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: By combining these,
Revised: By combining these,

Original: sup q
Revised: sup q

Original: [Equation]
Revised: [Equation]

Original: Using the previous lemma
Revised: Using the previous lemma

Original: [Equation]
Revised: [Equation]

Original: we will conclude the asymptotic behavior of ϕ ( f S SEM( τ ) ) in the following lemma:
Revised: we will conclude the asymptotic behavior of ϕ ( f S SEM( τ ) ) in the following lemma:

Original: Lemma 5.
Revised: Lemma 5.

Original: It holds that
Revised: It holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using Lemma
Revised: Using Lemma

Original: [Equation]
Revised: [Equation]

Original: /τ )
Revised: /τ )

Original: − 1 + e 2 /τ (1 + ( V − 2)
Revised: − 1 + e 2 /τ (1 + ( V − 2)

Original: e − ∆ /τ ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .
Revised: e − ∆ /τ ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .

Original: Moreover, lim
Revised: Moreover, lim

Original: [Equation]
Revised: [Equation]

Original: → 0 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 +
Revised: → 0 (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 1 +

Original: [Equation]
Revised: [Equation]

Original: τ −
Revised: τ −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: /τ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) = (cid:12)(cid:12)(cid:12)(cid:12)
Revised: /τ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) = (cid:12)(cid:12)(cid:12)(cid:12)

Original: 1 1 − 1 1 (cid:12)(cid:12)(cid:12)(cid:12)
Revised: 1 1 − 1 1 (cid:12)(cid:12)(cid:12)(cid:12)

Original: = 0 , and lim τ →(cid:12) (cid:12) (cid:12) (cid:12)(cid:12) 

Revised: = 0 , and lim τ →(cid:12) (cid:12) (cid:12) (cid:12)(cid:12)

Original: + e
Revised: + e

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) − 1 + e
Revised: ) − 1 + e

Original: [Equation]
Revised: [Equation]

Original: /τ (1 + (
Revised: /τ (1 + (

Original: [Equation]
Revised: [Equation]

Original: 2) e
Revised: 2) e

Original: [Equation]
Revised: [Equation]

Original: /τ ) (cid:12) (cid:12) (cid:12) (cid:12)(cid:12) =
Revised: /τ ) (cid:12) (cid:12) (cid:12) (cid:12)(cid:12) =

Original: | 0 − 0 | 2 = 0 .
Revised: | 0 − 0 | 2 = 0 .

Original: Therefore, lim τ → 0 ϕ ( f S SEM( τ ) ) ≤ 0 .
Revised: Therefore, lim τ → 0 ϕ ( f S SEM( τ ) ) ≤ 0 .

Original: Since ϕ ( f S SEM( τ ) ) ≥ 0 , this implies the statement of this lemma.
Revised: Since ϕ ( f S SEM( τ ) ) ≥ 0 , this implies the statement of this lemma.

Original: As we have analyzed ϕ ( f S SEM( τ ) ) in the previous two lemmas
Revised: As we have analyzed ϕ ( f S SEM( τ ) ) in the previous two lemmas

Original: [Equation]
Revised: [Equation]

Original: we are now ready to compare ϕ
Revised: we are now ready to compare ϕ

Original: ( f S SEM( τ ) ) and ϕ ( f S base ) , which is done in the following lemma:
Revised: ( f S SEM( τ ) ) and ϕ ( f S base ) , which is done in the following lemma:

Original: Lemma 6.
Revised: Lemma 6.

Original: For any τ > 0 ,
Revised: For any τ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Lemma 4, for any τ > 0 ,
Revised: From Lemma 4, for any τ > 0 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Recall the deﬁnition of
Revised: Recall the deﬁnition of

Original: [Equation]
Revised: [Equation]

Original: By choosing an element in the set over which the supremum is taken, for any δ ≥ ∆ > 0 , ϕ
Revised: By choosing an element in the set over which the supremum is taken, for any δ ≥ ∆ > 0 , ϕ

Original: [Equation]
Revised: [Equation]

Original: f S
Revised: f S

Original: [Equation]
Revised: [Equation]

Original: where ˆ q 1 = 1 , ˆ q j = 1 − δ for j ∈ { 2 , . .
Revised: where ˆ q 1 = 1 , ˆ q j = 1 − δ for j ∈ { 2 , . .

Original: , V } , ˆ q (cid:48) 

Revised: , V } , ˆ q (cid:48)

Original: = δ − 1 , and ˆ q (cid:48)
Revised: = δ − 1 , and ˆ q (cid:48)

Original: j = − 1 for j ∈ { 2 , . .
Revised: j = − 1 for j ∈ { 2 , . .

Original: , V } .
Revised: , V } .

Original: By combining those, for for any τ > 0 and δ ≥ ∆ > 0 ,
Revised: By combining those, for for any τ > 0 and δ ≥ ∆ > 0 ,

Original: [Equation]
Revised: [Equation]

Original: We combine the lemmas above to prove Theorem 1, which is restated below with its proof:
Revised: We combine the lemmas above to prove Theorem 1, which is restated below with its proof:

Original: Theorem 1.
Revised: Theorem 1.

Original: Let V ≥ 2 .
Revised: Let V ≥ 2 .

Original: For any 1 ≥ δ > 0 , with probability at least 1 − δ , the following holds for any f S ∈ { f S SEM( τ ) , f S base } :
Revised: For any 1 ≥ δ > 0 , with probability at least 1 − δ , the following holds for any f S ∈ { f S SEM( τ ) , f S base } :

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is a constant in ( n, f, H , δ, τ, S ) .
Revised: where c > 0 is a constant in ( n, f, H , δ, τ, S ) .

Original: Moreover,
Revised: Moreover,

Original: [Equation]
Revised: [Equation]

Original: →
Revised: →

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The ﬁrst statement directly follows from Lemma 3.
Revised: The ﬁrst statement directly follows from Lemma 3.

Original: The second statement is proven by
Revised: The second statement is proven by

Original: Lemma 5 and Lemma 6.
Revised: Lemma 5 and Lemma 6.

Original: B E XPERIMENT DETAILS FOR I MAGE N ET B. I MAGE AUGMENTATION
Revised: B E XPERIMENT DETAILS FOR I MAGE N ET B. I MAGE AUGMENTATION

Original: The augmentation applied in order during training are:
Revised: The augmentation applied in order during training are:

Original: • Random Resize crop to a 224 × 224 image.
Revised: • Random Resize crop to a 224 × 224 image.

Original: A random patch of the image is selected and resized to a 224 × 224 image.
Revised: A random patch of the image is selected and resized to a 224 × 224 image.

Original: • Random color jitter.
Revised: • Random color jitter.

Original: Modifying the brightness, the contrast, the saturation and the hue.
Revised: Modifying the brightness, the contrast, the saturation and the hue.

Original: • Random gray scale.
Revised: • Random gray scale.

Original: Randomly applying a gray scale ﬁlter to the image • Random Gaussian blur.
Revised: Randomly applying a gray scale ﬁlter to the image • Random Gaussian blur.

Original: Randomly applying a Gaussian bluer ﬁlter.
Revised: Randomly applying a Gaussian bluer ﬁlter.

Original: • Random solarization.
Revised: • Random solarization.

Original: Randomly applying a solarization ﬁlter.
Revised: Randomly applying a solarization ﬁlter.

Original: The parameters of the augmentations are presented in Table 16.
Revised: The parameters of the augmentations are presented in Table 16.

Original: At validation and test time, we resize the images to 256 × 256 and then center crop a patch of 224 × 224 .
Revised: At validation and test time, we resize the images to 256 × 256 and then center crop a patch of 224 × 224 .

Original: For both training and evaluation, we re-normalize the image using the statistic of the training set.g
Revised: For both training and evaluation, we re-normalize the image using the statistic of the training set.g

Original: B.
Revised: B.

Original: L INEAR EVALUATION
Revised: L INEAR EVALUATION

Original: We follow the evaluation protocol from (Chen et al., 2020b).
Revised: We follow the evaluation protocol from (Chen et al., 2020b).

Original: The linear evaluation is done by training a linear classiﬁer on the frozen representation of the ImageNet training samples.
Revised: The linear evaluation is done by training a linear classiﬁer on the frozen representation of the ImageNet training samples.

Original: We train a linear classiﬁer with a cross-entropy objective for 100 epochs using SGD with nesterov, a momentum of 0 .
Revised: We train a linear classiﬁer with a cross-entropy objective for 100 epochs using SGD with nesterov, a momentum of 0 .

Original: 9 and a batch size of 256.
Revised: 9 and a batch size of 256.

Original: We perform learning rate scheduling at epoch 60 and epoch 80 where we divide the learning rate by a factor of 10 .
Revised: We perform learning rate scheduling at epoch 60 and epoch 80 where we divide the learning rate by a factor of 10 .

Original: During training, we apply random resized crop to 224 ×pixels and random horizontal ﬂip.
Revised: During training, we apply random resized crop to 224 ×pixels and random horizontal ﬂip.

Original: We sweep over a set of 4 learning rates: { 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 } ,l 1 weight decays: { 0 , 1 e − 6 , 1 e − 5 } and 3 τ d for SEM: { 0 . 01 , 0 . 1 , 1 } , using a validation set ofimages per class and re-traing using the full training set.
Revised: We sweep over a set of 4 learning rates: { 0 . 5 , 0 . 1 , 0 . 05 , 0 . 01 } ,l 1 weight decays: { 0 , 1 e − 6 , 1 e − 5 } and 3 τ d for SEM: { 0 . 01 , 0 . 1 , 1 } , using a validation set ofimages per class and re-traing using the full training set.

Original: We report the results on the test set.
Revised: We report the results on the test set.

Original: B. R OBUSTNESS EXPERIMENTS
Revised: B. R OBUSTNESS EXPERIMENTS

Original: We follow the evaluation procedure from (Lee et al., 2021).
Revised: We follow the evaluation procedure from (Lee et al., 2021).

Original: We treated the robustness datasets as additional "test sets" in that we simply evaluated them using the evaluation procedure described above.
Revised: We treated the robustness datasets as additional "test sets" in that we simply evaluated them using the evaluation procedure described above.

Original: The images were resized to a 256 × 256 before being center cropped to a 224 × 224 image.
Revised: The images were resized to a 256 × 256 before being center cropped to a 224 × 224 image.

Original: The evaluation procedure was performed using the public robustness benchmark evaluation code of (Djolonga et al., 2020) 3 .
Revised: The evaluation procedure was performed using the public robustness benchmark evaluation code of (Djolonga et al., 2020) 3 .

Original: B. T RANSFER LEARNING LINEAR PROBE
Revised: B. T RANSFER LEARNING LINEAR PROBE

Original: We follow the linear evaluation protocol of (Kolesnikov et al., 2019; Chen et al., 2020b)
Revised: We follow the linear evaluation protocol of (Kolesnikov et al., 2019; Chen et al., 2020b)

Original: We train a linear classiﬁer using a regularized multinomial logistic regression from the scikit-learn package (Pedregosa et al., 2011).
Revised: We train a linear classiﬁer using a regularized multinomial logistic regression from the scikit-learn package (Pedregosa et al., 2011).

Original: The representation is frozen, so that we do not train the encoder backbone nor the batch-normalization statistics.
Revised: The representation is frozen, so that we do not train the encoder backbone nor the batch-normalization statistics.

Original: We do not perform any augmentations and the images are resized to 224 pixels using bicubic resampling and the normalized using the statistics on ImageNet’s training set.
Revised: We do not perform any augmentations and the images are resized to 224 pixels using bicubic resampling and the normalized using the statistics on ImageNet’s training set.

Original: We tune the regularizer term from a range of 45 logarithmically-spaced values between 10 − 6 and 10 5 using a small validation set and re-train using the full training set.
Revised: We tune the regularizer term from a range of 45 logarithmically-spaced values between 10 − 6 and 10 5 using a small validation set and re-train using the full training set.

Original: For SEM, we set τ d = 0 for all experiments.
Revised: For SEM, we set τ d = 0 for all experiments.

Original: B. T RANSFER LEARNING FINE - TUNING
Revised: B. T RANSFER LEARNING FINE - TUNING

Original: We follow the same ﬁne-tuning protocol of (Chen et al., 2020b; Grill et al., 2020).
Revised: We follow the same ﬁne-tuning protocol of (Chen et al., 2020b; Grill et al., 2020).

Original: We initialize the encoder with the pre-trained model and a classiﬁer head with random initialization.
Revised: We initialize the encoder with the pre-trained model and a classiﬁer head with random initialization.

Original: We train for 20,000 steps with a batch size of 256 using SGD with a Nesterov momentum of 0 .
Revised: We train for 20,000 steps with a batch size of 256 using SGD with a Nesterov momentum of 0 .

Original: We set the momentum parameter for the batch normalization to be max(1 − 10 /s, 0 . 9) where s is the number of steps per epoch.
Revised: We set the momentum parameter for the batch normalization to be max(1 − 10 /s, 0 . 9) where s is the number of steps per epoch.

Original: During pre-training, we use random resize to 224 × 224 pixels and random horizontal ﬂipping.
Revised: During pre-training, we use random resize to 224 × 224 pixels and random horizontal ﬂipping.

Original: At test time, we resize the images along the shortest size to 256 pixels using cubic resampling following by a center resize to 224 × 224 pixels.
Revised: At test time, we resize the images along the shortest size to 256 pixels using cubic resampling following by a center resize to 224 × 224 pixels.

Original: Due to computational constraint, we only tune the learning rate using a search of 7 values spaces on logarithmic scales between 0 .and 0 . 1 .
Revised: Due to computational constraint, we only tune the learning rate using a search of 7 values spaces on logarithmic scales between 0 .and 0 . 1 .

Original: For SEM, we set τ d = 1 .
Revised: For SEM, we set τ d = 1 .

Original: for all experiments After choosing the best learning rate of a validation set, we re-run the models using the full training set and evaluate it on the test set, which we use to report the numbers.
Revised: for all experiments After choosing the best learning rate of a validation set, we re-run the models using the full training set and evaluate it on the test set, which we use to report the numbers.

Original: B. S EMI - SUPERVISED LEARNING
Revised: B. S EMI - SUPERVISED LEARNING

Original: We follow the semi-supervised learning protocol of (Chen et al., 2020b; Grill et al., 2020).
Revised: We follow the semi-supervised learning protocol of (Chen et al., 2020b; Grill et al., 2020).

Original: We initialize the network using the pre-trained representation and initialize a classiﬁcation head using random initialization.
Revised: We initialize the network using the pre-trained representation and initialize a classiﬁcation head using random initialization.

Original: We ﬁne-tune the encoder while training the classiﬁcation head using a small subset of ImageNet.
Revised: We ﬁne-tune the encoder while training the classiﬁcation head using a small subset of ImageNet.

Original: We choose the same subset used in prior works which is deﬁned in the
Revised: We choose the same subset used in prior works which is deﬁned in the

Original: TensorFlow-Dataset software.
Revised: TensorFlow-Dataset software.

Original: During training, we random resize the images to 224 × 224 pixels along the shorter size using bicubic resampling followed by a center crop and random horizontal ﬂipping.
Revised: During training, we random resize the images to 224 × 224 pixels along the shorter size using bicubic resampling followed by a center crop and random horizontal ﬂipping.

Original: At test time, we resize the image to 224 × 224 .
Revised: At test time, we resize the image to 224 × 224 .

Original: We optimize the cross entropy loss with nestorov and a momentum of 0 .
Revised: We optimize the cross entropy loss with nestorov and a momentum of 0 .

Original: 9 using batch sizes of 224 .
Revised: 9 using batch sizes of 224 .

Original: We train models for { 30 , 50 } and take the best performing on the validation set.
Revised: We train models for { 30 , 50 } and take the best performing on the validation set.

Original: The learning rate used is chosen among a set of 5 learning rates: { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 005 } .
Revised: The learning rate used is chosen among a set of 5 learning rates: { 0 . 01 , 0 . 02 , 0 . 05 , 0 . 1 , 0 . 005 } .

Original: For SEM, we also search τ d ∈ { 0 . 01 , 0 .
Revised: For SEM, we also search τ d ∈ { 0 . 01 , 0 .

Original: We perform the search on the best performing one on the validation set and the number are returned are obtained using the test set after re-training using the full training set.
Revised: We perform the search on the best performing one on the validation set and the number are returned are obtained using the test set after re-training using the full training set.

Original: C H YPERPARAMETERS
Revised: C H YPERPARAMETERS

Original: The implementation of the SSL methods used in this work are taken from Solo-Learn (da Costa et al., 2021) to which we added the SEM module.
Revised: The implementation of the SSL methods used in this work are taken from Solo-Learn (da Costa et al., 2021) to which we added the SEM module.

Original: The pre-training hyper-parameters of every SSL methods trained on CIFAR-100 with ResNet-18 used in this work are the default provided in the companion repository of Solo-Learn.
Revised: The pre-training hyper-parameters of every SSL methods trained on CIFAR-100 with ResNet-18 used in this work are the default provided in the companion repository of Solo-Learn.

Original: The hyper-parameters are also provided in the launch scripts accompanying this work.
Revised: The hyper-parameters are also provided in the launch scripts accompanying this work.

Original: Due to the large number of SSL methods probed in this work and the amount of space it would require to exhaustively detail all of the hyper-parameters, we refer the reader to the code.
Revised: Due to the large number of SSL methods probed in this work and the amount of space it would require to exhaustively detail all of the hyper-parameters, we refer the reader to the code.

Original: For the CIFAR-100 results obtained with BYOL and a ResNet-50, we have slightly modiﬁed the default parameters.
Revised: For the CIFAR-100 results obtained with BYOL and a ResNet-50, we have slightly modiﬁed the default parameters.

Original: Otherwise, the baseline BYOL model would not obtain competitive results.
Revised: Otherwise, the baseline BYOL model would not obtain competitive results.

Original: The hyper-parameters were tuned using the BYOL baseline and the SEM module was not considered in the selection of the SSL hyper-parameters.
Revised: The hyper-parameters were tuned using the BYOL baseline and the SEM module was not considered in the selection of the SSL hyper-parameters.

Original: The BYOL hyper-parameters are presented in the launch script accompanying this work and presented below for completeness.
Revised: The BYOL hyper-parameters are presented in the launch script accompanying this work and presented below for completeness.

Original: For the ImageNet experiments, we took the hyper-parameters proposed in the launch scripts of
Revised: For the ImageNet experiments, we took the hyper-parameters proposed in the launch scripts of

Original: Solo-Learn to which we only modiﬁed the amount of epochs (100 epochs to 200 epochs.)
Revised: Solo-Learn to which we only modiﬁed the amount of epochs (100 epochs to 200 epochs.)

Original: Here, we present all of the SEM hyper-parameters used in every experiments.
Revised: Here, we present all of the SEM hyper-parameters used in every experiments.

Original: These hyper-parameters can also be found in the launch scripts accompanying this work.
Revised: These hyper-parameters can also be found in the launch scripts accompanying this work.

Original: We present the hype-parameters used to train for BYOL+SEM and MoCo+SEM on CIFAR100.
Revised: We present the hype-parameters used to train for BYOL+SEM and MoCo+SEM on CIFAR100.

Original: Unless mentioned otherwise, these are the parameters used.
Revised: Unless mentioned otherwise, these are the parameters used.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: C. C OMPUTATIONAL RESOURCES
Revised: C. C OMPUTATIONAL RESOURCES

Original: For all our CIFAR-100 training, we used 1 RTX-8000 per experiment.
Revised: For all our CIFAR-100 training, we used 1 RTX-8000 per experiment.

Original: For our ImageNet experiments, we used parallel training with 2 40GB A100 for the training with ResNet50 and ResNet50-x2 and40GB A100 for the training with ResNet50-x4.
Revised: For our ImageNet experiments, we used parallel training with 2 40GB A100 for the training with ResNet50 and ResNet50-x2 and40GB A100 for the training with ResNet50-x4.

Original: With this setup, the training takes about a week for the ResNet50 experiments and about 10 days for the ResNet50-x2 and ResNet50-x4 experiments.
Revised: With this setup, the training takes about a week for the ResNet50 experiments and about 10 days for the ResNet50-x2 and ResNet50-x4 experiments.

Original: D A DDITIONAL STUDIES OF SEM
Revised: D A DDITIONAL STUDIES OF SEM

Original: In Section 4.2, we discussed the effect of scaling L and V as well as changing the Softmax temperature during pre-training of the online network and changing the Softmax temperature for the downstream task.
Revised: In Section 4.2, we discussed the effect of scaling L and V as well as changing the Softmax temperature during pre-training of the online network and changing the Softmax temperature for the downstream task.

Original: Here, we propose additional studies of SEM to provide a better mastery of the method.
Revised: Here, we propose additional studies of SEM to provide a better mastery of the method.

Original: We provide a method for reducing the memory overhead of SEM and experiments demonstrating that despite this version still largely outperform the baseline.
Revised: We provide a method for reducing the memory overhead of SEM and experiments demonstrating that despite this version still largely outperform the baseline.

Original: We additionally present the effect of modifying the embedder contributing to the insight on how to get the most out of SEM.
Revised: We additionally present the effect of modifying the embedder contributing to the insight on how to get the most out of SEM.

Original: Next, we have discussion with a study of the spectrum of the covariance matrix of the SEM representation and the BYOL representation, showing insight how SEM can particularly improve the training signal
Revised: Next, we have discussion with a study of the spectrum of the covariance matrix of the SEM representation and the BYOL representation, showing insight how SEM can particularly improve the training signal

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Resnet-
Revised: Resnet-

Original: [Table]
Revised: [Table]

Original: BYOL
Revised: BYOL

Original: [Table]
Revised: [Table]

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: M
Revised: M

Original: [Table]
Revised: [Table]

Original: BYOL+SEM 313 .
Revised: BYOL+SEM 313 .

Original: 7 M
Revised: 7 M

Original: [Table] [Table]
Revised: [Table] [Table]

Original: M 13 .
Revised: M 13 .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: e
Revised: e

Original: [Table] [Table]
Revised: [Table] [Table]

Original: BYOL+SEM/ 51 .
Revised: BYOL+SEM/ 51 .

Original: 9 M
Revised: 9 M

Original: [Table]
Revised: [Table]

Original: 796 M
Revised: 796 M

Original: [Table] [Table]
Revised: [Table] [Table]

Original: e73 .
Revised: e73 .

Original: [Table]
Revised: [Table]

Original: Resnet-50 :
Revised: Resnet-50 :

Original: BYOL 35 M 4 .
Revised: BYOL 35 M 4 .

Original: M 11 .1 .
Revised: M 11 .1 .

Original: 65 e74 .
Revised: 65 e74 .

Original: BYOL+SEM 425 .
Revised: BYOL+SEM 425 .

Original: 6 M 4 .
Revised: 6 M 4 .

Original: 12 M 21 .2 .
Revised: 12 M 21 .2 .

Original: 04 e77 .
Revised: 04 e77 .

Original: BYOL+SEM/ 76 .
Revised: BYOL+SEM/ 76 .

Original: 7 M 4 .
Revised: 7 M 4 .

Original: 12 M 11 .1 .
Revised: 12 M 11 .1 .

Original: 69 e76 .during pre-training.
Revised: 69 e76 .during pre-training.

Original: We provide a scaling analysis of BYOL and BYOL + SEM on CIFAR-100.
Revised: We provide a scaling analysis of BYOL and BYOL + SEM on CIFAR-100.

Original: We end with an experiment showing that pre-training with SEM is necessary to get the best performance.
Revised: We end with an experiment showing that pre-training with SEM is necessary to get the best performance.

Original: D. A N EFFICIENT VARIANT OF SEM
Revised: D. A N EFFICIENT VARIANT OF SEM

Original: A large over-complete representation may induce a signiﬁcant memory footprint due to the additional parameters of the fully connected linear layer used to map to and from the representation.
Revised: A large over-complete representation may induce a signiﬁcant memory footprint due to the additional parameters of the fully connected linear layer used to map to and from the representation.

Original: For SEM we require two such mappings as depicted in Figure 2c for BYOL.
Revised: For SEM we require two such mappings as depicted in Figure 2c for BYOL.

Original: To reduce the amount of parameters, we propose to sparsify the weight matrix of the fully connected linear layer.
Revised: To reduce the amount of parameters, we propose to sparsify the weight matrix of the fully connected linear layer.

Original: We propose to do so by taking the block diagonal of the parameters of the matrix multiplication and setting the parameters outside the block diagonal to 0 .
Revised: We propose to do so by taking the block diagonal of the parameters of the matrix multiplication and setting the parameters outside the block diagonal to 0 .

Original: Formally, let v ∈ R b × m , w ∈ R m × o and y = v · w be the fully connected matrix multiplication.
Revised: Formally, let v ∈ R b × m , w ∈ R m × o and y = v · w be the fully connected matrix multiplication.

Original: Instead, we partition v into n blocks with v i ∈ R b
Revised: Instead, we partition v into n blocks with v i ∈ R b

Original: × mn and deﬁne
Revised: × mn and deﬁne

Original: n smaller w i ∈
Revised: n smaller w i ∈

Original: R mn × on , where i ∈
Revised: R mn × on , where i ∈

Original: [ L ] is the i th block.
Revised: [ L ] is the i th block.

Original: Then, we perform a batch matrix multiplication of v i and w i that we concatenate as follows:
Revised: Then, we perform a batch matrix multiplication of v i and w i that we concatenate as follows:

Original: y i = v i · w i and ¯ y i
Revised: y i = v i · w i and ¯ y i

Original: =
Revised: =

Original: Concat ([ y 1 , . . . , y n ]) .
Revised: Concat ([ y 1 , . . . , y n ]) .

Original: Thus, the amount of parameters of this matrix multiplication scales in O ( m · on ) , allowing us to reduce the memory consumption by increasing n , the number of blocks.
Revised: Thus, the amount of parameters of this matrix multiplication scales in O ( m · on ) , allowing us to reduce the memory consumption by increasing n , the number of blocks.

Original: We perform an experiment where we partition the embedder and the ﬁrst linear layer of the projector into 8 blocks.
Revised: We perform an experiment where we partition the embedder and the ﬁrst linear layer of the projector into 8 blocks.

Original: We present the results in Table 17 in which we compare the $ of parameters, the
Revised: We present the results in Table 17 in which we compare the # of parameters, the

Original: # of activations, the allocated vRAM by pytorch, the FLOPs/sample and the accuracy of BYOL,
Revised: # of activations, the allocated vRAM by pytorch, the FLOPs/sample and the accuracy of BYOL,

Original: BYOL+SEM and BYOL+SEM/8 representing the model with 8 blocks obtained following the method described above.
Revised: BYOL+SEM and BYOL+SEM/8 representing the model with 8 blocks obtained following the method described above.

Original: We observe that partitioning the matrix multiplications of SEM allows to vastly reduce the computation parameters while still yielding an important improvement over the baseline.
Revised: We observe that partitioning the matrix multiplications of SEM allows to vastly reduce the computation parameters while still yielding an important improvement over the baseline.

Original: This result demosntrate that SEM can be beneﬁcial while inducing minimal computational overhead.
Revised: This result demosntrate that SEM can be beneﬁcial while inducing minimal computational overhead.

Original: Attentive readers may notice that this performance is better compared to the ablation presented in
Revised: Attentive readers may notice that this performance is better compared to the ablation presented in

Original: Figure 3.
Revised: Figure 3.

Original: The difference in performance is due to probing the embedder’s output (i.e. z θ ) in Figureand probing the encoder’s output (i.e. e θ ) in Table 17.
Revised: The difference in performance is due to probing the embedder’s output (i.e. z θ ) in Figureand probing the encoder’s output (i.e. e θ ) in Table 17.

Original: Using the each ablation’s representation for probing to the other recovers the performance observed by each.
Revised: Using the each ablation’s representation for probing to the other recovers the performance observed by each.

Original: D. A DDITIONAL ABLATION OF THE SEM PARAMETERS
Revised: D. A DDITIONAL ABLATION OF THE SEM PARAMETERS

Original: Ablating the embedder In the main text, we mentioned that we use batch normalization at the output of the embedder.
Revised: Ablating the embedder In the main text, we mentioned that we use batch normalization)) at the output of the embedder.

Original: The reason we use batch normalization is mostly due to the fact that we wanted to avoid tuning any hyper-parameters that were not related to SEM to emphasize its contribution.
Revised: The reason we use batch normalization is mostly due to the fact that we wanted to avoid tuning any hyper-parameters that were not related to SEM to emphasize its contribution.

Original: Using BatchNorm gave the best performance without tuning the hyper-parameters of the baseline models.
Revised: Using BatchNorm gave the best performance without tuning the hyper-parameters of the baseline models.

Original: Here, we want to emphasize that SEM can be used without batch norm, but more hyper-parameters might need to be tuned for it to perform as well as the model with batch norm in the encoder.
Revised: Here, we want to emphasize that SEM can be used without batch norm, but more hyper-parameters might need to be tuned for it to perform as well as the model with batch norm in the encoder.

Original: For example, we found that using no weight decay was important to get better performance when we did not have batch normalization as illustrated in Table table 18.
Revised: For example, we found that using no weight decay was important to get better performance when we did not have batch normalization as illustrated in Table table 18.

Original: We leave the full study of the interaction of SEM with the SSL related parameters for future work.
Revised: We leave the full study of the interaction of SEM with the SSL related parameters for future work.

Original: [Table]
Revised: [Table]

Original: Another decision is to use a linear layer as the embedder.
Revised: Another decision is to use a linear layer as the embedder.

Original: Other alternative may include using the Identidy function (i.e. the output of the encoder is used for SEM).
Revised: Other alternative may include using the Identidy function (i.e. the output of the encoder is used for SEM).

Original: However, if we want to systematically use the same encoder as the SSL model, then we are constrained to a representation size that is the one of the ResNet encoder (i.e. 512 for a ResNet-18).
Revised: However, if we want to systematically use the same encoder as the SSL model, then we are constrained to a representation size that is the one of the ResNet encoder (i.e. 512 for a ResNet-18).

Original: Finally, we showcase that using a more expressive embedder leads to exacerbated performance and recommend practitioner to limit the expressivity of their embedder.
Revised: Finally, we showcase that using a more expressive embedder leads to exacerbated performance and recommend practitioner to limit the expressivity of their embedder.

Original: [Table]
Revised: [Table]

Revised: A very very large embedding Using a ResNet-18 encoder and the method proposed in Section D.1, we further scale the embedding size of SEM to see where the performance saturates for classiﬁcation.

Revised: In Figure 7 we observe that the performance saturates for L = 10000 for the classiﬁcation task.

Revised: We conjecture that the optimal L might be different for other tasks, but we leave that study for future work.

Original: D. A NALYZE OF THE SPECTRUM OF THE COVARIANCE MATRIX OF THE REPRESENTATION
Revised: D. A NALYZE OF THE SPECTRUM OF THE COVARIANCE MATRIX OF THE REPRESENTATION

Original: To obtain a better insight on why the SEM representation leads to better downstream performance, we analyze the spectrum of the covariance matrix of the representation using the methodology presented  in Jing et al.
Revised: To obtain a better insight on why the SEM representation leads to better downstream performance, we analyze the spectrum of the covariance matrix of the representation using the methodology presented [Figure] in Jing et al.

Original: That is, we collect the embedding vectors of the test set of CIFAR-100 using a pre-trained model using ResNet-50.
Revised: That is, we collect the embedding vectors of the test set of CIFAR-100 using a pre-trained model using ResNet-50.

Original: For BYOL, we have an additional embedder without softmax normalization (as done in Figure 3).
Revised: For BYOL, we have an additional embedder without softmax normalization (as done in Figure 3).

Original: For BYOL and BYOL+SEM we use the embedder’s output ( z θ ) to perform the evaluation.
Revised: For BYOL and BYOL+SEM we use the embedder’s output ( z θ ) to perform the evaluation.

Original: To compute the covariance matrix C ∈ R L · V × L · V of the embedding layer z , we deﬁne ¯ z : = (cid:80) Ni
Revised: To compute the covariance matrix C ∈ R L · V × L · V of the embedding layer z , we deﬁne ¯ z := (cid:80) Ni

Original: =

Revised: =

Original: z i /N
Revised: z i /N

Original: the average representation over the N samples and compute the covariance as follows:
Revised: the average representation over the N samples and compute the covariance as follows:

Original: [Equation]
Revised: [Equation]

Original: To plot the spectrum of the covariance matrix [Figure] we take the singular value decomposition of the matrix ( C = USV (cid:62) ) with S the diagonal of the singular values, which we plot in sorted order and logarithm scale in Figure [Figure]
Revised: To plot the spectrum of the covariance matrix, we take the singularalue decomposition of the matrix ( C = USV (cid:62) ) with S the diagonal of the singular values, which we plot in sorted order and logarithm scale in Figure 8.

Original: This experiment demonstrates that the softmax normalization counters the dimensionality collapse that was discussed in Jing et al.
Revised: This experiment demonstrates that the softmax normalization counters the dimensionality collapse that was discussed in Jing et al.

Original: Interestingly, the drop observed with SEM with L ≥occurs at the index 2048 which is the dimensionality output of the ResNet-50 encoder.
Revised: Interestingly, the drop observed with SEM with L ≥occurs at the index 2048 which is the dimensionality output of the ResNet-50 encoder.

Original: [Figure]
Revised: [Figure]

Revised: and BYOL + SEM obtained with a ResNet-50 encoder.

Original: D. S CALING THE R ES N ET ENCODER FOR CIFAR- 
Revised: D. S CALING THE R ES N ET ENCODER FOR CIFAR-

Original: [Figure] We perform a scaling experiment on CIFAR-100 where we compare the scaling behaviour of BYOL and BYOL + SEM.
Revised: We perform a scaling experiment on CIFAR-100 where we compare the scaling behaviour of BYOL and BYOL + SEM.

Original: We evaluate the computational cost of the methods and the resulting downstream accuracy for a range of four resnets: ResNet-18, ResNet-50, ResNet-50 x2 and ResNet-50 x4.
Revised: We evaluate the computational cost of the methods and the resulting downstream accuracy for a range of four resnets: ResNet-18, ResNet-50, ResNet-50 x2 and ResNet-50 x4.

Original: In Figure 8, we observe that SEM has a better scaling behaviour than the baseline, especially as we increase the width of the ResNet-50.
Revised: In Figure 9, we observe that SEM has a better scaling behaviour than the baseline, especially as we increase the width of the ResNet-50.

Original: For BYOL, we observe that the performance decays for
Revised: For BYOL, we observe that the performance decays for

Revised: [Figure]

Revised: [Table]

Original: ResNet-50 with width x2 and x4.
Revised: ResNet-50 with width x2 and x4.

Original: This is not unprecendented, as prior works as demonstrated other methods where scaling up the capacity of a model led to decrease in performance.
Revised: This is not unprecendented, as prior works as demonstrated other methods where scaling up the capacity of a model led to decrease in performance.

Original: When comparing the discrepancy with Figure 1, we attribute that to the fact that CIFAR-100 is a small dataset.
Revised: When comparing the discrepancy with Figure 1, we attribute that to the fact that CIFAR-100 is a small dataset.

Original: In fact, we observe that the training accuracy stays constant to about 79% for all the
Revised: In fact, we observe that the training accuracy stays constant to about 79% for all the

Original: ResNet-50 scales demonstrating overﬁtting for the baseline BYOL.
Revised: ResNet-50 scales demonstrating overﬁtting for the baseline BYOL.

Original: Nevertheless, SEM prevents the decrease in performance and even lead to further improved performance as we increase the scale of the ResNet-50.[Table]
Revised: Nevertheless, SEM prevents the decrease in performance and even lead to further improved performance as we increase the scale of the ResNet-50.

Original: D. T HE ROLE OF PRE - TRAINING WITH SEM
Revised: D. T HE ROLE OF PRE - TRAINING WITH SEM

Original: Here, we present the downstream accuracy obtained if one take a model pre-trained without SEM and add SEM normalization only for  downstream classiﬁcation.
Revised: We probe the downstream accuracy obtained of a model pre-trained without SEM and add SEM normalization only for the downstream classiﬁcation.

Original: For this experiment, we take a pre-trained model with embedder (i.e. BYOL + embed) with L = 5000 and V = 13 and add the softmax normalization  for  downstream classiﬁcation.
Revised: For this experiment, we take a pre-trained model with embedder (i.e. BYOL + embed) with L = 5000 and V = 13 and add the softmax normalization only for the downstream classiﬁcation.

Original: We  observe that such approach leads to an imprtant reduction in downstream accuracy in comparison to the model  with  SEM pre-training.
Revised: We do not use SEM during pre-training. We observe that using SEM for downstream classiﬁcation leads to an improvement even when the model is not pre-trained with SEM, demonstrating the utility of SEM downstream classiﬁcation. However, we note that the performance of the model pre-trained without SEM is much weaker and thus demonstrates the imprtance of also pre-training using SEM. E CIFAR-10 RESULTS

Revised: We conﬁrm that our method also yield improvement on simpler datasets such as CIFAR-10.

Revised: Here, we compare BYOL and BYOL + SEM on a ResNet-50 and observe and improvement of 1.6%.

Revised: [Table]

Original: E CIFAR100 SUPERCLASS
Revised: F CIFAR100 SUPERCLASS

Original: The 100 classes of CIFAR-100 (Krizhevsky, 2009) are grouped into 20 superclasses.
Revised: The 100 classes of CIFAR-100 (Krizhevsky, 2009) are grouped into 20 superclasses.

Original: The list of superclass for each class in Table
Revised: The list of superclass for each class in Table

Original: [Table]
Revised: [Table]

Original: vehicles
Revised: vehicles

Original: [Table]
Revised: [Table]

Original: vehicles
Revised: vehicles

Original: [Table]
Revised: [Table]

Original: F A DDITIONAL CIFAR-100 COHERENCE GRAPHS
Revised: G A DDITIONAL CIFAR-100 COHERENCE GRAPHS

Original: S otter kangaroobottle S lamp S beartank
Revised: S otter kangaroobottle S lamp S beartank

Original: S streetcarroad S249S train
Revised: S streetcarroad S249S train

Original: S lobster S boygirlhousewoman S29S cockroach S apple dolphin seal whale aquarium_fish flatfish ray shark orchid poppyrose sunflower tulip bowl can cup plate mushroom orange pear sweet_pepper clock computer_keyboard telephone television bed chair couch table wardrobe bee beetle butterfly caterpillar leopard tiger wolf bridge castle skyscraper cloud forest mountain plain sea camelcattle chimpanzee fox possum raccoon skunk crab snail worm baby man lizard snake turtle hamster mouse rabbit shrew squirrel maple_tree oak_tree palm_tree willow_tree bicycle bus motorcycle
Revised: S lobster S boygirlhousewoman S29S cockroach S apple dolphin seal whale aquarium_fish flatfish ray shark orchid poppyrose sunflower tulip bowl can cup plate mushroom orange pear sweet_pepper clock computer_keyboard telephone television bed chair couch table wardrobe bee beetle butterfly caterpillar leopard tiger wolf bridge castle skyscraper cloud forest mountain plain sea camelcattle chimpanzee fox possum raccoon skunk crab snail worm baby man lizard snake turtle hamster mouse rabbit shrew squirrel maple_tree oak_tree palm_tree willow_tree bicycle bus motorcycle

Original: pickup_truck lawn_mower rocket tractor
Revised: pickup_truck lawn_mower rocket tractor

Original: S55S
Revised: S55S

Original: S47S
Revised: S47S

Original: S
Revised: S

Original: S69S134S
Revised: S69S134S

Original: S
Revised: S

Original: S S
Revised: S S

Original: S S
Revised: S S

Original: S361S
Revised: S361S

Original: S
Revised: S

Original: S12S
Revised: S12S

Original: S S
Revised: S S

Original: S
Revised: S

Original: S190 S
Revised: S190 S

Original: S
Revised: S

Original: S183 S
Revised: S183 S

Original: S
Revised: S

Original: S S
Revised: S S

Original: S S
Revised: S S

Original: S
Revised: S

Original: S127 S
Revised: S127 S

Original: S111S
Revised: S111S

Original: S
Revised: S

Original: S
Revised: S

Original: S30S
Revised: S30S

Original: S
Revised: S

Original: S193 S
Revised: S193 S

Original: S
Revised: S

Original: S128S
Revised: S128S

Original: S
Revised: S

Original: S S
Revised: S S

Original: S S
Revised: S S

Original: S
Revised: S

Original: S298S260S
Revised: S298S260S

Original: S68S204S161S
Revised: S68S204S161S

Original: S177S
Revised: S177S

Original: S
Revised: S

Original: S227S
Revised: S227S

Original: S318S
Revised: S318S

Original: S
Revised: S

Original: S
Revised: S

Original: S126S
Revised: S126S

Original: S
Revised: S

Original: S S
Revised: S S

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]