Original: DINO AS A VON M ISES -F ISHER MIXTURE MODEL
Revised: DINO AS A VON M ISES -F ISHER MIXTURE MODEL

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-distillation methods using Siamese networks are popular for self-supervised pre-training.
Revised: Self-distillation methods using Siamese networks are popular for self-supervised pre-training.

Original: DINO is one such method based on a cross-entropy loss between
Revised: DINO is one such method based on a cross-entropy loss between

Original: K -dimensional probability vectors, obtained by applying a softmax function to the dot product between representations and learnt prototypes.
Revised: K -dimensional probability vectors, obtained by applying a softmax function to the dot product between representations and learnt prototypes.

Original: Given the fact that the learned representations are L 2 -normalized, we show that DINO  can be interpreted as a mixture model of von Mises-Fisher components.
Revised: Given the fact that the learned representations are L 2 -normalized, we show that DINO and its derivatives, such as iBOT, can be interpreted as a mixture model of von Mises-Fisher components.

Original: With this interpretation, DINO assumes equal precision for all components when the prototypes are also L 2 -normalized.
Revised: With this interpretation, DINO assumes equal precision for all components when the prototypes are also L 2 -normalized.

Original: Using this insight we propose DINO-vMF, that adds appropriate normalization constants when computing the cluster assignment probabilities.
Revised: Using this insight we propose DINOvMF, that adds appropriate normalization constants when computing the cluster assignment probabilities.

Original: Unlike DINO, DINO-vMF is stable also for the larger ViT-Base model with unnormalized prototypes.
Revised: Unlike DINO, DINO-vMF is stable also for the larger ViT-Base model with unnormalized prototypes.

Original: We show that the added flexibility of the mixture model is beneficial in terms of better image representations.
Revised: We show that the added flexibility of the mixture model is beneficial in terms of better image representations.

Original: The DINO-vMF pre-trained model consistently performs better than DINO on a range of downstream tasks.
Revised: The DINO-vMF pre-trained model consistently performs better than DINO on a range of downstream tasks.

Revised: We obtain similar improvements for iBOT-vMF vs iBOT and thereby show the relevance of our proposed modification also for other methods derived from DINO.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Self-supervised learning (SSL) is an effective approach for pre-training models on large unlabeled datasets.
Revised: Self-supervised learning (SSL) is an effective approach for pre-training models on large unlabeled datasets.

Original: The main objective of SSL pre-training is to learn representations that are transferable to a range of, so called, downstream tasks.
Revised: The main objective of SSL pre-training is to learn representations that are transferable to a range of, so called, downstream tasks.

Original: Early SSL methods achieved this through handcrafted pretext tasks that act as inductive biases in the representation learning process (Komodakis &
Revised: Early SSL methods achieved this through handcrafted pretext tasks that act as inductive biases in the representation learning process (Komodakis &

Original: Gidaris, 2018; Noroozi & Favaro, 2016; Kim et al., 2018; Doersch et al., 2015; Larsson et al., 2016;
Revised: Gidaris, 2018; Noroozi & Favaro, 2016; Kim et al., 2018; Doersch et al., 2015; Larsson et al., 2016;

Original: Zhang et al., 2016).
Revised: Zhang et al., 2016).

Original: Contrastive methods (Tian et al., 2020; Wu et al., 2018), using supervisory signals in the form of augmentation invariances and instance discrimination, have produced strong performance benchmarks.
Revised: Contrastive methods (Tian et al., 2020; Wu et al., 2018), using supervisory signals in the form of augmentation invariances and instance discrimination, have produced strong performance benchmarks.

Original: In practice, contrastive methods require large batch sizes (Chen et al., 2020) or specialized techniques like memory banks (He et al., 2020; Misra & Maaten, 2020) to achieve the best performance.
Revised: In practice, contrastive methods require large batch sizes (Chen et al., 2020) or specialized techniques like memory banks (He et al., 2020; Misra & Maaten, 2020) to achieve the best performance.

Original: Self-distillation methods based on the Mean Teacher (Tarvainen &
Revised: Self-distillation methods based on the Mean Teacher (Tarvainen &

Original: Valpola, 2017) framework are effective representation learners that do not require such large batch sizes for pre-training.
Revised: Valpola, 2017) framework are effective representation learners that do not require such large batch sizes for pre-training.

Original: A trivial solution where the network learns to output the same representation irrespective of the input is known as representation collapse.
Revised: A trivial solution where the network learns to output the same representation irrespective of the input is known as representation collapse.

Original: The negative samples in contrastive learning prevent representation collapse.
Revised: The negative samples in contrastive learning prevent representation collapse.

Original: In the absence of negative samples, self-distillation methods use explicit approaches to avoid collapse, such as asymmetric model architecture (Grill et al., 2020;
Revised: In the absence of negative samples, self-distillation methods use explicit approaches to avoid collapse, such as asymmetric model architecture (Grill et al., 2020;

Original: Chen & He, 2021) and whitening (Ermolov et al., 2021; Zbontar et al., 2021).
Revised: Chen & He, 2021) and whitening (Ermolov et al., 2021; Zbontar et al., 2021).

Original: Transformers (Vaswani et al., 2017), originally introduced in NLP, have emerged as a strong model architecture for vision tasks as well (Dosovitskiy et al., 2020; Liu et al., 2021).
Revised: Transformers (Vaswani et al., 2017), originally introduced in NLP, have emerged as a strong model architecture for vision tasks as well (Dosovitskiy et al., 2020; Liu et al., 2021).

Original: Current state-of-the-art
Revised: Current state-of-the-art

Original: SSL methods leverage the highly flexible Vision Transformers (ViTs) (Caron et al., 2021; Bao et al., 2021;
Revised: SSL methods leverage the highly flexible Vision Transformers (ViTs) (Caron et al., 2021; Bao et al., 2021;

Original: Li et al., 2021;
Revised: Li et al., 2021;

Original: Zhou et al., 2021; Xie et al., 2021; Chen et al., 2021).
Revised: Zhou et al., 2021; Xie et al., 2021; Chen et al., 2021).

Original: DINO (Caron et al., 2021) is a non-contrastive SSL method that is effective for pre-training ViT models.
Revised: DINO (Caron et al., 2021) is a non-contrastive SSL method that is effective for pre-training ViT models.

Original: Interestingly,
Revised: Interestingly,

Original: ViTs pre-trained using DINO outperformed ResNets
Revised: ViTs pre-trained using DINO outperformed ResNets

Original: (He et al., 2016) by a significant margin at kNN classification based on learned representations.
Revised: (He et al., 2016) by a significant margin at kNN classification based on learned representations.

Original:  MSN (Assran et al., 2022)is a recent method that adapts DINO to produce strong few-shot performance with enhanced training efficiency.
Revised: DINO is an influential SSL method with several state-of-the-art derivatives: MSN (Assran et al., 2022) adapts DINO to produce strong few-shot performance with enhanced training efficiency;

Revised: iBOT

Revised: (Zhou et al., 2021) extends DINO by adding an additional masked image modeling task; EsViT extends DINO by adding patch-level tasks that also use a DINO-like formulation.

Original: Both DINO and MSN are  trained using the Mean Teacher framework by learning to produce consistent outputs in the probability simplex.
Revised: These methods are all trained using the Mean Teacher framework by learning to produce consistent outputs in the probability simplex.

Original: The networks output softmax-logit scores based on an inner product between a learned representation and a setof prototypes.
Revised: The networks output softmax-logit scores based on an inner product between a learned representation and a set

Original: Many recent SSL methods have used a similar formulation but with other modifications (Li et al., 2021;

Original: Zhou et al., 2021). We provide a better understanding of DINO by taking a closer look at its inner-product formulation.
Revised: We provide a better understanding of DINO, and its derivatives, by taking a closer look at its inner-product formulation.

Original: We interpret DINO as a von Mises-Fisher mixture model under certain assumptions.
Revised: We interpret DINO as a von Mises-Fisher mixture model under certain assumptions.

Original: Based on this interpretation, we propose DINO-vMF, as a modified version of DINO, that adds flexibility in the learned latent space while keeping the training stable.
Revised: Based on this interpretation, we propose DINO-vMF, as a modified version of DINO, that adds flexibility in the learned latent space while keeping the training stable.

Original: DINO-vMF pre-training consistently improves performance on similar downstream tasks as DINO.
Revised: DINO-vMF pretraining consistently improves performance on similar downstream tasks as DINO.

Original: We also show that the larger ViT models achieve significantly improved few-shot classification performance with our pre-training.
Revised: We also show that the larger ViT models achieve significantly improved few-shot classification performance with our pre-training.

Revised: By incorporating our vMF modification in iBOT, we achieve significantly improved performance that suggests that our method is applicable to DINO-derived methods as well.

Original:  DINO
Revised:  DINO

Original: The self-distillation learning framework in DINO considers a teacher network g θ t and a student network g θ s , with parameters θ t and θ s , respectively.
Revised: The self-distillation learning framework in DINO considers a teacher network g θ t and a student network g θ s , with parameters θ t and θ s , respectively.

Original: In DINO, the student network is formulated to predict a vector in the ( K − 1)
Revised: In DINO, the student network is formulated to predict a vector in the ( K − 1)

Original: -dimensional probability simplex using a softmax function.
Revised: -dimensional probability simplex using a softmax function.

Original: The student probability distribution is obtained as follows:
Revised: The student probability distribution is obtained as follows:

Original: [Equation]
Revised: [Equation]

Original: where k ∝ indicates that the right-hand side is normalized w.r.t.
Revised: where k ∝ indicates that the right-hand side is normalized w.r.t.

Original: the index k (i.e. the equation above corresponds to a softmax).
Revised: the index k (i.e. the equation above corresponds to a softmax).

Original: The teacher probability distribution P t ( x ) is computed analogously.
Revised: The teacher probability distribution P t ( x ) is computed analogously.

Original: Given an unlabeled image dataset I , consider uniform samples x ∼ I and two random augmentations
Revised: Given an unlabeled image dataset I , consider uniform samples x ∼ I and two random augmentations

Original: A s ∼ A s , A t ∼ A t .
Revised: A s ∼ A s , A t ∼ A t .

Original: By applying these augmentations, we get two views x s =
Revised: By applying these augmentations, we get two views x s =

Original: A s ( x ) and x t =
Revised: A s ( x ) and x t =

Original: A t ( x ) .
Revised: A t ( x ) .

Original: The student network is trained using gradient updates to produce outputs P s ( x s ) that are consistent with those of the teacher network P t ( x t ) by minimizing a cross-entropy loss given by: min θ s (cid:80) Kk =1 − P ( k ) t ( x t ) log
Revised: The student network is trained using gradient updates to produce outputs P s ( x s ) that are consistent with those of the teacher network P t ( x t ) by minimizing a cross-entropy loss given by: min θ s (cid:80) Kk =1 − P ( k ) t ( x t ) log

Original: P ( k ) s ( x s ) .
Revised: P ( k ) s ( x s ) .

Original: The teacher network parameters θ t are only updated as an exponential moving average (EMA) of θ s .
Revised: The teacher network parameters θ t are only updated as an exponential moving average (EMA) of θ s .

Original: SSL methods based on Siamese networks (Bromley et al., 1993) face the representation collapse problem, where the network learns to produce the same output irrespective of the input.
Revised: SSL methods based on Siamese networks (Bromley et al., 1993) face the representation collapse problem, where the network learns to produce the same output irrespective of the input.

Original: One approach to address this is by introducing an asymmetry between the teacher and student networks.
Revised: One approach to address this is by introducing an asymmetry between the teacher and student networks.

Original: DINO uses the same model architecture for the student and teacher networks but instead shows that adding asymmetry through centering and sharpening operations is sufficient to avoid collapse.
Revised: DINO uses the same model architecture for the student and teacher networks but instead shows that adding asymmetry through centering and sharpening operations is sufficient to avoid collapse.

Original: The targets produced by the teacher network are centered to remove bias towards a cluster and sharpened using a temperature τ t as g θ t ( x t ) ← ( g θ t ( x t ) − c ) /τ t .
Revised: The targets produced by the teacher network are centered to remove bias towards a cluster and sharpened using a temperature τ t as g θ t ( x t ) ← ( g θ t ( x t ) − c ) /τ t .

Original: On the other hand, the student outputs are only sharpened as g θ s ( x s ) ← g θ s ( x s ) /τ s .
Revised: On the other hand, the student outputs are only sharpened as g θ s ( x s ) ← g θ s ( x s ) /τ s .

Original: The centering operation prevents one of the probability components from dominating but the solution could collapse to a uniform distribution instead.
Revised: The centering operation prevents one of the probability components from dominating but the solution could collapse to a uniform distribution instead.

Original: This is avoided by the sharpening operation, where the teacher uses a lower temperature value than the student, τ t < τ s = 0 .
Revised: This is avoided by the sharpening operation, where the teacher uses a lower temperature value than the student, τ t < τ s = 0 .

Original: The overall schematic of DINO is illustrated in Figure 1a.
Revised: The overall schematic of DINO is illustrated in Figure 1a.

Original:  M ETHOD 3. DINO: A CLOSER LOOK AT THE FINAL LAYER
Revised:  M ETHOD 3. DINO: A CLOSER LOOK AT THE FINAL LAYER

Original: The teacher and student networks are designed by combining a backbone network that can be transferred to downstream tasks and a prediction head that is specific to the SSL pre-training.
Revised: The teacher and student networks are designed by combining a backbone network that can be transferred to downstream tasks and a prediction head that is specific to the SSL pre-training.

Original: We take a closer look at the prediction head as shown in Figure 1b, which is found to be important to achieve good performance in ablation experiments (Caron et al., 2021).
Revised: We take a closer look at the prediction head as shown in Figure 1b, which is found to be important to achieve good performance in ablation experiments (Caron et al., 2021).

Original: The weight-normalization in the last linear layer (Salimans & Kingma, 2016) refers to a reparameterization of the weights W as 
Revised: The weight-normalization in the last linear layer (Salimans & Kingma, 2016) refers to a reparameterization of the weights W as w ( k )

Original: W ( k ) =
Revised: =

Original: g ( k ) v ( k ) where W ( k ) is the k th column of W , g ( k ) is a scalar magnitude, and v ( k ) is a unit vector.
Revised: g ( k ) v ( k ) where w ( k ) is the k th column of W , g ( k ) is a scalar magnitude, and v ( k ) is a unit vector.

Original: Optionally, the weights are L 2 -normalized by fixing g = 1 .
Revised: Optionally, the weights are L 2 -normalized by fixing g = 1 .

Original: Whether or not to L 2 -normalize the prototypes is a subtle, and in our opinion ad hoc , design choice which is not discussed enough in the literature.
Revised: Whether or not to L 2 -normalize the prototypes is a subtle, and in our opinion ad hoc , design choice which is not discussed enough in the literature.

Original: Most prior clustering-based SSL methods do use normalized prototypes (Caron et al., 2018; 2019; 2020;
Revised: Most prior clustering-based SSL methods do use normalized prototypes (Caron et al., 2018; 2019; 2020;

Original: Li et al., 2020; Asano et al., 2019), and this is not mentioned as a design choice in the paper by Caron et al.
Revised: Li et al., 2020; Asano et al., 2019), and this is not mentioned as a design choice in the paper by Caron et al.

Original: (2021), nor in their supplementary material.
Revised: (2021), nor in their supplementary material.

Original: However, from the public code repository of DINO 1 , it can be noted that not L 2 -normalizing 
Revised: However, from the public code repository of DINO 1 , it can be noted that not L 2 -normalizing [Figure]

Revised: The prediction head contains 3 MLP layers, an L 2 -normalization bottleneck and a weight-normalized

Revised: Salimans & Kingma

Revised: 2016) linear layer.

Revised: The weights of the weight-normalized linear layer are

Original:  W leads to a performance boost for ViT-Small models (these are the results reported in their paper).
Revised: L 2 -normalized in larger Base to W leads to a performance boost for ViT-Small models (these are the results reported in their paper).

Original: However, the larger ViT-Base models require the weights to be L 2 -normalized since the training is unstable otherwise.
Revised: However, the larger ViT-Base models require the weights to be L 2 -normalized since the training is unstable otherwise.

Original: Furthermore, comparing the ViT-Small and ViT-Base models at patch size 8, surprisingly the ViT-Small model achieves better kNN top-1 validation accuracy on ImageNet (Caron et al., 2021, Table 2).This suggests that L 2 -normalization is an important design choice and that the [Figure] The prediction head contains 3 MLP layers
Revised: Furthermore, comparing the ViT-Small and ViT-Base models at patch size 8, surprisingly the ViT-Small model achieves better kNN top-1 validation accuracy on ImageNet (Caron et al., 2021, Table 2).

Original:  L  -normalization bottleneck and weight- The weights theweight-normalized representations learned by the ViT-Base model (i.e. with L 2 -normalization) are sub-optimal and has room to improve.
Revised: This suggests that L 2 -normalization is an important design choice and that the representations learned by the ViT-Base model (i.e. with L 2 -normalization) are sub-optimal and has room to improve.

Original: In the next section we provide a better understanding of this from a mixture model perspective and investigate how we can better leverage prototypes that are not L 2 -normalized.
Revised: In the next section we provide a better understanding of this from a mixture model perspective and investigate how we can better leverage prototypes that are not L 2 -normalized.

Original: 3. DINO AS A M IXTURE M ODEL
Revised: 3. DINO AS A M IXTURE M ODEL

Original: DINO learns to predict a soft distribution over K components.
Revised: DINO learns to predict a soft distribution over K components.

Original: Let the intermediate output in the prediction head after the L 2 -normalization of the representation 2 be denoted as y ; see Figure 1b.
Revised: Let the intermediate output in the prediction head after the L 2 -normalization of the representation 2 be denoted as y ; see Figure 1b.

Original: Then, DINO can be interpreted as performing clustering in the latent space of y .
Revised: Then, DINO can be interpreted as performing clustering in the latent space of y .

Original: Specifically, since y is a unit vector, DINO performs clustering on the unit hypersphere.
Revised: Specifically, since y is a unit vector, DINO performs clustering on the unit hypersphere.

Original: Clustering is closely related to mixture modeling.
Revised: Clustering is closely related to mixture modeling.

Original: On the unit hypersphere, a mixture model can be defined using von Mises-Fisher (vMF) components.
Revised: On the unit hypersphere, a mixture model can be defined using von Mises-Fisher (vMF) components.

Original: For a random p -dimensional unit vector y , the vMF probability density function is given by f ( y ; µ, κ ) =
Revised: For a random p -dimensional unit vector y , the vMF probability density function is given by f ( y ; µ , κ ) =

Original: C p ( κ ) exp( κµ T y ) , where µ is a mean vector with | µ | = 1 , κ is a scalar concentration parameter that measures isotropic precision, and
Revised: C p ( κ ) exp( κ µ T y ) , where µ is a mean vector with ∥ µ ∥ = 1 , κ is a scalar concentration parameter that measures isotropic precision, and

Original: C p ( κ ) is a normalizing constant defined as: C p ( κ ) =
Revised: C p ( κ ) is a normalizing constant defined as: C p ( κ ) =

Original: κ p/ 2 − 1 / (cid:2) (2 π ) p/ 2 I p/ 2 − 1 ( κ ) (cid:3) , where I ν denotes the modified Bessel function of the first kind and order ν .
Revised: κ p/ 2 − 1 / (cid:2) (2 π ) p/ 2 I p/ 2 − 1 ( κ ) (cid:3) , where I ν denotes the modified Bessel function of the first kind and order ν .

Original: Assuming a mixture model containing K vMF components with mixture component ratios π ( k ) , the probability of assigning a sample y i to a mixture component k (also known as the responsibility of cluster k ) is given by:
Revised: Assuming a mixture model containing K vMF components with mixture component ratios π ( k ) , the probability of assigning a sample y i to a mixture component k (also known as the responsibility of cluster k ) is given by:

Original: [Equation]
Revised: [Equation]

Original: where ⟨ u, v ⟩ = u T v denotes inner product.
Revised: where ⟨ u , v ⟩ = u T v denotes inner product.

Original: In DINO, the probability distributions for the student and the teacher can be rewritten as:
Revised: In DINO, the probability distributions for the student and the teacher can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: Note that centering is only applied on the teacher outputs and that different sharpening temperatures are used for the teacher and the student.
Revised: Note that centering is only applied on the teacher outputs and that different sharpening temperatures are used for the teacher and the student.

Original: By comparing Eq.
Revised: By comparing Eq.

Original: (3) with Eq.
Revised: (3) with Eq.

Original: (2), we observe that the
Revised: (2), we observe that the

Original: DINO formulation resembles that of a mixture model, given some assumptions.
Revised: DINO formulation resembles that of a mixture model, given some assumptions.

Original: We explain these assumptions below to establish a more concrete connection.
Revised: We explain these assumptions below to establish a more concrete connection.

Original: This interpretation also applies to other recent works that use a similar inner-product-based prototype formulation (Caron et al., 2020;
Revised: This interpretation also applies to other recent works that use a similar inner-product-based prototype formulation (Caron et al., 2020;

Original: Assran et al., 2022; Li et al., 2021; 2020).
Revised: Assran et al., 2022; Li et al., 2021; 2020).

Original: The exponential term in DINO from Eq.
Revised: The exponential term in DINO from Eq.

Original: (3) corresponds to the unnormalized probability density of a vMF distribution if we identify κ ( k ) =
Revised: (3) corresponds to the unnormalized probability density of a vMF distribution if we identify κ ( k ) =

Original: ∥ W ( k ) ∥ /τ and µ ( k ) =
Revised: ∥ w ( k ) ∥ /τ and µ ( k ) =

Original: W ( k ) /
Revised: w ( k ) /

Original: ∥ W ( k ) ∥ .
Revised: ∥ w ( k ) ∥ .

Original: Similar to prior clustering-based SSL works, DINO avoids collapse by encouraging a uniform distribution of the data over the prototypes, which explains the absence of π ( k ) in Eq.
Revised: Similar to prior clustering-based SSL works, DINO avoids collapse by encouraging a uniform distribution of the data over the prototypes, which explains the absence of π ( k ) in Eq.

Original: In the teacher model, Eq.
Revised: In the teacher model, Eq.

Original: (4), we obtain an additional term through the centering operation which further encourages uniformity.
Revised: (4), we obtain an additional term through the centering operation which further encourages uniformity.

Original: We discuss more about the centering operation in section 3.4.
Revised: We discuss more about the centering operation in section 3.4.

Original: Following the mixture model interpretation, we note that a missing term in the DINO formulation is the normalization constant, C p ( κ ( k ) ) .
Revised: Following the mixture model interpretation, we note that a missing term in the DINO formulation is the normalization constant, C p ( κ ( k ) ) .

Original: However, this is inconsequential when the prototypes W ( k ) are L 2 -normalized as it will result in constant κ ( k )
Revised: However, this is inconsequential when the prototypes w ( k ) are L 2 -normalized as it will result in constant κ ( k )

Original: = 1 /τ .
Revised: = 1 /τ .

Original: Hence, the constant C p ( κ ( k ) ) would vanish in the softmax.
Revised: Hence, the constant C p ( κ ( k ) ) would vanish in the softmax.

Original: However, a constant κ ( k ) implies the assumption that all clusters should have a similar isotropic precision.
Revised: However, a constant κ ( k ) implies the assumption that all clusters should have a similar isotropic precision.

Original: This constraint reduces the flexibility of the mixture model, and this lack of flexibility in the latent space translates to the backbone mapping as well.
Revised: This constraint reduces the flexibility of the mixture model, and this lack of flexibility in the latent space translates to the backbone mapping as well.

Original: Indeed, as discussed above, unnormalized prototypes help in improving the performance achieved by DINO (for ViT-Small models) by enabling better representations (Caron et al., 2021).
Revised: Indeed, as discussed above, unnormalized prototypes help in improving the performance achieved by DINO (for ViT-Small models) by enabling better representations (Caron et al., 2021).

Original: However, larger ViT-Base models are affected by training instabilities when trained with unnormalized prototypes.
Revised: However, larger ViT-Base models are affected by training instabilities when trained with unnormalized prototypes.

Original: We hypothesise that this is due to the "missing" normalization constant and propose to modify DINO by including appropriate normalization according to a vMF mixture model.
Revised: We hypothesise that this is due to the "missing" normalization constant and propose to modify DINO by including appropriate normalization according to a vMF mixture model.

Original: 3. N ORMALIZING VON M ISES -F ISHER ( V MF) COMPONENTS
Revised: 3. N ORMALIZING VON M ISES -F ISHER ( V MF) COMPONENTS

Original: In DINO, a large ∥ W ( k ) ∥ scales the logit scores of a mixture component proportionally.
Revised: In DINO, a large ∥ w ( k ) ∥ scales the logit scores of a mixture component proportionally.

Original: In a properly normalized formulation, a large ∥ W ( k ) ∥ instead scales the κ ( k ) parameter proportionally and results in a sharper vMF distribution for the component.
Revised: In a properly normalized formulation, a large ∥ w ( k ) ∥ instead scales the κ ( k ) parameter proportionally and results in a sharper vMF distribution for the component.

Original: Hence, the model cannot naively increase ∥ W ( k ) ∥ to increase the responsibility of a component to a data sample—it also needs to map the data samples close to the component’s prototype.
Revised: Hence, the model cannot naively increase ∥ w ( k ) ∥ to increase the responsibility of a component to a data sample—it also needs to map the data samples close to the component’s prototype.

Original: If the images assigned to a component can be consistently mapped close to its prototype in the latent space, only then it is beneficial to increase κ ( k ) .
Revised: If the images assigned to a component can be consistently mapped close to its prototype in the latent space, only then it is beneficial to increase κ ( k ) .

Original: The vMF normalization constant C p ( κ ) includes the modified Bessel function of the first kind and order ν , denoted as I ν ( κ ) .
Revised: The vMF normalization constant C p ( κ ) includes the modified Bessel function of the first kind and order ν , denoted as I ν ( κ ) .

Original: It is easy to obtain κ = ∥ W ∥ /τ , but since I ν ( κ ) is a complicated function, we propose to use a differentiable approximation thereof.
Revised: It is easy to obtain κ = ∥ w ∥ /τ , but since I ν ( κ ) is a complicated function, we propose to use a differentiable approximation thereof.

Original: Let ν =
Revised: Let ν =

Original: p/ 2 − 1 and κ = νr .
Revised: p/ 2 − 1 and κ = νr .

Original: Then, I ν can be approximated by the following uniform expansion for large ν (DLMF, Eq.
Revised: Then, I ν can be approximated by the following uniform expansion for large ν (DLMF, Eq.

Original: [Equation]
Revised: [Equation]

Original: where ∼ denotes a Poincaré asymptotic expansion w.r.t.
Revised: where ∼ denotes a Poincaré asymptotic expansion w.r.t.

Original: ν , s = (1 + r 2 ) − 1 / 2 and polynomials U
Revised: ν , s = (1 + r 2 ) − 1 / 2 and polynomials U

Original: i ( s ) .
Revised: i ( s ) .

Original: Here, the bottleneck dimension in the prediction head is p = 256 , and this results in ν = 127 , which is large in this context.
Revised: Here, the bottleneck dimension in the prediction head is p = 256 , and this results in ν = 127 , which is large in this context.

Original: Empirically, evaluating only the first term in the sum, U 0 ( s ) = 1 seems to be sufficient, see Appendix A.1 for details.
Revised: Empirically, evaluating only the first term in the sum, U 0 ( s ) = 1 seems to be sufficient, see Appendix A.2 for details.

Original: With this approximation, we compute C p ( κ ( k ) ) for the k -th component and modify the logit scores for component k for both teacher and student from ( ⟨ W ( k ) , y ⟩ ) /τ to ( ⟨ W ( k ) , y ⟩ )
Revised: With this approximation, we compute C p ( κ ( k ) ) for the k -th component and modify the logit scores for component k for both teacher and student from ( ⟨ w ( k ) , y ⟩ ) /τ to ( ⟨ w ( k ) , y ⟩ )

Original: /τ
Revised: /τ

Original: + log C p ( κ ( k ) ) .
Revised: + log C p ( κ ( k ) ) .

Original: That is, we add log C p ( κ ( k ) ) to the logit scores in (3) and (4).
Revised: That is, we add log C p ( κ ( k ) ) to the logit scores in (3) and (4) to obtain the following expressions for DINO-vMF: [Equation] [Equation]

Original: 3. A VOIDING C OLLAPSE
Revised: 3. A VOIDING C OLLAPSE

Original: The probability distributions are computed based on the teacher and student network outputs using a softmax function.
Revised: The probability distributions are computed based on the teacher and student network outputs using a softmax function.

Original: With the motivation of avoiding collapse, centering and sharpening operations are applied to the network outputs to obtain the logit scores for the probability distribution.
Revised: With the motivation of avoiding collapse, centering and sharpening operations are applied to the network outputs to obtain the logit scores for the probability distribution.

Original: The centering parameter c in DINO is computed as an EMA of the teacher network outputs with a forgetting parameter m ≲ 1 as follows:
Revised: The centering parameter c in DINO is computed as an EMA of the teacher network outputs with a forgetting parameter m ≲ 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: However, when centering is done in the logit space as in (6) with our modified logit scores ( ⟨ W ( k ) t , y t ⟩ ) /τ
Revised: However, when centering is done in the logit space as in (8) with our modified logit scores ( ⟨ w ( k ) t , y t ⟩ ) /τ

Original: + log C p ( κ ( k ) t ) , an EMA of the log normalization constant will be added to c ( k ) .
Revised: + log C p ( κ ( k ) t ) , an EMA of the log normalization constant will be added to c ( k ) .

Original: When the centering operation is performed on the teacher outputs, the impact of the log normalization constant is dampened.
Revised: When the centering operation is performed on the teacher outputs, the impact of the log normalization constant is dampened.

Original: To avoid this, we propose to instead compute the probability distributions for the images in the batch, and then average the probability distributions instead.
Revised: To avoid this, we propose to instead compute the probability distributions for the images in the batch, and then average the probability distributions instead.

Original: This is also closer to how mixture proportions π ( k ) are computed in the M-step of the EM algorithm for a mixture model, although the estimated probabilities play a different role in DINO; see Appendix A.3 for a discussion.
Revised: This is also closer to how mixture proportions π ( k ) are computed in the M-step of the EM algorithm for a mixture model, although the estimated probabilities play a different role in DINO; see Appendix A.4 for a discussion.

Original: Our proposal for computing the centering parameter c is thus:
Revised: Our proposal for computing the centering parameter c is thus:

Original: [Equation]

Original: The computation of the probability distributions in DINO from (3) and (4) are modified to obtain the following expressions for DINO-vMF:

Original: [Equation]
Revised: [Equation]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Clustering-based and prototypical methods:Parallel to the advances in SSL discussed in Section 1, a line of work based on clustering have also evolved.
Revised: Clustering-based and prototypical methods: SSL has advanced significantly in recent times and a line of work based on clustering have also evolved.

Original: Caron et al.
Revised: Caron et al.

Original: (2018) proposed a predictive task using "pseudo-labels" generated by K-means clustering in the representation space.
Revised: (2018) proposed a predictive task using "pseudo-labels" generated by K-means clustering in the representation space.

Original: Following its success, several methods based on clustering have been proposed (Caron et al., 2020; Asano et al., 2019; Caron et al., 2019; Li et al., 2020; Zhuang et al., 2019; Assran et al., 2022; 2021).
Revised: Following its success, several methods based on clustering have been proposed (Caron et al., 2020; Asano et al., 2019; Caron et al., 2019; Li et al., 2020; Zhuang et al., 2019; Assran et al., 2022; 2021).

Original: These methods learn a set of prototype vectors or cluster centroids that correspond with the clusters.
Revised: These methods learn a set of prototype vectors or cluster centroids that correspond with the clusters.

Original: When combined with ViTs and self-distillation, Caron et al.
Revised: When combined with ViTs and self-distillation, Caron et al.

Original: (2021) demonstrated that the learned representations possess useful properties – strong nearest neighbor embeddings and rich spatial semantic features.
Revised: (2021) demonstrated that the learned representations possess useful properties – strong nearest neighbor embeddings and rich spatial semantic features.

Original: While the DINO formulation was only based on the [CLS] representation, several recent works have extendedthe DINOformulation to utilize patch-level representations in the objective formulation (Li et al., 2021;
Revised: While the DINO formulation was only based on the [CLS] representation, several recent works have extended DINO to utilize patch-level representations in the objective formulation (Li et al., 2021;

Original: Zhou et al., 2021).
Revised: Zhou et al., 2021).

Original: A common aspect among most recent methods is that they utilize L 2 -normalized prototypes.
Revised: A common aspect among most recent methods is that they utilize L 2 -normalized prototypes.

Original: Our work is based on the DINO formulation but our insights can prove beneficial to other methods as well.
Revised: Our work is based on the DINO formulation. By incorporating our work in iBOT, we show that our insights can prove beneficial to other methods as well.

Original: Our method differs from prior works by proposing a way to remove the L 2 -normalization constraint on the prototypes and hence, enable a flexible mixture model in the representation space.
Revised: Our method differs from prior works by proposing a way to remove the L 2 -normalization constraint on the prototypes and hence, enable a flexible mixture model in the representation space.

Original: Assran et al.
Revised: Assran et al.

Original: (2022) uses an entropy regularization to avoid collapse and our centering computation shares similarity with it, in terms of computing intra-batch averages of probability distributions.
Revised: (2022) uses an entropy regularization to avoid collapse and our centering computation shares similarity with it, in terms of computing intra-batch averages of probability distributions.

Original: However, we only average teacher probability distributions to compute the centering parameter instead of applying an explicit regularization of the student outputs.
Revised: However, we only average teacher probability distributions to compute the centering parameter instead of applying an explicit regularization of the student outputs.

Original: Mixture of von Mises-Fisher distributions: Banerjee et al.
Revised: Mixture of von Mises-Fisher distributions: Banerjee et al.

Original: (2005) proposed a mixture of vMF (movMF) distribution to perform clustering on the unit hypersphere.
Revised: (2005) proposed a mixture of vMF (movMF) distribution to perform clustering on the unit hypersphere.

Original: The EM updates are often done using an approximate reparameterization of the vMF precision parameter κ (Banerjee et al., 2005;
Revised: The EM updates are often done using an approximate reparameterization of the vMF precision parameter κ (Banerjee et al., 2005;

Original: Barbaro & Rossi, 2021).
Revised: Barbaro & Rossi, 2021).

Original: Instead, we learn κ directly through the prototype magnitudes.
Revised: Instead, we learn κ directly through the prototype magnitudes.

Original: Some image segmentation methods have used movMF formulations (Yang et al., 2020; Hwang et al., 2019) but assume a fixed κ for all mixture components to simplify computations.
Revised: Some image segmentation methods have used movMF formulations (Yang et al., 2020; Hwang et al., 2019) but assume a fixed κ for all mixture components to simplify computations.

Original: Taghia et al.
Revised: Taghia et al.

Original: ; Gopal &
Revised: ; Gopal &

Original: Yang (2014) use a complete Bayesian approach based on variational inference to model data as a movMF.
Revised: Yang (2014) use a complete Bayesian approach based on variational inference to model data as a movMF.

Original: Recent works relate the cosine similarity formulation commonly used in contrastive SSL to a vMF density (Shwartz-Ziv et al., 2022; Lee et al., 2021; Wang & Isola, 2020).
Revised: Recent works relate the cosine similarity formulation commonly used in contrastive SSL to a vMF density (Shwartz-Ziv et al., 2022; Lee et al., 2021; Wang & Isola, 2020).

Original: Our mixture model interpretation is closest to Hasnat et al.
Revised: Our mixture model interpretation is closest to Hasnat et al.

Original: (2017) that proposes a movMF loss to train a supervised face verification model, but they make similar uniformity assumptions as DINO.
Revised: (2017) that proposes a movMF loss to train a supervised face verification model, but they make similar uniformity assumptions as DINO.

Original: E XPERIMENTS
Revised: accuracy ablating on the impact of L 2normalization of prototypes, vMF normalization and probability centering. Average over 2 runs are reported. Refer

Original: We closely follow the training settings for different backbones as specified in the public repository for DINO 3 , in order to ensure fair comparison. The models are pre-trained on the ImageNet dataset accuracy ablating on the impact of L 2normalization of prototypes, vMF normalization and
Revised: A.5.1 for results of individual runs. [Table] default setting for DINO/ViT-S,

Original: [Table] [Table]
Revised: [Table] [Table]

Revised:  E XPERIMENTS

Revised: We closely follow the training settings for different backbones as specified in the public repository for DINO 3 , in order to ensure fair comparison.

Original:  (Deng et al., 2009) with the adamw optimizer (Loshchilov & Hutter, 2018).
Revised: The models are pre-trained on the ImageNet dataset (Deng et al., 2009) with the adamw optimizer (Loshchilov & Hutter, 2018).

Original: Weight decay with a cosine schedule from 0.04 to 0.4 is used.
Revised: Weight decay with a cosine schedule from 0.04 to 0.4 is used.

Original: The student temperature τ is set to 0 .
Revised: The student temperature τ is set to 0 .

Original: 1 and the teacher temperature is linearly scaled from 0 .
Revised: 1 and the teacher temperature is linearly scaled from 0 .

Original: 04 to 0 .
Revised: 04 to 0 .

Original: 07 over some initial epochs (50 epochs for ViT-Small/ and 30 epochs for ViT-Base/16).
Revised: 07 over some initial epochs (50 epochs for ViT-Small/ and 30 epochs for ViT-Base/16).

Original: We follow the same data augmentations as DINO and BYOL (Grill et al., 2020): color jittering, Gaussian blur and solarization.
Revised: We follow the same data augmentations as DINO and BYOL (Grill et al., 2020): color jittering, Gaussian blur and solarization.

Original: The multi-crop approach is used with the same definition of global and local crop sizes as DINO.
Revised: The multi-crop approach is used with the same definition of global and local crop sizes as DINO.

Original: The model trainings are done on a single
Revised: The model trainings are done on a single

Original: A100 node, consisting of 8 GPUs.
Revised: A100 node, consisting of 8 GPUs.

Original: The batch sizes are adapted to fit the node and adjusted based on the model architecture (batch size=64 per GPU for ViT-Base/16 and 128 for ViT-Small/16).
Revised: The batch sizes are adapted to fit the node and adjusted based on the model architecture (batch size=64 per GPU for ViT-Base/16 and 128 for ViT-Small/16).

Revised: The vMF normalization computation does not add any noticeable overheads to standard DINO and iBOT.

Original: Due to computational reasons, we  do not experiment with smaller patch sizes or ViTs larger than the Base variant.
Revised: Due to computational reasons, we train ViT-Small/8 for only 40 epochs (refer A.5.2) and do not consider ViTs larger than the Base variant.

Original: For comparison of results, we use the publicly released DINO and MSN 4 pre-trained models available in their respective repositories.
Revised: For comparison of results, we use the publicly released DINO and MSN 4 pre-trained models available in their respective repositories.

Original: 5. I MAGE N ET C LASSIFICATION 5.1. A BLATION STUDIES
Revised: 5. I MAGE N ET C LASSIFICATION 5.1. A BLATION STUDIES

Original: We conducted ablation experiments to study the impact of our proposed modifications to DINO.
Revised: We conducted ablation experiments to study the impact of our proposed modifications to DINO.

Original: We consider small-scale experiments by training a ViT-Small/16 backbone for 100 epochs with all other settings maintained similar to the standard training.
Revised: We consider small-scale experiments by training a ViT-Small/16 backbone for 100 epochs with all other settings maintained similar to the standard training.

Original: We report the kNN top-1 classification accuracy on ImageNet in Table 1.
Revised: We report the kNN top-1 classification accuracy on ImageNet in Table 1 by averaging over 2 runs.

Original: We observe that the performance is better when the cluster prototypes are not L 2 -normalized.
Revised: We observe that the performance is better when the cluster prototypes are not L 2 -normalized.

Original: We find that vMF normalization and probability centering individually lead to performance improvements.
Revised: We find that vMF normalization and probability centering individually lead to performance improvements.

Original: Maximal performanceimprovement is achieved when both vMF normalization and centering in probability space are used simultaneously.
Revised: Best performance is achieved when both vMF normalization and centering in probability space are used simultaneously.

Original: On this basis, we decide to use vMF normalization along with centering in the probability space in all other experiments.
Revised: On this basis, we decide to use vMF normalization along with centering in the probability space in all other experiments.

Revised: Note that maximal improvement is observed when compared to the default setting of DINO/ViT-B.

Original: 5.1. I MAGE N ET CLASSIFICATION WITH FULL DATASET
Revised: 5.1. I MAGE N ET CLASSIFICATION WITH FULL DATASET

Original: The quality of representations learned by self-supervised models are usually evaluated by linear classification benchmarks on top of a frozen pre-trained backbone model.
Revised: The quality of representations learned by self-supervised models are usually evaluated by linear classification benchmarks on top of a frozen pre-trained backbone model.

Original: DINO is also shown to be an efficient nearest neighbor classifier.
Revised: DINO is also shown to be an efficient nearest neighbor classifier.

Original: For linear evaluation, we follow the same protocol as
Revised: For linear evaluation, we follow the same protocol as

Original: DINO.
Revised: DINO.

Original: Additionally, we also evaluate the performance using only the [CLS] representation.
Revised: Additionally, we also evaluate the performance using only the [CLS] representation.

Original: For kNN evaluation, we consider a straightforward sweep over a range of values for nearest neighbors and  generally find k = 10 to work best with DINO-vMF models.
Revised: For kNN evaluation, we consider a straightforward sweep over a range of values for nearest neighbors and [Table] [Table] generally find k = 10 to work best with DINO-vMF models.

Original: The validation accuracy achieved by linear and kNN classifiers on ImageNet(Deng et al., 2009) are reported in Table 2.
Revised: The validation accuracy achieved by linear and kNN classifiers on ImageNet are reported in Table 2.

Original: With ViTBase/16, DINO-vMF  clearly performs better than DINO but with ViT-Small/16, the performance improvement is found to be marginal.
Revised: With ViT-Base/16, both DINO-vMF and iBOT-vMF clearly perform better than their standard counterparts. With ViT-Small/16, the performance improvement is found to be marginal.

Original: The MSN (Assran et al., 2022)pre-trained model also shows competitive performance on ViT-Small/16 but the performance is significantly worsethan both DINO and DINO-vMF on the larger ViT-Base/16.[Table] [Table] [Table]
Revised: The MSN (Assran et al., 2022) shows competitive performance on ViT-Small/16 but it is significantly worse on the larger ViT-Base/16.

Original:  I MAGE N ET FEW - SHOT EVALUATION
Revised: 5.1. I MAGE N ET FEW - SHOT EVALUATION

Original: Few-shot classification is an important application area for SSL pre-training.
Revised: Few-shot classification is an important application area for SSL pre-training.

Original: MSN (Assran et al., 2022) pre-training is found to be effective at few-shot classification tasks.
Revised: MSN (Assran et al., 2022) pre-training is found to be effective at few-shot classification tasks.

Original: We compare our DINOvMF, DINO and MSN pre-training using the same evaluation protocol as MSN.
Revised: We compare our DINO, iBOT, their vMF versions and MSN pre-training using the same evaluation protocol as MSN.

Original: A linear classifier is trained on features obtained using a center crop of the image from the frozen pre-trained model.
Revised: A linear classifier is trained on features obtained using a center crop of the image from the frozen pre-trained model.

Original: L 2 -regularization with a constant strength of 0 .
Revised: L 2 -regularization with a constant strength of 0 .

Original: 075 is used.
Revised: 075 is used.

Original: We consider different levels of labelled data availability: 1, 2 and 5 images per class, and 1% of all the training images.
Revised: We consider different levels of labelled data availability: 1, 2 and 5 images per class, and 1% of all the training images.

Original: We report the topaccuracy in Table 3.
Revised: We report the top-1 accuracy in Table 3.

Original: For the 1, 2 and 5 images per class experiments, we use three different splits and report the meanand standard deviations of the accuracy.
Revised: For the 1, 2 and 5 images per class experiments, we use three different splits and report the mean of the accuracy (refer A.5.3 for standard deviations).

Original: With ViT-Base/16 model, we observe a large improvement over standard DINO training and evenmarginally surpassing the performance achieved by MSN (Assran et al., 2022), which specifically targets the few-shot learning problem.
Revised: With ViT-Base/16 model , we observe a large improvement by adding the vMF modification and even surpassing the performance achieved by MSN (Assran et al., 2022), which specifically targets the few-shot learning problem.

Original: In comparison, the ViT-Small/16 model with DINO-vMF pre-training only improves marginally over DINO and is found to be lagging behind MSN by a large margin.
Revised: In comparison, the ViT-Small/16 model with DINO-vMF pre-training only improves marginally over DINO and is found to be lagging behind MSN by a large margin.

Original: 5. A NALYSIS OF LEARNED V MF MIXTURE MODEL
Revised: 5. A NALYSIS OF LEARNED V MF MIXTURE MODEL

Original: In this section, we take a deeper look at the mixture modeling task, which acts as the supervisory signal to train the self-supervised model.
Revised: In this section, we take a deeper look at the mixture modeling task, which acts as the supervisory signal to train the self-supervised model.

Original:  Inspecting the prototype vectors learned by DINO, we observed that many of them were almost identical.
Revised: Prototype utilization: Inspecting the prototype vectors learned by DINO, we observed that many of them were almost identical.

Original: We consider two prototypes W ( i ) and W ( j ) to be duplicates if their cosine similarity is greater than a threshold value.
Revised: We consider two prototypes w ( i ) and w ( j ) to be duplicates if their cosine similarity is greater than a threshold value.

Original: We identify unique prototypes by removing such duplicates.
Revised: We identify unique prototypes by removing such duplicates.

Original: Recall that DINO uses a large number of prototypes set to 65536 for all models.
Revised: Recall that DINO uses a large number of prototypes set to 65536 for all models.

Original: Assran et al.
Revised: Assran et al.

Original: (2022) does not observe a benefit in using more than 1024 prototypes, but fewer prototypes lead to worse performance.
Revised: (2022) does not observe a benefit in using more than 1024 prototypes, but fewer prototypes lead to worse performance.

Original: In Table 4, we show the number of unique prototypes and the size of the largest set of duplicate prototypes by considering a cosine similarity threshold of 0 .
Revised: In Table 4, we show the number of unique prototypes and the size of the largest set of duplicate prototypes by considering a cosine similarity threshold of 0 .

Original: 9 (in A.2, we show that our observations hold at other choices of similarity thresholds as well).
Revised: 9 (in A.3, we show that our observations hold at other choices of similarity thresholds as well).

Original: Interestingly, when pre-trained with DINO, the ViT-Base model produced one large duplicate set containing ≈ 83% of all the prototypes.
Revised: Interestingly, when pre-trained with DINO, the ViT-Base model produced one large duplicate set containing ≈ 83% of all the prototypes.

Original: We refer to this as the void prototype set .
Revised: We refer to this as the void prototype set .

Original: Interestingly, the void prototype has a cosine similarity of exactly −with the mean of y over all the training data.
Revised: Interestingly, the void prototype has a cosine similarity of exactly − 1 with the mean of y over all the training data.

Original: This means that the void prototype points in a direction exactly opposite to the data mean.
Revised: This means that the void prototype points in a direction exactly opposite to the data mean.

Original: The reason for this is because the model minimizes the logit scores of unused prototypes i.e. void prototypes.
Revised: The reason for this is because the model minimizes the logit scores of unused prototypes i.e. void prototypes.

Original: This is achieved by minimizing the inner-product of the void prototypes with all y .
Revised: This is achieved by minimizing the inner-product of the void prototypes with all y .

Original: With DINO-vMF pre-training, we do not observe that any such void prototype set is formed and more unique prototypes are utilized than standard DINO pre-training.
Revised: With DINO-vMF pre-training, we do not observe that any such void prototype set is formed and more unique prototypes are utilized than standard DINO pre-training.

Original: Utilizing more prototypes will increase the difficulty of the SSL task and we expect this to result in better image representations.
Revised: Utilizing more prototypes will increase the difficulty of the SSL task and we expect this to result in better image representations.

Original: 5.2. I NTERPRETING LEARNT V MF PRECISION

Original: [Figure]
Revised: [Figure]

Original:  In the DINO-vMF formulation, the L 2 -norm of the prototypes is directly related to the vMF precision as κ = ∥ W ( k ) ∥ /τ .
Revised: Interpreting learnt vMF precision: In the DINO-vMF formulation, the L 2 -norm of the prototypes is directly related to the vMF precision as κ = ∥ w ( k ) ∥ /τ .

Original: We investigate if the precision values can be indicative of image classification difficulty by evaluating downstream performance.
Revised: We investigate if the precision values can be indicative of image classification difficulty by evaluating downstream performance.

Original: We associate an image with the component k that has the maximum logit score under the DINO(-vMF) formulation.
Revised: We associate an image with the component k that has the maximum logit score under the DINO(-vMF) formulation.

Original: For each model, we divide the data based on percentile ranges of the precision values of the associated components.
Revised: For each model, we divide the data based on percentile ranges of the precision values of the associated components.

Original: In Figure 2, we show the kNN top-1 validation accuracy on ImageNet for subsets of data restricted to lie in the different precision percentile ranges.
Revised: In Figure 2, we show the kNN top-1 validation accuracy on ImageNet for subsets of data restricted to lie in the different precision percentile ranges.

Original: For DINO-vMF, we observe that the kNN classification accuracy is increasing with increasing precision values.
Revised: For DINOvMF, we observe that the kNN classification accuracy is increasing with increasing precision values.

Original: That is, data points associated with higher precision components appear to be easier to classify, and vice versa.
Revised: That is, data points associated with higher precision components appear to be easier to classify, and vice versa.

Original: The prototype magnitudes learned by DINO (only in ViT-Small/16) do not exhibit such a clear association.
Revised: The prototype magnitudes learned by DINO (only in ViT-Small/16) do not exhibit such a clear association.

Original: 5. T RANSFER LEARNING
Revised: 5. T RANSFER LEARNING

Original: An important benefit of SSL is that the learned model can be transferred to downstream tasks to achieve competitive performance with limited computational cost or limited training data.
Revised: An important benefit of SSL is that the learned model can be transferred to downstream tasks to achieve competitive performance with limited computational cost or limited training data.

Original: With this objective, we consider the following tasks: linear classification on small datasets, image retrieval and video object segmentation.
Revised: The tasks we consider are: linear classification on small datasets, image retrieval and video object segmentation.

Original:  We conduct linear classification experiments on a suite of datasets trained on features extracted from a frozen pre-trained model.
Revised: Linear classification on small datasets: We conduct linear classification experiments on a suite of datasets trained on features extracted from a frozen pre-trained model.

Original: We consider the following small datasets: Aircraft (Maji et al., 2013), Caltech101 (Li et al., 2022), CIFAR10, CIFAR100 (Krizhevsky, 2009), DTD (Cimpoi et al., 2014), Flowers (Nilsback & Zisserman, 2008), Food (Bossard et al., 2014), Pets (Parkhi et al., 2012) and SUN397 (Xiao et al., 2010).
Revised: The implementation details for this experiment are explained in A.5.4.

Original: The linear classifier is trained with L 2 -regularization and the strength is chosen for each dataset by using 5-fold cross-validation among a set of 45 values spaced linearly in the range [ − 6 , 5] in log-space following Ericsson et al.

Original: In Table 5, we report the linear classification accuracy on the test or validation dataset, depending on which is publicly available.
Revised: In Table 5, we report the linear classification accuracy on the test or validation dataset, depending on which is publicly available.

Original: First, we observe that the DINO-vMF model performs on par or better than the standard DINO model on most of the datasets with both ViT-Small/16 and ViT-Base/16 architectures. 
Revised: First, we observe that the DINO-vMF model performs on par or better than the standard DINO model on most of the datasets with both ViT-Small/16 and ViT-Base/16 architectures. The iBOT-vMF model consistently performs best for most datasets.

Original: We also observe that the MSN model performs worse than both the DINO and DINO-vMF models on all datasets.
Revised: We also observe that the MSN model performs worse than both the DINO and DINO-vMF models on all datasets.

Original: [Table]
Revised: [Table]

Original: I MAGE R ETRIEVAL

Original:  We consider the most recent, face-blurred versions (v1.0) of Oxford (Philbin et al., 2007) and Paris datasets (Philbin et al., 2008).
Revised: Image Retrieval: We consider the most recent, face-blurred versions (v1.0) of Oxford (Philbin et al., 2007) and Paris datasets (Philbin et al., 2008).

Original: We only consider the medium (M) and hard (H) data splits.We
Revised: We only consider the medium (M) and hard (H) data splits.

Original: [Table] [Table] perform nearest neighbour based image retrieval using ImageNet pre-trained frozen features and report the mean average precision (mAP).
Revised: We perform nearest neighbour based image retrieval using ImageNet pre-trained frozen features and report the mean average precision (mAP).

Original: We follow similar evaluation protocol to DINO but report performance metrics using the newer versions of the datasets. 
Revised: We follow similar evaluation protocol to DINO but report performance metrics using the newer versions of the datasets. We observe improved

Original: We observe improved performance  compared to  DINO  on both datasets when using the ViT-Base/16 model.
Revised: [Table] [Table] performance with the ViT-Base/16 model when using the vMF versions compared to standard DINO and iBOT on both datasets.

Original: However, the ViT-Small/ model pre-trained with DINO-vMF is only on par or slightly worse than standard DINO pre-training.Further
Revised: However, the ViT-Small/16 model pre-trained with DINO-vMF is only on par or slightly worse than standard DINO pre-training.

Original: [Table] we find that the MSN pre-trained model for ViT-Small/16 also performs competitively.
Revised: Further, we find that the MSN pre-trained model for ViT-Small/16 also performs competitively.

Original: 5.3. V IDEO O BJECT S EGMENTATION (VOS)
Revised: Video Object Segmentation (VOS):

Original: We consider the DAVIS-2017 video instance segmentation benchmark (Pont-Tuset et al., 2017).
Revised: We consider the DAVIS-2017 video instance segmentation benchmark (Pont-Tuset et al., 2017).

Original: We use the frozen features from the ImageNet pre-trained model and use a nearest neighbour based segmentation approach following the same experimental protocol as (Caron et al., 2021) and (Jabri et al., 2020).
Revised: We use the frozen features from the ImageNet pre-trained model and use a nearest neighbour based segmentation approach following the same experimental protocol as (Caron et al., 2021) and (Jabri et al., 2020).

Original: We compare our resultsto with DINO and MSN in Table 7.
Revised: We compare our results with standard DINO, iBOT and MSN in Table 7.

Original: We find that DINO-vMF marginally improves on standard DINO on both ViT-Small/16 and ViT-Base/16 backbones.
Revised: We find that the vMF versions show marginal improvements on both ViT backbones.

Original: We also find that the MSNpre-trained model transfers poorly to this task compared to DINO and DINO-vMF.
Revised: We also find that the MSN transfers poorly to this task compared to DINO and DINO-vMF.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We reinterpreted the DINO (Caron et al., 2021) SSL method as a mixture of von Mises-Fisher distributions in latent space.
Revised: We reinterpreted the DINO (Caron et al., 2021) SSL method as a mixture of von Mises-Fisher distributions in latent space.

Original: Based on this interpretation, we proposed to use unnormalized prototypes with appropriate normalization of the component logit scores to enable a more flexible mixture model.
Revised: Based on this interpretation, we proposed to use unnormalized prototypes with appropriate normalization of the component logit scores to enable a more flexible mixture model.

Original: We showed that our proposed modifications enable stable training of DINO using larger ViT models and better utilization of the clusters.
Revised: Our proposed modifications enable stable training of DINO using larger ViT models and better cluster utilization. The modification requires log-normalizing constants during training.

Revised: However, using the proposed approximation these are very fast to compute and do not result in any noticeable computational overhead.

Revised: At inference time the log-normalizer is not used at all.

Original: With empirical evidence, we demonstrated consistent performance gains on a range of transfer tasks.
Revised: We empirically demonstrated consistent performance gains on a range of transfer tasks.

Original: We also compared with the recent MSN method (Assran et al., 2022) on few-shot ImageNet classification and demonstrated better results with the larger ViT-Base/ model.

Original: The larger ViT-Base/16 model benefits the most from our proposed modifications and shows significant improvements on all the considered downstream tasks.
Revised: The larger ViT-Base/ model benefits the most from our proposed modifications and shows significant improvements on all the considered downstream tasks.

Original: We accredit this to the fact that, without the proposed modifications, the larger models require unfavourable L 2 -normalization of the prototypes to obtain stable training. 
Revised: We accredit this to the fact that, without the proposed modifications, the larger models require unfavourable L 2 -normalization of the prototypes to obtain stable training. The smaller models are already trained without L 2 -normalization and for these models we see more marginal gains.

Original: Thus, our study has put a spotlight on a seemingly important design choice that, in our opinion, has not been discussed enough in prior work.
Revised: Thus, our study has put a spotlight on a seemingly important design choice that, in our opinion, has not been discussed enough in prior work.

Revised: By improving iBOT using our vMF modification, we show that our work extends to other methods built on the DINO formulation.

Original: DINO-vMF was motivated by a mixture model interpretation of DINO, but we nevertheless use a quite similar training algorithm.
Revised: DINO-vMF was motivated by a mixture model interpretation of DINO, but we nevertheless use a quite similar training algorithm.

Original: For future work it would be interesting to investigate if this interpretation can be used to motivate additional modifications.
Revised: For future work it would be interesting to investigate if this interpretation can be used to motivate additional modifications.

Original: One could for instance relate the
Revised: One could for instance relate the

Original: DINO training scheme to stochastic versions of the EM algorithm (Delyon et al., 1999; Celeux &
Revised: DINO training scheme to stochastic versions of the EM algorithm (Delyon et al., 1999; Celeux &

Original: Diebolt, 1992).
Revised: Diebolt, 1992).

Original: The soft cluster assignments computed using the teacher network can be viewed as the E-step, where the EMA is similar to the stochastic approximation of the auxiliary quantity used in SAEM (Delyon et al., 1999).
Revised: The soft cluster assignments computed using the teacher network can be viewed as the E-step, where the EMA is similar to the stochastic approximation of the auxiliary quantity used in SAEM (Delyon et al., 1999).

Original: The stochasticity comes from mini-batch sampling and multi-crop training.
Revised: The stochasticity comes from mini-batch sampling and multi-crop training.

Original: The main difference is in terms of the objective that is optimized in the M-step, where
Revised: The main difference is in terms of the objective that is optimized in the M-step, where

Original: EM maximizes the ELBO whereas DINO(-vMF) minimizes the KL divergence between cluster probabilities.
Revised: EM maximizes the ELBO whereas DINO(-vMF) minimizes the KL divergence between cluster probabilities.

Original: Exploring this connection in more detail could possibly result in better understanding of clustering-based SSL methods, as well as additional performance boosts.
Revised: Exploring this connection in more detail could possibly result in better understanding of clustering-based SSL methods, as well as additional performance boosts.

Original: R EPRODUCIBILITY STATEMENT
Revised: R EPRODUCIBILITY STATEMENT

Original: We intend to make a public release of the code repository and pre-trained models in order to aid the research community to reproduce our experiments.
Revised: We intend to make a public release of the code repository and pre-trained models in order to aid the research community to reproduce our experiments.

Original: We do not introduce any new hyperparameters in our pre-training method and use the same hyperparameter setup as DINO (Caron et al., 2021)  which is already publicly available.
Revised: We do not introduce any new hyperparameters in our pre-training method and use the same hyperparameter setup as DINO (Caron et al., 2021) and iBOT (Zhou et al., 2021) which are already publicly available.

Original: For other downstream tasks, we closely follow the experimental protocol of other works and explicitly state any differences in the paper
Revised: For other downstream tasks, we closely follow the experimental protocol of other works and explicitly state any differences in the paper.

Original: A A PPENDIX A. VON M ISES -F ISHER N ORMALIZATION C ONSTANT
Revised: A A PPENDIX A. I NTUITIVE COMPARISON OF DINO AND DINO- V MF

Revised: [Figure]

Revised: In Section 3.

Revised: we stated that in a properly normalized formulation the model cannot naively increase ∥ w ( k ) ∥ to increase the probability of assigning a data sample to a cluster or mixture component.

Revised: We expand on this idea by providing an intuitive example.

Revised: Consider a 2-dimensional latent space on the unit circle.

Revised: Let y denote a vector in this latent space with ∥ y ∥

Revised: = 1 .

Revised: We consider a simple setting with only two clusters, z ∈ { 1 , 2 } and let w (1) and w (2) be prototypes representing these clusters.

Revised: The

Revised: DINO formulation proposes the following logit score l (1) DINO for assigning vector y to cluster 1:

Revised: [Equation]

Revised: For simplicity, let us ignore the temperature scaling that is applied in the sharpening step.

Revised: Here, the model can simply increase ∥ w (1) ∥ to increase l (1) DINO as long as cos θ (1) > 0 .

Revised: Instead, our proposed DINO-vMF uses the following l (1) DINO-vMF logit scores:

Revised: [Equation]

Revised: where log C p ( · ) < 0 and log C p ( ∥ w (1) ∥ ) monotonically decreases with ∥ w (1) ∥ .

Revised: So, merely increasing ∥ w (1) ∥ can increase the first term but this is counteracted by the second term.

Revised: For given values of ∥ w (1) ∥ and θ (1) there exists a threshold ˆ θ (1) below which increasing ∥ w (1) ∥ results in an increased logit score.

Revised: The logit score decreases when θ (1) is larger than this threshold.

Revised: So, the model needs to decrease θ (1) by mapping the vector y closer to the prototype in order to benefit from increasing ∥ w (1) ∥ .

Revised: We show an illustration of how the vMF density varies with the prototype magnitude in

Revised: Figure 3.

Revised: This implies that, if the DINO-vMF model can consistently map a set of images close to a prototype, only then it can benefit from increasing the magnitude of that prototype.

Revised: A. VON M ISES -F ISHER N ORMALIZATION C ONSTANT

Original: Given a von Mises-Fisher distribution in p -dimensions with parameters µ and κ  its normalization constant C p ( κ ) is given by: 
Revised: Given a von Mises-Fisher distribution in p -dimensions with parameters µ and κ , its normalization constant C p ( κ ) is given by: [Figure] [Figure]

Original: [Equation]
Revised: [Equation]

Original: where I ν denotes the modified Bessel function of the first kind and order ν .
Revised: where I ν denotes the modified Bessel function of the first kind and order ν .

Original: We approximate the
Revised: We approximate the

Original: I p/ 2 − 1 ( κ ) term as explained in section 3.3 by using an asymptotic expansion.
Revised: I p/ 2 − 1 ( κ ) term as explained in section 3.3 by using an asymptotic expansion.

Original: The softmax-logit scores for each mixture component is obtained by adding the network output and the log-normalization constant corresponding to that component.
Revised: The softmax-logit scores for each mixture component is obtained by adding the network output and the log-normalization constant corresponding to that component.

Original: Because of the softmax formulation, it is sufficient for the log-normalization constant approximation to be correct up to a constant.
Revised: Because of the softmax formulation, it is sufficient for the log-normalization constant approximation to be correct up to a constant.

Original: We compare the values of log C p ( κ ) obtained using our approximation to those obtained by using the scipy implementation of I ν instead.
Revised: We compare the values of log C p ( κ ) obtained using our approximation to those obtained by using the scipy implementation of I ν instead.

Original: We consider κ ∈ [20 , 500] and in Figure 3 we show a comparison of the values of log C p ( κ ) obtained using our approximation.
Revised: We consider κ ∈ [20 , 500] and in Figure 4 we show a comparison of the values of log C p ( κ ) obtained using our approximation.

Original: We did produce the same figure with the same values computed using the scipy-based implementation, but the lines appear to completely overlap with our approximation, so this is not shown.
Revised: We did produce the same figure with the same values computed using the scipy-based implementation, but the lines appear to completely overlap with our approximation, so this is not shown.

Original: We define an approximation error ϵ ( κ ) with respect to the scipy implementation as follows:
Revised: We define an approximation error ϵ ( κ ) with respect to the scipy implementation as follows:

Original: [Equation]
Revised: [Equation]

Original: where C ( a ) p ( κ ) denotes our approximation and C ( s ) p ( κ ) denotes the scipy-based implementation 5 .
Revised: where C ( a ) p ( κ ) denotes our approximation and C ( s ) p ( κ ) denotes the scipy-based implementation 5 .

Original: The error values are shown in Figure 4 for different bottleneck dimensions p in the DINO prediction head.
Revised: The error values are shown in Figure 5 for different bottleneck dimensions p in the DINO prediction head.

Original: For p = 256 , as in the case of DINO, we can observe that the approximation error is small compared to the actual function values.
Revised: For p = 256 , as in the case of DINO, we can observe that the approximation error is small compared to the actual function values.

Original: A. P ROTOTYPE UTILIZATION AT DIFFERENT SIMILARITY THRESHOLDS
Revised: A. P ROTOTYPE UTILIZATION AT DIFFERENT SIMILARITY THRESHOLDS

Original: As explained in section 5.2.1, we identify unique prototypes based on a cosine similarity threshold.
Revised: As explained in section 5.2, we identify unique prototypes based on a cosine similarity threshold.

Original: Here, we present a further discussion about how the prototype utilization varies with the choice of similarity threshold.
Revised: Here, we present a further discussion about how the prototype utilization varies with the choice of similarity threshold.

Original: In Figures 5 and 6, we show the number of unique prototypes and the size of the largest duplicate set of prototypes.
Revised: In Figures 6 and 7, we show the number of unique prototypes and the size of the largest duplicate set of prototypes.

Original: An unusually large duplicate set indicates a void prototype set .
Revised: An unusually large duplicate set indicates a void prototype set .

Original: We observe that the void prototype set in ViT-base/16 hardly changes in size even when the similarity threshold is increased up to 0 .
Revised: We observe that the void prototype set in ViT-base/16 hardly changes in size even when the similarity threshold is increased up to 0 .

Original: We observe a higher prototype utilization with DINOvMF compared to DINO for both the ViT-Base and ViT-Small models.
Revised: We observe a higher prototype utilization with DINOvMF compared to DINO for both the ViT-Base and ViT-Small models.

Original: Only exception is when we set the similarity threshold to a large value of 0 . 99 for ViT-Small/16 model trained with DINO.
Revised: Only exception is when we set the similarity threshold to a large value of 0 . 99 for ViT-Small/16 model trained with DINO.

Original: We expect these near-duplicates to further increase in similarity, if the model training is continued for more epochs.
Revised: We expect these near-duplicates to further increase in similarity, if the model training is continued for more epochs.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: N u m b e r o f un i q u e p r o t

Original: o 
Revised: o t

Original: [Figure]
Revised: y p

Original: un i
Revised: e s

Original: [Figure]
Revised: [Figure]

Original: u e
Revised: Base

Original: [Figure]
Revised: [Figure]

Original: o
Revised: Similarity threshold ViT-Small/

Original: [Figure]
Revised: [Figure]

Original: o

Original: [Figure]
Revised: [Figure]

Revised: S i z

Revised: e o f t

Original:  s 
Revised: h e l a r g e s t d u p li c a t e p r o t o t y p e s e t

Original: [Figure]
Revised: [Figure]

Original: A. R ELATIONSHIP BETWEEN CENTERING AND CLUSTER PRIOR
Revised: A. R ELATIONSHIP BETWEEN CENTERING AND CLUSTER PRIOR

Original: The centering parameter c ( k ) , computed according to Eq.
Revised: The centering parameter c ( k ) , computed according to Eq.

Original: (6) in DINO or Eq.
Revised: (8) in DINO or Eq.

Original: (7) in DINO-vMF, is related to an estimate of the cluster prior π ( k ) in our mixture model interpretation.
Revised: (9) in DINO-vMF, is related to an estimate of the cluster prior π ( k ) in our mixture model interpretation.

Original: Indeed, for
Revised: Indeed, for

Original: DINO, the centering parameter averages the logit scores of the components, scaled by the sharpening parameter τ t , so exp (cid:2) c ( k ) /τ
Revised: DINO, the centering parameter averages the logit scores of the components, scaled by the sharpening parameter τ t , so exp (cid:2) c ( k ) /τ

Original: t (cid:3) can be viewed as an estimate of π ( k ) up to normalization.
Revised: t (cid:3) can be viewed as an estimate of π ( k ) up to normalization.

Original: For DINOvMF we similarly average the cluster probability (i [Figure] e., responsibility) over a batch at each iteration, and update our estimate of c ( k ) using an EMA of the corresponding logits; see (7).
Revised: For DINOvMF we similarly average the cluster probability (i.e., responsibility) over a batch at each iteration, and update our estimate of c ( k ) using an EMA of the corresponding logits; see (9).

Original: Note that the sharpening parameter τ t is included in the expression for the probabilities, which means that, in this formulation, we can estimate π ( k ) as exp (cid:2) c ( k ) (cid:3) up to normalization, i.e. without the scaling τ t .
Revised: Note that the sharpening parameter τ t is included in the expression for the probabilities, which means that, in this formulation, we can estimate π ( k ) as exp (cid:2) c ( k ) (cid:3) up to normalization, i.e. without the scaling τ t .

Original: The key difference between the two estimates is thus whether the averaging is done in logit space or in probability space.
Revised: The key difference between the two estimates is thus whether the averaging is done in logit space or in probability space.

Original: The latter is closer to how one would typically estimate π ( k ) in a standard mixture model, e.g. using the EM algorithm, as an average of the responsibility for cluster k .
Revised: The latter is closer to how one would typically estimate π ( k ) in a standard mixture model, e.g. using the EM algorithm, as an average of the responsibility for cluster k .

Original: Combining the aforementioned estimates with the expressions for the teacher probabilities (Eq.
Revised: Combining the aforementioned estimates with the expressions for the teacher probabilities (Eq.

Original: (4) for DINO or Eq.
Revised: (4) for DINO or Eq.

Original: (9) for DINO-vMF) and comparing with the responsibilities as computed for a mixture model in Eq.
Revised: (7) for DINO-vMF) and comparing with the responsibilities as computed for a mixture model in Eq.

Original: (2), it is tempting to simply relate the factors involving c ( k ) with the prior  probability π ( k ) in (2).
Revised: (2), it is tempting to simply relate the factors involving c ( k ) with the prior [Table] [Table] probability π ( k ) in (2).

Original: However, this would be an incorrect interpretation since there is a minus sign in the exponents in Eqs.
Revised: However, this would be an incorrect interpretation since there is a minus sign in the exponents in Eqs.

Original: (4)and (9).
Revised: (4)and (7).

Original: Similarity threshold
Revised: To understand this discrepancy

Original: S i z e o f t
Revised: [Table]

Original: h e l a r g e s t d u p li c a t e p r o t o t y p e s e t
Revised: note first that in a mixture model we would estimate π ( k ) from the available data

Original: ViT-Base/ 0. Similarity threshold ViT-Small/ DINO
Revised: [Table]

Original: DINO-vMF To understand this discrepancy, note first that in a mixture model we would estimate π ( k ) from the available data, but also encourage data points to be assigned to components with higher π ( k ) -values.
Revised: but also encourage data points to be assigned to components with higher π ( k ) -values.

Original: This can be thought of as a "rich gets richer" approach.
Revised: This can be thought of as a "rich gets richer" approach.

Original: If a certain cluster is found to be dominant, then this information is fed back to the estimates of the responsibilities, increasing the probability of assigning data points to this cluster.
Revised: If a certain cluster is found to be dominant, then this information is fed back to the estimates of the responsibilities, increasing the probability of assigning data points to this cluster.

Original: Such an approach makes sense when the data is fixed, and we are simply trying to learn a clustering model that fits the data as well as possible.
Revised: Such an approach makes sense when the data is fixed, and we are simply trying to learn a clustering model that fits the data as well as possible.

Original: However, it is also distinctly different from the role of the centering in DINO(-vMF), where the data are mapped to latent representations and the clustering is carried out in this latent space.
Revised: However, it is also distinctly different from the role of the centering in DINO(-vMF), where the data are mapped to latent representations and the clustering is carried out in this latent space.

Original: For these models we fix π to be uniform.
Revised: For these models we fix π to be uniform.

Original: The centering can be viewed as a way to further encourage this uniformity.
Revised: The centering can be viewed as a way to further encourage this uniformity.

Original: This is achieved by generating targets using the inverse of the estimated cluster probabilities, i.e. instead of multiplying with π ( k ) ∝ exp (cid:2) c ( k ) (cid:3) we multiply with 1 /π ( k ) ∝ exp (cid:2) − c ( k ) (cid:3) .
Revised: This is achieved by generating targets using the inverse of the estimated cluster probabilities, i.e. instead of multiplying with π ( k ) ∝ exp (cid:2) c ( k ) (cid:3) we multiply with 1 /π ( k ) ∝ exp (cid:2) − c ( k ) (cid:3) .

Original: This can be viewed as a way to rebalance the component assignments, by encouraging the model to assign more samples to less used components and less samples to more highly used components.
Revised: This can be viewed as a way to rebalance the component assignments, by encouraging the model to assign more samples to less used components and less samples to more highly used components.

Original: We effectively obtain a "rich gets poorer" and "poor gets richer" strategy.
Revised: We effectively obtain a "rich gets poorer" and "poor gets richer" strategy.

Revised: A. A DDITIONAL EXPERIMENTS AND DETAILS A.5. D ETAILED ABLATION STUDIES

Revised: We run the ablation studies explained in 5.1.1 for 2 runs and all pre-trainings within a run are done with manually fixed seeds to ensure that the results are comparable and reproducible within each run.

Revised: The results for each run are reported in Table

Revised: A.5. I MAGE N ET CLASSIFICATION WITH FULL DATASET

Revised: In addition to the results presented based on DINO and DINO-vMF pre-trained ViT-Small/16 and

Revised: ViT-Base/16 models

Revised: , we consider the ViT-Small/8 backbone architecture that uses a smaller patch size of 8.

Revised: This model is more expensive to train compared to the ViT-Small/16 model.

Revised: DINO pre-trained this model for 800 epochs.

Revised: We conduct a smaller experiment for 40 epochs with the same hyperparameters as in DINO.

Revised: The results are reported in Table 9.

Revised: The results are similar to our observations regarding the ViT-Small/16 architecture.

Revised: [Table]

Revised: [Table]

Revised: I MAGE N ET FEW - SHOT EVALUATION

Revised: The few-shot evaluation involving 1, 2 and 5 images per class were conducted using three different splits.

Revised: In Table 10 and Table 11, we show the mean and standard deviation of the top-1 validation classification accuracy over the three splits.

Revised: A.5.

Revised: I MPLEMENTATION DETAILS FOR LINEAR CLASSIFICATION ON SMALL DATASETS

Revised: We consider the following small datasets: Aircraft (Maji et al., 2013), Caltech101 (Li et al., 2022), CIFAR10, CIFAR100 (Krizhevsky, 2009), DTD (Cimpoi et al., 2014), Flowers (Nilsback & Zisserman, 2008), Food (Bossard et al., 2014), Pets (Parkhi et al., 2012) and SUN397 (Xiao et al., 2010).

Revised: The linear classifier is trained with L 2 -regularization and the strength is chosen for each dataset by using 5-fold cross-validation among a set of 45 values spaced linearly in the range [ − 6 , 5] in log-space following Ericsson et al.