Original: O N

Original: D ATA -A UGMENTATION

Original: AND

Original:  C ONSISTENCY B ASED S EMI -S UPERVISED L EARNINGAnonymous authors
Revised: O N D ATA -A UGMENTATION AND C ONSISTENCYB ASED S EMI -S UPERVISED L EARNING

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently proposed consistency-based Semi-Supervised Learning (SSL) methods such as the Π -model, temporal ensembling, the mean teacher, or the virtual adversarial training, have advanced the state of the art in several SSL tasks.
Revised: Recently proposed consistency-based Semi-Supervised Learning (SSL) methods such as the Π -model, temporal ensembling, the mean teacher, or the virtual adversarial training, have advanced the state of the art in several SSL tasks.

Original: These methods can typically reach performances that are comparable to their fully supervised counterparts while using only a fraction of labelled examples.
Revised: These methods can typically reach performances that are comparable to their fully supervised counterparts while using only a fraction of labelled examples.

Original: Despite these methodological advances, the understanding of these methods is still relatively limited.
Revised: Despite these methodological advances, the understanding of these methods is still relatively limited.

Original: In this text, we analyse (variations of) the Π -model in settings where analytically tractable results can be obtained.
Revised: In this text, we analyse (variations of) the Π -model in settings where analytically tractable results can be obtained.

Original: We establish links with Manifold
Revised: We establish links with Manifold

Original: Tangent Classiﬁers and demonstrate that the quality of the perturbations is key to obtaining reasonable SSL performances.
Revised: Tangent Classiﬁers and demonstrate that the quality of the perturbations is key to obtaining reasonable SSL performances.

Original: Importantly, we propose a simple extension of the Hidden Manifold Model that naturally incorporates data-augmentation schemes and offers a framework for understanding and experimenting with SSL methods.
Revised: Importantly, we propose a simple extension of the Hidden Manifold Model that naturally incorporates data-augmentation schemes and offers a framework for understanding and experimenting with SSL methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Consider a dataset D
Revised: Consider a dataset D

Original: = D L ∪D U that is comprised of labelled samples D L
Revised: = D L ∪D U that is comprised of labelled samples D L

Original: = { x i , y i } i ∈ I L as well as
Revised: = { x i , y i } i ∈ I L as well as

Original: unlabelled samples D U
Revised: unlabelled samples D U

Original: = { x i } i ∈ I U .
Revised: = { x i } i ∈ I U .

Original: Semi-Supervised Learning (SSL) is concerned with the use of both the labelled and unlabeled data for training.
Revised: Semi-Supervised Learning (SSL) is concerned with the use of both the labelled and unlabeled data for training.

Original: In many scenarios, collecting labelled data is difﬁcult or time consuming or expensive so that the amount of labelled data can be relatively small when compared to the amount of unlabelled data.
Revised: In many scenarios, collecting labelled data is difﬁcult or time consuming or expensive so that the amount of labelled data can be relatively small when compared to the amount of unlabelled data.

Original: The main challenge of SSL is in the design of methods that can exploit the information contained in the distribution of the unlabelled data (Zhu05; CSZ09).
Revised: The main challenge of SSL is in the design of methods that can exploit the information contained in the distribution of the unlabelled data (Zhu05; CSZ09).

Original: In modern high-dimensional settings that are common to computer vision, signal processing, Natural
Revised: In modern high-dimensional settings that are common to computer vision, signal processing, Natural

Original: Language Processing (NLP) or genomics, standard graph/distance based methods (BC01; ZG02;
Revised: Language Processing (NLP) or genomics, standard graph/distance based methods (BC01; ZG02;

Original: ZGL03; BNS06; DSST19) that are successful in low-dimensional scenarios are difﬁcult to implement.
Revised: ZGL03; BNS06; DSST19) that are successful in low-dimensional scenarios are difﬁcult to implement.

Original: Indeed, in high-dimensional spaces, it is often difﬁcult to design sensible notions of distances that can be exploited within these methods.
Revised: Indeed, in high-dimensional spaces, it is often difﬁcult to design sensible notions of distances that can be exploited within these methods.

Original: We refer the interested reader to the book-length treatments (Zhu05; CSZ09) for discussion of other approaches.
Revised: We refer the interested reader to the book-length treatments (Zhu05; CSZ09) for discussion of other approaches.

Original: The manifold assumption is the fundamental structural property that is exploited in most modern approaches to SSL: high-dimensional data samples lie in a small neighbourhood of a low-dimensional manifold (TP91; BJ03; Pey09; Cay05; RDV + 11).
Revised: The manifold assumption is the fundamental structural property that is exploited in most modern approaches to SSL: high-dimensional data samples lie in a small neighbourhood of a low-dimensional manifold (TP91; BJ03; Pey09; Cay05; RDV + 11).

Original: In computer vision, the presence of this lowdimensional structure is instrumental to the success of (variational) autoencoder and generative adversarial networks: large datasets of images can often be parametrized by a relatively small number of degrees of freedom.
Revised: In computer vision, the presence of this lowdimensional structure is instrumental to the success of (variational) autoencoder and generative adversarial networks: large datasets of images can often be parametrized by a relatively small number of degrees of freedom.

Original: Exploiting the unlabelled data to uncover this low-dimensional structure is crucial to the design of efﬁcient SSL methods.
Revised: Exploiting the unlabelled data to uncover this low-dimensional structure is crucial to the design of efﬁcient SSL methods.

Original: A recent and independent evaluation of several modern methods for SSL can be found in (OOR + 18).
Revised: A recent and independent evaluation of several modern methods for SSL can be found in (OOR + 18).

Original: It is found there that consistency-based methods (BAP14; SJT16; LA16; TV17; MMIK18; LZL + 18; GSA + 20), the topic of this paper, achieve state-of-the art performances in many realistic scenarios.
Revised: It is found there that consistency-based methods (BAP14; SJT16; LA16; TV17; MMIK18; LZL + 18; GSA + 20), the topic of this paper, achieve state-of-the art performances in many realistic scenarios.

Original: Contributions: consistency-based semi-supervised learning methods have recently been shown to achieve state-of-the-art results.
Revised: Contributions: consistency-based semi-supervised learning methods have recently been shown to achieve state-of-the-art results.

Original: Despite these methodological advances, the understanding of these methods is still relatively limited when compared to the fully-supervised setting (SMG13;
Revised: Despite these methodological advances, the understanding of these methods is still relatively limited when compared to the fully-supervised setting (SMG13;

Original: AS17; SBD + 18; TZ15; SZT17).
Revised: AS17; SBD + 18; TZ15; SZT17).

Original: In this article, we do not propose a new SSL method.
Revised: In this article, we do not propose a new SSL method.

Original: Instead, we analyse consistency-based methods in settings where analytically tractable results can be obtained, when the data-samples lie in the neighbourhood of well-deﬁned and tractable low-dimensional manifolds, and simple and controlled experiments can be carried out.
Revised: Instead, we analyse consistency-based methods in settings where analytically tractable results can be obtained, when the data-samples lie in the neighbourhood of well-deﬁned and tractable low-dimensional manifolds, and simple and controlled experiments can be carried out.

Original: We establish links with
Revised: We establish links with

Original: Manifold Tangent Classiﬁers and demonstrate that consistency-based SSL methods are in general more powerful since they can better exploit the local geometry of the data-manifold if efﬁcient data-augmentation/perturbation schemes are used.
Revised: Manifold Tangent Classiﬁers and demonstrate that consistency-based SSL methods are in general more powerful since they can better exploit the local geometry of the data-manifold if efﬁcient data-augmentation/perturbation schemes are used.

Original: Furthermore, in section 4.1 we show that the popular Mean Teacher method and the conceptually more simple Π -model approach share the same solutions in the regime when the data-augmentations are small; this conﬁrms often reported claim that the data-augmentation schemes leveraged by the recent SSL, as well as fully unsupervised algorithms, are instrumental to their success.
Revised: Furthermore, in section 4.1 we show that the popular Mean Teacher method and the conceptually more simple Π -model approach share the same solutions in the regime when the data-augmentations are small; this conﬁrms often reported claim that the data-augmentation schemes leveraged by the recent SSL, as well as fully unsupervised algorithms, are instrumental to their success.

Original: Finally, in section 4.3 we propose an extension of the
Revised: Finally, in section 4.3 we propose an extension of the

Original: Hidden Manifold Model (GMKZ19; GLK + 20).
Revised: Hidden Manifold Model (GMKZ19; GLK + 20).

Original: This generative model allows us to investigate the properties of consistency-based SSL methods, taking into account the data-augmentation process and the underlying low-dimensionality of the data, in a simple and principled manner, and without relying on a speciﬁc dataset.
Revised: This generative model allows us to investigate the properties of consistency-based SSL methods, taking into account the data-augmentation process and the underlying low-dimensionality of the data, in a simple and principled manner, and without relying on a speciﬁc dataset.

Original: For gaining understanding of SSL, as well as self-supervised learning methods, we believe it to be important to develop a framework that (i) can take into account the geometry of the data (ii) allows the study of the inﬂuence of the quality of the data-augmentation schemes (iii) does not rely on any particular dataset.
Revised: For gaining understanding of SSL, as well as self-supervised learning methods, we believe it to be important to develop a framework that (i) can take into account the geometry of the data (ii) allows the study of the inﬂuence of the quality of the data-augmentation schemes (iii) does not rely on any particular dataset.

Original: While the understanding of fully-supervised methods have largely been driven by the analysis of simpliﬁed model architectures (eg. linear and two-layered models, large dimension asymptotic such as the Neural Tangent Kernel), these analytical tools alone are unlikely to be enough to explain the mechanisms responsible for the success of SSL and self-supervised learning methods (CKNH20; GSA + 20), since they do not, and cannot easily be extended to, account for the geometry of the data and data-augmentation schemes.
Revised: While the understanding of fully-supervised methods have largely been driven by the analysis of simpliﬁed model architectures (eg. linear and two-layered models, large dimension asymptotic such as the Neural Tangent Kernel), these analytical tools alone are unlikely to be enough to explain the mechanisms responsible for the success of SSL and self-supervised learning methods (CKNH20; GSA + 20), since they do not, and cannot easily be extended to, account for the geometry of the data and data-augmentation schemes.

Original: Our proposed framework offers a small step in that direction.
Revised: Our proposed framework offers a small step in that direction.

Original:  C ONSISTENCY -B ASED S EMI -S UPERVISED L EARNING
Revised:  C ONSISTENCY -B ASED S EMI -S UPERVISED L EARNING

Original: For concreteness and clarity of exposition, we focus the discussion on classiﬁcation problems.
Revised: For concreteness and clarity of exposition, we focus the discussion on classiﬁcation problems.

Original: The arguments described in the remaining of this article can be adapted without any difﬁculty to other situations such as regression or image segmentation.
Revised: The arguments described in the remaining of this article can be adapted without any difﬁculty to other situations such as regression or image segmentation.

Original: Assume that the samples x i ∈
Revised: Assume that the samples x i ∈

Original: X
Revised: X

Original: ⊂ R D can be represented as D -dimensional vectors and that the labels belong to C ≥ 2 possible classes,
Revised: ⊂ R D can be represented as D -dimensional vectors and that the labels belong to C ≥ 2 possible classes,

Original: y i ∈
Revised: y i ∈

Original: Y ≡ { 1 , . . .
Revised: Y ≡ { 1 , . . .

Original: , C } .
Revised: , C } .

Original: Consider a mapping F θ : R D → R C parametrized by θ ∈ Θ ⊂ R | Θ | .
Revised: Consider a mapping F θ : R D → R C parametrized by θ ∈ Θ ⊂ R | Θ | .

Original: This can be a neural network, although that is not necessary.
Revised: This can be a neural network, although that is not necessary.

Original: For x ∈ X , the quantity F θ ( x ) can represent probabilistic output of the classiﬁer, or , for example, the pre-softmax activations.
Revised: For x ∈ X , the quantity F θ ( x ) can represent probabilistic output of the classiﬁer, or , for example, the pre-softmax activations.

Original: Empirical risk minimization consists in minimizing the function
Revised: Empirical risk minimization consists in minimizing the function

Original: [Equation]
Revised: [Equation]

Original: for a loss function (cid:96) : R C × Y (cid:55)→ R .
Revised: for a loss function (cid:96) : R C × Y (cid:55)→ R .

Original: Maximum likelihood estimation corresponds to choosing the loss function as the cross entropy.
Revised: Maximum likelihood estimation corresponds to choosing the loss function as the cross entropy.

Original: The optimal parameter θ ∈ Θ is found by a variant of stochastic gradient descent (RM51) with estimated gradient
Revised: The optimal parameter θ ∈ Θ is found by a variant of stochastic gradient descent (RM51) with estimated gradient

Original: [Equation]
Revised: [Equation]

Original: for a mini-batch B L of labelled samples.
Revised: for a mini-batch B L of labelled samples.

Original: Consistency-based SSL algorithms regularize the learning by enforcing that the learned function x (cid:55)→ F θ ( x ) respects local derivative and invariance constraints.
Revised: Consistency-based SSL algorithms regularize the learning by enforcing that the learned function x (cid:55)→ F θ ( x ) respects local derivative and invariance constraints.

Original: For simplicity, assume that the mapping x (cid:55)→ F θ ( x ) is deterministic, although the use of dropout (SHK + 14) and other sources of stochasticity are popular in practice.
Revised: For simplicity, assume that the mapping x (cid:55)→ F θ ( x ) is deterministic, although the use of dropout (SHK + 14) and other sources of stochasticity are popular in practice.

Original: The Π -model (LA16; SJT16) makes use of a stochastic mapping S :
Revised: The Π -model (LA16; SJT16) makes use of a stochastic mapping S :

Original: X × Ω → X that maps a sample x ∈ X and a source of randomness ω ∈ Ω ⊂ R d Ω to another sample S ω ( x ) ∈ X .
Revised: X × Ω → X that maps a sample x ∈ X and a source of randomness ω ∈ Ω ⊂ R d Ω to another sample S ω ( x ) ∈ X .

Original: The mapping S describes a stochastic data augmentation process.
Revised: The mapping S describes a stochastic data augmentation process.

Original: In computer vision, popular data-augmentation schemes include random translations, rotations, dilatations, croppings, ﬂippings, elastic deformations, color jittering, addition of speckle noise, and many more domain-speciﬁc variants.
Revised: In computer vision, popular data-augmentation schemes include random translations, rotations, dilatations, croppings, ﬂippings, elastic deformations, color jittering, addition of speckle noise, and many more domain-speciﬁc variants.

Original: In NLP, synonym replacements, insertions and deletions, back-translations are often used although it is often more difﬁcult to implement these data-augmentation strategies.
Revised: In NLP, synonym replacements, insertions and deletions, back-translations are often used although it is often more difﬁcult to implement these data-augmentation strategies.

Original: In a purely supervised setting, data-augmentation can be used as a regularizer.
Revised: In a purely supervised setting, data-augmentation can be used as a regularizer.

Original: Instead of directly minimizing L L , one can minimize instead
Revised: Instead of directly minimizing L L , one can minimize instead

Original: [Equation]
Revised: [Equation]

Original: In practice, data-augmentation regularization, although a simple strategy, is often crucial to obtaining good generalization properties (PW17; CZM + 18; LBC17; PCZ + 19).
Revised: In practice, data-augmentation regularization, although a simple strategy, is often crucial to obtaining good generalization properties (PW17; CZM + 18; LBC17; PCZ + 19).

Original: The idea of regularizing by enforcing robustness to the injection of noise can be traced back at least to (Bis95).
Revised: The idea of regularizing by enforcing robustness to the injection of noise can be traced back at least to (Bis95).

Original: In the Π -model, the data-augmentation mapping S is used to deﬁne a consistency regularization term,
Revised: In the Π -model, the data-augmentation mapping S is used to deﬁne a consistency regularization term,

Original: [Equation]
Revised: [Equation]

Original: The notation θ (cid:63) designates a copy of the parameter θ , i.e. θ (cid:63) = θ , and emphasizes that when differentiating the consistency regularization term θ (cid:55)→ R ( θ ) , one does not differentiate through θ (cid:63) .
Revised: The notation θ (cid:63) designates a copy of the parameter θ , i.e. θ (cid:63) = θ , and emphasizes that when differentiating the consistency regularization term θ (cid:55)→ R ( θ ) , one does not differentiate through θ (cid:63) .

Original: In practice, a stochastic estimate of ∇R ( θ ) is obtained as follows.
Revised: In practice, a stochastic estimate of ∇R ( θ ) is obtained as follows.

Original: For a mini-batch B of samples { x i } i ∈B , the current value θ (cid:63) ∈ Θ of the parameter and the current predictions f i ≡ F θ (cid:63) ( x i ) , the quantity
Revised: For a mini-batch B of samples { x i } i ∈B , the current value θ (cid:63) ∈ Θ of the parameter and the current predictions f i ≡ [Equation] x i ) , the quantity

Original: [Equation]
Revised: [Equation]

Original: is an approximation of ∇R ( θ ) .
Revised: is an approximation of ∇R ( θ ) .

Original: There are indeed many variants (eg. use of different norms, different manners to inject noise), but the general idea is to force the learned function x (cid:55)→ F θ ( x ) to be locally invariant to the data-augmentation scheme S .
Revised: There are indeed many variants (eg. use of different norms, different manners to inject noise), but the general idea is to force the learned function x (cid:55)→ F θ ( x ) to be locally invariant to the data-augmentation scheme S .

Original: Several extensions such as the Mean Teacher (TV17) and the VAT (MMIK18) schemes have been recently proposed and have been shown to lead to good results in many SSL tasks.
Revised: Several extensions such as the Mean Teacher (TV17) and the VAT (MMIK18) schemes have been recently proposed and have been shown to lead to good results in many SSL tasks.

Original: The recently proposed and state-of-the-art BYOL approach (GSA + 20) is relying on mechanisms that are very close to the consistency regularization methods discussed on this text.
Revised: The recently proposed and state-of-the-art BYOL approach (GSA + 20) is relying on mechanisms that are very close to the consistency regularization methods discussed on this text.

Original: If one recalls the manifold assumption, this approach is natural: since the samples corresponding to different classes lie on separate manifolds, the function F θ : X → R C should be constant on each one of these manifolds.
Revised: If one recalls the manifold assumption, this approach is natural: since the samples corresponding to different classes lie on separate manifolds, the function F θ : X → R C should be constant on each one of these manifolds.

Original: Since the correct value of F θ is typically well approximated or known for labelled samples ( x i , y i )
Revised: Since the correct value of F θ is typically well approximated or known for labelled samples ( x i , y i )

Original: ∈ D L , the consistency regularization term equation 1 helps propagating these known values across these manifolds.
Revised: ∈ D L , the consistency regularization term equation 1 helps propagating these known values across these manifolds.

Original: This mechanism is indeed similar to standard SSL graph-based approaches such as label propagation (ZG02).
Revised: This mechanism is indeed similar to standard SSL graph-based approaches such as label propagation (ZG02).

Original: Graph-based methods are difﬁcult to directly implement in computer vision, or NLP, when a meaningful notion of distance is not available.
Revised: Graph-based methods are difﬁcult to directly implement in computer vision, or NLP, when a meaningful notion of distance is not available.

Original: This interpretation reveals that it is crucial to include the labelled samples in the regularization term equation 1 in order to help propagating the information contained in the labelled samples to the unlabelled samples.
Revised: This interpretation reveals that it is crucial to include the labelled samples in the regularization term equation 1 in order to help propagating the information contained in the labelled samples to the unlabelled samples.

Original: Our numerical experiments suggest that, in the standard setting when the number of labelled samples is much lower than the number of unlabeled samples, i.e. |D L | (cid:28) |D U | , the formulation equation 1 of the consistency regularization leads to sub-optimal results and convergence issues: the information contained in the labelled data is swamped by the number of unlabelled samples.
Revised: Our numerical experiments suggest that, in the standard setting when the number of labelled samples is much lower than the number of unlabeled samples, i.e. |D L | (cid:28) |D U | , the formulation equation 1 of the consistency regularization leads to sub-optimal results and convergence issues: the information contained in the labelled data is swamped by the number of unlabelled samples.

Original: In all our experiments, we have adopted instead the following regularization term
Revised: In all our experiments, we have adopted instead the following regularization term

Original: [Equation]
Revised: [Equation]

Original: that balances the labelled and unlabelled data samples more efﬁciently.
Revised: that balances the labelled and unlabelled data samples more efﬁciently.

Original: Furthermore, it is clear that the quality and variety of the data-augmentation scheme S :
Revised: Furthermore, it is clear that the quality and variety of the data-augmentation scheme S :

Original: X × Ω → X is pivotal to the success of consistency-based SSL methods.
Revised: X × Ω → X is pivotal to the success of consistency-based SSL methods.

Original: We argue in this article that it is the dominant factor contributing to the success of this class of methods.
Revised: We argue in this article that it is the dominant factor contributing to the success of this class of methods.

Original: Effort spent on building efﬁcient local data-augmentation schemes will be rewarded in terms of generalization performances.
Revised: Effort spent on building efﬁcient local data-augmentation schemes will be rewarded in terms of generalization performances.

Original: Designing good data-augmentation schemes is an efﬁcient manner of injecting expert/prior knowledge into the learning process.
Revised: Designing good data-augmentation schemes is an efﬁcient manner of injecting expert/prior knowledge into the learning process.

Original: It is done by leveraging the understanding of the local geometry of the data manifold.
Revised: It is done by leveraging the understanding of the local geometry of the data manifold.

Original: As usual and not surprisingly (NGP98; MHF + 12), in data-scarce settings, any type of domain-knowledge needs to be exploited and we argue that consistency regularization approaches to SSL are instances of this general principle.
Revised: As usual and not surprisingly (NGP98; MHF + 12), in data-scarce settings, any type of domain-knowledge needs to be exploited and we argue that consistency regularization approaches to SSL are instances of this general principle.

Original:  A PPROXIMATE M ANIFOLD T ANGENT C LASSIFIER
Revised:  A PPROXIMATE M ANIFOLD T ANGENT C LASSIFIER

Original: It has long been known (SLDV98) that exploiting the knowledge of derivatives, or more generally enforcing local invariance properties, can greatly enhance the performance of standard clas- siﬁers/regressors (HK02; CS02).
Revised: It has long been known (SLDV98) that exploiting the knowledge of derivatives, or more generally enforcing local invariance properties, can greatly enhance the performance of standard classiﬁers/regressors (HK02; CS02).

Original: In the context of deep-learning, the Manifold Tangent Classiﬁer (RDV + 11) is yet another illustration of this idea.
Revised: In the context of deep-learning, the Manifold Tangent Classiﬁer (RDV + 11) is yet another illustration of this idea.

Original: Consider the data manifold M ⊂ X ⊂ R D and assume that the data samples lie on a neighbourhood of it.
Revised: Consider the data manifold M ⊂ X ⊂ R D and assume that the data samples lie on a neighbourhood of it.

Original: For x ∈ M , consider as well the tangent plane T x to M at x .
Revised: For x ∈ M , consider as well the tangent plane T x to M at x .

Original: Assuming that the manifold M is of dimension 1 ≤ d ≤ D , the tangent plane
Revised: Assuming that the manifold M is of dimension 1 ≤ d ≤ D , the tangent plane

Original: T x is also of dimension d with an orthonormal basis e x 1 , . .
Revised: T x is also of dimension d with an orthonormal basis e x 1 , . .

Original: , e xd ∈ R D .
Revised: , e xd ∈ R D .

Original: This informally means that, for suitably small coefﬁcients ω 1 , . . . , ω d ∈ R , the transformed sample x ∈ X deﬁned as
Revised: This informally means that, for suitably small coefﬁcients ω 1 , . . . , ω d ∈ R , the transformed sample x ∈ X deﬁned as

Original: [Equation]
Revised: [Equation]

Original: also lies, or is very close to, the data manifold M .
Revised: also lies, or is very close to, the data manifold M .

Original: A possible stochastic data-augmentation scheme can therefore be deﬁned as S ω ( x ) = x + V ω where V ω
Revised: A possible stochastic data-augmentation scheme can therefore be deﬁned as S ω ( x ) = x + V ω where V ω

Original: =
Revised: =

Original: (cid:80) dj
Revised: (cid:80) dj

Original: =1 ω j e xj .
Revised: =1 ω j e xj .

Original: If ω is a multivariate ddimensional centred Gaussian random vector with suitably small covariance matrix, the perturbation vector V ω is also centred and normally distributed.
Revised: If ω is a multivariate ddimensional centred Gaussian random vector with suitably small covariance matrix, the perturbation vector V ω is also centred and normally distributed.

Original: To enforce that the function x → F θ ( x ) is locally approximately constant along the manifold M , one can thus penalize the derivatives of F θ at x in the directions V ω .
Revised: To enforce that the function x → F θ ( x ) is locally approximately constant along the manifold M , one can thus penalize the derivatives of F θ at x in the directions V ω .

Original: Denoting by J x ∈ R C,D the Jacobian with respect to x ∈ R D of F θ at x ∈ M , this can be implemented by adding a penalization term of the type E ω [ (cid:107) J x V ω (cid:107) 2 ] =
Revised: Denoting by J x ∈ R C,D the Jacobian with respect to x ∈ R D of F θ at x ∈ M , this can be implemented by adding a penalization term of the type E ω [ (cid:107) J x V ω (cid:107) 2 ] =

Original: Tr (cid:0) Γ ⊗ J Tx J x (cid:1) , where Γ ∈ R D,D is the covariance matrix of the random vector ω → V ω .
Revised: Tr (cid:0) Γ ⊗ J Tx J x (cid:1) , where Γ ∈ R D,D is the covariance matrix of the random vector ω → V ω .

Original: This type of regularization of the Jacobian along the data-manifold is for example used in (BNS06).
Revised: This type of regularization of the Jacobian along the data-manifold is for example used in (BNS06).

Original: More generally, if one assumes that for any x, ω ∈ X × Ω we have S εω ( x )
Revised: More generally, if one assumes that for any x, ω ∈ X × Ω we have S εω ( x )

Original: = x + ε D ( x, ω )
Revised: = x + ε D ( x, ω )

Original: + O ( ε 2 ) , for some derivative mapping D : X × Ω → X , it follows that
Revised: + O ( ε 2 ) , for some derivative mapping D : X × Ω → X , it follows that

Original: [Equation]
Revised: [Equation]

Original: where Γ x, S is the covariance matrix of the X -valued random vector ω (cid:55)→ D ( x, ω ) ∈ X .
Revised: where Γ x, S is the covariance matrix of the X -valued random vector ω (cid:55)→ D ( x, ω ) ∈ X .

Original: This shows that consistency-based methods can be understood as approximated Jacobian regularization methods, as proposed in (SLDV98; RDV + 11).
Revised: This shows that consistency-based methods can be understood as approximated Jacobian regularization methods, as proposed in (SLDV98; RDV + 11).

Original: 3. L IMITATIONS
Revised: 3. L IMITATIONS

Original: In practice, even if many local dimension reduction techniques have been proposed, it is still relatively difﬁcult to obtain a good parametrization of the data manifold.
Revised: In practice, even if many local dimension reduction techniques have been proposed, it is still relatively difﬁcult to obtain a good parametrization of the data manifold.

Original: The Manifold Tangent Classiﬁer (MTC) (RDV + 11) implements this idea by ﬁrst extracting in an unsupervised manner a good representation of the dataset D by using a Contractive-Auto-Encoder (CAE) (RVM + 11).
Revised: The Manifold Tangent Classiﬁer (MTC) (RDV + 11) implements this idea by ﬁrst extracting in an unsupervised manner a good representation of the dataset D by using a Contractive-Auto-Encoder (CAE) (RVM + 11).

Original: This CAE can subsequently be leveraged to obtain an approximate basis of each tangent plane T x
Revised: This CAE can subsequently be leveraged to obtain an approximate basis of each tangent plane T x

Original: i for x i ∈ D , which can then be used for penalizing the Jacobian of the mapping x
Revised: i for x i ∈ D , which can then be used for penalizing the Jacobian of the mapping x

Original: (cid:55)→ F θ ( x ) in the direction of the tangent plane to M at x .
Revised: (cid:55)→ F θ ( x ) in the direction of the tangent plane to M at x .

Original: The above discussion shows that the somewhat simplistic approach[Figure] consisting in adding an isotropic Gaussian noise to the data samples is unlikely to deliver  results.
Revised: The above discussion shows that the somewhat simplistic approach consisting in adding an isotropic Gaussian noise to the data samples is unlikely to deliver satisfying results.

Original: It is equivalent to penalizing the Frobenius norm (cid:107)  (cid:107) 2 F of the Jacobian of the mapping x (cid:55)→ F θ ( x ) ; in a linear model, that is equivalent to the standard ridge regularization .
Revised: It is equivalent to penalizing the Frobenius norm (cid:107) J x (cid:107) 2 F of the Jacobian of the mapping x (cid:55)→ F θ ( x ) ; in a linear model, that is equivalent to the standard ridge regularization .

Original: This mechanism does not take at all into account the local-geometry of the data-manifold.
Revised: This mechanism does not take at all into account the local-geometry of the data-manifold.

Original: Nevertheless, in medical imaging applications where scans are often contaminated by speckle noise, this class of approaches which can be thought off as adding artiﬁcial speckle noise, can help mitigate over-ﬁtting (DRS + 18).
Revised: Nevertheless, in medical imaging applications where scans are often contaminated by speckle noise, this class of approaches which can be thought off as adding artiﬁcial speckle noise, can help mitigate over-ﬁtting (DRS + 18).

Original: There are many situations where, because of data scarcity or the sheer difﬁculty of unsupervised representation learning in general, domain-speciﬁc data-augmentation schemes lead to much better regularization than Jacobian penalization.
Revised: There are many situations where, because of data scarcity or the sheer difﬁculty of unsupervised representation learning in general, domain-speciﬁc data-augmentation schemes lead to much better regularization than Jacobian penalization.

Original: Furthermore, as schematically illustrated in Figure 1,
Revised: Furthermore, as schematically illustrated in Figure 1,

Original: Jacobian penalization techniques are not efﬁcient at learning highly non-linear manifolds that are common, for example, in computer vision.
Revised: Jacobian penalization techniques are not efﬁcient at learning highly non-linear manifolds that are common, for example, in computer vision.

Original: For example, in “pixel space", a simple image translation is a highly non-linear transformation only well approximated by a ﬁrst order approximation for very small translations.
Revised: For example, in “pixel space", a simple image translation is a highly non-linear transformation only well approximated by a ﬁrst order approximation for very small translations.

Original: In other words, if x ∈ X represents an image and g ( x, v ) is its translated version by a vector v , the approximation g ( x, v ) ≈ x + ∇ v g ( x ) , with ∇ v g ( x ) ≡ lim ε → 0 ( g ( x, ε v ) − g ( x ) /ε , becomes poor as soon as the translation vector v is not extremely small.
Revised: In other words, if x ∈ X represents an image and g ( x, v ) is its translated version by a vector v , the approximation g ( x, v ) ≈ x + ∇ v g ( x ) , with ∇ v g ( x ) ≡ lim ε → 0 ( g ( x, ε v ) − g ( x ) /ε , becomes poor as soon as the translation vector v is not extremely small.

Original: In computer vision, translations, rotations and dilatations are often used as sole data-augmentation schemes: this leads to a poor local exploration of the data-manifold since this type transformations  only generate a very low dimensional exploration manifold.
Revised: In computer vision, translations, rotations and dilatations are often used as sole data-augmentation schemes: this leads to a poor local exploration of the data-manifold since this type transformations [Figure] only generate a very low dimensional exploration manifold.

Original: More precisely, the exploration manifold emanating from a sample x 0 ∈ X , i.e. { S ( x 0 , ω ) : ω ∈ Ω } , is very low dimensional: its dimension is much lower than the dimension d of the data-manifold M .
Revised: More precisely, the exploration manifold emanating from a sample x 0 ∈ X , i.e. { S ( x 0 , ω ) : ω ∈ Ω } , is very low dimensional: its dimension is much lower than the dimension d of the data-manifold M .

Original: Enriching the set of data-augmentation degrees of freedom with transformations such as elastic deformation or non-linear pixel intensity shifts is crucial to obtaining a high-dimensional local exploration manifold that can help propagating the information on the data-manifold efﬁciently (CZM + 19; PCZ + 19).
Revised: Enriching the set of data-augmentation degrees of freedom with transformations such as elastic deformation or non-linear pixel intensity shifts is crucial to obtaining a high-dimensional local exploration manifold that can help propagating the information on the data-manifold efﬁciently (CZM + 19; PCZ + 19).

Original:  A SYMPTOTIC P ROPERTIES 4. F LUID L IMIT
Revised:  A SYMPTOTIC P ROPERTIES 4. F LUID L IMIT

Original: Consider the standard Π -model trained with a standard Stochastic Gradient Descent (SGD).
Revised: Consider the standard Π -model trained with a standard Stochastic Gradient Descent (SGD).

Original: Denote by θ t ∈ Θ the current value of the parameter and η > 0 the learning rate.
Revised: Denote by θ t ∈ Θ the current value of the parameter and η > 0 the learning rate.

Original: We have
Revised: We have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for a parameter λ > 0 that controls the trade-off between supervised and consistency losses, as well as subsets B L and B U of labelled and unlabelled data samples, and f j≡ Fθ(cid:63) ( x j ) for θ(cid:63) ≡ θ k as discussed in Section 2.
Revised: for a parameter λ > 0 that controls the trade-off between supervised and consistency losses, as well as subsets B L and B U of labelled and unlabelled data samples, and f j F x j ) for θ ≡ θ k as discussed in Section 2.

Original: The right-hand-side is an unbiased estimate of η ∇ θ (cid:104)
Revised: The right-hand-side is an unbiased estimate of η ∇ θ (cid:104)

Original: L L ( θ k )+ λ R ( θ k ) (cid:105) with variance of order O ( η 2 ) , where the regularization term R ( θ k ) is described in equation 2.
Revised: L L ( θ k )+ λ R ( θ k ) (cid:105) with variance of order O ( η 2 ) , where the regularization term R ( θ k ) is described in equation 2.

Original: It follows from standard ﬂuid limit approximations (EK09)[Section 4.8] for Markov processes that, under mild regularity and growth assumptions and as η → 0 , the appropriately time-rescaled trajectory { θ k }
Revised: It follows from standard ﬂuid limit approximations (EK09)[Section 4.8] for Markov processes that, under mild regularity and growth assumptions and as η → 0 , the appropriately time-rescaled trajectory { θ k }

Original: k ≥can be approximated by the trajectory of the Ordinary Differential Equation (ODE).
Revised: k ≥can be approximated by the trajectory of the Ordinary Differential Equation (ODE).

Original: Proposition 4.1 Let D ([0 , T ] , R | Θ | ) be the usual space of càdlàg R | Θ | -valued functions on a bounded time interval [0 , T ] endowed with the standard Skorohod topology.
Revised: Proposition 4.1 Let D ([0 , T ] , R | Θ | ) be the usual space of càdlàg R | Θ | -valued functions on a bounded time interval [0 , T ] endowed with the standard Skorohod topology.

Original: Consider the update equation 3 with learning rate η > 0 and deﬁne the continuous time process θ η ( t )
Revised: Consider the update equation 3 with learning rate η > 0 and deﬁne the continuous time process θ η ( t )

Original: = θ [ t/η ] .
Revised: = θ [ t/η ] .

Original: The sequence of processes θ η ∈ D ([0 , T ] , R | Θ | ) converges weakly in D ([0 , T ] , R | Θ | ) and as η → 0 to the solution of the ordinary differential equation
Revised: The sequence of processes θ η ∈ D ([0 , T ] , R | Θ | ) converges weakly in D ([0 , T ] , R | Θ | ) and as η → 0 to the solution of the ordinary differential equation

Original: [Equation]
Revised: [Equation]

Original: The article (TV17) proposes the mean teacher model, an averaging approach related to the standard
Revised: The article (TV17) proposes the mean teacher model, an averaging approach related to the standard

Original: Polyak-Ruppert averaging scheme (Pol90; PJ92), which modiﬁes the consistency regularization term equation 2 by replacing the parameter θ (cid:63) by an exponential moving average (EMA).
Revised: Polyak-Ruppert averaging scheme (Pol90; PJ92), which modiﬁes the consistency regularization term equation 2 by replacing the parameter θ (cid:63) by an exponential moving average (EMA).

Original: In practical terms, this simply means that, instead of deﬁning f j = F θ (cid:63) ( x j ) , with θ (cid:63) = θ k in equation 3, one sets f j = F θ avg ,k ( x j ) where the EMA process { θ avg ,k } k ≥ 0 is deﬁned through the recursion θ avg ,k = (1 − α η ) θ avg ,k − 1 + α η θ k where the coefﬁcient α > 0 controls the time-scale of the averaging process.
Revised: In practical terms, this simply means that, instead of deﬁning f j = F θ (cid:63) ( x j ) , with θ (cid:63) = θ k in equation 3, one sets f j = F θ avg ,k ( x j ) where the EMA process { θ avg ,k } k ≥ 0 is deﬁned through the recursion θ avg ,k = (1 − α η ) θ avg ,k − 1 + α η θ k where the coefﬁcient α > 0 controls the time-scale of the averaging process.

Original: The use of the EMA process { θ avg ,k } k ≥ 0 helps smoothing out the stochasticity of the process θ k .
Revised: The use of the EMA process { θ avg ,k } k ≥ 0 helps smoothing out the stochasticity of the process θ k .

Original: Similarly to Proposition 4.1, as η → 0 , the joint process ( θ ηt , θ η avg ,t ) ≡ ( θ η [ t/η ] , θ η avg , [ t/η ] ) converges as η → 0 to the solution of the following ordinary differential equation
Revised: Similarly to Proposition 4.1, as η → 0 , the joint process ( θ ηt , θ η avg ,t ) ≡ ( θ η [ t/η ] , θ η avg , [ t/η ] ) converges as η → 0 to the solution of the following ordinary differential equation

Original: [Equation]
Revised: [Equation]

Original: where the notation R ( θ t , θ avg ,t ) designates the same quantity as the one described in equation 2, but with an emphasis on the dependency on the EMA process.
Revised: where the notation R ( θ t , θ avg ,t ) designates the same quantity as the one described in equation 2, but with an emphasis on the dependency on the EMA process.

Original: At convergence ( θ t , θ avg ,t ) → ( θ ∞ , θ avg , ∞ ) , one must necessarily have that θ ∞ =
Revised: At convergence ( θ t , θ avg ,t ) → ( θ ∞ , θ avg , ∞ ) , one must necessarily have that θ ∞ =

Original: θ avg , ∞ , conﬁrming that, in the regime of small learning rate η → 0 , the Mean Teacher method converges, albeit often more rapidly, towards the same solution as the more standard Π -model.
Revised: θ avg , ∞ , conﬁrming that, in the regime of small learning rate η → 0 , the Mean Teacher method converges, albeit often more rapidly, towards the same solution as the more standard Π -model.

Original: This indicates that the improved performances of the Mean
Revised: This indicates that the improved performances of the Mean

Original: Teacher approach sometimes reported in the literature are either not statistically meaningful, or due to poorly executed comparisons, or due to mechanisms not captured by the η → 0 asymptotic.
Revised: Teacher approach sometimes reported in the literature are either not statistically meaningful, or due to poorly executed comparisons, or due to mechanisms not captured by the η → 0 asymptotic.

Original: Indeed, several recently proposed consistency based SSL algorithms (BCG + 19; SBL + 20; XDH + 19) achieve state-of-the-art performance across diverse datasets without employing any exponential averaging processes.
Revised: Indeed, several recently proposed consistency based SSL algorithms (BCG + 19; SBL + 20; XDH + 19) achieve state-of-the-art performance across diverse datasets without employing any exponential averaging processes.

Original: These results are achieved by leveraging more sophisticated data augmentation schemes such as Rand-Augment (CZSL19) , Back Translation (ALAC17) or Mixup (ZCDLP17).
Revised: These results are achieved by leveraging more sophisticated data augmentation schemes such as Rand-Augment (CZSL19) , Back Translation (ALAC17) or Mixup (ZCDLP17).

Original: 4. M INIMIZERS ARE HARMONIC FUNCTIONS
Revised: 4. M INIMIZERS ARE HARMONIC FUNCTIONS

Original: To understand better the properties of the solutions, we consider a simpliﬁed setting further exploited in Section 4.3.
Revised: To understand better the properties of the solutions, we consider a simpliﬁed setting further exploited in Section 4.3.

Original: Assume that F : X ≡ R D
Revised: Assume that F : X ≡ R D

Original: → R and Y ≡ R and that, for every y
Revised: → R and Y ≡ R and that, for every y

Original: i ∈
Revised: i ∈

Original: Y ≡ R , the loss function f
Revised: Y ≡ R , the loss function f

Original: (cid:55)→ (cid:96) ( f, y i ) is uniquely minimized at f = y i .
Revised: (cid:55)→ (cid:96) ( f, y i ) is uniquely minimized at f = y i .

Original: We further assume that the data-manifold
Revised: We further assume that the data-manifold

Original: M ⊂ R D can be globally parametrized by a smooth and bijective mapping Φ : R d → M ⊂ R D .
Revised: M ⊂ R D can be globally parametrized by a smooth and bijective mapping Φ : R d → M ⊂ R D .

Original: Similarly to the Section 2, we consider a data-augmentation scheme that can be described as S εω ( x ) =
Revised: Similarly to the Section 2, we consider a data-augmentation scheme that can be described as S εω ( x ) =

Original: Φ( z + εω ) for z =
Revised: Φ( z + εω ) for z =

Original: Φ − 1 ( x ) and a sample ω from a R d -valued centred and isotropic
Revised: Φ − 1 ( x ) and a sample ω from a R d -valued centred and isotropic

Original: Gaussian distribution.
Revised: Gaussian distribution.

Original: We consider a ﬁnite set of labelled samples { x i , y i } i ∈ I L , with x i =
Revised: We consider a ﬁnite set of labelled samples { x i , y i } i ∈ I L , with x i =

Original: Φ( z i ) and z i ∈ R d for i ∈
Revised: Φ( z i ) and z i ∈ R d for i ∈

Original: I L .
Revised: I L .

Original: We choose to model the large number of unlabelled data samples as a continuum distributed on the data manifold M as the push-forward measure Φ (cid:93) µ ( dz ) of a probability distribution µ ( dz ) whose support is R d through the mapping Φ .
Revised: We choose to model the large number of unlabelled data samples as a continuum distributed on the data manifold M as the push-forward measure Φ (cid:93) µ ( dz ) of a probability distribution µ ( dz ) whose support is R d through the mapping Φ .

Original: This means that an empirical average of the type (1 / |D U | ) (cid:80) i ∈ I
Revised: This means that an empirical average of the type (1 / |D U | ) (cid:80) i ∈ I

Original: u ϕ ( x i ) can be replaced by (cid:82) ϕ
Revised: u ϕ ( x i ) can be replaced by (cid:82) ϕ

Original: [Φ( z )] µ ( dz ) .
Revised: [Φ( z )] µ ( dz ) .

Original: We investigate the regime ε → 0 and, similarly to Section 2, the minimization of the consistency-regularized objective
Revised: We investigate the regime ε → 0 and, similarly to Section 2, the minimization of the consistency-regularized objective

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For notational convenience, set f θ ≡ F θ ◦ Φ .
Revised: For notational convenience, set f θ ≡ F θ ◦ Φ .

Original: Since S εω [Φ( z )] =
Revised: Since S εω [Φ( z )] =

Original: Φ( z + ε ω ) , as ε → 0 the quantity 1 ε 2 E ω (cid:110)(cid:13)(cid:13) F θ [ S εω (Φ( z ))] − F θ (Φ( z )) (cid:13)(cid:13) 2 (cid:111) converges to (cid:107)∇ z f θ [Equation] 2 and the objective function equation 6 approaches the quantity
Revised: Φ( z + ε ω ) , as ε → 0 the quantity 1 ε 2 E ω (cid:110)(cid:13)(cid:13) F θ [ S εω (Φ( z ))] − F θ (Φ( z )) (cid:13)(cid:13) 2 (cid:111) converges to (cid:107)∇ z f θ (cid:107) 2 and the objective function equation 6 approaches the quantity

Original: [Equation]
Revised: [Equation]

Original: A minimizer f : R d → R of the functional G that is consistent with the labelled data, i.e. f ( z i ) =
Revised: A minimizer f : R d → R of the functional G that is consistent with the labelled data, i.e. f ( z i ) =

Original: y i for i ∈
Revised: y i for i ∈

Original: I L , is a minimizer of the energy functional f
Revised: I L , is a minimizer of the energy functional f

Original: (cid:55)→ (cid:82) R d (cid:107)∇ z f θ ( z ) (cid:107) 2 µ ( dz ) subject to the constraints f ( z i ) =
Revised: (cid:55)→ (cid:82) R d (cid:107)∇ z f θ ( z ) (cid:107) 2 µ ( dz ) subject to the constraints f ( z i ) =

Original: y i .
Revised: y i .

Original: It is the variational formulation of the Poisson equation
Revised: It is the variational formulation of the Poisson equation

Original: [Equation]
Revised: [Equation]

Original: Note that the solution does not depend on the regularization parameter λ in the regime of ε → 0 : this indicates, as will be discussed in Section 4.3 in detail, that the generalization properties of consistency-based SSL methods will typically be insensitive to this parameter, in the regime of small data-augmentation at least.
Revised: Note that the solution does not depend on the regularization parameter λ in the regime of ε → 0 : this indicates, as will be discussed in Section 4.3 in detail, that the generalization properties of consistency-based SSL methods will typically be insensitive to this parameter, in the regime of small data-augmentation at least.

Original: Furthermore, equation 8 shows that consistency-based SSL methods are indeed based on the same principles as more standard graph-based approaches such as Label
Revised: Furthermore, equation 8 shows that consistency-based SSL methods are indeed based on the same principles as more standard graph-based approaches such as Label

Original: Propagation (ZG02): solutions are gradient/Laplacian penalized interpolating functions.
Revised: Propagation (ZG02): solutions are gradient/Laplacian penalized interpolating functions.

Original: In Figure 2, we consider the case where D =
Revised: In Figure 2, we consider the case where D =

Original: d = 2 with trivial mapping Φ( x ) = x .
Revised: d = 2 with trivial mapping Φ( x ) = x .

Original: We consider labelled data situated on the right (resp.
Revised: We consider labelled data situated on the right (resp.

Original: left) boundary of the unit square and corresponding to the label y =(resp.
Revised: left) boundary of the unit square and corresponding to the label y =(resp.

Original: y = 1 ).
Revised: y = 1 ).

Original: For simplicity, we choose the loss function (cid:96) ( f, y )
Revised: For simplicity, we choose the loss function (cid:96) ( f, y )

Original: =
Revised: =

Original: 12 ( f − y ) 2 and parametrize
Revised: 12 ( f − y ) 2 and parametrize

Original: F θ ≡ f θ with a neural network with a single hidden layer with N = 100 neurons.
Revised: F θ ≡ f θ with a neural network with a single hidden layer with N = 100 neurons.

Original: As expected, the Π -model converges to the solution to the Poisson equation 8 in the unit square with boundary condition f
Revised: As expected, the Π -model converges to the solution to the Poisson equation 8 in the unit square with boundary condition f

Original: ( u, v )
Revised: ( u, v )

Original: = 0 for u = 

Revised: = 0 for u =

Original: and f ( u, v ) = 1 for u = 1 .
Revised: and f ( u, v ) = 1 for u = 1 .

Original: 4. G ENERATIVE MODEL FOR S EMI -S UPERVISED L EARNING
Revised: 4. G ENERATIVE MODEL FOR S EMI -S UPERVISED L EARNING

Original: As has been made clear throughout this text, SSL methods crucially rely on the dependence structure of the data.
Revised: As has been made clear throughout this text, SSL methods crucially rely on the dependence structure of the data.

Original: The existence and exploitation of a much lower-dimensional manifold M supporting the data-samples is instrumental to this class of methods.
Revised: The existence and exploitation of a much lower-dimensional manifold M supporting the data-samples is instrumental to this class of methods.

Original: Furthermore, the performance of consistencybased SSL approaches is intimately related to the data-augmentation schemes they are based upon.
Revised: Furthermore, the performance of consistencybased SSL approaches is intimately related to the data-augmentation schemes they are based upon.

Original: Consequently, in order to understand the mechanisms that are at play when consistency-based SSL methods are used to uncover the structures present in real datasets, it is important to build simpliﬁed and tractable generative models of data that (1) respect these low-dimensional structures and (2) allow the design of efﬁcient data-augmentation schemes.
Revised: Consequently, in order to understand the mechanisms that are at play when consistency-based SSL methods are used to uncover the structures present in real datasets, it is important to build simpliﬁed and tractable generative models of data that (1) respect these low-dimensional structures and (2) allow the design of efﬁcient data-augmentation schemes.

Original: Several articles have investigated the inﬂuence of the dependence structures that are present in the data on the learning algorithm (BM13; Mos16).
Revised: Several articles have investigated the inﬂuence of the dependence structures that are present in the data on the learning algorithm (BM13; Mos16).

Original: Here, we follow the Hidden Manifold Model (HMM) framework proposed in (GMKZ19; GLK + 20) where the authors describe a model of synthetic data concentrating near low-dimensional structures and analyze the learning curve associated to a class of two-layered neural networks.
Revised: Here, we follow the Hidden Manifold Model (HMM) framework proposed in (GMKZ19; GLK + 20) where the authors describe a model of synthetic data concentrating near low-dimensional structures and analyze the learning curve associated to a class of two-layered neural networks.

Original: [Figure]
Revised: [Figure]

Original: Low-dimensional structure: Similarly to Section 4.2, assume that the D -dimensional data-samples x i ∈
Revised: Low-dimensional structure: Similarly to Section 4.2, assume that the D -dimensional data-samples x i ∈

Original: X can be expressed as x i =
Revised: X can be expressed as x i =

Original: Φ( z i ) ∈
Revised: Φ( z i ) ∈

Original: R D for a ﬁxed smooth mapping Φ : R d → R D .
Revised: R D for a ﬁxed smooth mapping Φ : R d → R D .

Original: In other words, the data-manifold M is d -dimensional and the mapping Φ can be used to parametrize it.
Revised: In other words, the data-manifold M is d -dimensional and the mapping Φ can be used to parametrize it.

Original: The mapping Φ is chosen to be a neural network with a single hidden layer with H neurons, although other choices are indeed possible.
Revised: The mapping Φ is chosen to be a neural network with a single hidden layer with H neurons, although other choices are indeed possible.

Original: For z = ( z 1 , . . .
Revised: For z = ( z 1 , . . .

Original: , z d ) ∈ R d , set Φ( z ) =
Revised: , z d ) ∈ R d , set Φ( z ) =

Original: A 1 → 2 ϕ ( A 

Revised: A 1 → 2 ϕ ( A

Original: → 1 z + b 1 ) for matrices A 0 → 1 ∈ R H,d and A 1 → 2 ∈ R D,H , bias vector b 1 ∈ R H and non-linearity ϕ
Revised: → 1 z + b 1 ) for matrices A 0 → 1 ∈ R H,d and A 1 → 2 ∈ R D,H , bias vector b 1 ∈ R H and non-linearity ϕ

Original: R → R applied element-wise.
Revised: R → R applied element-wise.

Original: In all our experiments, we use the ELU non-linearity.
Revised: In all our experiments, we use the ELU non-linearity.

Original: We adopt the standard normalization A 

Revised: We adopt the standard normalization A

Original: → 1 i,j = w (1) i,j / √ d and A 1 → 2 i,j = w (2) i,j / √ H for weights w ( k ) i,j drawn i.i.d from a centred
Revised: → 1 i,j = w (1) i,j / √ d and A 1 → 2 i,j = w (2) i,j / √ H for weights w ( k ) i,j drawn i.i.d from a centred

Original: Gaussian distribution with unit variance; this ensures that, if the coordinate of the input vector z ∈ R
Revised: Gaussian distribution with unit variance; this ensures that, if the coordinate of the input vector z ∈ R

Original: d are all of order O (1) , so are the coordinates of x = Φ( z ) .
Revised: d are all of order O (1) , so are the coordinates of x = Φ( z ) .

Original: Data-augmentation: consider a data sample x i ∈ M on the data-manifold.
Revised: Data-augmentation: consider a data sample x i ∈ M on the data-manifold.

Original: It can also be expressed as x i =
Revised: It can also be expressed as x i =

Original: Φ( z i ) .
Revised: Φ( z i ) .

Original: We consider the natural data-augmentation process which consists in setting S εω ( x i ) =
Revised: We consider the natural data-augmentation process which consists in setting S εω ( x i ) =

Original: Φ( z i + εω ) for a sample ω ∈ R d from an isotropic Gaussian distribution with unit covariance and ε > 0 .
Revised: Φ( z i + εω ) for a sample ω ∈ R d from an isotropic Gaussian distribution with unit covariance and ε > 0 .

Original: Crucially, the data-augmentation scheme respect the low-dimensional structure of the data: the perturbed sample S εω ( x i ) belongs to the data-manifold M for any perturbation vector ε ω .
Revised: Crucially, the data-augmentation scheme respect the low-dimensional structure of the data: the perturbed sample S εω ( x i ) belongs to the data-manifold M for any perturbation vector ε ω .

Original: Note that, for any value of ε , the data-augmentation preserves the low-dimensional manifold: perturbed samples S εω ( x i ) exactly lie on the data-manifold.
Revised: Note that, for any value of ε , the data-augmentation preserves the low-dimensional manifold: perturbed samples S εω ( x i ) exactly lie on the data-manifold.

Original: The larger ε , the more efﬁcient the data-augmentation scheme; this property is important since it allows to study the inﬂuence of the amount of data-augmentation.
Revised: The larger ε , the more efﬁcient the data-augmentation scheme; this property is important since it allows to study the inﬂuence of the amount of data-augmentation.

Original: Classiﬁcation: we consider a balanced binary classiﬁcation problem with |D L | ≥ 2 labelled training examples { x i , y i }
Revised: Classiﬁcation: we consider a balanced binary classiﬁcation problem with |D L | ≥ 2 labelled training examples { x i , y i }

Original: i ∈
Revised: i ∈

Original: I
Revised: I

Original: L where x i =
Revised: L where x i =

Original: Φ( z i ) and y i ∈
Revised: Φ( z i ) and y i ∈

Original: Y ≡ {− 1 , +1 } .
Revised: Y ≡ {− 1 , +1 } .

Original: The sample z i ∈ R d corresponding to the positive (resp.
Revised: The sample z i ∈ R d corresponding to the positive (resp.

Original: negative) class are assumed to have been drawn i.i.d from a Gaussian distribution with identity covariance matrix and mean µ + ∈ R d (resp.
Revised: negative) class are assumed to have been drawn i.i.d from a Gaussian distribution with identity covariance matrix and mean µ + ∈ R d (resp.

Original: mean µ − ∈ R d ).
Revised: mean µ − ∈ R d ).

Original: The distance (cid:107) µ + − µ − (cid:107) quantiﬁes the hardness of the classiﬁcation task.
Revised: The distance (cid:107) µ + − µ − (cid:107) quantiﬁes the hardness of the classiﬁcation task.

Original: Neural architecture and optimization: Consider ﬁtting a two-layered neural network F θ : R D
Revised: Neural architecture and optimization: Consider ﬁtting a two-layered neural network F θ : R D

Original: → R by minimising the negative log-likelihood L L ( θ ) ≡ (1 / |D L | ) (cid:80) i (cid:96)
Revised: → R by minimising the negative log-likelihood L L ( θ ) ≡ (1 / |D L | ) (cid:80) i (cid:96)

Original: [ F θ ( x i ) , y i ] where (cid:96) ( f, y )
Revised: [ F θ ( x i ) , y i ] where (cid:96) ( f, y )

Original: =
Revised: =

Original: log(1 + exp[ − y f ]) .
Revised: log(1 + exp[ − y f ]) .

Original: We assume that there are |D L | = 10 labelled data pairs { x i , y i } i =
Revised: We assume that there are |D L | = 10 labelled data pairs { x i , y i } i =

Original: I L , as well as |D U | = 1000 unlabelled data samples, that the ambient space has dimension D = 100 and the data manifold M has dimension d = 10 .
Revised: I L , as well as |D U | = 1000 unlabelled data samples, that the ambient space has dimension D = 100 and the data manifold M has dimension d = 10 .

Original: The function Φ uses H = 30 neurons in its hidden layer.
Revised: The function Φ uses H = 30 neurons in its hidden layer.

Original: In all our experiments, we use a standard Stochastic Gradient Descent (SGD) method with constant learning rate and momentum β = 0 .
Revised: In all our experiments, we use a standard Stochastic Gradient Descent (SGD) method with constant learning rate and momentum β = 0 .

Original: For minimizing the consistency-based SSL objective L L ( θ ) + λ R ( θ ) , with regularization R ( θ ) given in equation 2, we use the standard strategy (TV17) consisting in ﬁrst minimizing the un-regularized objective alone L L for a few epochs in order for the function
Revised: For minimizing the consistency-based SSL objective L L ( θ ) + λ R ( θ ) , with regularization R ( θ ) given in equation 2, we use the standard strategy (TV17) consisting in ﬁrst minimizing the un-regularized objective alone L L for a few epochs in order for the function

Original: F θ to be learned in the neighbourhood of the few labelled data-samples before switching on the consistency-based regularization whose role is to propagate the information contained in the labelled samples along the data manifold M .
Revised: F θ to be learned in the neighbourhood of the few labelled data-samples before switching on the consistency-based regularization whose role is to propagate the information contained in the labelled samples along the data manifold M .

Original: [Figure] Insensitivity to λ : Figure 3 (Left) shows that this method is relatively insensitive to the parameter λ , as long as it is within reasonable bounds.
Revised: Insensitivity to λ : Figure 3 (Left) shows that this method is relatively insensitive to the parameter λ , as long as it is within reasonable bounds.

Original: This phenomenon can be read from equation 8 that does not depend on λ .
Revised: This phenomenon can be read from equation 8 that does not depend on λ .

Original: Much larger or smaller values (not shown in Figure 3) of λ do lead, unsurprisingly, to convergence and stability issues.
Revised: Much larger or smaller values (not shown in Figure 3) of λ do lead, unsurprisingly, to convergence and stability issues.

Original: Amount of Data-Augmentation: As is reported in many tasks (CZM + 18; ZCG + 19; KYF20), tuning the amount data-augmentation in deep-learning applications is often a delicate exercise that can greatly inﬂuence the resulting performances.
Revised: Amount of Data-Augmentation: As is reported in many tasks (CZM + 18; ZCG + 19; KYF20), tuning the amount data-augmentation in deep-learning applications is often a delicate exercise that can greatly inﬂuence the resulting performances.

Original: Figure 3 (Right) reports the generalization properties of the method for different amount of data-augmentation.
Revised: Figure 3 (Right) reports the generalization properties of the method for different amount of data-augmentation.

Original: Too low an amount of data-augmentation (i.e. ε = 0 . 03 ) and the ﬁnal performance is equivalent to the un-regularized method.
Revised: Too low an amount of data-augmentation (i.e. ε = 0 . 03 ) and the ﬁnal performance is equivalent to the un-regularized method.

Original: Too large an amount of data-augmentation (i.e. ε = 1 . 0 ) also leads to poor generalization properties.
Revised: Too large an amount of data-augmentation (i.e. ε = 1 . 0 ) also leads to poor generalization properties.

Original: This is because the choice of ε = 1 . 0 corresponds to augmented samples that are very different from the distribution of the training dataset (i.e. distributional shift), although these samples are still supported by the data-manifold.
Revised: This is because the choice of ε = 1 . 0 corresponds to augmented samples that are very different from the distribution of the training dataset (i.e. distributional shift), although these samples are still supported by the data-manifold.

Original: [Figure] Quality of the Data-Augmentation: to study the inﬂuence of the quality of the data-augmentation scheme, we consider a perturbation process implemented as S εω [ k ] ( x i ) =
Revised: Quality of the Data-Augmentation: to study the inﬂuence of the quality of the data-augmentation scheme, we consider a perturbation process implemented as S εω [ k ] ( x i ) =

Original: Φ( z i + ω
Revised: Φ( z i + ω

Original: [ k ]) for x i =
Revised: [ k ]) for x i =

Original: Φ( z i ) where the noise term ω [ k ] is deﬁned as follows.
Revised: Φ( z i ) where the noise term ω [ k ] is deﬁned as follows.

Original: For a data-augmentation dimension parameter 1 ≤ k ≤ d we have ω [ k ] =
Revised: For a data-augmentation dimension parameter 1 ≤ k ≤ d we have ω [ k ] =

Original: ( ξ 1 , . .
Revised: ( ξ 1 , . .

Original: . , ξ k , 0 , . .
Revised: . , ξ k , 0 , . .

Original: . , 0) for i.i.d standard Gaussian samples ξ 1 , . .
Revised: . , 0) for i.i.d standard Gaussian samples ξ 1 , . .

Original: . , ξ k ∈ R .
Revised: . , ξ k ∈ R .

Original: This data-augmentation scheme only explores the ﬁrst k dimensions of the d -dimensional datamanifold: the lower k , the poorer the exploration of the data-manifold. 
Revised: This data-augmentation scheme only explores the ﬁrst k dimensions of the d -dimensional datamanifold: the lower k , the poorer the exploration of the data-manifold. As demonstrated on Figure

Original: As demonstrated on Figure 4, lower quality data-augmentation schemes (i.e. lower values of k ∈ [0 , d ] ) hurt the generalization performance of the Π -model.
Revised: [Figure] [Figure] all the experiments, we used λ = 10 and used SGD with momentum β = 0 . 4, lower quality data-augmentation schemes (i.e. lower values of k ∈ [0 , d ] ) hurt the generalization performance of the Π -model.

Original: Mean-Teacher versus Π -model: we implemented the Mean-Teacher (MT) approach with an exponential moving average (EMA) process θ avg ,k = β MT θ avg ,k − 1 + (1 − β MT ) θ k for the MT parameter θ avg ,k with different scales β MT ∈ { 0 . 9 , 0 .
Revised: Mean-Teacher versus Π -model: we implemented the Mean-Teacher (MT) approach with an exponential moving average (EMA) process θ avg ,k = β MT θ avg ,k − 1 + (1 − β MT ) θ k for the MT parameter θ avg ,k with different scales β MT ∈ { 0 . 9 , 0 .

Original: 95 , 0 . 99 , 0 . 995 } , as well as a Π -model approach, with λ = 10 and ε = 0 .
Revised: 95 , 0 . 99 , 0 . 995 } , as well as a Π -model approach, with λ = 10 and ε = 0 .

Original: Figure 5 shows, in accordance with Section 4.1, that the different EMA schemes lead to generalization performances similar to a standard Π -model.
Revised: Figure 5 shows, in accordance with Section 4.1, that the different EMA schemes lead to generalization performances similar to a standard Π -model.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Consistency-based SSL methods rely on a subtle trade-off between the exploitation of the labelled samples and the discovery of the low-dimensional data-manifold.
Revised: Consistency-based SSL methods rely on a subtle trade-off between the exploitation of the labelled samples and the discovery of the low-dimensional data-manifold.

Original: The results presented in this article highlight the connections with more standard methods such as Jacobian penalization and graphbased approaches and emphasize the crucial role of the data-augmentation scheme.
Revised: The results presented in this article highlight the connections with more standard methods such as Jacobian penalization and graphbased approaches and emphasize the crucial role of the data-augmentation scheme.

Original: The analysis of consistency-based SSL methods is still in its infancy and our numerical simulations suggest that the variant of the Hidden Manifold Model described in this text is a natural framework to make progress in this direction.
Revised: The analysis of consistency-based SSL methods is still in its infancy and our numerical simulations suggest that the variant of the Hidden Manifold Model described in this text is a natural framework to make progress in this direction.