Original: S ELF - SUPERVISED R EPRESENTATION L EARNING WITH R ELATIVE P REDICTIVE C ODING
Revised: S ELF - SUPERVISED R EPRESENTATION L EARNING WITH R ELATIVE P REDICTIVE C ODING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity and downstream task performance.
Revised: This paper introduces Relative Predictive Coding (RPC), a new contrastive representation learning objective that maintains a good balance among training stability, minibatch size sensitivity and downstream task performance.

Original: The key to the success of RPC is two-fold.
Revised: The key to the success of RPC is two-fold.

Original: First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance.
Revised: First, RPC introduces the relative parameters to regularize the objective for boundedness and low variance.

Original: Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives.
Revised: Second, RPC contains no logarithm and exponential score functions, which are the main cause of training instability in prior contrastive objectives.

Original: We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks.
Revised: We empirically verify the effectiveness of RPC on benchmark vision and speech self-supervised learning tasks.

Original: Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance
Revised: Lastly, we relate RPC with mutual information (MI) estimation, showing RPC can be used to estimate MI with low variance

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Unsupervised learning has drawn tremendous attention recently because it can extract rich representations without label supervision.
Revised: Unsupervised learning has drawn tremendous attention recently because it can extract rich representations without label supervision.

Original: Self-supervised learning, a subset of unsupervised learning, learns representations by allowing the data to provide supervision (Devlin et al., 2018).
Revised: Self-supervised learning, a subset of unsupervised learning, learns representations by allowing the data to provide supervision (Devlin et al., 2018).

Original: Among its mainstream strategies, self-supervised contrastive learning has been successful in visual object recognition (He et al., 2020; Tian et al., 2019;
Revised: Among its mainstream strategies, self-supervised contrastive learning has been successful in visual object recognition (He et al., 2020; Tian et al., 2019;

Original: Chen et al., 2020c), speech recognition (Oord et al., 2018; Rivi ` ere et al., 2020), language modeling (Kong et al., 2019), graph representation learning (Velickovic et al., 2019) and reinforcement learning (Kipf et al., 2019).
Revised: Chen et al., 2020c), speech recognition (Oord et al., 2018; Rivi ` ere et al., 2020), language modeling (Kong et al., 2019), graph representation learning (Velickovic et al., 2019) and reinforcement learning (Kipf et al., 2019).

Original: The idea of self-supervised contrastive learning is to learn latent representations such that related instances (e.g., patches from the same image; deﬁned as positive pairs) will have representations within close distance, while unrelated instances (e.g., patches from two different images; deﬁned as negative pairs) will have distant representations (Arora et al., 2019).
Revised: The idea of self-supervised contrastive learning is to learn latent representations such that related instances (e.g., patches from the same image; deﬁned as positive pairs) will have representations within close distance, while unrelated instances (e.g., patches from two different images; deﬁned as negative pairs) will have distant representations (Arora et al., 2019).

Original: Prior work has formulated the contrastive learning objectives as maximizing the divergence between the distribution of related and unrelated instances.
Revised: Prior work has formulated the contrastive learning objectives as maximizing the divergence between the distribution of related and unrelated instances.

Original: In this regard, different divergence measurement often leads to different loss function design.
Revised: In this regard, different divergence measurement often leads to different loss function design.

Original: For example, variational mutual information (MI) estimation (Poole et al., 2019) inspires Contrastive Predictive Coding (CPC)
Revised: For example, variational mutual information (MI) estimation (Poole et al., 2019) inspires Contrastive Predictive Coding (CPC)

Original: (Oord et al., 2018).
Revised: (Oord et al., 2018).

Original: Note that MI is also the KL-divergence between the distributions of related and unrelated instances (Cover & Thomas, 2012).
Revised: Note that MI is also the KL-divergence between the distributions of related and unrelated instances (Cover & Thomas, 2012).

Original: While the choices of the contrastive learning objectives are abundant (Hjelm et al., 2018; Poole et al., 2019; Ozair et al., 2019), we point out that there are three challenges faced by existing methods.
Revised: While the choices of the contrastive learning objectives are abundant (Hjelm et al., 2018; Poole et al., 2019; Ozair et al., 2019), we point out that there are three challenges faced by existing methods.

Original: The ﬁrst challenge is the training stability, where an unstable training process with high variance may be problematic.
Revised: The ﬁrst challenge is the training stability, where an unstable training process with high variance may be problematic.

Original: For example, Hjelm et al.
Revised: For example, Hjelm et al.

Original: ; Tschannen et al.
Revised: ; Tschannen et al.

Original: ; Tsai et al.
Revised: ; Tsai et al.

Original: (2020b) show that the contrastive objectives with large variance cause numerical issues and have a poor downstream performance with their learned representations.
Revised: (2020b) show that the contrastive objectives with large variance cause numerical issues and have a poor downstream performance with their learned representations.

Original: The second challenge is the sensitivity to minibatch size, where the objectives requiring a huge minibatch size may restrict their practical usage.
Revised: The second challenge is the sensitivity to minibatch size, where the objectives requiring a huge minibatch size may restrict their practical usage.

Original: For instance, SimCLRv2 (Chen et al., 2020c) utilizes CPC as its contrastive objective and reaches state-of-the-art performances on multiple self-supervised and semi-supervised benchmarks.
Revised: For instance, SimCLRv2 (Chen et al., 2020c) utilizes CPC as its contrastive objective and reaches state-of-the-art performances on multiple self-supervised and semi-supervised benchmarks.

Original: Nonetheless, the objective is trained with a minibatch size of 8 , 192 , and this scale of training requires enormous computational power.
Revised: Nonetheless, the objective is trained with a minibatch size of 8 , 192 , and this scale of training requires enormous computational power.

Original: The third challenge is the downstream task performance, which is the one that we would like to emphasize the most.
Revised: The third challenge is the downstream task performance, which is the one that we would like to emphasize the most.

Original: For this reason, in most cases, CPC is the objective that we would adopt for contrastive representation learning, due to its favorable performance in downstream tasks (Tschannen et al., 2019; Baevski et al., 2020).
Revised: For this reason, in most cases, CPC is the objective that we would adopt for contrastive representation learning, due to its favorable performance in downstream tasks (Tschannen et al., 2019; Baevski et al., 2020).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: This paper presents a new contrastive representation learning objective: the Relative Predictive Coding ( RPC ) which attempts to achieve a good balance among these three challenges: training stability, sensitivity to minibatch size and downstream task performance.
Revised: This paper presents a new contrastive representation learning objective: the Relative Predictive Coding ( RPC ) which attempts to achieve a good balance among these three challenges: training stability, sensitivity to minibatch size and downstream task performance.

Original: At the core of RPC is the relative parameters , which are used to regularize RPC for its boundedness and low variance.
Revised: At the core of RPC is the relative parameters , which are used to regularize RPC for its boundedness and low variance.

Original: From a modeling perspective, the relative parameters act as a (cid:96) 2 regularization for RPC.
Revised: From a modeling perspective, the relative parameters act as a (cid:96) 2 regularization for RPC.

Original: From a statistical perspective, the relative parameters prevent RPC from growing to extreme values, as well as upper bound its variance.
Revised: From a statistical perspective, the relative parameters prevent RPC from growing to extreme values, as well as upper bound its variance.

Original: In addition to the relative parameters, RPC contains no logarithm and exponential, which are the main cause of the training instability for prior contrastive learning objectives (Song & Ermon, 2019).
Revised: In addition to the relative parameters, RPC contains no logarithm and exponential, which are the main cause of the training instability for prior contrastive learning objectives (Song & Ermon, 2019).

Original: To empirically verify the effectiveness of RPC, we consider benchmark self-supervised representation learning tasks, including visual object classiﬁcation on CIFAR-10/-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet (Russakovsky et al., 2015) and speech recognition on LibriSpeech (Panayotov et al., 2015).
Revised: To empirically verify the effectiveness of RPC, we consider benchmark self-supervised representation learning tasks, including visual object classiﬁcation on CIFAR-10/-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet (Russakovsky et al., 2015) and speech recognition on LibriSpeech (Panayotov et al., 2015).

Original: Comparing RPC to prior contrastive learning objectives, we observe a lower variance during training, a better minibatch size sensitivity, and consistent performance improvement.
Revised: Comparing RPC to prior contrastive learning objectives, we observe a lower variance during training, a better minibatch size sensitivity, and consistent performance improvement.

Original: Lastly, we also relate RPC with MI estimation, empirically showing that
Revised: Lastly, we also relate RPC with MI estimation, empirically showing that

Original: RPC can estimate MI with low variance.
Revised: RPC can estimate MI with low variance.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: This paper presents a new contrastive representation learning objective - the Relative Predictive
Revised: This paper presents a new contrastive representation learning objective - the Relative Predictive

Original: Coding (RPC).
Revised: Coding (RPC).

Original: At a high level, RPC 1) introduces the relative parameters to regularize the objective for boundedness and low variance; and 2) achieves a good balance among the three challenges in the contrastive representation learning objectives: training stability, sensitivity to minibatch size, and downstream task performance.
Revised: At a high level, RPC 1) introduces the relative parameters to regularize the objective for boundedness and low variance; and 2) achieves a good balance among the three challenges in the contrastive representation learning objectives: training stability, sensitivity to minibatch size, and downstream task performance.

Original: We begin by describing prior contrastive objectives along with their limitations on the three challenges in Section 2.1.
Revised: We begin by describing prior contrastive objectives along with their limitations on the three challenges in Section 2.1.

Original: Then, we detail our presented objective and its modeling beneﬁts in Section 2.2.
Revised: Then, we detail our presented objective and its modeling beneﬁts in Section 2.2.

Original: An overview of different contrastive learning objectives is provided in Table 1.
Revised: An overview of different contrastive learning objectives is provided in Table 1.

Original: We defer all the proofs in Appendix.
Revised: We defer all the proofs in Appendix.

Original: Notation We use an uppercase letter to denote a random variable (e.g., X ), a lower case letter to denote the outcome of this random variable (e.g., x ), and a calligraphy letter to denote the sample space of this random variable (e.g., X ).
Revised: Notation We use an uppercase letter to denote a random variable (e.g., X ), a lower case letter to denote the outcome of this random variable (e.g., x ), and a calligraphy letter to denote the sample space of this random variable (e.g., X ).

Original: Next, if the samples ( x, y ) are related (or positively-paired), we refer ( x, y ) ∼ P XY with P XY being the joint distribution of X × Y .
Revised: Next, if the samples ( x, y ) are related (or positively-paired), we refer ( x, y ) ∼ P XY with P XY being the joint distribution of X × Y .

Original: If the samples ( x, y ) are unrelated (negatively-paired), we refer ( x, y ) ∼ P X P Y with P X P Y being the product of marginal distributions over X × Y .
Revised: If the samples ( x, y ) are unrelated (negatively-paired), we refer ( x, y ) ∼ P X P Y with P X P Y being the product of marginal distributions over X × Y .

Original: Last, we deﬁne f ∈ F for F being any class of functions f : X ×Y → R .
Revised: Last, we deﬁne f ∈ F for F being any class of functions f : X ×Y → R .

Original: 2. P RELIMINARY
Revised: 2. P RELIMINARY

Original: Contrastive representation learning encourages the contrastiveness between the positive and negative pairs of the representations from the related data X and Y .
Revised: Contrastive representation learning encourages the contrastiveness between the positive and negative pairs of the representations from the related data X and Y .

Original: Speciﬁcally, when sampling a pair of representations ( x, y ) from their joint distribution ( ( x, y ) ∼ P XY ), this pair is deﬁned as a positive pair; when sampling from the product of marginals ( ( x, y ) ∼ P X P Y ), this pair is deﬁned as a negative pair.
Revised: Speciﬁcally, when sampling a pair of representations ( x, y ) from their joint distribution ( ( x, y ) ∼ P XY ), this pair is deﬁned as a positive pair; when sampling from the product of marginals ( ( x, y ) ∼ P X P Y ), this pair is deﬁned as a negative pair.

Original: Then, Tsai et al.
Revised: Then, Tsai et al.

Original: (2020b) formalizes this idea such that the contrastiveness of the representations can be measured by the divergence between P XY and P X P Y , where higher divergence suggests better contrastiveness.
Revised: (2020b) formalizes this idea such that the contrastiveness of the representations can be measured by the divergence between P XY and P X P Y , where higher divergence suggests better contrastiveness.

Original: To better understand prior contrastive learning objectives, we categorize them in terms of different divergence measurements between P XY and P X P Y , with their detailed objectives presented in Table 1.
Revised: To better understand prior contrastive learning objectives, we categorize them in terms of different divergence measurements between P XY and P X P Y , with their detailed objectives presented in Table 1.

Original: We instantiate the discussion using Contrastive Predictive Coding (Oord et al., 2018, J CPC ), which is a lower bound of D KL ( P XY (cid:107) P X P Y ) with D KL referring to the KL-divergence:
Revised: We instantiate the discussion using Contrastive Predictive Coding (Oord et al., 2018, J CPC ), which is a lower bound of D KL ( P XY (cid:107) P X P Y ) with D KL referring to the KL-divergence:

Original: [Equation]
Revised: [Equation]

Original: Then, Oord et al.
Revised: Then, Oord et al.

Original: (2018) presents to maximize J CPC ( X, Y ) , so that the learned representations X and Y have high contrastiveness.
Revised: (2018) presents to maximize J CPC ( X, Y ) , so that the learned representations X and Y have high contrastiveness.

Original: We note that J CPC has been commonly used in many recent self-supervised representation learning frameworks (He et al., 2020; Chen et al., 2020b), where they constrain the function to be f ( x, y )
Revised: We note that J CPC has been commonly used in many recent self-supervised representation learning frameworks (He et al., 2020; Chen et al., 2020b), where they constrain the function to be f ( x, y )

Original: =
Revised: =

Original: cosine( x, y ) with cosine( · ) being cosine similarity.
Revised: cosine( x, y ) with cosine( · ) being cosine similarity.

Original: Under this function design, maximizing J CPC leads the representations of related pairs to be close and representations of unrelated pairs to be distant.
Revised: Under this function design, maximizing J CPC leads the representations of related pairs to be close and representations of unrelated pairs to be distant.

Original: The category of modeling D KL ( P XY (cid:107) P X P Y ) also includes the Donsker-Varadhan objective ( J DV (Donsker & Varadhan, 1975; Belghazi et al., 2018)) and the Nguyen-Wainright-Jordan objective ( J NWJ (Nguyen et al., 2010; Belghazi et al., 2018)), where Belghazi et al.
Revised: The category of modeling D KL ( P XY (cid:107) P X P Y ) also includes the Donsker-Varadhan objective ( J DV (Donsker & Varadhan, 1975; Belghazi et al., 2018)) and the Nguyen-Wainright-Jordan objective ( J NWJ (Nguyen et al., 2010; Belghazi et al., 2018)), where Belghazi et al.

Original: ; Tsai et al.
Revised: ; Tsai et al.

Original: (2020b) show that J DV ( X, Y ) =
Revised: (2020b) show that J DV ( X, Y ) =

Original: J NWJ ( X, Y ) =
Revised: J NWJ ( X, Y ) =

Original: D KL ( P XY (cid:107) P X P Y ) .
Revised: D KL ( P XY (cid:107) P X P Y ) .

Original: The other divergence measurements considered in prior work are D JS ( P XY (cid:107) P X P Y ) (with D JS referring to the
Revised: The other divergence measurements considered in prior work are D JS ( P XY (cid:107) P X P Y ) (with D JS referring to the

Original: Jenson-Shannon divergence) and D Wass ( P XY (cid:107) P X P Y ) (with D Wass referring to the Wassersteindivergence).
Revised: Jenson-Shannon divergence) and D Wass ( P XY (cid:107) P X P Y ) (with D Wass referring to the Wassersteindivergence).

Original: The instance of modeling D JS ( P XY (cid:107) P X P Y ) is the Jensen-Shannon f-GAN objective (cid:0)
Revised: The instance of modeling D JS ( P XY (cid:107) P X P Y ) is the Jensen-Shannon f-GAN objective (cid:0)

Original: J JS (Nowozin et al., 2016; Hjelm et al., 2018) (cid:1) , where J JS ( X, Y ) =
Revised: J JS (Nowozin et al., 2016; Hjelm et al., 2018) (cid:1) , where J JS ( X, Y ) =

Original: 2 (cid:0) D JS ( P XY (cid:107) P X P Y ) − log 2 (cid:1) .
Revised: 2 (cid:0) D JS ( P XY (cid:107) P X P Y ) − log 2 (cid:1) .

Original: The instance of modeling D Wass ( P XY (cid:107) P X P Y ) is the Wasserstein Predictive Coding (cid:0)
Revised: The instance of modeling D Wass ( P XY (cid:107) P X P Y ) is the Wasserstein Predictive Coding (cid:0)

Original: J WPC (Ozair et al., 2019) (cid:1) , where J WPC ( X, Y ) modiﬁes J CPC ( X, Y ) objective (equation 1) by searching the function from F to F L .
Revised: J WPC (Ozair et al., 2019) (cid:1) , where J WPC ( X, Y ) modiﬁes J CPC ( X, Y ) objective (equation 1) by searching the function from F to F L .

Original: F L denotes any class of 1-Lipschitz continuous functions from ( X × Y ) to R , and thus F L ⊂ F .
Revised: F L denotes any class of 1-Lipschitz continuous functions from ( X × Y ) to R , and thus F L ⊂ F .

Original: Ozair et al.
Revised: Ozair et al.

Original: (2019) shows that J WPC ( X, Y ) is the lower bound of both D KL ( P XY (cid:107) P X P Y ) and
Revised: (2019) shows that J WPC ( X, Y ) is the lower bound of both D KL ( P XY (cid:107) P X P Y ) and

Original: D Wass ( P XY (cid:107) P X P Y ) .
Revised: D Wass ( P XY (cid:107) P X P Y ) .

Original: See Table 1 for all the equations.
Revised: See Table 1 for all the equations.

Original: To conclude, the contrastive representation learning objectives are unsupervised representation learning methods that maximize the distribution divergence between P XY and P X P Y .
Revised: To conclude, the contrastive representation learning objectives are unsupervised representation learning methods that maximize the distribution divergence between P XY and P X P Y .

Original: The learned representations cause high contrastiveness, and recent work (Arora et al., 2019; Tsai et al., 2020a) theoretically show that highly-contrastive representations could improve the performance on downstream tasks.
Revised: The learned representations cause high contrastiveness, and recent work (Arora et al., 2019; Tsai et al., 2020a) theoretically show that highly-contrastive representations could improve the performance on downstream tasks.

Original: After discussing prior contrastive representation learning objectives, we point out three challenges in their practical deployments: training stability, sensitivity to minibatch training size, and downstream task performance.
Revised: After discussing prior contrastive representation learning objectives, we point out three challenges in their practical deployments: training stability, sensitivity to minibatch training size, and downstream task performance.

Original: In particular, the three challenges can hardly be handled well at the same time, where we highlight the conclusions in Table 1.
Revised: In particular, the three challenges can hardly be handled well at the same time, where we highlight the conclusions in Table 1.

Original: Training Stability: The training stability highly relates to the variance of the objectives, where Song & Ermon (2019) shows that J DV and J NWJ exhibit inevitable high variance due to their inclusion of exponential function.
Revised: Training Stability: The training stability highly relates to the variance of the objectives, where Song & Ermon (2019) shows that J DV and J NWJ exhibit inevitable high variance due to their inclusion of exponential function.

Original: As pointed out by Tsai et al. (2020b), J CPC , J WPC , and J JS have better training stability because J CPC and J WPC can be realized as a multi-class classiﬁcation task and J JS can be realized as a binary classiﬁcation task.
Revised: As pointed out by Tsai et al. (2020b), J CPC , J WPC , and J JS have better training stability because J CPC and J WPC can be realized as a multi-class classiﬁcation task and J JS can be realized as a binary classiﬁcation task.

Original: The cross-entropy loss adopted in J CPC , J WPC , and J JS is highly-optimized and stable in existing optimization package (Abadi et al., 2016; Paszke et al., 2019).
Revised: The cross-entropy loss adopted in J CPC , J WPC , and J JS is highly-optimized and stable in existing optimization package (Abadi et al., 2016; Paszke et al., 2019).

Original: Sensitivity to minibatch training size: Among all the prior contrastive representation learning methods, J CPC is known to be sensitive to the minibatch training size (Ozair et al., 2019).
Revised: Sensitivity to minibatch training size: Among all the prior contrastive representation learning methods, J CPC is known to be sensitive to the minibatch training size (Ozair et al., 2019).

Original: Taking a closer look at equation 1,
Revised: Taking a closer look at equation 1,

Original: J CPC deploys an instance selection such that y 1 should be selected from { y 1 , y 2 , · · · , y N } , with ( x, y 1 ) ∼ P XY , ( x, y j> 1 ) ∼ P X P Y with N being the minibatch size.
Revised: J CPC deploys an instance selection such that y 1 should be selected from { y 1 , y 2 , · · · , y N } , with ( x, y 1 ) ∼ P XY , ( x, y j> 1 ) ∼ P X P Y with N being the minibatch size.

Original: Previous work (Poole et al., 2019; Song & Ermon, 2019; Chen et al., 2020b; Caron et al., 2020) showed that a large N results in a more challenging instance selection and forces J CPC to have a better contrastiveness of y 1 (related instance for x ) against { y j } N j =2 (unrelated instance for x ).
Revised: Previous work (Poole et al., 2019; Song & Ermon, 2019; Chen et al., 2020b; Caron et al., 2020) showed that a large N results in a more challenging instance selection and forces J CPC to have a better contrastiveness of y 1 (related instance for x ) against { y j } N j =2 (unrelated instance for x ).

Original: J DV , J NWJ , and J JS do not consider the instance selection, and J WPC reduces the minibatch training size sensitivity by enforcing 1Lipschitz constraint.
Revised: J DV , J NWJ , and J JS do not consider the instance selection, and J WPC reduces the minibatch training size sensitivity by enforcing 1Lipschitz constraint.

Original: Downstream Task Performance:
Revised: Downstream Task Performance:

Original: The downstream task performance is what we care the most among all the three challenges.
Revised: The downstream task performance is what we care the most among all the three challenges.

Original: J CPC has been the most popular objective as it manifests superior performance over the other alternatives (Tschannen et al., 2019; Tsai et al., 2020b;a).
Revised: J CPC has been the most popular objective as it manifests superior performance over the other alternatives (Tschannen et al., 2019; Tsai et al., 2020b;a).

Original: We note that although J WPC shows better performance on Omniglot (Lake et al., 2015) and CelebA (Liu et al., 2015) datasets, we empirically ﬁnd it not generalizing well to CIFAR-10/- 100 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015).
Revised: We note that although J WPC shows better performance on Omniglot (Lake et al., 2015) and CelebA (Liu et al., 2015) datasets, we empirically ﬁnd it not generalizing well to CIFAR-10/- 100 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015).

Original: 2. R ELATIVE P REDICTIVE C ODING
Revised: 2. R ELATIVE P REDICTIVE C ODING

Original: In this paper, we present Relative Predictive Coding (RPC), which achieves a good balance among the three challenges mentioned above:
Revised: In this paper, we present Relative Predictive Coding (RPC), which achieves a good balance among the three challenges mentioned above:

Original: [Equation]
Revised: [Equation]

Original: where α > 0 , β > 0 , γ > 0 are hyper-parameters and we deﬁne them as relative parameters .
Revised: where α > 0 , β > 0 , γ > 0 are hyper-parameters and we deﬁne them as relative parameters .

Original: Intuitively, J RPC contains no logarithm or exponential, potentially preventing unstable training due to numerical issues.
Revised: Intuitively, J RPC contains no logarithm or exponential, potentially preventing unstable training due to numerical issues.

Original: Now, we discuss the roles of α, β, γ .
Revised: Now, we discuss the roles of α, β, γ .

Original: At a ﬁrst glance, α acts to discourage the scores of P XY and P X P Y from being close, and β/γ acts as a (cid:96) 2 regularization coefﬁcient to stop f from becoming large.
Revised: At a ﬁrst glance, α acts to discourage the scores of P XY and P X P Y from being close, and β/γ acts as a (cid:96) 2 regularization coefﬁcient to stop f from becoming large.

Original: For a deeper analysis, the relative parameters act to regularize our objective for boundedness and low variance.
Revised: For a deeper analysis, the relative parameters act to regularize our objective for boundedness and low variance.

Original: To show this claim, we ﬁrst present the following lemma:
Revised: To show this claim, we ﬁrst present the following lemma:

Original: Lemma 1 (Optimal Solution for J RPC ) Let r ( x, y )
Revised: Lemma 1 (Optimal Solution for J RPC ) Let r ( x, y )

Original: = p ( x,y ) p ( x ) p ( y ) be the density ratio.
Revised: = p ( x,y ) p ( x ) p ( y ) be the density ratio.

Original: J RPC has the optimal solution f ∗ ( x, y )
Revised: J RPC has the optimal solution f ∗ ( x, y )

Original: =
Revised: =

Original: r ( x,y ) − α β r
Revised: r ( x,y ) − α β r

Original: ( x,y )
Revised: ( x,y )

Original: + γ :=
Revised: + γ :=

Original: r α,β,γ ( x, y ) with − αγ ≤ r α,β,γ ≤ 1 β .
Revised: r α,β,γ ( x, y ) with − αγ ≤ r α,β,γ ≤ 1 β .

Original: Lemma 1 suggests that J RPC achieves its supreme value at the ratio r α,β,γ ( x, y ) indexed by the relative parameters α, β, γ .
Revised: Lemma 1 suggests that J RPC achieves its supreme value at the ratio r α,β,γ ( x, y ) indexed by the relative parameters α, β, γ .

Original: We note that r α,β,γ ( x, y ) is an increasing function w.r.t.
Revised: We note that r α,β,γ ( x, y ) is an increasing function w.r.t.

Original: r ( x, y ) and is nicely bounded even when r ( x, y ) is large.
Revised: r ( x, y ) and is nicely bounded even when r ( x, y ) is large.

Original: We will now show that the bounded r α,β,γ suggests the empirical estimation of J RPC has boundeness and low variance.
Revised: We will now show that the bounded r α,β,γ suggests the empirical estimation of J RPC has boundeness and low variance.

Original: In particular, let { x i , y i } ni =1 be n samples drawn uniformly at random from P XY and { x (cid:48)
Revised: In particular, let { x i , y i } ni =1 be n samples drawn uniformly at random from P XY and { x (cid:48)

Original: j , y (cid:48)
Revised: j , y (cid:48)

Original: j } mj =1 be m samples drawn uniformly at random from P X P Y .
Revised: j } mj =1 be m samples drawn uniformly at random from P X P Y .

Original: Then, we use neural networks to empirically estimate J RPC as ˆ J m,n RPC :
Revised: Then, we use neural networks to empirically estimate J RPC as ˆ J m,n RPC :

Original: Deﬁnition 1 ( ˆ J m,n RPC , empirical estimation of J RPC )
Revised: Deﬁnition 1 ( ˆ J m,n RPC , empirical estimation of J RPC )

Original: We parametrize f via a family of neural networks F Θ :=
Revised: We parametrize f via a family of neural networks F Θ :=

Original: { f θ : θ ∈ Θ ⊆ R d } where d ∈ N and Θ is compact.
Revised: { f θ : θ ∈ Θ ⊆ R d } where d ∈ N and Θ is compact.

Original: Then, ˆ J m,n RPC =
Revised: Then, ˆ J m,n RPC =

Original: [Equation]
Revised: [Equation]

Original: Proposition 1 (Boundedness of ˆ J m,n RPC , informal) 

Revised: Proposition 1 (Boundedness of ˆ J m,n RPC , informal)

Original: ≤ J RPC ≤ 12 β + α 2 2 γ .
Revised: ≤ J RPC ≤ 12 β + α 2 2 γ .

Original: Then, with probability at least 1 − δ , | J RPC − ˆ J m,n RPC |
Revised: Then, with probability at least 1 − δ , | J RPC − ˆ J m,n RPC |

Original: =
Revised: =

Original: O ( (cid:113) d
Revised: O ( (cid:113) d

Original: +log(1 /δ ) n (cid:48) ) , where n (cid:48)
Revised: +log(1 /δ ) n (cid:48) ) , where n (cid:48)

Original: = min { n, m } .
Revised: = min { n, m } .

Original: Proposition 2 (Variance of ˆ J m,n RPC , informal)
Revised: Proposition 2 (Variance of ˆ J m,n RPC , informal)

Original: There exist universal constants c 1 and c 2 that depend only on α, β, γ , such that Var[ ˆ J m,n RPC ]
Revised: There exist universal constants c 1 and c 2 that depend only on α, β, γ , such that Var[ ˆ J m,n RPC ]

Original: [Equation]
Revised: [Equation]

Original: From the two propositions, when m and n are large, i.e., the sample sizes are large, ˆ J m,n RPC is bounded, and its variance vanishes to 0 .
Revised: From the two propositions, when m and n are large, i.e., the sample sizes are large, ˆ J m,n RPC is bounded, and its variance vanishes to 0 .

Original: First, the boundedness of ˆ J m,n RPC suggests ˆ J m,n RPC will not grow to extremely large or small values.
Revised: First, the boundedness of ˆ J m,n RPC suggests ˆ J m,n RPC will not grow to extremely large or small values.

Original: Prior contrastive learning objectives with good training stability (e.g.,
Revised: Prior contrastive learning objectives with good training stability (e.g.,

Original: J CPC / J JS / J WPC ) also have the boundedness of their objective values.
Revised: J CPC / J JS / J WPC ) also have the boundedness of their objective values.

Original: For instance, the empirical estimation of J CPC is less than log N (equation 1) (Poole et al., 2019).
Revised: For instance, the empirical estimation of J CPC is less than log N (equation 1) (Poole et al., 2019).

Original: Second, the upper bound of the variance implies the training of ˆ J m,n RPC can be stable, and in practice we observe a much smaller value than the stated upper bound.
Revised: Second, the upper bound of the variance implies the training of ˆ J m,n RPC can be stable, and in practice we observe a much smaller value than the stated upper bound.

Original: On the contrary, Song & Ermon (2019) shows that the empirical estimations of J DV and J NWJ exhibit inevitable variances that grow exponentially with the true
Revised: On the contrary, Song & Ermon (2019) shows that the empirical estimations of J DV and J NWJ exhibit inevitable variances that grow exponentially with the true

Original: D KL ( P XY (cid:107) P X P Y ) .
Revised: D KL ( P XY (cid:107) P X P Y ) .

Original: Lastly, similar to prior contrastive learning objective that are related to distribution divergence measurement, we associate J RPC with the Chi-square divergence D χ 2 ( P XY (cid:107) P X P Y ) =
Revised: Lastly, similar to prior contrastive learning objective that are related to distribution divergence measurement, we associate J RPC with the Chi-square divergence D χ 2 ( P XY (cid:107) P X P Y ) =

Original: E
Revised: E

Original: P X P Y [ r 2 ( x, y )] − 1 (Nielsen & Nock, 2013).
Revised: P X P Y [ r 2 ( x, y )] − 1 (Nielsen & Nock, 2013).

Original: The derivations are provided in Appendix.
Revised: The derivations are provided in Appendix.

Original: By having P (cid:48)
Revised: By having P (cid:48)

Original: =
Revised: =

Original: ββ + γ P XY + γβ + γ P X P Y as the mixture distribution of P XY and P X P Y , we can rewrite
Revised: ββ + γ P XY + γβ + γ P X P Y as the mixture distribution of P XY and P X P Y , we can rewrite

Original: J RPC ( X, Y ) as J RPC ( X, Y ) =
Revised: J RPC ( X, Y ) as J RPC ( X, Y ) =

Original: β + γ 2 E P (cid:48)
Revised: β + γ 2 E P (cid:48)

Original: [ r 2 α,β,γ ( x, y )] .
Revised: [ r 2 α,β,γ ( x, y )] .

Original: Hence, J RPC can be regarded as a generalization of D χ 2 with the relative parameters α, β, γ , where D χ 2 can be recovered from J RPC by specializing α = 0 , β = 0 and γ = 1 (e.g., D χ 2 = 2 J RPC | α =
Revised: Hence, J RPC can be regarded as a generalization of D χ 2 with the relative parameters α, β, γ , where D χ 2 can be recovered from J RPC by specializing α = 0 , β = 0 and γ = 1 (e.g., D χ 2 = 2 J RPC | α =

Original: β =0 ,γ =1 − 1 ).
Revised: β =0 ,γ =1 − 1 ).

Original: Note that J RPC may not be a formal divergence measure with arbitrary α, β, γ .
Revised: Note that J RPC may not be a formal divergence measure with arbitrary α, β, γ .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We provide an overview of the experimental section.
Revised: We provide an overview of the experimental section.

Original: First, we conduct benchmark self-supervised representation learning tasks spanning visual object classiﬁcation and speech recognition.
Revised: First, we conduct benchmark self-supervised representation learning tasks spanning visual object classiﬁcation and speech recognition.

Original: This set of experiments are designed to discuss the three challenges of the contrastive representation learning objectives: downstream task performance (Section 3.1), training stability (Section 3.2), and minibatch size sensitivity (Section 3.3).
Revised: This set of experiments are designed to discuss the three challenges of the contrastive representation learning objectives: downstream task performance (Section 3.1), training stability (Section 3.2), and minibatch size sensitivity (Section 3.3).

Original: We also provide an ablation study on the choices of the relative parameters in J RPC (Section 3.4).
Revised: We also provide an ablation study on the choices of the relative parameters in J RPC (Section 3.4).

Original: Second, we relate J RPC with mutual information (MI) estimation (Section 3.5).
Revised: Second, we relate J RPC with mutual information (MI) estimation (Section 3.5).

Original: The connection is that MI is an average statistic of the density ratio, and we have shown that the optimal solution of J RPC is the relative density ratio (see Lemma 1).
Revised: The connection is that MI is an average statistic of the density ratio, and we have shown that the optimal solution of J RPC is the relative density ratio (see Lemma 1).

Original: Thus we could estimate MI using the density ratio transformed from the optimal solution of J RPC .
Revised: Thus we could estimate MI using the density ratio transformed from the optimal solution of J RPC .

Original: On these two sets of experiments, we fairly compare J RPC with other contrastive learning objectives.
Revised: On these two sets of experiments, we fairly compare J RPC with other contrastive learning objectives.

Original: Particularly, across different objectives, we ﬁx the network, learning rate, optimizer, and batch size (we use the default conﬁgurations suggested by the original implementations from Chen et al.
Revised: Particularly, across different objectives, we ﬁx the network, learning rate, optimizer, and batch size (we use the default conﬁgurations suggested by the original implementations from Chen et al.

Original: (2020c), Rivi`ere et al.
Revised: (2020c), Rivi`ere et al.

Original: (2020) and Tsai et al.
Revised: (2020) and Tsai et al.

Original: (2020b).)
Revised: (2020b).)

Original: The only difference will be the objective itself.
Revised: The only difference will be the objective itself.

Original: In what follows, we perform the ﬁrst set of experiments.
Revised: In what follows, we perform the ﬁrst set of experiments.

Original: We defer experimental details in the Appendix.
Revised: We defer experimental details in the Appendix.

Original: Datasets.
Revised: Datasets.

Original: For the visual objective classiﬁcation, we consider CIFAR-10/-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet (Russakovsky et al., 2015).
Revised: For the visual objective classiﬁcation, we consider CIFAR-10/-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet (Russakovsky et al., 2015).

Original: CIFAR-10/- and ImageNet contain labeled images only, while STL-10 contains labeled and unlabeled images.
Revised: CIFAR-10/- and ImageNet contain labeled images only, while STL-10 contains labeled and unlabeled images.

Original: For the speech recognition, we consider LibriSpeech-100h (Panayotov et al., 2015) dataset, which contains 100 hours of 16kHz English speech from 251 speakers with 41 types of phonemes.
Revised: For the speech recognition, we consider LibriSpeech-100h (Panayotov et al., 2015) dataset, which contains 100 hours of 16kHz English speech from 251 speakers with 41 types of phonemes.

Original: Training and Evaluation Details.
Revised: Training and Evaluation Details.

Original: For the vision experiments, we follow the setup from SimCLRv2 (Chen et al., 2020c), which considers visual object recognition as its downstream task.
Revised: For the vision experiments, we follow the setup from SimCLRv2 (Chen et al., 2020c), which considers visual object recognition as its downstream task.

Original: For the speech experiments, we follow the setup from prior work (Oord et al., 2018; Rivi ` ere et al., 2020), which consider phoneme classiﬁcation and speaker identiﬁcation as the downstream tasks.
Revised: For the speech experiments, we follow the setup from prior work (Oord et al., 2018; Rivi ` ere et al., 2020), which consider phoneme classiﬁcation and speaker identiﬁcation as the downstream tasks.

Original: Then, we brieﬂy discuss the training and evaluation details into three modules: 1) related and unrelated data construction, 2) pre-training, and 3) ﬁne-tuning and evaluation.
Revised: Then, we brieﬂy discuss the training and evaluation details into three modules: 1) related and unrelated data construction, 2) pre-training, and 3) ﬁne-tuning and evaluation.

Original: For more details, please refer to
Revised: For more details, please refer to

Original: Appendix or the original implementations.
Revised: Appendix or the original implementations.

Original: (cid:46) Related and Unrelated Data Construction.
Revised: (cid:46) Related and Unrelated Data Construction.

Original: In the vision experiment, we construct the related images by applying different augmentations on the same image.
Revised: In the vision experiment, we construct the related images by applying different augmentations on the same image.

Original: Hence, when ( x, y ) ∼ P XY , x and y are the same image with different augmentations.
Revised: Hence, when ( x, y ) ∼ P XY , x and y are the same image with different augmentations.

Original: The unrelated images are two randomly selected samples.
Revised: The unrelated images are two randomly selected samples.

Original: In the speech experiment, we deﬁne the current latent feature (feature at time t ) and the future samples (samples at time > t ) as related data.
Revised: In the speech experiment, we deﬁne the current latent feature (feature at time t ) and the future samples (samples at time > t ) as related data.

Original: In other words, the feature in the latent space should contain information that can be used to infer future time steps.
Revised: In other words, the feature in the latent space should contain information that can be used to infer future time steps.

Original: The latent feature of randomly sampled data would be considered as unrelated data.
Revised: The latent feature of randomly sampled data would be considered as unrelated data.

Original: (cid:46)
Revised: (cid:46)

Original: Pre-training.
Revised: Pre-training.

Original: The pre-training stage refers to the self-supervised training by a contrastive learning objective.
Revised: The pre-training stage refers to the self-supervised training by a contrastive learning objective.

Original: Our training objective is deﬁned in Deﬁnition 1, where we use neural networks to parametrize the function using the constructed related and unrelated data.
Revised: Our training objective is deﬁned in Deﬁnition 1, where we use neural networks to parametrize the function using the constructed related and unrelated data.

Original: Convolutional neural networks are used for vision experiments.
Revised: Convolutional neural networks are used for vision experiments.

Original: Transformers (Vaswani et al., 2017) and LSTMs (Hochreiter & Schmidhuber, 1997) are used for speech experiments.
Revised: Transformers (Vaswani et al., 2017) and LSTMs (Hochreiter & Schmidhuber, 1997) are used for speech experiments.

Original: (cid:46) Fine-tuning and Evaluation.
Revised: (cid:46) Fine-tuning and Evaluation.

Original: After the pre-training stage, we ﬁx the parameters in the pre-trained networks and add a small ﬁne-tuning network on top of them.
Revised: After the pre-training stage, we ﬁx the parameters in the pre-trained networks and add a small ﬁne-tuning network on top of them.

Original: Then, we ﬁne-tune this small network with the downstream labels in the data’s training split.
Revised: Then, we ﬁne-tune this small network with the downstream labels in the data’s training split.

Original: For the ﬁne-tuning network, both vision and speech experiments consider multi-layer perceptrons.
Revised: For the ﬁne-tuning network, both vision and speech experiments consider multi-layer perceptrons.

Original: Last, we evaluate the ﬁne-tuned representations on the data’s test split.
Revised: Last, we evaluate the ﬁne-tuned representations on the data’s test split.

Original: 3. D OWNSTREAM T ASK P ERFORMANCES ON V ISION AND S PEECH
Revised: 3. D OWNSTREAM T ASK P ERFORMANCES ON V ISION AND S PEECH

Original: For the downstream task performance in the vision domain, we test the proposed J RPC and other contrastive learning objectives on CIFAR-10/-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet ILSVRC-2012 (Russakovsky et al., 2015).
Revised: For the downstream task performance in the vision domain, we test the proposed J RPC and other contrastive learning objectives on CIFAR-10/-100 (Krizhevsky et al., 2009), STL-10 (Coates et al., 2011), and ImageNet ILSVRC-2012 (Russakovsky et al., 2015).

Original:  Table 2 shows that the proposed J RPC outperforms otherobjectives on all datasets. Using J RPC on the largest network (ResNet with
Revised: Here we report the best performances J RPC can get on each dataset. Table 2 shows that the proposed J RPC outperforms other

Original: [Table]
Revised: [Table]

Original: results.
Revised: [Figure] [Figure] [Figure] [Figure]

Original: We also provide the results from fully supervised model as a comparison (Oord et al., 2018).
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Figure]

Original:  depth of 152 , channel width of 2 and selective kernels), the performance jumps from 77 .
Revised: objectives on all datasets. Using J RPC on the largest network (ResNet with depth of 152 , channel width of 2 and selective kernels), the performance jumps from 77 .

Original: 80% of J CPC to 78 .
Revised: 80% of J CPC to 78 .

Original: 40% of J RPC .
Revised: 40% of J RPC .

Original: Regarding speech representation learning, the downstream performance for phoneme and speaker classiﬁcation are shown in Table 3.
Revised: Regarding speech representation learning, the downstream performance for phoneme and speaker classiﬁcation are shown in Table 3.

Original: J RPC improves the phoneme classiﬁcation results witharound 4 . 5 percent and the speaker classiﬁcation results witharound 0 .
Revised: J RPC improves the phoneme classiﬁcation results with 4 .percent and the speaker classiﬁcation results with 0 .

Original: 3 percent, which is closer to the fully supervised model.
Revised: 3 percent, which is closer to the fully supervised model.

Original: Overall, the proposed J RPC performs better than other unsupervised learning objectives on both phoneme classiﬁcation and speaker classiﬁcation tasks.
Revised: Overall, the proposed J RPC performs better than other unsupervised learning objectives on both phoneme classiﬁcation and speaker classiﬁcation tasks.

Original: 3. T RAINING S TABILITY
Revised: 3. T RAINING S TABILITY

Original: We provide empirical training stability comparisons on J DV , J NWJ , J CPC and J RPC by plotting the values of the objectives as the training step increases.
Revised: We provide empirical training stability comparisons on J DV , J NWJ , J CPC and J RPC by plotting the values of the objectives as the training step increases.

Original: We apply the four objectives to the SimCLRv framework and train on the CIFAR-10 dataset.
Revised: We apply the four objectives to the SimCLRv framework and train on the CIFAR-10 dataset.

Original: All setups of training are exactly the same except the objectives.
Revised: All setups of training are exactly the same except the objectives.

Original: From our experiments, J DV and J NWJ soon explode to NaN and disrupt training (shown as early stopping in Figure 1a; extremely large values are not plotted due to scale constraints).
Revised: From our experiments, J DV and J NWJ soon explode to NaN and disrupt training (shown as early stopping in Figure 1a; extremely large values are not plotted due to scale constraints).

Original: On the other hand, J RPC and J CPC has low variance, and both enjoy stable training.
Revised: On the other hand, J RPC and J CPC has low variance, and both enjoy stable training.

Original: 3. M INIBATCH S IZE S ENSITIVITY
Revised: 3. M INIBATCH S IZE S ENSITIVITY

Original: We then provide the analysis on the effect of minibatch size on J RPC and J CPC , since J CPC is known to be sensitive to minibatch size (Poole et al., 2019).
Revised: We then provide the analysis on the effect of minibatch size on J RPC and J CPC , since J CPC is known to be sensitive to minibatch size (Poole et al., 2019).

Original: We train SimCLRv

Revised: We train SimCLRv

Original: (Chen et al., 2020c) on CIFAR-10 and the model from Rivi`ere et al.
Revised: (Chen et al., 2020c) on CIFAR-10 and the model from Rivi ` ere et al.

Original: (2020) on LibriSpeech-100h using J RPC and J CPC with different minibatch sizes.
Revised: (2020) on LibriSpeech-100h using J RPC and J CPC with different minibatch sizes.

Original: The settings of relative parameters are the same as Section 3.2.
Revised: The settings of relative parameters are the same as Section 3.2.

Original: From
Revised: From

Original: Figure 1b and 1c, we can observe that both J RPC and J CPC achieve their optimal performance at a large minibatch size.
Revised: Figure 1b and 1c, we can observe that both J RPC and J CPC achieve their optimal performance at a large minibatch size.

Original: However, when the minibatch size decreases, the performance of J CPC shows higher sensitivity and suffers more when the number of minibatch samples is small.
Revised: However, when the minibatch size decreases, the performance of J CPC shows higher sensitivity and suffers more when the number of minibatch samples is small.

Original: The result suggests that the proposed method might be less sensitive to the change of minibatch size compared to J CPC given the same training settings.
Revised: The result suggests that the proposed method might be less sensitive to the change of minibatch size compared to J CPC given the same training settings.

Original: 3. E FFECT OF R ELATIVE P ARAMETERS
Revised: 3. E FFECT OF R ELATIVE P ARAMETERS

Original: [Table]

Original: Wealso study the effect of different combinations of relative parameters in J RPC by showing downstream performance on visual object recognition.
Revised: We study the effect of different combinations of relative parameters in J RPC by comparing downstream performances on visual object recognition.

Original: We train SimCLRv on CIFAR-10 with different combination of α, β and γ in J RPC . All other experimental settings are identical.
Revised: We train SimCLRv2 on CIFAR-10 with different combinations of α, β and γ in J RPC and ﬁx all other experimental settings. We choose α ∈ { 0 , 0 .

Original: We ﬁnd that in Equation 2, a small β (in a scale of 10 − 3 ) anda large γ(close to 1 .  0 ) are crucial for training a high-performing encoder which extracts useful information for downstream tasks.
Revised: 001 , 1 . 0 } , β ∈ { 0 , 0 . 001 , 1 . 0 } , γ ∈ { 0 , 0 . and we report the best performances under each combination of α, β , and γ . From Figure 2, we ﬁrst observe that α > 0 has better downstream performance than α = 0 when β and γ are ﬁxed.

Original:  α = 0 inﬂuences less to  the downstream performance unless being equal to 0 .
Revised: This observation is as expected, since α > 0 encourages representations of related and unrelated samples to be pushed away. Then, we ﬁnd that a small but nonzero β ( β = 0 . 001 ) and a large γ ( γ = 1 . 0 ) give the best performance compared to other combinations.

Original: The major reason is that in the proposed J RPC , β and γ serve as  regularization coefﬁcients.
Revised: Since β and γ serve as the coefﬁcients of (cid:96) 2 regularization, the results imply that the regularization is a strong and sensitive factor that will inﬂuence the performance.

Original: α is less effective because it only regularizes a linear function of the negative terms, thus the gradients during backpropagation will be less dominant than the gradients from the quadratic term regularized by β . The results here are not as competitive as  the results in Table 2 because they are using different relative parameters.
Revised: The results here are not as competitive as Table 2 because the result reported in Table 2 is using a set of relative parameters ( α = 1 . 0 , β = 0 . 005 , γ = 1 . 0 ) that is different from the combinations in this subsection. In conclusion, we ﬁnd empirically that a non-zero α , a small β and a large γ will lead to the optimal representation for the downstream task.

Original: 3. R ELATION TO M UTUAL I NFORMATION E STIMATION
Revised: 3. R ELATION TO M UTUAL I NFORMATION E STIMATION

Original: The presented approach also closely relates to mutual information estimation.
Revised: The presented approach also closely relates to mutual information estimation.

Original: For random variables
Revised: For random variables

Original: X and Y with joint distribution P XY and product of marginals P X P Y , the mutual information is deﬁned as I ( X ; Y ) =
Revised: X and Y with joint distribution P XY and product of marginals P X P Y , the mutual information is deﬁned as I ( X ; Y ) =

Original: D KL ( P XY (cid:107) P X P Y ) .
Revised: D KL ( P XY (cid:107) P X P Y ) .

Original: Lemma 1 states that given optimal solution f ∗ ( x, y ) of
Revised: Lemma 1 states that given optimal solution f ∗ ( x, y ) of

Original: J RPC , we can get the density ratio r ( x, y ) :
Revised: J RPC , we can get the density ratio r ( x, y ) :

Original: = p ( x, y ) /p
Revised: = p ( x, y ) /p

Original: ( x ) p ( y ) as r ( x, y ) =
Revised: ( x ) p ( y ) as r ( x, y ) =

Original: γ/β + α 1 − βf ∗ ( x,y ) − γβ .
Revised: γ/β + α 1 − βf ∗ ( x,y ) − γβ .

Original: We can empirically estimate ˆ r ( x, y ) from the estimated ˆ f ( x, y ) via this transformation, and use ˆ r ( x, y ) to estimate mutual information (Tsai et al., 2020b).
Revised: We can empirically estimate ˆ r ( x, y ) from the estimated ˆ f ( x, y ) via this transformation, and use ˆ r ( x, y ) to estimate mutual information (Tsai et al., 2020b).

Original: Speciﬁcally, I ( X ; Y )
Revised: Speciﬁcally, I ( X ; Y )

Original: ≈ 1 n (cid:80) ni
Revised: ≈ 1 n (cid:80) ni

Original: =1 log ˆ r ( x i , y i ) with ( x i , y i ) ∼ P ⊗ nX,Y , where P ⊗ nX,Y is the uniformly sampled empirical distribution of P X,Y .
Revised: =1 log ˆ r ( x i , y i ) with ( x i , y i ) ∼ P ⊗ n X,Y , where P ⊗ n X,Y is the uniformly sampled empirical distribution of P X,Y .

Original: We follow prior work (Poole et al., 2019; Song & Ermon, 2019; Tsai et al., 2020b) for the experiments.
Revised: We follow prior work (Poole et al., 2019; Song & Ermon, 2019; Tsai et al., 2020b) for the experiments.

Original: We consider X and Y as two 20 -dimensional Gaussians with correlation ρ , and our goal is to estimate the mutual information
Revised: We consider X and Y as two 20 -dimensional Gaussians with correlation ρ , and our goal is to estimate the mutual information

Original: I ( X ; Y ) .
Revised: I ( X ; Y ) .

Original: Then, we perform a cubic transformation on y so that y (cid:55)→ y 3 .
Revised: Then, we perform a cubic transformation on y so that y (cid:55)→ y 3 .

Original: The ﬁrst task is referred to as Gaussian task and the second is referred to as Cubic task, where both have the ground truth
Revised: The ﬁrst task is referred to as Gaussian task and the second is referred to as Cubic task, where both have the ground truth

Original: I ( X ; Y ) =
Revised: I ( X ; Y ) =

Original: − 10log (1 − ρ 2 ) .
Revised: − 10log (1 − ρ 2 ) .

Original: The models are trained on 20 ,steps with I ( X ; Y ) starting at 2 and increased by 2 per 4 , 000 steps.
Revised: The models are trained on 20 ,steps with I ( X ; Y ) starting at 2 and increased by 2 per 4 , 000 steps.

Original: Our method is compared with baseline methods J CPC (Oord et al., 2018), J NWJ (Nguyen et al., 2010), J JS (Nowozin et al., 2016),
Revised: Our method is compared with baseline methods J CPC (Oord et al., 2018), J NWJ (Nguyen et al., 2010), J JS (Nowozin et al., 2016),

Original: SMILE (Song & Ermon, 2019) and Difference of Entropies (DoE) (McAllester & Stratos, 2020).
Revised: SMILE (Song & Ermon, 2019) and Difference of Entropies (DoE) (McAllester & Stratos, 2020).

Original: All approaches use the same network design, learning rate, optimizer and minibatch size for a fair comparison.
Revised: All approaches use the same network design, learning rate, optimizer and minibatch size for a fair comparison.

Original: First, we observe J CPC (Oord et al., 2018) has the smallest variance, while it exhibits a large bias (the estimated mutual information from J CPC has an upper bound log(batch size) ).
Revised: First, we observe J CPC (Oord et al., 2018) has the smallest variance, while it exhibits a large bias (the estimated mutual information from J CPC has an upper bound log(batch size) ).

Original: Second, J NWJ (Nguyen et al., 2010) and J JSD (Poole et al., 2019) have large variances, especially in the Cubic task.
Revised: Second, J NWJ (Nguyen et al., 2010) and J JSD (Poole et al., 2019) have large variances, especially in the Cubic task.

Original: Song & Ermon (2019) pointed out the limitations of J CPC , J NWJ , and J JSD , and developed the SMILE method, which clips the value of the estimated density function to reduce the variance of the estimators.
Revised: Song & Ermon (2019) pointed out the limitations of J CPC , J NWJ , and J JSD , and developed the SMILE method, which clips the value of the estimated density function to reduce the variance of the estimators.

Original: DoE (McAllester & Stratos, 2020) is neither a lower bound nor a upper bound of mutual information, but can achieve accurate estimates when underlying mutual information is large.
Revised: DoE (McAllester & Stratos, 2020) is neither a lower bound nor a upper bound of mutual information, but can achieve accurate estimates when underlying mutual information is large.

Original: J RPC exhibits comparable bias and lower variance compared to the SMILE method, and is more stable than the DoE method.
Revised: J RPC exhibits comparable bias and lower variance compared to the SMILE method, and is more stable than the DoE method.

Original: We would like to highlight our method’s low-variance property, where we neither clip the values of the estimated density ratio nor impose an upper bound of our estimated mutual information.
Revised: We would like to highlight our method’s low-variance property, where we neither clip the values of the estimated density ratio nor impose an upper bound of our estimated mutual information.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: As a subset of unsupervised representation learning, self-supervised representation learning (SSL) adopts self-deﬁned signals as supervision and uses the learned representation for downstream tasks, such as object detection and image captioning (Liu et al., 2020).
Revised: As a subset of unsupervised representation learning, self-supervised representation learning (SSL) adopts self-deﬁned signals as supervision and uses the learned representation for downstream tasks, such as object detection and image captioning (Liu et al., 2020).

Original: We categorize SSL work into two groups: when the signal is the input’s hidden property or the corresponding view of the input.
Revised: We categorize SSL work into two groups: when the signal is the input’s hidden property or the corresponding view of the input.

Original: For the ﬁrst group, for example, Jigsaw puzzle (Noroozi & Favaro, 2016) shufﬂes the image patches and deﬁnes the SSL task for predicting the shufﬂed positions of the image patches.
Revised: For the ﬁrst group, for example, Jigsaw puzzle (Noroozi & Favaro, 2016) shufﬂes the image patches and deﬁnes the SSL task for predicting the shufﬂed positions of the image patches.

Original: Other instances are
Revised: Other instances are

Original: Predicting Rotations (Gidaris et al., 2018) and Shufﬂe & Learn (Misra et al., 2016).
Revised: Predicting Rotations (Gidaris et al., 2018) and Shufﬂe & Learn (Misra et al., 2016).

Original: For the second group, the SSL task aims at modeling the co-occurrence of multiple views of data, via the contrastive or the predictive learning objectives (Tsai et al., 2020a).
Revised: For the second group, the SSL task aims at modeling the co-occurrence of multiple views of data, via the contrastive or the predictive learning objectives (Tsai et al., 2020a).

Original: The predictive objectives encourage reconstruction from one view of the data to the other, such as predicting the lower part of an image from its upper part (ImageGPT by Chen et al.
Revised: The predictive objectives encourage reconstruction from one view of the data to the other, such as predicting the lower part of an image from its upper part (ImageGPT by Chen et al.

Original: (2020a)).
Revised: (2020a)).

Original: Comparing the contrastive with predictive learning approaches, Tsai et al.
Revised: Comparing the contrastive with predictive learning approaches, Tsai et al.

Original: (2020a) points out that the former requires less computational resources for a good performance but suffers more from the over-ﬁtting problem.
Revised: (2020a) points out that the former requires less computational resources for a good performance but suffers more from the over-ﬁtting problem.

Original: Theoretical analysis (Arora et al., 2019; Tsai et al., 2020a;
Revised: Theoretical analysis (Arora et al., 2019; Tsai et al., 2020a;

Original: Tosh et al., 2020) suggests the contrastively learned representations can lead to a good downstream performance.
Revised: Tosh et al., 2020) suggests the contrastively learned representations can lead to a good downstream performance.

Original: Beyond the theory,
Revised: Beyond the theory,

Original: Tian et al.
Revised: Tian et al.

Original: (2020) shows what matters more for the performance are 1) the choice of the contrastive learning objective; and 2) the creation of the positive and negative data pairs in the contrastive objective.
Revised: (2020) shows what matters more for the performance are 1) the choice of the contrastive learning objective; and 2) the creation of the positive and negative data pairs in the contrastive objective.

Original: Recent work (Khosla et al., 2020) extends the usage of contrastive learning from the selfsupervised setting to the supervised setting.
Revised: Recent work (Khosla et al., 2020) extends the usage of contrastive learning from the selfsupervised setting to the supervised setting.

Original: The supervised setting deﬁnes the positive pairs as the data from the same class in the contrastive objective, while the self-supervised setting deﬁnes the positive pairs as the data with different augmentations.
Revised: The supervised setting deﬁnes the positive pairs as the data from the same class in the contrastive objective, while the self-supervised setting deﬁnes the positive pairs as the data with different augmentations.

Original: Our work also closely rates to the skewed divergence measurement between distributions (Lee, 1999; 2001; Nielsen, 2010; Yamada et al., 2013).
Revised: Our work also closely rates to the skewed divergence measurement between distributions (Lee, 1999; 2001; Nielsen, 2010; Yamada et al., 2013).

Original: Recall that the usage of the relative parameters plays a crucial role to regularize our objective for its boundness and low variance.
Revised: Recall that the usage of the relative parameters plays a crucial role to regularize our objective for its boundness and low variance.

Original: This idea is similar to the skewed divergence measurement, that when calculating the divergence between distributions P and Q , instead of considering D( P (cid:107) Q ) , these approaches consider D( P (cid:107) αP + (1 − α ) Q ) with
Revised: This idea is similar to the skewed divergence measurement, that when calculating the divergence between distributions P and Q , instead of considering D( P (cid:107) Q ) , these approaches consider D( P (cid:107) αP + (1 − α ) Q ) with

Original: D representing the divergence and 0 < α < 1 .
Revised: D representing the divergence and 0 < α < 1 .

Original: A natural example is that the Jensen-Shannon divergence is a symmetric skewed KL divergence: D JS ( P (cid:107) Q ) = 0 .
Revised: A natural example is that the Jensen-Shannon divergence is a symmetric skewed KL divergence: D JS ( P (cid:107) Q ) = 0 .

Original: 5 D KL ( P (cid:107) 0 .
Revised: 5 D KL ( P (cid:107) 0 .

Original: 5 P + 0 .
Revised: 5 P + 0 .

Original: 5 Q )
Revised: 5 Q )

Original: + 0 .
Revised: + 0 .

Original: 5 D KL ( Q (cid:107) 0 .
Revised: 5 D KL ( Q (cid:107) 0 .

Original: 5 P + 0 .
Revised: 5 P + 0 .

Original: 5 Q ) .
Revised: 5 Q ) .

Original: Compared to the non-skewed counterpart, the skewed divergence has shown to have a more robust estimation for its value (Lee, 1999; 2001; Yamada et al., 2013).
Revised: Compared to the non-skewed counterpart, the skewed divergence has shown to have a more robust estimation for its value (Lee, 1999; 2001; Yamada et al., 2013).

Original: Different from these works that focus on estimating the values of distribution divergence, we focus on learning self-supervised representations.
Revised: Different from these works that focus on estimating the values of distribution divergence, we focus on learning self-supervised representations.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we present RPC, the Relative Predictive Coding, that achieves a good balance among the three challenges when modeling a contrastive learning objective: training stability, sensitivity to minibatch size and downstream task performance.
Revised: In this work, we present RPC, the Relative Predictive Coding, that achieves a good balance among the three challenges when modeling a contrastive learning objective: training stability, sensitivity to minibatch size, and downstream task performance.

Original: We believe this work brings an appealing option for training self-supervised models and inspires future work to design objectives for balancing the aforementioned three challenges.
Revised: We believe this work brings an appealing option for training self-supervised models and inspires future work to design objectives for balancing the aforementioned three challenges.

Original: In the future, we are interested in applying RPC in other application domains and developing more principled approaches for better representation learning
Revised: In the future, we are interested in applying RPC in other application domains and developing more principled approaches for better representation learning.

Original: A A PPENDIX A. P ROOF 
Revised: A A PPENDIX A. P ROOF OF L EMMA 1 IN THE M AIN T EXT

Original: OF L EMMA 1 IN THE M AIN T EXT

Original: Lemma 2 (Optimal Solution for J RPC , restating Lemma 1 in the main text) Let 
Revised: Lemma 2 (Optimal Solution for J RPC , restating Lemma 1 in the main text) Let J

Original: [Equation]
Revised: [Equation]

Original: and r ( x, y )
Revised: and r ( x, y )

Original: = p ( x,y ) p ( x ) p ( y ) be the density ratio.
Revised: = p ( x,y ) p ( x ) p ( y ) be the density ratio.

Original: J RPC has the optimal solution
Revised: J RPC has the optimal solution

Original: [Equation]
Revised: [Equation]

Original: Proof: The second-order functional derivative of the objective is
Revised: Proof: The second-order functional derivative of the objective is

Original: [Equation]
Revised: [Equation]

Original: which is always negative.
Revised: which is always negative.

Original: The negative second-order functional derivative implies the objective has a supreme value.
Revised: The negative second-order functional derivative implies the objective has a supreme value.

Original: Then, take the ﬁrst-order functional derivative ∂J RPC ∂m and set it to zero:
Revised: Then, take the ﬁrst-order functional derivative ∂J RPC ∂m and set it to zero:

Original: [Equation]
Revised: [Equation]

Original: We then get
Revised: We then get

Original: [Equation]
Revised: [Equation]

Original: Since 0 ≤ r ( x, y ) ≤ ∞ [Equation] we have − αγ ≤ r ( x,y ) −
Revised: Since 0 ≤ r ( x, y ) ≤ ∞ , we have − αγ ≤ r ( x,y ) −

Original: α βr ( x,y )
Revised: α βr ( x,y )

Original: + γ ≤ 1 β .
Revised: + γ ≤ 1 β .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: γ ( x, y ) with − α γ ≤ r α,β,γ ≤ 1 β .

Original: (cid:3)

Original: A. R ELATION BETWEEN J RPC AND D χ
Revised: A. R ELATION BETWEEN J RPC AND D χ

Original: In this subsection, we aim to show the following: 1) D χ 2 ( P XY (cid:107) P X P Y ) =
Revised: In this subsection, we aim to show the following: 1) D χ 2 ( P XY (cid:107) P X P Y ) =

Original: E P X P Y [ r 2 ( x, y )] − 1 ; and 2) J RPC ( X, Y ) =
Revised: E P X P Y [ r 2 ( x, y )] − 1 ; and 2) J RPC ( X, Y ) =

Original: β + γ 2 E P (cid:48)
Revised: β + γ 2 E P (cid:48)

Original: [ r 2 α,β,γ ( x, y )] by having P (cid:48)
Revised: [ r 2 α,β,γ ( x, y )] by having P (cid:48)

Original: =
Revised: =

Original: ββ + γ P XY + γβ + γ P X P Y as the mixture distribution of P XY and P X P Y .
Revised: ββ + γ P XY + γβ + γ P X P Y as the mixture distribution of P XY and P X P Y .

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: Proof: By deﬁnition (Nielsen & Nock, 2013),
Revised: Proof: By deﬁnition (Nielsen & Nock, 2013),

Original: [Equation]
Revised: [Equation]

Original: Lemma 4 Deﬁning P (cid:48) =
Revised: Lemma 4 Deﬁning P (cid:48) =

Original: ββ + γ P XY + γβ + γ P X P Y as a mixture distribution of P XY and P X P Y ,
Revised: ββ + γ P XY + γβ + γ P X P Y as a mixture distribution of P XY and P X P Y ,

Original: J RPC [Equation]
Revised: J RPC ( X, Y ) =

Original: [Equation]
Revised: β + γ 2 E P (cid:48) [ r 2 α,β,γ ( x, y )] .

Original: Proof: Plug in the optimal solution f ∗ ( x, y ) =
Revised: Proof: Plug in the optimal solution f ∗ ( x, y ) =

Revised: dP X,Y − α · dP X P Y β · dP X,Y + γ · dP X P Y (see Lemma 2) into J RPC :

Original: [Equation]
Revised: [Equation]

Original: (see Lemma 2) into J
Revised: X,Y − α · dP X P Y β · dP X,Y + γ · dP X P Y (cid:17) [Equation] (cid:16) β ·

Original: [Equation]
Revised: [Equation]

Revised: γ ·

Original: [Equation]
Revised: [Equation]

Revised: X P Y (cid:17) =

Revised: β + γ (cid:90)

Original: =
Revised: [Equation] X,Y [Equation] α ·

Original: [Equation]
Revised: [Equation]

Original: Since we deﬁne r α,β,γ = dP X,Y − α · dP X P Y β · dP X,Y + γ · dP X P Y and P (cid:48) =
Revised: X P Y β · dP X,Y + γ · dP X P Y (cid:17) [Equation] (cid:16) β [Equation]

Original: ββ + γ P
Revised: [Equation] + γ [Equation] + γ ·

Revised: [Equation]

Revised: X P Y (cid:17) .

Revised: Since we deﬁne r α,β,γ = dP X,Y − α · dP X P Y β · dP X,Y + γ · dP X P Y and P (cid:48) =

Revised: ββ + γ P XY + γβ + γ P X P Y ,

Original: [Equation]
Revised: [Equation]

Revised: (cid:3)

Original: A. P ROOF OF P ROPOSITION 1 IN THE M AIN T EXT
Revised: A. P ROOF OF P ROPOSITION 1 IN THE M AIN T EXT

Original: The proof contains two parts: showing 0 ≤ J RPC ≤ 12 β + α 2 2 γ and ˆ J m,n RPC is a consistent estimator for
Revised: The proof contains two parts: showing 0 ≤ J RPC ≤ 12 β + α 2 2 γ and ˆ J m,n RPC is a consistent estimator for

Original: J RPC .
Revised: J RPC .

Original: A.3. B OUNDNESS OF J RPC 
Revised: A.3. B OUNDNESS OF J RPC Lemma

Original: Lemma 5 (Boundness of J RPC ) 0 ≤ J RPC ≤ 12 β + α 2 2 γ
Revised: Boundness of J RPC ) 0 ≤ J RPC ≤ 12 β + α 2 2 γ

Original: Proof: Lemma 4 suggests J RPC ( X, Y ) =
Revised: Proof: Lemma 4 suggests J RPC ( X, Y ) =

Original: β + γ 2 E P (cid:48)
Revised: β + γ 2 E P (cid:48)

Original: [ r 2 α,β,γ ( x, y )] with P (cid:48)
Revised: [ r 2 α,β,γ ( x, y )] with P (cid:48)

Original: =
Revised: =

Original: ββ + γ P XY + γβ + γ P X P Y as the mixture distribution of P XY and P X P Y .
Revised: ββ + γ P XY + γβ + γ P X P Y as the mixture distribution of P XY and P X P Y .

Original: Hence, it is obvious J RPC ( X, Y ) ≥ 0 .
Revised: Hence [Equation] it is obvious J RPC ( X, Y ) ≥ 0 .

Original: We leverage the intermediate results in the proof of Lemma 4:
Revised: We leverage the intermediate results in the proof of Lemma 4:

Original: [Equation]
Revised: [Equation]

Original: Since − αγ ≤ r α,β,γ ≤ 1 β , J RPC ( X, Y ) ≤ 12 β + α 2 2 γ .
Revised: Since − αγ ≤ r α,β,γ ≤ 1 β [Equation] J RPC ( X [Equation]

Revised: Y ) ≤

Revised: [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original: A.3. C ONSISTENCY
Revised: A.3. C ONSISTENCY

Original: We ﬁrst recall the deﬁnition of the estimation of J RPC :
Revised: We ﬁrst recall the deﬁnition of the estimation of J RPC :

Original: Deﬁnition 2 ( ˆ J m,n RPC , empirical estimation of J RPC , restating Deﬁnition 1 in the main text)
Revised: Deﬁnition 2 ( ˆ J m,n RPC , empirical estimation of J RPC , restating Deﬁnition 1 in the main text)

Original: We parametrize f via a family of neural networks F Θ :=
Revised: We parametrize f via a family of neural networks F Θ :=

Original: { f θ : θ ∈ Θ ⊆ R d } where d ∈ N and Θ is compact.
Revised: { f θ : θ ∈ Θ ⊆ R d } where d ∈ N and Θ is compact.

Original: Let { x i , y i } n i =1 be n samples drawn uniformly at random from P XY and { x (cid:48)
Revised: Let { x i , y i } ni =1 be n samples drawn uniformly at random from P XY and { x (cid:48)

Original: j , y (cid:48)
Revised: j , y (cid:48)

Original: j } m j =1 be m samples drawn uniformly at random from P X P Y .
Revised: j } mj =1 be m samples drawn uniformly at random from P X P Y .

Original: Then, ˆ 
Revised: Then, ˆ J m,n

Original: [Equation]
Revised: [Equation]

Original: Our goal is to show that ˆ J m,n RPC is a consistent estimator for J RPC .
Revised: Our goal is to show that ˆ J m,n RPC is a consistent estimator for J RPC .

Original: We begin with the following deﬁnition:
Revised: We begin with the following deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: and

Original: [Equation]
Revised: [Equation]

Original:  we follow the steps:
Revised: Then, we follow the steps:

Original: • The ﬁrst part is about estimation.
Revised: • The ﬁrst part is about estimation.

Original: We show that, with high probability, ˆ J m,n RPC ,θ is close to E (cid:104)
Revised: We show that, with high probability, ˆ J m,n RPC ,θ is close to E (cid:104)

Original: ˆ J RPC ,θ (cid:105) .
Revised: ˆ J RPC ,θ (cid:105) .

Original: • The second part is about approximation.
Revised: • The second part is about approximation.

Original: We will apply the universal approximation lemma of neural networks (Hornik et al., 1989) to show that there exists a network θ ∗ such that E (cid:104)
Revised: We will apply the universal approximation lemma of neural networks (Hornik et al., 1989) to show that there exists a network θ ∗ such that E (cid:104)

Original: ˆ J RPC ,θ ∗ (cid:105) is close to J RPC .
Revised: ˆ J RPC ,θ ∗ (cid:105) is close to J RPC .

Original: Part I - Estimation: With high probability, ˆ J m,n RPC ,θ is close to E (cid:104)
Revised: Part I - Estimation: With high probability, ˆ J m,n RPC ,θ is close to E (cid:104)

Original: ˆ J RPC ,θ (cid:105) .
Revised: ˆ J RPC ,θ (cid:105) .

Original: Throughout the analysis on the uniform convergence, we need the assumptions on the boundness and smoothness of the function f θ .
Revised: Throughout the analysis on the uniform convergence, we need the assumptions on the boundness and smoothness of the function f θ .

Original: Since we show the optimal function f is bounded in J RPC , we can use the same bounded values for f θ without losing too much precision.
Revised: Since we show the optimal function f is bounded in J RPC , we can use the same bounded values for f θ without losing too much precision.

Original: The smoothness of the function suggests that the output of the network should only change slightly when only slightly perturbing the parameters.
Revised: The smoothness of the function suggests that the output of the network should only change slightly when only slightly perturbing the parameters.

Original: The two assumptions are as follows:
Revised: The two assumptions are as follows:

Original: Assumption 1 (boundness of f θ )
Revised: Assumption 1 (boundness of f θ )

Original: There exist universal constants such that ∀ f θ ∈ F Θ , C L ≤ f θ ≤
Revised: There exist universal constants such that ∀ f θ ∈ F Θ , C L ≤ f θ ≤

Original: C U .
Revised: C U .

Original: For notations simplicity, we let M
Revised: For notations simplicity, we let M

Original: =
Revised: =

Original: C
Revised: C

Original: U − C L be the range of f θ and U = max {| C U | , | C L |} be the maximal absolute value of f θ .
Revised: U − C L be the range of f θ and U = max {| C U | , | C L |} be the maximal absolute value of f θ .

Original: In the paper, we can choose to constrain that ∀ f θ ∈ F Θ , − αγ ≤ f θ ≤ 1 β since the optimal function
Revised: In the paper, we can choose to constrain that ∀ f θ ∈ F Θ , − αγ ≤ f θ ≤ 1 β since the optimal function

Original: f ∗ has − αγ ≤ f ∗ ≤ 1 β .
Revised: f ∗ has − αγ ≤ f ∗ ≤ 1 β .

Original: Assumption 2 (smoothness of f θ )
Revised: Assumption 2 (smoothness of f θ )

Original: There exists constant ρ > 0 such that ∀ ( x, y ) ∈
Revised: There exists constant ρ > 0 such that ∀ ( x, y ) ∈

Original: ( X × Y ) and θ 1 , θ 2 ∈ Θ , | f θ 1 ( x, y )
Revised: ( X × Y ) and θ 1 , θ 2 ∈ Θ , | f θ 1 ( x, y )

Original: − f θ 2 ( x, y ) | ≤ ρ | θ 1 − θ 2 | .
Revised: − f θ 2 ( x, y ) | ≤ ρ | θ 1 − θ 2 | .

Original: Now, we can bound the rate of uniform convergence of a function class in terms of covering number (Bartlett, 1998):
Revised: Now, we can bound the rate of uniform convergence of a function class in terms of covering number (Bartlett, 1998):

Original: Lemma 6 (Estimation) Let (cid:15) > 0 and N (Θ , (cid:15) ) be the covering number of Θ with radius (cid:15) .
Revised: Lemma 6 (Estimation) Let (cid:15) > 0 and N (Θ , (cid:15) ) be the covering number of Θ with radius (cid:15) .

Original: Then,
Revised: Then,

Original: Pr
Revised: Pr

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Proof: For notation simplicity
Revised: (cid:15) [Equation] (cid:1) ) (cid:32) exp (cid:16) − n(cid:15)32 M(cid:17)

Original: [Equation]{++ exp (cid:16) − m(cid:15)32 M 2 α(cid:17) + exp (cid:16) − n(cid:15)32 U 2 β(cid:17) + exp (cid:16) − m(cid:15)
 U 2 γ(cid:17)(cid:33) .+}
Revised: {++ exp (cid:16) − m(cid:15)32 M 2 α(cid:17) + exp (cid:16) − n(cid:15)32 U 2 β(cid:17) + exp (cid:16) − m(cid:15)
 U 2 γ(cid:17)(cid:33) .+}

Original:  we deﬁne the operators
Revised: Proof: For notation simplicity, we deﬁne the operators

Original: • P ( f ) =
Revised: • P ( f ) =

Original: E P XY [ f ( x, y )] and P n ( f ) = 1 n (cid:80) ni
Revised: E P XY [ f ( x, y )] and P n ( f ) = 1 n (cid:80) ni

Original: =1 f ( x i , y i ) • Q ( f ) =
Revised: =1 f ( x i , y i ) • Q ( f ) =

Original: E P X P Y [ f ( x, y )] and Q m ( f ) = 1 m (cid:80) mj
Revised: E P X P Y [ f ( x, y )] and Q m ( f ) = 1 m (cid:80) mj

Original: =1 f ( x (cid:48)
Revised: =1 f ( x (cid:48)

Original: j , y (cid:48) j )
Revised: j , y (cid:48) j )

Original: Hence, [Equation] + [Equation]
Revised: Hence, (cid:12)(cid:12)(cid:12) ˆ J m,n RPC ,θ − E (cid:104)

Original: ( f

Original: [Equation] ( f [Equation] θ
Revised: ˆ J RPC ,θ (cid:105)(cid:12)(cid:12)(cid:12) =

Original: [Equation] + βP ( f2 θ [Equation] − γQ m ( f  2 θ ) +  γQ  ( f 
Revised: (cid:12)(cid:12) P n ( f θ ) − P ( f θ ) − αQ m ( f θ ) + αQ ( f θ ) − βP n ( f 2 θ ) + βP ( f 2 θ ) − γQ m ( f 2 θ )

Original: [Equation]

Original: θ

Original: [Equation] (cid:12)(cid:12) ≤ [Equation] P 
Revised: + γQ ( f 2 θ ) (cid:12)(cid:12) ≤ | P n ( f θ ) − P ( f θ ) | + α | Q m ( f θ ) − Q ( f θ ) |

Original: [Equation]

Original: ( f

Original: [Equation] − γQ ( f [Equation] ) (cid:12)(cid:12)
Revised: + β (cid:12)(cid:12) P n ( f 2 θ ) − P ( f 2 θ ) (cid:12)(cid:12) + γ (cid:12)(cid:12) Q m ( f 2 θ ) − γQ ( f 2 θ ) (cid:12)(cid:12)

Original: Let (cid:15) (cid:48)
Revised: Let (cid:15) (cid:48)

Original: = (cid:15) 4 ρ (cid:0) 1+ α +2( β + γ ) U (cid:1) and T :=
Revised: = (cid:15) 4 ρ (cid:0) 1+ α +2( β + γ ) U (cid:1) and T :=

Original: N (Θ , (cid:15) (cid:48) ) .
Revised: N (Θ , (cid:15) (cid:48) ) .

Original: Let C
Revised: Let C

Original: = { f θ 1 , f θ 2 , · · · , f θ T } with { θ 1 , θ 2 , · · · , θ T } be such that B ∞ ( θ 1 , (cid:15) (cid:48) ) , · · · , B ∞ ( θ T , (cid:15) (cid:48) ) are (cid:15) (cid:48) cover.
Revised: = { f θ 1 , f θ 2 , · · · , f θ T } with { θ 1 , θ 2 , · · · , θ T } be such that B ∞ ( θ 1 , (cid:15) (cid:48) ) , · · · , B ∞ ( θ T , (cid:15) (cid:48) ) are (cid:15) (cid:48) cover.

Original: Hence, for any f θ ∈ F Θ , there is an f θ k ∈ C such that (cid:107) θ − θ k (cid:107) ∞ ≤ (cid:15) (cid:48) . 
Revised: Hence, for any f θ ∈ F Θ , there is an f θ k ∈ C such that (cid:107) θ − θ k (cid:107) ∞ ≤ (cid:15) (cid:48) . Then [Table]

Revised: [Equation]

Original: Then, for any f θ k ∈ C : (cid:12)(cid:12)(cid:12)
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: 2 θ ) (cid:12)(cid:12) + γ (cid:12)(cid:12) Q m ( f 2 θ ) − γQ ( f 2 θ ) (cid:12)(cid:12) ≤ | P n ( f θ k ) − P ( f θ k ) | + | P n ( f θ ) − P n ( f θ k ) | + | P ( f θ ) − P ( f θ k )
Revised: [Equation] [Table] [Equation] [Table]

Original: |
Revised: [Equation] [Table] [Equation] [Table]

Original: + α (cid:18)
Revised: [Equation]

Original: | Q m ( f θ k ) − Q ( f θ k ) | + | Q m ( f θ ) − Q m ( f θ k ) | + | Q ( f θ ) − Q ( f θ k ) | (cid:19) + β (cid:18) (cid:12)(cid:12) P n ( f 2 θ k ) − P ( f 2 θ k ) (cid:12)(cid:12) + (cid:12)(cid:12) P n ( f 2 θ ) − P n ( f 2 θ k ) (cid:12)(cid:12) + (cid:12)(cid:12) P ( f 2 θ ) − P ( f 2 θ k ) (cid:12)(cid:12) (cid:19) + γ (cid:18) (cid:12)(cid:12) Q m ( f 2 θ k ) − Q ( f 2 θ k ) (cid:12)(cid:12) + (cid:12)(cid:12) Q m ( f 2 θ ) − Q m ( f 2 θ k ) (cid:12)(cid:12) + (cid:12)(cid:12) Q ( f 2 θ ) − Q ( f 2 θ k ) (cid:12)(cid:12) (cid:19) ≤ | P n ( f θ k ) − P ( f θ k ) |
Revised: [Table] [Equation]

Original: + ρ (cid:107) θ − θ k (cid:107) + ρ (cid:107) θ − θ k (cid:107) + α (cid:18)
Revised: [Table] [Equation] [Table] [Equation]

Original: | Q m ( f θ k ) − Q ( f θ k ) | + ρ (cid:107) θ − θ k (cid:107) + ρ (cid:107) θ
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original:  • | P n ( f θ ) − P n ( f θ k ) | ≤ ρ (cid:107) θ − θ k (cid:107) due to Assumption 2, and the result also applies for | P ( f θ ) − P ( f θ k ) | , | Q m ( f θ ) − Q m ( f θ k ) | , and | Q ( f θ ) − Q ( f θ k ) | • (cid:12)(cid:12) P n ( f 2 θ ) − P n ( f 2 θ k ) (cid:12)(cid:12) ≤ 2 (cid:107)f θ (cid:107) ∞
Revised: where • | P n ( f θ ) − P n ( f θ k ) | ≤ ρ (cid:107) θ − θ k (cid:107) due to Assumption 2, and the result also applies for | P ( f θ ) − P ( f θ k ) | , | Q m ( f θ ) − Q m ( f θ k ) | , and | Q ( f θ ) − Q ( f θ k ) | • (cid:12)(cid:12) P n ( f 2 θ ) − P n ( f 2 θ k ) (cid:12)(cid:12) ≤ 2 (cid:107)

Original:  ρ (cid:107) θ − θ k (cid:107) ≤ 2 ρU (cid:107) θ − θ k (cid:107) due to Assumption 1 and2.
Revised: f θ (cid:107) ∞ ρ (cid:107) θ − θ k (cid:107) ≤ 2 ρU (cid:107) θ − θ k (cid:107) due to Assumption 1 and

Original: The result also applies for (cid:12)(cid:12) P ( f 2 θ ) − P ( f 2 θ k ) (cid:12)(cid:12), (cid:12)(cid:12) Q m( f 2 θ ) − Q m ( f 2 θ k ) (cid:12)(cid:12) , and (cid:12)(cid:12) Q ( f 2 θ ) − Q ( f 2 θ k ) (cid:12)(cid:12) .
Revised: The result also applies for (cid:12)(cid:12) P ( f 2 θ ) − P ( f 2 θ k ) (cid:12)(cid:12) (cid:12)(cid:12) Q m

Revised: ( f 2 θ ) − Q m ( f 2 θ k ) (cid:12)(cid:12)

Revised: and (cid:12)(cid:12) Q ( f 2 θ ) − Q ( f 2 θ k )

Revised: (cid:12)(cid:12) .

Original: Hence,
Revised: Hence,

Original: Pr
Revised: Pr

Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Original: Pr
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: With Hoeffding’s inequality,
Revised: With Hoeffding’s inequality,

Original: • Pr (cid:0) | P n ( f θ k ) − P ( f θ k ) | ≥ (cid:15) 8 (cid:1) ≤ 2exp (cid:16) − n(cid:15) 2 32 M 2 (cid:17) •
Revised: • Pr (cid:0) | P n ( f θ k ) − P ( f θ k ) | ≥ (cid:15) 8 (cid:1) ≤ 2exp (cid:16) − n(cid:15) 2 32 M 2 (cid:17) •

Original: Pr (cid:0) α | Q
Revised: Pr (cid:0) α | Q

Original: m
Revised: m

Original: ( f θ k ) − Q ( f θ k ) | ≥ (cid:15) 8 (cid:1) ≤ 2exp (cid:16) − m(cid:15)
Revised: ( f θ k ) − Q ( f θ k ) | ≥ (cid:15) 8 (cid:1) ≤ 2exp (cid:16) − [Table]

Original: 2 32 M 2 α 2 (cid:17)• Pr (cid:0) β (cid:12)(cid:12)
Revised: 2 32 M 2 α 2 (cid:17)

Original: P n ( f 2 θ k ) − P ( f 2 θ k ) (cid:12)(cid:12) ≥ (cid:15) 8 (cid:1) ≤ 2exp (cid:16) − n(cid:15) 2 32 U 2 β 2 (cid:17) • Pr (cid:0) γ (cid:12)(cid:12) Q m ( f 2 θ k ) − Q ( f 2 θ k ) (cid:12)(cid:12) ≥ (cid:15) 8 (cid:1) ≤ 2exp (cid:16) − m(cid:15) 2 32 U 2 γ 2 (cid:17)
Revised: [Table]

Original: To conclude,
Revised: To conclude,

Original: Pr
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [-4 ρ (cid:0) 
 + α + 2( β + γ )-]
Revised: [-4 ρ (cid:0) 
 + α + 2( β + γ )-][Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:1) )
Revised: [Table]

Original: (cid:32) exp (cid:16) − n(cid:15)
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: 32 M(cid:17)
Revised: [Table] [Equation] [Table] [Equation]

Original: + exp (cid:16) − m(cid:15)32 M 2 α(cid:17) + exp (cid:16)
Revised: [Table] [Equation] [Table] [Equation]

Original: − n(cid:15)32 U 2 β(cid:17) + exp (cid:16)
Revised: [Table] [Equation] [Table]

Original: [-− m(cid:15)
-]
Revised: [-− m(cid:15)
-][Equation]

Original: U 2 γ(cid:17)(cid:33) .
Revised: [Table] [Equation] [Table]

Original: (cid:3)
Revised: [Equation]

Original: Part II - Approximation: Neural Network Universal Approximation.
Revised: Part II - Approximation: Neural Network Universal Approximation.

Original: We leverage the universal function approximation lemma of neural network
Revised: We leverage the universal function approximation lemma of neural network

Original: Lemma 7 (Hornik et al.
Revised: Lemma 7 (Hornik et al.

Original: (1989), Approximation) Let (cid:15) > 0 .
Revised: (1989), Approximation) Let (cid:15) > 0 .

Original: There exists d ∈ N and a family of neural networks F Θ :=
Revised: There exists d ∈ N and a family of neural networks F Θ :=

Original: { f θ : θ ∈ Θ ⊆ R d } where Θ is compact, such that
Revised: { f θ : θ ∈ Θ ⊆ R d } where Θ is compact, such that

Original: [Equation]
Revised: [Equation]

Original: Part III - Bringing everything together.
Revised: Part III - Bringing everything together.

Original: Now, we are ready to bring the estimation and approximation together to show that there exists a neural network θ ∗ such that, with high probability, ˆ J m,n RPC ,θ can approximate J RPC with n (cid:48)
Revised: Now, we are ready to bring the estimation and approximation together to show that there exists a neural network θ ∗ such that, with high probability, ˆ J m,n RPC ,θ can approximate J RPC with n (cid:48)

Original: = min { n, m } at a rate of O (1 / √ n (cid:48) ) :
Revised: = min { n, m } at a rate of O (1 / √ n (cid:48) ) :

Original: Proposition 3 With probability at least1 − δ , ∃ θ ∗ ∈ Θ , | J RPC − ˆ J m,n RPC ,θ | = O ( (cid:113) d
Revised: Proposition 3 With probability at least

Original: +log(1 /δ ) n (cid:48) ) , where n (cid:48)
Revised: [Equation] where n (cid:48)

Original: = min { n, m } .
Revised: = min { n, m } .

Original: Proof: The proof follows by combining Lemma 6 and 7.
Revised: Proof: The proof follows by combining Lemma 6 and 7.

Original: First, Lemma 7 suggests, ∃ θ ∗ ∈ Θ ,
Revised: First, Lemma 7 suggests, ∃ θ ∗ ∈ Θ ,

Original: [Equation]
Revised: [Equation]

Original: Next, we perform analysis on the estimation error, aiming to ﬁnd n, m and the corresponding probability, such that
Revised: Next, we perform analysis on the estimation error, aiming to ﬁnd n, m and the corresponding probability, such that

Original: [Equation]
Revised: [Equation]

Original: Applying Lemma 6 with the covering number of the neural network:(cid:18) N (Θ
Revised: Applying Lemma 6 with the covering number of the neural network:

Original: [Equation]
Revised: [Equation]

Original: (cid:15) [Equation] ρ ) =
Revised: =

Original: O (cid:16) exp (cid:0) d log ( ρ/(cid:15) )
Revised: O (cid:16) exp (cid:0) d log ( ρ/(cid:15) )

Original: (cid:1)(cid:17) (Anthony & Bartlett, 2009) (cid:19) and let n (cid:48)
Revised: (cid:1)(cid:17) (Anthony & Bartlett, 2009) (cid:19) and let n (cid:48)

Original: =
Revised: =

Original: min { n, m } :
Revised: min { n, m } :

Original: Pr

Original: [Equation]

Original: [Equation]

Original: m(cid:15)

Original: [Equation]

Original: [Equation]

Original: (cid:17)

Original: [Equation]

Original: n(cid:15)

Original: [Equation]

Original: [Equation]

Original: (cid:17) + exp

Original: [Equation]
Revised: [Equation]

Original:  where the big-O notation absorbs all the constants that do not require in the following derivation.
Revised: , where the big-O notation absorbs all the constants that do not require in the following derivation.

Original: Since we want to bound the probability with 1 − δ , we solve the (cid:15) such that
Revised: Since we want to bound the probability with 1 − δ , we solve the (cid:15) such that

Original: [Equation]
Revised: [Equation]

Original: With log ( x ) ≤ x − 1 , [Equation] where this inequality holds when
Revised: With log ( x ) ≤ x − 1 , n (cid:48) (cid:15) 2 + d ( (cid:15) − 1) ≥ n (cid:48) (cid:15) 2 + d log (cid:15) ≥ log (1 /δ ) , where this inequality holds when

Original: [Equation]
Revised: [Equation]

Original: A. P ROOF OF P ROPOSITION 2 IN THE M AIN T EXT - F ROM AN A SYMPTOTIC V IEWPOINT
Revised: A. P ROOF OF P ROPOSITION 2 IN THE M AIN T EXT - F ROM AN A SYMPTOTIC V IEWPOINT

Original: Here, we provide the variance analysis on ˆ J m,n RPC via an asymptotic viewpoint.
Revised: Here, we provide the variance analysis on ˆ J m,n RPC via an asymptotic viewpoint.

Original: First, assuming the network is correctly speciﬁed, and hence there exists a network parameter θ ∗ satisfying f ∗ ( x, y ) =
Revised: First, assuming the network is correctly speciﬁed, and hence there exists a network parameter θ ∗ satisfying f ∗ ( x, y ) =

Original: f θ ∗ ( x, y ) =
Revised: f θ ∗ ( x, y ) =

Original: r α,β,γ ( x, y ) .
Revised: r α,β,γ ( x, y ) .

Original: Then we recall that ˆ J m,n RPC is a consistent estimator of J RPC (see Proposition 3), and under regular conditions, the estimated network parameter ˆ θ in ˆ J m,n RPC satisfying the asymptotic normality in the large sample limit (see Theorem 5.23 in (Van der Vaart, 2000)).
Revised: Then we recall that ˆ J m,n RPC is a consistent estimator of J RPC (see Proposition 3), and under regular conditions, the estimated network parameter ˆ θ in ˆ J m,n RPC satisfying the asymptotic normality in the large sample limit (see Theorem 5.23 in (Van der Vaart, 2000)).

Original: We recall the deﬁnition of ˆ J m,n RPC ,θ in equation 3 and let n (cid:48)
Revised: We recall the deﬁnition of ˆ J m,n RPC ,θ in equation 3 and let n (cid:48)

Original: = min { n, m } , the asymptotic expansion of ˆ J m,n RPC has
Revised: = min { n, m } , the asymptotic expansion of ˆ J m,n RPC has

Original: [Equation]
Revised: [Equation]

Original:  where ˙ˆ J m,n RPC , ˆ θ = 0 since ˆ θ is the estimation from ˆ J m,n RPC = sup f
Revised: (5) where ˙ˆ J m,n RPC , ˆ θ = 0 since ˆ θ is the estimation from ˆ J m,n RPC = sup f

Original: [Equation]
Revised: [Equation]

Original: Next, we deﬁne
Revised: Next, we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Likewise, the asymptotic expansion of E [ ˆ J RPC ,θ ] has
Revised: Likewise, the asymptotic expansion of E [ ˆ J RPC ,θ ] has

Original: [Equation]
Revised: [Equation]

Original: where E [ ˙ˆ J RPC ,θ ∗ ] = 0 since E [ ˆ J RPC ,θ ∗ ] = J RPC and θ ∗ satisfying f ∗ ( x, y )
Revised: where E [ ˙ ˆ J RPC ,θ ∗ ] = 0 since E [ ˆ J RPC ,θ ∗ ] = J RPC and θ ∗ satisfying f ∗ ( x, y )

Original: =
Revised: =

Original: f θ ∗ ( x, y ) .
Revised: f θ ∗ ( x, y ) .

Original: Combining equations 5 and 6:

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: Therefore, the asymptotic Variance of ˆ J m,n RPC is

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: Var[ ˆ

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: m,n

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Therefore

Original: [Equation]
Revised: [Equation]

Original: the asymptotic Variance of ˆ J m,n RPC is Var[ ˆ J m
Revised: 1 n Var P XY [

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: x, y )]

Revised: [Equation]

Revised: 1 m Var

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ,γ ( x, y ) +

Revised: [Equation]

Original:  First, we look at Var P XY [ r α,β,γ ( x, y ) − β 2 r 2 α,β,γ ( x, y )] .
Revised: [Equation] 2 α,β,γ ( x, y )] + o ( 1 n (cid:48) ) . First, we look at Var P XY [ r α,β,γ ( x, y ) − β 2 r 2 α,β,γ ( x, y )] .

Original: Since β > 0 and − α γ ≤ r α,β,γ ≤ 1 β , simple calculation gives us − 2 αγ + βα 2 2 γ≤ r α,β,γ ( x, y ) − β 2 r 2 α,β,γ ( x, y ) ≤ 12 β .
Revised: Since β > 0 and − αγ ≤ r α,β,γ ≤ 1 β , simple calculation gives us − 2 αγ + βα 2 2 γ≤ r α,β,γ ( x, y ) − β 2 r 2 α,β,γ ( x, y ) ≤ 12 β .

Original: Hence,
Revised: Hence,

Revised: Var P XY

Original: [Equation]
Revised: [Equation]

Original: Next, we look at Var P X P Y [ αr α,β,γ ( x, y )+ γ 2 r 2 α,β,γ ( x, y )] .
Revised: Next, we look at Var P X P Y [ αr α,β,γ ( x, y )+ γ 2 r 2 α,β,γ ( x, y )] .

Original: Since α ≥ 0 , γ > 0 and − αγ ≤ r α,β,γ ≤ 1 β , simple calculation gives us − α 2 2 γ ≤ αr α,β,γ ( x, y ) + γ 2 r 2 α,β,γ ( x, y ) ≤ 2 αβ + γ 2 β 2 .
Revised: Since α ≥ 0 , γ > 0 and − αγ ≤ r α,β,γ ≤ 1 β , simple calculation gives us − α 2 2 γ ≤ αr α,β,γ ( x, y ) + γ 2 r 2 α,β,γ ( x, y ) ≤ 2 αβ + γ 2 β 2 .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Combining everything together, we restate the Proposition 2 in the main text: Proposition 4 (Asymptotic Variance of ˆ J m,n RPC ) Var[ ˆ J m,n RPC ] = 1 n Var P XY [ r α,β,γ ( x, y ) − β 2 r 2 α,β,γ ( x, y )] + 1 m Var P X P Y [ αr α,β,γ ( x, y ) + γ 2 r 2 α,β,γ ( x, y )]
Revised: Combining everything together

Original: + o ( 1 n (cid:48) )

Original: ≤n max (cid:26)(cid:16)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: γ

Original: [Equation]

Original: [Equation]

Original: (cid:17)

Original: [Equation]

Original: [Equation] max (cid:26)(cid:16) [Equation] 2 γ (cid:17)
Revised: we restate the Proposition 2 in the main text:

Original: [Equation] (cid:16) [Equation] γ
Revised: Proposition 4 (Asymptotic Variance of ˆ J m,n RPC )

Original: [Equation]
Revised: [Equation]

Original: (cid:17) 2 (cid:27)

Original: [Equation]

Original: (n (cid:48) )

Original: A. P ROOF OF P ROPOSITION 2 IN THE M AIN T EXT - F ROM B OUNDNESS OF f θ
Revised: A. P ROOF OF P ROPOSITION 2 IN THE M AIN T EXT - F ROM B OUNDNESS OF f θ

Original: As discussed in Assumption
Revised: As discussed in Assumption

Original: [Equation]
Revised: [Equation]

Original: for the estimation ˆ J m
Revised: for the estimation ˆ J m

Original: [Equation]
Revised: [Equation]

Original: n RPC
Revised: n RPC

Original: [Equation]
Revised: [Equation]

Original: we can bound the function f θ in F Θ within [ − αγ , 1 β ] without losing precision.
Revised: we can bound the function f θ in F Θ within [ − αγ , 1 β ] without losing precision.

Original: Then, re-arranging ˆ J m,n RPC :
Revised: Then, re-arranging ˆ J m,n RPC :

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: since − αγ ≤ f θ ( · , · ) ≤ 1 β
Revised: since − αγ ≤ f θ ( · , · ) ≤ 1 β

Original: [Equation]
Revised: [Equation]

Original: basic calculations give us − 2 αγ + βα2 γ
Revised: basic calculations give us − 2 αγ + βα2 γ

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: The resulting variances have
Revised: The resulting variances have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Taking the mean of m, n independent random variables gives the result:
Revised: Taking the mean of m, n independent random variables gives the result:

Original: Proposition 5 (Variance of ˆ J m,n RPC )
Revised: Proposition 5 (Variance of ˆ J m,n RPC )

Original: [Equation]
Revised: [Equation]

Original: A. I MPLEMENTATION OF E XPERIMENTS
Revised: A. I MPLEMENTATION OF E XPERIMENTS

Original: For visual representation learning, we follow the implementation in https://github.com/ google-research/simclr .
Revised: For visual representation learning, we follow the implementation in https://github.com/ google-research/simclr .

Original: For speech representation learning, we follow the implementation in https://github.com/facebookresearch/CPC_audio .
Revised: For speech representation learning, we follow the implementation in https://github.com/facebookresearch/CPC_audio .

Original: For MI estimation, we follow the implementation in https://github.com/yaohungt/Pointwise_
Revised: For MI estimation, we follow the implementation in https://github.com/yaohungt/Pointwise_

Original: Dependency_Neural_Estimation/tree/master/MI_Est_and_CrossModal ..
Revised: Dependency_Neural_Estimation/tree/master/MI_Est_and_CrossModal ..

Original: A. R ELATIVE P REDICTIVE C ODING ON V ISION
Revised: A. R ELATIVE P REDICTIVE C ODING ON V ISION

Original: The whole pipeline of pretraining contains the following steps: First, a stochastic data augmentation will transform one image sample x k to two different but correlated augmented views, x (cid:48) 2 k − 1 and x (cid:48) 2 k .
Revised: The whole pipeline of pretraining contains the following steps: First, a stochastic data augmentation will transform one image sample x k to two different but correlated augmented views, x (cid:48) 2 k − 1 and x (cid:48) 2 k .

Original: Then a base encoder f ( · ) implemented using ResNet (He et al., 2016) will extract representations from augmented views, creating representations h 2 k − 1 and h 2 k .
Revised: Then a base encoder f ( · ) implemented using ResNet (He et al., 2016) will extract representations from augmented views, creating representations h 2 k − 1 and h 2 k .

Original: Later a small neural network g ( · ) called projection head will map h 2 k − 1 and h 2 k to z 2 k − 1 and z 2 k in a different latent space.
Revised: Later a small neural network g ( · ) called projection head will map h 2 k − 1 and h 2 k to z 2 k − 1 and z 2 k in a different latent space.

Original: For each minibatch of N samples, there will be 2 N views generated.
Revised: For each minibatch of N samples, there will be 2 N views generated.

Original: For each image x k there will be one positive pair x (cid:48)
Revised: For each image x k there will be one positive pair x (cid:48)

Original: 2 k − 1 and x (cid:48) 2 k and 2( N − 1) negative samples.
Revised: 2 k − 1 and x (cid:48) 2 k and 2( N − 1) negative samples.

Original: The RPC loss between a pair of positive views, x (cid:48)
Revised: The RPC loss between a pair of positive views, x (cid:48)

Original: i and x (cid:48) j (augmented from the same image) , can be calculated by the substitution f θ ( x (cid:48)
Revised: i and x (cid:48) j (augmented from the same image) , can be calculated by the substitution f θ ( x (cid:48)

Original: i , x (cid:48) j ) =
Revised: i , x (cid:48) j ) =

Original: ( z i · z j ) /τ
Revised: ( z i · z j ) /τ

Original: =
Revised: =

Original: s i,j ( τ is a hyperparameter) to the deﬁnition of RPC:
Revised: s i,j ( τ is a hyperparameter) to the deﬁnition of RPC:

Original: [Equation]
Revised: [Equation]

Original: For losses other than RPC, a hidden normalization of s i,j is often required by replacing z i · z j with ( z i · z j ) / |
Revised: For losses other than RPC, a hidden normalization of s i,j is often required by replacing z i · z j with ( z i · z j ) / |

Original: z i ||
Revised: z i ||

Original: z j | .
Revised: z j | .

Original: CPC and WPC adopt this, while other objectives needs it to help stabilize training variance.
Revised: CPC and WPC adopt this, while other objectives needs it to help stabilize training variance.

Original: RPC does not need this normalization.
Revised: RPC does not need this normalization.

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: A. CIFAR-10/-100 AND I MAGE N ET E XPERIMENTS D ETAILS

Original: ImageNet Following the settings in (Chen et al., 2020b;c), we train the model on Cloud TPU with 128 cores, with a batch size of 4 , 096 and global batch normalization 2 (Ioffe & Szegedy, 2015).
Revised: ImageNet Following the settings in (Chen et al., 2020b;c), we train the model on Cloud TPU with 128 cores, with a batch size of 4 , 096 and global batch normalization 2 (Ioffe & Szegedy, 2015).

Original: Here we refer to the term batch size as the number of images (or utterances in the speech experiments)
Revised: Here we refer to the term batch size as the number of images (or utterances in the speech experiments)

Original: we use per GPU, while the term minibatch size refers to the number of negative samples used to calculate the objective, such as CPC or our proposed RPC.
Revised: we use per GPU, while the term minibatch size refers to the number of negative samples used to calculate the objective, such as CPC or our proposed RPC.

Original: The largest model we train is a 152-layer
Revised: The largest model we train is a 152-layer

Original: ResNet with selective kernels (SK) (Li et al., 2019) and 2 × wider channels.
Revised: ResNet with selective kernels (SK) (Li et al., 2019) and 2 × wider channels.

Original: We use the LARS optimizer (You et al., 2017) with momentum 0 .
Revised: We use the LARS optimizer (You et al., 2017) with momentum 0 .

Original: The learning rate linearly increases for the ﬁrst 20 epochs, reaching a maximum of 6 . 4 , then decayed with cosine decay schedule.
Revised: The learning rate linearly increases for the ﬁrst 20 epochs, reaching a maximum of 6 . 4 , then decayed with cosine decay schedule.

Original: The weight decay is 10 − 4 .
Revised: The weight decay is 10 − 4 .

Original: A MLP projection head g ( · ) with three layers is used on top of the ResNet encoder.
Revised: A MLP projection head g ( · ) with three layers is used on top of the ResNet encoder.

Original: Unlike Chen et al.
Revised: Unlike Chen et al.

Original: (2020c), we do not use a memory buffer, and train the model for only 100 epochs rather than 800 epochs due to computational constraints.
Revised: (2020c), we do not use a memory buffer, and train the model for only 100 epochs rather than 800 epochs due to computational constraints.

Original: These two options slightly reduce CPC’s performance benchmark for about 2% with the exact same setting.
Revised: These two options slightly reduce CPC’s performance benchmark for about 2% with the exact same setting.

Original: The unsupervised pre-training is followed by a supervised ﬁne-tuning.
Revised: The unsupervised pre-training is followed by a supervised ﬁne-tuning.

Original: Following SimCLRv

Revised: Following SimCLRv

Original: (Chen et al., 2020b;c), we ﬁne-tune the 3-layer g ( · ) for the downstream tasks.
Revised: (Chen et al., 2020b;c), we ﬁne-tune the 3-layer g ( · ) for the downstream tasks.

Original: We use learning rates 0 .
Revised: We use learning rates 0 .

Original: 16 and 0 .
Revised: 16 and 0 .

Original: 064 for standard 50-layer
Revised: 064 for standard 50-layer

Original: ResNet and larger 152-layer ResNet respectively, and weight decay and learning rate warmup are removed.
Revised: ResNet and larger 152-layer ResNet respectively, and weight decay and learning rate warmup are removed.

Original: Different from Chen et al.
Revised: Different from Chen et al.

Original: (2020c)
Revised: (2020c)

Original: , we use a batch size of 4 , 096 , and we do not use global batch normalization for ﬁne-tuning.
Revised: , we use a batch size of 4 , 096 , and we do not use global batch normalization for ﬁne-tuning.

Original: For J RPC we disable hidden normalization and use a temperature τ = 128 .
Revised: For J RPC we disable hidden normalization and use a temperature τ = 128 .

Original: For all other objectives, we use hidden normalization and τ = 0 .
Revised: For all other objectives, we use hidden normalization and τ = 0 .

Original: 5 following previous work (Chen et al., 2020c).
Revised: 5 following previous work (Chen et al., 2020c).

Original: CIFAR-10/-100 Following the settings in (Chen et al., 2020b), we train the model on a single
Revised: CIFAR-10/-100 Following the settings in (Chen et al., 2020b), we train the model on a single

Original: GPU, with a batch size of 512 and global batch normalization (Ioffe & Szegedy, 2015).
Revised: GPU, with a batch size of 512 and global batch normalization (Ioffe & Szegedy, 2015).

Original: We use
Revised: We use

Original: ResNet (He et al., 2016) of depth 18 and depth 50 , and does not use Selective Kernel (Li et al., 2019) or a multiplied width size.
Revised: ResNet (He et al., 2016) of depth 18 and depth 50 , and does not use Selective Kernel (Li et al., 2019) or a multiplied width size.

Original: We use the LARS optimizer (You et al., 2017) with momentum 0 .
Revised: We use the LARS optimizer (You et al., 2017) with momentum 0 .

Original: The learning rate linearly increases for the ﬁrst 20 epochs, reaching a maximum of 6 . 4 , then decayed with cosine decay schedule.
Revised: The learning rate linearly increases for the ﬁrst 20 epochs, reaching a maximum of 6 . 4 , then decayed with cosine decay schedule.

Original: The weight decay is 10 − 4 .
Revised: The weight decay is 10 − 4 .

Original: A MLP projection head g ( · ) with three layers is used on top of the ResNet encoder.
Revised: A MLP projection head g ( · ) with three layers is used on top of the ResNet encoder.

Original: Unlike Chen et al.
Revised: Unlike Chen et al.

Original: (2020c), we do not use a memory buffer.
Revised: (2020c), we do not use a memory buffer.

Original: We train the model for 1000 epochs.
Revised: We train the model for 1000 epochs.

Original: The unsupervised pre-training is followed by a supervised ﬁne-tuning.
Revised: The unsupervised pre-training is followed by a supervised ﬁne-tuning.

Original: Following SimCLRv

Revised: Following SimCLRv

Original: (Chen et al., 2020b;c), we ﬁne-tune the 3-layer g ( · ) for the downstream tasks.
Revised: (Chen et al., 2020b;c), we ﬁne-tune the 3-layer g ( · ) for the downstream tasks.

Original: We use learning rates 0 .
Revised: We use learning rates 0 .

Original: 16 for standard 50-layer ResNet , and weight decay and learning rate warmup are removed.
Revised: 16 for standard 50-layer ResNet , and weight decay and learning rate warmup are removed.

Original: For J RPC we disable hidden normalization and use a temperature τ = 128 .
Revised: For J RPC we disable hidden normalization and use a temperature τ = 128 .

Original: For all other objectives, we use hidden normalization and τ = 0 .
Revised: For all other objectives, we use hidden normalization and τ = 0 .

Original: 5 following previous work (Chen et al., 2020c).
Revised: 5 following previous work (Chen et al., 2020c).

Original: STL-10 We also perform the pre-training and ﬁne-tuning on STL-10 (Coates et al., 2011) using the model proposed in Chuang et al.
Revised: STL-10 We also perform the pre-training and ﬁne-tuning on STL-10 (Coates et al., 2011) using the model proposed in Chuang et al.

Original: Chuang et al.
Revised: Chuang et al.

Original: (2020) proposed to indirectly approximate the distribution of negative samples so that the objective is debiased .
Revised: (2020) proposed to indirectly approximate the distribution of negative samples so that the objective is debiased .

Original: However, their implementation of contrastive learning is consistent with Chen et al.
Revised: However, their implementation of contrastive learning is consistent with Chen et al.

Original: (2020b).
Revised: (2020b).

Original: We use a ResNet with depth 50 as an encoder for pre-training, with Adam optimizer, learning rate 0 .
Revised: We use a ResNet with depth 50 as an encoder for pre-training, with Adam optimizer, learning rate 0 .

Original: 001 and weight decay 10 − 6 .
Revised: 001 and weight decay 10 − 6 .

Original: The temperature τ is set to 0 .
Revised: The temperature τ is set to 0 .

Original: 5 for all objectives other than J RPC , which disables hidden normalization and use τ = 128 .
Revised: 5 for all objectives other than J RPC , which disables hidden normalization and use τ = 128 .

Original: The downstream task performance increases from 83 .
Revised: The downstream task performance increases from 83 .

Original: 4% of J CPC to 84 .
Revised: 4% of J CPC to 84 .

Original: 1% of
Revised: 1% of

Original: J RPC .
Revised: J RPC .

Original:  We also provide the conﬁdence interval of J RPC and J CPC on CIFAR-10, CIFAR-100 and ImageNet, using ResNet-18, ResNet-18 and ResNet-50 respectively (95% conﬁdence level is chosen) in Table 5: , the global batch
Revised: Conﬁdence Interval We also provide the conﬁdence interval of J RPC and J CPC on CIFAR-10, CIFAR-100 and ImageNet, using ResNet-18, ResNet-18 and ResNet-50 respectively (95% conﬁdence level is chosen) in Table 4.

Original: normalization during we Both CPC and RPC use the default parameters provided in this section.
Revised: Both CPC and RPC use the same experimental settings throughout the global this paper. Here we use the relative parameters ( α = 1 . 0 , β = 0 .

Revised: 005 , γ = 1 . 0 )

Revised: in J RPC which gives the best performance on CIFAR-10.

Original: The conﬁdence intervals of CPC do not overlap with the conﬁdence intervals of RPC, which means the difference of the downstream task performance between RPC and CPC is statistically signiﬁcant.
Revised: The conﬁdence intervals of CPC do not overlap with the conﬁdence intervals of RPC, which means the difference of the downstream task performance between RPC and CPC is statistically signiﬁcant.

Original: A. R ELATIVE P REDICTIVE C ODING ON S PEECH
Revised: A. R ELATIVE P REDICTIVE C ODING ON S PEECH

Original: For speech representation learning, we adopt the general architecture from Oord et al.
Revised: For speech representation learning, we adopt the general architecture from Oord et al.

Original: Given an input signal x 1: T with T time steps, we ﬁrst pass it through an encoder φ θ parametrized by θ to produce a sequence of hidden representations { h 1: T } where h t = φ θ ( x t ) .
Revised: Given an input signal x 1: T with T time steps, we ﬁrst pass it through an encoder φ θ parametrized by θ to produce a sequence of hidden representations { h 1: T } where h t = φ θ ( x t ) .

Original: After that, we obtain the contextual representation c t at time step t with a sequential model ψ ρ parametrized by ρ : c t = ψ ρ ( h 1 , . . .
Revised: After that, we obtain the contextual representation c t at time step t with a sequential model ψ ρ parametrized by ρ : c t = ψ ρ ( h 1 , . . .

Original: , h t ) , where c t contains context information before time step t .
Revised: , h t ) , where c t contains context information before time step t .

Original: For unsupervised pre-training, we use a multi-layer convolutional network as the encoder φ θ , and an LSTM with hidden dimension 256 as the sequential model ψ ρ .
Revised: For unsupervised pre-training, we use a multi-layer convolutional network as the encoder φ θ , and an LSTM with hidden dimension 256 as the sequential model ψ ρ .

Original: Here, the contrastiveness is between the positive pair ( h t + k , c t ) where k is the number of time steps ahead, and the negative pairs ( h i , c t ) , where h i is randomly sampled from N , a batch of hidden representation of signals assumed to be unrelated to c t .
Revised: Here, the contrastiveness is between the positive pair ( h t + k , c t ) where k is the number of time steps ahead, and the negative pairs ( h i , c t ) , where h i is randomly sampled from N , a batch of hidden representation of signals assumed to be unrelated to c t .

Original: The scoring function f based on Equation 2 at step t and look-ahead k will be f k = f k ( h , c t ) =
Revised: The scoring function f based on Equation 2 at step t and look-ahead k will be f k = f k ( h , c t ) =

Original: exp(( h ) (cid:62) W k c t ) , where W k is a learnable linear transformation deﬁned separately for each k ∈ { 1 , ..., K } and K is predetermined as 12 time steps.
Revised: exp(( h ) (cid:62) W k c t ) , where W k is a learnable linear transformation deﬁned separately for each k ∈ { 1 , ..., K } and K is predetermined as 12 time steps.

Original: The loss in Equation 2 will then be formulated as:
Revised: The loss in Equation 2 will then be formulated as:

Original: [Equation]
Revised: [Equation]

Original: 2 |N| (cid:88) h

Original: i ∈N f 2 k ( h i , c t ))

Original: As shown in Table 3, J RPC has better downstream task performance, and is closer to the performance from a fully supervised model.
Revised: As shown in Table 3, J RPC has better downstream task performance, and is closer to the performance from a fully supervised model.

Original: A. E MPIRICAL O BSERVATIONS ON V ARIANCE AND M INIBATCH S IZE
Revised: A. E MPIRICAL O BSERVATIONS ON V ARIANCE AND M INIBATCH S IZE

Original: Variance Experiment Setup We perform the variance comparison of J DV , J NWJ and the proposed J RPC .
Revised: Variance Experiment Setup We perform the variance comparison of J DV , J NWJ and the proposed J RPC .

Original: The empirical experiments are performed using SimCLRv2 (Chen et al., 2020c) on
Revised: The empirical experiments are performed using SimCLRv2 (Chen et al., 2020c) on

Original: CIFAR-10 dataset.
Revised: CIFAR-10 dataset.

Original: We use a ResNet of depth 18 , with batch size of 512 .
Revised: We use a ResNet of depth 18 , with batch size of 512 .

Original: We train each objective with 30 K training steps and record their value.
Revised: We train each objective with 30 K training steps and record their value.

Original: In Figure 1, we use a temperature τ = 128 for all objectives.
Revised: In Figure 1, we use a temperature τ = 128 for all objectives.

Original: Unlike other experiments, where hidden normalization is applied to other objectives, we remove hidden normarlization for all objectives due to the reality that objectives after normalization does not reﬂect their original values.
Revised: Unlike other experiments, where hidden normalization is applied to other objectives, we remove hidden normarlization for all objectives due to the reality that objectives after normalization does not reﬂect their original values.

Original: From Figure 1, J RPC enjoys lower variance and more stable training compared to J DV and J NWJ .
Revised: From Figure 1, J RPC enjoys lower variance and more stable training compared to J DV and J NWJ .

Original: Minibatch Size Experimental Setup We perform experiments on the effect of batch size on downstream performances for different objective.
Revised: Minibatch Size Experimental Setup We perform experiments on the effect of batch size on downstream performances for different objective.

Original: The experiments are performed using SimCLRv (Chen et al., 2020c) on CIFAR-10 dataset, as well as the model from Rivi`ere et al.
Revised: The experiments are performed using SimCLRv (Chen et al., 2020c) on CIFAR-10 dataset, as well as the model from Rivi ` ere et al.

Original: (2020) on
Revised: (2020) on

Original: LibriSpeech-100h dataset (Panayotov et al., 2015).
Revised: LibriSpeech-100h dataset (Panayotov et al., 2015).

Original: For vision task, we use the default temperature τ = 0 .
Revised: For vision task, we use the default temperature τ = 0 .

Original: 5 from Chen et al.
Revised: 5 from Chen et al.

Original: (2020c)
Revised: (2020c)

Original: and hidden normalization mentioned in Section 3 for J CPC .
Revised: and hidden normalization mentioned in Section 3 for J CPC .

Original: For J RPC in vision and speech tasks we use a temperature of τ = 128 and do not use hidden normalization.
Revised: For J RPC in vision and speech tasks we use a temperature of τ = 128 and do not use hidden normalization.

Original: A. M UTUAL I NFORMATION E STIMATION
Revised: A. M UTUAL I NFORMATION E STIMATION

Original: We follow prior work (Poole et al., 2019; Song & Ermon, 2019; Tsai et al., 2020b) for the experiments.
Revised: We follow prior work (Poole et al., 2019; Song & Ermon, 2019; Tsai et al., 2020b) for the experiments.

Original: We consider X and Y as two 20 -dimensional Gaussians with correlation ρ , and our goal is to estimate the mutual information
Revised: We consider X and Y as two 20 -dimensional Gaussians with correlation ρ , and our goal is to estimate the mutual information

Original: I ( X ; Y ) .
Revised: I ( X ; Y ) .

Original: Then, we perform a cubic transformation on y so that y (cid:55)→ y 3 .
Revised: Then, we perform a cubic transformation on y so that y (cid:55)→ y 3 .

Original: The ﬁrst task is referred as Gaussian task and the second is referred as Cubic task, where both have the ground truth
Revised: The ﬁrst task is referred as Gaussian task and the second is referred as Cubic task, where both have the ground truth

Original: I
Revised: I

Original: ( X ; Y ) =
Revised: ( X ; Y ) =

Original: − 10log (1 − ρ 2 ) .
Revised: − 10log (1 − ρ 2 ) .

Original: The models are trained on 20 ,steps with I ( X ; Y ) starting at 2 and increased by 2 per 4 , 000 steps.
Revised: The models are trained on 20 ,steps with I ( X ; Y ) starting at 2 and increased by 2 per 4 , 000 steps.

Original: Our method is compared with baseline methods CPC (Oord et al., 2018), NWJ (Nguyen et al., 2010), JSD (Nowozin et al., 2016), and SMILE (Song & Ermon, 2019).
Revised: Our method is compared with baseline methods CPC (Oord et al., 2018), NWJ (Nguyen et al., 2010), JSD (Nowozin et al., 2016), and SMILE (Song & Ermon, 2019).

Original: All the approaches consider the same design of f ( x, y ) , which is a 3-layer neural network taking concatenated ( x, y ) as the input.
Revised: All the approaches consider the same design of f ( x, y ) , which is a 3-layer neural network taking concatenated ( x, y ) as the input.

Original: We also ﬁx the learning rate, the optimizer, and the minibatch size across all the estimators for a fair comparison.
Revised: We also ﬁx the learning rate, the optimizer, and the minibatch size across all the estimators for a fair comparison.

Original: We present results of mutual information by Relative Predictive Coding using different sets of relative parameters.
Revised: We present results of mutual information by Relative Predictive Coding using different sets of relative parameters.

Original: In the ﬁrst row, we set β = 10 − 3 , γ = 1 , and experiment with different α values.
Revised: In the ﬁrst row, we set β = 10 − 3 , γ = 1 , and experiment with different α values.

Original: In the second row, we set α = 1 , γ = 1 and in the last row we set α = 1 , β = 10 − 3 .
Revised: In the second row, we set α = 1 , γ = 1 and in the last row we set α = 1 , β = 10 − 3 .

Original: From the ﬁgure, a small β around 10 − 3 and a large γ around 1 . 0 is crucial for an estimation that is relatively low bias and low variance.
Revised: From the ﬁgure, a small β around 10 − 3 and a large γ around 1 . 0 is crucial for an estimation that is relatively low bias and low variance.

Original: This conclusion is consistent with Section 3 in the main text.
Revised: This conclusion is consistent with Section 3 in the main text.

Original: We also performed comparison between J RPC and Difference of Entropies (DoE) (McAllester &
Revised: We also performed comparison between J RPC and Difference of Entropies (DoE) (McAllester &

Original: Stratos, 2020).
Revised: Stratos, 2020).

Original: We performed two sets of experiments: in the ﬁrst set of experiments we compare
Revised: We performed two sets of experiments: in the ﬁrst set of experiments we compare

Original: J RPC and DoE when MI is large ( > 100 nats), while in the second set of experiments we compare
Revised: J RPC and DoE when MI is large ( > 100 nats), while in the second set of experiments we compare

Original: J RPC and DoE using the setup in this section (MI < 12 nats and MI increases by 2 per 4k training steps).
Revised: J RPC and DoE using the setup in this section (MI < 12 nats and MI increases by 2 per 4k training steps).

Original: On the one hand, when MI is large ( > 100 nats), we acknowledge that DoE is performing well on MI estimation, compared to J RPC which only estimates the MI around 20 .
Revised: On the one hand, when MI is large ( > 100 nats), we acknowledge that DoE is performing well on MI estimation, compared to J RPC which only estimates the MI around 20 .

Original: This analysis is based on the code from https://github.com/karlstratos/doe .
Revised: This analysis is based on the code from https://github.com/karlstratos/doe .

Original: On the other hand, when the true MI is small, the DoE method is more unstable than J RPC , as shown in Figure 4.
Revised: On the other hand, when the true MI is small, the DoE method is more unstable than J RPC , as shown in Figure 5.

Original: Figure 4 illustrates the results of the DoE method when the distribution is isotropic Gaussian (correctly speciﬁed) or Logistic (mis-speciﬁed).
Revised: Figure 5 illustrates the results of the DoE method when the distribution is isotropic Gaussian (correctly speciﬁed) or Logistic (mis-speciﬁed).

Original: Figure 2 only shows the results using Gaussian.
Revised: Figure 3 only shows the results using Gaussian.