Original: P RETEXT T ASKS S ELECTION FOR M ULTITASK S ELFS UPERVISED S PEECH R EPRESENTATION L EARNING
Revised: P RETEXT T ASKS S ELECTION FOR M ULTITASK S ELFS UPERVISED S PEECH R EPRESENTATION L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Through solving pretext tasks, self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task.
Revised: Through solving pretext tasks, self-supervised learning leverages unlabeled data to extract useful latent representations replacing traditional input features in the downstream task.

Original: In audio/speech signal processing, a wide range of features where engineered through decades of research efforts.
Revised: In audio/speech signal processing, a wide range of features where engineered through decades of research efforts.

Original: As it turns out, learning to predict such features (a.k.a pseudo-labels) has proven to be a particularly relevant pretext task, leading to useful self-supervised representations which prove to be effective for downstream tasks.
Revised: As it turns out, learning to predict such features (a.k.a pseudo-labels) has proven to be a particularly relevant pretext task, leading to useful self-supervised representations which prove to be effective for downstream tasks.

Original: However, methods and common practices for combining such pretext tasks for better performance on the downstream task have not been explored and understood properly.
Revised: However, methods and common practices for combining such pretext tasks for better performance on the downstream task have not been explored and understood properly.

Original: In fact, the process relies almost exclusively on a computationally heavy experimental procedure, which becomes intractable with the increase of the number of pretext tasks.
Revised: In fact, the process relies almost exclusively on a computationally heavy experimental procedure, which becomes intractable with the increase of the number of pretext tasks.

Original: This paper introduces a method to select a group of pretext tasks among a set of candidates.
Revised: This paper introduces a method to select a group of pretext tasks among a set of candidates.

Original: The method we propose estimates calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process.
Revised: The method we propose estimates calibrated weights for the partial losses corresponding to the considered pretext tasks during the self-supervised training process.

Original: The experiments conducted on automatic speech recognition, speaker and emotion recognition validate our approach, as the groups selected and weighted with our method perform better than classic baselines, thus facilitating the selection and combination of relevant pseudo-labels for self-supervised representation learning.
Revised: The experiments conducted on automatic speech recognition, speaker and emotion recognition validate our approach, as the groups selected and weighted with our method perform better than classic baselines, thus facilitating the selection and combination of relevant pseudo-labels for self-supervised representation learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Self-supervised learning (SSL) methods usually rely on a supervision obtained from the data itself through solving speciﬁc pretext tasks leveraging the underlying structure of the considered data (Doersch et al., 2016; Arandjelovic & Zisserman, 2018).
Revised: Self-supervised learning (SSL) methods usually rely on a supervision obtained from the data itself through solving speciﬁc pretext tasks leveraging the underlying structure of the considered data (Doersch et al., 2016; Arandjelovic & Zisserman, 2018).

Original: This technique is used in various domains including image processing (Misra & Maaten, 2020; Jing & Tian, 2020;
Revised: This technique is used in various domains including image processing (Misra & Maaten, 2020; Jing & Tian, 2020;

Original: Grill et al., 2020), natural language understanding (Chen et al., 2020b; Du et al., 2020; Lan et al., 2019) or speech and audio processing (Baevski et al., 2020; Liu et al., 2020; Jiang et al., 2020).
Revised: Grill et al., 2020), natural language understanding (Chen et al., 2020b; Du et al., 2020; Lan et al., 2019) or speech and audio processing (Baevski et al., 2020b; Liu et al., 2020; Jiang et al., 2020).

Original: It offers numerous advantages, such as the independence from labeled data, stronger performance on downstream tasks, more robust models and an easier transfer to low-resource setups ( e.g. , low-resource languages)
Revised: It offers numerous advantages, such as the independence from labeled data, stronger performance on downstream tasks, more robust models and an easier transfer to low-resource setups ( e.g. , low-resource languages)

Original: (Baevski et al., 2020; Jing & Tian, 2020).
Revised: (Baevski et al., 2020b; Jing & Tian, 2020).

Original: The numerous existing SSL approaches are characterized by the nature of the pretext tasks they solve.
Revised: The numerous existing SSL approaches are characterized by the nature of the pretext tasks they solve.

Original: For instance, common techniques include predictive coding (Baevski et al., 2020; Liu et al., 2020; Song et al., 2020; Zhang et al., 2020; Hsu et al., 2021), pseudo-label learning (Pascual et al., 2019; Ravanelli et al., 2020), auto-encoding (Renshaw et al., 2015; Algayres et al., 2020),  generative modelling (Khurana et al., 2020) or contrastive learning (Saeed et al., 2020; Jiang et al., 2020).
Revised: For instance, common techniques include predictive coding (Baevski et al., 2020b; Liu et al., 2020; Song et al., 2020; Zhang et al., 2020; Hsu et al., 2021), pseudo-label learning (Pascual et al., 2019; Ravanelli et al., 2020), auto-encoding (Renshaw et al., 2015; Algayres et al., 2020), tripletloss learning (Shor et al., 2020; Peplinski et al., 2020), generative modelling (Khurana et al., 2020) or contrastive learning (Saeed et al., 2020; Jiang et al., 2020).

Original: More precisely, these pretext tasks may be deﬁned through the choice of pretext labels, hereafter referred to as pseudo-labels .
Revised: More precisely, these pretext tasks may be deﬁned through the choice of pretext labels, hereafter referred to as pseudo-labels .

Original: The automatic extraction of pseudo-labels for SSL ( i.e. from the data itself) is common in many application domains, such as computer vision (Noroozi & Favaro, 2017;
Revised: The automatic extraction of pseudo-labels for SSL ( i.e. from the data itself) is common in many application domains, such as computer vision (Noroozi & Favaro, 2017;

Original: Gidaris et al., 2018), music processing (Hung et al., 2019; Wu et al., 2021) and speech processing (Pascual et al., 2019; Shukla et al., 2020), and is commonly referred to as multitask self supervised learning .
Revised: Gidaris et al., 2018), music processing (Hung et al., 2019; Wu et al., 2021) and speech processing (Pascual et al., 2019; Shukla et al., 2020), and is commonly referred to as multitask self supervised learning .

Original: In the speciﬁc context of speech processing, the process of designing pseudo-labels may beneﬁt from decades of research in signal processing.
Revised: In the speciﬁc context of speech processing, the process of designing pseudo-labels may beneﬁt from decades of research in signal processing.

Original: For instance, potential candidates are pitch estimators, energy-based features, voicing state and many more.
Revised: For instance, potential candidates are pitch estimators, energy-based features, voicing state and many more.

Original: As demonstrated by Pascual et al.
Revised: As demonstrated by Pascual et al.

Original: (2019), multitask speech representation learning is a powerful tool to build representations that are beneﬁcial for a wide range of distinct downstream tasks, by com- bining different pseudo-labels which “intuitively” correspond to these tasks.
Revised: (2019), multitask speech representation learning is a powerful tool to build representations that are beneﬁcial for a wide range of distinct downstream tasks, by combining different pseudo-labels which “intuitively” correspond to these tasks.

Original: Unfortunately, there is no clear understanding of how these pseudo-labels may interact when optimised together, and therefore, no common practice of how to select groups of pseudo-labels to obtain better performance on a known downstream task.
Revised: Unfortunately, there is no clear understanding of how these pseudo-labels may interact when optimised together, and therefore, no common practice of how to select groups of pseudo-labels to obtain better performance on a known downstream task.

Original: As a matter of fact, this design process has been essentially driven by empirical validation and there is therefore no evidence that the obtained model is even the best one.
Revised: As a matter of fact, this design process has been essentially driven by empirical validation and there is therefore no evidence that the obtained model is even the best one.

Original: This empirical approach can rapidly become intractable with modern SSL architectures which may contain hundreds of millions or billions of parameters trained on thousands of hours of speech, not to mention the carbon footprint of such pseudo-label searches.
Revised: This empirical approach can rapidly become intractable with modern SSL architectures which may contain hundreds of millions or billions of parameters trained on thousands of hours of speech, not to mention the carbon footprint of such pseudo-label searches.

Original: For instance, the self-supervised training of a single state-of-the-art large wav2vec 2.0 model (Baevski et al., 2020) on 53 . 2 k hours of speech requires 128 GPUs for 5 . 2 days.
Revised: For instance, the self-supervised training of a single state-of-the-art large wav2vec 2.0 model (Baevski et al., 2020b) on 53 . 2 k hours of speech requires 128 GPUs for 5 . 2 days.

Original: This work aims at providing a clear, efﬁcient and theoretically motivated procedure for pseudolabel group selection and weighting based on conditional independence (CI).
Revised: This work aims at providing a clear, efﬁcient and theoretically motivated procedure for pseudolabel group selection and weighting based on conditional independence (CI).

Original: The method presented allows one to design ahead of training the most adapted multitask self-supervised speech representation learning model which perfectly suits the considered downstream tasks.
Revised: The method presented allows one to design ahead of training the most adapted multitask self-supervised speech representation learning model which perfectly suits the considered downstream tasks.

Original: Such an approach may also enable researchers to save a substantial amount of time and computation usually devoted to pseudo-label search.
Revised: Such an approach may also enable researchers to save a substantial amount of time and computation usually devoted to pseudo-label search.

Original: Hence, the contributions of this work are fourfold:
Revised: Hence, the contributions of this work are fourfold:

Original: Introduce a theoretically motivated and computationally efﬁcient method for the selection of pseudo-label groups among a set of candidates and with respect to the considered downstream tasks (Sections 3 and 4).
Revised: Introduce a theoretically motivated and computationally efﬁcient method for the selection of pseudo-label groups among a set of candidates and with respect to the considered downstream tasks (Sections 3 and 4).

Original: Validate empirically the proposed approach with a ﬁrst model SSL model relying on different sets of pseudo-labels corresponding to the ones obtained for three considered speech tasks.
Revised: Validate empirically the proposed approach with a ﬁrst model SSL model relying on different sets of pseudo-labels corresponding to the ones obtained for three considered speech tasks.

Original: (Sections 5).
Revised: (Sections 5).

Original: Extend our method to the SOTA wav2vec 2.0 to enhance its performance (Section 6).
Revised: Extend our method to the SOTA wav2vec 2.0 to enhance its performance (Section 6).

Original: Release the code base developed with SpeechBrain (Ravanelli et al., 2021) for replication and to encourage further investigations.
Revised: Release the code base developed with SpeechBrain (Ravanelli et al., 2021) for replication and to encourage further investigations.

Original: The conducted experiments demonstrate that the proposed method allows for a more intelligent, i.e. better informed, pseudo-label group selection for multitask SSL settings.
Revised: The conducted experiments demonstrate that the proposed method allows for a more intelligent, i.e. better informed, pseudo-label group selection for multitask SSL settings.

Original: Indeed, we ﬁnd that the models built with the proposed method obtain a word error rate and an equal error rate, respectively, 31 .
Revised: Indeed, we ﬁnd that the models built with the proposed method obtain a word error rate and an equal error rate, respectively, 31 .

Original: 6% and 27 .
Revised: 6% and 27 .

Original: 4% lower than the baseline, without the need for any empirical search.
Revised: 4% lower than the baseline, without the need for any empirical search.

Original:  R ELATED WORKS AND MOTIVATIONS
Revised:  R ELATED WORKS AND MOTIVATIONS

Original: SSL recently became a key component to achieve good performance on downstream tasks especially with low-resource setups, either in speech (Baevski et al., 2020; Conneau et al., 2020), natural language processing (Lan et al., 2019; Chen et al., 2020b) or computer vision (Gidaris et al., 2019;
Revised: SSL recently became a key component to achieve good performance on downstream tasks especially with low-resource setups, either in speech (Baevski et al., 2020b; Conneau et al., 2020), natural language processing (Lan et al., 2019; Chen et al., 2020b) or computer vision (Gidaris et al., 2019;

Original: Misra & Maaten, 2020; Jing & Tian, 2020).
Revised: Misra & Maaten, 2020; Jing & Tian, 2020).

Original: Due to its very nature, SSL relies on large amounts of unlabeled data used to train large deep neural networks over long periods of time.
Revised: Due to its very nature, SSL relies on large amounts of unlabeled data used to train large deep neural networks over long periods of time.

Original: It it thus crucial to understand properly what makes a good SSL model to lower the amount of computation and time needed to obtain the best downstream performance.
Revised: It it thus crucial to understand properly what makes a good SSL model to lower the amount of computation and time needed to obtain the best downstream performance.

Original: SSL for Speech.
Revised: SSL for Speech.

Original: Self-supervised learning for speech has recently enabled researchers to reach stateof-the-art results on various speech processing tasks (Fan et al., 2021).
Revised: Self-supervised learning for speech has recently enabled researchers to reach stateof-the-art results on various speech processing tasks (Fan et al., 2021).

Original: The most successful models rely on predictive and contrastive objectives (Baevski et al., 2020; Chung et al., 2019; Saeed et al., 2020) performing well across the different tasks even in low-resource settings.
Revised: The most successful models rely on predictive and contrastive objectives (Baevski et al., 2020b; Chung et al., 2019; Hsu et al., 2021; Shor et al., 2021) performing well across the different tasks even in low-resource settings.

Original: This led to the design of different benchmarks evaluating the self-supervised representations in different languages (Yang et al., 2021; Evain et al., 2021).
Revised: This led to the design of different benchmarks evaluating the self-supervised representations in different languages (Yang et al., 2021; Evain et al., 2021).

Original: However, in contrast to this proposition, these works have not tried to motivate beforehand the different choices made in the self-supervision pipeline.
Revised: However, in contrast to this proposition, these works have not tried to motivate beforehand the different choices made in the self-supervision pipeline.

Original: Understanding SSL.
Revised: Understanding SSL.

Original: A few works have tried to shed some theoretical light on the mainly empirical ﬁeld of self-supervised learning.
Revised: A few works have tried to shed some theoretical light on the mainly empirical ﬁeld of self-supervised learning.

Original: Following the different paradigms in SSL, various tracks havebeen followed to understand what makes for a good self-supervised representation, exploring different approaches (Lee et al., 2020; Arora et al., 2019; Wei et al., 2020). On the one hand, contrastive learning (Oord et al., 2018; Chen et al., 2020a) has been advocated both theoretically and empirically to achieve a balance in the mutual information between alternative representations of the data,
Revised: Following the different paradigms in SSL, various tracks have

Original: [Figure]
Revised: [Figure]

Original: Downstream
Revised: Pretext

Original: [Figure]
Revised: [Figure]

Original: Pretext
Revised: Self-Supervised Features

Original: [Figure]
Revised: [Figure]

Original: Conditional Independence Estimator
Revised: Pretraining Dataset Downstream Dataset Downstream Model 1. Pretext Tasks Group

Revised: [Figure]

Revised: and Weighting

Revised: Pretext Task Generation

Revised: Conditional Independence Estimator

Original: Selection of Pretext Tasks and
Revised: Selection of Pretext Tasks and

Original: Corresponding Weights
Revised: Corresponding Weights

Original: [Figure]
Revised: [Figure]

Revised: Encoder

Original:  keeping just enough shared information to keep the class-related content (Tschannen et al., 2020;
Revised: Downstream [Figure] been followed to understand what makes for a good self-supervised representation, exploring different approaches (Lee et al., 2020; Arora et al., 2019; Wei et al., 2020). On the one hand, contrastive learning (Oord et al., 2018; Chen et al., 2020a) has been advocated both theoretically and empirically to achieve a balance in the mutual information between alternative representations of the data, keeping just enough shared information to keep the class-related content (Tschannen et al., 2020;

Original: Tian et al., 2020; Bachman et al., 2019).
Revised: Tian et al., 2020; Bachman et al., 2019).

Original: In a recent work from Li et al.
Revised: In a recent work from Li et al.

Original: (2021), independence testing has been used to produce better transformations in contrastive learning settings for image representations.
Revised: (2021), independence testing has been used to produce better transformations in contrastive learning settings for image representations.

Original: Predictive learning, on the other hand, requires the model to predict masked elements in sequential data.
Revised: Predictive learning, on the other hand, requires the model to predict masked elements in sequential data.

Original: This technique is powerful on downstream tasks that can be reduced to a masking problem, as suggested by research on language modeling (Saunshi et al., 2020).
Revised: This technique is powerful on downstream tasks that can be reduced to a masking problem, as suggested by research on language modeling (Saunshi et al., 2020).

Original: However, all these works have been focusing on computer vision or text-related applications, and none of them addressed the multi-tasked self supervision problem.
Revised: However, all these works have been focusing on computer vision or text-related applications, and none of them addressed the multi-tasked self supervision problem.

Original: Multi-task self-supervised learning.
Revised: Multi-task self-supervised learning.

Original: While the literature on multi-tasking in self-supervised learning remains scarce, it has been shown in classic supervised learning settings, that through estimates of similarity between tasks or thorough empirical testing, several tasks can take advantage of being solved with a common encoder (Zamir et al., 2018; Dwivedi & Roig, 2019; Shafey et al., 2019;
Revised: While the literature on multi-tasking in self-supervised learning remains scarce, it has been shown in classic supervised learning settings, that through estimates of similarity between tasks or thorough empirical testing, several tasks can take advantage of being solved with a common encoder (Zamir et al., 2018; Dwivedi & Roig, 2019; Shafey et al., 2019;

Original: Chen et al., 2015).
Revised: Chen et al., 2015).

Original: More speciﬁcally, combining pretext tasks with SSL has been mainly explored in computer vision and speech (Pascual et al., 2019; Ravanelli et al., 2020).
Revised: More speciﬁcally, combining pretext tasks with SSL has been mainly explored in computer vision and speech (Pascual et al., 2019; Ravanelli et al., 2020).

Original: Pretext tasks such as
Revised: Pretext tasks such as

Original: Jigsaw (Doersch et al., 2016), colourisation and rotation (Gidaris et al., 2018) have been combined successfully to improve downstream performance (Kim et al., 2018; Shin’ya Yamaguchi et al.).
Revised: Jigsaw (Doersch et al., 2016), colourisation and rotation (Gidaris et al., 2018) have been combined successfully to improve downstream performance (Kim et al., 2018; Shin’ya Yamaguchi et al.).

Original: The two closest works to our line of research are from Lee et al.
Revised: The two closest works to our line of research are from Lee et al.

Original: (2020) and Doersch et al.
Revised: (2020) and Doersch et al.

Original: The former shows that a theoretical link can be established between conditional independence and an improvement of the performance on the downstream task, while the latter proposes to select layers from a multitask self-supervised encoder according to the pretext task to be solved.
Revised: The former shows that a theoretical link can be established between conditional independence and an improvement of the performance on the downstream task, while the latter proposes to select layers from a multitask self-supervised encoder according to the pretext task to be solved.

Original: However, in both cases, the studies do not offer practical and theoretical solutions to select groups of pseudo-labels to build an adapted SSL model that will perform well on the considered downstream tasks.
Revised: However, in both cases, the studies do not offer practical and theoretical solutions to select groups of pseudo-labels to build an adapted SSL model that will perform well on the considered downstream tasks.

Original: Group feature selection.
Revised: Group feature selection.

Original: Finally, feature selection, and especially group feature selection is another close and inspiring ﬁeld given the problem we consider.
Revised: Finally, feature selection, and especially group feature selection is another close and inspiring ﬁeld given the problem we consider.

Original: The relationship and interactions between features have been largely investigated in the supervised learning literature (Guyon & Elisseeff, 2003).
Revised: The relationship and interactions between features have been largely investigated in the supervised learning literature (Guyon & Elisseeff, 2003).

Original: This led to multiple solutions to the feature group selection problem, including LASSObased techniques (Yuan & Lin, 2006), or multiple kernel formulations (Sonnenburg et al., 2006;
Revised: This led to multiple solutions to the feature group selection problem, including LASSObased techniques (Yuan & Lin, 2006), or multiple kernel formulations (Sonnenburg et al., 2006;

Original: Rakotomamonjy et al., 2007).
Revised: Rakotomamonjy et al., 2007).

Original: However, these works do not involve any self-supervision, and links between feature selection and self-supervision design and pretext-task selection are yet to be proved.
Revised: However, these works do not involve any self-supervision, and links between feature selection and self-supervision design and pretext-task selection are yet to be proved.

Original: We will further consider these lines of works for concurrent baselines.
Revised: We will further consider these lines of works for concurrent baselines.

Original: With this work, we aim at shortening the process of designing SSL models while giving insights on the pseudo-label importance and the underlying mechanisms between pretext and downstream tasks at the same time.
Revised: With this work, we aim at shortening the process of designing SSL models while giving insights on the pseudo-label importance and the underlying mechanisms between pretext and downstream tasks at the same time.

Original: We decided to experiment with speech due to the lack of literature on this domain for multitask SSL, and for the various pseudo-labels available, which are based on decades of signal processing research.
Revised: We decided to experiment with speech due to the lack of literature on this domain for multitask SSL, and for the various pseudo-labels available, which are based on decades of signal processing research.

Original: The whole pipeline starting from the acoustic feature extraction to the downstream task scoring follows three major steps summarized in Figure 1.
Revised: The whole pipeline starting from the acoustic feature extraction to the downstream task scoring follows three major steps summarized in Figure 1.

Original: First, for every downstream task, our method produces a pretext task selection and weighting.
Revised: First, for every downstream task, our method produces a pretext task selection and weighting.

Original: Then, a SSL model is trained, before being used as a feature extractor front-end to one or many downstream tasks.
Revised: Then, a SSL model is trained, before being used as a feature extractor front-end to one or many downstream tasks.

Original:  C ONDITIONAL INDEPENDENCE FOR UTILITY ESTIMATION
Revised:  C ONDITIONAL INDEPENDENCE FOR UTILITY ESTIMATION

Original: As a ﬁrst step, we require a function that estimates the utility of learning to solve a pretext-task to improve the performance on the downstream task.
Revised: As a ﬁrst step, we require a function that estimates the utility of learning to solve a pretext-task to improve the performance on the downstream task.

Original: We use an estimation of the conditional independence between the pseudo-label values and the downstream data points given the downstream labels.
Revised: We use an estimation of the conditional independence between the pseudo-label values and the downstream data points given the downstream labels.

Original: Hereafter, we explain the theoretical motivations and describe the computation steps.
Revised: Hereafter, we explain the theoretical motivations and describe the computation steps.

Original: 3. P ROBLEM DEFINITION AND INTUITION
Revised: 3. P ROBLEM DEFINITION AND INTUITION

Original: Let X , Y and Z be, respectively, the downstream data points, their downstream labels and their pseudo-labels.
Revised: Let X , Y and Z be, respectively, the downstream data points, their downstream labels and their pseudo-labels.

Original: Let also C be the set of possible downstream classes.
Revised: Let also C be the set of possible downstream classes.

Original: As an example, if one considers speaker recognition as a downstream task, X would be the speech samples, Y the speaker IDs, C the set of unique speaker IDs, and Z a computed signal feature, such as the fundamental frequency.
Revised: As an example, if one considers speaker recognition as a downstream task, X would be the speech samples, Y the speaker IDs, C the set of unique speaker IDs, and Z a computed signal feature, such as the fundamental frequency.

Original: As stated in Section 2, Lee et al.
Revised: As stated in Section 2, Lee et al.

Original: (2020) linked the utility of a pseudo-label ( Z ) to the conditional independence (CI) between Z and X given Y .
Revised: (2020) linked the utility of a pseudo-label ( Z ) to the conditional independence (CI) between Z and X given Y .

Original: The approach prescribes that, given the labels Y , one may seek to quantify how much it is possible to predict the pseudo-labels Z without knowing much about X .
Revised: The approach prescribes that, given the labels Y , one may seek to quantify how much it is possible to predict the pseudo-labels Z without knowing much about X .

Original: The authors bounded, under certain assumptions, the downstream classiﬁer error with a function of the downstream training set size, and a measure of the CI.
Revised: The authors bounded, under certain assumptions, the downstream classiﬁer error with a function of the downstream training set size, and a measure of the CI.

Original: More precisely, the main theorem shows that the bounding function decreases linearly with the downstream-task dataset size ( M ) and quadratically with the CI, thus making it a potential estimator for pseudo-label utility.
Revised: More precisely, the main theorem shows that the bounding function decreases linearly with the downstream-task dataset size ( M ) and quadratically with the CI, thus making it a potential estimator for pseudo-label utility.

Original: The proposed function depends on the ﬁnal downstream task to be solved.
Revised: The proposed function depends on the ﬁnal downstream task to be solved.

Original: This is motivated by two main reasons.
Revised: This is motivated by two main reasons.

Original: First, it can be seen through the large literature on feature selection for various speech or computer vision tasks (Liu et al., 2020;
Revised: First, it can be seen through the large literature on feature selection for various speech or computer vision tasks (Liu et al., 2020;

Original: Serizel et al., 2017; Schuller et al., 2007; Wang et al., 2019), that different tasks require the description of different aspects of the data.
Revised: Serizel et al., 2017; Schuller et al., 2007; Wang et al., 2019), that different tasks require the description of different aspects of the data.

Original: This suggests that different downstream tasks may perform better after different pre-trainings.
Revised: This suggests that different downstream tasks may perform better after different pre-trainings.

Original: A second argument is the difﬁculty to evaluate representations quality intrinsically, i.e. independently from the choice of a particular downstream task.
Revised: A second argument is the difﬁculty to evaluate representations quality intrinsically, i.e. independently from the choice of a particular downstream task.

Original: A few metrics and tests (Schatz et al., 2013; Carlin et al., 2011; Lakhotia et al., 2021) have been proposed for speech, but the correlation between these and downstream-task performance has not been clearly identiﬁed (Algayres et al., 2020;
Revised: A few metrics and tests (Schatz et al., 2013; Carlin et al., 2011; Lakhotia et al., 2021) have been proposed for speech, but the correlation between these and downstream-task performance has not been clearly identiﬁed (Algayres et al., 2020;

Original: Gump et al., 2020).
Revised: Gump et al., 2020).

Original: The principal issue with CI is the difﬁculty of computing good estimates of how much two variables are independent given a third one on realistic data (Shah & Peters, 2018).
Revised: The principal issue with CI is the difﬁculty of computing good estimates of how much two variables are independent given a third one on realistic data (Shah & Peters, 2018).

Original: In a previous work (double-blind reviewing process) , we proposed a simple way to get an estimation of the conditional independence.
Revised: In a previous work (not cited to respect the double blind reviewing), we proposed a simple way to get an estimation of the conditional independence.

Original: This method has proven effective for individual pretext task selection, as the utility estimator correlated highly with the ﬁnal downstream performances.
Revised: This method has proven effective for individual pretext task selection, as the utility estimator correlated highly with the ﬁnal downstream performances.

Original: In our case, the considered pseudo-labels are not independent of the speech samples, as they are signal features.
Revised: In our case, the considered pseudo-labels are not independent of the speech samples, as they are signal features.

Original: The approach resorts to performing classic independence testing on data sliced by the downstream classes, to check whether this dependence remains given the downstream labels information.
Revised: The approach resorts to performing classic independence testing on data sliced by the downstream classes, to check whether this dependence remains given the downstream labels information.

Original: 3. C ONDITIONAL INDEPENDENCE ESTIMATOR COMPUTATION
Revised: 3. C ONDITIONAL INDEPENDENCE ESTIMATOR COMPUTATION

Original: This section details the computation of the conditional independence estimate that is used as a measure of pseudo-label utility.
Revised: This section details the computation of the conditional independence estimate that is used as a measure of pseudo-label utility.

Original: Let X
Revised: Let X

Original: = { x i } i ∈{ 0 ,...,M } with M being the cardinal of X and x i data samples ( e.g. Mel-band spectrogram for audio and speech).
Revised: = { x i } i ∈{ 0 ,...,M } with M being the cardinal of X and x i data samples ( e.g. Mel-band spectrogram for audio and speech).

Original: Every sample x i has a corresponding downstream label y i and an automatically generated pseudo-label z i .
Revised: Every sample x i has a corresponding downstream label y i and an automatically generated pseudo-label z i .

Original: We assume that y i is discrete reducing the task to a classiﬁcation problem such as with speaker ID for speaker recognition.
Revised: We assume that y i is discrete reducing the task to a classiﬁcation problem such as with speaker ID for speaker recognition.

Original: We also assume that for every pretext-task Z , a single z i value corresponds to each x i .
Revised: We also assume that for every pretext-task Z , a single z i value corresponds to each x i .

Original: In our case, z i values are the mean of the frame-wise pseudo-label values.
Revised: In our case, z i values are the mean of the frame-wise pseudo-label values.

Original: For independence testing, we decided to rely on the kernel-based Hilbert Schmidt Independence
Revised: For independence testing, we decided to rely on the kernel-based Hilbert Schmidt Independence

Original: Criterion (HSIC) (Gretton et al., 2007) for two reasons.
Revised: Criterion (HSIC) (Gretton et al., 2007) for two reasons.

Original: First, HSIC has already proven successful for textual data in testing statistical dependence between translated sentences (Gretton et al., 2007).
Revised: First, HSIC has already proven successful for textual data in testing statistical dependence between translated sentences (Gretton et al., 2007).

Original: Second, kernel-based techniques facilitate the handling of multivariate and varying-length data such as speech, as the estimation then boils down to the computation of a similarity measure between the considered variables.
Revised: Second, kernel-based techniques facilitate the handling of multivariate and varying-length data such as speech, as the estimation then boils down to the computation of a similarity measure between the considered variables.

Original: Computation steps.
Revised: Computation steps.

Original: The estimation of the CI of a pseudo-label Z for a downstream task ( X, Y ) consists of three steps.
Revised: The estimation of the CI of a pseudo-label Z for a downstream task ( X, Y ) consists of three steps.

Original: We start by splitting the data samples X according to the downstream (discrete) classes.
Revised: We start by splitting the data samples X according to the downstream (discrete) classes.

Original: Then, we compute for every downstream class c ∈ C , the kernel matrices K c and
Revised: Then, we compute for every downstream class c ∈ C , the kernel matrices K c and

Original: L c representing the similarity measures for the data samples, and the pseudo-labels, respectively.
Revised: L c representing the similarity measures for the data samples, and the pseudo-labels, respectively.

Original: Finally, we perform the independence test for every split group using K c and L c and aggregate the estimates with a weighted mean taking into account the number of samples per downstream class.
Revised: Finally, we perform the independence test for every split group using K c and L c and aggregate the estimates with a weighted mean taking into account the number of samples per downstream class.

Original: Thus, for two speech samples x i and x j , holding two pseudo-label values z i and z j , the coefﬁcients of the similarity matrices K c and L c are computed as follows:
Revised: Thus, for two speech samples x i and x j , holding two pseudo-label values z i and z j , the coefﬁcients of the similarity matrices K c and L c are computed as follows:

Original: [Equation]
Revised: [Equation]

Original: with GD ( . )
Revised: with GD ( . )

Original: the Gaussian Downsampling function (more details in the appendix A.6) , cos( ., . )
Revised: the Gaussian Downsampling function (more details in the appendix A.7) , cos( ., . )

Original: the cosine similarity, and RBF ( ., . )
Revised: the cosine similarity, and RBF ( ., . )

Original: the Radial Basis Function kernel, deﬁned as:
Revised: the Radial Basis Function kernel, deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where σ is the width of the RBF kernel and trace ( . )
Revised: where σ is the width of the RBF kernel and trace ( . )

Original: the sum of elements of the main diagonal.
Revised: the sum of elements of the main diagonal.

Original: Note that we compute the matrices K c and L c , for each group of samples sharing the same downstream class c ∈ C .
Revised: Note that we compute the matrices K c and L c , for each group of samples sharing the same downstream class c ∈ C .

Original: Hence, K c and L c correspond to the deﬁnitions above, but restricted to the points with c as a downstream label.
Revised: Hence, K c and L c correspond to the deﬁnitions above, but restricted to the points with c as a downstream label.

Original: For each downstream class c , and as in Gretton et al.
Revised: For each downstream class c , and as in Gretton et al.

Original: (2007), the HSIC value is given by:
Revised: (2007), the HSIC value is given by:

Original: [Equation]
Revised: [Equation]

Original: with H c
Revised: with H c

Original: =
Revised: =

Original: I n c − 1 n c 1 n c 1 Tn c , n c being the number of points with downstream label c , and 1 n c a vector of ones of size n c × 1 .
Revised: I n c − 1 n c 1 n c 1 Tn c , n c being the number of points with downstream label c , and 1 n c a vector of ones of size n c × 1 .

Original: The HSIC value is non-negative and corresponds to the Hilbert norm of their cross-covariance matrix.
Revised: The HSIC value is non-negative and corresponds to the Hilbert norm of their cross-covariance matrix.

Original: It is used to characterize the independence of the two considered quantities.
Revised: It is used to characterize the independence of the two considered quantities.

Original: Intuitively, the
Revised: Intuitively, the

Original: HSIC value is high if samples similar in K c are similar in L c .
Revised: HSIC value is high if samples similar in K c are similar in L c .

Original: Therefore, the lower this value is, the more independent the two arguments of HSIC are and the better the pseudo-label should be for self-supervision before ﬁne-tuning on the downstream class.
Revised: Therefore, the lower this value is, the more independent the two arguments of HSIC are and the better the pseudo-label should be for self-supervision before ﬁne-tuning on the downstream class.

Original: The ﬁnal value for a given pseudo-label and a downstream task is expressed as:
Revised: The ﬁnal value for a given pseudo-label and a downstream task is expressed as:

Original: [Equation]
Revised: [Equation]

Revised: Computational efﬁciency.

Revised: Efﬁciency is one of the key motivations of this work, and the gain in time observed with our approach is considerable.

Revised: The K and L matrices used for the CI estimate are only computed for the downstream datasets.

Revised: But since these datasets may get very large, we can sample among the downstream classes to keep the computations tractable as shown in appendix A.3.

Original:  P RETEXT TASK GROUP SELECTION AND WEIGHTING
Revised:  P RETEXT TASK GROUP SELECTION AND WEIGHTING

Original: While we now are able to predict the utility of every considered pretext task independently, the next step remains to deﬁne a clever way to combine them optimally within the same pre-training process.
Revised: While we now are able to predict the utility of every considered pretext task independently, the next step remains to deﬁne a clever way to combine them optimally within the same pre-training process.

Original: Hence, we introduce a method to select a group of pseudo-labels and weight their respective losses to increase or decrease their importance in the self-supervised representation.
Revised: Hence, we introduce a method to select a group of pseudo-labels and weight their respective losses to increase or decrease their importance in the self-supervised representation.

Original: Such an optimisation of the latent representation is expected to provide better downstream performance.
Revised: Such an optimisation of the latent representation is expected to provide better downstream performance.

Original: Our method minimises the conditional dependence between the resulting group pretext task, entailing the prediction of a selected pseudo-label group and the downstream samples given the downstream labels.
Revised: Our method minimises the conditional dependence between the resulting group pretext task, entailing the prediction of a selected pseudo-label group and the downstream samples given the downstream labels.

Original: Given a set of k possible pseudo-labels ( Z i ) i ∈
Revised: Given a set of k possible pseudo-labels ( Z i ) i ∈

Original: [0 ,k ] , we seek to estimate a set of parameters ( λ i ) i ∈ [0 ,k ] weighting the loss of every pseudo-label k during the pre-training phase.
Revised: [0 ,k ] , we seek to estimate a set of parameters ( λ i ) i ∈ [0 ,k ] weighting the loss of every pseudo-label k during the pre-training phase.

Original: Hence, we deﬁne a grouping pseudo-label Z λ as an orthogonal concatenation of ( Z i ) i ∈ [0 ,k ] weighted with ( λ i ) i ∈ [0 ,k ] :
Revised: Hence, we deﬁne a grouping pseudo-label Z λ as an orthogonal concatenation of ( Z i ) i ∈ [0 ,k ] weighted with ( λ i ) i ∈ [0 ,k ] :

Original: [Equation]
Revised: Z λ = ( λ 1 Z 1 , ..., λ k Z k ) .

Original: The custom conditional HSIC computation pipeline described above is fully differentiable with respect to ( λ i ) i ∈ [0 ,k ] as proved in appendix A.1.
Revised: The custom conditional HSIC computation pipeline described above is fully differentiable with respect to ( λ i ) i ∈ [0 ,k ] as proved in appendix A.1.

Original: In the HSIC computation, the data similarity matrices { K c } c ∈ C are independent of Z and therefore of λ .
Revised: In the HSIC computation, the data similarity matrices { K c } c ∈ C are independent of Z and therefore of λ .

Original: Only the pseudo-label similarity matrices { L c } c ∈ C are changed.
Revised: Only the pseudo-label similarity matrices { L c } c ∈ C are changed.

Original: For every downstream class c , L c is deﬁned as:
Revised: For every downstream class c , L c is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: with z h,i denotes the mean value of the h -th pseudo-label for the i -th data point in the dataset.
Revised: with z h,i denotes the mean value of the h -th pseudo-label for the i -th data point in the dataset.

Original: 4. C ONSTRAINTS ON THE WEIGHTS
Revised: 4. C ONSTRAINTS ON THE WEIGHTS

Original: The conditional-independence based utility estimator must be optimized with respect to the weighting parameters ( λ i ) i ∈ [0 ,k ] and three constraints.
Revised: The conditional-independence based utility estimator must be optimized with respect to the weighting parameters ( λ i ) i ∈ [0 ,k ] and three constraints.

Original: First, the parameters ( λ i ) i ∈ [0 ,k ] must be positive, as they are used as weights for the corresponding losses.
Revised: First, the parameters ( λ i ) i ∈ [0 ,k ] must be positive, as they are used as weights for the corresponding losses.

Original: A negative weighting loss would lack interpretability as it could imply that the selfsupervised model should “unlearn” the corresponding pretext task.
Revised: A negative weighting loss would lack interpretability as it could imply that the selfsupervised model should “unlearn” the corresponding pretext task.

Original: This may be the case for adversarial learning methods, but we are not considering this case in the present work.
Revised: This may be the case for adversarial learning methods, but we are not considering this case in the present work.

Original: Second, the value of the weights must be high enough.
Revised: Second, the value of the weights must be high enough.

Original: Indeed, the presented method for estimating the conditional independence assumes that the considered pseudo-label Z is not independent of X .
Revised: Indeed, the presented method for estimating the conditional independence assumes that the considered pseudo-label Z is not independent of X .

Original: It is fortunately true for speech features, as Z is a function of X , but not always valid.
Revised: It is fortunately true for speech features, as Z is a function of X , but not always valid.

Original: For instance, with ( λ i ) i ∈ [0 ,k ] = 0 , the utility estimator would be null and thus the lowest ( i.e. the best), but it would break the assumption of non independence between Z and X .
Revised: For instance, with ( λ i ) i ∈ [0 ,k ] = 0 , the utility estimator would be null and thus the lowest ( i.e. the best), but it would break the assumption of non independence between Z and X .

Original: Furthermore, the HSIC value decreases with positive decreasing values of ( λ i ) i ∈ [0 ,k ] and we thus need to ensure that the sum of the weights is signiﬁcantly higher than zero, or it would mean that we are not really doing multi-task learning as the losses of the pseudo-labels would be barely considered.
Revised: Furthermore, the HSIC value decreases with positive decreasing values of ( λ i ) i ∈ [0 ,k ] and we thus need to ensure that the sum of the weights is signiﬁcantly higher than zero, or it would mean that we are not really doing multi-task learning as the losses of the pseudo-labels would be barely considered.

Original: Finally, for a fair comparison between the weighting choices during the optimization, the sum of the weights should remain constant.
Revised: Finally, for a fair comparison between the weighting choices during the optimization, the sum of the weights should remain constant.

Original: In the following, the sum of the ( λ i ) i ∈ [0 ,k ] is ﬁxed to one and the problem is summarized as follows:min [Equation] R k
Revised: In the following, the sum of the ( λ i ) i ∈ [0 ,k ] is ﬁxed to one and the problem is summarized as follows:

Original: [Equation]
Revised: [Equation]

Original: To minimize the estimator quantity while respecting the constraints, the weights used in the computation of the CI value are the softmax output of free learnable parameters ( W i ) i ∈
Revised: To minimize the estimator quantity while respecting the constraints, the weights used in the computation of the CI value are the softmax output of free learnable parameters ( W i ) i ∈

Original: [0 ,k ] .
Revised: [0 ,k ] .

Original: Softmax enforces the conditions while being differentiable and the problem becomes:
Revised: Softmax enforces the conditions while being differentiable and the problem becomes:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  W EIGHTS SPARSITY
Revised: 4. W EIGHTS SPARSITY

Original: Another trait that is desirable for the weighting vector is sparsity.
Revised: Another trait that is desirable for the weighting vector is sparsity.

Original: If a few pseudo-labels are not needed for the given downstream task, they would rather be discarded than given a low weight.
Revised: If a few pseudo-labels are not needed for the given downstream task, they would rather be discarded than given a low weight.

Original: First, this would save computation time including the extraction of the pseudo-labels, and their extraction and prediction during the self-supervised training process.
Revised: First, this would save computation time including the extraction of the pseudo-labels, and their extraction and prediction during the self-supervised training process.

Original: Second, it would help with transparency to understand what features are included or not in the latent space.
Revised: Second, it would help with transparency to understand what features are included or not in the latent space.

Original: This sparsity property is also related to feature selection such as with LASSO (Yuan & Lin, 2006).
Revised: This sparsity property is also related to feature selection such as with LASSO (Yuan & Lin, 2006).

Original: To ensure the sparsity of the output weighting vector, while maintaining the desired property of differentiability, we choose the sparsemax function (Martins & Astudillo, 2016) to replace the softmax in eq.
Revised: To ensure the sparsity of the output weighting vector, while maintaining the desired property of differentiability, we choose the sparsemax function (Martins & Astudillo, 2016) to replace the softmax in eq.

Original:  E XPERIMENTAL STUDY
Revised:  E XPERIMENTAL STUDY

Original: This section details the experiments validating the introduced estimator.
Revised: This section details the experiments validating the introduced estimator.

Original: It describes the selection and weighting processes (Section 5.1), the SSL models (Section 5.2), the downstream tasks (Section 5.3), and the obtained results (Section 5.4).
Revised: It describes the selection and weighting processes (Section 5.1), the SSL models (Section 5.2), the downstream tasks (Section 5.3), and the obtained results (Section 5.4).

Original: 5. G ROUP SELECTION AND WEIGHTING
Revised: 5. G ROUP SELECTION AND WEIGHTING

Original: Individual pseudo-labels of interest are obtained with the OpenSmile library (Eyben et al., 2010).
Revised: Individual pseudo-labels of interest are obtained with the OpenSmile library (Eyben et al., 2010).

Original: We decided to focus on markers mostly related to prosody and spectral descriptors as the signal processing literature commonly associates them to the three considered downstream tasks ( i.e. speech, speaker and emotion recognition).
Revised: We decided to focus on markers mostly related to prosody and spectral descriptors as the signal processing literature commonly associates them to the three considered downstream tasks ( i.e. speech, speaker and emotion recognition).

Original: Selected pseudo-labels include: Loudness, F0, Voicing, α Ratio (Sundberg & Nordenberg, 2006), Zero Crossing Rate, L1 Norm of Rasta Spectrum (Hermansky et al., 1992), log of Harmonicity to Noise Ratio (Murphy & Akande, 2005).
Revised: Selected pseudo-labels include: Loudness, F0, Voicing, α Ratio (Sundberg & Nordenberg, 2006), Zero Crossing Rate, L1 Norm of Rasta Spectrum (Hermansky et al., 1992), log of Harmonicity to Noise Ratio (Murphy & Akande, 2005).

Original: Then, and according to
Revised: Then, and according to

Original: Figure 1 ( step 1 ), we group these pseudo-labels based on their weights, i.e. by optimising equationto obtain the different λ values associated to each one of them.
Revised: Figure 1 ( step 1 ), we group these pseudo-labels based on their weights, i.e. by optimising equationto obtain the different λ values associated to each one of them.

Original: Comparative baselines follow common feature group selection strategies or natural intuitions.
Revised: Comparative baselines follow common feature group selection strategies or natural intuitions.

Original: The ﬁrst one simply bundles all the pseudo-labels together without any weighting ( i.e. λ = 1 for all pseudo-labels) as proposed for PASE (Pascual et al., 2019).
Revised: The ﬁrst one simply bundles all the pseudo-labels together without any weighting ( i.e. λ = 1 for all pseudo-labels) as proposed for PASE (Pascual et al., 2019).

Original: As SSL group pretext-task selection is yet to be fully explored, the two other baselines come from the feature selection literature as it represents the closest ﬁeld with well established techniques.
Revised: As SSL group pretext-task selection is yet to be fully explored, the two other baselines come from the feature selection literature as it represents the closest ﬁeld with well established techniques.

Original: The CI-based pseudo-label selection is thus compared to Recursive Feature Elimination (RFE, Guyon et al.
Revised: The CI-based pseudo-label selection is thus compared to Recursive Feature Elimination (RFE, Guyon et al.

Original: and Maximum Relevance Minimum Redundancy (MRMR, Peng et al.
Revised: and Maximum Relevance Minimum Redundancy (MRMR, Peng et al.

Revised: We also add an experiment with the remaining pretext tasks after the MRMR selection, this is to show the effect of learning supposedly unrelated workers, we will call it ”Remaining”.

Original: More details about these baseline algorithms are given in Appendix A.7, while Appendix A.9 shows the workers selected and their corresponding weights for every experiment.
Revised: More details about these baseline algorithms are given in Appendix A.9, while Appendix A.11 shows the workers selected and their corresponding weights for every experiment.

Original: Noise (HNR) seems to be the most important information to learn to predict for speaker recognition while fundamental frequency is privileged for ASR.
Revised: Noise (HNR) seems to be the most important information to learn to predict for speaker recognition while fundamental frequency is privileged for ASR.

Original: 5. S ELF - SUPERVISED TRAINING
Revised: 5. S ELF - SUPERVISED TRAINING

Original: In the second step of Figure 1, the SSL model learns to predict the selected pseudo-labels.
Revised: In the second step of Figure 1, the SSL model learns to predict the selected pseudo-labels.

Original: For every one of those, the loss is multiplied by the corresponding assigned weight.
Revised: For every one of those, the loss is multiplied by the corresponding assigned weight.

Original: Based on previous work conclusions (Ravanelli et al., 2020; Jiang et al., 2020) and apart from the considered pretext task, the network learns to reconstruct the input Mel spectrograms, and to compute 40 -dimensional MelFrequency Cepstral Coefﬁcients (MFCC) feature vectors.
Revised: Based on previous work conclusions (Ravanelli et al., 2020; Jiang et al., 2020) and apart from the considered pretext task, the network learns to reconstruct the input Mel spectrograms, and to compute 40 -dimensional MelFrequency Cepstral Coefﬁcients (MFCC) feature vectors.

Original: These targets are usually kept to avoid information loss harming heavily downstream performance and are used in all our experiments.
Revised: These targets are usually kept to avoid information loss harming heavily downstream performance and are used in all our experiments.

Original: For a given weighting vector ( λ i ) i ∈ [0 ,k ] , the self-supervised loss is deﬁned as:
Revised: For a given weighting vector ( λ i ) i ∈ [0 ,k ] , the self-supervised loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original:  with MSE the classic mean squared error computed for Mel spectra ( MSE mel ) and MFCC ( MSE mfcc ), and (cid:96) 

Revised: (8) with MSE the classic mean squared error computed for Mel spectra ( MSE mel ) and MFCC ( MSE mfcc ), and (cid:96)

Original: ( Z ) the (cid:96) 1 -loss of the pretext task related to pseudo-label Z .
Revised: ( Z ) the (cid:96) 1 -loss of the pretext task related to pseudo-label Z .

Original: Prior to extending our method to state-of-the-art architectures such as wav2vec 2.0 that are particularly costly to train, we propose to ﬁrst employ a PASE-like model to empirically validate the approach.
Revised: Prior to extending our method to state-of-the-art architectures such as wav2vec 2.0 that are particularly costly to train, we propose to ﬁrst employ a PASE-like model to empirically validate the approach.

Original: Hence, the encoder is composed of three distinct parts: a VGG-like feature extractor (Simonyan & Zisserman, 2015), a bidirectional LSTM, and a two-layered dense neural network.
Revised: Hence, the encoder is composed of three distinct parts: a VGG-like feature extractor (Simonyan & Zisserman, 2015), a bidirectional LSTM, and a two-layered dense neural network.

Original: All the details of the architecture are given in the appendix A.4.
Revised: All the details of the architecture are given in the appendix A.5.

Original: Then, and inspired by Ravanelli et al.
Revised: Then, and inspired by Ravanelli et al.

Original: (2020), the encoder is followed by simple one-layered predictors voluntarily limited in capacity.
Revised: (2020), the encoder is followed by simple one-layered predictors voluntarily limited in capacity.

Original: SSL dataset.
Revised: SSL dataset.

Original: The SSL model is optimised on the training set of the English Common Voice dataset (version 5 . 1 , 700 hours of training, Ardila et al.
Revised: The SSL model is optimised on the training set of the English Common Voice dataset (version 5 . 1 , 700 hours of training, Ardila et al.

Original: Common Voice is a collection of speech utterances from worldwide users recording themselves from their own devices.
Revised: Common Voice is a collection of speech utterances from worldwide users recording themselves from their own devices.

Original: Hence, the closeness to natural settings makes it a suitable choice for self-supervised learning.
Revised: Hence, the closeness to natural settings makes it a suitable choice for self-supervised learning.

Original: 700 hours of speech is a relatively small amount compared to what is generally used for state-of-the-art SSL models.
Revised: 700 hours of speech is a relatively small amount compared to what is generally used for state-of-the-art SSL models.

Original: However, we believe it is a sound choice as this is generally greater than what is typically available in SSL use-cases like low-resource languages.
Revised: However, we believe it is a sound choice as this is generally greater than what is typically available in SSL use-cases like low-resource languages.

Original: We decided to not use the LibriSpeech dataset for pre-training as it is part of our downstream evaluation protocol hence alleviating a strong bias.
Revised: We decided to not use the LibriSpeech dataset for pre-training as it is part of our downstream evaluation protocol hence alleviating a strong bias.

Original: 5. D OWNSTREAM TASKS
Revised: 5. D OWNSTREAM TASKS

Original: Our proposed pseudo-label selection strategy is compared with the two baselines on two different downstream tasks leading to different groups of pseudo-labels: automatic speech recognition (ASR, with LibriSpeech 100 hours)and speaker recognition (SR, with VoxCeleb 1).
Revised: Our proposed pseudo-label selection strategy is compared with the two baselines on three different downstream tasks leading to different groups of pseudo-labels: automatic speech recognition (ASR, with LibriSpeech 100 hours) speaker recognition (SR, with VoxCeleb 1), and emotion recognition (ER with IEMOCAP).

Original: Datasets and downstream architectures are inspired from the SUPERB  benchmark (Yang et al., 2021) for selfsupervised learning representations and carefully described in Appendix A.4.3.
Revised: Datasets and downstream architectures are inspired from the SUPERB [Table] benchmark (Yang et al., 2021) for self-supervised learning representations and carefully described in Appendix A.5.3.

Original: Prior to downstream training, the SSL model are frozen to be used as a feature extractor with the new pipeline that is taskdependent.
Revised: Prior to downstream training, the SSL model are frozen to be used as a feature extractor with the new pipeline that is task-dependent.

Original: We do not use any data augmentation for a pristine comparison of the learned models.[Table] [Table]
Revised: We do not use any data augmentation for a pristine comparison of the learned models.

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: Baselines detailed in Section 4 are respectively referred to as “ All ”, “ RFE ” and “ MRMR ”.
Revised: [Table] detailed in Section 4 are respectively referred to as “ All ”, “ RFE ” and “ MRMR ”.

Original: All the details about the selection and weights are available in Appendix A.9.
Revised: All the details about the selection and weights are available in Appendix A.11.

Original: First, it is clear from the results reported in Table 1 that, for the considered downstream tasks, the two introduced strategies ( Sparsemax and Softmax ) perform better than the group selection baselines with a gain of 3 .of
Revised: First, it is clear from the results reported in Table 1 that, for the considered downstream tasks, the two introduced strategies ( Sparsemax and Softmax ) perform better than the group selection baselines with a gain of 3 . 28 of

Original: EER for Sparsemax against the RFE approach on VoxCeleb, and 8 .
Revised: EER for Sparsemax against the RFE approach on VoxCeleb, and 8 .

Original: 81 of WER with Softmax compared to the All baseline.
Revised: 81 of WER with Softmax compared to the All baseline.

Original: Interestingly, simply bundling all the pseudo-labels together may lead to poor performance as observed on LibriSpeech with a very high 21 .
Revised: Interestingly, simply bundling all the pseudo-labels together may lead to poor performance as observed on LibriSpeech with a very high 21 .

Original: 98% of WER obtained.
Revised: 98% of WER obtained.

Original: Hence, intuitively building sets of labels could be harmful for the ﬁnal representation.
Revised: Hence, intuitively building sets of labels could be harmful for the ﬁnal representation.

Original: This motivates the need for a better pseudo-label selection strategy such as the one introduced in this work, as the WER dropped to 13 .
Revised: This motivates the need for a better pseudo-label selection strategy such as the one introduced in this work, as the WER dropped to 13 .

Original: As a comparison, the exact same architecture trained with Mel spectra only ( i.e. no SSL) obtains a WER of 17 .
Revised: As a comparison, the exact same architecture trained with Mel spectra only ( i.e. no SSL) obtains a WER of 17 .

Original: 3 % without LM.
Revised: 3 % without LM.

Original: Hence, our method even further decreases the WER while being only pretrained with a reasonable amount of data ( i.e. only 700 hours compared to a few thousands for common SSL techniques (Baevski et al., 2020)).
Revised: Hence, our method even further decreases the WER while being only pretrained with a reasonable amount of data ( i.e. only 700 hours compared to a few thousands for common SSL techniques (Baevski et al., 2020b)).

Original: As expected, introducing the joint decoding with a language model strongly decreases the WER but also introduces a bias in our comparison as probabilities are smoothed with a third-party neural model.
Revised: As expected, introducing the joint decoding with a language model strongly decreases the WER but also introduces a bias in our comparison as probabilities are smoothed with a third-party neural model.

Original: Nevertheless, and even in this scenario, our weighting strategy outperforms all the baselines.
Revised: Nevertheless, and even in this scenario, our weighting strategy outperforms all the baselines.

Original: In the context of speaker recognition, Sparsemax beats Softmax with an EER 0 .
Revised: In the context of speaker recognition, Sparsemax beats Softmax with an EER 0 .

Original: 61 lower.
Revised: 61 lower.

Revised: For IEMOCAP, Softmax and Sparsemax weighting still perform the best among all methods.

Revised: To investigate how strongly improvements are correlated to the task, we took the best learned model for LibriSpeech (i.e. softmax weighting) and ﬁne-tuned it on VoxCeleb1 and IEMOCAP.

Revised: It reaches an EER of 10 .

Original:   E XTENDING WAV 2 VEC 2.0 TO MULTITASK SSL
Revised: 55% and an accuracy of 59 . 9% respectively. While it performs better than the baselines, the difference between these results and the best performing selections shows that the weightings are indeed task-related.  E XTENDING WAV 2 VEC 2.0 TO MULTITASK SSL

Original: To the best of our knowledge, multi-task speech representation learning has not been scaled to a point where it could represent a state-of-the-art alternative.
Revised: To the best of our knowledge, multi-task speech representation learning has not been scaled to a point where it could represent a state-of-the-art alternative.

Original: Contrastive predictive coding (Oord et al., 2018)
Revised: Contrastive predictive coding (Oord et al., 2018)

Original: based techniques like wav2vec 2.0 (Baevski et al., 2020), on the other hand, currently trust most of the leader-boards for speech-related tasks.
Revised: based techniques like wav2vec 2.0 (Baevski et al., 2020b), on the other hand, currently trust most of the leader-boards for speech-related tasks.

Original: Recently, Sadhu et al.
Revised: Recently, Sadhu et al.

Original: (2021) showed that combining a consistency loss and contrastive predictive coding improves the results of the wav2vec 2.
Revised: (2021) showed that combining a consistency loss and contrastive predictive coding improves the results of the wav2vec 2.

Original: architectures in noisy conditions.
Revised: architectures in noisy conditions.

Original: Following this idea, we propose to further validate our selection method with an extension of wav2vec 2.0 to multitask SSL to demonstrate its scaling capabilities. 
Revised: Following this idea, we propose to further validate our selection method with an extension of wav2vec 2.0 to multitask SSL to demonstrate its scaling capabilities. [Table]

Original: Hence, the training loss is extended to:
Revised: Hence, the training loss is extended to:

Original: [Equation]
Revised: [Equation]

Original:  We use the standard BASE wav2vec 2.0 ﬁrst described in (Baevski et al., 2020) as a SSL model and train it with the same Common Voice dataset.
Revised: [Table] [Equation] [Table] We use the standard BASE wav2vec 2.0 ﬁrst described in (Baevski et al., 2020b) as a SSL model and train it with the same Common Voice dataset.

Original: The pre-training pipeline is implemented within
Revised: The pre-training pipeline is implemented within

Original: SpeechBrain.
Revised: SpeechBrain.

Original: The trained BASE model has been compared to one obtained with the ofﬁcial Fairseq implementation from Baevski et al.
Revised: The trained BASE model has been compared to one obtained with the ofﬁcial Fairseq implementation from Baevski et al.

Original: (2020), and
Revised: (2020b), and

Original: results are strictly equivalent.
Revised: results are strictly equivalent.

Original: The entire recipe alongside with the large set of hyperparameters needed to properly train a wav2vec 2.0 model are released under our anonymous repository and will be made available with SpeechBrain afterwards.
Revised: The entire recipe alongside with the large set of hyperparameters needed to properly train a wav2vec 2.0 model are released under our anonymous repository and will be made available with SpeechBrain afterwards.

Original: To further investigate our methodology, we extend this analysis to emotion recognition (ER) with the

Original: IEMOCAP (Busso et al., 2008) dataset, as prosody has shown important in ER literature (Luengo et al., 2005).

Original: We follow the SUPERB benchmark conventions (Yang et al., 2021)
Revised: We follow the SUPERB benchmark conventions (Yang et al., 2021)

Original: both at the data and downstream architecture levels.
Revised: both at the data and downstream architecture levels.

Original: Hence, and conversely to the previous experiments, the ASR system[Table] solely optimises the CTC criterion over characters.
Revised: Hence, and conversely to the previous experiments, the ASR system solely optimises the CTC criterion over characters.

Original: For each of the three tasks ( i.e. ASR, SV, ER) we compare the standard BASE wav2vec 2.0 model with one trained following the sparsemax selection of multitask SSL.
Revised: For each of the three tasks ( i.e. ASR, SV, ER) we compare the standard BASE wav2vec 2.0 model with one trained following the sparsemax selection of multitask SSL.

Original: Sparsemax is chosen over softmax because it enforces the sparsity criterion and removes completely a few pseudo-labels from the training, which is one of the objectives of this work.
Revised: Sparsemax is chosen over softmax because it enforces the sparsity criterion and removes completely a few pseudo-labels from the training, which is one of the objectives of this work.

Revised: Another experiment is led with a “naive” pretext-task selection where a constant weight of 0 . 5 is used across all signal-based pretext-tasks.

Original: As for the other experiments, the exact weights of each pseudo-label are reported in Appendix A.9.
Revised: Like the other experiments, the exact weights of each pseudo-label are reported in Appendix A.11.

Original: Each wav2vec 2.0 model required 24 NVIDIA Tesla V100 GPUs to train for 150 epochs (hours).
Revised: Each wav2vec 2.0 model required 24 NVIDIA Tesla V100 GPUs to train for 150 epochs ( 40 hours).

Original: Finally, we also propose to compare frozen and unfrozen ( i.e. where the wav2vec 2. encoder is ﬁne-tuned with the downstream task) SSL models.
Revised: Finally, we also propose to compare frozen and unfrozen ( i.e. where the wav2vec 2.0 encoder is ﬁne-tuned with the downstream task) SSL models.

Original: It is clear from the results reported in Table 2 that our approach improves the performance over the standard wav2vec 2.0 framework for every considered downstream task.
Revised: It is clear from the results reported in Table 2 that our approach improves the performance over the standard wav2vec 2.0 framework for every considered downstream task.

Revised: While adding pretext tasks naively improves the ﬁnal performance, the difference in performance between the naive selection and the sparsemax weighting shows the beneﬁt of our method in getting the best downstream performance.

Revised: Unsurprisingly this difference is small (though statistically signiﬁcant in all but one case), as the wav2vec 2.0 BASE is already powerful and the additional workers are anyway useful.

Original: Here, it is worth noting that the difference in performance compared to the literature mostly comes from the training conditions.
Revised: Here, it is worth noting that the difference in performance compared to the literature mostly comes from the pre-training conditions.

Original: For instance, wav2vec 2.0 is commonly pre-trained with larger models onLibriSpeech to achieve lower WER on this dataset. We decided to avoid this comparison since including the downstream dataset in the pre-training phase would have introduced biases in the evaluation of our method.  F URTHER DISCUSSIONS Computational efﬁciency.
Revised: For instance, wav2vec 2.0 is commonly pre-trained with larger models on

Original: Efﬁciency is one of the key motivations of this work, and the gain in time observed with our approach is considerable. The K and L matrices used for the CI estimate are only computed for the downstream datasets. But since these datasets may get bigger and bigger, we can sample among the considered downstream classes to keep the computations tractable and quick. For instance, the CI testing of a considered pretext task ( i.e. pseudo-labels selection) takes less than half an hour on 20 CPUs whether it be for LibriSpeech or VoxCeleb.
Revised: LibriSpeech to achieve lower WER on this dataset.

Original: This is to be compared to 40 hours of GPU training ( i.e. 24 Nvidia Tesla V100 for wav2vec 2.0 BASE ) for a pre-training experiment. The exhaustive search for proper pseudo-label weighting is even more dramatically computationally and energy consuming, if we considered s values per pseudo-label in a grid search involving k pseudo-labels, we would need s k experiments. Pseudo-label interactions. A thorough analysis of the evolution of the CI estimate as a function of the weights, as given in Appendix A.4.3, provides interesting insights on which pseudo-labels seem to be interchangeable, complementary or even harmful with respect to a downstream task.
Revised: This comparison can be found in appendix A. as it could bias the evaluation of our method with pre-training and target datasets being the same.

Original: This suggests that the design of good pseudo-labels and sets of pseudo-labels is not a trivial task.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we introduce a method to quickly and simply combine pseudo-labels into a useful pretext task for multitask self-supervised learning settings.
Revised: In this work, we introduce a method to quickly and simply combine pseudo-labels into a useful pretext task for multitask self-supervised learning settings.

Original: Our approach allows for an optimal selection of pseudo-labels following a cheap optimisation process drastically decreasing the time and compute needed to design the best performing multitask SSL model.
Revised: Our approach allows for an optimal selection of pseudo-labels following a cheap optimisation process drastically decreasing the time and compute needed to design the best performing multitask SSL model.

Original: Our method is validated on three speech-related downstream tasks and outperforms common pseudo-label selection strategies when combined with simple and state-of-the-art SSL models.
Revised: Our method is validated on three speech-related downstream tasks and outperforms common pseudo-label selection strategies when combined with simple and state-of-the-art SSL models.

Original: This opens a range of possibilities for ﬁnding and selecting new pretext tasks in self-supervised learning for speech or other types of data.
Revised: This opens a range of possibilities for ﬁnding and selecting new pretext tasks in self-supervised learning for speech or other types of data.

Original: A A PPENDIX A. D IFFERENTIABILITY PROOF
Revised: A A PPENDIX A. D IFFERENTIABILITY PROOF

Original: We want to show that the utility estimate is differentiable with respect to the weighting parameters ( λ i ) i ∈ [0 ,k ] .
Revised: We want to show that the utility estimate is differentiable with respect to the weighting parameters ( λ i ) i ∈ [0 ,k ] .

Original: Since the ﬁnal estimate is a weighted mean of the in-class independent tests, the problem boils down to showing that within a downstream class c , HSIC c ( X, Z λ ) is differentiable.
Revised: Since the ﬁnal estimate is a weighted mean of the in-class independent tests, the problem boils down to showing that within a downstream class c , HSIC c ( X, Z λ ) is differentiable.

Original: Let us recall the deﬁnition of the considered quantities:
Revised: Let us recall the deﬁnition of the considered quantities:

Original: [Equation]
Revised: [Equation]

Original: where K c and H c are independent of λ and L c coefﬁcients are deﬁned as:
Revised: where K c and H c are independent of λ and L c coefﬁcients are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Therefore for p ∈ [0 , k ] :
Revised: Therefore for p ∈ [0 , k ] :

Original: [Equation]
Revised: [Equation]

Original: This allowed us to minimize the conditional-independence based utility estimator according to the weighting values. 
Revised: This allowed us to minimize the conditional-independence based utility estimator according to the weighting values. [Figure] Loudness [Figure]

Revised: approx.

Original: A. C ONSIDERED SIGNAL FEATURES AND DESCRIPTIONS
Revised: loudness [Figure] Alpha Ratio (Sundberg & Nordenberg, 2006) Ratio of spectrum intensity % 1000 Hz

Revised: Zero Crossing Rate

Revised: Zero crossing number per frame

Revised: RastaSpec L1Norm

Revised: L1 Norm of Rasta Spectrum (Hermansky et al., 1992) log HNR (Murphy & Akande, 2005) log of Harmonicity to Noise Ratio

Revised: [Figure]

Revised: A. C ONSIDERED SIGNAL FEATURES AND DESCRIPTIONS

Original: Table 3 contains the descriptions of the signal features used as pseudo-labels in this work.
Revised: Table 3 contains the descriptions of the signal features used as pseudo-labels in this work.

Revised: A. E FFECTS OF SAMPLING ON THE C ONDITIONAL I NDEPENDENCE ESTIMATION

Original: [Table]
Revised: Two limitations related to the size of the downstream dataset may be faced using our technique. First, very small downstream datasets could not be sufﬁcient for a good estimate of the conditional independence. Second, very large downstream datasets may render the CI estimation intractable. This section shows experimentally on VoxCeleb1 that our technique is robust to these two situations.

Revised: First, we show by taking small subsets of VoxCeleb1, that in case of downstream data scarcity, the

Revised: CI estimations obtained with our method are close to the ﬁnal estimations, and the ranking of the pretext tasks is not altered even when we take only 100 speakers among the 1 251 in VC.

Revised: Second, as one of the main motivations of this work is the reduction of the computation needed to get the best selection of pretext-tasks in self-supervised learning settings, we show that the CI estimation converges quickly with a growing number of speakers considered, and is thus resilient to sampling.

Revised: Considering one pretext task at a time, we consider subsets of VoxCeleb1 using a growing number of considered speakers ( total = 1 251 ).

Revised: For each of these considered numbers, we run 10 experiments with sampled speakers.

Revised: We get the CI estimation for every subset and plot the boxplot of the obtained values.

Revised: Results are shown in Fig.

Revised: We can see that using only 20 speakers showcases results that are already close to those with 1 000 speakers.

Revised: Furthermore, we plot the boxplots of CI values obtained using more than 200 speakers to show the separability between the considered features in

Revised: Fig.

Revised: While values for Voicing and Loudness are slightly overlapping, all the other pretext tasks are already separated and rankable using only 200 random speakers among the whole dataset.

Revised: [Figure]

Revised: C I v a l u e s

Revised: [Figure]

Revised: Z C R

Original: A. S
Revised: L o u d n e ss V o i c i n g R a s t a

Original: [Table]
Revised: [Figure]

Original: When initialized with random parameters W , and if one parameter is high enough compared to the other, leading with the Sparsemax function to a weighting value close to 1, we observed that the minimization process falls into local minima selecting only one pseudo-label with weight 1.
Revised: When initialized with random parameters W , and if one parameter is high enough compared to the other, leading with the Sparsemax function to a weighting value close to 1, we observed that the minimization process falls into local minima selecting only one pseudo-label with weight 1.

Original: To avoid this, we initialize all the free parameters W with the same unitary value to which we add some
Revised: To avoid this, we initialize all the free parameters W with the same unitary value to which we add some

Original: Gaussian noise.
Revised: Gaussian noise.

Original: Hence, W init =
Revised: Hence, W init =

Original: (1) + N (0 , (cid:15) ) with (cid:15)
Revised: (1) + N (0 , (cid:15) ) with (cid:15)

Original: = 0 .
Revised: = 0 .

Original: A. T RAINING AND ARCHITECTURES DETAILS
Revised: A. T RAINING AND ARCHITECTURES DETAILS

Original: All the considered audio ﬁles are sampled at 16kHz.
Revised: All the considered audio ﬁles are sampled at 16kHz.

Original: We feed the SSL models with 80 -band Mel spectrograms, with 25 ms windows and 10 ms stride.
Revised: We feed the SSL models with 80 -band Mel spectrograms, with 25 ms windows and 10 ms stride.

Original: To every Mel band corresponds a learned vector of size 256 obtained at the output of the SSL model.
Revised: To every Mel band corresponds a learned vector of size 256 obtained at the output of the SSL model.

Original: So if the input spectrogram is of size ( N , 80 ) with N the number of frames, the representation fed to the downstream pipeline is of size ( N , 256 ).
Revised: So if the input spectrogram is of size ( N , 80 ) with N the number of frames, the representation fed to the downstream pipeline is of size ( N , 256 ).

Original: All models including SSL and downstream ones are developed with SpeechBrain (Ravanelli et al., 2021).
Revised: All models including SSL and downstream ones are developed with SpeechBrain (Ravanelli et al., 2021).

Original: A.4. P RETRAINING OF THE SSL ENCODER.
Revised: A.5. P RETRAINING OF THE SSL ENCODER

Original: The encoder is a succession of 2D CNN layers, LSTM layers and a ﬁnal dense network.
Revised: The encoder is a succession of 2D CNN layers, LSTM layers and a ﬁnal dense network.

Original: This representation is then fed to one dense layer that predict the selected pretext task labels.
Revised: This representation is then fed to one dense layer that predict the selected pretext task labels.

Original: There aresuccessive CNN blocks containing each 2 CNN layers with kernel size (3 , 3) and 128 , 200 andchannels for each block respectively.
Revised: There aresuccessive CNN blocks containing each 2 CNN layers with kernel size (3 , 3) and 128 , 200 andchannels for each block respectively.

Original: No time pooling is performed in order to preserve the input sequence length.
Revised: No time pooling is performed in order to preserve the input sequence length.

Original: 5 bidirectional LSTM layers of size 256 are then stacked.
Revised: 5 bidirectional LSTM layers of size 256 are then stacked.

Original: Finally, a MLP with one hidden layer with 256 neurons.
Revised: Finally, a MLP with one hidden layer with 256 neurons.

Original: The LeakyReLU activation is used across all the layers except for the LSTM.
Revised: The LeakyReLU activation is used across all the layers except for the LSTM.

Original: We use a dropout rate of 0 . 15 during the training.
Revised: We use a dropout rate of 0 . 15 during the training.

Original: The AdaDelta optimizer is used to update the weights with an initial learning rate of 1 . 0 , ρ = 0 .
Revised: The AdaDelta optimizer is used to update the weights with an initial learning rate of 1 . 0 , ρ = 0 .

Original: 8 and (cid:15) = 10 − 8 .
Revised: 8 and (cid:15) = 10 − 8 .

Original: For every experiment, the SSL model is trained for 10 epochs ( leading to the convergence of the validation loss).
Revised: For every experiment, the SSL model is trained for 10 epochs ( leading to the convergence of the validation loss).

Original: A.4. D OWNSTREAM TRAININGS : FIRST EXPERIMENTS
Revised: A.5. D OWNSTREAM TRAININGS : FIRST EXPERIMENTS

Original: Speaker recognition details.
Revised: Speaker recognition details.

Original: VoxCeleb1 (Nagrani et al., 2017) is used for the speaker recognition task.
Revised: VoxCeleb1 (Nagrani et al., 2017) is used for the speaker recognition task.

Original: The training set contains 148 , 642 utterances from 1 , 251 different speakers.
Revised: The training set contains 148 , 642 utterances from 1 , 251 different speakers.

Original: To compute the conditional independence estimates while limiting the computational load, we restricted ourselves to the utterances of 50 different speakers (the detailed list is given in the released repository).
Revised: To compute the conditional independence estimates while limiting the computational load, we restricted ourselves to the utterances of 50 different speakers (the detailed list is given in the released repository).

Original: A standard xvector model (Snyder et al., 2018) is trained following the available VoxCeleb SpeechBrain recipe.
Revised: A standard xvector model (Snyder et al., 2018) is trained following the available VoxCeleb Speech- Brain recipe.

Original: The extracted speaker embeddings are tested on the enrol and test splits using PLDA (Ioffe, 2006) as a similarity metric.
Revised: The extracted speaker embeddings are tested on the enrol and test splits using PLDA (Ioffe, 2006) as a similarity metric.

Original: Performance is reported in terms of equal error rate (EER).
Revised: Performance is reported in terms of equal error rate (EER).

Original: While architecture details are given in appendix A.4, it is worth noticing that the whole pipeline is fully integrated to Speechbrain and can thus easily be extended.
Revised: While architecture details are given in appendix A.5, it is worth noticing that the whole pipeline is fully integrated to Speechbrain and can thus easily be extended.

Original: We train an embedding model (XVector) until the validation loss converges, on top of the self supervised representations using 5 successive layers of time-delay neural networks (TDNN) (Peddinti et al., 2015).
Revised: We train an embedding model (XVector) until the validation loss converges, on top of the self supervised representations using 5 successive layers of time-delay neural networks (TDNN) (Peddinti et al., 2015).

Original: The number of channels is (512 , 512 , 512 , 512 , 1500) , with kernel sizes of (5 , 3 , 3 , 1 , 1) and dilations of (1 , 2 , 3 , 1 , 1) .
Revised: The number of channels is (512 , 512 , 512 , 512 , 1500) , with kernel sizes of (5 , 3 , 3 , 1 , 1) and dilations of (1 , 2 , 3 , 1 , 1) .

Original: The architecture is inspired by successful works on embeddings for speaker recognition (Snyder et al., 2015).
Revised: The architecture is inspired by successful works on embeddings for speaker recognition (Snyder et al., 2015).

Original: The learned embeddings are therefore used on a list of pairs of samples to predict whether they are from the same speaker or not.
Revised: The learned embeddings are therefore used on a list of pairs of samples to predict whether they are from the same speaker or not.

Original: The details of the recipe can be found in the given GitHub repository.
Revised: The details of the recipe can be found in the given GitHub repository.

Original: We train every embedding model on 10 epochs with an Adam Optimizer starting with a learning rate of 0.001 decaying linearly to 0.0001.
Revised: We train every embedding model on 10 epochs with an Adam Optimizer starting with a learning rate of 0.001 decaying linearly to 0.0001.

Original: Speech recognition details.
Revised: Speech recognition details.

Original: ASR is conducted with the 100 -hour clean subset of the LibriSpeech dataset (Panayotov et al., 2015) to simulate the low-resource scenario commonly encountered with
Revised: ASR is conducted with the 100 -hour clean subset of the LibriSpeech dataset (Panayotov et al., 2015) to simulate the low-resource scenario commonly encountered with

Original: SSL settings.
Revised: SSL settings.

Original: CI estimations are obtained with word-level alignments from the Montreal Forced
Revised: CI estimations are obtained with word-level alignments from the Montreal Forced

Original: Aligner (McAuliffe et al., 2017).
Revised: Aligner (McAuliffe et al., 2017).

Original: The ASR pipeline follows the LibriSpeech recipe of SpeechBrain (Ravanelli et al., 2021) and therefore contains a CRDNN encoder ( i.e. CNN, RNN, DNN) trained jointly with CTC (Graves, 2012) and attention (L¨uscher et al., 2019)
Revised: The ASR pipeline follows the LibriSpeech recipe of SpeechBrain (Ravanelli et al., 2021) and therefore contains a CRDNN encoder ( i.e. CNN, RNN, DNN) trained jointly with CTC (Graves, 2012) and attention (L ¨ uscher et al., 2019)

Original: (details in appendix A.4).
Revised: (details in appendix A.5).

Original: The decoding process is based on beam-search with and without shallow fusion with a pretrained recurrent language model that is publicly available and obtained from SpeechBrain.
Revised: The decoding process is based on beam-search with and without shallow fusion with a pretrained recurrent language model that is publicly available and obtained from SpeechBrain.

Original: 2 Performance is expressed in word error rate (WER).
Revised: 2 Performance is expressed in word error rate (WER).

Original: The CRDNN starts with three CNN blocks composed each with 2 2D CNN layers, layernormalisation and (2 , 2) maxpooling along the frequency dimension.
Revised: The CRDNN starts with three CNN blocks composed each with 2 2D CNN layers, layernormalisation and (2 , 2) maxpooling along the frequency dimension.

Original: The ﬁlter dimensions for each block are 64 , 100 , 100 .
Revised: The ﬁlter dimensions for each block are 64 , 100 , 100 .

Original: Then, maxpooling of 4 is applied on the time dimension to reduce the sequence length before being fed to the RNN.
Revised: Then, maxpooling of 4 is applied on the time dimension to reduce the sequence length before being fed to the RNN.

Original: The latter is made of 5 bidirectional LSTM layers of 1 , 024 neurons.
Revised: The latter is made of 5 bidirectional LSTM layers of 1 , 024 neurons.

Original: Finally two dense layers are connected (with batch-normalisation in between).
Revised: Finally two dense layers are connected (with batch-normalisation in between).

Original: The
Revised: The

Original: LeakyReLU activation function is used across all the layers except for the LSTM.
Revised: LeakyReLU activation function is used across all the layers except for the LSTM.

Original: A dropout rate of 0.15 is employed with the encoder.
Revised: A dropout rate of 0.15 is employed with the encoder.

Original: The CTC decoder is a simple dense linear layer of size equal to the vocabulary.
Revised: The CTC decoder is a simple dense linear layer of size equal to the vocabulary.

Original: The vocabulary is obtained with byte pair encoding or sub-words units (BPE) and is of size 1 , 000 .
Revised: The vocabulary is obtained with byte pair encoding or sub-words units (BPE) and is of size 1 , 000 .

Original: The attentional decoder is a one-layered location-aware GRU ( 1 , 024 neurons).
Revised: The attentional decoder is a one-layered location-aware GRU ( 1 , 024 neurons).

Original: Then, a beam search of depth 60 is applied to obtain the output transcripts.
Revised: Then, a beam search of depth 60 is applied to obtain the output transcripts.

Original: The model is trained forepochs.
Revised: The model is trained forepochs.

Original: The learning rate ( 1 . 0 ) is multiplied with a factor of 0 .
Revised: The learning rate ( 1 . 0 ) is multiplied with a factor of 0 .

Original: 8 every time the validation loss is not decreasing to ensure an optimal convergence of all the models.
Revised: 8 every time the validation loss is not decreasing to ensure an optimal convergence of all the models.

Original: A.4. SUPERB SETTINGS
Revised: A.5. SUPERB SETTINGS

Original: SUPERB (Yang et al., 2021) is a recent benchmark for self-supervised representations of speech data.
Revised: SUPERB (Yang et al., 2021) is a recent benchmark for self-supervised representations of speech data.

Original: We use this benchmark for our experiments in combining wav2vec with our selected pretext tasks.
Revised: We use this benchmark for our experiments in combining wav2vec with our selected pretext tasks.

Original: We detail here the downstream models as detailed in the benchmark paper :
Revised: We detail here the downstream models as detailed in the benchmark paper :

Original: Emotion Recognition.
Revised: Emotion Recognition.

Original: IEMOCAP (Busso et al., 2008) is used for the Emotion Recognition (ER) task.
Revised: IEMOCAP (Busso et al., 2008) is used for the Emotion Recognition (ER) task.

Original: 4 classes are considered (neutral, happy, sad, angry), and only the audio data is used.
Revised: 4 classes are considered (neutral, happy, sad, angry), and only the audio data is used.

Original: The learned representations are mean-pooled then fed to a ﬁnal linear classiﬁer to compute a crossentropy loss.
Revised: The learned representations are mean-pooled then fed to a ﬁnal linear classiﬁer to compute a crossentropy loss.

Original: We cross-validate on ﬁve folds of the standard splits.
Revised: We cross-validate on ﬁve folds of the standard splits.

Original: The result shown is the average of the ﬁve attempts.
Revised: The result shown is the average of the ﬁve attempts.

Original: The evaluation metric is accuracy (ACC).
Revised: The evaluation metric is accuracy (ACC).

Original: Automatic Speech Recognition For ASR, the decoder is a vanilla 2-layer 1024-unit BLSTM fed with our self-supervised representations and optimized by CTC loss on characters.
Revised: Automatic Speech Recognition For ASR, the decoder is a vanilla 2-layer 1024-unit BLSTM fed with our self-supervised representations and optimized by CTC loss on characters.

Original: We use the same language model for decoding as in the ﬁrst experiments.
Revised: We use the same language model for decoding as in the ﬁrst experiments.

Original: LibriSpeech Clean-100 only is used for downstream training.
Revised: LibriSpeech Clean-100 only is used for downstream training.

Original: Speaker Recognition The model and the dataset splits used in the ﬁrst experiment correspond to the SUPERB ones, so we kept the same settings.
Revised: Speaker Recognition The model and the dataset splits used in the ﬁrst experiment correspond to the SUPERB ones, so we kept the same settings.

Original: The results are therefore comparable.
Revised: The results are therefore comparable.

Original: A. I NTUITION AROUND THE USE OF C ONDITIONAL I NDEPENDENCE
Revised: A. I NTUITION AROUND THE USE OF C ONDITIONAL I NDEPENDENCE

Original: To get an intuitive understanding of the motivations of this choice, let us consider the example of image classiﬁcation as the downstream task, and image colourization as the pretext task.
Revised: To get an intuitive understanding of the motivations of this choice, let us consider the example of image classiﬁcation as the downstream task, and image colourization as the pretext task.

Original: In this case, this pretext task would be suited to the downstream one if the ﬁnal classiﬁcation label can help implying the colours.
Revised: In this case, this pretext task would be suited to the downstream one if the ﬁnal classiﬁcation label can help implying the colours.

Original: For instance, if there are only two classes ”Blue skies” and ”Yellow deserts”, then colourisation is an interesting pretext task, as knowing the ﬁnal label helps a lot for the pretext task, independently of the image.
Revised: For instance, if there are only two classes ”Blue skies” and ”Yellow deserts”, then colourisation is an interesting pretext task, as knowing the ﬁnal label helps a lot for the pretext task, independently of the image.

Original: However, if all the classes share the same colour palette, colourization may not be an interesting task.
Revised: However, if all the classes share the same colour palette, colourization may not be an interesting task.

Original: (In this simple example, we are ignoring the edge detection aspect of colourization, and only focusing on the colour choice part.
Revised: (In this simple example, we are ignoring the edge detection aspect of colourization, and only focusing on the colour choice part.

Original: Obviously the former aspect plays a big part on why the colourization pretext task has been successful.)
Revised: Obviously the former aspect plays a big part on why the colourization pretext task has been successful.)

Original: Concerning our estimation method, as the pseudo-labels considered in this work are data features, they are indeed functions of the original data samples.
Revised: Concerning our estimation method, as the pseudo-labels considered in this work are data features, they are indeed functions of the original data samples.

Original: This ensures that the data samples are not independent of the pseudo-labels.
Revised: This ensures that the data samples are not independent of the pseudo-labels.

Original: The idea behind the estimator of conditional independence is that it will test whether this remains true when the considered points share the same downstream class.
Revised: The idea behind the estimator of conditional independence is that it will test whether this remains true when the considered points share the same downstream class.

Original: A. K ERNELS U SED FOR THE SIMILARITY MATRICES
Revised: A. K ERNELS U SED FOR THE SIMILARITY MATRICES

Original: The computation of the similarity matrices used in our kernel-based independence test, requires ﬁxed-size embeddings for the data speech samples.
Revised: The computation of the similarity matrices used in our kernel-based independence test, requires ﬁxed-size embeddings for the data speech samples.

Original: These embeddings allow the use of classic kernels on top.
Revised: These embeddings allow the use of classic kernels on top.

Original: However, in the case of sequential data, as it is the case with audio/speech signals, one may want to avoid the additional burden of learning ﬁxed-size embeddings (for possibly variablelength audio sequences).
Revised: However, in the case of sequential data, as it is the case with audio/speech signals, one may want to avoid the additional burden of learning ﬁxed-size embeddings (for possibly variablelength audio sequences).

Original: One possible solution to this, which we conveniently exploited in our application to speech data (see Section 5) is the Gaussian Downsampling method (Holzenberger et al., 2018) detailed thereafter.
Revised: One possible solution to this, which we conveniently exploited in our application to speech data (see Section 5) is the Gaussian Downsampling method (Holzenberger et al., 2018) detailed thereafter.

Original: In this instance, after the Mel spectrogram extraction, a speech sample is a sequence of varying length input feature vectors.
Revised: In this instance, after the Mel spectrogram extraction, a speech sample is a sequence of varying length input feature vectors.

Original: Therefore, to obtain ﬁxed size embeddings aggregating the input frame-wise Mel spectrum vectors into a ﬁxed number N of input vectors, N being a ﬁxed hyper-parameter, we ﬁrst divide the sequence into N equal length segments.
Revised: Therefore, to obtain ﬁxed size embeddings aggregating the input frame-wise Mel spectrum vectors into a ﬁxed number N of input vectors, N being a ﬁxed hyper-parameter, we ﬁrst divide the sequence into N equal length segments.

Original: Then, in each segment, a Gaussian average of the input spectra is computed around the center of the considered segment with the standard deviation σ gd being another hyper-parameter.
Revised: Then, in each segment, a Gaussian average of the input spectra is computed around the center of the considered segment with the standard deviation σ gd being another hyper-parameter.

Original: Denoting by D the dimension of the input frame-wise Mel spectrum vectors, this leads, for any speech excerpt, to a
Revised: Denoting by D the dimension of the input frame-wise Mel spectrum vectors, this leads, for any speech excerpt, to a

Original: N × D tensor, without any training procedure.
Revised: N × D tensor, without any training procedure.

Original: As in the work presenting the gaussian downsampling method (Holzenberger et al., 2018), we set N = 20 and σ gd = 0 .
Revised: As in the work presenting the gaussian downsampling method (Holzenberger et al., 2018), we set N = 20 and σ gd = 0 .

Original: For the RBF kernel on the pseudo-labels mean value per ﬁle, we ﬁxed he RBF kernel width to σ = 0 .
Revised: For the RBF kernel on the pseudo-labels mean value per ﬁle, we ﬁxed he RBF kernel width to σ = 0 .

Revised: We will know motivate the gaussian downsampling choice.

Original: A. L INKS WITH F EATURE S ELECTION
Revised: At a high-level, we were looking for a method giving ﬁxed size speech embeddings that would not rely on any learning. As we were ﬁrst working with ASR and Speaker Id, we wanted a representation that captures phonetic and speaker content. The GD has been, quite successfully, applied for unsupervised word discovery (Holzenberger et al., 2018), thus capturing this phonetic aspect. In unpublished experiments when working on unsupervised work discovery, we also found out that GD holds speaker-related information, this made GD a good candidate for our unlearned embedding method.

Revised: Moreover, after the ﬁrst results, another downsampling method was tested in the beginning of this work : SVCCA (Raghu et al., 2017).

Revised: We calculated the CI estimates using GD and SVCCA on the considered pretext tasks for

Revised: LibriSpeech and Voxceleb1.

Revised: It resulted in a relative difference of 3% between the averaged HSIC.

Revised: The robustness to this downsampling method change comforted us into using GD.

Revised: A. I MPACT OF THE PRETRAINING DATASET ON SPEAKER RECOGNITION

Revised: We developed in section 5.2 the reasons behind our choice of CommonVoice as a pre-training dataset.

Revised: In this section, we study the impact of this choice on the results.

Revised: Speciﬁcally, it is common in the speech SSL literature to train on LibriSpeech 960 before ﬁne-tuning on LibriSpeech100.

Revised: As explained before, we believe that this introduces a bias due to the closeness of pretraining and ﬁnetuning data.

Revised: To verify this, we train our best multitask BASE wav2vec 2.0 architecture with the best performing pretext tasks and their weights on LibriSpeech 960.

Revised: The model follows the exact same training procedure as for Table 2.

Revised: We ﬁne-tune the models on LibriSpeech 100 exactly as it has been done with the other models.

Revised: Table 4 shows the results.

Revised: Two observations deserve to be noted.

Revised: First, in this case also, adding a selected set of pretext tasks improves the ﬁnal downstream performance in the frozen and ﬁnetuned cases.

Revised: Second, as expected, the results obtained after training on Lib-

Revised: [Table]

Revised: rispeech960 are better than those with CommonVoice, reaching the lowest 6 .

Revised: 01% of WER with the ﬁne-tuned version compared to 9 .

Revised: 18% of WER in table

Revised: A. L INKS WITH F EATURE S ELECTION

Original: We also studied the link between classic feature selection and pretext task selection through two experiments.
Revised: We also studied the link between classic feature selection and pretext task selection through two experiments.

Original: The ﬁrst one was made to check how hard it was to estimate the utility of a pseudo-label, we computed the mutual information between the pseudo-labels and the downstream labels, and checked how much it would correlate with downstream performance.
Revised: The ﬁrst one was made to check how hard it was to estimate the utility of a pseudo-label, we computed the mutual information between the pseudo-labels and the downstream labels, and checked how much it would correlate with downstream performance.

Original: It led to very low correlation values, with even changing signs between VoxCeleb and LibriSpeech.
Revised: It led to very low correlation values, with even changing signs between VoxCeleb and LibriSpeech.

Original: This seems to indicate that
Revised: This seems to indicate that

Original: Mutual Information is not related directly to self-supervision utility.
Revised: Mutual Information is not related directly to self-supervision utility.

Original: The maximum relevance minimum redundancy (MRMR) technique (Peng et al., 2005) used as a baseline in this work relies on the Conditional Independence based estimator.
Revised: The maximum relevance minimum redundancy (MRMR) technique (Peng et al., 2005) used as a baseline in this work relies on the Conditional Independence based estimator.

Original: It is a close to a naive selection of the best pretext tasks according to the CI based criterion, but it furthermore penalizes the mutual information between the selected pretext tasks.
Revised: It is a close to a naive selection of the best pretext tasks according to the CI based criterion, but it furthermore penalizes the mutual information between the selected pretext tasks.

Original: More precisely, we select the group of pseudo-labels ( Z ) i ∈
Revised: More precisely, we select the group of pseudo-labels ( Z ) i ∈

Original: [0 , p ] maximizing :
Revised: [0 , p ] maximizing :

Original: [Equation]
Revised: [Equation]

Original: Recursive Feature Elimination (RFE) (Guyon et al., 2002) relies on a classiﬁer that provides information concerning the importance of a given feature in the decision.
Revised: Recursive Feature Elimination (RFE) (Guyon et al., 2002) relies on a classiﬁer that provides information concerning the importance of a given feature in the decision.

Original: This classiﬁer is ﬁrst trained with the whole set of pseudo-labels as features, and the least important feature is eliminated.
Revised: This classiﬁer is ﬁrst trained with the whole set of pseudo-labels as features, and the least important feature is eliminated.

Original: The process is repeated until only 4 pseudo-labels are kept.
Revised: The process is repeated until only 4 pseudo-labels are kept.

Original: We use the scikit-learn implementation with the C-Support Vector Classiﬁcation as the the classiﬁer providing the feature importance values.
Revised: We use the scikit-learn implementation with the C-Support Vector Classiﬁcation as the the classiﬁer providing the feature importance values.

Original: We use the default scikit-learn hyperparameters.
Revised: We use the default scikit-learn hyperparameters.

Original: These two baselines perform worse than the proposed techniques.
Revised: These two baselines perform worse than the proposed techniques.

Original: This suggests that despite the apparent similarity, feature selection and self-supervision pretext task design do not necessarily involve the same mechanisms.
Revised: This suggests that despite the apparent similarity, feature selection and self-supervision pretext task design do not necessarily involve the same mechanisms.

Original: A. P SEUDO - LABELS ’ INTERACTIONS .
Revised: A. P SEUDO - LABELS ’ INTERACTIONS .

Original: To understand the interactions between pseudo-labels, studying the evolution of the CI estimate as a function of the weights shows which pseudo-labels seem interchangeable, which ones are complementary and which ones seem only harmful to the considered downstream task.
Revised: To understand the interactions between pseudo-labels, studying the evolution of the CI estimate as a function of the weights shows which pseudo-labels seem interchangeable, which ones are complementary and which ones seem only harmful to the considered downstream task.

Original: Figure 2 shows the
Revised: Figure 4 shows the

Original: CI estimates for weighted combinations of groups of three pseudo-labels.
Revised: CI estimates for weighted combinations of groups of three pseudo-labels.

Original: As the weights sum up to one, two pretext tasks’ values are shown on the x and y axes, while the value of the remaining one, whose name is in the title, is equal to 1 − x − y .
Revised: As the weights sum up to one, two pretext tasks’ values are shown on the x and y axes, while the value of the remaining one, whose name is in the title, is equal to 1 − x − y .

Original: For instance, at the origin point (0 , 0) , only 
Revised: For instance, at the origin point (0 , 0) , only the third pseudo-label is selected with a weight equal to one, while its weight is equal to zero on the hypotenuse of the right triangle. Figure 4 illustrates that the relationship leading to a lower CI-based utility estimator is not always straightforward. For instance, if we consider the second plot on the second row ( i.e. α -ratio, F0, logHNR ), we can see that selecting only one element is always worse

Original: [Figure]
Revised: [Figure]

Original: Selection
Revised: Selection

Original: [Table]
Revised: [Table]

Original: the third pseudo-label is selected with a weight equal to one, while its weight is equal to zero on the hypotenuse of the right triangle. Figure 2 illustrates that the relationship leading to a lower CI-based utility estimator is not always straightforward. For instance, if we consider the second plot on the second row ( i
Revised: than selecting a weighted concatenation

Original: [Table]
Revised: [Table]

Original: e. α -ratio, F0, logHNR ), we can see that selecting only one element is always worse than selecting a weighted concatenation, because the areas around the origin and the points (1 , 0) and (0 , 1) are brighter than the central area.
Revised: because the areas around the origin and the points (1 , 0) and (0 , 1) are brighter than the central area.

Original: A. F ULL WEIGHTS FOR THE CONSIDERED EXPERIMENTS
Revised: A. F ULL WEIGHTS FOR THE CONSIDERED EXPERIMENTS

Original: Table 4 shows the weights computed using the proposed selection and weighting process for every considered downstream task.
Revised: Table 5 shows the weights computed using the proposed selection and weighting process for every considered downstream task.

Original: It also shows the pretext tasks selected by the baseline methods.
Revised: It also shows the pretext tasks selected by the baseline methods.