Original: On Combining Global and Localized Self-Supervised Models of Speech
Revised: On Combining Global and Localized Self-Supervised Models of Speech

Original: Anonymous Authors

Original: Abstract
Revised: Abstract

Original: Self supervised learning involves learning generalpurpose representations that can be useful in a variety of downstream tasks.
Revised: Self supervised learning involves learning generalpurpose representations that can be useful in a variety of downstream tasks.

Original: In this work, we study the application of speech-embeddings derived from popular self-supervised learning frameworks such as wav2vec-2.0 and HuBERT over four different speech-classification tasks such as sentiment classification, command detection, emotion classification and depression detection.
Revised: In this work, we study the application of speech-embeddings derived from popular self-supervised learning frameworks such as wav2vec-2.0 and HuBERT over four different speech-classification tasks such as sentiment classification, command detection, emotion classification and depression detection.

Original: We distinguish between and discuss self-supervised training tasks that induce localized and global features of speech based on their temporal granularity: noting that self-supervised representation learning frameworks based on the masked languagemodeling objective – such as wav2vec-2.0 and
Revised: We distinguish between and discuss self-supervised training tasks that induce localized and global features of speech based on their temporal granularity: noting that self-supervised representation learning frameworks based on the masked languagemodeling objective – such as wav2vec-2.0 and

Original: HuBERT – induce localized embeddings, we define a self-supervised learning framework based on SimSiam for learning global features of speech.
Revised: HuBERT – induce localized embeddings, we define a self-supervised learning framework based on SimSiam for learning global features of speech.

Original: Through our evaluations, we find that these global representations are better suited for tasks such as depression detection and emotion classification while the localized embeddings of speech can be very useful in tasks such as speech-command detection; we also find that our proposed model outperforms TRILL – a popular model for learning global representations.
Revised: Through our evaluations, we find that these global representations are better suited for tasks such as depression detection and emotion classification while the localized embeddings of speech can be very useful in tasks such as speech-command detection; we also find that our proposed model outperforms TRILL – a popular model for learning global representations.

Original: Finally, we also propose and confirm empirically that combining the global and localized representations of speech helps obtain better performance across a range of downstream tasks than each of the individual embedding methods.
Revised: Finally, we also propose and confirm empirically that combining the global and localized representations of speech helps obtain better performance across a range of downstream tasks than each of the individual embedding methods.

Revised: First Workshop of Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022 , Baltimore, Maryland, USA, PMLR 162, 2022.

Revised: Copyright 2022 by

Original: 1. Introduction
Revised: 1. Introduction

Original: Self-supervised learning frameworks aim to learn general purpose representations from large amounts of data through strategically defined training tasks that do not require perinstance manual annotations.
Revised: Self-supervised learning frameworks aim to learn general purpose representations from large amounts of data through strategically defined training tasks that do not require perinstance manual annotations.

Original: Popular self-supervised learning frameworks include: contrastive learning frameworks (Chen et al.
Revised: Popular self-supervised learning frameworks include: contrastive learning frameworks (Chen et al.

Original: , 2020b;c; He et al., 2020; Henaff, 2020), clustering frameworks (Asano et al., 2019; Caron et al., 2020) and non-contrastive learning frameworks (Grill et al., 2020;
Revised: , 2020b;c; He et al., 2020; Henaff, 2020), clustering frameworks (Asano et al., 2019; Caron et al., 2020) and non-contrastive learning frameworks (Grill et al., 2020;

Original: Chen & He, 2021).
Revised: Chen & He, 2021).

Original: In this work we will explore the use of simple siamese representation learning (SimSiam)
Revised: In this work we will explore the use of simple siamese representation learning (SimSiam)

Original: (Chen &
Revised: (Chen &

Original: He, 2021), a non-contrastive representation learning framework, for speech.
Revised: He, 2021), a non-contrastive representation learning framework, for speech.

Original: As part of the self-supervised learning setup, models are trained on cleverly defined pretext tasks to obtain general purpose representations which can then be used for the downstream tasks.
Revised: As part of the self-supervised learning setup, models are trained on cleverly defined pretext tasks to obtain general purpose representations which can then be used for the downstream tasks.

Original: In the context of learning self-supervised representations of speech, we can categorize the pretext tasks as:
Revised: In the context of learning self-supervised representations of speech, we can categorize the pretext tasks as:

Original: • Localized pretext tasks focus on learning segmentlevel representations of speech conditional on remaining contextual segments of the speech.
Revised: • Localized pretext tasks focus on learning segmentlevel representations of speech conditional on remaining contextual segments of the speech.

Original: As an example, self-supervised learning frameworks over raw speech waves that use masked prediction as the pretext task (Baevski et al., 2020; Hsu et al., 2021; Shor et al., 2020) are said to induce localized representations of speech as the task involves learning to represent and identify segments of speech conditional on the remaining segments; these localized representations of speech can be used to obtain state-of-the-art performance in automatic speech recognition (ASR)
Revised: As an example, self-supervised learning frameworks over raw speech waves that use masked prediction as the pretext task (Baevski et al., 2020; Hsu et al., 2021; Shor et al., 2020) are said to induce localized representations of speech as the task involves learning to represent and identify segments of speech conditional on the remaining segments; these localized representations of speech can be used to obtain state-of-the-art performance in automatic speech recognition (ASR)

Original: (Van den Oord et al., 2018; Baevski et al., 2020; Hsu et al., 2021).
Revised: (Van den Oord et al., 2018; Baevski et al., 2020; Hsu et al., 2021).

Original: In summary, examples of localized pretext tasks for speech include: predicting the content of the unseen regions (Chung & Glass, 2020; Liu et al., 2020), language model-style pre-training (Chung et al., 2019), predicting discrete targets of masked regions (Baevski et al., 2019; 2020; Hsu et al., 2021), combining 
Revised: In summary, examples of localized pretext tasks for speech include: predicting the content of the unseen regions (Chung & Glass, 2020; Liu et al., 2020), language model-style pre-training (Chung et al., 2019), predicting discrete targets of masked regions (Baevski et al., 2019; 2020; Hsu et al., 2021), combining self-

Original: selfsupervised learning with adversarial training (Baevski et al., 2021).
Revised: supervised learning with adversarial training (Baevski et al., 2021).

Original: • Global pretext tasks focus on learning global representations of entire speech wave.
Revised: • Global pretext tasks focus on learning global representations of entire speech wave.

Original: For example, (Saeed et al., 2021) proposes to contrastively learn representations of mel-spectrograms derived from speech segments – on the other hand, wav2vec-2.0 contrastively learns representations of speech segments conditional on remaining segments of the raw speech.
Revised: For example, (Saeed et al., 2021) proposes to contrastively learn representations of mel-spectrograms derived from speech segments – on the other hand, wav2vec-2.0 contrastively learns representations of speech segments conditional on remaining segments of the raw speech.

Original: In a similar vein, (Niizumi et al., 2021) proposes a self-supervised learning strategy based on BYOL (Grill et al., 2020) for learning representations of spectrograms derived from speech segments without requiring the use of negative samples (i.e. without contrastive learning).
Revised: In a similar vein, (Niizumi et al., 2021) proposes a self-supervised learning strategy based on BYOL (Grill et al., 2020) for learning representations of spectrograms derived from speech segments without requiring the use of negative samples (i.e. without contrastive learning).

Original: Some other examples of self-supervised learning frameworks based on global pretext tasks (Chen et al., 2021; Shor et al., 2020; Jiang et al., 2020; Shor et al., 2021) are usually 1) applied to mel-spectrograms; and 2) use contrastive learning frameworks which rely on negative samples, thus requiring large batch sizes to train the models.
Revised: Some other examples of self-supervised learning frameworks based on global pretext tasks (Chen et al., 2021; Shor et al., 2020; Jiang et al., 2020; Shor et al., 2021) are usually 1) applied to mel-spectrograms; and 2) use contrastive learning frameworks which rely on negative samples, thus requiring large batch sizes to train the models.

Original: Intuitively, localized features of speech effectively represent fine details of speech segments conditional on remaining speech segments – the state-of-the-art performance in ASR obtained by the use of localized features support this intuition.
Revised: Intuitively, localized features of speech effectively represent fine details of speech segments conditional on remaining speech segments – the state-of-the-art performance in ASR obtained by the use of localized features support this intuition.

Original: On the other hand, global features of speech represent more temporally stable elements of speech and as we show in our results, these global representations of speech are better suited than localized representations to downstream tasks such as emotion classification and depression detection.
Revised: On the other hand, global features of speech represent more temporally stable elements of speech and as we show in our results, these global representations of speech are better suited than localized representations to downstream tasks such as emotion classification and depression detection.

Original: In fact, TRILL(Shor et al., 2020) was proposed to learn such non-semantic representations of speech.
Revised: In fact, TRILL(Shor et al., 2020) was proposed to learn such non-semantic representations of speech.

Original: Accordingly, the localized representations of speech depend on the speech duration and change quickly throughout a speech sample, whereas global representations do not.
Revised: Accordingly, the localized representations of speech depend on the speech duration and change quickly throughout a speech sample, whereas global representations do not.

Original: In general, a global representation of speech can be obtained from the localized features of speech – such as those obtained from
Revised: In general, a global representation of speech can be obtained from the localized features of speech – such as those obtained from

Original: BERT-style training – by averaging across the constituent speech segments.
Revised: BERT-style training – by averaging across the constituent speech segments.

Original: We note that we can draw a parallel to text-representation learning: for example, BERT training over text corpora learns context-sensitive embeddings of words whereas some downstream tasks require sentence embeddings; see (Reimers & Gurevych, 2019) for a novel fine-tuning method over BERT for deriving sentence embeddings from pre-trained BERT.
Revised: We note that we can draw a parallel to text-representation learning: for example, BERT training over text corpora learns context-sensitive embeddings of words whereas some downstream tasks require sentence embeddings; see (Reimers & Gurevych, 2019) for a novel fine-tuning method over BERT for deriving sentence embeddings from pre-trained BERT.

Original: In this paper, we will propose SimSiam-speech, a simple siamese self-supervised learning framework for speech to learn global features of speech.
Revised: In this paper, we will propose SimSiam-speech, a simple siamese self-supervised learning framework for speech to learn global features of speech.

Original: We show that these global features achieve better performance than localized features on some downstream tasks.
Revised: We show that these global features achieve better performance than localized features on some downstream tasks.

Original: We will also show that combining these global features with localized features will achieve improved performance on various downstream tasks.
Revised: We will also show that combining these global features with localized features will achieve improved performance on various downstream tasks.

Original: The main contributions of this work are as follows:
Revised: The main contributions of this work are as follows:

Original: • We introduce a SimSiam-based framework for speech (SimSiam-Speech): Most previous self-supervised learning work on speech has been based on frameworks such as masked prediction similar to BERT (Kenton & Toutanova, 2019), simple contrastive learning (Chen et al., 2020b) and bootstrap your own latent (Grill et al., 2020).
Revised: • We introduce a SimSiam-based framework for speech (SimSiam-Speech): Most previous self-supervised learning work on speech has been based on frameworks such as masked prediction similar to BERT (Kenton & Toutanova, 2019), simple contrastive learning (Chen et al., 2020b) and bootstrap your own latent (Grill et al., 2020).

Original: In this work, we have extended the SimSiam framework for speech representation learning.
Revised: In this work, we have extended the SimSiam framework for speech representation learning.

Original: • Building on the intuition that global and localized features of speech have different temporal granularity, we propose to combine these two representations to obtain better performance on downstream tasks.
Revised: • Building on the intuition that global and localized features of speech have different temporal granularity, we propose to combine these two representations to obtain better performance on downstream tasks.

Original: In our evaluations, we report on and analyse the application of localized and global – both, independently and together – self supervised representations of speech on a diverse set of speech classification tasks.
Revised: In our evaluations, we report on and analyse the application of localized and global – both, independently and together – self supervised representations of speech on a diverse set of speech classification tasks.

Original: 2. Proposed Approach
Revised: 2. Proposed Approach

Original: SimSiam Framework for Speech builds on top of SimSiam (see Figure 2 in Appendix A) and is shown in Figure 1: given a raw speech waveform S as input, two random augmented sequences of speech segments – X and Y – are used to enforce invariance across a sequence of speech augmentations described below.
Revised: SimSiam Framework for Speech builds on top of SimSiam (see Figure 2 in Appendix A) and is shown in Figure 1: given a raw speech waveform S as input, two random augmented sequences of speech segments – X and Y – are used to enforce invariance across a sequence of speech augmentations described below.

Original: We obtain the representations of speech by average pooling the encoder outputs across the time steps; accordingly, these representations are used as input to the Projector f .
Revised: We obtain the representations of speech by average pooling the encoder outputs across the time steps; accordingly, these representations are used as input to the Projector f .

Original: We refer to these representations as global representations of speech and apply them in downstream speech classification tasks.
Revised: We refer to these representations as global representations of speech and apply them in downstream speech classification tasks.

Original: Given a raw speech waveform S with sampling frequency f s , we obtain a sequence of segments s
Revised: Given a raw speech waveform S with sampling frequency f s , we obtain a sequence of segments s

Original: i ∈ R d such that s i spans d/f s seconds – in other words, each segment s i contains d waveform amplitudes and the waveform S containing D amplitude samples gets partitioned into T
Revised: i ∈ R d such that s i spans d/f s seconds – in other words, each segment s i contains d waveform amplitudes and the waveform S containing D amplitude samples gets partitioned into T

Original: = D/d segments.
Revised: = D/d segments.

Original: We generate the two augmented views by first par- titioning the sequence of segments { s i } i ∈ [1 ,T ] into two contiguous sequences X
Revised: We generate the two augmented views by first partitioning the sequence of segments { s i } i ∈ [1 ,T ] into two contiguous sequences X

Original: = { x i } i ∈ [1 ,T X ] and Y =
Revised: = { x i } i ∈ [1 ,T X ] and Y =

Original: { y i } i ∈ [1 ,T Y ] with an overlap of 50 - 80 % segments as shown in Figure 1.
Revised: { y i } i ∈ [1 ,T Y ] with an overlap of 50 - 80 % segments as shown in Figure 1.

Original: The duration of the two augmented views X
Revised: The duration of the two augmented views X

Original: and Y is in the range of 2 . 0 - 2 .
Revised: and Y is in the range of 2 . 0 - 2 .

Original: 5 seconds, and T X and T Y can be different.
Revised: 5 seconds, and T X and T Y can be different.

Original: We can now define the outputs obtained after applying the encoder b (i.e. the transformer backbone), projector f and predictor h as z X
Revised: We can now define the outputs obtained after applying the encoder b (i.e. the transformer backbone), projector f and predictor h as z X

Original: = h
Revised: = h

Original: ( f (ˆ b ( X ))) and z Y = stopgrad { f (ˆ b ( X )) } where ˆ b refers to the averagepooled outputs obtained after applying the encoder b .
Revised: ( f (ˆ b ( X ))) and z Y = stopgrad { f (ˆ b ( X )) } where ˆ b refers to the averagepooled outputs obtained after applying the encoder b .

Original: The training objective involves maximizing the cosine similarity between z X and z Y as shown:
Revised: The training objective involves maximizing the cosine similarity between z X and z Y as shown:

Original: [Equation]
Revised: [Equation]

Original: where ∥·∥ 2 refers to l 2 -norm.
Revised: where ∥·∥ 2 refers to l 2 -norm.

Original: Finally, to have a symmetric loss as defined in (Chen & He, 2021), the loss function is defined as:
Revised: Finally, to have a symmetric loss as defined in (Chen & He, 2021), the loss function is defined as:

Original: [Equation]
Revised: [Equation]

Original: Augmentations: For each of the sub-sequences X and Y , we apply the following sequence of augmentations: Gaussian Noise, Shuffling, Masking, and adding silence segments.
Revised: Augmentations: For each of the sub-sequences X and Y , we apply the following sequence of augmentations: Gaussian Noise : To each segment, apply additive white Gaussian noise (AWGN) with SNR between 0 − 10 db

Revised: and then normalize the amplitude of the resulting segment using min-max normalization.

Original: For details, check Appendix refaugs
Revised: Shuffling : Randomly select and shuffle 20 − 40 % of the segments. Masking : Randomly select 20 − 40 % of the segments and replace the segments with an R d vector sampled uniformly from the hypercube [0 . 9 , 1 . 1] d . 4. Add silence segments : Select N random locations and insert silence segments whose R d vector has all components set equal to the lowest amplitude of the original speech waveform S .

Revised: We choose N to be equal to 10 % of the sequence length (i.e. T X or T Y ).

Original: Encoder network b : We use a multi-layer Transformer architecture (Kenton & Toutanova, 2019) as the encoder network.
Revised: Encoder network b : We use a multi-layer Transformer architecture (Kenton & Toutanova, 2019) as the encoder network.

Original: The transformer model consists oftransformer-encoder layers, each containing 12 attention heads.
Revised: The transformer model consists oftransformer-encoder layers, each containing 12 attention heads.

Original: Each segment (i.e. x i and y i ) is projected to a 768-dimensional vector before passing as input to the transformer; for example, the input and output of the transformer for sequence X is of shape T X × 768 .
Revised: Each segment (i.e. x i and y i ) is projected to a 768-dimensional vector before passing as input to the transformer; for example, the input and output of the transformer for sequence X is of shape T X × 768 .

Original: The non-linear projection networks used within the transformer consist of one hidden layer having 2048 units with ReLU as the non-linearity.
Revised: The non-linear projection networks used within the transformer consist of one hidden layer having 2048 units with ReLU as the non-linearity.

Original: Finally, we use the standard self-attention (Vaswani et al., 2017) as the Multi-Head-Attention module with layer-norm normalization.
Revised: Finally, we use the standard self-attention (Vaswani et al., 2017) as the Multi-Head-Attention module with layer-norm normalization.

Original: Projector ( f ) and Predictor ( h ) networks: We follow the architecture of the projector and predictor as defined in (Chen & He, 2021).
Revised: Projector ( f ) and Predictor ( h ) networks: We follow the architecture of the projector and predictor as defined in (Chen & He, 2021).

Original: The projector network takes a 768-dimensional global representation as input and has 2048-dimensional output vector while the prediction network has both input and output dimensions equal to 2048: the projector has two hidden layers, each having 2048 ReLU units and batch normalization (BN) (Ioffe &
Revised: The projector network takes a 768-dimensional global representation as input and has 2048-dimensional output vector while the prediction network has both input and output dimensions equal to 2048: the projector has two hidden layers, each having 2048 ReLU units and batch normalization (BN) (Ioffe &

Original: Szegedy, 2015) in between; the prediction network has one hidden layer with 512 ReLU units and BN.
Revised: Szegedy, 2015) in between; the prediction network has one hidden layer with 512 ReLU units and BN.

Original: Pre-training dataset: For a fair comparison with the previous self-supervised learning models, our SimsSiam-speech model is pre-trained using the LibriSpeech dataset (Panayotov et al., 2015).
Revised: Pre-training dataset: For a fair comparison with the previous self-supervised learning models, our SimSiam-speech model is pre-trained using the LibriSpeech dataset (Panayotov et al., 2015).

Original: LibriSpeech dataset derived from public domain audio books contain 960 hours of speech data collected from 2338 different speakers ( 1128 female and 1210 male speakers).
Revised: LibriSpeech dataset derived from public domain audio books contain 960 hours of speech data collected from 2338 different speakers ( 1128 female and 1210 male speakers).

Original: In this work, we discard utterances with duration less than 3 seconds as we select two speech segments each of duration ranging from 2 . 0 - 2 . 5 seconds, with an overlap of 50 - 80 % in duration, to train our models.
Revised: In this work, we discard utterances with duration less than 3 seconds as we select two speech segments each of duration ranging from 2 . 0 - 2 . 5 seconds, with an overlap of 50 - 80 % in duration, to train our models.

Original: Pre-training Details: Input speech is sampled at f s = 16 kHz.
Revised: Pre-training Details: Input speech is sampled at f s = 16 kHz.

Original: We take raw speech waveform as input and use d = 1000 for creating segments.
Revised: We take raw speech waveform as input and use d = 1000 for creating segments.

Original: Our models are pre-trained with a batch size of 480 using Adam optimizer with a learning rate of 3 e − 4 and a weight decay of 5 e − 5 .
Revised: Our models are pre-trained with a batch size of 480 using Adam optimizer with a learning rate of 3 e − 4 and a weight decay of 5 e − 5 .

Original: Furthermore, the learning rate has a cosine decay schedule (Chen et al., 2020a).
Revised: Furthermore, the learning rate has a cosine decay schedule (Chen et al., 2020a).

Original: 3. Experiments 3.1. Downstream Tasks
Revised: 3. Experiments 3.1. Downstream Tasks

Original: We compare and evaluate the representations extracted from the pre-trained SimSiam-speech model on a diverse set of downstream tasks to test different aspects of the representations.
Revised: We compare and evaluate the representations extracted from the pre-trained SimSiam-speech model on a diverse set of downstream tasks to test different aspects of the representations.

Original: The downstream tasks are summarized in Table 2 (In Appendix B) and are as follows:
Revised: The downstream tasks are summarized in Table 1 and are as follows:

Original: Sentiment classification: We use CMU Multimodal Opinion-Level Sentiment Intensity (CMU-MOSI)
Revised: Sentiment classification: We use CMU Multimodal Opinion-Level Sentiment Intensity (CMU-MOSI)

Revised: [Table]

Original: (Zadeh et al., 2016) and (2) CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) (Zadeh et al., 2018) datasets to evaluate the pre-trained models for the task of sentiment classification.
Revised: (Zadeh et al., 2016) and (2) CMU Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) (Zadeh et [Table] 2018) datasets to evaluate the pre-trained models for the task of sentiment classification.

Original: Low-vocabulary speech recognition: We use speech commands (Warden, 2018) dataset which contains short spoken commands for the task of low-vocabulary speech recognition.
Revised: Low-vocabulary speech recognition: We use speech commands (Warden, 2018) dataset which contains short spoken commands for the task of low-vocabulary speech recognition.

Original: Speech emotion recognition: We use IEMOCAP (Busso et al., 2008) dataset for the downstream task of speech emotion recognition.
Revised: Speech emotion recognition: We use IEMOCAP (Busso et al., 2008) dataset for the downstream task of speech emotion recognition.

Original: Depression detection: DAIC-WOZ (Gratch et al., 2014) dataset is used for the downstream task of depression detection.
Revised: Depression detection: DAIC-WOZ (Gratch et al., 2014) dataset is used for the downstream task of depression detection.

Original: 3.2. Models for Comparison
Revised: 3.2. Models for Comparison

Original: We compare the performance of the pre-trained SimSiamspeech model with various publicly available SOTA speech representation models i.e, Mockingjay (Liu et al., 2020),
Revised: We compare the performance of the pre-trained SimSiamspeech model with various publicly available SOTA speech representation models: • Mockingjay (Liu et al., 2020): Mockingjay, a transformer-based model, was trained using BERT-like loss to predict masked frames.

Original:  VQ-wav2vec (Baevski et al., 2019), Wav2vec-2.0 (Baevski et al., 2020), HuBERT (Hsu et al., 2021), TRILL (Shoret al., 2020).
Revised: • VQ-wav2vec (Baevski et al., 2019): VQ-wav2vec uses online k-means clustering to quantize the dense representations for training a deep convolutional network with a BERT-like loss. • Wav2vec-2.0 (Baevski et al., 2020): Wav2vec-2.0, a transformer-based model, was trained to predict the discrete representation of the masked frames using a BERT-like loss. • HuBERT (Hsu et al., 2021): HuBERT, an extension of wav2vec-2.0, utilizes an offline clustering step to provide aligned target labels to train transformer model using BERT-like loss. • TRILL (Shor

Revised: , 2020): TRILL, a transformerbased model, was trained on mel spectrograms using a triplet loss.

Original: Note that VQ-wav2vec, wav2vec-2.0 and HuBERT takes raw speech waveform as input whereas Mockingjay and TRILL takes mel spectrogram as input.
Revised: Note that VQ-wav2vec, wav2vec-2.0 and HuBERT takes raw speech waveform as input whereas Mockingjay and TRILL takes mel spectrogram as input.

Original: While Mockingjay,VQ-wav2vec, wav2vec-2.0 and HuBERT learn [Table]
Revised: While Mockingjay,

Original: [Table] [Table] localized features , TRILL learns global features.
Revised: VQ-wav2vec, wav2vec-2.0 and HuBERT learn localized features, TRILL learns global features.

Original: Further, we also use the openSMILE (Eyben et al., 2010) and COVAREP (Degottex et al., 2014) features as a baseline.See
Revised: Further, we also use the openSMILE (Eyben et al., 2010) and COVAREP (Degottex et al., 2014) features as a baseline.

Original: Appendix D for details.

Original: 3.3. Evaluation Details
Revised: 3.3. Evaluation Details

Original: For each downstream dataset, we freeze the weights of the pre-trained models, and extract representations from the encoder for the downstream task.
Revised: For each downstream dataset, we freeze the weights of the pre-trained models, and extract representations from the encoder for the downstream task.

Original: We apply a non-linear projection (i.e. linear projection followed by non-linearity) to these representations and then pass it as input to the softmax classifier.
Revised: We apply a non-linear projection (i.e. linear projection followed by non-linearity) to these representations and then pass it as input to the softmax classifier.

Original: The number of hidden units are selected based on the performance on the validation set.
Revised: The number of hidden units are selected based on the performance on the validation set.

Original: For the case of combining the representations extracted from different pre-trained models, we simply concatenate the non-linear projections applied to the representations before passing it as input to the softmax layer.
Revised: For the case of combining the representations extracted from different pre-trained models, we simply concatenate the non-linear projections applied to the representations before passing it as input to the softmax layer.

Original: 3.4. Results
Revised: 3.4. Results

Original: We provide results obtained by performing 5-fold cross validation on each dataset, unless otherwise mentioned.
Revised: We provide results obtained by performing 5-fold cross validation on each dataset, unless otherwise mentioned.

Original: Table 1 shows the performance of the proposed SimSiamSpeech (SimSiam-S) compared to previous self-supervised models across four diverse set of downstream tasks.
Revised: Table 2 shows the performance of the proposed SimSiamSpeech (SimSiam-S) compared to previous self-supervised models across four diverse set of downstream tasks.

Original: For the downstream tasks of emotion recognition and depression detection, proposed SimSiam-S and TRILL performs better than the other pre-trained models, with SimSiam-S outperforming TRILL.
Revised: For the downstream tasks of emotion recognition and depression detection, proposed SimSiam-S and TRILL performs better than the other pre-trained models, with SimSiam-S outperforming TRILL.

Original: For the downstream tasks of sentiment classification and low-vocabulary speech recognition, HuBERT and wav2vec-2.0 performs better.
Revised: For the downstream tasks of sentiment classification and low-vocabulary speech recognition, HuBERT and wav2vec-2.0 performs better.

Original: This shows that the models learning global representations (SimSiam-S and TRILL) perform better on some tasks while models learning localized representations (HuBERT and wav2vec-2.0) perform better on other tasks.
Revised: This shows that the models learning global representations (SimSiam-S and TRILL) perform better on some tasks while models learning local- [Table] ized representations (HuBERT and wav2vec-2.0) perform better on other tasks.

Original: We can also observe in Table 1 that combining global and localized representations (HuBERT + SimSiam-S and wav2vec-2.0 + SimSiam-S) achieve significant improvements in performance compared to the case of combining two global (wav2vec-2.0 + HuBERT) or two localized representations (TRILL + SimSiam-S).
Revised: We can also observe in Table 2 that combining global and localized representations (HuBERT + SimSiam-S and wav2vec-2.0 + SimSiam-S) achieve significant improvements in performance compared to the case of combining two global (wav2vec-2.0 + HuBERT) or two localized representations (TRILL + SimSiam-S).

Original: For all the downstream tasks, combining SimSiam-S with global representations (HuBERT or wav2vec-2.0) achieves the best performance.
Revised: For all the downstream tasks, combining SimSiam-S with local representations (HuBERT or wav2vec-2.0) achieves the best performance.

Original: 4. Conclusion
Revised: 4. Ablation Studies

Revised: We conducted ablation studies to analyze the importance of stop-gradient, projector and predictor networks when

Revised: SimSiam framework is extended to speech–which is a sequential data.

Revised: We trained different SimSiam-Speech models by discarding one component at a time.

Revised: All the models were pre-trained using the 960 hours of LibriSpeech dataset.

Revised: It can be observed from Table 3 that all the three components i.e., projector, predictor and stop-gradient are required to avoid the model collapse.

Revised: Even discarding a single component will lead to the model collapse.

Revised: These observations are in agreement with those reported in (Chen & He, 2021).

Revised: 5. Conclusion

Original: In this work, we discuss the application of self-supervised embeddings to speech classification tasks as the downstream tasks.
Revised: In this work, we discuss the application of self-supervised embeddings to speech classification tasks as the downstream tasks.

Original: In doing so, we highlight the distinction between selfsupervised learning frameworks that favor global versus localized embeddings of speech: in this study, pretext tasks which operate on smaller speech units are said to yield localized embeddings, while those operating on entire speech sequences are said to yield global embeddings.
Revised: In doing so, we highlight the distinction between selfsupervised learning frameworks that favor global versus localized embeddings of speech: in this study, pretext tasks which operate on smaller speech units are said to yield lo- [Table] calized embeddings, while those operating on entire speech sequences are said to yield global embeddings.

Original: We define a self-supervised learning framework for learning global features of speech and propose combining both global and localized features of speech for obtaining more robust downstream performance.
Revised: We define a self-supervised learning framework for learning global features of speech and propose combining both global and localized features of speech for obtaining more robust downstream performance.

Original: Our empirical experiments confirm that global embeddings of speech capture certain paralinguistic elements and can be particularly useful in tasks such as emotion classification and depression detection, while localized embeddings of speech can be effective in tasks such as sentiment classification and speech-command detection.
Revised: Our empirical experiments confirm that global embeddings of speech capture certain paralinguistic elements and can be particularly useful in tasks such as emotion classification and depression detection, while localized embeddings of speech can be effective in tasks such as sentiment classification and speech-command detection.

Original: Importantly, we find that combining the global and localized representation of speech leads to better performance across a range of tasks than each of the individual embedding methods alone.
Revised: Importantly, we find that combining the global and localized representation of speech leads to better performance across a range of tasks than each of the individual embedding methods alone.

Original: A. Background
Revised: A. Background

Original: The SimSiam framework (Chen & He, 2021) is an extension of BYOL (Grill et al., 2020) for learning self-supervised representations of images.
Revised: The SimSiam framework (Chen & He, 2021) is an extension of BYOL (Grill et al., 2020) for learning self-supervised representations of images.

Original: Like other recently proposed self-supervised learning frameworks, SimSiam induces representations invariant over the set of pre-defined data augmentations.
Revised: Like other recently proposed self-supervised learning frameworks, SimSiam induces representations invariant over the set of pre-defined data augmentations.

Original: The core components of the SimSiam framework are as shown in Figure 2: given an input image x , the two random augmented views – x 1 and x 2 – of the input x are first passed through a series of Encoder-Projector-Predictor, and Encoder-Projector networks respectively; the training objective involves maximizing the cosine similarity between these final representations.
Revised: The core components of the SimSiam framework are as shown in Figure 2: given an input image x , the two random augmented views – x 1 and x 2 – of the input x are first passed through a series of Encoder-Projector-Predictor, and Encoder-Projector networks respectively; the training objective involves maximizing the cosine similarity between these final representations.

Original: The predictor and the stop-gradient are essential to avoid model collapse, which we will also show in the ablation studies.
Revised: The predictor and the stop-gradient are essential to avoid model collapse, which we will also show in the ablation studies.

Original: Finally, the encoder representations are used as the general-purpose representations of images in downstream tasks.
Revised: Finally, the encoder representations are used as the general-purpose representations of images in downstream tasks.

Original: In summary, SimSiam is a simplified self-supervised learning framework which has been shown to achieve SOTA performance on image downstream tasks without relying on negative sample pairs, large batch sizes and momentum encoder.
Revised: In summary, SimSiam is a simplified self-supervised learning framework which has been shown to achieve SOTA performance on image downstream tasks without relying on negative sample pairs, large batch sizes and momentum encoder.

Original: In this paper, we adapt the SimSiam framework to learn self-supervised global representations of speech.B. Downstream Task Details [Table] [Table]
Revised: In this paper, we adapt the SimSiam framework to learn self-supervised global representations of speech.

Original: D

Original: [Table] C. Augmentations Following augmentations are applied to the speech sequences: Gaussian Noise : To each segment, apply additive white Gaussian noise (AWGN) with SNR between 0 − 10 db
Revised: Siamese neural networks, trained using supervised learning framework, were used for speech-based tasks such as voice casting (Gresse et al., 2019) and emotion recognition in speech (Lian et al., 2018).

Original: and then normalize the amplitude of the resulting segment using min-max normalization.

Original: Shuffling : Randomly select and shuffle 20 − 40 % of the segments.

Original: Masking : Randomly select 20 − 40 % of the segments and replace the segments with an R d vector sampled uniformly from the hypercube [0 . 9 , 1 . 1] d . 4.

Original: Add silence segments : Select N random locations and insert silence segments whose R d vector has all components set equal to the lowest amplitude of the original speech waveform S .

Original: We choose N to be equal to 10 % of the sequence length (i.e. T X or T Y ).

Original: D. Models for comparison

Original: • Mockingjay (Liu et al., 2020): Mockingjay, a transformer-based model, was trained using BERT-like loss to predict masked frames.

Original: • VQ-wav2vec (Baevski et al., 2019): VQ-wav2vec uses online k-means clustering to quantize the dense representations for training a deep convolutional network with a BERT-like loss.

Original: • Wav2vec-2.0 (Baevski et al., 2020): Wav2vec-2.0, a transformer-based model, was trained to predict the discrete representation of the masked frames using a BERT-like loss.

Original: • HuBERT (Hsu et al., 2021): HuBERT, an extension of wav2vec-2.0, utilizes an offline clustering step to provide aligned target labels to train transformer model using BERT-like loss.

Original: • TRILL

Original: (Shor et al., 2020): TRILL, a transformer-based model, was trained on mel spectrograms using a triplet loss.

Original: E. Ablation Studies

Original: We conducted ablation studies to analyze the importance of stop-gradient, projector and predictor networkswhen SimSiam framework is extended to speech–which is a sequential data. We trained different SimSiam-Speech models by discarding one component at a time. All the models were pre-trained using the 960 hours of LibriSpeech dataset. It can be observed from
Revised: Further, siamese neural networks trained in a unsupervised fashion were used for speech-based tasks such as spoken keyword spotting (Riad et al., 2018) and speaker verification (Khan & Hernando Peric´as, 2020).

Original: Table 3 that all the three components i.e., projector, predictor and stop-gradient are required to avoid the model collapse.

Original: Even discarding a single component will lead to the model collapse.

Original: These observations are in agreement with those reported in (Chen & He, 2021).

Original: [Table]