Original: Learned in Speech Recognition: Contextual Acoustic Word Embeddings Anonymous Author
Revised: LEARNED IN SPEECH RECOGNITION: CONTEXTUAL ACOUSTIC WORD EMBEDDINGS

Original: Abstract
Revised: ABSTRACT

Original: End-to-end acoustic-to-word speech recognition models have recently gainedpopularity because they are easy to train, scale well to large amounts of trainingdata, and do not require a lexicon.
Revised: End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not require a lexicon.

Original: In addition, word models may also be easier tointegrate with downstream tasks such as spoken language understanding, becauseinference (search) is much simpliﬁed compared to phoneme, character or anyother sort of sub-word units.
Revised: In addition, word models may also be easier to integrate with downstream tasks such as spoken language understanding, because inference (search) is much simpliﬁed compared to phoneme, character or any other sort of sub-word units.

Original: In this paper, we describe methods to constructcontextual acoustic word embeddings directly from a supervised sequence-tosequence acoustic-to-word speech recognition model using the learned attentiondistribution.
Revised: In this paper, we describe methods to construct contextual acoustic word embeddings directly from a supervised sequence-to-sequence acoustic-to-word speech recognition model using the learned attention distribution.

Original: On a suite of 16 standard sentence evaluation tasks, our embeddingsshow competitive performance against a word2vec model trained on the speechtranscriptions.
Revised: On a suite of 16 standard sentence evaluation tasks, our embeddings show competitive performance against a word2vec model trained on the speech transcriptions.

Original: In addition, we evaluate these embeddings on a spoken languageunderstanding task, and observe that our embeddings match the performance oftext-based embeddings in a pipeline of ﬁrst performing speech recognition andthen constructing word embeddings from transcriptions.
Revised: In addition, we evaluate these embeddings on a spoken language understanding task, and observe that our embeddings match the performance of text-based embeddings in a pipeline of ﬁrst performing speech recognition and then constructing word embeddings from transcriptions

Original: Introduction
Revised: 1. INTRODUCTION

Original: The task of learning ﬁxed-size representations for variable length data like words or sentences, eithertext or speech based, is an interesting problem and a focus of much current research.
Revised: The task of learning ﬁxed-size representations for variable length data like words or sentences, either text or speech-based, is an interesting problem and a focus of much current research.

Original: In the naturallanguage processing community, methods like word2vec Mikolov et al.
Revised: In the natural language processing community, methods like word2vec [1],

Original: [2013], GLoVE [Penningtonet al., 2014], CoVe [McCann et al., 2017] and ELMo [Peters et al., 2018] have become increasinglypopular due to their utility in several natural language processing tasks.
Revised: GLoVE [2], CoVe [3] and ELMo [4] have become increasingly popular, due to their utility in several natural language processing tasks.

Original: Similar research hasprogressed in the speech recognition community, where however the input is a sequence of short-termaudio features, rather than words or characters.
Revised: Similar research has progressed in the speech recognition community, where however the input is a sequence of short-term audio features, rather than words or characters.

Original: Therefore, the variability in speakers, acoustics ormicrophones for different occurrences of the same word or sentence adds to the challenge.
Revised: Therefore, the variability in speakers, acoustics or microphones for different occurrences of the same word or sentence adds to the challenge.

Original: Prior work towards the problem of learning word representations from variable length acoustic framesinvolved either providing word boundaries to align speech and text [Chung and Glass, 2018] orchunking (“chopping” or “padding”) input speech into ﬁxed-length segments that usually span onlyone word [Kamper et al., 2016, Bengio and Heigold, 2014, Harwath and Glass, 2015, He et al.,2016].
Revised: Prior work towards the problem of learning word representations from variable length acoustic frames involved either providing word boundaries to align speech and text [5], or chunking (“chopping” or “padding”) input speech into ﬁxed-length segments that usually span only one word [6, 7, 8, 9].

Original: Since these techniques learn acoustic word embeddings from audio fragment and wordpairs obtained via a given segmentation of the audio data, they ignore the speciﬁc audio contextassociated with a particular word.
Revised: Since these techniques learn acoustic word embeddings from audio fragment and word pairs obtained via a given segmentation of the audio data, they ignore the speciﬁc audio context associated with a particular word.

Original: So the resulting word embeddings do not capture the contextualdependencies in speech.
Revised: So the resulting word embeddings do not capture the contextual dependencies in speech.

Original: In contrast, our work constructs individual acoustic word embeddingsgrounded in utterance-level acoustics.
Revised: In contrast, our work constructs individual acoustic word embeddings grounded in utterance-level acoustics.

Original: In this paper, we present different methods of obtaining acoustic word embeddings from an attentionbased sequence-to-sequence model [Sutskever et al., 2014, Chan et al., 2016a, Chorowski et al. , 2015]trained for direct Acoustic-to-Word (A2W) speech recognition [Palaskar and Metze, 2018].
Revised: In this paper, we present different methods of obtaining acoustic word embeddings from an attention-based sequence-to-sequence model [10, 11, 12] trained for direct Acoustic-to-Word (A2W) speech recognition [13].

Original: Usingthis model, we jointly learn to automatically segment and classify input speech into individual words, hence getting rid of the problem of chunking or requiring pre-deﬁned word boundaries.
Revised: Using this model, we jointly learn to automatically segment and classify input speech into individual words, hence getting rid of the problem of chunking or requiring pre-deﬁned word boundaries.

Original: As our A2Wmodel is trained at the utterance level, we show that we can not only learn acoustic word embeddings,but also learn them in the proper context of their containing sentence.
Revised: As our A2W model is trained at the utterance level, we show that we can not only learn acoustic word embeddings, but also learn them in the proper context of their containing sentence.

Original: We also evaluate our contextualacoustic word embeddings on a spoken language understanding task, demonstrating that they can beuseful in non-transcription downstream tasks.
Revised: We also evaluate our contextual acoustic word embeddings on a spoken language understanding task, demonstrating that they can be useful in non-transcription downstream tasks.

Original: Our main contributions in this paper are the following:
Revised: Our main contributions in this paper are the following:

Original: • We demonstrate the usability of attention not only for aligning words to acoustic frames without any forced alignment but also for constructing Contextual Acoustic Word Embeddings(CAWE).• We demonstrate that our methods to construct word representations (CAWE) directly from aspeech recognition model are highly competitive with the text-based word2vec embeddings[Mikolov et al., 2013], as evaluated on 16 standard sentence evaluation benchmarks.•
Revised: We demonstrate the usability of attention not only for aligning words to acoustic frames without any forced alignment but also for constructing Contextual Acoustic Word Embeddings (CAWE). We demonstrate that our methods to construct word representations (CAWE) directly from a speech recognition model are highly competitive with the text-based word2vec embeddings [1], as evaluated on 16 standard sentence evaluation benchmarks.

Original: We demonstrate the utility of CAWE on the speech-based downstream task of SpokenLanguage Understanding showing thatthe pretrained speech models could be used fortransfer learning similar toVGG in vision [Simonyan and Zisserman, 2014] or CoVe innatural language understanding [McCann et al., 2017]. In the next section, we survey the related works in A2W modeling, as well as learning vectorrepresentations for speech.
Revised: We demonstrate the utility of CAWE on a speech-based downstream task of Spoken Language Understanding showing that pretrained speech models could be used for transfer learning similar to

Original: Our A2W speech recognition model is explained in Section 3. Ourmethods to construct Contextual Acoustic Word Embeddings (CAWE) from the trained A2W speechrecognition model are explained in Section 4. Experimental results and conclusions follow thereafter.
Revised: VGG in vision [14] or CoVe in natural language understanding [3].

Original: Related Work
Revised: 2. RELATED WORK

Original: A2W modeling has been largely pursued using Connectionist Temporal Classiﬁcation (CTC) models[Graves et al., 2006] and Sequence-to-Sequence (S2S) models [Sutskever et al., 2014].
Revised: A2W modeling has been largely pursued using Connectionist Temporal Classiﬁcation (CTC) models [15] and Sequence-to-Sequence (S2S) models [10].

Original: Prior workshows the need for large amounts of training data for these models (thousands of hours of speech)with large word vocabularies containing frequently occurring words [Soltau et al., 2016, Li et al. ,2018, Chan et al., 2016b, Chiu et al., 2017, Rao et al., 2017, Zeyer et al., 2018].
Revised: Prior work shows the need for large amounts of training data for these models (thousands of hours of speech) with large word vocabularies of frequently occurring words [16, 17, 18, 19, 20].

Original: Progress in the ﬁeldshowed the possibility of training these models with smaller amount of data (300 hours Switchboardcorpus [Godfrey et al., 1992]) but restricting the vocabulary to words occurring atleast 5 or 10 times[Audhkhasi et al., 2017b,a, Ueno et al., Toshniwal et al., 2017].
Revised: Progress in the ﬁeld showed the possibility of training these models with smaller amount of data (300 hours Switchboard corpus [21]) but restricting the vocabulary to words occurring atleast 5 ortimes [22, 23].

Original: The solutions to generate out-ofvocabulary words have revolved around backing off to smaller units like characters or sub-words[Audhkhasi et al., 2017a, Sennrich et al., 2015, Li et al., 2018, Chan et al., 2016b, Chiu et al., 2017, Ueno et al., Zeyer et al., 2018].
Revised: The solutions to generate out-of-vocabulary words have revolved around backing off to smaller units like characters or sub-words [22, 17, 18, 23, 20].

Original: While this solves the problem of rare word generation, the modelsare no longer pure-word models.[Lu et al., 2016] present one of the ﬁrst S2S models for pure-word large vocabulary A2W recognitionwith the 300 hour Switchboard corpus with a vocabulary of  30,000 words.[Chen et al., 2018,
Revised: While this solves the problem of rare word generation, the models are no longer pure-word models. [24] present one of the ﬁrst S2S models for pure-word large vocabulary A2W recognition with the 300 hour Switchboard corpus with a vocabulary of about 30,000 words.

Original: Palaskar and Metze, 2018] build upon their work and improve the training of these models for thelarge vocabulary task.
Revised: [25, 13] build upon their work and improve the training of these models for the large vocabulary task.

Original: [Palaskar and Metze, 2018] show that the direct A2W model is also able toautomatically learn word boundaries without any supervision and is the current best pure-wordmodelin S2S models.
Revised: [13] is one of our previous works where we show that the direct A2W model is also able to automatically learn word boundaries without any supervision and is the current best pure-word S2S model.

Original: We use their model in this work. Another area of research that our paper is highly relevant to is learning acoustic word embeddings.[Chung and Glass, 2018, Kamper et al., 2016, Harwath and Glass, 2015, Bengio and Heigold, 2014,
Revised: We use the same model in this work and expand it towards learning acoustic embeddings.

Original: Yuan et al., 2018, He et al., 2016] all explore ways to learn acoustic word embeddings.
Revised: [5, 6, 8, 7, 26, 9] all explore ways to learn acoustic word embeddings.

Original: All abovemethods except [Bengio and Heigold, 2014] use unsupervised learning based methods to obtain theseembeddings where they do not use the transcripts or do not perform speech recognition.
Revised: All above methods except [7] use unsupervised learning based methods to obtain these embeddings where they do not use the transcripts or do not perform speech recognition.

Original: [Bengioand Heigold, 2014] use a supervised Convolutional Neural Network based speech recognition modelbut with short speech frames as input that usually correspond to a single word.
Revised: [7] use a supervised Convolutional Neural Network based speech recognition model but with short speech frames as input that usually correspond to a single word.

Original: This is the commonpractice in most prior work that simpliﬁes training but prevents the models to scale to learn contextualword embeddings grounded in utterance level acoustics.
Revised: This is the common practice in most prior work that simpliﬁes training but prevents the models to scale to learn contextual word embeddings grounded in utterance level acoustics.

Original: [Chung and Glass, 2018] propose anunsupervised method to learn speech embeddings using a ﬁxed context of words in the past andfuture.
Revised: [5] propose an unsupervised method to learn speech embeddings using a ﬁxed context of words in the past and future.

Original: The drawbacks of their method are the ﬁxed context andthe need to obtain forced alignmentbetween speech and words for training.
Revised: The drawbacks of their method are the ﬁxed context and need for forced alignment between speech and words for training.

Original: Learning text-based word embeddings is also a rich area of research with well established techniquessuch as [Mikolov et al., 2013, Pennington et al., 2014].
Revised: Learning text-based word embeddings is also a rich area of research with well established techniques such as [1, 2].

Original: Research has further progressed into learning contextualized word embeddings [McCann et al., 2017, Peters et al., 2018] that are useful in manytext-based downstream tasks as detailed in [Conneau and Kiela, 2018].
Revised: Research has further progressed into learning contextualized word embeddings [3, 4] that are useful in many text-based downstream tasks [27].

Original: [McCann et al., 2017] learnscontextual word embeddings from a fully trained machine translation model and depict re-use oftheir encoder in other downstream tasks.
Revised: [3] learns contextual word embeddings from a fully trained machine translation model and depict re-use of their encoder in other downstream tasks.

Original: Our work ties A2W speech recognition model with learningcontextual word embeddings from speech.
Revised: Our work ties A2W speech recognition model with learning contextual word embeddings from speech.

Original: Acoustic-to-Word Speech Recognition Model3.1 Sequence-to-Sequence Model
Revised: 3. ACOUSTIC-TO-WORD RECOGNITION

Original: Our S2S model is similar in structure to the Listen, Attend and Spell model [Chan et al. , 2016a]which consists of 3 components: the encoder network, a decoder network and an attention model.
Revised: Our S2S model is similar in structure to the Listen, Attend and Spell model [11] which consists of 3 components: the encoder network, a decoder network and an attention model.

Original: The encoder maps the input acoustic features vectors x = ( x 1 , x 2 , ..., x T ) where x i ∈ R d , into asequence of higher-level features h
Revised: The encoder maps the input acoustic features vectors x = ( x 1 , x 2 , ..., x T ) where x i ∈ R d , into a sequence of higher-level features h

Original: = ( h 1 , h 2 , ..., h T (cid:48) ) .
Revised: = ( h 1 , h 2 , ..., h T (cid:48) ) .

Original: The encoder is a  multi-layer bi-directional 
Revised: The encoder is a pyramidal multi-layer bi-directional Long Short Term

Original: Long Short Term Memory (BLSTM)  that is structured as a pyramid by skipping every other framebetween certainencoder layers.
Revised: Memory (BLSTM) network that skips every other frame between certain layers.

Original: This reduces the length of the input from T to T (cid:48) .
Revised: This reduces the length of the input from T to T (cid:48) .

Original: This encodernetwork is analogous to the traditional acoustic model of a speech recognizer. The decoder network isalso an LSTM network that learns to model the output distribution over the next target conditioned onsequence of previous predictions i.e. P ( y l | y ∗ l − 1 , y ∗ l − 2 , ..., y ∗ 0 , x ) where y ∗ =
Revised: The decoder network is also an LSTM network that learns to model the output distribution over the next target conditioned on sequence of previous predictions i.e. P ( y l | y ∗ l − 1 , y ∗ l − 2 , ..., y ∗ 0 , x ) where y ∗ =

Original: ( y ∗ 0 , y ∗ 1 , ..., y ∗ L +1 ) is theground-truth label sequence.
Revised: ( y ∗ 0 , y ∗ 1 , ..., y ∗ L +1 ) is the ground-truth label sequence.

Original: In this work, y ∗ i ∈
Revised: In this work, y ∗ i ∈

Original: U is from a word vocabulary.
Revised: U is from a word vocabulary.

Original: This decodernetworkis similar to the language model in traditional speech recognizer as it generates targets y from husing an attention mechanism.
Revised: This decoder generates targets y from h using an attention mechanism.

Original: The attention model learns an alignment weight vector between theencoding h and the current output of decoder y l .
Revised: The attention model learns an alignment weight vector between the encoding h and the current output of decoder y l .

Original: At each time step, the attention module computes acontext vector that is fed into the decoder together with the previous ground-truth label y ∗ l − 1 .Location-aware Attention
Revised: At each time step, the attention module computes a context vector that is fed into the decoder together with the previous ground-truth label y ∗ l − 1 .

Original: We use a location-aware attention mechanism [Chorowski et al., 2015]that enforces monotonicity in the alignments, which may be beneﬁcial for speech recognition. To doso, the location-aware attention applies a convolution across time to the attention of previous timestep using trainable ﬁlters.
Revised: We use the location-aware attention mechanism [12] that enforces monotonicity in the alignments by applying a convolution across time to the attention of previous time step using trainable ﬁlters.

Original: This convolved attention feature is used for calculating the attention forthe current time step. We apply a one-dimensional convolution K along the input feature axis t togeta set of T features { f } Tt =1 described as follows: [Equation] where a l − 1 =
Revised: This convolved attention feature is used for calculating the attention for the current time step which leads to a peaky distribution [13].

Original: [ a l − 1 , 1 , ..., a l − 1 ,T ] T , g is a learnable vector parameter, { e lt } Tt =1 is a T -dimensionalvector, Lin() is a linear layer with learnable matrix parameters without bias vectors, LinB() is a linearlayer with learnable matrix and bias parameters.

Original: The S2S model is trained by optimizing the cross entropy loss function which maximizes the loglikelihood of the training data.

Original: We use beam search to perform inference.

Original: We also apply unigramlabel smoothing that distributes the probability of most-probable token to prevent the over-conﬁdenceof the model [Pereyra et al., 2017, Chorowski and Jaitly, 2016].

Original: 3.2 Experimental Setup

Original: Datasets We use a commonly used speech recognition setup, the 300 hour Switchboard corpus(LDC97S62)

Original: [Godfrey et al., 1992] which consists of 2,430 two-sided telephonic conversationsbetween 500 different speakers and contains 3 million words of text.

Original: We evaluate on the Eval2000(LDC2002S09, LDC2002T43) set, which includes the Switchboard (SW) subset similar to trainingdata, and the “harder” CallHome (CH) subset.

Original: There are 196,656 total utterances out of which weuse the ﬁrst 4,000 utterances as a validation set, as is commonly done in the Kaldi recipe [Poveyet al., 2011].

Original: Our second dataset is a 300 hour subset of the How2 [Sanabria et al., 2018]

Original: dataset ofinstructional videos, which contains planned, but free speech, often outdoor and recorded with distantmicrophones, as opposed to the indoor, telephony, conversational speech of Switchboard.

Original: There are13,662 videos in total leading to 184,286 utterances for training, 2019 for validation and 2031 fortesting.

Original: There are a total of 3.5 million words (tokens) in this corpus.

Original: Our input features are 80-dimensional log-mel ﬁlter banks normalized with per-speaker mean andvariance.

Original: We also use 3-dimensional pitch features [Ghahremani et al., 2014].

Original: For Switchboard, weframe a large vocabulary task by including all the words in the dataset leading to a vocabulary of29,874 words.

Original: In How2, as the dataset is domain speciﬁc, we restrict the vocabulary to words thatoccur at least 5 times, resulting in 14,242 types. Model Architecture Our implementation is similar to the architecture described in [Palaskar and Metze, 2018]. The encoder consists of 6 layers each with  300 bi-directional LSTM cells (150 in eachdirection).
Revised: Our model follows the same experimental setup and model hyperparameters as the word-based models described in our previous work [13] with the difference of learning 300 dimensional acoustic feature vectors instead of 320 dimensional.

Original: The encoder is a pyramidal or subsampling encoder that skips every other frame in ﬁrst 2layers to get a reduction of T/ 4 in input frames.

Original: We use the AdaDelta [Zeiler, 2012] optimizer.

Original: Thelocation-aware attention convolution uses 10 ﬁlters with width 100.

Original: There is a projection layer of300 dimensions after each layer of the encoder.

Original: The decoder is a single layer LSTM containing 300cells.

Original: We initialize all parameters uniformly within [ − 0 .

Original: and use unigram label smoothing withweight 0.05.

Original: The beam size used for all experiments is 10 and the length penalty in decoding is set to0.1.

Original: We use the ESPnet toolkit [Kim et al., 2017, Watanabe et al., 2017] for our experiments.

Original: 3.3 Results

Original: [Table]

Original: and Metze [2018] 29, 22.

Original: Our SW Word model 29, 22.

Original: How

Original: Our

Original: [Table]

Original: model 14, - - 24. Our S2S A2W model is trained on large vocabulary whole-word targets. To restrict the scope of thispaper to learning acoustic word embeddings, we do not compare our model with other A2W modelstrained using Connectionist Temporal Classiﬁcation [Graves et al., 2006] or Byte-Pair Units [Zeyeret al., 2018]. While those techniques may perform well for speech-to-text tasks, they cannot be usedfor learning acoustic word embeddings, whereas S2S can [Palaskar and Metze, 2018].
Revised: 4. CONTEXTUAL ACOUSTIC WORD EMBEDDINGS

Original: Table 1 showsthat we closely match the performance of other A2W models on the Switchboard task, proving thestrength of our model for speech recognition.

Original: We now describe our method to obtain the acoustic word embeddings from the end-to-end trainedspeech recognition system described in Section 3.
Revised: We now describe our method to obtain the acoustic word embeddings from the end-to-end trained speech recognition system described in Section 3.

Original: The model is as shown in Figure 1 where theembeddings are constructed using the hidden representations obtained from the encoder and theattention weights from the decoder.
Revised: The model is as shown in Figure 1 where the embeddings are constructed using the hidden representations obtained from the encoder and the attention weights from the decoder.

Original: The main challenge that separates our method from CoVe[McCann et al., 2017] in learning embeddings from a supervised task, is the problem of alignmentbetween input speech and output words.
Revised: The main challenge that separates our method from CoVe [3] in learning embeddings from a supervised task, is the problem of alignment between input speech and output words.

Original: We use the location-aware attention mechanism that hasthe property to assign higher probability to certain frames leading to a peaky attention distribution.
Revised: We use the locationaware attention mechanism that has the property to assign higher probability to certain frames leading to a peaky attention distribution.

Original: We exploit this property of location-aware attention in an A2W model to automatically segmentcontinuous speech into words as shown by [Palaskar and Metze, 2018], and then use this segmentationto obtain word embeddings. In the next two subsections, we formalize this process of constructingcontextual acoustic word embeddings. Intuitively, attention weights on the acoustic frames hiddenrepresentations reﬂect their importance in classifying a particular word. They thereby provide acorrespondence between the frame and the word within a given acoustic context.
Revised: We exploit this property of location-aware attention in an A2W model to automatically segment continuous speech into words as shown in our previous work [13], and then use this segmentation to

Original: We can thusconstruct word representations by weighing the hidden representations of these acoustic frames interms of their importance to the word i.e. the attention weight.

Original: We show this in the Figure 1 whereinthe hidden representations and their attention weights are colored according to their correspondancewith a particular word.

Original: AttentionDecoder
Revised: AttentionDecoder

Original: Vocabulary ...
Revised: Vocabulary ...

Original: Pyramidal BiLSTM Encoder
Revised: [Figure]

Original: Speech
Revised: Speech

Original: CAWE
Revised: CAWE

Original: W
Revised: W

Original: W 3 W
Revised: W 3 W

Original: W
Revised: W

Original: W 
Revised: W W obtain word embeddings. In the next two subsections, we formalize this process of constructing contextual acoustic word embeddings. Intuitively, attention weights on the acoustic frames hidden representations reﬂect their importance in classifying a particular word.

Original: W
Revised: They thereby provide a correspondence between the frame and the word within a given acoustic context. We can thus construct word representations by weighing the hidden representations of these acoustic frames in terms of their importance to the word i.e. the attention weight. We show this in the Figure 1 wherein the hidden representations and their attention weights are colored according to their correspondance with a particular word.

Original: 4.1 Automatic Segmentation of Speech into Words
Revised: 4.1. Automatic Segmentation of Speech into Words

Original: Given that a j represents the acoustic frame j , let encoder ( a j ) represent the higher-level features obtained for the frame a j ( i.e. a j = h
Revised: Given that a j represents the acoustic frame j , let encoder ( a j ) represent the higher-level features obtained for the frame a j ( i.e. a j = h

Original: = ( h 1 , h 2 , ..., h T (cid:48) ) , as explained in Section 3.1).
Revised: = ( h 1 , h 2 , ..., h T (cid:48) ) , as explained in Section 3).

Original: Then, for the i th word w i our model ﬁrst obtains the mappings of w i to acoustic frames a K where K is the set such that ∀ [Equation]
Revised: Then, for the i th word w i our model ﬁrst obtains the mappings of w i to acoustic frames a K where K is the set such that ∀ k ∈ K k = argmax j ( attention ( a j )) over all utterances U containing the word w i in the training set.

Original: over all utterances U containing the word w i in the training set.4.2 Investigating the Utility of Attention Distribution for Constructing Embeddings: CAWE
Revised: 4.2. Investigating the Utility of Attention Distribution for Constructing Embeddings: CAWE

Original: Below we describe three different ways of using attention to obtain acoustic word embeddings for aword w i (here, n ( K ) represents the cardinality of the set K ):Unweighted Average (U-AVG): w i =
Revised: Below we describe three different ways of using attention to obtain acoustic word embeddings for a word w i (here, n ( K ) represents the cardinality of the set K ):

Original: (cid:80) k encoder ( a k ) n ( K ) Attention Weighted Average (CAWE-W): w i = (cid:80) k attention ( a k ) · encoder ( a k ) n ( K )
Revised: [Equation]

Original: Maximum Attention (CAWE-M): w i = encoder ( argmax k ∈ K ( a k )) Therefore, unweighted Average (U-AVG) is just the unweighted combination of all the hiddenrepresentations of acoustic frames mapped to a particular word.
Revised: Therefore, unweighted Average (U-AVG, Equation 1) is just the unweighted combination of all the hidden representations of acoustic frames mapped to a particular word.

Original: Attention weighted Average (CAWEW) is the weighted average of the hidden representations of all acoustic frames using the attentionweights for a given word.
Revised: Attention weighted Average (CAWE-W, Equation 2) is the weighted average of the hidden representations of all acoustic frames using the attention weights for a given word.

Original: Finally, maximum attention (CAWE-M) is the hidden representation ofthe acoustic frame with the highest attention score for a given word across all utterances in thetraining data.
Revised: Finally, maximum attention (CAWE-M, Equation 3) is the hidden representation of the acoustic frame with the highest attention score for a given word across all utterances in the training [Table]

Original:  We call the attention-weighted average and the maximum attention based representationtechnique as Contextual Acoustic Word Embeddings (CAWE) since they are highly contextual owingto the use of attention scores (over all acoustic frames for a given word) in their construction.
Revised: data. We call the attention-weighted average and the maximum attention based techniques as Contextual Acoustic Word Embeddings (CAWE) since they are highly contextual owing to the use of attention scores (over all acoustic frames for a given word).

Original: Experiments and Results
Revised: 5. EXPERIMENTS AND RESULTS

Revised: We use a commonly used speech recognition setup, the 300hour

Revised: Switchboard corpus (LDC97S62) [21] which consists of 2,430 twosided telephonic conversations between 500 different speakers and contains 3 million words of text.

Revised: Our second dataset is a 300 hour subset of the How2 [28] dataset of instructional videos, which contains planned, but free speech, often outdoor and recorded with distant microphones, as opposed to the indoor, telephony, conversational speech of Switchboard.

Original: 5.1 Comparing Methods for Constructing Embeddings
Revised: There are 13,662 videos with a total of 3.5 million words in this corpus. The A2W obtains a word error rate of 22.2% on Switchboard and 36.6% on CallHome set from the Switchboard Eval2000 test set and 24.3% on dev5 test set of How2. Comparing Methods for Constructing Embeddings

Original: Datasets for Downstream Tasks We evaluate our embeddings by using them as features for 16benchmark sentence evaluation tasks that cover Semantic Textual Similarity (STS 2012-2016 and
Revised: Datasets for Downstream Tasks: We evaluate our embeddings by using them as features for 16 benchmark sentence evaluation tasks that cover Semantic Textual Similarity (STS 2012-2016 and

Original: STS B),binary and multi-class classiﬁcation: Movie Review (MR), product review (CJ), sentimentanalysis (SST, SST-FG), question type (TREC), Subjectivity/Objectivity (SUBJ), and opinion polarity(MPQA), entailment and semantic relatedness using the SICK dataset for SICK-E (entailment) [Table] SICK
Revised: STS B), classiﬁcation: Movie Review (MR), product review (CJ), sentiment analysis (SST, SST-FG), question type (TREC), Subjectivity/Objectivity (SUBJ), and opinion polarity (MPQA), entailment and semantic relatedness using the SICK dataset for SICK-E (entailment) and SICK-R (relatedness) and paraphrase detection (MRPC).

Original: [Table] and SICK-R (relatedness) and paraphrase detection (MRPC). The STS and SICK-R tasks measure Spearman’s coefﬁcient of correlation between embedding based similarity and human scores, hencethe scores range from [ − 1 , 1] where higher number denotes high correlation.
Revised: The STS and SICK-R tasks measure Spearman’s coefﬁcient of correlation between embedding based similarity and human scores, hence the scores range from [ − 1 , 1] where higher number denotes high correlation.

Original: All the remaining tasksare measured on test classiﬁcation accuracies.
Revised: All the remaining tasks are measured on test classiﬁcation accuracies.

Original: We use the SentEval toolkit [Conneau and Kiela,2018] 1 to evaluate.
Revised: We use the SentEval toolkit [27] to evaluate.

Original: Training Details In all downstream evaluations involving classiﬁcation tasks, we have used thesimplest model i.e. logistic regression for classiﬁcation since a better representation should lead tobetter scores without using complicated models (hence abstracting away model complexities fromour evaluations).
Revised: Training Details: In all downstream evaluations involving classiﬁcation tasks, we have used a simple logistic regression for classiﬁcation since a better representation should lead to better scores without using complicated models (hence abstracting away model complexities from our evaluations).

Original: This also means that we can use the concatenation of CAWE and CBOW as featuresto the logistic regression model without adding tunable embedding parameters.
Revised: This also means that we can use the concatenation of CAWE and CBOW as features to the logistic regression model without adding tunable embedding parameters.

Original: Discussion From the results in Table 2 we see that CAWE-M outperforms U-AVG by 34% and 13%and CAWE-W by 33.9% and 12% on Switchboard and How2 datasets respectively in terms of averageperformance on STS tasks and leads to better or slightly worse performance on the classiﬁcationtasks.
Revised: Discussion: From the results in Table 1 we see that CAWE-M outperforms U-AVG by 34% and 13% and CAWE-W by 33.9% and 12% on Switchboard and How2 datasets respectively in terms of average performance on STS tasks and leads to better or slightly worse performance on the classiﬁcation tasks.

Original: We observe that CAWE-W usually performs worse than CAWE-M which could be attributed toa noisy estimation of the word embeddings on the account of taking even the less conﬁdent attentionscores while constructing the embedding.
Revised: We observe that CAWE-W usually performs worse than CAWE-M which could be attributed to a noisy estimation of the word embeddings on the account of taking even the less conﬁdent attention scores while constructing the embedding.

Original: In constrast, CAWE-M is constructed using the mostconﬁdent attention score obtained over all the occurrences of the acoustic frames corresponding to aparticular word.
Revised: In constrast, CAWE-M is constructed using the most conﬁdent attention score obtained over all the occurrences of the acoustic frames corresponding to a particular word.

Original: We also observe that U-AVG performs worse than CAWE-W on STS and SICK-Rtasks since it is constructed using an even noisier process in which all encoder hidden representationsare weighted equally irrespective of their attention scores.
Revised: We also observe that U-AVG performs worse than CAWE-W on STS and SICK-R tasks since it is constructed using an even noisier process in which all encoder hidden representations are weighted equally irrespective of their attention scores.

Original: 5.2 Comparing with
Revised: Comparing with

Original: Text-based Embeddings
Revised: Embeddings

Original: Datasets for Downstream Tasks The datasets are the same as described in Section 5.1.
Revised: Datasets for Downstream Tasks: The datasets are the same as described in Section 5.1.

Original: Training Details In all the following comparisons, we compare embeddings obtained only from thetraining set of the speech recognition model, while the text-based word embeddings are obtained bytraining Continuous Bag-of-Words (CBOW) word2vec model on all the transcripts (train, validationand test).
Revised: Training Details: In all the following comparisons, we compare embeddings obtained only from the training set of the speech recognition model, while the text-based word embeddings are obtained by training Continuous Bag-of-Words (CBOW) word2vec model on all the transcripts (train, validation and test).

Original: This was done to ensure a fair comparison between our supervised technique and theunsupervised word2vec method.
Revised: This was done to ensure a fair comparison between our supervised technique and the unsupervised word2vec method.

Original: This naturally leads to a smaller vocabulary for CAWE.
Revised: This naturally leads to a smaller vocabulary for CAWE.

Original: Further,one of the drawbacks of A2W speech recognition model is that it fails to capture entire vocabulary, [Table] [Table] [Table]
Revised: Further, one of the drawbacks of A2W speech recognition model is that it fails to capture entire vocabulary, recognizing only 3044 words out of 29874 (out of which 18800 words occur less than 5 times) and 4287 out of 14242 total vocabulary for Switchboard and How2 respectively.

Original: [Table] TREC
Revised: Despite this fact, the performance of

Original: [Table] [Table] [Table] [Table]
Revised: CAWE is very competitive with word2vec CBOW which does not

Original: [Table]
Revised: [Table]

Original: SICK-E 74. recognizing only 3044 words out of 29874 (out of which 18800 words occur less than 5 times)and 4287 out of 14242 total vocabulary for Switchboard and How2 respectively. Despite this fact,the performance of CAWE is very competitive with word2vec CBOW which does not suffer fromreduced vocabulary problem.
Revised: suffer from reduced vocabulary problem.

Original: Discussion From Table 3, it is clear that our embeddings perform as well as the text-embeddings.
Revised: Discussion: From Table 2, it is clear that our embeddings perform as well as the text-embeddings.

Original: Evaluations using CAWE-M extracted from Switchboard based training show that the acousticembeddings when concatenated with the text embeddings outperform the word2vec embeddings on10 out of 16 tasks.
Revised: Evaluations using CAWE-M extracted from Switchboard based training show that the acoustic embeddings when concatenated with the text embeddings outperform the word2vec embeddings on 10 out of 16 tasks.

Original: This concatenated embedding shows that we add more information with CAWE-Mthat improves the CBOW embedding as well.
Revised: This concatenated embedding shows that we add more information with CAWEM that improves the CBOW embedding as well.

Original: The gains are more prominent in Switchboard ascompared to the How2 dataset since How2 is planned instructional speech whereas Switchboard isspontaneous conversational speech (thereby making the How2 characteristics closer to text leading toa stronger CBOW model).
Revised: The gains are more prominent in Switchboard as compared to the How2 dataset since How2 is planned instructional speech whereas Switchboard is spontaneous conversational speech (thereby making the How2 characteristics closer to text leading to a stronger CBOW model).

Original: 5.3 Evaluation on Spoken Language Understanding
Revised: 5.3. Evaluation on Spoken Language Understanding

Original: Dataset In addition to generic sentence-level evaluations, we also evaluate CAWE on the widelyused [Tur et al., 2010] ATIS dataset [Price, 1990] for Spoken Language Understanding (SLU).
Revised: Dataset: In addition to generic sentence-level evaluations, we also evaluate CAWE on the widely used ATIS dataset [29] for Spoken Language Understanding (SLU).

Original: ATISdataset [Price, 1990] is comprised of spoken language queries for airline reservations that have intentand named entities.
Revised: ATIS dataset is comprised of spoken language queries for airline reservations that have intent and named entities.

Original: Hence, it is similar in domain to Switchboard, making it a useful test bed forevaluating CAWE on a speech-based downstream evaluation task.
Revised: Hence, it is similar in domain to Switchboard, making it a useful test bed for evaluating CAWE on a speech-based downstream evaluation task.

Original: Training Details For this task, our model is similar to the simple Recurrent Neural Network (RNN)based model architecture as investigated in [Mesnil et al., 2013].
Revised: Training Details: For this task, our model is similar to the simple Recurrent Neural Network (RNN) based model architecture as investigated in [30].

Original: Our architecture is comprised of anembedding layer, a single layer RNN-variant (Simple RNN, Gated Recurrent Unit (GRU) [Chunget al., 2014]) along with a dense layer and softmax.
Revised: Our architecture is comprised of an embedding layer, a single layer RNN-variant (Simple RNN, Gated Recurrent Unit (GRU)) along with a dense layer and softmax.

Original: In each instance, we train our model for 10epochs with RMSProp (learning rate 0.001).
Revised: In each instance, we train our model for 10 epochs with RMSProp (learning rate 0.001).

Original: We train each model 3 times with different seed valuesand report average performance.
Revised: We train each model 3 times with different seed values and report average performance.

Original: Discussion [Mesnil et al., 2013] concluded that text-based word embeddings trained on large textcorpora consistently lead to better performance on the ATIS dataset.
Revised: Discussion: [30] concluded that text-based word embeddings trained on large text corpora consistently lead to better performance on the ATIS dataset.

Original: We demonstrate that directspeech-based word embeddings could lead to matching performance when compared to text-basedword embeddings in this speech-based downstream task, thus highlighting the utility of our speech 
Revised: We demonstrate that direct speech-based word embeddings could lead to matching performance when compared to text-based word embeddings in this speech-based downstream task, thus highlighting the utility of our speech based embeddings. Specifically, we compare the test scores obtained by initializing the model

Original: [Table]
Revised: [Table]

Original: based embeddings.

Original: Speciﬁcally, we compare the initializing

Original: CAWE-M, CAWE-W and CBOW embeddings and ﬁne-tuning them based on the task.

Original: Conclusion
Revised: 6. CONCLUSION

Original: We present a method to learn contextual acoustic word embeddings from an encoder-decoder basedacoustic-to-word speech recognition model that learns to jointly segment and classify speech.
Revised: We present a method to learn contextual acoustic word embeddings from a sequence-to-sequence acoustic-to-word speech recognition model that learns to jointly segment and classify speech.

Original: Weanalyze the role of attention in constructing contextual acoustic word embeddings, and ﬁnd ouracoustic embeddings to be highly competitive with word2vec (CBOW) text embeddings.
Revised: We analyze the role of attention in constructing contextual acoustic word embeddings, and ﬁnd our acoustic embeddings to be highly competitive with word2vec (CBOW) text embeddings.

Original: We discusstwo variants of such contextual acoustic word embeddings which outperform the simple unweightedaverage method by upto 34% on semantic textual similarity tasks.
Revised: We discuss two variants of such contextual acoustic word embeddings which outperform the simple unweighted average method by upto 34% on semantic textual similarity tasks.

Original: The embeddings also matchedthe performance of text-based embeddings spoken language understanding, showing the use of thismodel as a pretrained model for other speech-based downstream tasks.
Revised: The embeddings also matched the performance of text-based embeddings spoken language understanding, showing the use of this model as a pretrained model for other speech-based downstream tasks.

Original: We surmise that contextualaudio embeddings will generalize and improve downstream tasks in a way that is similar to their textcounterparts, despite the additional complexity presented by noisy audio input.
Revised: We surmise that contextual audio embeddings will generalize and improve downstream tasks in a way that is similar to their text counterparts, despite the additional complexity presented by noisy audio input.

Original: In the future, we willexplore ways to scale our model to larger corpora and larger vocabularies.
Revised: In the future, we will explore ways to scale our model to larger corpora and larger vocabularies.