Original: LEARNED IN SPEECH RECOGNITION: CONTEXTUAL ACOUSTIC WORD EMBEDDINGS
Revised: LEARNED IN SPEECH RECOGNITION: CONTEXTUAL ACOUSTIC WORD EMBEDDINGS

Original: ABSTRACT
Revised: ABSTRACT

Original: End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not require a lexicon.
Revised: End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not require a lexicon.

Original: In addition, word models may also be easier to integrate with downstream tasks such as spoken language understanding, because inference (search) is much simpliﬁed compared to phoneme, character or any other sort of sub-word units.
Revised: In addition, word models may also be easier to integrate with downstream tasks such as spoken language understanding, because inference (search) is much simpliﬁed compared to phoneme, character or any other sort of sub-word units.

Original: In this paper, we describe methods to construct contextual acoustic word embeddings directly from a supervised sequence-to-sequence acoustic-to-word speech recognition model using the learned attention distribution.
Revised: In this paper, we describe methods to construct contextual acoustic word embeddings directly from a supervised sequence-to-sequence acoustic-to-word speech recognition model using the learned attention distribution.

Original: On a suite of 16 standard sentence evaluation tasks, our embeddings show competitive performance against a word2vec model trained on the speech transcriptions.
Revised: On a suite of 16 standard sentence evaluation tasks, our embeddings show competitive performance against a word2vec model trained on the speech transcriptions.

Original: In addition, we evaluate these embeddings on a spoken language understanding task, and observe that our embeddings match the performance of text-based embeddings in a pipeline of ﬁrst performing speech recognition and then constructing word embeddings from transcriptions
Revised: In addition, we evaluate these embeddings on a spoken language understanding task, and observe that our embeddings match the performance of text-based embeddings in a pipeline of ﬁrst performing speech recognition and then constructing word embeddings from transcriptions.

Original: 1. INTRODUCTION
Revised: 1. INTRODUCTION

Original: The task of learning ﬁxed-size representations for variable length data like words or sentences, either text or speech-based, is an interesting problem and a focus of much current research.
Revised: The task of learning ﬁxed-size representations for variable length data like words or sentences, either text or speech-based, is an interesting problem and a focus of much current research.

Original: In the natural language processing community, methods like word2vec [1],
Revised: In the natural language processing community, methods like word2vec [1],

Original: GLoVE [2], CoVe [3] and ELMo [4] have become increasingly popular, due to their utility in several natural language processing tasks.
Revised: GLoVE [2], CoVe [3] and ELMo [4] have become increasingly popular, due to their utility in several natural language processing tasks.

Original: Similar research has progressed in the speech recognition community, where however the input is a sequence of short-term audio features, rather than words or characters.
Revised: Similar research has progressed in the speech recognition community, where however the input is a sequence of short-term audio features, rather than words or characters.

Original: Therefore, the variability in speakers, acoustics or microphones for different occurrences of the same word or sentence adds to the challenge.
Revised: Therefore, the variability in speakers, acoustics or microphones for different occurrences of the same word or sentence adds to the challenge.

Original: Prior work towards the problem of learning word representations from variable length acoustic frames involved either providing word boundaries to align speech and text [5], or chunking (“chopping” or “padding”) input speech into ﬁxed-length segments that usually span only one word [6, 7, 8, 9].
Revised: Prior work towards the problem of learning word representations from variable length acoustic frames involved either providing word boundaries to align speech and text [5], or chunking (“chopping” or “padding”) input speech into ﬁxed-length segments that usually span only one word [6, 7, 8, 9].

Original: Since these techniques learn acoustic word embeddings from audio fragment and word pairs obtained via a given segmentation of the audio data, they ignore the speciﬁc audio context associated with a particular word.
Revised: Since these techniques learn acoustic word embeddings from audio fragment and word pairs obtained via a given segmentation of the audio data, they ignore the speciﬁc audio context associated with a particular word.

Original: So the resulting word embeddings do not capture the contextual dependencies in speech.
Revised: So the resulting word embeddings do not capture the contextual dependencies in speech.

Original: In contrast, our work constructs individual acoustic word embeddings grounded in utterance-level acoustics.
Revised: In contrast, our work constructs individual acoustic word embeddings grounded in utterance-level acoustics.

Original: In this paper, we present different methods of obtaining acoustic word embeddings from an attention-based sequence-to-sequence model [10, 11, 12] trained for direct Acoustic-to-Word (A2W) speech recognition [13].
Revised: In this paper, we present different methods of obtaining acoustic word embeddings from an attention-based sequence-to-sequence model [10, 11, 12] trained for direct Acoustic-to-Word (A2W) speech recognition [13].

Original: Using this model, we jointly learn to automatically segment and classify input speech into individual words, hence getting rid of the problem of chunking or requiring pre-deﬁned word boundaries.
Revised: Using this model, we jointly learn to automatically segment and classify input speech into individual words, hence getting rid of the problem of chunking or requiring pre-deﬁned word boundaries.

Original: As our A2W model is trained at the utterance level, we show that we can not only learn acoustic word embeddings, but also learn them in the proper context of their containing sentence.
Revised: As our A2W model is trained at the utterance level, we show that we can not only learn acoustic word embeddings, but also learn them in the proper context of their containing sentence.

Original: We also evaluate our contextual acoustic word embeddings on a spoken language understanding task, demonstrating that they can be useful in non-transcription downstream tasks.
Revised: We also evaluate our contextual acoustic word embeddings on a spoken language understanding task, demonstrating that they can be useful in non-transcription downstream tasks.

Original: Our main contributions in this paper are the following:
Revised: Our main contributions in this paper are the following:

Original: We demonstrate the usability of attention not only for aligning words to acoustic frames without any forced alignment but also for constructing Contextual Acoustic Word Embeddings (CAWE).
Revised: We demonstrate the usability of attention not only for aligning words to acoustic frames without any forced alignment but also for constructing Contextual Acoustic Word Embeddings (CAWE).

Original: We demonstrate that our methods to construct word representations (CAWE) directly from a speech recognition model are highly competitive with the text-based word2vec embeddings [1], as evaluated on 16 standard sentence evaluation benchmarks.
Revised: We demonstrate that our methods to construct word representations (CAWE) directly from a speech recognition model are highly competitive with the text-based word2vec embeddings [1], as evaluated on 16 standard sentence evaluation benchmarks.

Original: We demonstrate the utility of CAWE on a speech-based downstream task of Spoken Language Understanding showing that pretrained speech models could be used for transfer learning similar to
Revised: We demonstrate the utility of CAWE on a speech-based downstream task of Spoken Language Understanding showing that pretrained speech models could be used for transfer learning similar to

Original: VGG in vision [14] or CoVe in natural language understanding [3].
Revised: VGG in vision [14] or CoVe in natural language understanding [3].

Original: 2. RELATED WORK
Revised: 2. RELATED WORK

Original: A2W modeling has been largely pursued using Connectionist Temporal Classiﬁcation (CTC) models [15] and Sequence-to-Sequence (S2S) models [10].
Revised: A2W modeling has been largely pursued using Connectionist Temporal Classiﬁcation (CTC) models [15] and Sequence-to-Sequence (S2S) models [10].

Original: Prior work shows the need for large amounts of training data for these models (thousands of hours of speech) with large word vocabularies of frequently occurring words [16, 17, 18, 19, 20].
Revised: Prior work shows the need for large amounts of training data for these models (thousands of hours of speech) with large word vocabularies of frequently occurring words [16, 17, 18, 19, 20].

Original: Progress in the ﬁeld showed the possibility of training these models with smaller amount of data (300 hours Switchboard corpus [21]) but restricting the vocabulary to words occurring atleast 5 ortimes [22, 23].
Revised: Progress in the ﬁeld showed the possibility of training these models with smaller amount of data (300 hours Switchboard corpus [21]) but restricting the vocabulary to words occurring atleast 5 ortimes [22, 23].

Original: The solutions to generate out-of-vocabulary words have revolved around backing off to smaller units like characters or sub-words [22, 17, 18, 23, 20].
Revised: The solutions to generate out-of-vocabulary words have revolved around backing off to smaller units like characters or sub-words [22, 17, 18, 23, 20].

Original: While this solves the problem of rare word generation, the models are no longer pure-word models.
Revised: While this solves the problem of rare word generation, the models are no longer pure-word models.

Original: [24] present one of the ﬁrst S2S models for pure-word large vocabulary A2W recognition with the 300 hour Switchboard corpus with a vocabulary of about 30,000 words.
Revised: [24] present one of the ﬁrst S2S models for pure-word large vocabulary A2W recognition with the 300 hour Switchboard corpus with a vocabulary of about 30,000 words.

Original: [25, 13] build upon their work and improve the training of these models for the large vocabulary task.
Revised: [25, 13] build upon their work and improve the training of these models for the large vocabulary task.

Original: [13] is one of our previous works where we show that the direct A2W model is also able to automatically learn word boundaries without any supervision and is the current best pure-word
Revised: [13] is one of our previous works where we show that the direct A2W model is also able to automatically learn word boundaries without any supervision and is the current best pure-word

Original: S2S model.
Revised: S2S model.

Original: We use the same model in this work and expand it towards learning acoustic embeddings.
Revised: We use the same model in this work and expand it towards learning acoustic embeddings.

Original: [5, 6, 8, 7, 26, 9] all explore ways to learn acoustic word embeddings.
Revised: [5, 6, 8, 7, 26, 9] all explore ways to learn acoustic word embeddings.

Original: All above methods except [7] use unsupervised learning based methods to obtain these embeddings where they do not use the transcripts or do not perform speech recognition.
Revised: All above methods except [7] use unsupervised learning based methods to obtain these embeddings where they do not use the transcripts or do not perform speech recognition.

Original: [7] use a supervised Convolutional Neural Network based speech recognition model but with short speech frames as input that usually correspond to a single word.
Revised: [7] use a su- pervised Convolutional Neural Network based speech recognition model but with short speech frames as input that usually correspond to a single word.

Original: This is the common practice in most prior work that simpliﬁes training but prevents the models to scale to learn contextual word embeddings grounded in utterance level acoustics.
Revised: This is the common practice in most prior work that simpliﬁes training but prevents the models to scale to learn contextual word embeddings grounded in utterance level acoustics.

Original: [5] propose an unsupervised method to learn speech embeddings using a ﬁxed context of words in the past and future.
Revised: [5] propose an unsupervised method to learn speech embeddings using a ﬁxed context of words in the past and future.

Original: The drawbacks of their method are the ﬁxed context and need for forced alignment between speech and words for training.
Revised: The drawbacks of their method are the ﬁxed context and need for forced alignment between speech and words for training.

Original: Learning text-based word embeddings is also a rich area of research with well established techniques such as [1, 2].
Revised: Learning text-based word embeddings is also a rich area of research with well established techniques such as [1, 2].

Original: Research has further progressed into learning contextualized word embeddings [3, 4] that are useful in many text-based downstream tasks [27].
Revised: Research has further progressed into learning contextualized word embeddings [3, 4] that are useful in many text-based downstream tasks [27].

Original: [3] learns contextual word embeddings from a fully trained machine translation model and depict re-use of their encoder in other downstream tasks.
Revised: [3] learns contextual word embeddings from a fully trained machine translation model and depict re-use of their encoder in other downstream tasks.

Original: Our work ties A2W speech recognition model with learning contextual word embeddings from speech.
Revised: Our work ties A2W speech recognition model with learning contextual word embeddings from speech.

Original: 3. ACOUSTIC-TO-WORD RECOGNITION
Revised: 3. ACOUSTIC-TO-WORD RECOGNITION

Original: Our S2S model is similar in structure to the Listen, Attend and Spell model [11] which consists of 3 components: the encoder network, a decoder network and an attention model.
Revised: Our S2S model is similar in structure to the Listen, Attend and Spell model [11] which consists of 3 components: the encoder network, a decoder network and an attention model.

Original: The encoder maps the input acoustic features vectors x = ( x 1 , x 2 , ..., x T ) where x i ∈ R d , into a sequence of higher-level features h
Revised: The encoder maps the input acoustic features vectors a = ( a 1 , a 2 , ..., a T ) where a i ∈ R d , into a sequence of higher-level features h

Original: = ( h 1 , h 2 , ..., h T (cid:48) ) .
Revised: = ( h 1 , h 2 , ..., h T (cid:48) ) .

Original: The encoder is a pyramidal  multi-layer bi-directional LongShort Term
Revised: The encoder is a pyramidal (sub-sampling) multi-layer bi-directional Long

Original:  Memory (BLSTM) network that skips every other frame between certain layers.
Revised: Short Term Memory (BLSTM) network.

Original: This reduces the length of the input from T to T (cid:48) .

Original: The decoder network is also an LSTM network that learns to model the output distribution over the next target conditioned on sequence of previous predictions i.e. P ( y l | y ∗ l − 1 , y ∗ l − 2 , ..., y ∗ 0 , x ) where y ∗ =
Revised: The decoder network is also an LSTM network that learns to model the output distribution over the next target conditioned on sequence of previous predictions i.e. P ( y l | y ∗ l − 1 , y ∗ l − 2 , ..., y ∗ 0 , x ) where y ∗ =

Original: ( y ∗ 0 , y ∗ 1 , ..., y ∗ L +1 ) is the ground-truth label sequence.
Revised: ( y ∗ 0 , y ∗ 1 , ..., y ∗ L +1 ) is the ground-truth label sequence.

Original: In this work, y ∗ i ∈
Revised: In this work, y ∗ i ∈

Original: U is from a word vocabulary.
Revised: U is from a word vocabulary.

Original: This decoder generates targets y from h using an attention mechanism.
Revised: This decoder generates targets y from h using an attention mechanism.

Original: The attention model learns an alignment weight vector between the encoding h and the current output of decoder y l .

Original: At each time step, the attention module computes a context vector that is fed into the decoder together with the previous ground-truth label y ∗ l − 1 .

Original: We use the location-aware attention mechanism [12] that enforces monotonicity in the alignments by applying a convolution across time to the attention of previous time step using trainable ﬁlters.
Revised: We use the location-aware attention mechanism [12] that enforces monotonicity in the alignments by applying a convolution across time to the attention of previous time step.

Original: This convolved attention feature is used for calculating the attention for the current time step which leads to a peaky distribution [13].
Revised: This convolved attention feature is used for calculating the attention for the current time step which leads to a peaky distribution [12, 13].

Original: Our model follows the same experimental setup and model hyperparameters as the word-based models described in our previous work [13] with the difference of learning 300 dimensional acoustic feature vectors instead of 320 dimensional.
Revised: Our model follows the same experimental setup and model hyper-parameters as the word-based models described in our previous work [13] with the difference of learning 300 dimensional acoustic feature vectors instead of 320 dimensional.

Original: 4. CONTEXTUAL ACOUSTIC WORD EMBEDDINGS
Revised: 4. CONTEXTUAL ACOUSTIC WORD EMBEDDINGS

Original: We now describe our method to obtain the acoustic word embeddings from the end-to-end trained speech recognition system described in Section 3.
Revised: We now describe our method to obtain the acoustic word embeddings from the end-to-end trained speech recognition system described in Section 3.

Original: The model is as shown in Figure 1 where the embeddings are constructed using the hidden representations obtained from the encoder and the attention weights from the decoder.
Revised: The model is as shown in Figure 1 where the embeddings are constructed using the hidden representations obtained from the encoder and the attention weights from the decoder.

Revised: Our method of constructing “contextual” acoustic word embeddings is similar to a method proposed for text embeddings, CoVe [3].

Original: The main challenge that separates our method from CoVe [3] in learning embeddings from a supervised task, is the problem of alignment between input speech and output words.
Revised: The main challenge that separates our method from CoVe [3] in learning embeddings from a supervised task, is the problem of alignment between input speech and output words.

Original: We use the locationaware attention mechanism that has the property to assign higher probability to certain frames leading to a peaky attention distribution.
Revised: We use the location-aware attention mechanism that has the property to assign higher probability to certain frames leading to a peaky attention distribution.

Original: We exploit this property of location-aware attention in an A2W model to automatically segment continuous speech into words as shown in our previous work [13], and then use this segmentation to 
Revised: We exploit this property of location-aware attention in an A2W model to automatically segment continuous speech into words as shown in our previous work [13], and then use this segmentation to obtain word embeddings. In the next two subsections, we formalize this process

Original: AttentionDecoder
Revised: AttentionDecoder

Original: Vocabulary...
Revised: Vocabulary

Original: [Figure]
Revised: [Figure]

Original: Speech

Original: CAWE
Revised: CAWE

Original: W
Revised: W

Original: W 3 W
Revised: W [Figure]

Original: W
Revised: W

Original: W
Revised: W

Original: Wobtain word embeddings. In the next two subsections, we formalize this process of constructing contextual acoustic word embeddings.
Revised: W of constructing contextual acoustic word embeddings.

Original: Intuitively, attention weights on the acoustic frames hidden representations reﬂect their importance in classifying a particular word.
Revised: Intuitively, attention weights on the acoustic frames hidden representations reﬂect their importance in classifying a particular word.

Original: They thereby provide a correspondence between the frame and the word within a given acoustic context.
Revised: They thereby provide a correspondence between the frame and the word within a given acoustic context.

Original: We can thus construct word representations by weighing the hidden representations of these acoustic frames in terms of their importance to the word i.e. the attention weight.
Revised: We can thus construct word representations by weighing the hidden representations of these acoustic frames in terms of their importance to the word i.e. the attention weight.

Original: We show this in the Figure 1 wherein the hidden representations and their attention weights are colored according to their correspondance with a particular word.
Revised: We show this in the Figure 1 wherein the hidden representations and their attention weights are colored according to their correspondence with a particular word.

Original: 4.1. Automatic Segmentation of Speech into Words
Revised: 4.1. Automatic Segmentation of Speech into Words

Original: Given that a j represents the acoustic frame j , let encoder ( a j ) represent the higher-level features obtained for the frame a j ( i.e.  a j  = h
Revised: Given that a j represents the acoustic frame j , let encoder ( a j ) represent the higher-level features obtained for the frame a j ( i.e. encoder ( a j ) = h

Original: = ( h 1 , h 2 , ..., h T (cid:48) ) , as explained in Section 3).
Revised: = ( h 1 , h 2 , ..., h T (cid:48) ) , as explained in Section 3).

Original: Then, for the i th word w i our model ﬁrst obtains the mappings of w i to acoustic frames a K where K is the set such that ∀ k ∈ K k = argmax j ( attention ( a j )) over all utterances U containing the word w i in the training set.
Revised: Then, for the i th word w i our model ﬁrst obtains the mappings of w i to acoustic frames a K where K is the set such that ∀ k ∈ K [Equation] over all utterances U containing the word w i in the training set.

Original: 4.2. Investigating the Utility of Attention Distribution for Constructing Embeddings: CAWE
Revised: 4.2. Investigating the Utility of Attention Distribution for Constructing Embeddings: CAWE

Original: Below we describe three different ways of using attention to obtain acoustic word embeddings for a word w i (here, n ( K ) represents the cardinality of the set K ):
Revised: Below we describe three different ways of using attention to obtain acoustic word embeddings for a word w i (here, n ( K ) represents the cardinality of the set K ):

Original: [Equation]
Revised: [Equation]

Original: Therefore, unweighted Average (U-AVG, Equation 1) is just the unweighted combination of all the hidden representations of acoustic frames mapped to a particular word.
Revised: Therefore, unweighted Average (U-AVG, Equation 1) is just the unweighted combination of all the hidden representations of acoustic frames mapped to a particular word.

Original: Attention weighted Average (CAWE-W, Equation 2) is the weighted average of the hidden representations of all acoustic frames using the attention weights for a given word.
Revised: Attention weighted Average (CAWE-W, Equation 2) is the weighted average of the hidden representations of all acoustic frames using the attention weights for a given word.

Original: Finally, maximum attention (CAWE-M, Equation 3) is the hidden representation of the acoustic frame with the highest attention score fora given word across all utterances in the training
Revised: Finally, maximum attention (CAWE-M, Equation 3) is the hidden representation of the acoustic frame with the highest attention score for

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: data.
Revised: data.

Original: We call the attention-weighted average and the maximum attention based techniques as Contextual Acoustic Word Embeddings (CAWE) since they arehighly contextual owing to the use of attention scores (over all acoustic frames for a given word).
Revised: We call the attention-weighted average and the maximum attention based techniques as Contextual Acoustic Word Embeddings (CAWE) since they are contextual owing to the use of attention scores (over all acoustic frames for a given word).

Original: 5. EXPERIMENTS AND RESULTS
Revised: 5. EXPERIMENTS AND RESULTS

Original: We use a commonly used speech recognition setup, the 300hour
Revised: We use a commonly used speech recognition setup, the 300hour

Original: Switchboard corpus (LDC97S62) [21] which consists of 2,430 twosided telephonic conversations between 500 different speakers and contains 3 million words of text.
Revised: Switchboard corpus (LDC97S62) [21] which consists of 2,430 twosided telephonic conversations between 500 different speakers and contains 3 million words of text.

Original: Our second dataset is a 300 hour subset of the How2 [28] dataset of instructional videos, which contains planned, but free speech, often outdoor and recorded with distant microphones, as opposed to the indoor, telephony, conversational speech of Switchboard.
Revised: Our second dataset is a 300 hour subset of the How2 [28] dataset of instructional videos, which contains planned, but free speech, often outdoor and recorded with distant microphones, as opposed to the indoor, telephony, conversational speech of Switchboard.

Original: There are 13,662 videos with a total of 3.5 million words in this corpus.
Revised: There are 13,662 videos with a total of 3.5 million words in this corpus.

Original: The A2W obtains a word error rate of 22.2% on Switchboard and 36.6% on CallHome set from the
Revised: The A2W obtains a word error rate of 22.2% on Switchboard and 36.6% on CallHome set from the

Original: Switchboard Eval2000 test set and 24.3% on dev5 test set of How2.
Revised: Switchboard Eval2000 test set and 24.3% on dev5 test set of How2.

Original:  Comparing Methods for Constructing Embeddings
Revised: 5.1. Comparing Methods for Constructing Embeddings

Original: Datasets for Downstream Tasks:
Revised: Datasets for Downstream Tasks:

Original: We evaluate our embeddings by using them as features for 16 benchmark sentence evaluation tasks that cover Semantic Textual Similarity (STS 2012-2016 and
Revised: We evaluate our embeddings by using them as features for 16 benchmark sentence evaluation tasks that cover Semantic Textual Similarity (STS 2012-2016 and

Original: STS B), classiﬁcation: Movie Review (MR), product review (CJ), sentiment analysis (SST, SST-FG), question type (TREC), Subjectivity/Objectivity (SUBJ), and opinion polarity (MPQA), entailment and semantic relatedness using the SICK dataset for SICK-E (entailment) and SICK-R (relatedness) and paraphrase detection (MRPC).
Revised: STS B), classiﬁcation: Movie Review (MR), product review (CJ), sentiment analysis (SST, SST-FG), question type (TREC), Subjectivity/Objectivity (SUBJ), and opinion polarity (MPQA), entailment and semantic relatedness using the SICK dataset for SICK-E (entailment) and SICK-R (relatedness) and paraphrase detection (MRPC).

Original: The STS and SICK-R tasks measure Spearman’s coefﬁcient of correlation between embedding based similarity and human scores, hence the scores range from [ − 1 , 1] where higher number denotes high correlation.
Revised: The STS and SICK-R tasks measure Spearman’s coefﬁcient of correlation between embedding based similarity and human scores, hence the scores range from [ − 1 , 1] where higher number denotes high correlation.

Original: All the remaining tasks are measured on test classiﬁcation accuracies.
Revised: All the remaining tasks are measured on test classiﬁcation accuracies.

Original: We use the SentEval toolkit [27] to evaluate.
Revised: We use the SentEval toolkit [27] to evaluate.

Original: Training Details: In all downstream evaluations involving classiﬁcation tasks, we have used a simple logistic regression for classiﬁcation since a better representation should lead to better scores without using complicated models (hence abstracting away model complexities from our evaluations).
Revised: Training Details: In all downstream evaluations involving classiﬁcation tasks, we have used a simple logistic regression for classiﬁcation since a better representation should lead to better scores without using complicated models (hence abstracting away model complexities from our evaluations).

Original: This also means that we can use the concatenation of CAWE and CBOW as features to the logistic regression model without adding tunable embedding parameters.
Revised: This also means that we can use the concatenation of CAWE and CBOW as features to the logistic regression model without adding tunable embedding parameters.

Original: Discussion:
Revised: Discussion:

Original: From the results in Table 1 we see that CAWE-M outperforms U-AVG by 34% and 13% and CAWE-W by 33.9% and 12% on Switchboard and How2 datasets respectively in terms of average performance on STS tasks and leads to better or slightly worse performance on the classiﬁcation tasks.
Revised: From the results in Table 1 we see that CAWE-M outperforms U-AVG by 34% and 13% and CAWE-W by 33.9% and 12% on Switchboard and How2 datasets respectively in terms of average performance on STS tasks and leads to better or slightly worse performance on the classiﬁcation tasks.

Original: We observe that CAWE-W usually performs worse than CAWE-M which could be attributed to a noisy estimation of the word embeddings on the account of taking even the less conﬁdent attention scores while constructing the embedding.
Revised: We observe that CAWE-W usually performs worse than CAWE-M which could be attributed to a noisy estimation of the word embeddings on the account of taking even the less conﬁdent attention scores while constructing the embedding.

Original: In constrast, CAWE-M is constructed using the most conﬁdent attention score obtained over all the occurrences of the acoustic frames corresponding to a particular word.
Revised: In contrast, CAWE-M is constructed using the most conﬁdent attention score obtained over all the occurrences of the acoustic frames corresponding to a particular word.

Original: We also observe that U-AVG performs worse than CAWE-W on STS and SICK-R tasks since it is constructed using an even noisier process in which all encoder hidden representations are weighted equally irrespective of their attention scores.
Revised: We also observe that U-AVG performs worse than CAWE-W on STS and SICK-R tasks since it is constructed using an even noisier process in which all encoder hidden representations are weighted equally irrespective of their attention scores.

Original:  Comparing with Text-based Embeddings
Revised: 5.2. Comparing with Text-based Embeddings

Original: Datasets for Downstream Tasks:
Revised: Datasets for Downstream Tasks:

Original: The datasets are the same as described in Section 5.1.
Revised: The datasets are the same as described in Section 5.1.

Original: Training Details: In all the following comparisons, we compare embeddings obtained only from the training set of the speech recognition model, while the text-based word embeddings are obtained by training Continuous Bag-of-Words (CBOW) word2vec model on all the transcripts (train, validation and test).
Revised: Training Details: In all the following comparisons, we compare embeddings obtained only from the training set of the speech recognition model, while the text-based word embeddings are obtained by training Continuous Bag-of-Words (CBOW) word2vec model on all the transcripts (train, validation and test).

Original: This was done to ensure a fair comparison between our supervised technique and the unsupervised word2vec method.
Revised: This was done to ensure a fair comparison between our supervised technique and the unsupervised word2vec method.

Original: This naturally leads to a smaller vocabulary for CAWE.
Revised: This naturally leads to a smaller vocabulary for CAWE.

Original: Further, one of the drawbacks of A2W speech recognition model is that it fails to capture entire vocabulary, recognizing only 3044 words out of 29874 (out of which 18800 words occur less than 5 times) and 4287 out of 14242 total vocabulary for Switchboard and How2 respectively.
Revised: Further, one of the drawbacks of A2W speech recognition model is that it fails to capture entire vocabulary, recognizing only 3044 words out of 29874 (out of which 18800 words occur less than 5 times) and 4287 out of 14242 total vocabulary for Switchboard and How2 respectively.

Original: Despite this fact, the performance of
Revised: Despite this fact, the performance of

Original: CAWE is very competitive with word2vec CBOW which does not
Revised: CAWE is very competitive with word2vec CBOW which does not

Original: [Table]
Revised: [Table]

Original: suffer from reduced vocabulary problem.
Revised: suffer from reduced vocabulary problem.

Original: Discussion: From Table 2, it is clear that our embeddings perform as well as the text-embeddings.
Revised: Discussion: In Table 2, we see that our embeddings perform as well as the text-embeddings.

Original: Evaluations using CAWE-M extracted from Switchboard based training show that the acoustic embeddings when concatenated with the text embeddings outperform the word2vec embeddings on 10 out of 16 tasks.
Revised: Evaluations using CAWE-M extracted from Switchboard based training show that the acoustic embeddings when concatenated with the text embeddings outperform the word2vec embeddings on 10 out of 16 tasks.

Original: This concatenated embedding shows that we add more information with CAWEM that improves the CBOW embedding as well.
Revised: This concatenated embedding shows that we add more information with CAWEM that improves the CBOW embedding as well.

Original: The gains are more prominent in Switchboard as compared to the How2 dataset since
Revised: The gains are more prominent in Switchboard as compared to the How2 dataset since

Original: How2 is planned instructional speech whereas Switchboard is spontaneous conversational speech (thereby making the How2 characteristics closer to text leading to a stronger CBOW model).
Revised: How2 is planned instructional speech whereas Switchboard is spontaneous conversational speech (thereby making the How2 characteristics closer to text leading to a stronger CBOW model).

Original: 5.3. Evaluation on Spoken Language Understanding
Revised: 5.3. Evaluation on Spoken Language Understanding

Original: Dataset: In addition to generic sentence-level evaluations, we also evaluate CAWE on the widely used ATIS dataset [29] for Spoken Language Understanding (SLU).
Revised: Dataset: In addition to generic sentence-level evaluations, we also evaluate CAWE on the widely used ATIS dataset [29] for Spoken Language Understanding (SLU).

Original: ATIS dataset is comprised of spoken language queries for airline reservations that have intent and named entities.
Revised: ATIS dataset is comprised of spoken language queries for airline reservations that have intent and named entities.

Original: Hence, it is similar in domain to Switchboard, making it a useful test bed for evaluating CAWE on a speech-based downstream evaluation task.
Revised: Hence, it is similar in domain to Switchboard, making it a useful test bed for evaluating CAWE on a speech-based downstream evaluation task.

Original: Training Details: For this task, our model is similar to the simple Recurrent Neural Network (RNN) based model architecture as investigated in [30].
Revised: Training Details: For this task, our model is similar to the simple Recurrent Neural Network (RNN) based model architecture as investigated in [30].

Original: Our architecture is comprised of an embedding layer, a single layer RNN-variant (Simple RNN, Gated Recurrent
Revised: Our architecture is comprised of an embedding layer, a single layer RNN-variant (Simple RNN, Gated Recurrent

Original: Unit (GRU)) along with a dense layer and softmax.
Revised: Unit (GRU)) along with a dense layer and softmax.

Original: In each instance, we train our model for 10 epochs with RMSProp (learning rate 0.001).
Revised: In each instance, we train our model for 10 epochs with RMSProp (learning rate 0.001).

Original: We train each model 3 times with different seed values and report average performance.
Revised: We train each model 3 times with different seed values and report average performance.

Original: Discussion: [30] concluded that text-based word embeddings trained on large text corpora consistently lead to better performance on the ATIS dataset.
Revised: Discussion: [30] concluded that text-based word embeddings trained on large text corpora consistently lead to better performance on the ATIS dataset.

Original: We demonstrate that direct speech-based word embeddings could lead to matching performance when compared to text-based word embeddings in this speech-based downstream task, thus highlighting the utility of our speech based embeddings.
Revised: We demonstrate that direct speech-based word embeddings could lead to matching performance when compared to text-based word embeddings in this speech-based downstream task, thus highlighting the utility of our speech based embeddings.

Original: Specifically, we compare the test scores obtained by initializing the model
Revised: Specifically, we compare the test scores obtained by initializing the model

Original: [Table]
Revised: [Table]

Original: 6. CONCLUSION
Revised: 6. CONCLUSION

Original: We present a method to learn contextual acoustic word embeddings from a sequence-to-sequence acoustic-to-word speech recognition model that learns to jointly segment and classify speech.
Revised: We present a method to learn contextual acoustic word embeddings from a sequence-to-sequence acoustic-to-word speech recognition model that learns to jointly segment and classify speech.

Original: We analyze the role of attention in constructing contextual acoustic word embeddings, and ﬁnd our acoustic embeddings to be highly competitive with word2vec (CBOW) text embeddings.
Revised: We analyze the role of attention in constructing contextual acoustic word embeddings, and ﬁnd our acoustic embeddings to be highly competitive with word2vec (CBOW) text embeddings.

Original: We discuss two variants of such contextual acoustic word embeddings which outperform the simple unweighted average method by upto 34% on semantic textual similarity tasks.
Revised: We discuss two variants of such contextual acoustic word embeddings which outperform the simple unweighted average method by upto 34% on semantic textual similarity tasks.

Original: The embeddings also matched the performance of text-based embeddings  spoken language understanding, showing the use of this model as a pretrained model for other speech-based downstream tasks.
Revised: The embeddings also matched the performance of text-based embeddings in spoken language understanding, showing the use of this model as a pre-trained model for other speech-based downstream tasks.

Original: We surmise that contextual audio embeddings will generalize and improve downstream tasks in a way that is similar to their text counterparts, despite the additional complexity presented by noisy audio input.
Revised: We surmise that contextual audio embeddings will generalize and improve downstream tasks in a way that is similar to their text counterparts, despite the additional complexity presented by noisy audio input.

Original: In the future, we will explore ways to scale our model to larger corpora and larger vocabularies.
Revised: In the future, we will explore ways to scale our model to larger corpora, larger vocabularies and compare with non-contextual acoustic word embedding methods. 7. ACKNOWLEDGEMENTS This work was supported by the Center for Machine Learning and Health (CMLH) at Carnegie