Original: Learned in Speech Recognition: Contextual Acoustic Word Embeddings
Revised: Learned in Speech Recognition: Contextual Acoustic Word Embeddings

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress
Revised: AfﬁliationAddress

Original: Abstract
Revised: Abstract

Original: End-to-end acoustic-to-word speech recognition models have recently gainedpopularity because they are easy to train, scale well to large amounts of trainingdata, and do not require a lexicon.
Revised: End-to-end acoustic-to-word speech recognition models have recently gainedpopularity because they are easy to train, scale well to large amounts of trainingdata, and do not require a lexicon.

Original: In addition, word models may also be easier tointegrate with downstream tasks such as spoken language understanding, becauseinference (search) is much simpliﬁed compared to phoneme, character or anyother sort of sub-word units.
Revised: In addition, word models may also be easier tointegrate with downstream tasks such as spoken language understanding, becauseinference (search) is much simpliﬁed compared to phoneme, character or anyother sort of sub-word units.

Original: In this paper, we describe methods to constructcontextual acoustic word embeddings directly from a supervised sequence-tosequence acoustic-to-word speech recognition model using the learned attentiondistribution.
Revised: In this paper, we describe methods to constructcontextual acoustic word embeddings directly from a supervised sequence-tosequence acoustic-to-word speech recognition model using the learned attentiondistribution.

Original: On a suite of 16 standard sentence evaluation tasks, our embeddingsshow competitive performance against a word2vec model trained on the speechtranscriptions.
Revised: On a suite of 16 standard sentence evaluation tasks, our embeddingsshow competitive performance against a word2vec model trained on the speechtranscriptions.

Original: In addition, we evaluate these embeddings on a spoken languageunderstanding task, and observe that our embeddings match the performance oftext-based embeddings in a pipeline of ﬁrst performing speech recognition andthen constructing word embeddings from transcriptions.
Revised: In addition, we evaluate these embeddings on a spoken languageunderstanding task, and observe that our embeddings match the performance oftext-based embeddings in a pipeline of ﬁrst performing speech recognition andthen constructing word embeddings from transcriptions.

Original:  Introduction
Revised:  Introduction

Original: The task of learning ﬁxed-size representations for variable length data like words or sentences, eithertext or speech based, is an interesting problem and a focus of much current research.
Revised: The task of learning ﬁxed-size representations for variable length data like words or sentences, eithertext or speech based, is an interesting problem and a focus of much current research.

Original: In the naturallanguage processing community, methods like word2vec Mikolov et al.
Revised: In the naturallanguage processing community, methods like word2vec Mikolov et al.

Original: [2013], GLoVE [Penningtonet al., 2014], CoVe [McCann et al., 2017] and ELMo [Peters et al., 2018] have become increasinglypopular due to their utility in several natural langauge processing tasks.
Revised: [2013], GLoVE [Penningtonet al., 2014], CoVe [McCann et al., 2017] and ELMo [Peters et al., 2018] have become increasinglypopular due to their utility in several natural langauge processing tasks.

Original: Similar research hasprogressed in the speech recognition community, where however the input is a sequence of short-termaudio features, rather than words or characters.
Revised: Similar research hasprogressed in the speech recognition community, where however the input is a sequence of short-termaudio features, rather than words or characters.

Original: Therefore, the variability in speakers, acoustics ormicrophones for different occurrences of the same word or sentence adds to the challenge.
Revised: Therefore, the variability in speakers, acoustics ormicrophones for different occurrences of the same word or sentence adds to the challenge.

Original: Prior work towards the problem of learning word representations from variable length acoustic framesinvolved either providing word boundaries to align speech and text [Chung and Glass, 2018] orchunking (“chopping” or “padding”) input speech into ﬁxed-length segments that span only oneword [Kamper et al., 2016, Bengio and Heigold, 2014, Harwath and Glass, 2015, He et al., 2016].
Revised: Prior work towards the problem of learning word representations from variable length acoustic framesinvolved either providing word boundaries to align speech and text [Chung and Glass, 2018] orchunking (“chopping” or “padding”) input speech into ﬁxed-length segments that span only oneword [Kamper et al., 2016, Bengio and Heigold, 2014, Harwath and Glass, 2015, He et al., 2016].

Original: Since these techniques learn acoustic word embeddings from audio fragment and word pairs obtainedvia a given segmentation of the audio data, they ignore the speciﬁc audio context associated witha particular word.
Revised: Since these techniques learn acoustic word embeddings from audio fragment and word pairs obtainedvia a given segmentation of the audio data, they ignore the speciﬁc audio context associated witha particular word.

Original: So the resulting word embeddings do not capture the contextual dependenciesin speech.
Revised: So the resulting word embeddings do not capture the contextual dependenciesin speech.

Original: In contrast, our work constructs individual acoustic word embeddings grounded inutterance-level acoustics.
Revised: In contrast, our work constructs individual acoustic word embeddings grounded inutterance-level acoustics.

Original: In this paper, we present different methods of obtaining acoustic word embeddings from an attentionbased sequence-to-sequence model [Sutskever et al., 2014, Chan et al., 2016a, Chorowski et al.
Revised: In this paper, we present different methods of obtaining acoustic word embeddings from an attentionbased sequence-to-sequence model [Sutskever et al., 2014, Chan et al., 2016a, Chorowski et al.

Original: , 2015]trained for direct Acoustic-to-Word (A2W) speech recognition [Palaskar and Metze, 2018, Uenoet al.,
Revised: , 2015]trained for direct Acoustic-to-Word (A2W) speech recognition [Palaskar and Metze, 2018, Uenoet al.,

Original: Chen et al., 2018, Zeyer et al., 2018, Toshniwal et al., 2017, Lu et al., 2016].
Revised: Chen et al., 2018, Zeyer et al., 2018, Toshniwal et al., 2017, Lu et al., 2016].

Original: Using this model, we jointly learn to automatically segment and classify input speech into individual words, hencegetting rid of the problem of chunking or requiring pre-deﬁned word boundaries.
Revised: Using this model, we jointly learn to automatically segment and classify input speech into individual words, hencegetting rid of the problem of chunking or requiring pre-deﬁned word boundaries.

Original: As our A2W modelis trained at the utterance level, we show that we can not only learn acoustic word embeddings, butalso learn them in the proper context of their containing sentence.
Revised: As our A2W modelis trained at the utterance level, we show that we can not only learn acoustic word embeddings, butalso learn them in the proper context of their containing sentence.

Original: We also evaluate our contextualacoustic word embeddings on a spoken language understanding task, demonstrating that they can beuseful in non-transcription downstream tasks.
Revised: We also evaluate our contextualacoustic word embeddings on a spoken language understanding task, demonstrating that they can beuseful in non-transcription downstream tasks.

Original: Our main contributions in this paper are the following:
Revised: Our main contributions in this paper are the following:

Original: • We demonstrate the usability of attention not only for aligning words to acoustic frames without any forced alignment but also for constructing Contextual Acoustic Word Embeddings(CAWE).
Revised: • We demonstrate the usability of attention not only for aligning words to acoustic frames without any forced alignment but also for constructing Contextual Acoustic Word Embeddings(CAWE).

Original: • We demonstrate that our methods to construct word representations (CAWE) directly from aspeech recognition model are highly competitive with the text-based word2vec
Revised: • We demonstrate that our methods to construct word representations (CAWE) directly from aspeech recognition model are highly competitive with the text-based word2vec

Original: embeddings[Mikolov et al., 2013], as evaluated on 16 standard sentence evaluation benchmarks.
Revised: embeddings[Mikolov et al., 2013], as evaluated on 16 standard sentence evaluation benchmarks.

Original: • We demonstrate the utility of CAWE on the speech-based downstream task of SpokenLanguage Understanding showing that the pretrained speech models could be used fortransfer learning similar to VGG in vision [Simonyan and Zisserman, 2014] or CoVe innatural language understanding [McCann et al., 2017].
Revised: • We demonstrate the utility of CAWE on the speech-based downstream task of Spoken Language Understanding showing that the pretrained speech models could be used for transfer learning similar to VGG in vision [Simonyan and Zisserman, 2014] or CoVe innatural language understanding [McCann et al., 2017].

Original: In the next section, we survey the related works in Acoustic-to-Word modeling, as well as learningvector representations for speech.
Revised: In the next section, we survey the related works in Acoustic-to-Word modeling, as well as learningvector representations for speech.

Original: Our A2W Speech Recognition model is explained in Section 3.
Revised: Our A2W Speech Recognition model is explained in Section 3.

Original: Our methods to construct Contextual Acoustic Word Embeddings from the trained A2W speechrecognition model are explained in Section 4.
Revised: Our methods to construct Contextual Acoustic Word Embeddings from the trained A2W speechrecognition model are explained in Section 4.

Original: Experimental results and conclusions follow thereafter.
Revised: Experimental results and conclusions follow thereafter.

Original:  Related Work
Revised:  Related Work

Original: A2W modeling has been largely pursued using Connectionist Temporal Classiﬁcation [Graves et al. ,2006] models.
Revised: A2W modeling has been largely pursued using Connectionist Temporal Classiﬁcation (CTC) models[Graves et al., 2006] and Sequence-to-Sequence (S2S) models [Sutskever et al., 2014].

Original: [Soltau et al., 2016] introduced the ﬁrst A2W model but needed a very large trainingcorpus (125,000 hours) due to the  large vocabulary of A2W models, 100000 words in their case. Theseword vocabularies are noticeably much larger than character or sub-word vocabularies.
Revised: Prior workshows the need for large amounts of training data for these models (thousands of hours of speech)with large word vocabularies containing frequently occurring words [Soltau et al., 2016, Li et al.

Original: Audhkhasi etal. [Audhkhasi et al., 2017b] found that ﬁltering out rare words and replacing them with an OOVsymbol alleviates the need for large data. But producing OOV would lead to higher word error rates. A common technique to solve this OOV problem has been to use word-level prediction for frequentwords and revert to the character or sub-word prediction for rare words [Li et al., 2018, Audhkhasiet al., 2017a].
Revised: ,2018, Chan et al., 2016b, Chiu et al., 2017, Rao et al., 2017, Zeyer et al., 2018].

Original: This is a two-step procedure, straying from the regular sequence-to-sequence mappingof acoustics to units. Recently, [Li et al., 2018] proposed a hybrid-CTC model where an A2W modelconsulted an Acoustic-to-Letter model upon generation of the OOV symbol. They also proposeda mixed-unit CTC model using frequent words, letters and sub-words although again with largeamounts of data (approximately 3,400 hours).
Revised: Progress in the ﬁeldshowed the possibility of training these models with smaller amount of data (300 hours Switchboardcorpus)

Original: In [Audhkhasi et al., 2017a], the authors proposea Spell and Recognize model where they ﬁrst predict the character sequence of the word beforecomposing it into a word unit (if a frequent word), or preserving the “spelling" as character units.
Revised: but restricting the vocabulary to words occurring atleast 5 or 10 times [Audhkhasi et al. ,2017b,a, Ueno et al.]. The solutions to generate out-of-vocabulary words have revolved aroundbacking off to smaller units like characters or sub-words [Audhkhasi et al., 2017a, Sennrich et al.

Original: Allthese methods described above use the CTC loss function. S2S models have also been used for recognizing sub-word [Sennrich et al., 2015] or word-piece unitsin ASR [Chan et al., 2016b, Chiu et al., 2017, Rao et al., 2017] that no longer have OOV wordsbut these results were presented with 12,500 hours of in-house speech data. Lu et al.
Revised: ,2015, Li et al., 2018, Chan et al., 2016b, Chiu et al., 2017].

Original: [Lu et al.,2016] presents one of the ﬁrst S2S models for  large vocabulary A2W recognition with the 300 hour Switchboard corpus with a vocabulary of 30,000 words.
Revised: While this solves the problem of rareword generation, the models are no longer pure-word models.[Lu et al., 2016] present one of the ﬁrst S2S models for pure-word large vocabulary A2W recognitionwith the 300 hour Switchboard corpus with a vocabulary of 30,000 words.

Original: [Palaskar and Metze, 2018] build upon theirwork improve the training of these models for the large vocabulary task.
Revised: [Palaskar and Metze,2018] build upon their work improve the training of these models for the large vocabulary task.

Original: Their model is also able toautomatically learn word boundaries without any supervision; we use their model in this work.
Revised: Theirmodel is also able to automatically learn word boundaries without any supervision; we use theirmodel in this work.

Revised: The large vocabulary task also represents a pure word model for A2W models.

Original: Another area of research that our paper is highly relevant to is learning acoustic word embeddings.[Chung and Glass, 2018, Kamper et al., 2016, Harwath and Glass, 2015, Bengio and Heigold, 2014,
Revised: Another area of research that our paper is highly relevant to is learning acoustic word embeddings.[Chung and Glass, 2018, Kamper et al., 2016, Harwath and Glass, 2015, Bengio and Heigold, 2014,

Original: Yuan et al., 2018] all explore ways to learn acoustic word embeddings.
Revised: Yuan et al., 2018] all explore ways to learn acoustic word embeddings.

Original: All above methods except[Bengio and Heigold, 2014] use unsupervised learning based methods to obtain these embeddingswhere they do not use the transcripts or do not perform speech recognition.
Revised: All above methods except[Bengio and Heigold, 2014] use unsupervised learning based methods to obtain these embeddingswhere they do not use the transcripts or do not perform speech recognition.

Original: [Bengio and Heigold,2014] use a supervised Convolutional Neural Network based speech recognition model but with shortspeech frames as input that usually correspond to a single word.
Revised: [Bengio and Heigold,2014] use a supervised Convolutional Neural Network based speech recognition model but with shortspeech frames as input that usually correspond to a single word.

Original: [Chung and Glass, 2018] propose anunsupervised method to learn speech embeddings using a ﬁxed context of 10 words in the past and future.
Revised: [Chung and Glass, 2018] propose anunsupervised method to learn speech embeddings using a ﬁxed context of 10 words in the past andfuture.

Original: The drawback of their method is that they need to obtain forced alignment between speechand words again restricting their input to a single speech fragment.
Revised: The drawback of their method is that they need to obtain forced alignment between speechand words again restricting their input to a single speech fragment.

Original: This is the common practicein most prior work that simpliﬁes training but prevents the models to scale to learning contextualsentence-level acoustic word embeddings.
Revised: This is the common practice inmost prior work that simpliﬁes training but prevents the models to scale to learn contextual wordembeddings grounded in utterance level acoustics.

Original: Learning text-based word embeddings is a rich area of research with well established techniquessuch as [Mikolov et al., 2013, Pennington et al., 2014].
Revised: Learning text-based word embeddings is a rich area of research with well established techniquessuch as [Mikolov et al., 2013, Pennington et al., 2014].

Original: Research has further progressed into learningcontextual word embeddings [McCann et al., 2017] or sentence-level word embeddings [Peterset al., 2018] that are useful in many text-based downstream tasks as detailed in [Conneau and Kiela,2018].
Revised: Research has further progressed into learning contextualized word embeddings [McCann et al., 2017, Peters et al., 2018] that are useful in manytext-based downstream tasks as detailed in [Conneau and Kiela, 2018].

Original: [McCann et al., 2017] learns contextual word embeddings from a fully trained machinetranslation model and depict re-use of their encoder in other downstream tasks.
Revised: [McCann et al., 2017] learnscontextual word embeddings from a fully trained machine translation model and depict re-use oftheir encoder in other downstream tasks.

Original: Our work ties A2Wspeech recognition model with learning contextual word embeddings from speech in a similar way as[McCann et al., 2017].
Revised: Our work ties A2W speech recognition model with learningcontextual word embeddings from speech.

Original:  Acoustic-to-Word Speech Recognition  Model
Revised:  Acoustic-to-Word Speech Recognition Model3.1 Sequence-to-Sequence Model

Original: [Figure] 3.1 Sequence-to-Sequence Model

Original: Our S2S model is similar in structure to the Listen, Attend and Spell model [Chan et al.
Revised: Our S2S model is similar in structure to the Listen, Attend and Spell model [Chan et al.

Original: , 2016a]which consists of 3 components: the encoder network, a decoder network and an attention model.
Revised: , 2016a]which consists of 3 components: the encoder network, a decoder network and an attention model.

Original: Theencoder maps the input acoustic features vectors x = ( x 1 , x 2 , ..., x T ) where x i ∈ R d , into a sequenceof higher-level features h
Revised: Theencoder maps the input acoustic features vectors x = ( x 1 , x 2 , ..., x T ) where x i ∈ R d , into a sequenceof higher-level features h

Original: = ( h 1 , h 2 , ..., h T (cid:48) ) .
Revised: = ( h 1 , h 2 , ..., h T (cid:48) ) .

Original: The encoder is a multi-layer bi-directional Long Short
Revised: The encoder is a multi-layer bi-directional Long Short

Original: Term Memory (BLSTM) RNN that is structured as a pyramid by skipping every other frame betweencertain encoder layers for efﬁcient training.
Revised: Term Memory (BLSTM) RNN that is structured as a pyramid by skipping every other frame betweencertain encoder layers for efﬁcient training.

Original: This reduces the length of the input from T to T (cid:48) .
Revised: This reduces the length of the input from T to T (cid:48) .

Original: Thisencoder network is analogous to the traditional acoustic model of an ASR.
Revised: Thisencoder network is analogous to the traditional acoustic model of an ASR.

Original: The decoder network isalso an LSTM network that learns to model the output distribution over the next target conditionedon sequence of previous predictions i.e. P ( y l | y ∗ l − 1 , y ∗ l − 2 , ..., y ∗ 0 , x ) where y ∗ =
Revised: The decoder network isalso an LSTM network that learns to model the output distribution over the next target conditionedon sequence of previous predictions i.e. P ( y l | y ∗ l − 1 , y ∗ l − 2 , ..., y ∗ 0 , x ) where y ∗ =

Original: ( y ∗ 0 , y ∗ 1 , ..., y ∗ L +1 )
Revised: ( y ∗ 0 , y ∗ 1 , ..., y ∗ L +1 )

Original: isthe ground-truth label sequence.
Revised: isthe ground-truth label sequence.

Original: In this work, y ∗ i ∈ U can be a token from a character, sub-wordor word vocabulary.
Revised: In this work, y ∗ i ∈ U can be a token from a character, sub-wordor word vocabulary.

Original: This decoder network is similar to the language model in traditional ASR as itgenerates targets y from h using an attention mechanism.
Revised: This decoder network is similar to the language model in traditional ASR as itgenerates targets y from h using an attention mechanism.

Original: The attention model learns an alignmentweight vector between the encoding h and the current output of decoder y l .
Revised: The attention model learns an alignmentweight vector between the encoding h and the current output of decoder y l .

Original: At each time step, theattention module computes a context vector that is fed into the decoder together with the previousground-truth label y ∗ l − 1 .
Revised: At each time step, theattention module computes a context vector that is fed into the decoder together with the previousground-truth label y ∗ l − 1 .

Original: Location-aware Attention
Revised: Location-aware Attention

Original: We use a location-aware attention mechanism [Chorowski et al., 2015]that enforces monotonicity in the alignments, which may be beneﬁcial for speech recognition.
Revised: We use a location-aware attention mechanism [Chorowski et al., 2015]that enforces monotonicity in the alignments, which may be beneﬁcial for speech recognition.

Original: To doso, the location-aware attention applies a convolution across time to the attention of previous timestep using trainable ﬁlters.
Revised: To doso, the location-aware attention applies a convolution across time to the attention of previous timestep using trainable ﬁlters.

Original: This convolved attention feature is used for calculating the attention forthe current time step.
Revised: This convolved attention feature is used for calculating the attention forthe current time step.

Original: We apply a one-dimensional convolution K along the input feature axis t to geta set of T features { f } Tt =1 described as follows:
Revised: We apply a one-dimensional convolution K along the input feature axis t to geta set of T features { f } Tt =1 described as follows:

Original: [Equation]
Revised: [Equation]

Original: where a l − 1 =
Revised: where a l − 1 =

Original: [ a l − 1 , 1 , ..., a l − 1 ,T ] T , g is a learnable vector parameter, { e lt } Tt =1 is a T -dimensionalvector, Lin() is a linear layer with learnable matrix parameters without bias vectors, LinB() is a linearlayer with learnable matrix and bias parameters.
Revised: [ a l − 1 , 1 , ..., a l − 1 ,T ] T , g is a learnable vector parameter, { e lt } Tt =1 is a T -dimensionalvector, Lin() is a linear layer with learnable matrix parameters without bias vectors, LinB() is a linearlayer with learnable matrix and bias parameters.

Original: The S2S model is trained by optimizing the cross entropy loss function which maximizes the loglikelihood of the training data.
Revised: The S2S model is trained by optimizing the cross entropy loss function which maximizes the loglikelihood of the training data.

Original: We use beam search to perform inference.
Revised: We use beam search to perform inference.

Original: We also apply unigramlabel smoothing that distributes the probability of most-probable token to prevent the over-conﬁdenceof the model [Pereyra et al., 2017, Chorowski and Jaitly, 2016].
Revised: We also apply unigramlabel smoothing that distributes the probability of most-probable token to prevent the over-conﬁdenceof the model [Pereyra et al., 2017, Chorowski and Jaitly, 2016].

Original: 3.2 Experimental Setup
Revised: 3.2 Experimental Setup

Original: Datasets We use a commonly used speech recognition setup, the 300 hour Switchboard corpus(LDC97S62)
Revised: Datasets We use a commonly used speech recognition setup, the 300 hour Switchboard corpus(LDC97S62)

Original: [Godfrey et al., 1992] which consists of 2,430 two-sided telephonic conversationsbetween 500 different speakers and contains 3 million words of text.
Revised: [Godfrey et al., 1992] which consists of 2,430 two-sided telephonic conversationsbetween 500 different speakers and contains 3 million words of text.

Original: We evaluate on the [-Eval
 [Table] (LDC2002S09,-] LDC2002T43) set, which includes the Switchboard (SW) subset similar to trainingdata, and the “harder” CallHome (CH) subset.
Revised: We evaluate on the [-Eval
 [Table] (LDC2002S09,-]Eval2000(LDC2002S09, LDC2002T43) set, which includes the Switchboard (SW) subset similar to trainingdata, and the “harder” CallHome (CH) subset.

Original: There are 196,656 total utterances out of which weuse the ﬁrst 4,000 utterances as a validation set, as is commonly done in the Kaldi recipe [Poveyet al., 2011].
Revised: There are 196,656 total utterances out of which weuse the ﬁrst 4,000 utterances as a validation set, as is commonly done in the Kaldi recipe [Poveyet al., 2011].

Original: Our second dataset is a 300 hour subset of the How2 [Sanabria et al., 2018]
Revised: Our second dataset is a 300 hour subset of the How2 [Sanabria et al., 2018]

Original: dataset ofinstructional videos, which contains planned, but free speech, often outdoor and recorded with distantmicrophones, as opposed to the indoor, telephony, conversational speech of Switchboard.
Revised: dataset ofinstructional videos, which contains planned, but free speech, often outdoor and recorded with distantmicrophones, as opposed to the indoor, telephony, conversational speech of Switchboard.

Original: There are13,662 videos in total leading to 184,286 utterances for training 2019 for validation and 2361 fortesting.
Revised: There are13,662 videos in total leading to 184,286 utterances for training, 2019 for validation and 2361 fortesting.

Original: There are a total of 3.5 million words (tokens) in this corpus.
Revised: There are a total of 3.5 million words (tokens) in this corpus.

Revised: [Table]

Original: Our input features are 80-dimensional log-mel ﬁlter banks normalized with per-speaker mean andvariance.
Revised: Our input features are 80-dimensional log-mel ﬁlter banks normalized with per-speaker mean andvariance.

Original: We also use 3-dimensional pitch features [Ghahremani et al., 2014].
Revised: We also use 3-dimensional pitch features [Ghahremani et al., 2014].

Original: For Switchboard, weframe a large vocabulary task by including all the words in the dataset leading to a vocabulary of29,874 words.
Revised: For Switchboard, weframe a large vocabulary task by including all the words in the dataset leading to a vocabulary of29,874 words.

Original: In How2, as the dataset is domain speciﬁc, we restrict the vocabulary to words thatoccur at least 5 times, resulting in 14,242 types.
Revised: In How2, as the dataset is domain speciﬁc, we restrict the vocabulary to words thatoccur at least 5 times, resulting in 14,242 types.

Original: Model Architecture Our implementation is similar to the architecture described in [Palaskar and
Revised: Model Architecture Our implementation is similar to the architecture described in [Palaskar and

Original: Metze, 2018].
Revised: Metze, 2018].

Original: The encoder consists of 6 layers each with 300 bi-directional LSTM cells (150 in eachdirection).
Revised: The encoder consists of 6 layers each with 300 bi-directional LSTM cells (150 in eachdirection).

Original: The encoder is a pyramidal or subsampling encoder that skips every other frame in ﬁrst 2layers to get a reduction of T/ 4 in input frames.
Revised: The encoder is a pyramidal or subsampling encoder that skips every other frame in ﬁrst 2layers to get a reduction of T/ 4 in input frames.

Original: We use the AdaDelta [Zeiler, 2012] optimizer.
Revised: We use the AdaDelta [Zeiler, 2012] optimizer.

Original: Thelocation-aware attention convolution uses 10 ﬁlters with width 100.
Revised: Thelocation-aware attention convolution uses 10 ﬁlters with width 100.

Original: There is a projection layer of300 dimensions after each layer of the encoder.
Revised: There is a projection layer of300 dimensions after each layer of the encoder.

Original: The decoder is a single layer LSTM containing 300cells.
Revised: The decoder is a single layer LSTM containing 300cells.

Original: We initialize all parameters uniformly within [ − 0 .
Revised: We initialize all parameters uniformly within [ − 0 .

Original: and use unigram label smoothing withweight 0.05.
Revised: and use unigram label smoothing withweight 0.05.

Original: The beam size used for all experiments is 10 and the length penalty in decoding is set to0.1.
Revised: The beam size used for all experiments is 10 and the length penalty in decoding is set to0.1.

Original: We use the ESPnet toolkit [Kim et al., 2017, Watanabe et al., 2017] for our experiments.
Revised: We use the ESPnet toolkit [Kim et al., 2017, Watanabe et al., 2017] for our experiments.

Original: 3.3 Results
Revised: 3.3 Results

Original: Our S2S A2W model is trained on large vocabulary whole-word targets.
Revised: Our S2S A2W model is trained on large vocabulary whole-word targets.

Original: To restrict the scope of thispaper to learning acoustic word embeddings, we do not compare our model with other A2W modelstrained using Connectionist Temporal Classiﬁcation [Graves et al., 2006] or Byte-Pair Units [Zeyeret al., 2018].
Revised: To restrict the scope of thispaper to learning acoustic word embeddings, we do not compare our model with other A2W modelstrained using Connectionist Temporal Classiﬁcation [Graves et al., 2006] or Byte-Pair Units [Zeyeret al., 2018].

Original: While those techniques may perform well for speech-to-text tasks, they cannot be usedfor learning acoustic word embeddings, whereas S2S can.
Revised: While those techniques may perform well for speech-to-text tasks, they cannot be usedfor learning acoustic word embeddings, whereas S2S can.

Original: Table 1 shows that we closely match theperformance of other A2W models on the Switchboard task, proving the strength of our model forspeech recognition.
Revised: Table 1 shows that we closely match theperformance of other A2W models on the Switchboard task, proving the strength of our model forspeech recognition.

Original:  Learning Contextual Acoustic Word Embeddings (CAWE)
Revised:  Learning Contextual Acoustic Word Embeddings (CAWE)

Original: We now describe our method to obtain the acoustic word embeddings from the end-to-end trainedspeech recognition system described in Section 3.
Revised: We now describe our method to obtain the acoustic word embeddings from the end-to-end trainedspeech recognition system described in Section 3.

Original: The model is as shown in Figure 1 where theembeddings are constructed using the hidden representations obtained from the encoder and theattention weights from the decoder.
Revised: The model is as shown in Figure 1 where theembeddings are constructed using the hidden representations obtained from the encoder and theattention weights from the decoder.

Original: The main challenge that separates our method from CoVe[McCann et al., 2017] is the problem of alignment between input speech and output words.
Revised: The main challenge that separates our method from CoVe[McCann et al., 2017] is the problem of alignment between input speech and output words.

Original: Weuse the location-aware attention mechanism that has the property to assign higher probability tocertain frames leading to a peaky attention distribution.
Revised: Weuse the location-aware attention mechanism that has the property to assign higher probability tocertain frames leading to a peaky attention distribution.

Original: We exploit this property of location-awareattention in an A2W model to automatically segment continuous speech into words as shown by
Revised: We exploit this property of location-awareattention in an A2W model to automatically segment continuous speech into words as shown by[Palaskar and Metze, 2018], and then using this segmentation to obtain word embeddings. In the nexttwo subsections, we formalize this process of constructing contextual acoustic word embeddings. Intuitively, attention weights on the acoustic frames hidden representations reﬂect their importancein classifying a particular word. They thereby provide a correspondance between the frame and theword within a given acoustic context.

Revised: We can thus construct word representations by weighing thehidden representations of these acoustic frames in terms of their importance to the word i.e. theattention weight.

Revised: We show this in the Figure 1 wherein the hidden representations and their attentionweights are colored according to their correspondance with a particular word.

Original: AttentionDecoder
Revised: AttentionDecoder

Original: Vocabulary ...
Revised: Vocabulary ...

Original: Pyramidal BiLSTM
Revised: Pyramidal BiLSTM

Original: Encoder
Revised: Encoder

Original: Speech
Revised: Speech

Original: CAWE
Revised: CAWE

Original: W
Revised: W

Original: W 3 W
Revised: W 3 W

Original: W
Revised: W

Original: WW [Palaskar and Metze, 2018], and then using this segmentation to obtain word embeddings. In the nexttwo subsections, we formalize this process of constructing contextual acoustic word embeddings.
Revised: W

Original: Intuitively, attention weights on the acoustic frames hidden representations reﬂect their importancein classifying a particular word. They thereby provide a correspondance between the frame and theword within a given acoustic context. We can thus construct word representations by weighing thehidden representations of these acoustic frames in terms of their importance to the word i.e. theattention weight. We show this in the Figure 1 wherein the hidden representations and their attentionweights are colored according to their correspondance with a particular word.
Revised: W

Original: 4.1 Automatic Segmentation of Speech into Words
Revised: 4.1 Automatic Segmentation of Speech into Words

Original: Given that a j represents the acoustic frame j , let encoder ( a j ) represent the higher-level features obtained for the frame a j ( i.e. a j = h
Revised: Given that a j represents the acoustic frame j , let encoder ( a j ) represent the higher-level features obtained for the frame a j ( i.e. a j = h

Original: = ( h 1 , h 2 , ..., h T (cid:48) ) , as explained in Section 3.1).
Revised: = ( h 1 , h 2 , ..., h T (cid:48) ) , as explained in Section 3.1).

Original: Then, for the i th word w i our model ﬁrst obtains the mappings of w i to acoustic frames a K where K is the set such that∀
Revised: Then, for the i th word w i our model ﬁrst obtains the mappings of w i to acoustic frames a K where K is the set such that

Original: [Equation]
Revised: [Equation]

Original: over all utterances U containing the word w i in the training set. 4.2 Investigating the Utility of Attention Distribution for Constructing Embeddings: CAWE
Revised: over all utterances U containing the word w i in the training set.4.2 Investigating the Utility of Attention Distribution for Constructing Embeddings: CAWE

Original: Below we describe three different ways of using attention to obtain acoustic word embeddings for aword w i that occurs n ( K ) times
Revised: Below we describe three different ways of using attention to obtain acoustic word embeddings for aword w i (here, n ( K ) represents the cardinality of the set K ):

Original: Unweighted Average (U-AVG): w i =
Revised: Unweighted Average (U-AVG): w i =

Original: [Equation] k encoder ( a k ) n ( K )
Revised: (cid:80) k encoder ( a k ) n ( K )

Original: Attention Weighted Average (CAWE-W): w i =
Revised: Attention Weighted Average (CAWE-W): w i =

Original: [Equation] k attention ( a k [Equation]
Revised: (cid:80) k attention ( a k ) · encoder ( a k ) n ( K )

Original: Maximum Attention (CAWE- M): w i = encoder ( argmax k ∈ K ( a k ))
Revised: Maximum Attention (CAWE-M): w i = encoder ( argmax k ∈ K ( a k ))

Original: Therefore, unweighted Average (U-AVG) is just the unweighted combination of all the hiddenrepresentations of acoustic frames mapped to a particular word.
Revised: Therefore, unweighted Average (U-AVG) is just the unweighted combination of all the hiddenrepresentations of acoustic frames mapped to a particular word.

Original: Attention weighted Average (CAWEW) is the weighted average of the hidden representations of all acoustic frames using the attentionweights for a given word.
Revised: Attention weighted Average (CAWEW) is the weighted average of the hidden representations of all acoustic frames using the attentionweights for a given word.

Original: Finally, maximum attention (CAWE-M) is the hidden representation ofthe acoustic frame with the highest attention score for a given word across all utterances in thetraining data.
Revised: Finally, maximum attention (CAWE-M) is the hidden representation ofthe acoustic frame with the highest attention score for a given word across all utterances in thetraining data.

Original: We call the attention-weighted average and the maximum attention based representationtechnique as Contextual Acoustic Word Embeddings (CAWE) since they are highly contextual owingto the use of attention scores (over all acoustic frames for a given word) in their construction.[Table]
Revised: We call the attention-weighted average and the maximum attention based representationtechnique as Contextual Acoustic Word Embeddings (CAWE) since they are highly contextual owingto the use of attention scores (over all acoustic frames for a given word) in their construction.

Original:  Experiments and Results
Revised:  Experiments and Results

Original: 5.1 Comparing Methods for Constructing Embeddings
Revised: 5.1 Comparing Methods for Constructing Embeddings

Original: Datasets for Downstream Tasks We evaluate our embeddings by using them as features for 16benchmark sentence evaluation tasks that cover Semantic Textual Similarity (STS 2012-2016 and
Revised: Datasets for Downstream Tasks We evaluate our embeddings by using them as features for 16benchmark sentence evaluation tasks that cover Semantic Textual Similarity (STS 2012-2016 and

Original: STS B), binary and multi-class classiﬁcation: Movie Review (MR), product review (CJ), sentimentanalysis (SST, SST-FG), question type (TREC), Subjectivity/Objectivity (SUBJ), and opinion polarity(MPQA), entailment and semantic relatedness using the SICK dataset for SICK-E (entailment)and SICK-R (relatedness) and paraphrase detection (MRPC).
Revised: STS B), binary and multi-class classiﬁcation: Movie Review (MR), product review (CJ), sentimentanalysis (SST, SST-FG), question type (TREC), Subjectivity/Objectivity (SUBJ), and opinion polarity(MPQA), entailment and semantic relatedness using the SICK dataset for SICK-E (entailment) [Table] SICK [Table]

Revised: and SICK-R (relatedness) and paraphrase detection (MRPC).

Original: The STS and SICK-R tasks measure
Revised: The STS and SICK-R tasks measure

Original: Spearman’s coefﬁcient of correlation between embedding based similarity and human scores, hencethe scores range from [ − 1 , 1] where higher number denotes high correlation.
Revised: Spearman’s coefﬁcient of correlation between embedding based similarity and human scores, hencethe scores range from [ − 1 , 1] where higher number denotes high correlation.

Original: All the remaining tasksare measured on test classiﬁcation accuracies.
Revised: All the remaining tasksare measured on test classiﬁcation accuracies.

Original: We use the SentEval toolkit [Conneau and Kiela,2018] 1 to evaluate.
Revised: We use the SentEval toolkit [Conneau and Kiela,2018] 1 to evaluate.

Original: Training Details In all downstream evaluations involving classiﬁcation tasks, we have used thesimplest model i.e. logistic regression for classiﬁcation since a better representation should lead tobetter scores without using complicated models (hence abstracting away model complexities fromour evaluations).
Revised: Training Details In all downstream evaluations involving classiﬁcation tasks, we have used thesimplest model i.e. logistic regression for classiﬁcation since a better representation should lead tobetter scores without using complicated models (hence abstracting away model complexities fromour evaluations).

Original: This also means that we can use the concatenation of CAWE and CBOW as featuresto the logistic regression model without adding tunable embedding parameters.
Revised: This also means that we can use the concatenation of CAWE and CBOW as featuresto the logistic regression model without adding tunable embedding parameters.

Original: Discussion From the results in Table 2 we see that CAWE-M outperforms U-AVG and CAWE-Wby a large margin on STS and SICK-R tasks and leads to better or slightly worse performance on theclassiﬁcation tasks.
Revised: Discussion From the results in Table 2 we see that CAWE-M outperforms U-AVG and CAWE-Wby a large margin on STS and SICK-R tasks and leads to better or slightly worse performance on theclassiﬁcation tasks.

Original: We observe that CAWE-W usually performs worse than CAWE-M which couldbe attributed to a noisy estimation of the word embeddings on the account of taking even the lessconﬁdent attention scores while constructing the embedding.
Revised: We observe that CAWE-W usually performs worse than CAWE-M which couldbe attributed to a noisy estimation of the word embeddings on the account of taking even the lessconﬁdent attention scores while constructing the embedding.

Original: In constrast, CAWE-M is constructedusing the most conﬁdent attention score obtained over all the occurrences of the acoustic framescorresponding to a particular word.
Revised: In constrast, CAWE-M is constructedusing the most conﬁdent attention score obtained over all the occurrences of the acoustic framescorresponding to a particular word.

Original: We also observe that U-AVG performs worse than CAWE-Won STS and SICK-R tasks since it is constructed using an even noisier process in which all encoder [Table] TREC 67. SICK-E 74.
Revised: We also observe that U-AVG performs worse than CAWE-Won STS and SICK-R tasks since it is constructed using an even noisier process in which all encoderhidden representations are weighted equally irrespective of their attention scores.

Original: 5.2 Comparing with Text-based Embeddings
Revised: 5.2 Comparing with Text-based Embeddings

Original: Datasets for Downstream Tasks The datasets are the same as described in Section 5.1.
Revised: Datasets for Downstream Tasks The datasets are the same as described in Section 5.1.

Original: Training Details In all the following comparisons, we compare embeddings obtained only from thetraining set of the speech recognition model, while the text-based word embeddings are obtained bytraining Continuous Bag-of-Words (CBOW) word2vec model on all the transcripts (train, validationand test).
Revised: Training Details In all the following comparisons, we compare embeddings obtained only from thetraining set of the speech recognition model, while the text-based word embeddings are obtained bytraining Continuous Bag-of-Words (CBOW) word2vec model on all the transcripts (train, validationand test).

Original: This was done to ensure a fair comparison between our supervised technique and theunsupervised word2vec method.
Revised: This was done to ensure a fair comparison between our supervised technique and theunsupervised word2vec method.

Original: This naturally leads to a smaller vocabulary for CAWE.
Revised: This naturally leads to a smaller vocabulary for CAWE.

Original: Further,one of the drawbacks of A2W speech recognition model is that it fails to capture entire vocabulary,recognizing only 3044 words out of 29874 (out of which 18800 words occur less than 5 times)and 4287 out of 14242 total vocabulary for Switchboard and How2 respectively.
Revised: Further,one of the drawbacks of A2W speech recognition model is that it fails to capture entire vocabulary,recognizing only 3044 words out of 29874 (out of which 18800 words occur less than 5 times) [Table] [Table] [Table]

Revised: [Table]

Revised: TREC

Original: Despite this fact,the performance of CAWE is very competitive with word2vec CBOW which does not suffer fromreduced vocabulary problem.
Revised: [Table] [Table] [Table] [Table]

Revised: [Table]

Revised: SICK

Revised: [Table]

Revised: E

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: and 4287 out of 14242 total vocabulary for Switchboard and How2 respectively.

Revised: Despite this fact,the performance of CAWE is very competitive with word2vec CBOW which does not suffer fromreduced vocabulary problem.

Original: Discussion From Table 3, it is clear that our embeddings perform as well as the text-embeddings.
Revised: Discussion From Table 3, it is clear that our embeddings perform as well as the text-embeddings.

Original: Evaluations using CAWE-M extracted from Switchboard based training show that the acousticembeddings when concatenated with the text embeddings outperform the word2vec embeddings on10 out of 16 tasks.
Revised: Evaluations using CAWE-M extracted from Switchboard based training show that the acousticembeddings when concatenated with the text embeddings outperform the word2vec embeddings on10 out of 16 tasks.

Original: This concatenated embedding shows that we add more information with CAWE-Mthat improves the CBOW embedding as well.
Revised: This concatenated embedding shows that we add more information with CAWE-Mthat improves the CBOW embedding as well.

Original: The gains are more prominent in Switchboard ascompared to the How2 dataset since How2 is planned instructional speech whereas Switchboard isspontaneous conversational speech (thereby making the How2 characteristics closer to text leading toa stronger CBOW model).
Revised: The gains are more prominent in Switchboard ascompared to the How2 dataset since How2 is planned instructional speech whereas Switchboard isspontaneous conversational speech (thereby making the How2 characteristics closer to text leading toa stronger CBOW model).

Original: 5.3 Evaluation on Spoken Language Understanding
Revised: 5.3 Evaluation on Spoken Language Understanding

Original: Dataset In addition to generic sentence-level evaluations, we also evaluate CAWE on the widelyused [Tur et al., 2010]
Revised: Dataset In addition to generic sentence-level evaluations, we also evaluate CAWE on the widelyused [Tur et al., 2010]

Original: ATIS dataset [Price, 1990] for Spoken Language Understanding (SLU).
Revised: ATIS dataset [Price, 1990] for Spoken Language Understanding (SLU).

Original: ATISdataset [Price, 1990] is comprised of spoken language queries for airline reservations that have intentand named entities.
Revised: ATISdataset [Price, 1990] is comprised of spoken language queries for airline reservations that have intentand named entities.

Original: Hence, it is similar in domain to Switchboard, making it a useful test bed forevaluating CAWE on a speech-based downstream evaluation task.
Revised: Hence, it is similar in domain to Switchboard, making it a useful test bed forevaluating CAWE on a speech-based downstream evaluation task.

Original: Training Details For this task, our model is similar to the simple Recurrent Neural Network (RNN)based model architecture as investigated in [Mesnil et al., 2013].
Revised: Training Details For this task, our model is similar to the simple Recurrent Neural Network (RNN)based model architecture as investigated in [Mesnil et al., 2013].

Original: Our architecture is comprised of an embedding layer, a single layer RNN-variant (Simple RNN, Gated Recurrent Unit (GRU) [Chunget al., 2014]) along with a dense layer and softmax.
Revised: Our architecture is comprised of anembedding layer, a single layer RNN-variant (Simple RNN, Gated Recurrent Unit (GRU) [Chunget al., 2014]) along with a dense layer and softmax.

Original: In each instance, we train our model for 10epochs with RMSProp (learning rate 0.001).
Revised: In each instance, we train our model for 10epochs with RMSProp (learning rate 0.001).

Original: We train each model 3 times with different seed valuesand report average performance.
Revised: We train each model 3 times with different seed valuesand report average performance.

Original: [Table] Discussion [Mesnil et al., 2013] concluded that text-based word embeddings trained on large textcorpora consistently lead to better performance on the ATIS dataset.
Revised: Discussion [Mesnil et al., 2013] concluded that text-based word embeddings trained on large textcorpora consistently lead to better performance on the ATIS dataset.

Original: We demonstrate that directspeech-based word embeddings could lead to matching performance when compared to text-basedword embeddings in this speech-based downstream task, thus highlighting the utility of our speechbased embeddings.
Revised: We demonstrate that directspeech-based word embeddings could lead to matching performance when compared to text-basedword embeddings in this speech-based downstream task, thus highlighting the utility of our speechbased embeddings.

Original: Speciﬁcally, we compare the test scores obtained by initializing the model with
Revised: Speciﬁcally, we compare the test scores obtained by initializing the model with

Original: CAWE-M, CAWE-W and CBOW embeddings and ﬁne-tuning them based on the task.
Revised: CAWE-M, CAWE-W and CBOW embeddings and ﬁne-tuning them based on the task.

Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: We presented a method to learn contextual acoustic word embeddings from an end-to-end trainedacoustic-to-word speech recognition model that learns to jointly segment and classify speech intoindividual words.
Revised: We presented a method to learn contextual acoustic word embeddings from an end-to-end trainedacoustic-to-word speech recognition model that learns to jointly segment and classify speech intoindividual words.

Original: Using this property of automatic segmentation, we present different methods tolearn acoustic word embeddings from a speech recognition model.
Revised: Using this property of automatic segmentation, we present different methods tolearn acoustic word embeddings from a speech recognition model.

Original: We demonstrate the utility ofattention in constructing contextual acoustic word embeddings which are highly competitive withword2vec (CBOW) text embeddings.
Revised: We demonstrate the utility ofattention in constructing contextual acoustic word embeddings which are highly competitive withword2vec (CBOW) text embeddings.

Original: We presented two variants of such contextual acoustic wordembeddings which outperform the simple unweighted average method.
Revised: We presented two variants of such contextual acoustic wordembeddings which outperform the simple unweighted average method.

Original: The embeddings also matchedthe performance of text embeddings in a speech-based downstream spoken language understandingtask showing that we can use this model as a pretrained model for other tasks requiring speechembeddings.
Revised: The embeddings also matchedthe performance of text embeddings in a speech-based downstream spoken language understandingtask showing that we can use this model as a pretrained model for other tasks requiring speechembeddings.

Original: This is particularly challenging for speech since unlike machine translation, which onlyhas to learn alignments between words, we have to additionally learn alignments between multipleacoustic frames and words.
Revised: This is particularly challenging for speech since unlike machine translation, which onlyhas to learn alignments between words, we have to additionally learn alignments between multipleacoustic frames and words.

Original: Despite the additional challenges, our demonstration of utility wordembeddings obtained directly from a supervised speech recognition model can serve as a precursor tospeech recognition models being extensively used for downstream tasks like CoVe in natural languageprocessing and VGG in computer vision.
Revised: Despite the additional challenges, our demonstration of utility wordembeddings obtained directly from a supervised speech recognition model can serve as a precursor tospeech recognition models being extensively used for downstream tasks like CoVe in natural languageprocessing and VGG in computer vision.

Original: In the future, we will explore ways to scale our model tolarge corpora to generate contextual acoustic word embeddings for larger vocabularies
Revised: In the future, we will explore ways to scale our model tolarge corpora to generate contextual acoustic word embeddings for larger vocabularies.