From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Reject

From: NIPS.cc/2018/Workshop/IRASL/Paper54/AnonReviewer3
title: reviewer feedback
comment: >>  It is not clear what means 'contextual' in 'contextual acoustic word embedding'.
>        The "contextual" term is used to explain an important distinction against the >previous acoustic word embedding techniques (and is similar in spirit to [1]). As explained in the introduction, since earlier techniques learn acoustic word embeddings using audio fragments and word pairs obtained via a word-level segmentation of the audio data, they only use a limited audio context based on word boundaries for deriving the embedding for a particular word, eg. 3 words in past and 3 words in future [4], not capturing the contextual dependencies in speech at an entire utterance-level. In contrast, our work constructs individual acoustic word embeddings using attention scores (without needing word boundaries), grounded in long-term utterance-level acoustics and does not need any hyper-parameters to define the context of a word. Further, speaker and acoustic variability will be reflected in the attention scores (eg. noisy utterance will have less peaky attention [3]) and our CAWE-W and CAWE-M methods make use of this information for constructing embeddings which prior work does not have access to. Please refer to line 189-191 in the manuscript as well.

I understand your point of view, you suggest that your acoustic word embedding capture high-level linguistic information. It should be interesting to compare the mapping between the h higher-level features taken into account to compute a word embeddings to the audio frames corresponding to this word (by using a forced alignments). I have the intuition that the audio frames targeted by the attention mechanism are included in the audio frames aligned to the word. And so the context you capture is a minor part coming from the word level history modeled in the bLSTM.
Your experiments show that your CAWE get better results that CBOW but some it should be interesting to compare also to 1-hot vector (since document classification takes also care of co-occurences) and character-based embeddings (I guess that acoustic word embeddings is able to capture some morphological features that helps to generalize, while the training corpus you use for CBOW is small finally).

Here are some papers that evaluates acoustic word embeddings. Usually, they don't try to compete to linguistic word embedding, but try to represent acoustic phenomena: [5] [6] [7].
It is true that the paper [4] you cite proposes also a comparison between their AWE and w2v embeddings, but I don't consider their experiments as convincing (they trained their w2v embedding only on the transcriptions of 250h of Librispeech): same remarks as the ones I wrote above.



>> Even if the comparison to the use of CBOW w2v seems to be at the advantage of the AWE, it seems like a non-sense to build CBOW embedding on very restricted text (one of the main advantages of such embeddings is to be able to be built through an unsupervised way on huge amount of textual data).
>        We used the Switchboard 300h dataset for training the A2W model, since it is a common benchmark in speech recognition and specifically for acoustic-to-word models.



>> In addition, it is not clear what is the impact of reducing the size of AWE vocabulary: maybe this is the cause of some side effects.
>        We did not reduce the size of the AWE vocabulary. The lesser vocabulary size of CAWE embeddings appears due to the fact that the acoustic-to-word model is trained without any word level segmentation, thereby incurring cases where some words do not get aligned.

I agree, but by nature of this construction, this implies that the AWE vocabulary is smaller than the CBOW one.


Whatever, even I am not convinced by the experiments you carried on in this paper, I consider the approach you propose to build acoustic word embeddings as very interesting. It should be relevant to compare it to other kinds of acoustic word embeddings, particularly to evaluate their capacity to represent acoustic phenomena in a fixed length vector. 


[5] Bengio, Samy, and Georg Heigold. "Word embeddings for speech recognition." Fifteenth Annual Conference of the International Speech Communication Association. 2014.

[6] Settle, Shane, et al. "Query-by-example search with discriminative neural acoustic word embeddings." arXiv preprint arXiv:1706.03818 (2017).

[7] Ghannay, S., Esteve, Y., Camelin, N., & Deléglise, P. (2016). Evaluation of acoustic word embeddings. In Proceedings of the 1st Workshop on Evaluating Vector-Space Representations for NLP (pp. 62-66).

From: NIPS.cc/2018/Workshop/IRASL/Paper54/Authors
title: Author response to reviewers comments
comment: Thanks for your feedback.

We request you to kindly consider the following remarks in the score for our paper:

>  It is not clear what means 'contextual' in 'contextual acoustic word embedding'.
        The "contextual" term is used to explain an important distinction against the previous acoustic word embedding techniques (and is similar in spirit to [1]). As explained in the introduction, since earlier techniques learn acoustic word embeddings using audio fragments and word pairs obtained via a word-level segmentation of the audio data, they only use a limited audio context based on word boundaries for deriving the embedding for a particular word, eg. 3 words in past and 3 words in future [4], not capturing the contextual dependencies in speech at an entire utterance-level. In contrast, our work constructs individual acoustic word embeddings using attention scores (without needing word boundaries), grounded in long-term utterance-level acoustics and does not need any hyper-parameters to define the context of a word. Further, speaker and acoustic variability will be reflected in the attention scores (eg. noisy utterance will have less peaky attention [3]) and our CAWE-W and CAWE-M methods make use of this information for constructing embeddings which prior work does not have access to. Please refer to line 189-191 in the manuscript as well.

> I am not convinced by the experiments while the way to build acoustic word embeddings is really very interesting. First, there are probably a lot of relevant tasks on which applying acoustic word embeddings (AWE): what is the justification of applying them on TEXTUAL document classification?
        Downstream evaluations have been the standard method for evaluating word embedding quality [2]. Applying the embeddings learned directly from speech recognition on the standard downstream embedding evaluation benchmarks is important to evaluate the quality of the "word embeddings". We have also evaluated our embedding on a spoken language understanding task, to demonstrate that speech-derived embeddings could be used to bypass the need for constructing word embeddings after converting speech to text in an application pipeline. We request you to let us know what speech-based tasks would be relevant for this evaluation in addition to Spoken Language Understanding.

> Even if the comparison to the use of CBOW w2v seems to be at the advantage of the AWE, it seems like a non-sense to build CBOW embedding on very restricted text (one of the main advantages of such embeddings is to be able to be built through an unsupervised way on huge amount of textual data).
        We used the Switchboard 300h dataset for training the A2W model, since it is a common benchmark in speech recognition and specifically for acoustic-to-word models. The word2vec CBOW embeddings were trained on the entire (test, train, and dev) corpus of the transcripts, with full vocabulary (not dropping words based on frequency), while the speech embeddings were constructed using attention scores, without any word segmentation information, only from the training set. We show that acoustic word embeddings are competitive with word2vec CBOW, despite the small vocabulary disadvantage. The reason we did not use word embeddings trained on large amounts of (external) textual corpora was to eliminate the impact of vocabulary differences, to ensure fair comparisons of the embedding techniques. Additionally, given that the concatenation of AWE and CBOW embeddings perform better, we show that speech and text are adding complementary information to the embeddings. 

> In addition, it is not clear what is the impact of reducing the size of AWE vocabulary: maybe this is the cause of some side effects.
        We did not reduce the size of the AWE vocabulary. The lesser vocabulary size of CAWE embeddings appears due to the fact that the acoustic-to-word model is trained without any word level segmentation, thereby incurring cases where some words do not get aligned.

Thank you very much for your feedback on our paper!

[1] McCann, Bryan, et al. "Learned in translation: Contextualized word vectors." Advances in Neural Information Processing Systems. 2017.

[2] Conneau, Alexis, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. "Supervised Learning of Universal Sentence Representations from Natural Language Inference Data." In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670-680. 2017.

[3] Palaskar, Shruti, and Florian Metze. "Acoustic-to-Word Recognition with Sequence-to-Sequence Models." arXiv preprint arXiv:1807.09597 (2018).

[4] Chung, Yu-An, and James Glass. "Speech2vec: A sequence-to-sequence framework for learning word embeddings from speech." arXiv preprint arXiv:1803.08976 (2018).

From: NIPS.cc/2018/Workshop/IRASL/Paper54/AnonReviewer3
title: A new way to build acoustic word embeddings, but not convincing experimental protocol
review: This paper proposes a new way to build acoustic word embeddings, and uses them to process classification on textual documents.  It is not clear what means 'contextual' in 'contextual acoustic word embedding'. 
I am not convinced by the experiments while the way to build acoustic word embeddings is really very interesting. 
First, there are probably a lot of relevant tasks on which applying acoustic word embeddings (AWE): what is the justification of applying them on TEXTUAL document classification? Even if the comparison to the use of CBOW w2v seems to be at the advantage of the AWE, it seems like a non-sense to build CBOW embedding on very restricted text (one of the main advantages of such embeddings is to be able to be built through an unsupervised way on huge amount of textual data). 
In addition, it is not clear what is the impact of reducing the size of AWE vocabulary: maybe this is the cause of some side effects.

rating: 2: Marginally below acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper54/Authors
title: Thank you for your feedback. Please consider the following responses to your review. 
comment: 
Dear Reviewer and Committee,

Thank you very much for your feedback on our paper. Please consider the following responses regarding the reviewers’ remarks:
 
> word targets don't make much sense on small data sets and WER performance is much worse than conventional baseline...On the other hand, the speech recognition experiments are pretty weak. Word units are an interesting exploration, but only if you have sufficient training data and has been done before. Building such models on SWB doesn’t make any sense, and the results in table 1 show this. But the author fail to compare against any conventional baseline. They state “Table 1 shows that we closely match the performance of other A2W models on the Switchboard task, proving the strength of our model for speech recognition.” This is very hard to accept given the error rates are 3 time higher than normal.

        The authors are fully aware that A2W models (indeed, most end-to-end models) do not nearly perform as well as a thoroughly optimized "hybrid" phoneme-based HMM model in terms of WER, certainly with hundreds of hours of training data. Their advantage is that they do not need a dictionary, time alignments, etc - and it is understood that results will always be better on larger datasets. We chose that setup to be able to show that our implementation is on par with the state of the art and recently published work doing the same.
         There seems to be confusion regarding the main contribution of the paper. We believe that the main contribution of this paper (submitted to IRASL) is that we present the first method to extract word embeddings directly from speech without requiring a segmentation to be given. We also show that these embeddings are highly competitive (and even complementary) to text-based word embeddings. We hope that our work will contribute to a better understanding of A2W models, which might make them more competitive with conventional systems in the future.
         In Table 1, for evaluation against comparable prior work, we compare our model with other A2W encoder-decoder models trained with large vocabulary whole word units (not phonemes, characters, or sub-words targets). We have replicated the model of Palaskar and Metze 2018 for this work in which there is a deeper exploration of A2W models with characters, sub-words and pure-words units, and theirs is the current state of the art result on the pure-word large vocabulary task. Our methods of learning acoustic “word embeddings” are possible only with such encoder-decoder models trained to predict “words” and not with conventional models or models trained on units other than pure-words.

>The interesting part of this work is section 5.2, where they compare against text based embeddings on sentence evaluation tasks but a lot of details are missing.

	We apologize for any missing details (probably the hyperparameters of the trained word embedding models) and would like to provide them in the next revision. Could you please let us know what more information would be helpful? 

> The authors also don’t properly cite original work, for example Bahdanau’s work is missing.
         
         We apologize for this oversight on our part. We will add the citation to Bahdanau’s paper. We have cited a number of speech recognition papers which make use of location-aware attention mechanism (Chorowski et al. 2015) which is a specialized type of attention for speech recognition to enforce monotonicity. We didn’t cite Bahdanau since it is a machine translation based work, without any speech based evaluations. However, we will add the citation, due to its historical significance in proposing attention.

We will include these clarifying remarks in the next revision of the paper and thank you once again for your feedback.

- Paper54 Authors

From: NIPS.cc/2018/Workshop/IRASL/Paper54/Authors
title: Thank you for your feedback. Please consider the following responses to your review.
comment: 
Dear Reviewer and Committee,

Thank you very much for your feedback on our paper. Please consider the following responses regarding the reviewers’ remarks:

> Since maximum attention based embeddings, the best-performing method, picks only one word representation whose attention value is the highest in the training set, the contextual information from other utterances that are not chosen seems to be discarded. So I think a reasonable rationale for its outperformance should be provided.

 	Our best embedding variant (CAWE-M, based on the maximum attention score) performs better (by 34% and 13%, relatively) on semantic textual similarity benchmarks than the unweighted average (U-AVG) and the weighted average variants (CAWE-W) respectively. We hypothesize that since the word to acoustic frame alignments derived using low attention scores themselves may not be correct (as indicated by the high word error rates of acoustic-to-word models in general), including the encoder representations for such frames amounts to adding noise. This is indicated by the poor performance of the unweighted average variant of the word embeddings (U-AVG). Even weighing such encoder representations by their corresponding attention scores does not lead to avoiding noise, as indicated by the lower performance of the attention weighted embeddings (CAWE-W). In contrast, CAWE-M suffers from no such noise and is based on the encoded representation of the model’s most confident acoustic frame for a particular word. We will include an analysis of interpolating between the maximum and weighted average embedding variants in the subsequent revision to further consolidate our findings.

> The results are too preliminary…..Also, the paper claims that the concatenation of speech- and text-based word embeddings performs the best, but the application where both inputs are available at the same time is not clear.

        We evaluated the acoustic word embeddings on 16 widely used standard downstream embedding evaluation benchmarks [1] as well as on a downstream Spoken Language Understanding task, where our embeddings learned only on speech data are highly competitive with text-based embeddings. Further, the evaluation of concatenated embeddings (speech + text) shows that the acoustic word embeddings complement the text-based word embeddings, leading to even better performance. The concatenation experiment does not indeed target any specific application, but it shows that there is complementary information in speech and text embeddings - which is relevant in and by itself. We will clarify this in the text.

We will include all these clarifying remarks in the next revision of the paper and also correct the minor typos you pointed out.

Thank you,
Paper54 Authors

[1] Conneau, Alexis, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. "Supervised Learning of Universal Sentence Representations from Natural Language Inference Data." In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 670-680. 2017.

From: NIPS.cc/2018/Workshop/IRASL/Paper54/AnonReviewer2
title: word targets don't make much sense on small data sets and WER performance is much worse than conventional baselines
review: The authors build a conventional encoder-decoder-attention model with word targets and use the learned word embeddings for other tasks.

The interesting part of this work is section 5.2, where they compare against text based embeddings on sentence evaluation tasks but a lot of details are missing.

On the other hand, the speech recognition experiments are pretty weak. Word units are an interesting exploration, but only if you have sufficient training data and has been done before.
Building such models on SWB doesn’t make any sense, and the results in table 1 show this.
But the author fail to compare against any conventional baseline. 

They state “Table 1 shows that we closely match the performance of other A2W models on the Switchboard task, proving the strength of our model for speech recognition.” This is very hard to accept given the error rates are 3 time higher than normal.

The authors also don’t properly cite original work, for example Bahdanau’s work is missing.


rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper54/AnonReviewer1
title: Learned in Speech Recognition: Contextual Acoustic Word Embeddings
review: 
The paper proposes a method to construct acoustic word embeddings, which are learned directly from audio. By training an encoder-decoder model with a location-aware attention mechanism, high-level representation for a sentence can be obtained. This sentence-level representation is segmented into word-level representation according to the attention value. Then, the word-level embeddings are obtained from word-level representation by three ways: unweighted average, attention weighted average, and maximum attention. Among these, maximum attention based embeddings shows competitive performance with text-based word embeddings, and the concatenation of both embeddings performs the best in several datasets.

- pros:
The main advantage of this method is that any predefined word boundary is not needed. It is an interesting idea that the speech can be split into words by using the attention value only, because the attention tends to be highly localized.

- cons:
The results are too preliminary. Since maximum attention based embeddings, the best-performing method, picks only one word representation whose attention value is the highest in the training set, the contextual information from other utterances that are not chosen seems to be discarded. So I think a reasonable rationale for its outperformance should be provided. Also, the paper claims that the concatenation of speech- and text-based word embeddings performs the best, but the application where both inputs are available at the same time is not clear.

- typos (minor):
  1) Since the length of the input is reduced from T to T’, I think T (line 116-117) should be T’.
  2) The acoustic frame ‘a’ (line 176) is confusable with the attention ‘a’ (line 117), because it says “as explained in Section 3.1” (line 176).
  3) (line 176) If a_j is the acoustic frame j, a_j cannot be h=(h_1, h_2, …, h_T’). Isn’t it h_j?
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct