From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper31/Authors
title: Response to AnonReviewer2
comment: Thank you for your review and suggestions. Regarding your comments:

“The technical content of the paper is correct. Nonetheless, it does not include any technical novelty. ”

- In this paper we focused on understanding established models better. We hope that this can inform the design of better models.

“The paper is clear and well written. The only remark concerns colors of Figure 1. Indeed, if printed in black-white, it is very hard to distinguish the combined and word-level curves on Figure 1. ”

- Thank you. We will update the figure to increase its readability.

“The only problem I can see with this work is that, based on the fact that it claims to provide a better understanding of the performances of this multi-task based model, i would like the paper to contains more experiments, more investigations, more insights, in fact, a paper with 8 pages of content for a deeper overview. ”

- The paper presents work in progress and we are working on more experiments and hope to publish an extended version at a later point.

From: NIPS.cc/2018/Workshop/IRASL/Paper31/Authors
title: Response to AnonReviewer3
comment: Thank you for your review and suggestions. Regarding your comments:

“For this reason, it would also be interesting how the discussed biases apply to larger data sets and how much influence they have when controlling for gains derived from dealing with oov words.”

- Thanks for the suggestion. We are currently looking into how the effects we observe for a relatively small dataset replicate on a larger one. We focused on a smaller dataset for now where these regularizing effects are even more important.

“The text could be more clear about which exact instantiations of MTL learning have been shown to work and which ones not in the literature review but otherwise the paper is easy to follow. ”

- We are happy to add further details to the related work section.

From: NIPS.cc/2018/Workshop/IRASL/Paper31/Authors
title: Response to AnonReviewer1
comment: Thank you for your review and suggestions. Regarding your comments:

“The paper’s main weakness is probably its scope. Although interesting, one is left wondering: what might be done with this understanding of inductive bias?”

- We agree that the paper can be easily extended and more interactions can be analyzed. From our point of view, a better understanding of the inductive biases can help us possibly design these, as you suggest in the remainder.

“The paper suggests follow-up questions. In equation (2), the authors introduce a hyper-parameter, “lambda”, which trades-off between the word and char losses used by the MTL model. In all of their experiments, the authors set lambda to 1. It might be interesting though to see if tweaking lambda would result in changes to the inductive bias. ”

- This is an excellent idea and we will look into this.

From: NIPS.cc/2018/Workshop/IRASL/Paper31/AnonReviewer1
title: Choosing a word, character, or multi-task based loss in an ASR model yields distinct inductive biases
review: This paper aims at understanding why recent ASR results that use multi-task learning (MTL) to combine both word and character-level losses can improve on word error rates (relative to models that include only a word or character-based loss; see references in paper). 

Fundamental to the paper is an assumption that models trained with SGD learn "easier" examples first (Arpit et al. 2017). So, understanding what the model learns first should reveal something about its biases. Following this line of reasoning, the authors plot how the distribution of words learned by word-level, character-level, and MTL models evolve in the early stages of training. They conclude the following:

(A) Word-based models learn high-frequency words first.
(B) Character-based models learn short words first.
(C) MTL models interpolate between (A) and (B).

Overall the paper is well written and contains sufficient detail for the study to be replicated. It also fits nicely within the theme of the workshop.

The paper's main weakness is probably its scope. Although interesting, one is left wondering: what might be done with this understanding of inductive bias?

Random thoughts:

The paper suggests follow-up questions. In equation (2), the authors introduce a hyper-parameter, "lambda", which trades-off between the word and char losses used by the MTL model. In all of their experiments, the authors set lambda to 1. It might be interesting though to see if tweaking lambda would result in changes to the inductive bias. 

Perhaps the obvious hypothesis is that varying lambda should result in MTL models that will still interpolate between the word- and char-level biases in (A) and (B), but with more emphasis on the "word-frequency" bias (A) if lambda<1, and more emphasis on the "word-length" bias (B) if lambda>1. 

A case where this level of control might be relevant is in models with a disproportionate amount of low-frequency words, such as the medical communication data that the authors mention. One could imagine using lambda as a prior to improve performance on smaller datasets like this. Or perhaps one could learn an optimal lambda as a way to characterise different datasets.
rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper31/AnonReviewer3
title: Somewhat obvious results and limited in scope but still worth knowing about
review: The authors investigate the properties of speech recognition systems which are trained to optimize for sequences of both word-level symbols and character-level symbols in a multi-task setup. An ablation study is carried out on WSJ with a CTC-based system. The results indicate that world-level training is biased towards the recognition of frequent words, while character-level training is biased towards the recognition of short words, while the multi-task training has a more uniform distribution over words. 

While the combination of character-level and word-level modelling is not new by itself, it is interesting to learn more about why it may be beneficial to use such a multi-task setup. The results in the paper are a bit limited in scope because they are on a small, clean data set. However, the choice for a small data set makes some sense to investigate how the addition of a character-based loss can help with the data scarcity at the word level. The notion that word-level training is biased towards frequent words while character-level training is biased towards shorter words is somewhat obvious. It is still worth verifying that training both objectives lead to a model that is somewhat in between these two extremes because of the features it learned. 

Normally, an emphasis on more frequent words should also lead to better WER scores unless the frequency spectrum between the train and test set differs substantially. This leaves one to wonder whether simpler regularization methods could be used which don’t require a character-level modelling module. Improved results on larger datasets are perhaps more due to the ability to deal with OOV words than to this frequency smoothing. For this reason, it would also be interesting how the discussed biases apply to larger data sets and how much influence they have when controlling for gains derived from dealing with oov words.

All in all, the results are limited in scope and somewhat predictable but perhaps still worth knowing about. The text could be more clear about which exact instantiations of MTL learning have been shown to work and which ones not in the literature review but otherwise the paper is easy to follow. The subject matter of the paper does fit the scope of the workshop quite well.

rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper31/AnonReviewer2
title: Good but short investigation of the inductive bias of Word-Character-Level Multi-Task Learning for Speech Recognition
review: Recent works have proposed to merge the word and character levels (multi-task learning) with the CTC function in a single neural model, to achieve better performances on speech recognition tasks. This paper proposes to investigate how the introduction of multi-task learning through the character level CTC helps word-based CTC models to perform better. It is shown that the character-level supervision allows the investigated model to interpolates between more frequent words (word-based) and shorter words(character-based), resulting in better and smoother generalization capabilities. 

Quality : 
The technical content of the paper is correct. Nonetheless, it does not include any technical novelty. 

Clarity : 
The paper is clear and well writen. The only remark concerns colors of Figure 1. Indeed, if printed in black-white, it is very hard to distinguish the combined and word-level curves on Figure 1.  

Originality : 
The investigated model is definitly not novel since many architectures already proposed multi-task learning at word and character level for speech recognition. However, this is a good investigation on « Why such models work better ? ». The only problem I can see with this work is that, based on the fact that it claims to provide a better understanding of the performances of this multi-task based model, i would like the paper to contains more experiments, more investigations, more insights, in fact, a paper with 8 pages of content for a deeper overview. 

Significance :
While the paper won’t have a huge impact. It fits particularly well to the the IRASL workshop with a better understanding of  « what is going on under the hood ». 

Pros :
+ Clearly connected to the IRASL workshop goals
+ Clear, well-written and self-contained

Cons :
- Such an investigation could have been extended to 8 pages for a deeper overview
- No technical novelty 




rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature