From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper41/Authors
title: Author response
comment: Thank you for review and insightful comments. We will incorporate your suggestions in the extension of this article as full paper. 

EM and F1 measures could indeed be explained in the text. EM (Exact Match) measures whether the predicted start and end of answer matches exactly with the target start and end.

From: NIPS.cc/2018/Workshop/IRASL/Paper41/AnonReviewer3
title: Sensitivity of deep Question-Answering systems to perturbations in embedding and semantic space
review: The paper shows results to compare the stability of 3 deep QA models to various types of perturbations. This is done replacing the most salient word in the question as identified by LIME with either close neighbors in embedding space ("numeric") or wordnet ("semantic") or random. The 3 algorithms show greater stability to embedding space perturbations compared to semantic or random changes. The paper argues that this is undesirable. The paper proposes two techniques to address this problem : a) dropping random words from training, similar to dropout, and b) training embeddings along with the model. Both these technique improve the stability on semantic changes to some extent. However, they still have better stability on embedding-based perturbations.

Overall: This is an important problem worthy of investigation. Specifically, the sensitivity of neural models in NLP  has been less explored compared to their computer vision counterparts.

The paper could be improved by additional empirical studies to assess other factors that can help semantic stability: 
e.g.  
a) effect of more and diverse training data. This could include a training curve for in-domain data and examining the effect of out-of-domain data.
b) within the numerical perturbations, are there differences in stability for antonyms vs synonyms given that some of the neighbors in embedding space are actually antonyms ?

Typos:
'sensibility' -> 'sensitivity' ?
coeffiencts -> coefficients?

Table 1:
What does 'Original EM' mean? The later part of the paper mentions that this is a metric but it would be good to provide the expansion and also mention this upfront.


rating: 3: Marginally above acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper41/Authors
title: Author response
comment: Thank you for the review and comments. We are happy to hear that the paper was considered interesting overall.

We will have the paper checked by a professional proofreader before the final release.

"and GRAD (propagation of gradients to inputs) on 10.000 and 30.000 top words." - indeed, here we mean 10K and 30K most common words found in the training set. This wording will be corrected in the paper.

"(...) evidence that this is indeed the effect and not random fluctuations" - we provide scores on 4 improved models, and we assumed that most likely a random fluctuation would show in this setting if this was indeed the case. We will include more examples in the paper. Here we provide some cases of synonymic questions which were answered wrongly by the original model (baseline), while receiving correct answer from the improved models:

Original question: What is St. John's Cathedral an example of, stylistically?
Synonymic question: What is St. John's Cathedral an instance of, stylistically?

Original quesion: What do some sources say Nikola's brother died from?
Synonymic question: What do some sources say Nikola's brother perished from?

Original question: What band headlined half-time during Super Bowl 50?
Synonymic question: What musician headlined half-time during Super Bowl 50?

From: NIPS.cc/2018/Workshop/IRASL/Paper41/Authors
title: Author response
comment: Thank you for the review and comments. We are happy to hear that the work was considered valuable.

As to the cons:
- As this is short paper category, we restricted the problem to Q&A systems. We mean to extend the work with similar tests on other NLP tasks such as emotion recognition, possibly VQA systems, and also on more Q&A models.
- It is true that training embeddings at the same time as the task model is a common practice but we want to highlight that this solution increases robustness. We believe that each system feature should be looked at from 2 independent perspectives: system performance and system robustness. Our study suggests that even if a model with fine-tuned embeddings does not exhibit increase in accuracy (and indeed, fine-tuning of embeddings is omitted in many systems), this feature should still be considered an upgrade to robustness alone.

From: NIPS.cc/2018/Workshop/IRASL/Paper41/AnonReviewer2
title: A look at the potential effects of adversarial information on question answering accuracy
review: The manuscript addresses the problem of robustness to adversarial information in question answering. It investigates the robustness of 3 state-of-the-art Q&A models to attacks generated by swapping words with their synonyms, embedding neighbors, or random words. It then proposes two approaches to adversarial training which reduce the gap between synonyms and embedding neighbors. Finally, it makes available the relatively small number of carefully constructed test cases used in the study for further work on the matter.

- Pro's:

The paper is easy to follow and the basic conclusion of the study (models perform better with embedding neighbors than synonyms resulting in semantically correct questions), while expected, is still worth raising. In addition, the experimental results are competently set up and reasonably informative. 

- Con's:

I have two major reservations:
* the small scale of the study limits its applicability — making it hard in particular to determine the practical usefulness of the proposed enhancements across different use cases
* the proposed solution of training embeddings at the same time as the task model offers zero novel insight — this has been well-documented over the past few years across many setups and many different tasks.

rating: 3: Marginally above acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: NIPS.cc/2018/Workshop/IRASL/Paper41/AnonReviewer1
title: Improving robustness of machine comprehension/question answering
review: - Nice paper but it is filled with English mistakes -- I'm not going to enumerate them here (there's so many that it distracts from the presentation)

"Previous studies showed sensibility of the model to changes"
"and GRAD (propagation of gradients to inputs) on 10.000 and 30.000 top words." -- I think they meant 10K and 30K as sizes of vocabulary(?).

- Authors have nice ideas (leave out words in questions / propagate gradient conditionally) but just not well presented and the impact of these changes -- while we measure the final number, it would be good to see a couple of examples where there is evidence that this is indeed the effect and not random fluctuations.
rating: 2: Marginally below acceptance threshold
confidence: 3: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature