Original: Are you tough enough? Framework for Robustness Validation of Machine Comprehension Systems
Revised: Are you tough enough? Framework for Robustness Validation of Machine Comprehension Systems

Original: Abstract
Revised: Abstract

Original: Deep Learning NLP domain lacks procedures for the analysis of model robustness.
Revised: Deep Learning NLP domain lacks procedures for the analysis of model robustness.

Original: In this paper we propose a framework which validates robustness of any
Revised: In this paper we propose a framework which validates robustness of any

Original: Question Answering model through model explainers.
Revised: Question Answering model through model explainers.

Original: We propose that a robust model should transgress the initial notion of semantic similarity induced by word embeddings to learn a more human-like understanding of meaning.
Revised: We propose that a robust model should transgress the initial notion of semantic similarity induced by word embeddings to learn a more human-like understanding of meaning.

Original: We test this property by manipulating questions in two ways: swapping important question word for 1) its semantically correct synonym and 2) for word vector that is close in embedding space.
Revised: We test this property by manipulating questions in two ways: swapping important question word for 1) its semantically correct synonym and 2) for word vector that is close in embedding space.

Original: We estimate importance of words in asked questions with
Revised: We estimate importance of words in asked questions with

Original: Locally Interpretable Model Agnostic Explanations method (LIME).
Revised: Locally Interpretable Model Agnostic Explanations method (LIME).

Original: With these two steps we compare state-of-the-art Q&A models.
Revised: With these two steps we compare state-of-the-art Q&A models.

Original: We show that although accuracy of state-of-the-art models is high, they are very fragile to changes in the input.
Revised: We show that although accuracy of state-of-the-art models is high, they are very fragile to changes in the input.

Original: Moreover, we propose 2 adversarial training scenarios which raise model sensitivity to true synonyms by up to 7% accuracy measure.
Revised: Moreover, we propose 2 adversarial training scenarios which raise model sensitivity to true synonyms by up to 7% accuracy measure.

Original: Our ﬁndings help to understand which models are more stable and how they can be improved.
Revised: Our ﬁndings help to understand which models are more stable and how they can be improved.

Original: In addition, we have created and published a new dataset that may be used for validation of robustness of a Q&A model 1 .
Revised: In addition, we have created and published a new dataset that may be used for validation of robustness of a Q&A model 1 .

Original:  Introduction
Revised:  Introduction

Original: Up-to-date advancements in natural language processing show that it is possible to achieve high accuracy on tasks that from human point of view require language understanding.
Revised: Up-to-date advancements in natural language processing show that it is possible to achieve high accuracy on tasks that from human point of view require language understanding.

Original: However, recent research on adversarial examples revealed shortcomings of neural network models, despite their great performance on selected datasets [11].
Revised: However, recent research on adversarial examples revealed shortcomings of neural network models, despite their great performance on selected datasets [11].

Original: Most adversarial examples studies in Question Answering (Q&A) task focus on designing sophisticated attacks that prove overstability understood as an inability of the model to distinguish sentences that answer the question from sentences that have words in common with the question [4].
Revised: Most adversarial examples studies in Question Answering (Q&A) task focus on designing sophisticated attacks that prove overstability understood as an inability of the model to distinguish sentences that answer the question from sentences that have words in common with the question [4].

Original: [6] and [3] also show that the whole question itself is not crucial to get the right answer.
Revised: [6] and [3] also show that the whole question itself is not crucial to get the right answer.

Original: Despite growing research in the area of adversarial attacks, little is still known about the source of lacking robustness of Q&A models and effective ways of overcoming this problem.
Revised: Despite growing research in the area of adversarial attacks, little is still known about the source of lacking robustness of Q&A models and effective ways of overcoming this problem.

Original: In order to come up with a solution it is crucial to understand the inner workings of task speciﬁc architectures and benchmark their performance on adversarial attacks.
Revised: In order to come up with a solution it is crucial to understand the inner workings of task speciﬁc architectures and benchmark their performance on adversarial attacks.

Original: In this work we propose a model agnostic framework that checks stability of three state-of-the-art
Revised: In this work we propose a model agnostic framework that checks stability of three state-of-the-art

Original: Q&A architectures in terms of their ability to respond to semantically similar questions.
Revised: Q&A architectures in terms of their ability to respond to semantically similar questions.

Original: We formulate two aspects which are subject to tests: semantic input stability (relying on true semantics) and numerical input stability (purely induced by similarity between word embeddings).
Revised: We formulate two aspects which are subject to tests: semantic input stability (relying on true semantics) and numerical input stability (purely induced by similarity between word embeddings).

Original: These terms are explained later on in the article.
Revised: These terms are explained later on in the article.

Original: We state that a robust and reliable Q&A model should be invariant to changes in the input until they induce semantic changes.
Revised: We state that a robust and reliable Q&A model should be invariant to changes in the input until they induce semantic changes.

Original: Having said that, we state that a stable model must display higher semantic input stability measure than numerical input stability measure.
Revised: Having said that, we state that a stable model must display higher semantic input stability measure than numerical input stability measure.

Original: We claim that a robust Q&A model should possess high sensitivity to true semantics, as opposed to closeness between word embeddings.
Revised: We claim that a robust Q&A model should possess high sensitivity to true semantics, as opposed to closeness between word embeddings.

Original: Otherwise, attacks based on commonly accessible sets of embeddings become possible, including antonymous questions, as antonyms are often close in embedding space.
Revised: Otherwise, attacks based on commonly accessible sets of embeddings become possible, including antonymous questions, as antonyms are often close in embedding space.

Original: We make the following contributions:
Revised: We make the following contributions:

Original: • We investigate robustness of the model in the face of semantic and numerical changes to the input.
Revised: • We investigate robustness of the model in the face of semantic and numerical changes to the input.

Original: • We use output of Locally Interpretable Model Agnostic Explanations [10] to create attacks.
Revised: • We use output of Locally Interpretable Model Agnostic Explanations [10] to create attacks.

Original: • We offer 2 approaches to adversarial training which increase model sensitivity to true semantic differences by a maximum of 7% in accuracy.
Revised: • We offer 2 approaches to adversarial training which increase model sensitivity to true semantic differences by a maximum of 7% in accuracy.

Original: • We release a collection of 1500 semantically coherent questions from SQuAD dataset [9] preprocessed by LIME and human annotators, that we used in this study, as a reference dataset for further works on this problem.
Revised: • We release a collection of 1500 semantically coherent questions from SQuAD dataset [9] preprocessed by LIME and human annotators, that we used in this study, as a reference dataset for further works on this problem.

Original: In section 2 we explain measures and tools that we used in the study.
Revised: In section 2 we explain measures and tools that we used in the study.

Original: In section 3 we conduct experiments on three popular Q&A architectures, then we introduce two modiﬁcations to one of the models in order to increase ability to answer semantically similar questions and ﬁnally we refer to related work and conclude.
Revised: In section 3 we conduct experiments on three popular Q&A architectures, then we introduce two modiﬁcations to one of the models in order to increase ability to answer semantically similar questions and ﬁnally we refer to related work and conclude.

Original:  Robustness measures
Revised:  Robustness measures

Original: Importance of question words.
Revised: Importance of question words.

Original: Previous studies have shown sensitivity of the model to changes in the input.
Revised: Previous studies have shown sensitivity of the model to changes in the input.

Original: Based on this notion, we use the popular LIME (Locally Interpretable Model Agnostic
Revised: Based on this notion, we use the popular LIME (Locally Interpretable Model Agnostic

Original: Explanations) framework to assess the contribution of each question word and construct adversaries later on, by replacing the most important word.
Revised: Explanations) framework to assess the contribution of each question word and construct adversaries later on, by replacing the most important word.

Original: LIME ﬁts a linear model in local surrounding of each example and estimates coefﬁcients that serve as a importance measure in the process of model interpretability.
Revised: LIME ﬁts a linear model in local surrounding of each example and estimates coefﬁcients that serve as a importance measure in the process of model interpretability.

Original: We call the most important word the keyword .
Revised: We call the most important word the keyword .

Original: Numerical input stability.
Revised: Numerical input stability.

Original: Deep learning models usually have input words represented in the form of embedded vectors, which exhibit proximity to other words, which are not necessarily synonyms.
Revised: Deep learning models usually have input words represented in the form of embedded vectors, which exhibit proximity to other words, which are not necessarily synonyms.

Original: This results from their creation procedure, which represents a word by its neighbors.
Revised: This results from their creation procedure, which represents a word by its neighbors.

Original: In GloVe [7] and Fasttext [1] antonyms can be found to be close neighbors, e.g. victory is located near to defeat , as well as up to down (0.85 cosine similarity in GloVe, they are nearest neighbors).
Revised: In GloVe [7] and Fasttext [1] antonyms can be found to be close neighbors, e.g. victory is located near to defeat , as well as up to down (0.85 cosine similarity in GloVe, they are nearest neighbors).

Original: Likewise, other kinds of similarity which bring words close together in embedding space can be identiﬁed, such as common hypernym, e.g. cat is very close to dog and puppy .
Revised: Likewise, other kinds of similarity which bring words close together in embedding space can be identiﬁed, such as common hypernym, e.g. cat is very close to dog and puppy .

Original: Overstability to numerical inputs should be thus regarded as major vulnerability to adversarial attacks.
Revised: Overstability to numerical inputs should be thus regarded as major vulnerability to adversarial attacks.

Original: We use the idea of similarity in embedding space to ﬁnd words that are the closest to the keyword in
Revised: We use the idea of similarity in embedding space to ﬁnd words that are the closest to the keyword in

Original: SQuAD questions.
Revised: SQuAD questions.

Original: We swap keywords with closest words and make predictions.
Revised: We swap keywords with closest words and make predictions.

Original: GloVe embeddings are used for extraction of closest words, since they were the input used to train the tested models.
Revised: GloVe embeddings are used for extraction of closest words, since they were the input used to train the tested models.

Original: Overall, we ﬁnd that 37% of our created questions can be seen as potentially vulnerability-inducing, containing antonyms or other severe meaning modiﬁcations with regard to original questions.
Revised: Overall, we ﬁnd that 37% of our created questions can be seen as potentially vulnerability-inducing, containing antonyms or other severe meaning modiﬁcations with regard to original questions.

Original: Semantic input stability.
Revised: Semantic input stability.

Original: As stated above, robust model should be invariant to semantic changes.
Revised: As stated above, robust model should be invariant to semantic changes.

Original: Thus, we disregard common word embeddings and construct genuinely synonymous sentences based on Wordnet [5] synonyms end context dependent ELMO [8] embeddings that are later on validated by human annotators and corrected.
Revised: Thus, we disregard common word embeddings and construct genuinely synonymous sentences based on Wordnet [5] synonyms end context dependent ELMO [8] embeddings that are later on validated by human annotators and corrected.

Original: This manually improved dataset is released for the community.
Revised: This manually improved dataset is released for the community.

Original: The procedure together with comparison with numerical input stability measure allows us to inspect whether neural network understands the whole question or simply relies on numerical representation of keywords .
Revised: The procedure together with comparison with numerical input stability measure allows us to inspect whether neural network understands the whole question or simply relies on numerical representation of keywords .

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Experiments
Revised:  Experiments

Original: We investigate three popular architectures: DrQA [2], BiDAF [12] and QANet [14].
Revised: We investigate three popular architectures: DrQA [2], BiDAF [12] and QANet [14].

Original: Each model is tested on a prepared dataset, which comprises of examples of contexts, questions and answers extracted from SQuAD dataset.
Revised: Each model is tested on a prepared dataset, which comprises of examples of contexts, questions and answers extracted from SQuAD dataset.

Original: Each question is modiﬁed so that the most important word computed with LIME is swapped for 1.
Revised: Each question is modiﬁed so that the most important word computed with LIME is swapped for 1.

Original: its synonym appropriate in the given context ( syn-Dataset ) 2.
Revised: its synonym appropriate in the given context ( syn-Dataset ) 2.

Original: a word which has the shortest cosine distance in GloVe embedding space ( num-Dataset ) 3.
Revised: a word which has the shortest cosine distance in GloVe embedding space ( num-Dataset ) 3.

Original: a word random ( random-Dataset ).
Revised: a word random ( random-Dataset ).

Original: Table 1 shows an example of the resulting questions.
Revised: Table 1 shows an example of the resulting questions.

Original: In total, we producetest examples, with 500 generated by each of the 3 algorithms.
Revised: In total, we producetest examples, with 500 generated by each of the 3 algorithms.

Original: Note that modiﬁcations induced by various algorithms vary since they attend to different words in the question (as measured by LIME), so in fact we release 3 separate datasets.
Revised: Note that modiﬁcations induced by various algorithms vary since they attend to different words in the question (as measured by LIME), so in fact we release 3 separate datasets.

Original: In our experiments we only test on examples which scored a correct answer in each dataset, treating wrongly tagged examples as of ambiguous merit for this task.
Revised: In our experiments we only test on examples which scored a correct answer in each dataset, treating wrongly tagged examples as of ambiguous merit for this task.

Original: A merge of these three datasets can be seen as an approximation of our original datasets to be used with any new model other than the ones we test here.
Revised: A merge of these three datasets can be seen as an approximation of our original datasets to be used with any new model other than the ones we test here.

Original: As shown in table 2, stability with regard to both semantic and numeric changes varies between models.
Revised: As shown in table 2, stability with regard to both semantic and numeric changes varies between models.

Original: In general however, we ﬁnd that all models respond better to changes induced by numDataset than syn-Dataset, which is an unwelcome result.
Revised: In general however, we ﬁnd that all models respond better to changes induced by numDataset than syn-Dataset, which is an unwelcome result.

Original: Indeed, all tested models are found to be overstable to numeric manipulations while recognizing true synonyms at a lower rate.
Revised: Indeed, all tested models are found to be overstable to numeric manipulations while recognizing true synonyms at a lower rate.

Original: Moreover, there is a big difference in performance once we swap keyword with random words which are on average more distant to the keyword in embedding space.
Revised: Moreover, there is a big difference in performance once we swap keyword with random words which are on average more distant to the keyword in embedding space.

Original: This heavy decrease in performance also proves that indeed, models are overstable once attacked with words close to the keyword .
Revised: This heavy decrease in performance also proves that indeed, models are overstable once attacked with words close to the keyword .

Original:  Adversarial training
Revised:  Adversarial training

Original: In the face of our ﬁndings, a much-needed next step is to render models more robust by increasing their ability to give right answer to the semantically similar questions.
Revised: In the face of our ﬁndings, a much-needed next step is to render models more robust by increasing their ability to give right answer to the semantically similar questions.

Original: In order to increase this aspect of model robustness we introduce two separate changes to the algorithm and choose to retrain DrQA model, which turned out to be weak in terms of semantic similarity recognition.
Revised: In order to increase this aspect of model robustness we introduce two separate changes to the algorithm and choose to retrain DrQA model, which turned out to be weak in terms of semantic similarity recognition.

Original: New models are validated on all datasets.
Revised: New models are validated on all datasets.

Original: These changes are of model-agnostic nature and can be implemented in any algorithm.
Revised: These changes are of model-agnostic nature and can be implemented in any algorithm.

Original: [Table]
Revised: [Table]

Original: Removing words from training dataset (REM).
Revised: Removing words from training dataset (REM).

Original: Inspired by logic behind dropout [13] we train
Revised: Inspired by logic behind dropout [13] we train

Original: Q&A models with enriched training dataset.
Revised: Q&A models with enriched training dataset.

Original: Each question is present in multiple forms - one question with all words preserved and 2 or 3 questions with one random word removed.
Revised: Each question is present in multiple forms - one question with all words preserved and 2 or 3 questions with one random word removed.

Original: This enrichment of the dataset enables to overcome phenomenon that may be the reason behind the success of adversarial attacks - heavy dependence on individual words in the question.
Revised: This enrichment of the dataset enables to overcome phenomenon that may be the reason behind the success of adversarial attacks - heavy dependence on individual words in the question.

Original: Thanks to this procedure we increase the performance on syn-Dataset in comparison to training only on original questions.
Revised: Thanks to this procedure we increase the performance on syn-Dataset in comparison to training only on original questions.

Original: However, it must be noted that although this improvement is a step towards answering semantically similar questions, it does not overcome the problem of higher numeric input stability than semantic input stability, deﬁned in section 2.
Revised: However, it must be noted that although this improvement is a step towards answering semantically similar questions, it does not overcome the problem of higher numeric input stability than semantic input stability, deﬁned in section 2.

Original: Propagating gradients back to input embeddings (GRAD).
Revised: Propagating gradients back to input embeddings (GRAD).

Original: Intuitively, a part of the Q&A model’s knowledge of synonymy acquired during training can be transmitted to embeddings themselves.
Revised: Intuitively, a part of the Q&A model’s knowledge of synonymy acquired during training can be transmitted to embeddings themselves.

Original: Fine-tuning is sometimes done to increase model performance, and may be abandoned when no rise in performance is observed.
Revised: Fine-tuning is sometimes done to increase model performance, and may be abandoned when no rise in performance is observed.

Original: We conclude that ﬁne-tuning the embeddings can increase the semantic stability, independently from inﬂuencing model performance.
Revised: We conclude that ﬁne-tuning the embeddings can increase the semantic stability, independently from inﬂuencing model performance.

Original: Similarly as in REM procedure, we obtain a rise in semantic synonym recognition, coupled with a rise in numeric accuracy.
Revised: Similarly as in REM procedure, we obtain a rise in semantic synonym recognition, coupled with a rise in numeric accuracy.

Original: To check whether the increased numeric accuracy scores were indeed a problem in our adversarial settings, we tested which percentage of semantically incorrect numDataset examples receive a negative score from our models.
Revised: To check whether the increased numeric accuracy scores were indeed a problem in our adversarial settings, we tested which percentage of semantically incorrect numDataset examples receive a negative score from our models.

Original: For base model, 37% problematic examples received decision change with respect to correct original examples.
Revised: For base model, 37% problematic examples received decision change with respect to correct original examples.

Original: With scores 38%, 36% for GRADx10 and GRADx respectively, these models suffer less overstability to numeric modiﬁcations with regard to base model, while REM models (34%, 30% for REMx2 and REMx3 respectively) did exhibit increased acceptance to manipulations in numeric representations.
Revised: With scores 38%, 36% for GRADx10 and GRADx respectively, these models suffer less overstability to numeric modiﬁcations with regard to base model, while REM models (34%, 30% for REMx2 and REMx3 respectively) did exhibit increased acceptance to manipulations in numeric representations.

Original: Both methods inﬂuenced model robustness without a noticeable difference in model quality measured in traditional metrics - EM and F1.
Revised: Both methods inﬂuenced model robustness without a noticeable difference in model quality measured in traditional metrics - EM and F1.

Original: It suggests that models do need speciﬁc, established procedures for testing robustness, which should be conducted independently from traditional performance tests, since these scores are not necessarily related.
Revised: It suggests that models do need speciﬁc, established procedures for testing robustness, which should be conducted independently from traditional performance tests, since these scores are not necessarily related.

Original:  Related work
Revised:  Related work

Original: Researching true semantics with respect to numeric manipulations is a novel topic, although recent research is rich with papers dealing with manipulating semantics for Q&A systems.
Revised: Researching true semantics with respect to numeric manipulations is a novel topic, although recent research is rich with papers dealing with manipulating semantics for Q&A systems.

Original: [11] show effectiveness of adversarial attacks on Q&A system which drop performance without signiﬁcant semantic change.
Revised: [11] show effectiveness of adversarial attacks on Q&A system which drop performance without signiﬁcant semantic change.

Original: [4] show a keyword detection method in Q&A using Integrated Gradients, although they do not venture on to create attacks using important words in Q&A task.
Revised: [4] show a keyword detection method in Q&A using Integrated Gradients, although they do not venture on to create attacks using important words in Q&A task.

Original: [15] generate paraphrases of whole adversarial questions and test model robustness on these grammatically correct and semantically similar adversaries.
Revised: [15] generate paraphrases of whole adversarial questions and test model robustness on these grammatically correct and semantically similar adversaries.

Original: In our work, apart from studying semantic similarity on a more detailed level (individual words), we propose an augmentation framework.
Revised: In our work, apart from studying semantic similarity on a more detailed level (individual words), we propose an augmentation framework.

Original: [14] attempt to generate adversarial examples in a non-gradient based way, in order to be able able to attack a black-box model, a problem which we bypass using model-agnostic explainer.
Revised: [14] attempt to generate adversarial examples in a non-gradient based way, in order to be able able to attack a black-box model, a problem which we bypass using model-agnostic explainer.

Original:  Conclusion
Revised:  Conclusion

Original: In our study we focused on one particular aspect of model robustness - ability to answer semantically coherent questions.
Revised: In our study we focused on one particular aspect of model robustness - ability to answer semantically coherent questions.

Original: We show that performance of all tested models decreases once we change important question words indicated by LIME.
Revised: We show that performance of all tested models decreases once we change important question words indicated by LIME.

Original: However, we observe that models have higher performance once we swap tokens in questions for words close in their embedding space deﬁned by word vectors they were trained on, in comparison to accuracy obtained by asking semantically correct questions.
Revised: However, we observe that models have higher performance once we swap tokens in questions for words close in their embedding space deﬁned by word vectors they were trained on, in comparison to accuracy obtained by asking semantically correct questions.

Original: We manage to increase ability of the model to answer this kind of questions but it does not mean that they understand semantics - it is reﬂected in increased accuracy of questions with GloVe embeddings once we test our newly trained models on num-Dataset.
Revised: We manage to increase ability of the model to answer this kind of questions but it does not mean that they understand semantics - it is reﬂected in increased accuracy of questions with GloVe embeddings once we test our newly trained models on num-Dataset.

Original: We show that the reason behind the success of some adversarial examples lies in the way input words are represented.
Revised: We show that the reason behind the success of some adversarial examples lies in the way input words are represented.

Original: Popular embeddings do not include knowledge about real meaning of the words, but rather incorporate knowledge about their context.
Revised: Popular embeddings do not include knowledge about real meaning of the words, but rather incorporate knowledge about their context.

Original: Our work serves as a starting point for future research on more semantically-conscious representation of words.
Revised: Our work serves as a starting point for future research on more semantically-conscious representation of words.

Original: Acknowledgements
Revised: Acknowledgements

Original: This research wasﬁnancially supported by the NCN grant no.2017/27/B/ST6/01307 .
Revised: This research was supported by the Polish National Science Centre DALEX grant no. 2017/27/B/ST6/01307.