Original: Are you tough enough? Framework for Robustness Validation of Machine Comprehension Systems
Revised: Are you tough enough? Framework for Robustness Validation of Machine Comprehension Systems

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Deep Learning NLP domain lacks procedures for the analysis of model robustness.
Revised: Deep Learning NLP domain lacks procedures for the analysis of model robustness.

Original: In this paper we propose a framework which validates robustness of anyQuestion
Revised: In this paper we propose a framework which validates robustness of any

Original:  Answering model through model explainers.
Revised: Question Answering model through model explainers.

Original: We propose thatoutput of a robustmodel should be invariant to alterations that do not change its semantics.
Revised: We propose that a robust model should transgress the initial notion of semantic similarity induced by word embeddings to learn a more human-like understanding of meaning.

Original: Wetest this property by manipulating question in two ways: swapping importantquestion word for 1) its semantically correct synonym and 2) for word vector thatis close in embedding space.
Revised: We test this property by manipulating questions in two ways: swapping important question word for 1) its semantically correct synonym and 2) for word vector that is close in embedding space.

Original: We estimate importance of words in asked questionswith Locally Interpretable Model Agnostic Explanations method (LIME).
Revised: We estimate importance of words in asked questions with Locally Interpretable Model Agnostic Explanations method (LIME).

Original: Withthese two steps we compare state-of-the-art Q&A models.
Revised: With these two steps we compare state-of-the-art Q&A models.

Original: We show that althoughaccuracy of state-of-the-art models is high, they are very fragile to changes in theinput.
Revised: We show that although accuracy of state-of-the-art models is high, they are very fragile to changes in the input.

Original: We can choose architecture that is more immune to attacks and thus morerobust and stable in production environment.

Original: Morevoer, we propose 2 adversarialtraining scenarios which raise model sensitivity to true synonyms by up to 7%accuracy measure.
Revised: Moreover, we propose 2 adversarial training scenarios which raise model sensitivity to true synonyms by up to 7% accuracy measure.

Original: Our ﬁndings help to understand which models are more stableand how they can be improved.
Revised: Our ﬁndings help to understand which models are more stable and how they can be improved.

Original: In addition, we have created and published a newdataset that may be used for validation of robustness of a Q&A model 1 .
Revised: In addition, we have created and published a new dataset that may be used for validation of robustness of a Q&A model 1 .

Original:  Introduction
Revised:  Introduction

Original: Up-to-date advancements in natural language processing show that it is possible to get high accuracyon tasks that from human point of view require language understanding.
Revised: Up-to-date advancements in natural language processing show that it is possible to achieve high accuracy on tasks that from human point of view require language understanding.

Original: However, recent research onadversarial examples revealed shortcomings of neural network models, despite their great performanceon selected datasets [10].
Revised: However, recent research on adversarial examples revealed shortcomings of neural network models, despite their great performance on selected datasets [11].

Original: To narrow the scope of our study, we focus on Question Answering(Q&A) task. Most adversarial examples studies  focus on designing sophisticated attacks that proveoverstability understood as an inability of the model to distinguish sentence that answers the questionfrom a sentence that has words in common with the question [4].
Revised: Most adversarial examples studies in Question Answering (Q&A) task focus on designing sophisticated attacks that prove overstability understood as an inability of the model to distinguish sentences that answer the question from sentences that have words in common with the question [4].

Original: Research also show that thequestion itself is not crucial to get the right answer ([5], [3]).
Revised: [6] and [3] also show that the whole question itself is not crucial to get the right answer.

Original: Despite growing research in the areaof adversarial attacks, little is still known about the source of lack of robustness of Q&A modelsand effective ways of overcoming this problem.
Revised: Despite growing research in the area of adversarial attacks, little is still known about the source of lacking robustness of Q&A models and effective ways of overcoming this problem.

Original: In order to come  with the solution it is crucial tounderstand the inner workings of task speciﬁc architectures and benchmark their performance atadversarial attack.
Revised: In order to come up with a solution it is crucial to understand the inner workings of task speciﬁc architectures and benchmark their performance on adversarial attacks.

Original: That is why we propose a model agnostic framework that checks stability of different architecturesin terms of their ability to respond to semantically similar questions.
Revised: In this work we propose a model agnostic framework that checks stability of three state-of-the-art Q&A architectures in terms of their ability to respond to semantically similar questions.

Original: In our work we inspect twoaspects: semantic input stability  and numerical input stability that we explain later on in the article.
Revised: We formulate two aspects which are subject to tests: semantic input stability (relying on true semantics) and numerical input stability (purely induced by similarity between word embeddings). These terms are explained later on in the article.

Original: We state that a robust and reliable Q&A model should be invariant to changes in the input until theyinduce semantic changes.
Revised: We state that a robust and reliable Q&A model should be invariant to changes in the input until they induce semantic changes.

Original: Having said that, we state that a stable model must display higher semantic input stability measure than numerical input stability measure.
Revised: Having said that, we state that a stable model must display higher semantic input stability measure than numerical input stability measure.

Original: We claim that a robust Q&A modelshould possess high sensitivity to true semantics, as opposed to closeness between word embeddings.
Revised: We claim that a robust Q&A model should possess high sensitivity to true semantics, as opposed to closeness between word embeddings.

Original: Otherwise, attacks based on commonly accessible sets of embeddings become possible, includingantonymous questions, as antonyms are often close in embedding space.
Revised: Otherwise, attacks based on commonly accessible sets of embeddings become possible, including antonymous questions, as antonyms are often close in embedding space.

Original: We make the followingcontributions:
Revised: We make the following contributions:

Original: • We investigate robustness of the model in the face of semantic and numerical changes to theinput.• We use output of Locally Interpretable Model Agnostic Explanations [9] to create attacks.•
Revised: • We investigate robustness of the model in the face of semantic and numerical changes to the input. • We use output of Locally Interpretable Model Agnostic Explanations [10] to create attacks.

Original:  We offer 2 approaches to adversarial training which increase model sensitivity to truesemantic differences by a maximum of 7%.
Revised: • We offer 2 approaches to adversarial training which increase model sensitivity to true semantic differences by a maximum of 7% in accuracy.

Original: • We release a collection of 1500 semantically coherent questions from SQuAD dataset [8]preprocessed by LIME  that we used in this study, as a reference dataset for further works on this problem.
Revised: • We release a collection of 1500 semantically coherent questions from SQuAD dataset [9] preprocessed by LIME and human annotators, that we used in this study, as a reference dataset for further works on this problem.

Original: In section 2 we explain measures and tools that we used in the study.
Revised: In section 2 we explain measures and tools that we used in the study.

Original: In section 3 we conductexperiments on 3 popular Q&A architectures, then we introduce two modiﬁcations to one of themodels in order to increase ability to answer semantically similar questions and ﬁnally we refer torelated work and conclude.
Revised: In section 3 we conduct experiments on three popular Q&A architectures, then we introduce two modiﬁcations to one of the models in order to increase ability to answer semantically similar questions and ﬁnally we refer to related work and conclude.

Original:  Robustness measures
Revised:  Robustness measures

Original: Importance of question words.
Revised: Importance of question words.

Original: Previous studies showed sensibility of the model to changes in theinput, although it is not clear why some attacks succeed and others do not.
Revised: Previous studies have shown sensitivity of the model to changes in the input.

Original: We use a popular LIME(Locally Interpretable Model Agnostic Explanations) framework to assess the contribution of eachquestion word andbased on result, we construct adversaries later on.
Revised: Based on this notion, we use the popular LIME (Locally Interpretable Model Agnostic Explanations) framework to assess the contribution of each question word and construct adversaries later on, by replacing the most important word.

Original: LIME ﬁts a linear model inlocal surrounding of each example and estimates coefﬁencts that serve as a importance measure inthe process of model interpretability.
Revised: LIME ﬁts a linear model in local surrounding of each example and estimates coefﬁcients that serve as a importance measure in the process of model interpretability.

Original: We call the most important word the keyword .
Revised: We call the most important word the keyword .

Original: Numerical input stability.
Revised: Numerical input stability.

Original: Deep learning models usually have input words represented in the formof embedded vectors, which exhibit proximity of other words, which are not necessarily synonyms.
Revised: Deep learning models usually have input words represented in the form of embedded vectors, which exhibit proximity to other words, which are not necessarily synonyms.

Original: That results from their creation procedure that represents  word by its neighbours.
Revised: This results from their creation procedure, which represents a word by its neighbors.

Original: In GloVe [6] and Fasttext [1]) antonyms can be found to be close neighbors, e.g. victory is located near to defeat , aswell as up to down (0.85 cosine similarity in GloVe, they are nearest neighbors).
Revised: In GloVe [7] and Fasttext [1] antonyms can be found to be close neighbors, e.g. victory is located near to defeat , as well as up to down (0.85 cosine similarity in GloVe, they are nearest neighbors).

Original: Likewise, otherkinds of similarity which bring words close together in embedding space can be identiﬁed, suchas common hypernym, e.g. cat is very close to dog and puppy .
Revised: Likewise, other kinds of similarity which bring words close together in embedding space can be identiﬁed, such as common hypernym, e.g. cat is very close to dog and puppy .

Original: Overstability to numerical inputsshould be thus regarded as major vulnerability to adversarial attacks.
Revised: Overstability to numerical inputs should be thus regarded as major vulnerability to adversarial attacks.

Original: We use the idea of similarity in embedding space to ﬁnd words that are the closest to the keyword .
Revised: We use the idea of similarity in embedding space to ﬁnd words that are the closest to the keyword in

Revised: SQuAD questions.

Original: Weswap keywords with closest words and make predictions.
Revised: We swap keywords with closest words and make predictions.

Original: GloVe embeddings were used for extractionof closest words, since they were the input used to train the tested models.
Revised: GloVe embeddings are used for extraction of closest words, since they were the input used to train the tested models.

Original: Overall, we ﬁnd that 37%of our created examples can be seen as adversarial, containing antonyms or other severe meaningmodiﬁcations.
Revised: Overall, we ﬁnd that 37% of our created questions can be seen as potentially vulnerability-inducing, containing antonyms or other severe meaning modiﬁcations with regard to original questions.

Original: Semantic input stability.
Revised: Semantic input stability.

Original: As stated above, robust model should be invariant to semantic changes.
Revised: As stated above, robust model should be invariant to semantic changes.

Original: Thus, we disregard common word embeddings and construct  sentences based on Wordnet synonymsend context dependent ELMO [7] embeddings that are later on validated by human annotators andcorrected.
Revised: Thus, we disregard common word embeddings and construct genuinely synonymous sentences based on Wordnet [5] synonyms end context dependent ELMO [8] embeddings that are later on validated by human annotators and corrected.

Original: We release manually improved dataset  for the community.
Revised: This manually improved dataset is released for the community.

Original: This procedure together withcomparison with numerical input stability measure allows us to inspect whether neural networkunderstands the whole question or simply relies on numerical representation of keywords . 
Revised: The procedure together with comparison with numerical input stability measure allows us to inspect whether neural network understands the whole question or simply relies on numerical representation of keywords . [Table] [Table]

Original:  Experiments
Revised:  Experiments

Original: We investigate 3 popular architectures: DrQA [2], BiDAF [11] and QANet [13].
Revised: We investigate three popular architectures: DrQA [2], BiDAF [12] and QANet [14].

Original: Each model is testedon a prepared dataset, which comprises of examples of contexts, questions and answers extractedfrom SQuAD[8] dataset.
Revised: Each model is tested on a prepared dataset, which comprises of examples of contexts, questions and answers extracted from SQuAD dataset.

Original: Each question is modiﬁed so that the most important word computedwith LIME is swapped for: 1) its synonym appropriate in the given context (syn-Dataset) 2) a wordwhich has the shortest cosine distance in GloVe embedding space (num-Dataset) 3) a  random word(random-Dataset).
Revised: Each question is modiﬁed so that the most important word computed with LIME is swapped for 1. its synonym appropriate in the given context ( syn-Dataset ) 2. a word which has the shortest cosine distance in GloVe embedding space ( num-Dataset ) 3. a word random ( random-Dataset ).

Original: Table 1 shows an example of the resulting questions.
Revised: Table 1 shows an example of the resulting questions.

Original: [-In total, we produce 
-]
Revised: [-In total, we produce 
-]

Original: [Table]

Original: [Table] [Table] test examples, with 500 generated by each of the 3 algorithms.
Revised: In total, we producetest examples, with 500 generated by each of the 3 algorithms.

Original: Note that modiﬁcations induced byvarious algorithms vary since they attend to different words in the question (as measured by LIME),so in fact we release 3 separate datasets.
Revised: Note that modiﬁcations induced by various algorithms vary since they attend to different words in the question (as measured by LIME), so in fact we release 3 separate datasets.

Original: In our experiments we only test on examples which scored acorrect answer in each dataset, treating wrongly tagged examples as of ambiguous merit for this task.
Revised: In our experiments we only test on examples which scored a correct answer in each dataset, treating wrongly tagged examples as of ambiguous merit for this task.

Original: A merge of these three datasets can be seen as an approximation of our original datasets to be usedwith any new model other than the ones we test here.
Revised: A merge of these three datasets can be seen as an approximation of our original datasets to be used with any new model other than the ones we test here.

Original: As shown in table 2, stability with regard to both semantic and numeric changes varies betweenmodels.
Revised: As shown in table 2, stability with regard to both semantic and numeric changes varies between models.

Original: In general however, we ﬁnd that all models respond better to changes induced by numDataset than syn-Dataset, which is an unwelcome result.
Revised: In general however, we ﬁnd that all models respond better to changes induced by numDataset than syn-Dataset, which is an unwelcome result.

Original: Indeed, all tested models are found to beoverstable to numeric manipulations while recognizing true synonyms at a lower rate.
Revised: Indeed, all tested models are found to be overstable to numeric manipulations while recognizing true synonyms at a lower rate.

Original: Moreover,there is a big difference in performance once we swap keyword with random words which are onaverage more distant to the keyword in embedding space.
Revised: Moreover, there is a big difference in performance once we swap keyword with random words which are on average more distant to the keyword in embedding space.

Original: This heavy decrease in performance alsoproves that indeed, models are overstable once attacked with words close to the keyword .
Revised: This heavy decrease in performance also proves that indeed, models are overstable once attacked with words close to the keyword .

Original:  Adversarial training
Revised:  Adversarial training

Original: In the face of our ﬁndings, a much-needed next step is to render models more robust by increasingtheir ability to give right answer to the semantically similar questions.
Revised: In the face of our ﬁndings, a much-needed next step is to render models more robust by increasing their ability to give right answer to the semantically similar questions.

Original: In order toto increase thisaspect of model robustness we introduced two separate changes to the algorithm and chose to retrain DrQA model, which turned out to be weak in terms of semantic similarity recognition.
Revised: In order to increase this aspect of model robustness we introduce two separate changes to the algorithm and choose to retrain DrQA model, which turned out to be weak in terms of semantic similarity recognition.

Original: New modelswere validated on syn-Dataset.
Revised: New models are validated on all datasets.

Original: These changes are of model-agnostic nature and can be implementedin any algorithm. 
Revised: These changes are of model-agnostic nature and can be implemented in any algorithm. [Table]

Original: Removing words from training dataset (REM).
Revised: Removing words from training dataset (REM).

Original: Inspired by logic behind dropout [12] we train
Revised: Inspired by logic behind dropout [13] we train

Original: Q&A models with enriched training dataset.
Revised: Q&A models with enriched training dataset.

Original: Each question is present in multiple forms - one questionwith all words preserved and 2 or 3 questions with one random word removed.
Revised: Each question is present in multiple forms - one question with all words preserved and 2 or 3 questions with one random word removed.

Original: This enrichment ofthe dataset enables to overcome phenomenon that is the reason behind the success of adversarialattack -  dependence on individual words in the question.
Revised: This enrichment of the dataset enables to overcome phenomenon that may be the reason behind the success of adversarial attacks - heavy dependence on individual words in the question.

Original: Thanks to this procedure we increase theperformance on syn-Dataset in comparison to training only on original questions.
Revised: Thanks to this procedure we increase the performance on syn-Dataset in comparison to training only on original questions.

Original: However, it mustbe noted that although this improvement is a step towards answering semantically similar questions,it does not overcome the problem of higher numerical input stability than semantic input stability,deﬁned in section 2.[Table]
Revised: However, it must be noted that although this improvement is a step towards answering semantically similar questions, it does not overcome the problem of higher numeric input stability than semantic input stability, deﬁned in section 2.

Original: Propagating gradients back to input embeddings (GRAD).
Revised: Propagating gradients back to input embeddings (GRAD).

Original: Intuitively, a part of the Q&A model’sknowledge of synonymy acquired during training can be transmitted to embeddings themselves.
Revised: Intuitively, a part of the Q&A model’s knowledge of synonymy acquired during training can be transmitted to embeddings themselves.

Revised: Fine-tuning is sometimes done to increase model performance, and may be abandoned when no rise in performance is observed.

Original: Weconclude that training the embeddingstogether with the system can increase the semantic stability.
Revised: We conclude that ﬁne-tuning the embeddings can increase the semantic stability, independently from inﬂuencing model performance.

Original: Similarly as in REM procedure, we obtain a rise in semantic synonym recognition, coupled with arise in numeric accuracy.
Revised: Similarly as in REM procedure, we obtain a rise in semantic synonym recognition, coupled with a rise in numeric accuracy.

Original: To test whether the increased numeric accuracy scoreswere indeed a problem in our adversarial settings, wetested which percentage of semantically incorrect numDataset examples receive a negative score from our models.
Revised: To check whether the increased numeric accuracy scores were indeed a problem in our adversarial settings, we tested which percentage of semantically incorrect numDataset examples receive a negative score from our models.

Original: For base model, 37% problematic examples receiveddecision change with respect to correct original examples.
Revised: For base model, 37% problematic examples received decision change with respect to correct original examples.

Original: With scores 38%, 36% for GRADx10 and GRADx30respectively, these models suffer less overstability to numeric modiﬁcations with regard to base model, while REMmodels (34%, 30% for [-REMx
-] and REMx3 respectively)did exhibit increased acceptance to manipulations in numeric representations.
Revised: With scores 38%, 36% for GRADx10 and GRADx respectively, these models suffer less overstability to numeric modiﬁcations with regard to base model, while REM models (34%, 30% for [-REMx
-]REMx2 and REMx3 respectively) did exhibit increased acceptance to manipulations in numeric representations.

Original: Both methods inﬂuenced model robustness without a noticeable difference in model quality measured in traditionalmetrics - EM and F1.
Revised: Both methods inﬂuenced model robustness without a noticeable difference in model quality measured in traditional metrics - EM and F1.

Original: It suggests that models do need speciﬁc, established procedures for testing robustness, whichshould be conducted independently from traditional performance tests, since these scores are not necessarily related.
Revised: It suggests that models do need speciﬁc, established procedures for testing robustness, which should be conducted independently from traditional performance tests, since these scores are not necessarily related.

Original:  Related work
Revised:  Related work

Original: Researching true semantics with respect to numeric manipulations is a novel topic, although recentresearch is rich with papers dealing with manipulating semantics for Q&A systems.
Revised: Researching true semantics with respect to numeric manipulations is a novel topic, although recent research is rich with papers dealing with manipulating semantics for Q&A systems.

Original: [10] showeffectiveness of adversarial attacks on Q&A system which drop performance without signiﬁcantsemantic change.
Revised: [11] show effectiveness of adversarial attacks on Q&A system which drop performance without signiﬁcant semantic change.

Original: [4] show a keyword detection method in Q&A using Integrated Gradients, althoughthey do not venture on to create attacks using important words in Q&A task.
Revised: [4] show a keyword detection method in Q&A using Integrated Gradients, although they do not venture on to create attacks using important words in Q&A task.

Original: [14] generate paraphrases of whole adversarial questions and test model robustness on these grammatically correct andsemantically similar adversaries.
Revised: [15] generate paraphrases of whole adversarial questions and test model robustness on these grammatically correct and semantically similar adversaries.

Original: In our work, apart from studying semantic similarity on a moredetailed level (individual words), we propose an augmentation framework.
Revised: In our work, apart from studying semantic similarity on a more detailed level (individual words), we propose an augmentation framework.

Original: [13] attempt to generateadversarial examples in a non-gradient based way, in order to be able able to attack a black-boxmodel, a problem which we bypass using model-agnostic explainer.
Revised: [14] attempt to generate adversarial examples in a non-gradient based way, in order to be able able to attack a black-box model, a problem which we bypass using model-agnostic explainer.

Original:  Conclusion
Revised:  Conclusion

Original: In our study we focused on one particular aspect of model robustness - ability to answer semanticallycoherent questions.
Revised: In our study we focused on one particular aspect of model robustness - ability to answer semantically coherent questions.

Original: We show that performance of all three models decreases once we change important question words indicated by LIME.
Revised: We show that performance of all tested models decreases once we change important question words indicated by LIME.

Original: However, we observe that models have higher performanceonce we swap tokens in questions to words close in their embedding space deﬁned by word vectorsthey were trained on, in comparison to accuracy obtained by asking semantically correct questions.
Revised: However, we observe that models have higher performance once we swap tokens in questions for words close in their embedding space deﬁned by word vectors they were trained on, in comparison to accuracy obtained by asking semantically correct questions.

Original: We manage to increase ability of the model to answer this kind of questions but it does not mean thatthey understand semantics - it is reﬂected in increased accuracy of questions with GloVe embeddingsonce we test our newly trained models on num-Dataset.
Revised: We manage to increase ability of the model to answer this kind of questions but it does not mean that they understand semantics - it is reﬂected in increased accuracy of questions with GloVe embeddings once we test our newly trained models on num-Dataset.

Original: This study shows that the reason behind the success of some adversarial examples lies in the wayinput words are represented.
Revised: We show that the reason behind the success of some adversarial examples lies in the way input words are represented.

Original: Popular embeddings do not include knowledge about real meaning ofthe words, but rather incorporate knowledge about their context.
Revised: Popular embeddings do not include knowledge about real meaning of the words, but rather incorporate knowledge about their context.

Original: It serves as a starting point for futureresearch on more semantically-conscious representation of words. 
Revised: Our work serves as a starting point for future research on more semantically-conscious representation of words. Acknowledgements

Revised: This research was supported by the Polish National Science Centre DALEX grant no.2017/27/B/ST6/01307.