  While numerous methods have been proposed as defenses against adversarial
examples in question answering (QA), these techniques are often model specific,
require retraining of the model, and give only marginal improvements in
performance over vanilla models. In this work, we present a simple
model-agnostic approach to this problem that can be applied directly to any QA
model without any retraining. Our method employs an explicit answer candidate
reranking mechanism that scores candidate answers on the basis of their content
overlap with the question before making the final prediction. Combined with a
strong base QAmodel, our method outperforms state-of-the-art defense
techniques, calling into question how well these techniques are actually doing
and strong these adversarial testbeds are.
