The measurement of true progress in multihop question-answering has been muddled by the strong ability of models to exploit artifacts and other reasoning shortcuts. Models can produce the correct answer, and even independently identify the supporting facts, without necessarily connecting the information between the facts. This  defeats the purpose of building multihop QA datasets. We make three contributions towards addressing this issue . First, we formalize this form of disconnected reasoning and propose contrastive support sufficiency as a better test of multifact reasoning. To this end , we introduce an automated sufficiency-based dataset transformation that considers all possible partitions of supporting facts, capturing disconnected reasoning. Second, we develop a probe to measure how much can a model cheat (via non-multifact reasoning) on existing tests and our sufficiency test. Third, we conduct experiments using a transformer based model (XLNet), demonstrating that the sufficiency transform not only reduces the amount of non-multifact reasoning in this model by 6.5\% but is also harder to cheat -- a non-multifact model sees a 20.8\% (absolute) reduction in score compared to previous metrics .
Has there been real progress in multi-hop question-answering ? Models often exploit dataset artifacts to produce correct answers, without connecting information across multiple supporting facts. This limits our ability to measure true progress and defeats the purpose of building multihop QA datasets. We make three contributions towards addressing this . First, we formalize such undesirable behavior as disconnected reasoning across subsets of supporting facts. This allows developing a model-agnostic probe for measuring how much any model can cheat via disconnected reasoning. Second, using a notion of contrastive support sufficiency , we introduce an automatic transformation of existing datasets that reduces the amount of disconnected reasoning. Third, our experiments demonstrate that there hasn't been much progress in multifact reasoning. For a recent large-scale model (XLNet), we show that only 18\% of its answer score is obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline. Our transformation shows a substantial reduction in disconnected reasoning (nearly 19 points in answer F1). It is complementary to adversarial approaches, yielding further reductions in conjunction .
R
The measurement of true progress in multihop
Has there been real progress in multi-hop
R
has been muddled by the strong ability of models to exploit artifacts and other reasoning shortcuts. Models can produce the correct answer, and even independently identify the supporting facts, without necessarily connecting the information between the
? Models often exploit dataset artifacts to produce correct answers, without connecting information across multiple supporting
A
limits our ability to measure true progress and
D
issue
R
this form of disconnected reasoning and propose
such undesirable behavior as disconnected reasoning across subsets of supporting facts. This allows developing a model-agnostic probe for measuring how much any model can cheat via disconnected reasoning. Second, using a notion of
D
as a better test of multifact reasoning. To this end
R
automated sufficiency-based dataset transformation that considers all possible partitions of supporting facts, capturing
automatic transformation of existing datasets that reduces the amount of
R
Second, we develop a probe to measure how much can a model cheat (via non-multifact reasoning) on existing tests and our sufficiency test. Third, we conduct experiments using a transformer based
Third, our experiments demonstrate that there hasn't been much progress in multifact reasoning. For a recent large-scale
R
demonstrating that the sufficiency transform not only reduces the amount of non-multifact reasoning in this model by 6.5\% but is also harder to cheat -- a non-multifact model sees a 20.8\% (absolute) reduction in score compared to previous metrics
we show that only 18\% of its answer score is obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline. Our transformation shows a substantial reduction in disconnected reasoning (nearly 19 points in answer F1). It is complementary to adversarial approaches, yielding further reductions in conjunction