  We release FoolMeTwice (FM2 for short), a large dataset of challenging
entailment pairs collected through a fun multi-player game. Gamification
encourages adversarial examples, drastically lowering the number of examples
that can be solved using "shortcuts" compared to other popular entailment
datasets. Players are presented with two tasks. The first task asks the player
to write a plausible claim based on the evidence from a Wikipedia page. The
second one shows two plausible claims written by other players, one of which is
false, and the goal is to identify it before the time runs out. Players "pay"
to see clues retrieved from the evidence pool: the more evidence the player
needs, the harder the claim. Game-play between motivated players leads to
diverse strategies for crafting claims, such as temporal inference and
diverting to unrelated evidence, and results in higher quality data for the
entailment and evidence retrieval tasks. We open source the dataset and the
game code.
