Recent progress in pretrained language model "solved" many reading comprehension benchmark datasets. Yet information-seeking Question Answering (QA) datasets , where questions are written without the evidence document , remain unsolved . We analyze two such datasets ( Natural Questions and TyDi QA ) to identify remaining headrooms : paragraph selection and answerability classification , i.e. determining whether the paired evidence document contains the answer to the query or not. In other words, given a gold paragraph and knowing whether it contains an answer or not, models easily outperform a single annotatorin both datasets. After identifying unanswerability as a bottleneck, we further inspect what makes questions unanswerable. Our study points to avenues for future research  , both for dataset creation and model development.
Recent pretrained language models "solved" many reading comprehension benchmarks , where questions are written with access to the evidence document . However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging . We analyze why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on Natural Questions and TyDi QA . Our controlled experiments suggest two headrooms -- paragraph selection and answerability prediction , i.e. whether the paired evidence document contains the answer to the query or not. When provided with a gold paragraph and knowing when to abstain from answering, existing models easily outperform a human annotator. However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per-category answerability prediction, revealing issues in the current dataset collection as well as task formulation. Together, our study points to avenues for future research in information-seeking question answering , both for dataset creation and model development.
R
progress in pretrained language model
pretrained language models
R
benchmark datasets. Yet information-seeking Question Answering (QA) datasets
benchmarks
R
without
with access to
R
, remain unsolved
. However, datasets containing information-seeking queries where evidence documents are provided after the queries are written independently remain challenging
R
two such datasets (
why answering information-seeking queries is more challenging and where their prevalent unanswerabilities arise, on
R
) to identify remaining headrooms :
. Our controlled experiments suggest two headrooms --
R
classification
prediction
D
determining
R
In other words, given
When provided with
R
whether it contains an answer or not,
when to abstain from answering, existing
R
single annotatorin both datasets. After identifying unanswerability as a bottleneck, we further inspect what makes questions unanswerable. Our
human annotator. However, predicting answerability itself remains challenging. We manually annotate 800 unanswerable examples across six languages on what makes them challenging to answer. With this new data, we conduct per-category answerability prediction, revealing issues in the current dataset collection as well as task formulation. Together, our
A
in information-seeking question answering