Khandelwal et al. (2020) show that a k-nearest-neighbor (kNN) component improves language modeling performance. We use this idea for open domain question answering (QA). To improve the recall of facts stated in the trainingtext , we combine BERT (Devlin et al., 2019) with a  kNN search over a large corpus . Our contributions are as follows . i) We outperform BERT on cloze-style QA by large margins without any further training. ii) We show that BERT often identifies the correct response category (e.g., central European city), but only kNN recovers the factually correct answer (e.g., " Vienna ")  .
Khandelwal et al. (2020) use a k-nearest-neighbor (kNN) component to improve language model performance. We show that this idea is beneficial for open-domain question answering (QA). To improve the recall of facts encountered during training , we combine BERT (Devlin et al., 2019) with a traditional information retrieval step (IR) and a kNN search over a large datastore of an embedded text collection . Our contributions are as follows : i) BERT-kNN outperforms BERT on cloze-style QA by large margins without any further training. ii) We show that BERT often identifies the correct response category (e.g., US city), but only kNN recovers the factually correct answer (e.g., " Miami ") . iii) Compared to BERT, BERT-kNN excels for rare facts. iv) BERT-kNN can easily handle facts not covered by BERT's training set, e.g., recent events .
R
show that
use
R
improves language modeling
to improve language model
R
use this idea for open domain
show that this idea is beneficial for open-domain
R
stated in the trainingtext
encountered during training
A
traditional information retrieval step (IR) and a
R
corpus
datastore of an embedded text collection
R
. i) We outperform
: i) BERT-kNN outperforms
R
central European
US
R
Vienna
Miami
A
. iii) Compared to BERT, BERT-kNN excels for rare facts. iv) BERT-kNN can easily handle facts not covered by BERT's training set, e.g., recent events