Paper:
The open-domain question answering (OpenQA) task aims to extract answers that match specific questions from a distantly supervised corpus. Unlike supervised reading comprehension (RC) datasets where questions are designed for particular paragraphs, background sentences in OpenQA datasets are more prone to noise. We observe that most existing OpenQA approaches are vulnerable to noise since they simply regard those sentences that contain the answer span as ground truths and ignore the plausible correlation between the sentences and the question. To address this deficiency, we introduce a unified and collaborative model that leverages alignment information from query-sentence pairs in a small-scale supervised RC dataset and aggregates relevant evidence from distantly supervised corpus to answer open-domain questions. We evaluate our model on several real-world OpenQA datasets, and experimental results show that our collaborative learning methods outperform the existing baselines significantly.

Review:
This paper aims for open-domain question answering with distant supervision. First, the authors proposed an aggregation-based openQA model with sentence discriminator and sentence reader. Second, they use a semantic labeler to handle distant supervision problem by utilizing other span supervision tasks, and propose two different denoising methods. They run experiments on 3 open-domain QA datasets and achieve SOTA.
Strengths
1) Their semantic labeler and exploration of two different denoising methods are interesting and meaningful.
2) They conducted experiments on 3 widely-used open-domain datasets, and the performance gain is impressive.
Weakness
Although there is an impressive performance gain, the contribution of the paper seems to be marginal.
1) First of all, it is hard to say there is a contribution to the idea of sentence discriminator and sentence reader — people have used this framework for large-scale QA a lot. Also, the architecture of the models in this paper are almost identical to Chen et al (ACL 2017) and Lin et al (ACL 2018).
2) Thus, the contribution is more on semantic labeler and denoising method. However, this contribution is marginal as well since its role is almost the same as sentence discriminator plus pretraining methods which have widely used already.
Questions
1) What exactly is the difference between semantic labeler and sentence discriminator? For me, it seems like both of them label each sentence `yes` or `no`. My thought is sentence discriminator is only trained on the target dataset (distant supervision dataset) while semantic labeler is also trained (either jointly or separately) trained on the source dataset (span supervision dataset). (If my thought is wrong, please let me know, I would like to update my score.)
2) Chen et al (ACL 2017) have shown that pretraining QA model on span supervision dataset (SQuAD) is effective to train the model on distant supervision dataset. Similarly, Min et al (ACL 2018) have pretrained both QA model and sentence selector on SQuAD. While I think pretraining sentence selector on SQuAD is almost identical to sentence labeler with SSL method, could you give exact comparison of these different methods? For example, remove sentence labeler, and pretrain both sentence discriminator and reader on SQuAD, or jointly train them on SQuAD & target dataset.
Marginal comments
1) At the beginning of Section 2.4.1, it says the semantic labeler is able to transfer knowledge from the span supervised data — however, the authors should be careful since people usually refers to `knowledge` as an external knowledge. This method is more like better learning of accurate sentence selection, not transferring knowledge.
2) Please mention the TriviaQA data you used is Wikipedia domain, since there are two different domains (Wikipedia and Web).
3) In References section, the conference venues in many papers are omitted.
Overall comments
The paper explored several different methods to deal with distant supervision via sentence labeling, and I really appreciate their efforts. While the result is impressive, the idea in the paper is similar to the methods that have widely used already.