Paper:
Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.

Review:
This paper aims to incorporate world knowledge for the pretraining approach so that (1) pretrained models contain useful information about the world, and (2) benefit downstream NLP tasks. The paper does so by introducing the objective which distinguishes the groundtruth entity and the false entity in the Wikipedia text. This was carefully done by detecting entities in the text, find the corresponding entity in Wikidata, randomly choose another entity which has the same type as the original entity, and make sure this doesn’t happen for neighboring entities or too much in order to avoid context change. Adding this objective to the original masked LM objective, this pretrained model is shown to be effective and outperform baselines significantly in many tasks such as zero-shot fact completion, question answering, and fine-grained entity typing.
 
Particularly, I appreciate (1) the fact that they compare with other knowledge-aware pretrained models such as ERNIE, and (2) their ablation in Table 6 which compares only knowledge learning objective, more masking or finetuning with knowledge learning objective starting from MLM instead of multi-task learning (actually, is BERT+1M MLM updates a correct term? Since MLM is masked LM, shouldn’t it be BERT+1M knowledge learning?): it clearly shows that it is important to do both MLM and knowledge learning, appropriate ratio for masking, and multi-task learning between MLM and knowledge learning instead of some kind of pretrain-finetune approach.
Some marginal concerns I have is that some settings for downstream tasks are not clear. For example, for WebQuestions the number of validation examples is not specified, but I believe they should have split the train set into train/valid for development (such as early stopping or hyperparameter tuning), and conventionally people have split the train set into 90/10 for train/valid. In addition, as far as I know, TriviaQA has a bunch of settings such as Wiki setting, Web setting, unfiltered setting and open setting. What exactly is this setting? The paper mentions they follow Lin et al. 2018, but the statistics shown in Table 2 are different from their statistics. In addition, the authors compare results in different settings in Table 4 (for example, ORQA is in open setting, while I think the setting in this paper is not).
Some clarification questions:
1. I understand Wikipedia anchor links means hyperlinks. How do you know that the mention with a hyperlink is always an entity? Also, there are many cases that the mention with the hyperlink does not match in meaning with the linked article. For example, in “https://en.wikipedia.org/wiki/Barack_Obama”, there is a sentence “He was elected over Republican John McCain and was inaugurated on January 20, 2009” where “elected” is linked to “https://en.wikipedia.org/wiki/2008_United_States_presidential_election” which is probably not what we want.
2. If there are same mentions in the text chunk, are all of them replaced? Probably with the same negative entity?
3. How often were entities to replace chosen? E.g. is one entity in the text chunk replaced at a time? Or multiple ones?
4. How do you look up the type of the entity? For instance, in the example of “Marvel Comics” illustrated in Figure 1, I see there are two triples, <Marvel Comics, instance of, business> and <Marvel Comics, instance of, book publishing company>. Is "instance of" used? How did you choose “book publishing company” over “business”? Or is it randomly chosen? I think ideally more fine-grained type should be chosen, but wonder if there is a way to find which one is more fine-grained.
5. Perhaps not necessary, but I wonder what happens if an entity is detected, and there is another masked LM objective which only predicts this entity, without the process of looking up its type and randomly choose the negative entity.
- I’m curious because I’m not sure how much effect do negative entities have. Although they are the same type of entities, it’s hard that the chosen entity is a strong negative entity. For instance, Barack Obama (Q76) is an instance of ‘human’, and I don’t think another entity of human that is randomly chosen is helpful. A similar observation was found in WikiHop dataset, which is a multi-choice QA dataset that all negative candidates are guaranteed to be the same type, but still question-candidates-only baseline (without context paragraphs) outperforms state-of-the-art models (https://openreview.net/forum?id=B1lf43A5Y7). It indicates that entities with the same type are not strong negative candidates.
- For this reason, I think it is possible that negative entities do not help, but the fact that the training objective is more entity-centric helps the performance on various different downstream tasks. That’s why I think the ablation with entry-centric masked LM can be good to see.
- Of course, this is just one of my hypotheses, and I believe the results in this paper are significant regardless, but I just wonder what authors think about this.