Paper:
Language modeling tasks, in which words, or word-pieces, are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the text contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn high quality entity representations, and we demonstrate successful results on four domains: (1) existing entity-level typing benchmarks, including a 64% error reduction over previous work on TypeNet ; (2) a novel few-shot category reconstruction task; (3) existing entity linking benchmarks, where we match the state-of-the-art on CoNLLAida without linking-specific features and obtain a score of 89.8% on TAC-KBP 2010 without using any alias table, external knowledge base or in domain training data and (4) answering trivia questions, which uniquely identify entities. Our global entity representations encode fine-grained type categories, such as Scottish footballers, and can answer trivia questions such as Who was the last inmate of Spandau jail in Berlin?.

Review:
This paper proposes to learn entity representations by matching entities to the context it occurs in. It also shows that using these representations is very effective for a wide variety of down-stream entity-centric tasks such as entity typing, linking, and answering entity centric trivia questions. They train the model using a corpus of entity linked Wikipedia contexts (sentences unto length 128 tokens). The context is encoded with a BERT model and the CLS representation is used as the representation of context. After obtaining the representation, they train the entity embedding (present in the sentence) to be similar to the context embedding. They test their embeddings on few down-stream entity-centric tasks — linking, typing and trivia question answering.
Strengths:
1. They try the entity representations on a wide variety of entity-centric tasks and get reasonable results.
Weaknesses:
1. The biggest weakness of the paper is wrt novelty. Masking out entities and training to context is not a new idea. As pointed by the paper, Yamada et al., 2017 have a very similar objective and it is not very clear from the paper what is the additional contribution that this paper makes. Is using pretrained LMs the major difference? If not, it would have been nice to see Yamada et al’s results with BERT. Over all, this paper needs to make its own contribution clear compared to Yamada et al., 2017.
2. The paper needs to be written more clearly at several places. Few examples are, even though in entity linking results (Table 1) the model achieves 83.0 with other papers achieving 90.9. I didnt see a discussion on how to close the gap. Even in the coNLL benchmark, the initial results of the paper is significantly behind. Claims like “CoNLL -Aida is known to be restricted and idiotic-synctatic domain” should be backed by detailed analysis or atleast a citation. Even after finetuning on the CoNLL benchmark, the result is 2.2 points behind state of the art and no discussions have been provided. As a result, I think the entity linking section needs major re-writing and explanation of the results.
3. The paper makes an interesting observation that masking of entities is better for typing tasks and it affects linking performance, because spelling features are really important for linking. It would be interesting to see a discussion on what could be done to remedy this. Because if we have to retrain entity embeddings for different tasks, then it goes against the hypothesis of the paper which is to use entity representations for a wide variety of down-stream tasks.
4. I found it confusing to read the setup in sec 5.4. especially where it says we represent each category with three random exemplars. Initially I thought 3 randomly sampled entities formed a category, which didnt make sense, but from figure 3, I think I understood that you first pick a category from Typenet and Wikipedia and then 3 entities are sampled from there. Is that correct? Regarding the results, can the poor results of Yamada et al., can be understood by the fact that it was trained using smaller number of categories? Also, why are the numbers wrt All entities left blank in Table 4. Given that your model is similar, I am assuming its easy to retrain Yamada et al and test it on the all entities benchmark?