DOCENT : Learning Self-Supervised Entity Representations from Large Document Collections (2021)

Abstract: This paper explores learning rich self-supervised entity representations from large amounts of associated text. Once pre-trained, these models become applicable to multiple entity-centric tasks such as ranked retrieval, knowledge base completion, question answering, and more. Unlike other methods that harvest self-supervision signals based merely on a local context within a sentence, we radically expand the notion of context to include any available text related to an entity. This enables a new class of powerful, high-capacity representations that can ultimately distill much of the useful information about an entity from multiple text sources, without any human supervision. We present several training strategies that, unlike prior approaches, learn to jointly predict words and entities – strategies we compare experimentally on downstream tasks in the TV-Movies domain, such as MovieLens tag prediction from user reviews and natural language movie search. As evidenced by results, our models match or outperform competitive baselines, sometimes with little or no fine-tuning, and are also able to scale to very large corpora. Finally, we make our datasets and pre-trained models publicly available. This includes Reviews2Movielens, mapping the ~1B word corpus of Amazon movie reviews (He and McAuley, 2016) to MovieLens tags (Harper and Konstan, 2016), as well as Reddit Movie Suggestions with natural language queries and corresponding community recommendations.

## introduction much of the online information describing entities in domains such as music, movies, venues or review 1: "this movie develops its power best if you don't try to look out for the "real" and "true" events behind the four versions of the narration... shown in a very intelligent and artistic way, no silly plot-twists, no explanation in the end -it is open to your fantasy... "$movie" is an important piece of cinematic storytelling and a really interesting way to reflect on the origin of tales... some scenes even remind me of andrej tarkovskijs intensive style..". review 2: "just rented this, and at first i didn't like very much, but then it starts to sink in for how good it is, the acting is great especially toshiro mifune, it was shot very good for an older movie... it's #62 on the top 250" review 3: "saw this movie at my local video store... was placed on a waiting list, but when i returned to check it out the video store had closed down over night. actually whent out of business" ... more reviews ... ## self-supervised entity representations inspired by the success of self-supervised language models, we seek to extend them to jointly compute text and entity representations. recall that our input is a set of entities e where for every entity @xmath0, we have a collection of sentences, denoted by s e , from all documents related to e. intuitively, we want the representation of e to be influenced by each associated sentence @xmath1, and vice versa. to that end, we explore two (self-) supervision signals: p (e | s) and p (s | e). ## tasks in this section, we define the three tasks used to evaluate pre-trained entity representations. ## related work much of the prior art in text-based entity understanding is motivated by the entity linking (el) problem: predict a unique entity from its mention in text, assuming a single right answer. by contrast, tasks like entity retrieval and tag prediction imply multiple valid matches and emphasize understanding entities through the prism of their attributes, expressed in natural language. still, recent el works propose dual encoder approaches similar to ours @xcite @xcite @xcite @xcite @xcite , with @xcite already discussed in section 2.1. dual encoders have also been explored in zero-shot scenarios @xcite @xcite , with entity embeddings computed dynamically based on metadata such as dictionary definitions, entity name and/or category. others incorporate entity representations directly in the transformer by retrieving from an external memory @xcite . while clearly useful for el, e.g., in sentences with multiple entity mentions, the benefits to our applications are unclear. finally, there is ernie @xcite ) -a language model trained with awareness of entity mentions. alas, the lack of explicit entity representation limits its use in our tasks. ## conclusion & future work this paper proposes a family of models to learn self-supervised entity representations from large document collections. we motivate these dedicated representations by contrasting them with naive textas-a-proxy approaches, with clear gains on entitycentric tasks such as natural language search and movie tag prediction. we then show that achieving superior performance requires optimizing both p (entity | t ext) and p (t ext | entity)-in contrast to the baseline relic model (and similar prior dual encoders) having only a single objective. to that end, we propose two novel models and study them in zero-shot, few-shot and supervised settings. we match or outperform competitive baselines, where available, with little or no fine-tuning. future work as shown qualitatively in sec. 3.3, docent has the potential for being a hybrid approach to bridge entity retrieval and recommendation, an application worth exploring in depth (e.g., on the movielens recommendation task which can be readily integrated with docent thanks to reviews2movielens). a larger entity retrieval study with heterogeneous entity types is another useful direction. lastly, extending docent to additional entity understanding tasks such as qa and summarization is yet another promising avenue.