All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing (2025)

Abstract: Due to their capacity to acquire world knowledge from large corpora, pre-trained language models (PLMs) are extensively used in ultra-fine entity typing tasks where the space of labels is extremely large. In this work, we explore the limitations of the knowledge acquired by PLMs by proposing a novel heuristic to approximate the pre-training distribution of entities when the pre-training data is unknown. Then, we systematically demonstrate that entity-typing approaches that rely solely on the parametric knowledge of PLMs struggle significantly with entities at the long tail of the pre-training distribution, and that knowledge-infused approaches can account for some of these shortcomings. Our findings suggest that we need to go beyond PLMs to produce solutions that perform well for infrequent entities.

## introduction ultra-fine entity typing (ufet) is the task of inferring the type of an entity with high specificity @xcite . for example, in the sentence "barack obama attended biden's presidential inauguration.", the entity "barack obama" would have a coarse entity-type of "person", but a more specific, ultra-fine entity type such as "politician", "democrat" or "ex-president". previous work on ufet relies mostly on pretrained language models (plms) to elicit entity types @xcite @xcite . the common approach is to position the ufet task close to the learning objective of the plm. for example, @xcite exploit the masked language modeling objective by appending an entity-mention and mask to a sentence, allowing bert to fill in the mask to elicit a type (e.g., barack obama attended biden's presidential inauguration, barack obama is a @xcite ). such approaches are attractive because they benefit from the large amount of pre-training data the plm has seen to make common associations, thus helping them determine the most likely type of entity based on the context provided. albeit successful, plms are limited by the data that they have been exposed to, so their reliability can be affected when working with infrequently used language constructions such as rarely/never seen entities. since plms rarely encounter these entities, they have fewer opportunities to capture knowledge about them compared to frequent entities. past work has mostly overlooked this issue, assuming that the embedding space will be enough to capture similarities between rare and less rare entities and generalize across them. studies that have looked at infrequent entities usually characterize them in terms of their availability in task-specific training data @xcite , rather than the data used to pre-train plms. in this paper, we investigate the extent to which plm-based methods struggle when determining the ultra-fine grained type of entities that occur infrequently in their pre-training data. this is a challenging undertaking, as the data used for pretraining plms are often unavailable @xcite . to address this challenge, we set out to answer the following research questions: rq1: do internet search hits provide a good proxy for estimating relative entity frequency? to answer this question, we follow the conventional wisdom that modern plms have been trained on "all of the internet", and we estimate the frequency of an entity by querying the google search api to obtain the number of documents containing that entity. we validate this proxy by correlating it with real-world datasets known to be used in plm pretraining, acknowledging that such disclosures are limited to only a few models. rq2: do internal model representations reflect the entity frequencies in the pre-training data? to answer this question, we measure the correlation between the number of google search hits obtained for an entity and the model's probability of eliciting that entity in numerous contexts. we perform this test across various plms, including both masked lms and causal lms, and find a strong correlation across the board. rq3: do plms struggle to type entities that are in the long tail of the pre-training distribution? finally, we design a benchmark to answer this question. * to do this, we divide entities in various ufet datasets into bins based on their resulting frequencies from our google search. our hypothesis is that the bins consisting of the least frequent entities "in the wild" will be much harder to predict than the rest. we test this assertion by looking at the performance of two types of published approaches for ufet; plm-only approaches, which we expect to struggle at the long tail of the pre-training distribution, and knowledgeinfused approaches, which we expect can level the playing field across frequencies by leveraging external sources of information. our results confirm our hypothesis, suggesting that we need to go beyond plms to produce solutions that perform well for rare, new, or infrequent entities. ## related work ufet was proposed to generate free-form noun phrases that appropriately describe the type of a target entity @xcite . ufet has found several downstream applications, such as coreference resolution @xcite , entity linking @xcite and relation extraction @xcite . plm solutions for ufet. plms have been shown to capture world knowledge in their parameter space @xcite . this ability has allowed them to perform well on 'fill-in-the-blank' problems, where the goal is to elicit an answer to a query based on the context provided. many ufet approaches have capitalized on this ability; @xcite looked at introducing hearst patterns with [mask] tokens to describe entity types using bert, while @xcite primed bert to produce an ultra-fine type by appending the entity mention and a [mask] token to a sentence. while these methods report some of the best metrics for ufet, they do not explore how their approaches fare on infrequent, rare entities that plms may not have been sufficiently * data and code available at: https://github.com/ blast-cu/all-entities-are-not-created-equal exposed to. in this work, we explore the effect of entity representation strength on a variety of plms in order to determine whether rarer entities cause significant issues for plm-based approaches. knowledge infused ufet. the ufet dataset has 9 general, 121 fine, and 10, 201 ultra-fine types. while a large type vocabulary and the scarcity of annotated examples per type make this task especially challenging, type labels often consist of rich semantics. @xcite leverage the type semantics and formulate the task as an nli problem. others have exploited the dependencies between labels @xcite and hierarchies within types @xcite to supplement the plm objectives. such techniques often perform significantly better than plm-only approaches. approaches to estimate entity frequencies in large corpora. while there have been efforts to efficiently index large pre-training corpora to better estimate entity frequencies @xcite , we argue that there are two clear limitations with this approach: (1) the assumption that training data for models is always available and (2) the need to re-index datasets as they are updated over time. therefore, we need simple proxies that can approximate the entity distribution "seen" by plms without direct access to their training data. ## experimental design our study is composed of three experiments. first, we evaluate a proxy to establish the relative frequency of entities in pre-training data using search engine api indexing. then, we examine whether parametric representations match these frequencies by correlating the likelihood of a plm suggesting a target entity to how the term is indexed on the web. lastly, we select seven models to perform ultra-fine entity typing; three based on plm objectives, three that leverage some additional knowledge, and one simple baseline. we measure typing performance across entity groups of varying frequency across the internet data. establishing the long tail (rq1, rq2) the long tail of an entity distribution is characterized by a large number of entities that are rare to encounter in the real world. assuming that the internet is a fairly balanced representation of a real-world distribution, we use the internet search hits as a proxy for frequency. we quantify the occurrence "in the wild" of each entity in our dataset by performing strict searches using the google search api. as most datasets used to pre-train modern plms come from a subset of the internet, we hypothesize that such "long-tail" entities would be underrepresented in the parameter space of these models. to test this hypothesis, we query models and assess whether an entity is underrepresented based on its ability to be predicted or reconstructed accurately. we then measure the correlation between internet frequencies and model-estimated probabilities. to perform model estimations reliably, we must balance different considerations. first, given that plms learn word representations from in-context examples, we need a representative set of incontext examples for each of the entities that we want to estimate. to do this, we prompt llama3-8b-instruct to generate 10 different sentences that include the target entity. second, we need to consider the training objective of the plm when querying it for the probability of a given entity in a given context. to recover entity probabilities with masked lms, we first replace entity tokens with [mask] tokens. then, we generate a probability distribution over all candidate tokens. to deal with causal lms, we reframe the task as a fill-inthe-blank problem. we provide the model with a prompt (see app e) and compute the probabilities through a conditional generation process. in both cases, we use the probability that the model assigns to the tokens of the target entity to represent its salience in the model's parametric knowledge. more details about the probability estimation process can be found in app. d. ## evaluation and discussion long tail analysis (rq1, rq2) we compare average plm probability estimates with the number of hits the search engine api has for the target entity. we do this for three plms with both mlm and causal lm learning objectives: bert (devlin et al., 2019), bart @xcite and qwen3 @xcite . we calculate the probability assigned by the plms per in-context example using the method described in section 3, and take the average across all sentences for its respective entity. we perform a correlation analysis between plm probability estimates and the entity frequencies obtained from the google search api and observe high spearman correlation coefficients (0.885 for bert, 0.716 for bart and 0.897 for qwen3). to visualize this, we plot the hits from the search engine api against the average probability for the entity obtained by each model in app. f. the high correlation between the plm probability estimates and the number of api hits supports our hypothesis: entities that occur more/less frequently in the real world are more/less salient in plms. temporal shifts. we recognize that the internet is constantly evolving and that temporal dynamics could potentially alter the distribution of entities. for this reason, we performed our analysis with api data capped at 2018 and at 2024, and found the results to be consistent over time, with minor changes in correlation coefficients (see app. b). search api vs. real pre-training datasets. we test the validity of using internet counts to estimate the pre-training distribution by performing a correlation study between search api hits and real-world pre-training datasets such as bookcorpus @xcite , c4 @xcite , pile @xcite , redpajama @xcite , dolma-v1.7 @xcite . we estimate the entity counts for bookcorpus by performing strict searches. for pile, we rely on the infini-gram-mini hosted api @xcite . for the remaining datasets (c4, redpajama, dolma); we use the infini-gram hosted api @xcite . n-gram counts in infinigram and infini-gram-mini are case-sensitive and therefore noisy. regardless, we see high spearman coefficients (>0.9) for all of them. while the correlation coefficient for bookcorpus is significantly lower, we find that the largest contributing factor is its much smaller scale. as a result, many entities are not present in bookcorpus. when restricting only to entities that are present in bookcorpus, we find that the correlation jumps as high as 0.88. for completeness, we also report the spearman correlation coefficients between the frequency of ufet entities in different pre-training datasets and the average entity recovery probability given surrounding context (see app. a). typing performance (rq3) we benchmark the seven models introduced in sec. 3: ufet-lstm @xcite , few-shot llama3 @xcite /qwen3 @xcite , bert (plm-based), and lite @xcite , lrn @xcite and box4types @xcite (knowledge-infused), and plot results in fig. 2 . we find that most of the examined approaches perform better in determining the ultra-fine entity type when the target entity is more frequent (bin4), and worse when the entity is rarer (bin1). we note that most plm approaches show a major decline in performance as we go down the bins (or underperforming completely, as in the case of bert for ontonotes). this performance decline is most underlined when moving from bin 4 to bin 3. looking at the knowledge-infused approaches, we find that the enrichment from auxiliary tasks or label dependencies helps overcome the overreliance on learned entity representations for bin 1. lite performs the best, beating the rest by achieving an f 1 score of 45.6 on infrequent entities and 54.2 on frequent entities for ufet, and an f 1 score of 77.8 on infrequent entities and 81.8 on frequent entities for ontonotes. this further highlights the advantage of the auxiliary task (nli), which allows the model to transfer more knowledge than approaches that only rely on the plms parameters. ## conclusion and future work we showed the effectiveness of internet search api hits as a proxy for entity frequency in large pretraining corpora (rq1). we also showed that, as expected, this distribution significantly affects entity representations for different plms (rq2). finally, we measured the performance of several plmbased and knowledge-infused entity typing systems on entities with different frequencies and found that all models performed worse on rarer/lessprobable entities (rq3). however, we found that all knowledge-infused systems are considerably more robust to frequency shifts than plm-based approaches. our results show that for plms to fulfill their promise for long-tail entity typing, we need better strategies to inject knowledge about rare entities into plms by using external resources and other forms of domain knowledge.