Recall, Expand, and Multi-Candidate Cross-Encode: Fast and Accurate Ultra-Fine Entity Typing (2023)

Abstract: Ultra-fine entity typing (UFET) predicts extremely free-formed types (e.g., president, politician ) of a given entity mention (e.g., Joe Biden ) in context. State-of-the-art (SOTA) methods use the cross-encoder (CE) based architecture. CE concatenates a mention (and its context) with each type and feeds the pair into a pretrained language model (PLM) to score their relevance. It brings deeper interaction between the mention and the type to reach better performance but has to perform N (the type set size) forward passes to infer all the types of a single mention. CE is therefore very slow in inference when the type set is large (e.g., N=10k for UFET). % Cross-encoder also ignores the correlation between different types.To this end, we propose to perform entity typing in a recall-expand-filter manner. The recall and expansion stages prune the large type set and generate K (typically much smaller than N ) most relevant type candidates for each mention. At the filter stage, we use a novel model called {pasted macro ‘NAME’} to concurrently encode and score all these K candidates in only one forward pass to obtain the final type prediction. We investigate different model options for each stage and conduct extensive experiments to compare each option, experiments show that our method reaches SOTA performance on UFET and is thousands of times faster than the CE-based architecture. We also found our method is very effective in fine-grained (130 types) and coarse-grained (9 types) entity typing. Our code is available at {pasted macro ‘CODE’}.

## introduction ultra-fine entity typing (ufet) @xcite aims to predict extremely fine-grained types (e.g., president, politician) of a given entity mention within its context. it provides detailed semantic plm [cls] mention & context [sep] h cls 0/1 entailment for plm [cls] mention & context [sep] understandings of entity mentions and is a fundamental step in fine-grained named entity recognition @xcite . it can also be utilized to assist various downstream tasks such as relation extraction @xcite , keyword extraction @xcite and content recommendation @xcite . most recently, the cross-encoder (ce) based method @xcite achieves the sota performance in ufet. specifically, @xcite proposes to treat the mention with its context as a premise, and each ultra-fine-grained type as a hypothesis. they then concatenate them together as input and feed it into a pretrained language model (plm) (e.g., roberta @xcite ) to score the entailment between the mention-type pair as illustrated in figure 1 @xcite . compared with the traditional multi-label classification method (shown in figure 1 (a)) that simultaneously scores all types using the same mention representation, ce has the advantage of incorporating type semantics in the encoding and inference process (by taking words in type labels as input) and enabling deeper interactions between each type and the mention via cross-encoding. however, the ce-based method is slow in inference because it has to enumerate all the types (up to 10k types in ufet) and score entailment for each of them given the mention as a premise. there is also no direct interaction between types in ce for modeling correlations between types (e.g., one has to be a person if he or she is categorized as a politician), which has been proven to be useful in previous works @xcite . to this end, we propose a recall-expand-filter paradigm for ufet (illustrated in figure 2 ) for faster and more accurate ultra-fine entity typing. as the name suggests, we first train a multi-label classification (mlc) model to efficiently recall top candidate types, which reduces the number of potential types from 10k to hundreds. as the mlc model recalls candidates based on representations learned from the training data, it may not be able to recall candidates that are scarce or unseen in the training set. consequently, we apply a type candidate expansion step utilizing lexical information and weak supervision from masked language models @xcite to improve the recall rate of the candidate set. finally, we propose a novel method called multi-candidate cross-encoder (mcce) to concurrently encode and filter the expanded type candidate set. different from ce, mcce concatenates all the recalled type candidates with the mention and its context. the concatenated input is then fed into a plm to obtain candidate representations and candidate scores. the mcce allows us to simultaneously encode and infer all the types from the candidate set and is thus much faster than the ce-based method, but it still preserves the advantages of ce in modeling interactions between the types and the mention. concatenating all the candidates also enables mcce to implicitly learn correlations between types. the advantages of mcce over existing methods are summarized in figure 3 . experiments on two ufet datasets show that our recall-expand-filter paradigm reaches sota performance and mcce is thousands of times 2 background ## method inspired by techniques in information retrieval (larson, 2010) and entity linking (ledell et al., 2020), we decompose the inference of ufet into three stages as illustrated in figure 2: (1) a recall stage to reduce the type candidate number (e.g., from @xmath0. (2) an expansion stage to improve the recall rate by incorporating lexical information using exact matching and weak supervision @xcite from large pretrained language models such as bert-large @xcite . (3) a filter stage to filter the expanded type candidates to obtain the final prediction. for the filter stage, we propose an efficient model, multi-candidate cross-encoder (mcce), to concurrently encode and filter type candidates of a given mention with only a single forward pass. ## experiments we conduct experiments on two ultra-fine entity typing datasets, ufet (english) and cfet (chinese). their statistics are shown in ## analysis 6.1 importance of expansion stage we perform an ablation study on the importance of the expansion stage by comparing the results of mcce-s with and without the expansion stage in ## related work while writing this paper, we noticed that a paper @xcite that has similar ideas to our work was submitted to arxiv. they target the task of selecting from multiple options, of which ufet is a special case. their second model, parallel-te, is similar to our mcce-b. in addition, when applying their model to ufet in their experiments, they prune types in a similar manner to our recall stage. below we summarize the differences between our method and theirs when applied to ufet. ## conclusion we propose a recall-expand-filter paradigm for ultra-fine entity typing. we