Original: A UTOREGRESSIVE E NTITY R ETRIEVAL
Revised: A UTOREGRESSIVE E NTITY R ETRIEVAL

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Entities are at the center of how we represent and aggregate knowledge.
Revised: Entities are at the center of how we represent and aggregate knowledge.

Original: For instance, Encyclopedias such as Wikipedia are structured by entities (e.g., one per
Revised: For instance, Encyclopedias such as Wikipedia are structured by entities (e.g., one per

Original: Wikipedia article).
Revised: Wikipedia article).

Original: The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering.
Revised: The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering.

Original: One way to understand current approaches is as classiﬁers among atomic labels, one for each entity.
Revised: One way to understand current approaches is as classiﬁers among atomic labels, one for each entity.

Original: Their weight vectors are dense entity representations produced by encoding entity meta information such as their descriptions.
Revised: Their weight vectors are dense entity representations produced by encoding entity meta information such as their descriptions.

Original: This approach leads to several shortcomings: (i) context and entity afﬁnity is mainly captured through a vector dot product, potentially missing ﬁne-grained interactions between the two; (ii) a large memory footprint is needed to store dense representations when considering large entity sets; (iii) an appropriately hard set of negative data has to be subsampled at training time.
Revised: This approach leads to several shortcomings: (i) context and entity afﬁnity is mainly captured through a vector dot product, potentially missing ﬁne-grained interactions between the two; (ii) a large memory footprint is needed to store dense representations when considering large entity sets; (iii) an appropriately hard set of negative data has to be subsampled at training time.

Original: In this work, we propose
Revised: In this work, we propose

Original: GENRE, the ﬁrst system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion and conditioned on the context.
Revised: GENRE, the ﬁrst system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion and conditioned on the context.

Original: This enables us to mitigate the aforementioned technical issues since: (i) the autoregressive formulation allows us to directly capture relations between context and entity name, effectively cross encoding both; (ii) the memory footprint is greatly reduced because the parameters of our encoder-decoder architecture scale with vocabulary size, not entity count; (iii) the exact softmax loss can be efﬁciently computed without the need to subsample negative data.
Revised: This enables us to mitigate the aforementioned technical issues since: (i) the autoregressive formulation allows us to directly capture relations between context and entity name, effectively cross encoding both; (ii) the memory footprint is greatly reduced because the parameters of our encoder-decoder architecture scale with vocabulary size, not entity count; (iii) the exact softmax loss can be efﬁciently computed without the need to subsample negative data.

Original: We show the efﬁcacy of the approach, experimenting with more than 20 datasets on entity disambiguation, end-to-end entity linking and document retrieval tasks, achieving new state-of-the-art or very competitive results while using a tiny fraction of the memory footprint of competing systems.
Revised: We show the efﬁcacy of the approach, experimenting with more than 20 datasets on entity disambiguation , end-to-end entity linking and document retrieval tasks, achieving new state-of-the-art or very competitive results while using a tiny fraction of the memory footprint of competing systems.

Original: Finally, we demonstrate that new entities can be added by simply specifying their unambiguous name.
Revised: Finally, we demonstrate that new entities can be added by simply specifying their unambiguous name.

Revised: Code and pre-trained models at https://github.com/facebookresearch/GENRE .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ability to retrieve the correct entity from large Knowledge Bases (KBs) given a textual input is a fundamental building block for several applications (Ferrucci, 2012; Slawski, 2015; Yang et al., 2018b).
Revised: The ability to retrieve the correct entity from large Knowledge Bases (KBs) given a textual input is a fundamental building block for several applications (Ferrucci, 2012; Slawski, 2015; Yang et al., 2018a).

Original: Most commercial recommendation systems, for instance, include in their pipelines components to detect and disambiguate entity mentions in open text, in order to isolate relevant concepts from non-meaningful data (Slawski, 2015; Yang et al., 2018b).
Revised: Most commercial recommendation systems, for instance, include in their pipelines components to detect and disambiguate entity mentions in open text, in order to isolate relevant concepts from non-meaningful data (Slawski, 2015; Yang et al., 2018a).

Original: Another example are chat-bots and question answering systems, that are often equipped with retrieval components to surface speciﬁc
Revised: Another example are chat-bots and question answering systems, that are often equipped with retrieval components to surface speciﬁc

Original: KB entries (e.g., Wikipedia articles) to ﬁnd knowledge for sustaining a conversation or answering a question (Ferrucci, 2012; Chen et al., 2017; Lewis et al., 2020; Roller et al., 2020).
Revised: KB entries (e.g., Wikipedia articles) to ﬁnd knowledge for sustaining a conversation or answering a question (Ferrucci, 2012; Chen et al., 2017; Lewis et al., 2020b; Roller et al., 2020).

Original: Although there has been extensive previous work on entity retrieval (e.g. Hoffart et al., 2011; Piccinno & Ferragina, 2014; Huang et al., 2015; Le & Titov, 2018; Logeswaran et al., 2019; Broscheit, 2019; Wu et al., 2019, to name just a few) there is a common design choice to most current solutions: entities are associated with a unique atomic label and the retrieval problem can be interpreted as multi-class classiﬁcation across these labels.
Revised: Although there has been extensive previous work on entity retrieval (e.g. Hoffart et al., 2011; Piccinno & Ferragina, 2014; Huang et al., 2015; Le & Titov, 2018; Logeswaran et al., 2019; Broscheit, 2019; Wu et al., 2020, to name just a few) there is a common design choice to most current solutions: entities are associated with a unique atomic label and the retrieval problem can be interpreted as multi-class classiﬁcation across these labels.

Original: The match between input and label is calculated through a bi-encoder (Wu et al., 2019; Karpukhin et al., 2020): a dot product between dense vector encodings of the input and the entity’s meta information (such as title and description).
Revised: The match between input and label is calculated through a bi-encoder (Wu et al., 2020; Karpukhin et al., 2020): a dot product between dense vector encodings of the input and the entity’s meta information (such as title and description).

Original:  Critically, this formulation enables sub-linear search using modern maximum-inner-product-search libraries (Johnson et al., 2019) and hence supports retrieving from large entity databases.
Revised: [Figure] Critically, this formulation enables sub-linear search using modern maximum-inner-product-search libraries (Johnson et al., 2019) and hence supports retrieving from large entity databases.

Original: Unfortunately, the classiﬁer approach to entity retrieval also has several shortcomings.
Revised: Unfortunately, the classiﬁer approach to entity retrieval also has several shortcomings.

Original: First, unless a costly cross-encoder is used for re-ranking (Wu et al., 2019), the dot-product can miss ﬁne-grained [Figure] From 1905 to [Figure]
Revised: First, unless a costly cross-encoder is used for re-ranking (Wu et al., 2020), the dot-product can miss ﬁne-grained interactions between input and entity meta information (Humeau et al., 2020).

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: most

Original: [Figure]

Original: Rome

Original: [Figure]

Original: dei Normanni [Figure] interactions between input and entity meta information (Humeau et al., 2019). Second, storing dense vectors for the whole KB requires a large memory footprint, especially in real-world scenarios (i.e., ∼ 24GB to store 1024-dimensional vectors for all of the ∼ 6M Wikipedia pages), and the size linearly grows with the addition of new entities.
Revised: Second, storing dense vectors for the whole KB requires a large memory footprint, especially in real-world scenarios (i.e., ∼ 24GB to store 1024-dimensional vectors for all of the ∼ 6M Wikipedia pages), and the size linearly grows with the addition of new entities.

Original: Third, computing an exact softmax over all entities is very expensive, hence current solutions need to subsample negative data (Logeswaran et al., 2019;
Revised: Third, computing an exact softmax over all entities is very expensive, hence current solutions need to subsample negative data (Logeswaran et al., 2019;

Original: Karpukhin et al., 2020) at training time.
Revised: Karpukhin et al., 2020) at training time.

Original: Tuning an appropriately hard set of negative instances can be challenging and time-consuming.
Revised: Tuning an appropriately hard set of negative instances can be challenging and time-consuming.

Original: Finally, existing systems can suffer from a cold-start problem since they cannot represent entities about which they have not yet gathered sufﬁcient information, in the form, for instance, of a textual description or a set of relations with the existing entities.
Revised: Finally, existing systems can suffer from a cold-start problem since they cannot represent entities about which they have not yet gathered sufﬁcient information, in the form, for instance, of a textual description or a set of relations with the existing entities.

Original: The treatment of entity identiﬁers as atomic labels in a classiﬁer ignores the fact that we often have unambiguous, highly structured and compositional entity names.
Revised: The treatment of entity identiﬁers as atomic labels in a classiﬁer ignores the fact that we often have unambiguous, highly structured and compositional entity names.

Original: Wikipedia, for instance, associates unique titles to articles, 1 that may be the name of the subject or a description of its topic, as well as potential distinctive information to disambiguate 2 (see Figure 1 for some examples).
Revised: Wikipedia, for instance, associates unique titles to articles, 1 that may be the name of the subject or a description of its topic, as well as potential distinctive information to disambiguate 2 (see Figure 1 for some examples).

Original: These entity names often interact with mention contexts in a predictable and regular fashion.
Revised: These entity names often interact with mention contexts in a predictable and regular fashion.

Original: For example, often entity names are identical with the mention strings that refer to them (e.g., Fig. 1f).
Revised: For example, often entity names are identical with the mention strings that refer to them (e.g., Fig. 1f).

Original: When this is not possible, they might be composed of tokens in the context (e.g., Fig. 1b), include a type speciﬁcation that can inferred (e.g., Fig. 1a), be the translation of the string mention (e.g.,
Revised: When this is not possible, they might be composed of tokens in the context (e.g., Fig. 1b), include a type speciﬁcation that can inferred (e.g., Fig. 1a), be the translation of the string mention (e.g.,

Original: Fig. 1c), require ‘normalization’ such as referring to the correct alias of a mention (e.g., Fig. 1d), or require factual knowledge that might be stored in the parameters of a model (e.g., Fig. 1e).
Revised: Fig. 1c), require ‘normalization’ such as referring to the correct alias of a mention (e.g., Fig. 1d), or require factual knowledge that might be stored in the parameters of a model (e.g., Fig. 1e).

Original: These observations suggest that inputs could be translated into unique entity names, word by word, instead of being classiﬁed among a huge set of options.
Revised: These observations suggest that inputs could be translated into unique entity names, word by word, instead of being classiﬁed among a huge set of options.

Original: In this paper, we propose GENRE (for Generative ENtity REtrieval ), the ﬁrst entity retriever that exploit a sequence-to-sequence architecture to generate entity names in an autoregressive fashion conditioned on the context.
Revised: In this paper, we propose GENRE (for Generative ENtity REtrieval ), the ﬁrst entity retriever that exploits a sequence-to-sequence architecture to generate entity names in an autoregressive fashion conditioned on the context.

Original: Concretely, GENRE uses a transformer-based architecture, pre-trained with a language modeling objective (i.e., we use BART weights from Lewis et al. 
Revised: Concretely, GENRE uses a transformer-based architecture, pre-trained with a language modeling objective (i.e., we use BART weights from Lewis et al. (2020a))

Original: and ﬁnetuned to generate entity names.
Revised: and ﬁne-tuned to generate entity names.

Original: This architecture has been shown to retain factual knowledge to some extent (Petroni et al., 2019) and language translation skills (Radford et al., 2019) among other things, both desirable properties for an entity retriever.
Revised: This architecture has been shown to retain factual knowledge to some extent (Petroni et al., 2019) and language translation skills (Radford et al., 2019) among other things, both desirable properties for an entity retriever.

Original: Naturally, the generated output might not always be a valid entity name.
Revised: Naturally, the generated output might not always be a valid entity name.

Original: To solve this problem, GENRE employs a constrained decoding strategy that forces each generated name to be in a predeﬁned candidate set.
Revised: To solve this problem, GENRE employs a constrained decoding strategy that forces each generated name to be in a predeﬁned candidate set.

Original: The autoregressive formulation allows us to directly capture the aforementioned relations between context and entity name, effectively cross encoding both.
Revised: The autoregressive formulation allows us to directly capture the aforementioned relations between context and entity name, effectively cross encoding both.

Original: Also, the memory footprint required is orders of magnitude smaller than current systems, since the parameters of a sequence-to-sequence model scale linearly with the vocabulary size, not entity count.
Revised: Also, the memory footprint required is orders of magnitude smaller than current systems, since the parameters of a sequence-to-sequence model scale linearly with the vocabulary size, not entity count.

Original: Moreover, the exact softmax can be computed efﬁciently for each output token (i.e., all non-gold tokens are considered negative), thereby eliminating the need for negative data downsampling.
Revised: Moreover, the exact softmax can be computed efﬁciently for each output token (i.e., all non-gold tokens are considered negative), thereby eliminating the need for negative data downsampling.

Original: Finally, our model never accesses any explicit meta-information about the entity beyond their title, hence new entities can be added by simply appending their unambiguous name to the candidate set (e.g., Fig.
Revised: Finally, our model never accesses any explicit meta-information about the entity beyond their title, hence new entities can be added by simply appending their unambiguous name to the candidate set (e.g., Fig.

Original: 1b refers to an entity added after training).
Revised: 1b refers to an entity added after training).

Original: We empirically evaluate the performance of GENRE on more than 20 datasets, spanning three families of tasks: (i) entity disambiguation, using popular datasets and settings (both in and out-of– domain); (ii) end-to-end entity linking, with the GERBIL benchmarking tool (R¨oder et al., 2018), by using a novel dynamically markup-constrained decoding strategy; (iii) document retrieval, with the recently proposed KILT benchmark (Petroni et al., 2020b) which spans 5 different sub-tasks.
Revised: We empirically evaluate the performance of GENRE on more than 20 datasets, spanning three families of tasks: (i) entity disambiguation, using popular datasets and settings (both in and out-of– domain); (ii) end-to-end entity linking, with the GERBIL benchmarking tool (R¨oder et al., 2018), by using a novel dynamically markup-constrained decoding strategy; (iii) document retrieval, with the recently proposed KILT benchmark (Petroni et al., 2020b) which spans 5 different sub-tasks.

Original: Our models achieve state-of-the-art or very competitive results on nearly all datasets, often with substantial improvement (+13.7 precision points on KILT for retrieval on average).
Revised: Our models achieve state-of-the-art or very competitive results on nearly all datasets, often with substantial improvement (+13.7 precision points on KILT for retrieval on average).

Original: Further, we show that compared with recent models, GENRE requires substantially less memory ( ∼ 20 times smaller footprint on average).
Revised: Further, we show that compared with recent models, GENRE requires substantially less memory ( ∼ 20 times smaller footprint on average).

Original: Finally, we demonstrate that our model can be applied in scenarios where the only entity information available is its name.
Revised: Finally, we demonstrate that our model can be applied in scenarios where the only entity information available is its name.

Original: We organize the paper as follows: in Section 2 we describe our problem formulation.
Revised: We organize the paper as follows: in Section 2 we describe our problem formulation.

Original: Then, in
Revised: Then, in

Original: Section 3 we present GENRE and eventually in Section 4 we extensively evaluate our method on the aforementioned settings.
Revised: Section 3 we present GENRE and eventually in Section 4 we extensively evaluate our method on the aforementioned settings.

Original: We will release code and pre-processed data to reproduce our experiments.
Revised: We will release code and pre-processed data to reproduce our experiments.

Original:  E NTITY R ETRIEVAL
Revised:  E NTITY R ETRIEVAL

Original: We assume to have a collection of entities E (e.g., Wikipedia articles) where each entity is an entry in a Knowledge Base (KB) such as Wikipedia.
Revised: We assume to have a collection of entities E (e.g., Wikipedia articles) where each entity is an entry in a Knowledge Base (KB) such as Wikipedia.

Original: We want to approach the following retrieval problem: given a textual input source x (e.g
Revised: We want to approach the following retrieval problem: given a textual input source x (e.g., question), a model has to return the most relevant entities from

Original: question

Original: ), a

Original: model

Original: has to return the most relevant entities from

Original: E with respect to x .
Revised: E with respect to x .

Original: We assume that each e ∈ E is uniquely assigned to a textual representation (i.e., its name): a sequence of tokens
Revised: We assume that each e ∈ E is uniquely assigned to a textual representation (i.e., its name): a sequence of tokens

Original: y (e.g., Wikipedia pages are identiﬁed by their titles).
Revised: y (e.g., Wikipedia pages are identiﬁed by their titles).

Original: A particular instance of this problem is Entity Disambiguation (ED) (see Figure 1 for an example) where an input x is annotated with a mention and a system has to select either its corresponding entity from E , or to predict that there is no corresponding entry in the KB.
Revised: A particular instance of this problem is Entity Disambiguation (ED) (see Figure 1 for an example) where an input x is annotated with a mention and a system has to select either its corresponding entity from E , or to predict that there is no corresponding entry in the KB.

Original: Another instance is pagelevel Document Retrieval (DR) where the input x is intended as a query and E as a collection of documents identiﬁed by their unique titles (e.g., Wikipedia articles).
Revised: Another instance is pagelevel Document Retrieval (DR) where the input x is intended as a query and E as a collection of documents identiﬁed by their unique titles (e.g., Wikipedia articles).

Original:  M ETHOD
Revised:  M ETHOD

Original: We address the retrieval problem with an sequence-to-sequence model that generates textual entity identiﬁers (i.e., entity names).
Revised: We address the retrieval problem with an sequence-to-sequence model that generates textual entity identiﬁers (i.e., entity names).

Original: Concretely, GENRE ranks each e ∈ E by computing a score with an autoregressive formulation: score( e | x ) =
Revised: Concretely, GENRE ranks each e ∈ E by computing a score with an autoregressive formulation: score( e | x ) =

Original: p θ ( y | x ) =
Revised: p θ ( y | x ) =

Original: (cid:81)
Revised: (cid:81)

Original: Ni =1 p θ ( y i |
Revised: Ni =1 p θ ( y i |

Original: y <
Revised: y <

Original: i , x ) where y is the set of N tokens in the identiﬁer of e , and θ the parameters of the model.
Revised: i , x ) where y is the set of N tokens in the identiﬁer of e , and θ the parameters of the model.

Original: We take advantage of ﬁne-tuning the BART (Lewis et al., 2019) pre-trained language model.
Revised: We take advantage of ﬁne-tuning the BART (Lewis et al., 2020a) pre-trained language model.

Original: We train GENRE using a standard seq2seq objective, i.e., maximizing the output sequence likelihood with teacher forcing (Sutskever et al., 2011; 2014) and regularized with dropout (Srivastava et al., 2014) and label smoothing (Szegedy et al., 2016).
Revised: We train GENRE using a standard seq2seq objective, i.e., maximizing the output sequence likelihood with teacher forcing (Sutskever et al., 2011; 2014) and regularized with dropout (Srivastava et al., 2014) and label smoothing (Szegedy et al., 2016).

Original: Concretely, we use the objective that is typically used for neural machine translation (NMT, Wu et al., 2016), that is maximizing log p θ ( y | x ) with respect to model’s parameters θ which, due to the factorized formulation, can be calculated exactly.
Revised: Concretely, we use the objective that is typically used for neural machine translation (NMT, Wu et al., 2016), that is maximizing log p θ ( y | x ) with respect to model’s parameters θ which, due to the factorized formulation, can be calculated exactly.

Original: We do not need negative sampling to approximate the loss normalizer.
Revised: We do not need negative sampling to approximate the loss normalizer.

Original: 3. I NFERENCE WITH C ONSTRAINED B EAM S EARCH
Revised: m e n ti on c on ti nu e In 1503, _ Leonardo [ [Figure] In

Revised: [Figure]

Revised: [Leonardo_ began c on ti nu e e n tit y ]

Revised: [Figure]

Revised: In 1503, [Leonardo](Leonardo_ da Vinci DiCaprio

Revised: [Figure]

Revised: c on ti nu e c l o s e )

Revised: [Figure]

Revised: 3. I NFERENCE WITH C ONSTRAINED B EAM S EARCH

Original: Naturally, at test time, we could compute a score for every element in E and then sort them.
Revised: Naturally, at test time, we could compute a score for every element in E and then sort them.

Original: Unfortunately, this might be prohibitively expensive when E is very large (e.g., Wikipedia has ∼ 6M entities).
Revised: Unfortunately, this might be prohibitively expensive when E is very large (e.g., Wikipedia has ∼ 6M entities).

Original: Hence, we exploit Beam Search (BS, Sutskever et al., 2014), an established approximate decoding strategies to efﬁciently navigate the search space.
Revised: Hence, we exploit Beam Search (BS, Sutskever et al., 2014), an established approximate decoding strategies to efﬁciently navigate the search space.

Original: Instead of explicitly scoring all entities in E , we search for the top- k entities in E decoding from our model using BS with k beams.Note m e n ti on c on ti nu e In 1503, _ Leonardo [ [Figure]
Revised: Instead of explicitly scoring all entities in E , we search for the top- k entities in E decoding from our model using BS with k beams.

Original: In 1503, [Leonardo_ began c on ti nu e e n tit y ]

Original: [Figure]

Original: In 1503, [Leonardo](Leonardo_ da Vinci DiCaprio (

Original: [Figure] ) c on ti nu e c l o s e ) [Figure] that using BS implies that the time cost of our retriever does not depend on the size of E , but only on the size of the beams and the average length of entity representations as we do autoregressive generation.
Revised: Note that using BS implies that the time cost of our retriever does not depend on the size of E , but only on the size of the beams and the average length of entity representations as we do autoregressive generation.

Original: The average length of entity representations is tractable (e.g., Wikipedia titles have
Revised: The average length of entity representations is tractable (e.g., Wikipedia titles have

Original: BPE tokens on average) and we follow standard NMT settings where k is small (e.g., 10).
Revised: BPE tokens on average) and we follow standard NMT settings where k is small (e.g., 10).

Original: Since we want to output only entities from E we cannot use traditional BS while decoding.
Revised: Since we want to output only entities from E we cannot use traditional BS while decoding.

Original: Indeed, allowing to generate any token from the vocabulary at every decoding step might lead the model to generate output strings that are not valid identiﬁers.
Revised: Indeed, allowing to generate any token from the vocabulary at every decoding step might lead the model to generate output strings that are not valid identiﬁers.

Original: Hence, we resort to Constrained BS, forcing to only decode valid entity identiﬁers.
Revised: Hence, we resort to Constrained BS, forcing to only decode valid entity identiﬁers.

Original: BS only considers one step ahead during decoding so we can only constrain the generation of a single next token conditioned on the previous ones.
Revised: BS only considers one step ahead during decoding so we can only constrain the generation of a single next token conditioned on the previous ones.

Original: Thus, we deﬁne our constrain in terms of a preﬁx tree T (aka trie) (Cormen et al., 2009) where nodes are annotated with tokens from the vocabulary.
Revised: Thus, we deﬁne our constrain in terms of a preﬁx tree T (aka trie) (Cormen et al., 2009) where nodes are annotated with tokens from the vocabulary.

Original: For each node t ∈ T , its children indicate all the allowed continuations from the preﬁx deﬁned traversing the trie from the root to t .
Revised: For each node t ∈ T , its children indicate all the allowed continuations from the preﬁx deﬁned traversing the trie from the root to t .

Original: See Figure 9 in Appendix C for an exampled of a trie.
Revised: See Figure 9 in Appendix C for an exampled of a trie.

Original: When the number of allowed outputs is tractable (e.g., generating a Wikipedia title among ∼ 6M) the trie is relatively small it can be precomputed and stored into memory (e.g., constraining on Wikipedia titles using the BART tokenizer produces a trie with ∼ 6M leaves, ∼ 17M internal nodes that occupied ∼ 600MB of disk space).
Revised: When the number of allowed outputs is tractable (e.g., generating a Wikipedia title among ∼ 6M) the trie is relatively small it can be precomputed and stored into memory (e.g., constraining on Wikipedia titles using the BART tokenizer produces a trie with ∼ 6M leaves, ∼ 17M internal nodes that occupied ∼ 600MB of disk space).

Original: We employed the constraints masking the log-probabilities of the invalid tokens and not their logits (i.e., we do not re-normalize the probability over the vocabulary).
Revised: We employed the constraints masking the log-probabilities of the invalid tokens and not their logits (i.e., we do not re-normalize the probability over the vocabulary).

Original: 3. A UTOREGRESSIVE E ND - TO -E ND E NTITY L INKING
Revised: 3. A UTOREGRESSIVE E ND - TO -E ND E NTITY L INKING

Original: We additionally extend our autoregressive framework to address end-to-end Entity Linking (EL) where, given a document, a system has to both detect entity mentions and link those mentions to their respective KB entities.
Revised: We additionally extend our autoregressive framework to address end-to-end Entity Linking (EL) where, given a document, a system has to both detect entity mentions and link those mentions to their respective KB entities.

Original: In this setting, we train the model to predict the source input again but with annotated spans.
Revised: In this setting, we train the model to predict the source input again but with annotated spans.

Original: We use a Markup annotation where spans boundaries are ﬂagged with special tokens and accompanied by their corresponding entity identiﬁers.
Revised: We use a Markup annotation where spans boundaries are ﬂagged with special tokens and accompanied by their corresponding entity identiﬁers.

Original: Differently from a setting where the output space is relatively small (e.g., a pre-deﬁned set E ), the space of annotated outputs is exponentially large.
Revised: Differently from a setting where the output space is relatively small (e.g., a pre-deﬁned set E ), the space of annotated outputs is exponentially large.

Original: Hence, it is intractable to pre-compute a trie for decoding, and we compute it dynamically instead.
Revised: Hence, it is intractable to pre-compute a trie for decoding, and we compute it dynamically instead.

Original: In Figure 2 we show an example.
Revised: In Figure 2 we show an example.

Original: At each generation step, the decoder is either generating a mention span, generating a link to a mention, or continuing from the input source.
Revised: At each generation step, the decoder is either generating a mention span, generating a link to a mention, or continuing from the input source.

Original: When outside a mention/entity step the decoder has only two options: (i) to continue by copying the next token from the input source, or (ii) to generate the start of mention token (i.e., ‘[’) which makes the decoder enter the mention generating phase.
Revised: When outside a mention/entity step the decoder has only two options: (i) to continue by copying the next token from the input source, or (ii) to generate the start of mention token (i.e., ‘[’) which makes the decoder enter the mention generating phase.

Original: While generating a mention, the decoder has either to continue with the next token in the input source or to generate the end of mention token (i.e., ‘]’) which makes the decoder enter the entity generating phase.
Revised: While generating a mention, the decoder has either to continue with the next token in the input source or to generate the end of mention token (i.e., ‘]’) which makes the decoder enter the entity generating phase.

Original: Finally, when generating an entity, the decoder employs the entities trie such that it can only output a valid entity identiﬁer as in Constrained Beam Search explained above.
Revised: Finally, when generating an entity, the decoder employs the entities trie such that it can only output a valid entity identiﬁer as in Constrained Beam Search explained above.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We extensively evaluate GENRE on more than 20 datasets across 3 tasks: Entity Disambiguation, end-to-end Entity Linking (EL), and page-level Document Retrieval.
Revised: We extensively evaluate GENRE on more than 20 datasets across 3 tasks: Entity Disambiguation, end-to-end Entity Linking (EL), and page-level Document Retrieval.

Original: We describe the experimental settings in Section 4.1 where we discuss results in Section 4.2.
Revised: We describe the experimental settings in Section 4.1 where we discuss results in Section 4.2.

Original: All experiments are in English.
Revised: All experiments are in English.

Original: 4. S ETTINGS
Revised: 4. S ETTINGS

Original: Entity Disambiguation (ED)
Revised: Entity Disambiguation (ED)

Original: We reproduce the setting of Le & Titov (2018) using the same candidate sets, in-domain and out-of-domain datasets, and evaluating using the InKB micro- F 1 .
Revised: We reproduce the setting of Le & Titov (2018) using the same candidate sets, in-domain and out-of-domain datasets, and evaluating using the InKB micro- F 1 .

Original: We train GENRE feeding each document where a single mention is ﬂagged with two special start and end tokens and the target output is the textual representation of the corresponding entity.
Revised: We train GENRE feeding each document where a single mention is ﬂagged with two special start and end tokens and the target output is the textual representation of the corresponding entity.

Original: At test time, we decode using constrained beam search with a trie obtained using the provided candidate set (i.e., a subset of E ).
Revised: At test time, we decode using constrained beam search with a trie obtained using the provided candidate set (i.e., a subset of E ).

Original: As large generative models beneﬁt from large amount of data, we ﬁrst pre-train
Revised: As large generative models beneﬁt from large amount of data, we ﬁrst pre-train

Original: GENRE on the BLINK data (Wu et al., 2019), i.e., 9M unique triples document-mention-entity from
Revised: GENRE on the BLINK data (Wu et al., 2020), i.e., 9M unique triples document-mention-entity from

Original: Wikipedia.
Revised: Wikipedia.

Original: Then, for the in-domain scenario, we ﬁne-tune using the AIDA-CoNLL dataset (Hoffart et al., 2011).
Revised: Then, for the in-domain scenario, we ﬁne-tune using the AIDA-CoNLL dataset (Hoffart et al., 2011).

Original: For the out-of-domain scenario, we evaluate on ﬁve test sets: MSNBC, AQUAINT,
Revised: For the out-of-domain scenario, we evaluate on ﬁve test sets: MSNBC, AQUAINT,

Original: ACE2004, WNED-CWEB (CWEB) and WNED-WIKI (WIKI)
Revised: ACE2004, WNED-CWEB (CWEB) and WNED-WIKI (WIKI)

Original: (Gabrilovich et al., 2013; Guo &
Revised: (Gabrilovich et al., 2013; Guo &

Original: Barbosa, 2018).
Revised: Barbosa, 2018).

Original: More task details and hyperparameters setting are reported in Appendix A.1.
Revised: More task details and hyperparameters setting are reported in Appendix A.1.

Original: End-to-End Entity Linking (EL)
Revised: End-to-End Entity Linking (EL)

Original: For EL, we reproduce the setting of Kolitsas et al.
Revised: For EL, we reproduce the setting of Kolitsas et al.

Original: (2018) using the same in-domain and out-of-domain datasets as well as evaluating the InKB micro- F 1 on the
Revised: (2018) using the same in-domain and out-of-domain datasets as well as evaluating the InKB micro- F 1 on the

Original: GERBIL benchmark platform (R¨oder et al., 2018).
Revised: GERBIL benchmark platform (R ¨ oder et al., 2018).

Original: Similarly to the ED setting, we ﬁrst pre-traine our model on all abstract sections from Wikipedia 4 enriched by a string matching heuristic to solve co-references (i.e., if there is a string that matches exactly with another hyperlink we also add it to the dataset as a mention/entity pairs).
Revised: Similarly to the ED setting, we ﬁrst pre-traine our model on all abstract sections from Wikipedia 4 enriched by a string matching heuristic to solve co-references (i.e., if there is a string that matches exactly with another hyperlink we also add it to the dataset as a mention/entity pairs).

Original: Then, for the in-domain scenario, we ﬁne-tune using the AIDACoNLL dataset.
Revised: Then, for the in-domain scenario, we ﬁne-tune using the AIDACoNLL dataset.

Original: We evaluate on seven out-of-domain test sets: MSNBC, Derczynski (Der)
Revised: We evaluate on seven out-of-domain test sets: MSNBC, Derczynski (Der)

Original: (Derczynski et al., 2015), KORE 50 (K50) (Hoffart et al., 2012), N3-Reuters-128 (R128), N3-RSS- (R500) (R ¨ oder et al., 2014), and OKE challenge 2015 and 2016 (OKE15 and OKE16) (Nuzzolese etal., 2015).
Revised: (Derczynski et al., 2015), KORE 50 (K50) (Hoffart et al., 2012), N3-Reuters-128 (R128), N3-RSS- (R500) ( oder et al., 2014), and OKE challenge 2015 and 2016 (OKE15 and OKE (Nuzzolese et

Original: More task details and hyperparameters setting are reported in Appendix A.2.

Original: Page-level Document Retrieval (DR) For this setting
Revised: Page-level Document Retrieval (DR) For this setting

Original: , we test GENRE on all the KILT benchmark tasks (Petroni et al., 2020b).
Revised: , we test GENRE on all the KILT benchmark tasks (Petroni et al., 2020b).

Original: Here, whole Wikipedia is used as the candidate set and we evaluate using R-precision (Beitzel et al., 2009).
Revised: Here, whole Wikipedia is used as the candidate set and we evaluate using R-precision (Beitzel et al., 2009).

Original: KILT consists of ﬁve tasks that use the same Wikipedia dump as a knowledge source: fact checking with FEVER (Thorne et al., 2018); open domain question answering using Natural Questions (Kwiatkowski et al., 2019), HotpotQA
Revised: KILT consists of ﬁve tasks that use the same Wikipedia dump as a knowledge source: fact checking with FEVER (Thorne et al., 2018); open domain question answering using Natural Questions (Kwiatkowski et al., 2019), HotpotQA

Original: (Yang et al., 2018c),
Revised: (Yang et al., 2018b),

Original: TriviaQA (Joshi et al., 2017), ELI5 (Fan et al., 2019); slot ﬁlling with T-REx (Elsahar et al., 2018),
Revised: TriviaQA (Joshi et al., 2017), ELI5 (Fan et al., 2019); slot ﬁlling with T-REx (Elsahar et al., 2018),

Original: Zero Shot RE (Levy et al., 2017); entity disambiguation on AIDA CoNLL-YAGO, WNED-WIKI and WNED-CWEB; dialogue with Wizard of Wikipedia (Dinan et al., 2019).
Revised: Zero Shot RE (Levy et al., 2017); entity disambiguation on AIDA CoNLL-YAGO, WNED-WIKI and WNED-CWEB; dialogue with Wizard of Wikipedia (Dinan et al., 2019).

Original: We train GENRE on
Revised: We train GENRE on

Original: BLINK and all KILT data simultaneously with a single model.
Revised: BLINK and all KILT data simultaneously with a single model.

Original: 5 More details on the hyperparameter setting are reported in Appendix A.3.
Revised: 5 More details on the hyperparameter setting are reported in Appendix A.3.

Original: 4. R ESULTS
Revised: 4. R ESULTS

Original: Overall, GENRE achieves very competitive results in all of the three settings being the best performing system on average across all of them.
Revised: Overall, GENRE achieves very competitive results in all of the three settings being the best performing system on average across all of them.

Original: See Appendix C for examples of inputs, ground truth and model predictions for all of the three tasks.
Revised: See Appendix C for examples of inputs, ground truth and model predictions for all of the three tasks.

Original: In the following, we discuss how GENRE compares to SOTA systems as well as showing some quantitative analysis on its memory footprint, how  it exploits the structured of the entity name space, and how it behaves on a cold-start scenario where new unseen entities are added to the KB (descriptions of those entities are unobserved).
Revised: In the following, we discuss how GENRE compares to SOTA systems as well as showing some quantitative analysis on its memory footprint, how [Table] it exploits the structured of the entity name space, and how it behaves on a cold-start scenario where new unseen entities are added to the KB (descriptions of those entities are unobserved).

Original: Comparing GENRE to SOTA systems In ED the difference in average F 1 score between
Revised: Comparing GENRE to SOTA systems In ED the difference in average F 1 score between

Original: GENRE and the second best performing system is small (i.e., +0.8) however, ED is an established task with more than a decade of research that benchmarked on those datasets.
Revised: GENRE and the second best performing system is small (i.e., +0.8) however, ED is an established task with more than a decade of research that benchmarked on those datasets.

Original: Indeed all systems reported in Table 1 achieved high and similar results even if they were taken from three years back.[Table] [Table]
Revised: Indeed all systems reported in Table 1 achieved high and similar results even if they were taken from three years back.

Original: The improvements on EL are instead move evident.
Revised: The improvements on EL are instead more evident.

Original: GENRE is the best in-domain system for
Revised: GENRE is the best in-domain system for

Original: AIDA while performing remarkably well also on the out-of-domain setting (e [Table] g., +13 F 1 points on
Revised: AIDA while performing remarkably well also on the out-of-domain setting (e.g., +13 F 1 points on

Original: Derczynski, and +4.7 on KORE50).
Revised: Derczynski, and +4.7 on KORE50).

Original: Noticeably, in two datasets (OKE15 and OKE16) our model performs poorly.
Revised: Noticeably, in two datasets (OKE15 and OKE16) our model performs poorly.

Original: However, these datasets are annotated with coreference (pronouns and common nouns are linked to entities) while our model was not speciﬁcally trained for that.
Revised: However, these datasets are annotated with coreference (pronouns and common nouns are linked to entities) while our model was not speciﬁcally trained for that.

Original: Conversely, most of the other systems, have a mention detection component in their pipelines that can be trained or biased to also solve these cases.
Revised: Conversely, most of the other systems, have a mention detection component in their pipelines that can be trained or biased to also solve these cases.

Original: We considered out of the aim of this work to additional train and evaluate on coreference and we leave it for future work.
Revised: We considered out of the aim of this work to additional train and evaluate on coreference and we leave it for future work.

Original: On page-level DR, the superiority of GENRE is remarkable.
Revised: On page-level DR, the superiority of GENRE is remarkable.

Original: Our model is the best performing system across all 5 KILT tasks and all datasets except on Natural Questions where it is the second best.
Revised: Our model is the best performing system across all 5 KILT tasks and all datasets except on Natural Questions where it is the second best.

Original: We achieve +13.7 R-precision points on average with respect to the best performing baseline.
Revised: We achieve +13.7 R-precision points on average with respect to the best performing baseline.

Original: In Table 3 we compare GENRE against all methods reported in the public leaderboard:
Revised: In Table 3 we compare GENRE against all methods reported in the public leaderboard:

Original: DPR (Karpukhin et al., 2020), DPR+BERT (Devlin et al., 2019), DPR+BART, tf-idf (Leskovec et al., 2014), RAG (Lewis et al., 2020), and BLINK+ﬂair (Wu et al., 2019; Akbik et al., 2019).
Revised: DPR (Karpukhin et al., 2020), DPR+BERT (Devlin et al., 2019), DPR+BART, tf-idf (Leskovec et al., 2014), RAG (Lewis et al., 2020b), and BLINK+ﬂair (Wu et al., 2020; Akbik et al., 2019).

Original: No model except ours was trained on the entire KILT dataset at the same time.
Revised: No model except ours was trained on the entire KILT dataset at the same time.

Original: A RAG model was trained for every single task as well as for DPR+BERT.
Revised: A RAG model was trained for every single task as well as for DPR+BERT.

Original: Note that this gives and advantage to RAG and DPR+BERT to specialize on single tasks where we have only a single model to solve all of them which still performs better.
Revised: Note that this gives and advantage to RAG and DPR+BERT to specialize on single tasks where we have only a single model to solve all of them which still performs better.

Original: We speculate that multi-task training could have helped since the all tasks share a common objective to retrieve entities.
Revised: We speculate that multi-task training could have helped since the all tasks share a common objective to retrieve entities.

Original: Both DPR and BLINK+ﬂair were not trained speciﬁcally on KILT.
Revised: Both DPR and BLINK+ﬂair were not trained speciﬁcally on KILT.

Original: However, DPR was trained using several QA datasets which include Natural
Revised: However, DPR was trained using several QA datasets which include Natural

Original: Question and TriviaQA.
Revised: Question and TriviaQA.

Original: In Appendix B we report additional results where we do not pre-train or ﬁne-tune our models for both the ED and retrieval setting in Table 7 and 8 respectively.
Revised: In Appendix B we report additional results where we do not pre-train or ﬁne-tune our models for both the ED and retrieval setting in Table 1 and 8 respectively.

Original: When we train GENRE only in the DPR or BLINK data, our model still outperforms them. 
Revised: When we train GENRE only in the DPR or BLINK data, our model still outperforms them. [Table]

Original: [Table]
Revised: [Table]

Original: Memory Footprint
Revised: Memory Footprint

Original: GENRE is not only performing better than other SOTA models on DR but it has a signiﬁcant reduction of memory footprint (disk space).
Revised: GENRE is not only performing better than other SOTA models on DR but it has a signiﬁcant reduction of memory footprint (disk space).

Original: In Figure 4 we compare the number of model/index parameter against DPR, RAG [Table] and BLINK.
Revised: In Figure 4 we compare the number of model/index parameter against DPR, RAG, and BLINK.

Original: GENRE uses an order of magnitude less parameters (millions instead of billions) to store the entity index because it just has to use a preﬁx tree of the entity names as opposed to a dense vector for each entity.
Revised: GENRE uses an order of magnitude less parameters (millions instead of billions) to store the entity index because it just has to use a preﬁx tree of the entity names as opposed to a dense vector for each entity.

Original: Concretely, GENRE occupied 12 times less memory than BLINK and almost 30 times less memory than DPR.
Revised: Concretely, GENRE occupied 12 times less memory than BLINK and almost 30 times less memory than DPR.

Original: Exploiting the Structured Name Space
Revised: Exploiting the Structured Name Space

Original: We investigated some properties of GENRE, comparing two variants of our model to BLINK on the ED task (using WNED-KILT validation set): one trained to generate entity names and another to generate numerical identiﬁers (IDs).
Revised: We investigated some properties of GENRE, comparing two variants of our model to BLINK on the ED task (using WNED-KILT validation set): one trained to generate entity names and another to generate numerical identiﬁers (IDs).

Original: All models are trained on the same data and we report results in Figure 5.
Revised: All models are trained on the same data and we report results in Figure 5.

Original: When there is an exact match between a mention and its entity name, both BLINK and GENRE almost always make an accurate prediction.
Revised: When there is an exact match between a mention and its entity name, both BLINK and GENRE almost always make an accurate prediction.

Original: Different is the case of partial and no match:
Revised: Different is the case of partial and no match:

Original: GENRE performance is much higher suggesting that our model uses the context more effectively, as the autoregressive formulation allows to cross-encode mention context and entity candidates directly capturing ﬁne-grained interactions between the two.
Revised: GENRE performance is much higher suggesting that our model uses the context more effectively, as the autoregressive formulation allows to cross-encode mention context and entity candidates directly capturing ﬁne-grained interactions between the two.

Original: Moreover, when we switch to predicting IDs, the performance drops drastically (-20.3 points on average) indicating that it is important that entity names are meaningful, structured and compositional (as they are in Wikipedia) conversely to atomic IDs.
Revised: Moreover, when we switch to predicting IDs, the performance drops drastically (-20.3 points on average) indicating that it is important that entity names are meaningful, structured and compositional (as they are in Wikipedia) conversely to atomic IDs.

Original: Surprisingly, when there is no overlap between a mention-entity pair, performance are still relatively high by using IDs.
Revised: Surprisingly, when there is no overlap between a mention-entity pair, performance are still relatively high by using IDs.

Original: This suggests that the model is good at memorizing and recalling identiﬁers even if numeric.
Revised: This suggests that the model is good at memorizing and recalling identiﬁers even if numeric.

Original: Cold-start We manually collect 50 Wikipedia articles that were created in 2020 6 to simulate a cold-start setting where new entities are added to  the KB and the  only entity information available is their names.
Revised: Ablation study We here discuss an ablation study on the entity disambiguation task (see Table 1). Due to space limitation, we discuss an ablation study on document retrieval in Appendix B.2. In Table 1, GENRE only AIDA or BLINK data indicates the ablation for which we only train on one of the two datasets (i.e., only ﬁne-tuning).

Revised: GENRE (full) is also used with constrained decoding (see

Revised: Section 3) and in combination with a candidate set (as provided by Le & Titov (2018)).

Revised: GENRE without candidate set denotes ablating the provided (and small) candidate set and therefore using all the entities in the KB (in our case Wikipedia) as candidates.

Revised: GENRE without constraints indicates ablating constrained decoding which implies no use of the provided candidates set but also unconstrained generation (i.e., the model may generate entity names that are not in the KB).

Revised: Eventually, using constrained generation and exploiting the candidate sets proved useful.

Revised: Training only on AIDA data is insufﬁcient to get high F 1 (but AIDA is quite small compared to the 9M datapoints of BLINK data).

Revised: Entity frequency The performance of a model naturally depends on how many times entities appear in the training data.

Revised: We show the data distribution of the mention-entity frequency in Figure 3.

Revised: Most of the pairs appears in Wikipedia (10931 / 13354) where 2423 do not (ﬁrst bin).

Revised: The average accuracy is 82.5% but noticeable it is higher for mention-entity pairs that are more frequent (right side of the plot).

Revised: The accuracy for pairs that do not appear in Wikipedia is substantially lower than the average suggesting that those are harder cases (the very end tail of the distribution).

Revised: The degradation in performance is minimal indicating that our model is good at predicting rare entities.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Cold-start We manually collect 50 Wikipedia articles that were created in 2020 6 to simulate a cold-start setting where new entities are added to the KB and the only entity information available is their names.

Original: To create ED instances we resort to hyperlinks pointing to those entities in other
Revised: To create ED instances we resort to hyperlinks pointing to those entities in other

Original: Wikipedia articles.
Revised: Wikipedia articles.

Original: 19 out of 50 mentions have an exact match with their respective entity names and all of them were correctly classiﬁed by GENRE.
Revised: 19 out of 50 mentions have an exact match with their respective entity names and all of them were correctly classiﬁed by GENRE.

Original: In combination with the results from Tablewe can conclude that GENRE has a bias on exactly copying the mention, and this helps on unseen data.
Revised: In combination with the results from Tablewe can conclude that GENRE has a bias on exactly copying the mention, and this helps on unseen data.

Original: GENRE also correctly classiﬁed 14/31 of the remaining mentions (45.2%).
Revised: GENRE also correctly classiﬁed 14/31 of the remaining mentions (45.2%).

Original: This demonstrates the ability of our solution to be applied in scenarios where entity metadata is unavailable (apart his name), a setting where, to the best of our knowledge, no existing system is capable to operate.
Revised: This demonstrates the ability of our solution to be applied in scenarios where entity metadata is unavailable (apart his name), a setting where, to the best of our knowledge, no existing system is capable to operate.

Original: We additionally test how GENRE performs on unseen mention-entity pairs on WikilinksNED
Revised: We additionally test how GENRE performs on unseen mention-entity pairs on WikilinksNED

Original: Unseen-Mentions data (Onoe & Durrett, 2020)
Revised: Unseen-Mentions data (Onoe & Durrett, 2020)

Original: and we report all results in Table 6 in Appendix B.1.
Revised: and we report all results in Table 6 in Appendix B.1.

Original: Surprisingly, GENRE performs almost the same for seen and unseen entity pairs (64.4 vs 63.2 accuracy)
Revised: Surprisingly, GENRE performs almost the same for seen and unseen entity pairs (64.4 vs 63.2 accuracy)

Original: However, in the Onoe & Durrett (2020) setting we cannot guarantee entity descriptions have not been seen by BART during pre-training (given his training data contains Wikipedia).
Revised: However, in the Onoe & Durrett (2020) setting we cannot guarantee entity descriptions have not been seen by BART during pre-training (given his training data contains Wikipedia).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Casting NLP tasks with a structured input or output into sequence-to-sequence problems has been explored for different problems, including semantic parsing (Rongali et al., 2020), semantic role labelling (Daza & Frank, 2018), discourse representation structure parsing (Liu et al., 2018), generation of ﬂuent natural language responses from structured semantic representations (Balakrishnan [Table] [Table] [Table]
Revised: Casting NLP tasks with a structured input or output into sequence-to-sequence problems has been explored for different problems, including semantic parsing (Rongali et al., 2020), semantic role labelling (Daza & Frank, 2018), discourse representation structure parsing (Liu et al., 2018), generation of ﬂuent natural language responses from structured semantic representations (Balakrishnan et al., 2019), generation and parsing of abstract meaning representation (Konstas et al., 2017).

Original: et al., 2019), generation and parsing of abstract meaning representation (Konstas et al., 2017).

Original: In these works a structured representation, a tree or a graph for instance, is linearized into a sequence of symbols compatible with a seq2seq architecture.
Revised: In these works a structured representation, a tree or a graph for instance, is linearized into a sequence of symbols compatible with a seq2seq architecture.

Original: To the best of our knowledge, we are the ﬁrst to cast entity retrieval as a sequence-to-sequence problem while decoding with an autoregressive formulation during inference.
Revised: To the best of our knowledge, we are the ﬁrst to cast entity retrieval as a sequence-to-sequence problem while decoding with an autoregressive formulation during inference.

Original: Related to our constrained generation mechanism, Daza & Frank (2018); Rongali et al.
Revised: Related to our constrained generation mechanism, Daza & Frank (2018); Rongali et al.

Original: (2020) use a copying mechanism in order to limit lexical deviations between the input and output strings.
Revised: (2020) use a copying mechanism in order to limit lexical deviations between the input and output strings.

Original: In these tasks, as well as for our problem, it is natural to promote a copying mechanism due to the input and the output proximity.
Revised: In these tasks, as well as for our problem, it is natural to promote a copying mechanism due to the input and the output proximity.

Original: A different type of constraint, a structural constraint, is used in Balakrishnan et al.
Revised: A different type of constraint, a structural constraint, is used in Bal-

Revised: None 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 2 11 2 12 2 13 2 14 2 15 2 16 2 17 Mention-entity pair appearances 0.

Revised: A cc u r a c y

Revised: Accuracy Data distribution 0.0e+ 6.9e+ 1.4e+ 2.1e+ 2.8e+ 3.4e+ 4.1e+ 4.8e+ 5.5e+ 6.2e+ 6.9e+

Revised: F r e q u e n c y akrishnan et al.

Original: (2019) to maintain a valid tree structure.
Revised: (2019) to maintain a valid tree structure.

Original: Our constrained beam search encompasses both aspects, a copying mechanism that restrains the vocabulary and a structural constraint to obtain a well-formed annotated output.
Revised: Our constrained beam search encompasses both aspects, a copying mechanism that restrains the vocabulary and a structural constraint to obtain a well-formed annotated output.

Original: In addition to these tasks with close input and output, the integration of a mechanism to guide the output of neural networks has been explored in various settings.
Revised: In addition to these tasks with close input and output, the integration of a mechanism to guide the output of neural networks has been explored in various settings.

Original: Lexically constrained decoding has been used to force the inclusion of pre-speciﬁed words for machine translation (Hokamp & Liu, 2017; Post & Vilar, 2018), and image captioning (Anderson et al., 2017).
Revised: Lexically constrained decoding has been used to force the inclusion of pre-speciﬁed words for machine translation (Hokamp & Liu, 2017; Post & Vilar, 2018), and image captioning (Anderson et al., 2017).

Original: To the best of our knowledge, we are the ﬁrst to exploit constrained generation for entity disambiguation, end-to-end entity linking, and query-based entity retrieval.
Revised: To the best of our knowledge, we are the ﬁrst to exploit constrained generation for entity disambiguation, end-to-end entity linking, and query-based entity retrieval.

Original: Nogueira et al.
Revised: Nogueira et al.

Original: (2020) propose to use a sequence-to-sequence model to re-rank document.
Revised: (2020) propose to use a sequence-to-sequence model to re-rank document.

Original: Given a query and a document the model is trained to output the words ”true” or ”false” depending on whether the document is relevant or not.
Revised: Given a query and a document the model is trained to output the words ”true” or ”false” depending on whether the document is relevant or not.

Original: Differently from our approach for entity retrieval, it requires a limited list of candidates documents, obtained with BM25 for instance, in order to be computationally possible.
Revised: Differently from our approach for entity retrieval, it requires a limited list of candidates documents, obtained with BM25 for instance, in order to be computationally possible.

Original: Massarelli et al.
Revised: Massarelli et al.

Original: ; Petroni et al.
Revised: ; Petroni et al.

Original: (2020a) explore the idea of using an autoregressive language model as neural retriever, by exploiting the implicit knowledge stored in their parameters to generate relevant sentences given a query.
Revised: (2020a) explore the idea of using an autoregressive language model as neural retriever, by exploiting the implicit knowledge stored in their parameters to generate relevant sentences given a query.

Original: While intriguing, such solutions still lag behind retrievers with an explicit knowledge access (e.g., an explicit Wikipedia index).
Revised: While intriguing, such solutions still lag behind retrievers with an explicit knowledge access (e.g., an explicit Wikipedia index).

Original: The idea of using a generative model for entity disambiguation was proposed in Petroni et al.
Revised: The idea of using a generative model for entity disambiguation was proposed in Petroni et al.

Original: (2020b) as they trained both BART and T5 in a seq2seq fashion on all KILT tasks (including ED).
Revised: (2020b) as they trained both BART and T5 in a seq2seq fashion on all KILT tasks (including ED).

Original: We expanded that intuition generalizing on multiple tasks (end-to-end EL and page-level retrieval) as well as introducing constrained decoding for an efﬁcient and effective search.
Revised: We expanded that intuition generalizing on multiple tasks (end-to-end EL and page-level retrieval) as well as introducing constrained decoding for an efﬁcient and effective search.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we propose GENRE, a novel paradigm to addresses entity retrieval: generate entity names autoregressively.
Revised: In this work, we propose GENRE, a novel paradigm to addresses entity retrieval: generate entity names autoregressively.

Original: Entity names have several properties that might help (even humans) retrieving them, including a compositional structure and a predictable interaction with the context.
Revised: Entity names have several properties that might help (even humans) retrieving them, including a compositional structure and a predictable interaction with the context.

Original: The autoregressive formulation allows us to directly capture some of these properties, leading to several advantages with respect to current solutions, including an efﬁcient way to cross encode mention context and entity candidates, a much smaller memory footprint, and the ability to compute an exact softmax without the need to subsample negative data.
Revised: The autoregressive formulation allows us to directly capture some of these properties, leading to several advantages with respect to current solutions, including an efﬁcient way to cross encode mention context and entity candidates, a much smaller memory footprint, and the ability to compute an exact softmax without the need to subsample negative data.

Original: We empirically show that these characteristics, combined with constrained decoding strategies, led to state-of-the-art performance on a plethora of entity retrieval datasets, spanning entity disambiguation, end-to-end entity linking, and page-level document retrieval, while resulting in systems with a remarkably contained memory footprint, a space reduction by a factor of twenty on average.
Revised: We empirically show that these characteristics, combined with constrained decoding strategies, led to state-of-the-art performance on a plethora of entity retrieval datasets, spanning entity disambiguation, end-to-end entity linking, and page-level document retrieval, while resulting in systems with a remarkably contained memory footprint, a space reduction by a factor of twenty on average.

Original: We additionally demonstrate that new entities can be effectively considered in our system by simply appending their unambiguous name to the candidate set. 
Revised: We additionally demonstrate that new entities can be effectively considered in our system by simply appending their unambiguous name to the candidate set. A CKNOWLEDGMENTS

Revised: Authors thank Patrick Lewis, Aleksandra Piktus, Michael Schlichtkrull, Ivan Titov, Jean Maillard,

Revised: Edouard Grave, Sergio De Cao, Luisa Quarta for helpful discussions and technical support.

Original: A E XPERIMENTAL D ETAILS
Revised: A E XPERIMENTAL D ETAILS

Original: We implemented, trained, and evaluate our model using the fariseq library (Ott et al., 2019).
Revised: We implemented, trained, and evaluate our model using the fariseq library (Ott et al., 2019).

Original: We trained GENRE for every task using Adam (Kingma & Ba, 2014) with a learning rate 3 · 10 −with a linear warm-up for 500 steps and then liner decay.
Revised: We trained GENRE for every task using Adam (Kingma & Ba, 2014) with a learning rate 3 · 10 −with a linear warm-up for 500 steps and then liner decay.

Original: The objective is sequence-to-sequence categorical cross-entropy loss with 0.1 of label smoothing.
Revised: The objective is sequence-to-sequence categorical cross-entropy loss with 0.1 of label smoothing.

Original: A. N AMED E NTITY D ISAMBIGUATION
Revised: A. N AMED E NTITY D ISAMBIGUATION

Original: Setting Given a document d j (e.g., a sentence) containing a set of entity mentions m 1 , . .
Revised: Setting Given a document d j (e.g., a sentence) containing a set of entity mentions m 1 , . .

Original: , m N , a system either has to assign, to each mention m i , either a KB entity (i.e., e i ∈ E ), or predicts that there is no corresponding entry in the KB (i.e., e i = NIL ).
Revised: , m N , a system either has to assign, to each mention m i , either a KB entity (i.e., e i ∈ E ), or predicts that there is no corresponding entry in the KB (i.e., e i = NIL ).

Original: Moreover, a restricted candidates set
Revised: Moreover, a restricted candidates set

Original: C i =
Revised: C i =

Original: { ˆ e i 1 , . . .
Revised: { ˆ e i 1 , . . .

Original: , ˆ e iK } ⊆ E ∪ { NIL } for each mention m i is provided.
Revised: , ˆ e iK } ⊆ E ∪ { NIL } for each mention m i is provided.

Original: Training We pre-trained GENRE on BLINK data for 200k steps
Revised: Training We pre-trained GENRE on BLINK data for 200k steps

Original: and then we do model selection on the validation set.
Revised: and then we do model selection on the validation set.

Original: Afterward, we ﬁne-tuned on AIDA without resetting the learning rate nor the optimizer statistics for 10k steps and we do model selection on the validation set.
Revised: Afterward, we ﬁne-tuned on AIDA without resetting the learning rate nor the optimizer statistics for 10k steps and we do model selection on the validation set.

Original: Following previous works (Yamada et al., 2016; Ganea & Hofmann, 2017; Le & Titov, 2018), we considered only mentions that have entities in the KB (i.e., Wikipedia).
Revised: Following previous works (Yamada et al., 2016a; Ganea & Hofmann, 2017; Le & Titov, 2018), we considered only mentions that have entities in the KB (i.e., Wikipedia).

Original: Training was done on 32 GPUs (with 32GB of memory) and it completed in ∼ 24h for a total of ∼ 32 GPU/day.
Revised: Training was done on 32 GPUs (with 32GB of memory) and it completed in ∼ 24h for a total of ∼ 32 GPU/day.

Original: Inference At test time, we use Constrained Beam Search with 10 beams, and maximum decoding steps of 15.
Revised: Inference At test time, we use Constrained Beam Search with 10 beams, and maximum decoding steps of 15.

Original: We restrict the input sequence to be at most 384 tokens cutting the left, right, or both parts of the context around a mention.
Revised: We restrict the input sequence to be at most 384 tokens cutting the left, right, or both parts of the context around a mention.

Original: We normalize the log-probabilities by sequence length.
Revised: We normalize the log-probabilities by sequence length.

Original: A. E NTITY L INKING
Revised: A. E NTITY L INKING

Original: Setting Given a document d j (e.g., a sentence) a system has to return a set of tuples (cid:104) m i , e i (cid:105) where m i is a entity mentions (a span contained in d j ) and e
Revised: Setting Given a document d j (e.g., a sentence) a system has to return a set of tuples (cid:104) m i , e i (cid:105) where m i is a entity mentions (a span contained in d j ) and e

Original: i ∈
Revised: i ∈

Original: E its corresponding entity in the
Revised: E its corresponding entity in the

Original: KB.
Revised: KB.

Original: Following Kolitsas et al.
Revised: Following Kolitsas et al.

Original: (2018), we considered only mentions that have entities in the KB (i.e.,
Revised: (2018), we considered only mentions that have entities in the KB (i.e.,

Original: Wikipedia)
Revised: Wikipedia)

Original: and we used their candidate sets with the additions of the table computed by Hoffart et al.
Revised: and we used their candidate sets with the additions of the table computed by Hoffart et al.

Original: Training We pre-trained GENRE on all abstract sections from Wikipedia 7 enriched by a string matching heuristic to solve co-references (i.e., if there is a string that matches exactly with another hyperlink we also add it to the dataset as a mention/entity pairs) data for 200k steps.
Revised: Training We pre-trained GENRE on all abstract sections from Wikipedia 7 enriched by a string matching heuristic to solve co-references (i.e., if there is a string that matches exactly with another hyperlink we also add it to the dataset as a mention/entity pairs) data for 200k steps.

Original: Then we do model selection on the validation set.
Revised: Then we do model selection on the validation set.

Original: Afterward, we ﬁne-tuned on AIDA resetting the learning rate and the optimizer statistics for 10k steps and we do model selection on the validation set.
Revised: Afterward, we ﬁne-tuned on AIDA resetting the learning rate and the optimizer statistics for 10k steps and we do model selection on the validation set.

Original: Again, following previous works (Kolitsas et al., 2018), we considered only mentions that have entities in
Revised: Again, following previous works (Kolitsas et al., 2018), we considered only mentions that have entities in

Original: Wikipedia.
Revised: Wikipedia.

Original: Training was done on 64 GPUs (with 32GB of memory) and it completed in ∼ 30h for a total of ∼ 80 GPU/day.
Revised: Training was done on 64 GPUs (with 32GB of memory) and it completed in ∼ 30h for a total of ∼ 80 GPU/day.

Original: Inference At test time, we use Constrained Beam Search with 6 beams, and a maximum decoding step of 384.
Revised: Inference At test time, we use Constrained Beam Search with 6 beams, and a maximum decoding step of 384.

Original: When the input sequence is too long, we split the input into multiple chunks of equal size.
Revised: When the input sequence is too long, we split the input into multiple chunks of equal size.

Original: We normalize the log-probabilities by sequence length.
Revised: We normalize the log-probabilities by sequence length.

Original: A. P AGE - LEVEL D OCUMENT R ETRIEVAL
Revised: A. P AGE - LEVEL D OCUMENT R ETRIEVAL

Original: Setting Given a query q (e.g., a question) and a collection of documents D (in KILT are Wikipedia pages), a system has to rank documents in D based on their relevance to q .
Revised: Setting Given a query q (e.g., a question) and a collection of documents D (in KILT are Wikipedia pages), a system has to rank documents in D based on their relevance to q .

Original: Training We trained GENRE on all KILT data simultaneously for 200k steps and we do model selection on the validation set averaging the score across tasks.
Revised: Training We trained GENRE on all KILT data simultaneously for 200k steps and we do model selection on the validation set averaging the score across tasks.

Original: Training was done on 128 GPUs (with 32GB of memory) and it completed in ∼ 33h for a total of ∼ 176 GPU/day.
Revised: Training was done on 128 GPUs (with 32GB of memory) and it completed in ∼ 33h for a total of ∼ 176 GPU/day.

Original: Inference At test time, we use Constrained Beam Search with 10 beams.
Revised: Inference At test time, we use Constrained Beam Search with 10 beams.

Original: For the ED sub-task, we restrict the input sequence to be at most 384 tokens cutting the left, right, or both parts of the context around a mention.
Revised: For the ED sub-task, we restrict the input sequence to be at most 384 tokens cutting the left, right, or both parts of the context around a mention.

Original: We normalize the log-probabilities by sequence length.
Revised: We normalize the log-probabilities by sequence length [Table]

Original: B A DDITIONAL R ESULTS B. N AMED E NTITY D ISAMBIGUATION
Revised: B A DDITIONAL R ESULTS B. N AMED E NTITY D ISAMBIGUATION

Original: Table 6 reports evaluation of GENRE on on WikilinksNED Unseen-Mentions data (Onoe & Durrett, 2020).
Revised: Table 6 reports evaluation of GENRE on on WikilinksNED Unseen-Mentions data (Onoe & Durrett, 2020).

Original: Table 7 extends Table 1 with additional resultsand an ablation study on  the entity disambiguation task.
Revised: We also report additional results on AIDA from the literature in Table 7.

Original: [Table]
Revised: [Table]

Original: We train on the provided train set
Revised: We train on the provided train set

Original: and we report accuracy scores (i.e., precision at 1) alongside with the number of supporting datapoints.
Revised: and we report accuracy scores (i [Table] e., precision at 1) alongside with the number of supporting datapoints.

Original: We report scores splitting the test set  and unseen entities  well as in three different matchings  a mention 
Revised: We report scores splitting the test set in seen and unseen entities as well as in three different matchings between a mention and its gold entity

Original: [Table]
Revised: [Table]

Original: Shahbazi [Table] GENRE (full) is also used with constrained decoding (see Section 3) and in combination with a candidate set (as provided by Le & Titov (2018)). GENRE without candidate set ablating the provided (and small) candidate set and therefore using all the entities in the KB (
Revised: Methods

Original: [Table]
Revised: [Table]

Original: com [Table] facebookresearch [Table]
Revised: AIDA.

Original: BLINK [Table] to accommodate entities not in KB;
Revised: We report Micro InKB F 1 on test sets.

Original: B. D OCUMENT R ETRIEVAL
Revised: B. D OCUMENT R ETRIEVAL

Original: Table 8 extends Table 3 with additional results (i
Revised: Table 8 extends Table 3 with additional results (i.e., training GENRE on the numerical identiﬁers) and an ablation study on the document retrieval task. The purpose of the experiment is to see whether GENRE beneﬁts from the entity names to be meaningful as well as compositional. Numerical IDs do not have that property.

Original: [Table]
Revised: In both cases, the model uses its memorizing capabilities but when using IDs the performance is signiﬁcantly low. Indeed, with IDs the model has no way to generalize nor to use “implicit knowledge” acquired during the unsupervised pre-training. We also ablate the training data.

Original: e., training GENRE on the numerical identiﬁers) and an ablation study onthe document retrieval task
Revised: DPR data corresponds to training only on Natural Questions (NQ) and TriviaQA (TQA) as DPR was trained only for QA tasks on those datasets and two extra ones. Note that training on

Original: [Table]
Revised: BLINK data corresponds to only training for entity disambiguation. However, every other task share similarities with entity disambiguation and thus the model is also capable to address the other tasks with non-zero performance. For the ablations, underlined cells indicate what are the results on the respective task on which a model was trained for (i.e., GENRE only BLINK data was trained only for ED where GENRE only DPR data was trained only for QA). The ablation on data suggests that it is beneﬁcial to train on all tasks simultaneously.

Original: , the model may generate entity names that are not in the KB).
Revised: GENRE without constraints indicates ablating constrained decoding which implies unconstrained generation (i.e., the model may generate entity names that are not in the KB).

Original: [Figure]
Revised: [Table]

Original: [Figure] [Figure]
Revised: 43.8 GENRE w/o constraints 78. GENRE full 83.

Original: [Figure]
Revised: 18 Number of Tokens 0.

Original: [Figure] cc 
Revised: A cc u r a c y

Original: [Figure] 0.0e [Figure] 1.7e+ 3.3e+ 5.0e+ 6.6e
Revised: Accuracy Data distribution 0.0e+ 1.7e+ 3.3e+ 5.0e+ 6.6e+ 8.3e+ 1.0e+ 1.2e+ 1.3e+ 1.5e+ 1.7e+

Original: [Figure]
Revised: F r e q u e n c y

Original: 8.3e+ 1.0e [Figure] 1.2e+ 1.3e
Revised: heavily affected (e

Original: [Figure] 1.5e
Revised: g., for length > 15)

Original: [Figure] 1.7e [Figure]
Revised: None 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 2 10 2 11 2 12 2 13 2 14 2 15 2 16 2 17 Number of incoming links in Wikipedia 0.

Original: F r [Figure]
Revised: A cc u r a c y

Original: [Figure] [Figure] [Figure] 1.3e+ 1.5e+ 1.7e+
Revised: Accuracy Data distribution 0.0e+ 1.7e+ 3.3e+ 5.0e+ 6.6e+ 8.3e+ 1.0e+ 1.2e+ 1.3e+ 1.5e+ 1.7e+

Original: F r e q u e n c y
Revised: F r e q u e n c y

Original: C E XAMPLES
Revised: C E XAMPLES

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Table]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]