Paper: ReLiK: Retrieve, Read and LinK: Fast and Accurate Entity Linking and Relation Extraction on an Academic Budget (ICLR-2024)

Reviewer comment (Novelty):
Although ReLiK integrates entity linking and relation extraction together into one framework, the design for each module is relatively simple and similar to previous works.

Author response:
To the best of your knowledge, we are the first to design an architecture capable of performing EL and RE in a single forward step in the Retriever-Reader paradigm, while achieving state-of-the-art performances and inference speed. Furthermore, **simplicity, in our view, is not a drawback**. While individual components might bear resemblance to existing models (such as using a Retriever for EL), the real innovation of ReLiK lies in the synergy of these components, which, to the best of our knowledge, we are the first to consider for closed Information Extraction. Moreover, the modularity of ReLiK is a key advantage with respect to other approaches, since ReLiK can be easily enhanced and expanded by leveraging any improvement on each of its components separately.

Reviewer comment (Experiments):
As one of the emphases of this paper is the integration of EL and RE tasks, the mutual influence between EL and RE should be more clearly demonstrated in the experimental analysis section.

Author response:
We agree, and that is why **we explored such interaction in Appendix A.4.2** and showed how EL helped RE, evidenced by a notable drop in F1 score when RE is isolated. Performance in Table 2 for EL is reported solely on entities present in triplets, but if we check the F1 score for entities in general, validation F1 reached 87.46 for ReLiK small. We ran an experiment where EL was trained with the same data without RE (similar to the experiment reported in A.4.2) and F1 reached 87.5. While, in this case, RE doesn’t seem to provide a benefit to EL, it does not affect negatively either. We would have liked to devote more space to such interactions in the main body; however, in its current shape, the paper already contains a vast array of experiments and ablations that we considered more relevant, at least within the main body. We welcome suggestions for additional experiments to explore this interaction further, and we will happily provide results if there’s time during this rebuttal period, or in the final version of the paper otherwise.

Reviewer comment (Writing):
"Recent approaches only focus on at most two out of the three properties simultaneously." I don't quite understand this sentence. What are the "three properties" referring to?

Author response:
It refers to “Speed, Flexibility, and Performance.”, which are mentioned in the previous paragraph. We will make it more clear.

Reviewer comment (Experiments):
The baseline systems used for comparison were not comprehensive enough in Section 4, and it was recommended that more baseline systems be added for comparison [1], [2], [3].

Author response:
Unfortunately, we prioritized including the most current and impactful EL systems for our baseline/comparison systems due to the limited space available. The systems suggested by the reviewer, although valuable, exhibit lower performance than our baselines. However, we will include these lower baselines in a more comprehensive EL comparison in the Appendix.

Reviewer comment (Novelty):
Since the contribution of this paper lies in the novel paradigm, the authors could have devoted a chapter to a brief overview of the developmental lineage of the relevant paradigm in order to describe more clearly the special features of this paper.

Author response:
Again, space constraints led us to focus on the previous trends and systems for each task in sections 4.1.2  and 5.1.2, but we did not have the space to delve into the explanation of Retriever-Reader systems in general. We will add a more detailed explanation of the “developmental lineage” of RR systems in the Appendix.

Reviewer comment (Experiments):
The author mentioned that “ReLiK excels in this regard, surpassing previous systems in terms of performance, memory requirements, and speed”, but did not provide a quantitative comparison of memory requirements with other systems.

Author response:
We reported on the GPU requirements of each system, which is indirectly related to memory. However, we will make sure to include a more quantitative comparison for memory. For our systems, training never requires more than 24GB of memory. At inference time, the retriever with e5 base and batch size of 128 uses 881 MB of VRAM. The reader on its larger model (debate-v3-large) with a batch size of 4096 tokens uses 3985 MB of VRAM for EL, 4497 MB of VRAM for RE. As for the passages index for the retriever, it takes 10 GB of VRAM if loaded on the GPU with fp16, and 17 GB of RAM if loaded on the CPU with fp32. For comparison, [Zhang et al. (2022)] index takes 22 GB of RAM with fp32.

Reviewer comment (Evaluation):
Is there error propagation when the retriever fails to retrieve relationships and entities from top-k results?

Author response:
Yes, whenever the Reader component fails to retrieve an entity/relation, the Reader component cannot link/extract it, and thus either it does not predict it at all (false negative) or it links/extracts a wrong one. However, the retriever has a very high recall, as shown in Table 6 of the Appendix: 99.2 R@100 and 98.8 R@50, and it rarely is the source of errors. We didn’t report on recall for cIE in the paper, but it is R@20 of 99% for relations and R@25 of 98% for entities on the REBEL dataset. We will add these numbers to the paper as well.

Reviewer comment (Evaluation):
The proposed approach is underpinned by access to external knowledge since ReLiK is given as input the text together with entities and relations from the KB. This impacts the performance and efficiency of the model and raises concerns about the fairness of the proposed benchmarks.

Author response:
The statements:

>  1. “The proposed approach is underpinned by access to external knowledge since ReLiK is given as input the text together with entities and relations from the KB. **This impacts the performance and efficiency of the model and raises concerns about the fairness of the proposed benchmarks.**”, and 
2. “More specifically, the fact that ReLiK relies on the entities and relations from the KB already provides the model with the set of possible entities that can be extracted from the text, which can help for demarcating and disambiguating entities”

Makes us believe this viewpoint might stem from a misunderstanding, which we are eager to resolve. **Every system under comparison uses external knowledge and has access to the possible entities that can be extracted from the text**. Specifically, systems in [De Cao et al. 2021 a] and [De Cao et al. 2021b] both use an external mention-entity index in which, as stated in our paper, every possible mention (e.g., Barack Obama or Washington) is associated with the list of possible entity classes (i.e. Wikipedia titles) that can be associated with them. [Zhang et al. (2022)] and [1], sharing our Retriever approach, access the same non-parametric external knowledge as ours and the set of extractable entities from the text. Therefore, within the tasks of EL and RE, **external knowledge in the form of entity and relation titles, or their definitions, is not a source of unfairness but rather available information for the task**, analogous to a training set with labels. Finally, along with the previous reasoning, we want to be very clear that, **at inference time, ReLiK receives the text as input, and solely the text**. It is the combination of the Retriever and the Reader that is able first to reduce the number of candidates and then link them. The initial search space encompasses the entire set of entities or relations, as is standard in EL and RE systems, thus ensuring fairness in task comparisons against other systems.

Reviewer comment (Evaluation):
The fact that ReLiK relies on the entities and relations from the KB already provides the model with the set of possible entities that can be extracted from the text, which can help for demarcating and disambiguating entities, and also for extracting relations.

Author response:
The statements:

>  1. “The proposed approach is underpinned by access to external knowledge since ReLiK is given as input the text together with entities and relations from the KB. **This impacts the performance and efficiency of the model and raises concerns about the fairness of the proposed benchmarks.**”, and 
2. “More specifically, the fact that ReLiK relies on the entities and relations from the KB already provides the model with the set of possible entities that can be extracted from the text, which can help for demarcating and disambiguating entities”

Makes us believe this viewpoint might stem from a misunderstanding, which we are eager to resolve. **Every system under comparison uses external knowledge and has access to the possible entities that can be extracted from the text**. Specifically, systems in [De Cao et al. 2021 a] and [De Cao et al. 2021b] both use an external mention-entity index in which, as stated in our paper, every possible mention (e.g., Barack Obama or Washington) is associated with the list of possible entity classes (i.e. Wikipedia titles) that can be associated with them. [Zhang et al. (2022)] and [1], sharing our Retriever approach, access the same non-parametric external knowledge as ours and the set of extractable entities from the text. Therefore, within the tasks of EL and RE, **external knowledge in the form of entity and relation titles, or their definitions, is not a source of unfairness but rather available information for the task**, analogous to a training set with labels. Finally, along with the previous reasoning, we want to be very clear that, **at inference time, ReLiK receives the text as input, and solely the text**. It is the combination of the Retriever and the Reader that is able first to reduce the number of candidates and then link them. The initial search space encompasses the entire set of entities or relations, as is standard in EL and RE systems, thus ensuring fairness in task comparisons against other systems.

Reviewer comment (Novelty):
The following recent prior work [1], which uses an end-to-end Reader-Retrieval approach for EL, should be cited in the paper. It would be interesting to see how both methods compare.

Author response:
> *“Also, the following recent prior work [1], which uses an end-to-end Reader-Retrieval approach for EL, should be cited in the paper. It would be interesting to see how both methods compare.”*

[1] was published in July 2023 as a preprint on Arxiv, with the deadline for ICLR 2024 being in September, and therefore, we were not able to discuss it in the paper at submission time. It is an extension of [Zhang et al. (2022)] with an interesting new end-to-end training approach. Unfortunately, they do not report results on datasets other than AIDA, and the absence of publicly available code impedes direct comparison. Their results on AIDA are in the same ballpark as ours, but the model suffers from the same shortcomings as [Zhang et al. (2022)] in terms of efficiency, or even worse, as they discuss in their Limitations section. For instance, their system requires 50GB of GPU memory, and takes 3.5 hours per epoch on 4 A100 GPUs, while ours can be trained on a single 4090 and takes half an hour per epoch for the Retriever, and the same for the Reader, on its largest version. Exploring the potential integration of our lightweight paradigm into their end-to-end approach remains an intriguing prospect. We will make sure to cover this paper in the final version of the paper.

Reviewer comment (Writing):
Xs and Xt are not defined in section 3.2, in the definition.

Author response:
> Xs and Xt are not defined in section 3.2, in the definition.

We will make it more explicit that *s* and *t* denote the indices of start and end tokens of a span in *X*.

Reviewer comment (Writing):
What is the point of $<ST_{0}>$? It is not associated with any passage and we already have the [SEP] special token to dissociate between the text and the retrieved passages.

Author response:
> What is the point of ⟨ST$_0$⟩? It is not associated with any passage and we already have the [SEP] special token to dissociate between the text and the retrieved passages.

This is explained in the third footnote, found in page 4:
Here e$_0$ symbolizes NME, i.e. mentions for which the gold entity is not in E, represented by ⟨ST$_0$⟩

Therefore, it is used for EL and cIE when a span cannot be linked to any entity candidate because either it is not in the KB or the Retriever module failed to retrieve it, and therefore it is not present in the candidates. To avoid confusion we will make sure to mention it earlier, where it is first shown in Equation 2.