Paper: Know2BIO: A Comprehensive Dual-View Benchmark for Evolving Biomedical Knowledge Graphs (ICLR-2024)

Reviewer comment (Experiments):
The proposed graph has several multi-modal attributes, but they are not used in the evaluation and the compared methods are limited.

Author response:
*The proposed graph has several multi-modal attributes, but they are not used in the evaluation and the compared methods are limited.*

Thank you for the question.

We highlight the significance of our benchmark dataset, which encompasses not only triple-based biomedical knowledge but also incorporates multi-modal attributes. In other words, Know2BIO emphasizes more on the dual-view feature (ontology and instance views) of biomedical KG as a novel benchmark.

While our initial benchmark evaluations didn't encompass these multi-modal attributes, we have decided to include this valuable information with our release. Subsequently, we plan to conduct model evaluations that harness the potential of this data at a later stage, including BERT_KG and LLM models.

Reviewer comment (Theory):
Are the edges for each of the categories homogeneous in nature?

Author response:
Thank you for the question.

Yes, the edges for each category are homogenous in nature. In total, there are 108 unique edge types, across 11 biomedical categories. In the Appendix, the final table(s) illustrate the number of relations between pairs of biomedical categories, with the most relationships between drug and protein nodes.

Reviewer comment (Reproducibility):
Can users easily query this KG? If yes, will the authors be extending support to using LLMs on this?

Author response:
Thank you for your constructive suggestion.

- Querying the KG is easy for a 1-hop query: simply use grep to find the relationship/node of interest.

- For k-hop queries, we recommend using a graph database management system, such as Neo4j. We have included code to load the resulting knowledge graph into Neo4j and query using its Cypher query language, with an example query. We have uploaded the codebase to deploy Neo4j on Know2BIO to the repository.

- Support for integrating this knowledge graph with LLMs will be explored in a subsequent publication, currently in the works (LLaMA2).

Reviewer comment (Presentation):
It would have been better if some examples were present that would explain the integration of specific similar entities with different names/representation in two KGs, and steps of resolving those.

Author response:
Entity alignment was performed by identifier mapping. In other words, we relied on the entity mappings provided by the knowledge bases we used, not on a computational method that would be less reliable such as fuzzy matching or embedding-based approaches. Once we identify the relevant data sources, the process to align the entities is somewhat straightforward.

An overview of the mappings from each data source is shown in the Appendix Figure 2. An example of the mapping process is described in the beginning of section 3.1. An overview of the mappings for each biomedical category is shown in the Appendix, starting at the bottom of page 21. The organized code to perform the mappings is provided in the GitHub repository under Know2BIO/dataset/create_edge_files_utils which we provide as a footnote in a few locations throughout the paper. 

Given this explanation, does the initial weakness still remain?

Reviewer comment (Presentation):
Future work should provide concrete methods that could be tried.

Author response:
Here is our new "Future Works" section with more concrete methods that could be tried:

Future benchmarking work on Know2BIO can test various models in graph learning
(e.g., KG embeddings, GNNs), multi-modal models (e.g., text and graph embedding models), and
training strategies (e.g., parameter initialization, curriculum learning). Multi-view graph learning
approaches can take advantage of the separate instance and ontology views. Multi-modal models
can use the non-KG data modalities to further enrich the KG with a wider array of information. To
elaborate, the non-KG data modalities can be extracted (e.g., by language models, graph learning
models) and integrated as node information into knowledge graph learning models via different
feature fusion approaches (e.g., early, mixed, late fusion). Some feature fusion approaches can also
double as a pre-training parameter initialization strategy, one of many initialization strategies to
test. The edge weights can be utilized by graph learning approaches that incorporate weights into
the training objective. Edge weights can also be used for curriculum learning. This will enable
researchers to access a more extensive and holistic view of biomedical knowledge, providing a wider
range of benchmarking tasks and challenges.

Additionally, other emerging technologies can take advantage of Know2BIO. In particular, large
language models (LLMs) can leverage the up-to-date information and structure of Know2BIO. This
can facilitate reasoning and mitigate hallucination for the LLM, both for predictive analyses (e.g.,
in-context learning) and retrieval augmented generation (e.g., multi-hop question answering). We
plan to explore each of these topics in future works.

Reviewer comment (Reproducibility):
How did you integrate multiple knowledge graphs with different ontology?

Author response:
Entity alignment was performed by identifier mapping. In other words, we relied on the entity mappings provided by the knowledge bases we used, not on a computational method that would be less reliable such as fuzzy matching or embedding-based approaches. Once we identify the relevant data sources, the process to align the entities is somewhat straightforward.

An overview of the mappings from each data source is shown in the Appendix Figure 2. An example of the mapping process is described in the beginning of section 3.1. An overview of the mappings for each biomedical category is shown in the Appendix, starting at the bottom of page 21. The organized code to perform the mappings is provided in the GitHub repository under Know2BIO/dataset/create_edge_files_utils which we provide as a footnote in a few locations throughout the paper. 

Given this explanation, does the initial weakness still remain?

Reviewer comment (Reproducibility):
What techniques did you use to perform entity resolution, entity disambiguation?

Author response:
Entity alignment was performed by identifier mapping. In other words, we relied on the entity mappings provided by the knowledge bases we used, not on a computational method that would be less reliable such as fuzzy matching or embedding-based approaches. Once we identify the relevant data sources, the process to align the entities is somewhat straightforward.

An overview of the mappings from each data source is shown in the Appendix Figure 2. An example of the mapping process is described in the beginning of section 3.1. An overview of the mappings for each biomedical category is shown in the Appendix, starting at the bottom of page 21. The organized code to perform the mappings is provided in the GitHub repository under Know2BIO/dataset/create_edge_files_utils which we provide as a footnote in a few locations throughout the paper. 

Given this explanation, does the initial weakness still remain?

Reviewer comment (Experiments):
What challenges did you face while merging the different KGs?

Author response:
identify the relevant data sources, the process to align the entities is somewhat straightforward.
However, it was arduous to curate the proper data sources that provide the mappings (e.g., to find a path of “is_a” relationships between the head and tail entities from two data sources). We had certain biomedical relationships we were interested in (e.g.,  gene-disease, protein-function, or drug-side effect), based on our domain knowledge (our data curation team has backgrounds in biological sciences.) To find data sources that provide this information, we searched online, looked into data sources used by other well-known biomedical knowledge graphs, and investigated data sources we had known previously. Through this process, we became very familiar with the available resources and which ones were commonly used and reputable. To choose the main identifier to use for each data type (e.g., UniProt ID for all protein nodes), we depended on the identifier mappings that were available and used the most. For example, UniProt IDs are used almost universally, though some data sources have their IDs (e.g., STRING, TTD). Therefore, we chose to use UniProt and map the less commonly used IDs to UniProt. To reiterate, the decisions about biomedical data types were informed by our biomedical understanding and data availability; the decisions about the identifier mappings were informed by our understanding of and wrangling with the data sources.

There are other KBs with additional data not contained in our 30 data sources which could be useful (e.g., clue.io). We are considering supplementing the KG with this information. However, the data is currently quite extensive, both in terms of biomedical information and data sources, and includes numerous major biomedical knowledge bases. 

Does this answer clarify our process or do the questions remain? We would be happy to provide more information and update the paper accordingly if needed.

Reviewer comment (Experiments):
Since you have mentioned the use of 30 KBs for your KG construction, how did you select these? What about other KGs present? Is there any significant KB not utilized?

Author response:
identify the relevant data sources, the process to align the entities is somewhat straightforward.
However, it was arduous to curate the proper data sources that provide the mappings (e.g., to find a path of “is_a” relationships between the head and tail entities from two data sources). We had certain biomedical relationships we were interested in (e.g.,  gene-disease, protein-function, or drug-side effect), based on our domain knowledge (our data curation team has backgrounds in biological sciences.) To find data sources that provide this information, we searched online, looked into data sources used by other well-known biomedical knowledge graphs, and investigated data sources we had known previously. Through this process, we became very familiar with the available resources and which ones were commonly used and reputable. To choose the main identifier to use for each data type (e.g., UniProt ID for all protein nodes), we depended on the identifier mappings that were available and used the most. For example, UniProt IDs are used almost universally, though some data sources have their IDs (e.g., STRING, TTD). Therefore, we chose to use UniProt and map the less commonly used IDs to UniProt. To reiterate, the decisions about biomedical data types were informed by our biomedical understanding and data availability; the decisions about the identifier mappings were informed by our understanding of and wrangling with the data sources.

There are other KBs with additional data not contained in our 30 data sources which could be useful (e.g., clue.io). We are considering supplementing the KG with this information. However, the data is currently quite extensive, both in terms of biomedical information and data sources, and includes numerous major biomedical knowledge bases. 

Does this answer clarify our process or do the questions remain? We would be happy to provide more information and update the paper accordingly if needed.