Paper: Encoding Ontologies with Holographic Reduced Representations for Transformers (ICLR-2024)

Reviewer comment (Reproducibility):
**Reproducibility**: To faithfully replicate the study's findings, more detailed information is necessary. Specifically, clarity on the dataset employed, the design of the architectural models, and the implementation specifics of all the variant baselines would be indispensable.

Author response:
Regarding reproducibility, we have improved clarity regarding both the mapping of ICD to SNOMED and data preprocessing. Furthermore, we have provided supplementary materials which contain all code necessary for implementation. The code is properly documented and will be released open source on publication of the manuscript. We aim with our open source code to eliminate any ambiguity that remains in the manuscript.

Reviewer comment (Evaluation):
Fig. 1: Why does the HRRBase have such a noticeably lower score compared to the other methods? Does this suggest that HRRBase is the least effective?

Author response:
It suggests that HRRBase is the least effective in terms of the MLM pre-training metric of accuracy, although performance on fine-tuning tasks is of more interest. However, the main take-away from this is the comparison of HRRBase with HRRFreq. HRRFreq is identical to HRRBase except that it explicitly encodes token-frequency information, and it performs on par with unstructured embeddings. As we now explain in Section 3.1, this suggests that the MLM performance difference is due to frequency information, which (as shown in Figure 2) is prominently represented in the unstructured embeddings.  

Embedded code frequency information can help models make predictions and classifications. However, code frequency information is not medically causal, so using such information to aid medical prediction may not be desirable. At least, the influence of frequency should be explicit. Our HRRFreq method allows toggling the inclusion and exclusion of frequency information to understand its influence on the model’s predictions.

Reviewer comment (Evaluation):
Table 1: While the performances of all methods seem to be on par, what distinguishes HRR from the unstructured approach in terms of advantages?

Author response:
Table 1 and the associated statistical analysis shows that there are modest performance increases with our proposed methodology, particularly for mortality prediction. Our new experiments show additional situations where HRR outperforms unstructured embeddings, including a strong advantage when the input consists of codes that were unseen in training. 

Overall, the key advantages are improved generalization to codes that are unseen in training, and explainability, particularly with respect to code-frequency effects.

There are additional differences in representation that may provide further practical advantages, although further work is needed to understand these: 
- HRR embeddings produce broader distributions of cosine similarities
- HRR embeddings do not learn a co-occurrence bias that is common in other methods of learning medical embeddings.

Reviewer comment (Presentation):
Figure 2: Why is there no visualization for the control method, 'Unstructured'? Given that the embedding spaces in the current Figure 2 appear quite analogous, what's the primary takeaway or message from this representation?

Author response:
We do include visualizations for ‘unstructured’, there are 4 graphs in figure 2, (a) unstructured, (b) HRRAdd, (c) HRRCat, and (d) HRRBase. Graphs (b), (c), and (d) are quite analogous but appear different from (a).

In (a) unstructured, we can see that there is a larger gap between frequent and infrequent codes when compared to (b), (c), and (d) which demonstrates that a frequency bias is learned in the unstructured approach that wasn’t learned in any of the HRR approaches. Frequency bias is not desired in the embedding of medical codes as it does not respect medical similarity/reasoning. Medical codes that have similar embeddings should also be medically similar with respect to human knowledge.

Reviewer comment (Experiments):
Evaluation is only performed on the MIMIC-IV dataset collected from a single medical center. Further evaluation of multi-center datasets such as eICU could present further information on the method's usefulness.

Author response:
Thank you for your review. We appreciate your feedback and suggestions. As per your primary suggestions, we have added additional evaluations on the multi-center eICU dataset to A.1. As data from eICU cannot be ordered sequentially, we were not able to apply disease prediction to the dataset. We performed mortality prediction on the eICU dataset with the MIMIC-trained model. There is a distribution shift because eICU counts mortality is defined in a different time window than MIMIC. However, our method showed a small performance improvement. Due to the time constraint, we could not redo the entire training process and analysis with eICU. MIMIC has advantages for our analysis because it has more patients with longer sequences of visits. However, further work with eICU would be a good direction for the future.

Reviewer comment (Evaluation):
Given the small performance boosts demonstrated in Table 1, I wonder under what circumstances we need to adopt the proposed method.

Author response:
Addressing your question:

**Given the small performance boosts demonstrated in Table 1, I wonder under what circumstances we need to adopt the proposed method.**

Our methods should be applied in use cases with imbalanced datasets and where frequency bias is undesirable. Given the existence of frequency bias in the unstructured embeddings, it is unclear how frequency bias learned by the unstructured model contributes to correct or incorrect predictions in fine-tuning tasks. The embeddings produced by our method are less dominated by frequency (as shown in the tSNE plots, Figure 2). As frequency is not a desirable bias in clinical settings, reducing frequency bias in medical AI models may improve trust. 

Relatedly, the method should also be used when it is important to understand a network’s reliance on token frequency in a particular decision. Our method disentangles frequency information from semantic information. Transformers have a well-known frequency bias, and we have shown that frequency representation is prominent in learned embeddings. Our method incorporates token-frequency information in a structured way. Although frequency information can be important for accurate predictions (as we have shown by comparing HRRBase to HRRFreq), the frequency information in our embeddings can be added, removed, and modified as needed to examine the role of frequency on individual decisions by the network, enhancing its explainability. In principle, other elements of our embeddings could be treated in a similar way, e.g. to determine the impact of a condition’s category vs. its site in the body. We acknowledge that these important points were unclear in our original submission.

Reviewer comment (Writing):
The manuscript could benefit from additional explanations of abbreviations such as "SNOMED CT." A glossary or expanded definitions on first use would aid comprehension, especially for readers unfamiliar with the terminology.

Author response:
**1. The manuscript frequently uses the abbreviation "SNOMED CT" without providing a full explanation or definition for readers who may be unfamiliar with the term. Could you please provide a brief description of "SNOMED CT" and its relevance to your work in the introduction or the first instance where it is mentioned?**

We’ve added a brief introduction about SNOMED CT in section 1.1 at the end of paragraph 5. More information on SNOMED CT is included in A.1 as well.

Reviewer comment (Experiments):
References to critical resources like the "MIMIC-IV dataset" are absent. Citing such resources would provide context and allow readers to assess the relevance and applicability of the data.

Author response:
**2. Dataset Citation:The "MIMIC-IV dataset" is a crucial element in your research, yet it lacks a proper citation or reference. Could you please add a citation for the dataset to allow readers to trace the source and potentially reproduce your study?**

We’ve added citations to MIMIC-IV in paragraph 3 of section 1.

Reviewer comment (Reproducibility):
Section 2.1 lacks detailed descriptions of dataset handling procedures, which is not reader-friendly for those not acquainted with the "MIMIC-IV dataset." Providing more detail would enhance reproducibility and understanding.

Author response:
**3. Rationale Behind Mapping Method: Section 2.2.1 does not sufficiently explain the reasoning behind the chosen ICD mapping method. Why did you opt for the particular mapping approach used in the study instead of mapping all ICD-10 codes to ICD-9 codes, and then to "SNOMED CT"? Clarifying the rationale behind this decision would be helpful.**

We’ve added more detail on dataset handling to section 2.1. Briefly:
- We used the ICD-9 and ICD-10 diagnostic codes from the diagnosis table from the MIMIC-IV hosp module. 
- We filtered patients who did not have at least one diagnostic code associated with their records. 
- Sequences of codes were generated per patient by sorting their hospital visits by time. 
- Each unique code was assigned a token.

Reviewer comment (Experiments):
Section 2.2.1 does not adequately explain the rationale behind the chosen ICD mapping approach. Specifically, it is unclear why the authors did not map all ICD-10 codes directly to ICD-9 codes and subsequently to SNOMED CT, but rather employed the separate mapping method described. This decision is perplexing and warrants a thorough justification to understand the advantages or the necessity of the approach taken.

Author response:
**4. Dataset Processing Details: In Section 2.1, the description of how the dataset was processed is somewhat brief. Providing a more detailed account of the preprocessing steps would be beneficial, especially for readers who are not familiar with the dataset. Could you elaborate on this process?**

We’ve moved some of our mapping methodology from section 2.2.1 to the appendix where we’ve added in additional rationale (A.8). Describing the rationale:
- ICD-10 was not mapped directly to ICD-9 to maintain preciseness of ICD-10 codes while potentially still keeping codes without mapping from ICD-10 to ICD-9.
- OMOP mapping greatly improved the coverage (i.e. codes that have a valid mapping) of our ICD-to-SNOMED mapping to 96.5% from 77.6% when only using ICD-9/10 to SNOMED mappings.

Reviewer comment (Evaluation):
The choice of employing a "one-sided Dunnett’s test" in Section 3.2 is not substantiated with an explanation. It is essential for the authors to clarify this methodological decision, especially since the results in Table 1 suggest an overlap in the confidence intervals and a close proximity of mean values between the experimental and control groups. The absence of such a justification leaves readers questioning the appropriateness of the statistical test used in the analysis.

Author response:
**5. The choice of a "one-sided Dunnett’s test" in Section 3.2 requires further explanation. Could you elaborate on the reasons for selecting this test and whether your method adheres to the assumptions of the test? Additionally, if the decision to use Dunnett's test was deliberate, why opt for a one-sided test instead of a two-sided test? Do you have supporting rationale to support this test? The clarity on these points is crucial, especially in light of the overlapping confidence intervals and closely aligned mean values between the experimental and control groups as presented in Table 1.**

We’ve added reasoning as to why we chose one-sidedness over two-sidedness in paragraph 1 of section 3.2. We chose a one-sided test because we suspected that HRR methods would improve performance above unstructured controls. 
As the Dunnett’s test is based on computing multiple t-tests, the same assumptions apply for both. Our methods follow with assumptions required by both the Dunnett’s test and t-test (i.e. data is continuous, data is representative, homogeneity of variance, and distribution is approximately normal).


As Dunnett’s test is designed for multiple comparisons of treatments to a control, it suited our case where we wished to compare our HRR configuration performance to the unstructured control. 
We have also computed t-tests of our results which agree with our Dunnett’s test analysis.

Reviewer comment (Evaluation):
While Section 3.3 commendably visualizes "highly frequent codes" and "infrequent codes," the lack of separate predictive performance displays for these two categories in Table 1 is a missed opportunity. This differentiation is, after all, one of the paper's stated goals.

Author response:
**6. While the visualization of "highly frequent codes" and "infrequent codes" in Section 3.3 is commendable, the performance metrics under these two categories are not separately displayed in Table 1. Displaying results across various scenarios could effectively showcase the effectiveness of your approach, especially since addressing the representation of infrequent codes is one of your stated objectives. Could you include these performance metrics in your results?**

Thank you for this suggestion, we have added this analysis to A.2 (and to some extent, A.3) of our manuscript. We were unclear whether our approach would improve prediction of rare codes, as that may require experience with rare codes in training context. It should more directly improve predictions that have rare codes in their input.

Reviewer comment (Presentation):
Finally, the paper does not sufficiently discuss the limitations of the proposed method. Acknowledging and addressing potential shortcomings would strengthen the paper by providing a balanced view and suggesting directions for future research.

Author response:
**7. A discussion on the limitations of the proposed method is noticeably absent. An acknowledgement of potential limitations and constraints of your approach would provide a more balanced view and contribute to the paper’s integrity. Could you add a section discussing these aspects?**

We’ve added discussions of limitations into section 4. The first limitation is the complexity of creating HRR compatible knowledge graphs from ontologies. The second limitation is that the HRR method is still 1.5x slower compared to unstructured. Finally, our method relies on rare-code HRRs sharing atomic elements with common codes. However, rare codes are also likely to contain rare atomic elements. This is the likely explanation for separation of rare and common codes in the tSNE reduction of our HRRBase embeddings, which do not contain unstructured elements or explicit frequency information. This limitation suggests a way to improve the method in the future, by incorporating additional information about rare atomic elements (e.g. initializing with pretrained language embeddings of the description words).