Paper: CICD-Coder: Chinese EMRs Based ICD Coding With Multi-axial  Supported Clinical Evidence (ICLR-2024)

Reviewer comment (Reproducibility):
Will the data used for evaluation will be publicly avilable?

Author response:
Thank you for your valuable feedback.
The Chinese electronic medical record (EMR) dataset we utilized in our study was collected from a collaborative research effort involving four different hospitals. As we are conducting joint research with these institutions, the issue of dataset availability needs to be discussed and agreed upon with our collaborative partners. We are actively engaged in these discussions to ensure that any decisions regarding the dataset's public availability align with the interests and privacy considerations of all involved parties.

Reviewer comment (Novelty):
The paper overlooks crucial existing research, particularly the study outlined in [1]. That seminal work similarly employs prompt-based mask prediction for ICD code probability, and its omission here represents a significant gap in the literature review.

Author response:
Thank you for your valuable feedback.

Prompt tuning is indeed an effective framework for integrating downstream tasks with pretraining, and it has been widely adopted and utilized in the field of automated ICD coding in recent years.  In the original manuscript, we discussed this point and cited a paper that uses this framework, namely "Multi-Label Few-Shot ICD Coding as Autoregressive Generation with Prompt" published in AAAI 2023. We have taken your feedback into account and have now included the study you mentioned in our revised manuscript.

Reviewer comment (Reproducibility):
The explanation of key methodologies, especially the evidence retrieval module, is vague. Given its critical role in enhancing ICD coding performance, a more in-depth discussion of its design and functionality is essential for readers to fully understand and replicate the study.

Author response:
In response to your concern regarding the explanation of key methodologies, particularly the evidence retrieval module, we have provided a more detailed discussion in the appendix to enhance the understanding and replicability of our study.

Reviewer comment (Experiments):
Certain claims appear unsubstantiated, creating potential confusion. For instance, the assertion regarding the brevity of diagnoses in Chinese EMRs contradicts common characteristics seen in datasets like MIMIC. This discrepancy necessitates clarification to maintain the paper's credibility.

Author response:
To address this concern and substantiate our claim, we conducted a thorough comparison between Chinese and English EMR datasets. Specifically, we collected data from Chinese EMRs and analyzed the average length of diagnostic descriptions and discharge summary. Our findings revealed that the average length of Chinese diagnostic descriptions was 3.74 words, whereas the average length of the discharge summary was 731.80 words. In contrast, when using the widely used MIMIC-III dataset for English EMRs, we observed that the average length of English diagnostic descriptions was 14.73 words, and the average length of discharge summary was 1600.36 words. To provide further evidence, we have included more comprehensive statistical information in Table 1 of the paper.

Based on these extensive statistics, it is evident that diagnostic descriptions in Chinese EMRs are significantly briefer and lack sufficient information compared to their English counterparts. Moreover, discharge summary in Chinese EMRs are also shorter and insufficient to support automated ICD coding. We believe that these additional details and comparisons will enhance the credibility and clarity of our paper.

Reviewer comment (Novelty):
The paper exhibits limited novelty when viewed against the backdrop of existing studies like [1]. The underdeveloped evidence retrieval module further diminishes the perceived innovativeness of the CICD-Coder framework. A more thorough exploration of these elements could help underline the unique contributions of the current study.

Author response:
Our research primarily focuses on designing an automatic ICD coding framework that addresses the challenges posed by the characteristics of Chinese electronic medical records (EMRs). In Chinese EMRs, diagnoses are often concise, and discharge summary may lack sufficient information, resulting in scattered evidence supporting ICD coding throughout the entire medical record. It is in response to these characteristics that we have designed a methodological system that combines a domain-specific knowledge-driven evidence retrieval module with a multi-axes evidence-supported ICD coding inference module.

Certainly, there is room for further optimization of each specific technical module in our approach. The retrieval module we have developed is primarily based on the characteristics of Chinese EMRs, as well as the prior knowledge and expert experience related to diseases. Through the evidence evaluation module, the retrieved evidence has already demonstrated its effectiveness in supporting ICD coding inference and providing good interpretability. Therefore, we did not pursue further detailed optimization in this particular aspect. However, we appreciate your suggestion, as further optimization could potentially lead to performance improvements. The main focus of this study is to propose a framework that is more suitable for automatic ICD coding in Chinese EMRs. Subsequent work can delve into exploring each module in more detail.

Reviewer comment (Reproducibility):
Will the dataset be available? If not, have you considered applying the proposed method to some public datasets e.g. MIMIC-III and IV? If not, what makes it impossible?

Author response:
The Chinese electronic medical record (EMR) dataset we utilized in our study was collected from a collaborative research effort involving four different hospitals. As we are conducting joint research with these institutions, the issue of dataset availability needs to be discussed and agreed upon with our collaborative partners. We are actively engaged in these discussions to ensure that any decisions regarding the dataset's public availability align with the interests and privacy considerations of all involved parties.

While we recognize the value of applying our method to English EMRs, it would require significant modifications and adaptations to account for the differences in data structure and language characteristics. Exploring the application of our method to English EMRs is an intriguing avenue for future research, and we appreciate your suggestion in this regard.

Reviewer comment (Evaluation):
While the proposed method in the paper does improve over its baselines, the improvement does not have a large margin and is not significant.

Author response:
Firstly, we would like to emphasize that ICD coding is a challenging task due to the large number of codes involved. Therefore, achieving significant improvements in performance metrics can be difficult.

In our study, we compared our proposed method, CICD-Coder, with baselines that also take the evidence set as input. On average, our method outperforms these baselines by 26.70\%, 23.48\%, 24.03\%, 5.28\%, and 4.86\% in precision, recall, F1, accuracy, and p@5 metrics, respectively. The significant improvement in macro-averaged metrics indicates that our method performs better in recommending rare or tail ICD codes and the enhanced discriminative ability for distinguishing between confusing ICD codes. This improvement can be attributed to our approach of finding supporting evidence sets under each candidate code axis and reasoning about the overall code support based on this evidence. At the same time, our method also exhibits distinct evidence-based differences for codes that are prone to confusion, ultimately improving code discrimination.

Regarding the lower performance of our method compared to MSMN in the p@5 metric, we would like to clarify that MSMN performs classification within the entire ICD code label space, while our method performs binary classification for each candidate code individually within the label space of all recalled candidates. Therefore, the upper limit of our method's performance in p@5 is essentially the recall rate of the candidate codes. The recall rate for candidate codes is approximately 95\%-96\%. Consequently, this limitation can explain the difference in performance for p@5 compared to MSMN. However, there is a solution to address this issue. The recall rate of candidate codes can be effectively improved by increasing the number of candidate codes.

Reviewer comment (Theory):
In 2.2.1, the author says that **We have prior information about the likely appearance location of the supporting evidence in the medical record by professional ICD coders**. In the last sentence, they say that this prior information makes the process **fast and efficient**. Given that the paper is application-oriented, I wonder what would happen when practitioners do not have this prior information. It seems to be that having such prior information is a strong assumption.

Author response:
Thank you for your valuable feedback regarding the assumption of prior information in our paper. To address your concern, we provided a detailed explanation in the first paragraph of section 2.2.2 regarding the acquisition method of prior knowledge. 

In the retrieval module, this prior information plays a crucial role in improving the efficiency of evidence retrieval. With prior knowledge, we can narrow down the search scope from the entire EMR to the likely appearance locations, and only resort to searching the entire medical record when the retrieved evidence is insufficient. We have estimated that the evidence retrieval strategy empowered by this prior knowledge can achieve a speed improvement of approximately 18 times compared to directly searching the entire medical record.

In terms of reliability, evidence retrieval based on the entire medical record may yield a higher recall rate of reliable evidence. However, it is indeed a trade-off between recall and reliability. We have evaluated that, under the same number of supporting pieces of evidence, the evidence retrieval strategy based on prior knowledge can achieve approximately 33\% higher reliability of the evidence set compared to the strategy without prior knowledge.

Reviewer comment (Writing):
Minor issue: part of the paper contains grammar issues.

Author response:
In the revised version that we have submitted, we have conducted a thorough grammar check and made the necessary corrections throughout the paper.

Reviewer comment (Novelty):
Given that the paper's focus is on Chinese EMRs, I am not sure whether some setups made in the paper are relevant or specific to Chinese EMR. For instance, for the two principles made in the paper, the first principle is **The source of the evidence limits what the evidence can describe and how credible it is.**, and the second principle is **The number of repeating times of one piece of evidence in the Chinese EMR determines its importance**. I am not sure whether those two principles are specific to Chinese EMRs. It seems to me that those two principles could broadly apply to EMRs in most countries. Thus, it is debatable whether the design of the principles is specific to Chinese EMR, which makes those two design principles seem a bit too general. I wonder whether the authors have any idea about the specific design of the principles.

Author response:
Thank you for your thoughtful comments and suggestions. We agree that the two principles mentioned in our paper could potentially apply to electronic medical records in other languages as well, and yield benefits.

The reason we emphasized these principles in the context of Chinese EMRs is that most of the existing automatic ICD coding methods were originally developed for English EMRs. We found that the length of discharge summary in English EMRs is more than twice that of Chinese EMRs. As a result, many automatic ICD coding methods for English EMRs directly use the discharge summary as input. However, in Chinese EMRs, the supporting evidence for each ICD code may be scattered throughout different sections of the medical record. Therefore, the source and frequency of appearance of each piece of evidence become meaningful.

By incorporating these two pieces of information into our prompt, we aim to provide additional valuable information for determining whether a set of evidence can support the reasoning task of candidate code selection. Based on the ablative experiments in the last two rows of Table 3, it is indeed observable that incorporating this information can lead to performance improvements.

Reviewer comment (Novelty):
The motivation for focusing on Chinese EMRs in the study is inadequately substantiated. The problems highlighted by the authors are not unique to Chinese EMRs and are also prevalent in other languages, including English. A stronger rationale is required to establish the relevance and significance of this specific focus.

Author response:
Thank you for providing your valuable feedback. While we acknowledge that the problems we highlighted are not unique to Chinese EMRs and can be found in other languages, including English, we believe that these issues are more pronounced in Chinese EMRs due to the specific characteristics of the Chinese language.

To support our claim, we conducted a comparison between Chinese and English EMR datasets. We collected data from Chinese EMRs and found that the average length of Chinese diagnostic descriptions was 3.74 words, while the average length of the discharge summary was 731.80 words. In contrast, using the commonly used MIMIC-III dataset for English EMRs, we found that the average length of English diagnostic descriptions was 14.73 words, and the average length of discharge summary was 1600.36 words. We have provided more detailed statistical information in Table 1 of the revised manuscript.

Based on the aforementioned statistics, it is evident that diagnostic descriptions in Chinese EMRs are significantly shorter and lack sufficient information. Moreover, discharge summary are also shorter and insufficient to support automated ICD coding. Additionally, clinical evidence supporting ICD coding across different axes is scattered throughout the entire Chinese EMR, making existing automated ICD coding methods that are effective in English EMRs less effective in Chinese EMRs.

Therefore, we believe there is a strong motivation to propose a specialized automatic ICD coding method for Chinese EMRs.

Reviewer comment (Reproducibility):
In Section 2.2.1 the authors mention that they obtain the ICD codes. Which method is used to obtain the initial ICD codes?

Author response:
Thank you for your comment. We appreciate your observation that there is a paragraph below Section 2.2 and above Section 2.2.1 that explains how we obtain the candidate codes, but it lacks a specific subheading. To improve the overall logical flow and organization of the paper, we have now added a new subheading, "2.2.1: Obtaining Candidate Codes for Each Diagnosis," to address the method used to obtain the initial ICD codes.