Paper: Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML (ICLR-2024)

Reviewer comment (Experiments):
The baseline table is useful for future work to compare against. However, this table does not seem relevant to motivating the proposed framework and could be included in the appendix. To better motivate the value of the proposed unifying framework, it would be useful to show more experiments regarding significant differences in model performance when different experiment methods are used (outcome labels, cohort selection, etc.). For example, you could consider an ablation study to show that model performance significantly improves or decreases based on various outcome definitions for datasets with consistent preprocessing, model choice, hyperparameters. The experiment mentioned in the paper regarding exclusion criteria would also be a good example to elaborate on and include table results comparing impact of different exclusion criteria.

Author response:
**Baseline table**

We acknowledge the possibility of more experiments and thank the reviewer for providing examples. Unfortunately, we have had to be selective in the experiments we included. While writing the paper, we performed several experiments, that partially ended up in the appendix due to the lack of space. The appendix contains results for 1) the use of dynamic feature generation, 2) excluding static features, 3) out-of-the-box results with demo cohorts provided with our repository, 4) Length of Stay external validation, and 5) AUPRC results for the fine-tuning experiment in the main text. 

  

Additionally, we recognize that many experiments can be performed in addition to the baseline. However, we believe one of our main contributions is demonstrating a baseline of different tasks across datasets and models to compare their work. We, therefore, respectfully disagree with the reviewer that a table with baseline results across models, tasks, and datasets does not motivate our framework. Future work can directly utilize these results to get an understanding for the performance of their model, dataset, preprocessing pipeline, or clinical task. Furthermore, including it in the main text is important for interpreting our work as a reference paper. If we were to include it in the appendix, we risk misinterpretation of our work, whereas we want to communicate the goal of YAIB as a baseline as clearly as possible.

Reviewer comment (Evaluation):
Be specific about performance gains and losses (example: improved by XX%) and their statistical significance. A significant difference in model performance due to unaligned data preprocessing and processing pipeline would help motivate the proposed unifying framework. Stating that “differences were bigger for some datasets than others” is not a robust explanation.

Author response:
**Performance gains and losses**

We thank the reviewer for pointing out the comment about performance gains and losses but are not exactly sure what the reviewer means. In the mentioned section, we have removed the sentence mentioned and added percentage-wise difference ranges when comparing approaches against each other. We also provide standard deviations for each result.  We hope this addresses the reviewer's comment.

Reviewer comment (Reproducibility):
Novel ML research often proposes a new model that is evaluated across multiple datasets. Enabling support for new models to be easily integrated into this framework is important. The appendix provides example code for training a specific model, such as those in scikit-learn or implemented in pytorch. Including more thorough explanation and/or code examples for how to train a novel/custom model, such as pytorch-based model implemented in a separate python file, would be useful.

Author response:
**Integration of Novel ML**

The reviewer would like additional explanation on the ease of integrating novel ML. As the main paper claims, adding new models to YAIB is straightforward. We provide examples in Appendix E.2 on extending YAIB concerning datasets, concepts, tasks, models, and metrics to demonstrate this. Among these examples is a Temporal Fusion Transformer ([https://arxiv.org/abs/1912.09363](https://arxiv.org/abs/1912.09363)), which may be considered state-of-the-art and was recently added by a benchmark user. 

  

We additionally refer to the work added to the supplementary materials (Closing Gaps: An Imputation Analysis of ICU Vital Signs) which has been accepted to the First Workshop on Generative Modelling for Health at NeurIPS 2023 and uses YAIB as a base. 

  

To accommodate the reviewers' comments, we have added three files; these are *rnn.gin, rnn.py, and instructions.md* in the folder *Adding a model* of the supplemental material. to the supplemental material to demonstrate the ease of adding an RNN model. Additionally, we have added a wiki to show how to add new models; this usually comes down to adding just two files (model specification and hyperparameter bindings). Finally, Appendix F.3 shows how users can transparently share new setups with YAIB as core.

Reviewer comment (Reproducibility):
The author mentions flexibility of the repo in allowing users to configure data preprocessing in a streamlined fashion across multiple datasets. However, does this somewhat deflate the original purpose to enable reproducible research through consistent dataset preprocessing and processing. Researchers may still conduct research with various preprocessing steps that differ across literature. The value of this framework hinges on if research exactly states the detailed use of YAIB in their paper/methods AND if future work follows these exact same steps. It would be useful if the author's address this point and potentially expand their unifying framework to include recommended approaches for how to reference/cite use of YAIB in future work that could promote reproducible research.

Author response:
**Flexibility or tradeoff of extensibility and reproducibility**

The reviewer argues that providing extensibility could sacrifice the reproducibility of YAIB. We provide an out-of-the-box end-to-end pipeline that can be used for benchmarking. However, many users would like to tweak aspects of this pipeline to create novel benchmarks. We allow for this use case as well a “pure” benchmarking against a new method without changing any of the code.  We acknowledge that we can never create a benchmark that fits the requirements of all users going forward.

  

We designed YAIB to be as extensible as possible while retaining full reproducibility. This means easy support of new databases, clinical concepts, tasks, experiment configurations, preprocessing pipelines, imputation methods, models, and evaluation metrics. If changes are necessary, they need to be reproducible and easily shareable across research teams. 

If the user only requires a few default ICU tasks from a single i.i.d. dataset to test their new method, any existing ICU benchmarks could be sufficient. Users do not need to apply for access to multiple datasets and do not have to deal with the intricacies of the clinical task definition. As long as the integration of a new model is seamless, such simple frameworks are fit-for-purpose and abstract much of the complexity, allowing the user to only worry about one thing: their model. If multiple papers used the exact same benchmark, results are also directly comparable between papers (an “apples-to-apples” comparison).

  

However, we found that this setup tends to be too restrictive and thus unrealistic. Users often want to highlight a particular aspect of their model, prompting them to adapt to the default task. At other times, they want to show clinical impact and need to adapt the default task to make it more realistic. Given the lack of successful translation of prediction models into clinical practice, reviewers are also increasingly requesting external validation – sometimes with multiple endpoints – which is difficult to shoehorn into most existing solutions.

YAIB embraces the need to tweak experimental setups. Results will no longer be directly comparable between papers, but we argue that actual apples-to-apples comparisons were inherently rare. Instead of forcing users into a rigid framework, it allows for adaptations but requires them to be done transparently. Absolute performance should be compared only within the same paper or among papers with the same task setup (see our examples in Tables 5 and 6).

  

To facilitate the transparency of adaptations, we rely on a sophisticated framework to define clinical concepts across multiple datasets (ricu). We have adapted and extended ricu to provide a standard workflow for YAIB to integrate new databases and define new clinical concepts. To date, it has been successfully used to bring 4/5 +1 ICU datasets into a common format (including our addition of the Salzburg Intensive Care Database, which is currently in quality control). This approach is flexible enough that we have yet to encounter significant restrictions in mapping admissions, demographics, vital signs, laboratory values, medication (including rates and durations), clinical scores, and outcomes at different time scales across datasets. The main restriction of ricu is that it is currently implemented in the R language only, but we provide guidance on how to access it via rpy2, and we are in the process of porting it to Python; this will make our pipeline even more accessible, especially to clinical researchers. Our cohort definition functionality provides helper functions to apply inclusion/exclusion criteria on top of ricu and report step-by-step attrition numbers. The cohorts can be used in a modular fashion with custom preprocessing steps, imputations, prediction models, and evaluation metrics, all using the exact same code across multiple datasets.

  

Even so, there will likely be situations where the user may be better off with a custom solution. We expect this to occur once their use case diverges significantly from standard supervised learning. For example, federated learning or reinforcement learning setups may require significantly different training and evaluation loops. These are not currently supported, but we consider this as future work. In any case, the user can still use our data processing, cohort generation, and possibly other parts of YAIB (e.g., by exchanging the default training module with a custom module). Authors using YAIB should, therefore, provide their code and a detailed list of the changes they have made to the repository; modern version control allows us to verify this against the original YAIB repository easily. The newly added Appendix F.3 addresses the exact steps to provide transparency for novel work.

Reviewer comment (Reproducibility):
It is possible that thresholds used to define outcomes from time series data may update over time based on new knowledge, thus impacting outcome labels. Does the repo have support to enable users to define their own outcome definitions across datasets? For example, a patient AKI outcome may occur when patient creatinine falls below a specific threshold. Or are outcome labels strictly based on definitions included in the datasets?

Author response:
**Support for defining new outcomes**

As we use the ricu framework, one can define new tasks on the basis of the existing data. This means that we can redefine thresholds and generate new cohorts that reflect a different clinical prediction task. We have added a YAIB wiki to show the process of adding new tasks with the updated supplemental material. Additionally, we refer to the ricu user guide ([https://eth-mds.github.io/ricu/](https://eth-mds.github.io/ricu/)) and Appendix E.3. We note that most outcomes are not a label in the data; they are a complex combination of events. For example, for sepsis use of it is the use of antibiotics while there is a suspicion of infection (for details see YAIB-cohorts/R/sepsis.R in the supplementary materials).

Reviewer comment (Reproducibility):
Are models other than those in scikit-learn hardcoded into the repo or are there wrappers for these as well? Is there support for users to include and evaluate new/customized model architectures in the pipeline?

Author response:
**Adding new models**

We provide extensive support for adding new models as this is one of the main use cases of YAIB. Both ML and DL model have one common wrapper from which they inherent. We have a separate wrapper for deep learning models as they have the mechanics of a neural network (requiring e.g., a forward method). Adding a model and a new dataloader can be done in a straightforward manner by subclassing them.  We refer to the newly added wiki in the supplementary materials (Adding-a-new-model.md) and Appendix E.4.

Reviewer comment (Novelty):
Is there a reason why this framework is limited to ICU/healthcare settings and not time series in general. I could see a pipeline like this being applicable to time series from many domains. It may be worth elaborating on why the framework is ICU-specific. It may also be useful to compare YAIB to similar non-healthcare related frameworks.

Author response:
**Generalizability to other domains**

Healthcare is a complicated application area of ML. We see this in the lack of adoption of ML in this domain [1,2]. The complete end-to-end pipeline that YAIB provides has been designed for temporal ICU data. The data harmonization and cohort creation are currently specific to ICU data. We acknowledge that extending the framework to general EHR data and other time-series data can be straightforward in some cases. However, we believe that is infeasible to achieve a completely domain-agnostic pipeline; we have instead focused on the end-to-end pipeline that is able to unite both clinicians and ML researchers. One of our main motivations is to solve the lack of reproducibility in clinical ML [1,2], which impedes clinical studies with ML models.  Moreover, given the specific use cases for YAIB, we believe that it is not useful to compare to more works than the 13 works in Table 1. 

  

*[1] Lucas M. Fleuren, Patrick Thoral, Duncan Shillan, Ari Ercole, Paul W. G. Elbers, Mark Hoogendoorn, Ben Gibbison, Thomas L. T. Klausch, Tingjie Guo, Luca F. Roggeveen, Eleonora L. Swart, Armand R. J. Girbes, and Right Data Right Now Collaborators. Machine learning in intensive care medicine: Ready for take-off? Intensive Care Medicine, 46(7):1486–1488, July 2020b. ISSN 1432-1238.doi: 10.1007/s00134-020-06045-y.*

*[2] Bar Eini-Porat, Ofra Amir, Danny Eytan, and Uri Shalit. Tell me something interesting: Clinical utility of machine learning prediction models in the ICU. Journal of Biomedical Informatics, 132: 104107, August 2022. ISSN 15320464. doi: 10.1016/j.jbi.2022.104107.*

*[3] Tabinda Sarwar, Sattar Seifollahi, Jeffrey Chan, Xiuzhen Zhang, Vural Aksakalli, Irene Hudson, Karin Verspoor, and Lawrence Cavedon. The Secondary Use of Electronic Health Records for Data Mining: Data Characteristics and Challenges. ACM Computing Surveys, 55(2):1–40, March 2023. ISSN 0360-0300, 1557-7341. doi: 10.1145/3490234.*

*[4] Christopher J. Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King. Key challenges for delivering clinical impact with artificial intelligence. BMC Medicine, 17(1): 195, December 2019. ISSN 1741-7015. doi: 10.1186/s12916-019-1426-2.*

  
  

**Proposed actions:**

- Added a wiki that provides a low barrier of entry to using and extending YAIB.
    
- Added sentence to the design philosophy that indicates that users of YAIB should openly provide their methodology and code and compare it with the original benchmark.
    
- Added standalone Python files in the folder docs/adding_model to demonstrate adding a model to YAIB. Adding just two short files allows us to use a new model in most cases. For other models, we have included instructions in the wiki (Adding-a-new-model.md).
    
- Added Appendix F.3 which serves as a guideline for users to use YAIB in their work to benchmark new models (or other parts of the pipeline). 
    
- Adjusted the description of the experiments to remove vague claims and use percentage-wise differences where appropriate.
    

We hope this rebuttal addresses the reviewer's points. We invite them to respond with additional points that could be used to improve our work further.

Reviewer comment (Experiments):
Although the authors incorporate a wide range of measurements from the ICU datasets, many therapeutic interventions and comorbidities are missing, which can be crucial for predicting clinical outcomes of interest. Moreover, the time-varying features are mostly continuous while there exist many binary and categorical features in these ICU datasets.

Author response:
**Additional features** 

We chose the 52 most common clinical features shared by all datasets. They were readily available in all benchmarked datasets, demonstrating YAIB's adaptability. Our work focuses on the interoperability of datasets and the opportunity for experiments with a.o. transfer learning and domain adaption. We strongly believe there is the most value in providing a modular setup where the user can add or remove features to suit their needs better and, most importantly, do so reproducibly. 

  

Nevertheless, several medications for eICU and MIMIC-IV are readily available; the ricu package ([https://github.com/eth-mds/ricu/blob/main/inst/extdata/config/concept-dict.json](https://github.com/eth-mds/ricu/blob/main/inst/extdata/config/concept-dict.json)) maintains a complete list of the currently available native concepts which are available. Complex concepts, dependent on several native concepts, such as SOFA scores, are additionally available. Each concept that is available in ricu can be readily used in YAIB. Some medications that are already implemented, such as antibiotics and vasopressors, are used in the definition of the complex Sepsis endpoint. Therefore, we decided to leave those out to have the same features for each task. This also includes categorical medications. Comorbidities are missing in HiRID and AUMCdb; we decided not to include these for a fair comparison across datasets. However, they can certainly be extracted from eICU and MIMIC; this is also part of our current efforts. 

We note, additionally, that it is straightforward to implement new concepts in our pipeline; Appendix E.2 describes the addition of Potassium Chloride to the ICU harmonization package ricu. A similar process can be followed for adding new medications, which immediately improves the usability of YAIB. Moreover, we are actively working on integrating more features, including therapeutic interventions, comorbidities, and medications. We want to note that many features are not available in each dataset; this does not mean they can not be valuable in clinical prediction tasks. 

  

Finally, we would like to point out that YAIB's end-to-end pipeline is designed as a solid starting point for 1) clinicians looking for external validation to employ ML in practice, 2) dataset creators looking for a solid platform to facilitate widespread use, and 3) the ML community to contribute novel prediction models. They can use a mature and externally developed framework, which adds to the credibility of any experiment results. Adding new feature concepts for their datasets can also increase the adoption of their datasets. They are likely domain experts for their respective datasets, meaning fewer errors are made in this process. This process will improve the usability of YAIB as an end-to-end benchmarking tool and improve the confidence of health experts in clinical ML. We have added our argumentation to Appendix F.2 as well.

Reviewer comment (Novelty):
There exists a harmonized ICU dataset called BlendedICU [A] that incorporates AmsterdamUMCdb, eICU, HiRID, and MIMIC-IV. While the reviewer acknowledges that [A] has not been available online when the proposed work was submitted, what is the distinction and contribution of the proposed work from [A]?

Author response:
**BlendedICU**

Thank you for bringing our attention to this work. BlendedICU seems like a good effort as far as a static data combination pipeline is concerned. However, the authors of BlendedICU seem to have completely bypassed the ricu package that does harmonization for our work. This is a major problem when it comes to extensibility. Whereas we are close to adding more datasets to our benchmark, they have a static pipeline that does not seem to provide extensibility. They do not provide cohort generation functionality to define adaptable tasks. Moreover, there is no benchmarking framework which is a crucial part of our end-to-end pipeline to ensure comparable results. We would gladly adapt YAIB to be able to work with this work as well and will attempt to contact the authors to integrate their work into our pipeline to increase the usability of both works; this is why OMOP compatibility is on our list of future work. The work has been added to the related work section in the manuscript to show the distinction between it and YAIB.

Reviewer comment (Experiments):
Regarding Weakness #1: As far as the reviewer is aware, variations in units and features can occur both within and across datasets due to differences in observation circumstances (e.g., using different medical devices). How did the authors handle such issues? Additionally, what is the reason for the extracted datasets having missing therapeutic interventions and comorbidities (while some of them are mentioned in Appendix C), which are often extremely important for predicting clinical outcomes?

Author response:
**Unit changes**

Unit changes are handled by mapping to a common unit within ricu. This makes our harmonization work in a reproducible manner and allows other datasets to be added and harmonized to the common format. Moreover, different machines can indeed lead to different kinds of erroneous data. The datasets were therefore extensively checked for unrealistic values. Further information is available in Appendix B and C as well as the ricu documentation ([https://eth-mds.github.io/ricu/](https://eth-mds.github.io/ricu/)). We hope to have answered the question about comorbidities in the above section of this response (Additional features).

Reviewer comment (Writing):
The definition of several terms used in the paper is not clear, including "harmonization", "clinical concepts". Though it can be inferred from later text what they may refer to, I think it would still benefit the readers if you can clearly define them the first time you use them in the paper.

Author response:
**Unclear concepts and terms**

We thank the reviewer for the suggestions to clarify certain terms in our manuscript. The manuscript has been updated to reflect these changes. If any other terms are unclear, please feel free to address them in a response.

Reviewer comment (Experiments):
Based on the experiments, it seems that the time series features are extracted from numeric data rather than waveform data (please correct me if I am wrong). Currently, most ICU benchmarks used numeric data for prediction tasks and only very few used waveform data, but waveform data are very informative and may greatly help improve the predictive performance. Thus I think the contribution would be more solid if waveform data can be utilized for a new ICU benchmark.

Author response:
**Waveform data**

The reviewer has brought up that our work could be improved if we included support for waveform data. We understand waveform data to be time series data with a certain sampling frequency (e.g., 10 Hz). Waveform data are already included in our work as time-series features. These have been discretized to the same period for each dataset. As an example of highly frequent measurements, we include discretized heart rate (hr), respiratory rate (resp), oxygen saturation (o2sat), mean arterial pressure (map), systolic blood pressure (sbp), and diastolic blood pressure (dbp). Table 13 in the appendix shows that, e.g., MIMIC has a lower resolution for some features. It is relatively straightforward to create a preprocessing pipeline that can include waveform data, and it can be considered as future work once more authors publish waveform data. Moreover, we refer to the general response (Choice of Features) and the newly added Appendix F.2 for considerations on the chosen features.

Reviewer comment (Experiments):
Basic statistics of the datasets, e.g. number of patients, ICU stays, and the prevalence of classes or the range of regression targets, are missing in the main paper (found them in appendix). Inclusion of such information will help readers get a general idea of the datasets and understand some of the experiment results, e.g. the discrepancy in the performances shown in Figure 2 may be due to the difference in prevalence in mortality across the datasets. Also, for the pooled dataset, is there any possibility that one dataset is dominating the training set for any task?

Author response:
**Basic statistics for the datasets**

The reviewer argues that the dataset statistics should be in the main text. We acknowledge the importance of providing basic statistics for each dataset. We have adjusted the description of the mentioned experiment by providing clarification and reference to the mentioned table. However, due to the page limit, we only see a way of providing this information in the main text if we cut important manuscript parts. 

The reviewer points out the possibility of one dataset dominating another in pooled training for external validation. We have taken measures to prevent this. For the external validation experiments, we take a stratified sample of the same amount of patients for each dataset (10,000 for the mortality task). A short elaboration on this measure has been added to the experiments section. Further details can be found in Appendix G.5.

Reviewer comment (Writing):
In Section 4, you investigated the effects of small variations in task definitions. For the three variations you tried, the first and second are variations to the cohort and the feature selection but not to the clinical task definition, only the third is varying the definition of sepsis - which I understand is the only one directly relevant to clinical task definition. In this case, you may need to change how you describe the variations or redefine what you mean by "task" to avoid the confusion.

Author response:
1. We have changed the subsection header to: "Using YAIB as an experimental framework" to avoid the confusion of using "task" when referring to a prediction task. We hope this clears up any confusion.

Reviewer comment (Reproducibility):
In Section 4 Preprocessing, for the aggregated features, what are the time windows for aggregation and the frequencies that you update those features? Is there any specific consideration in the choice for them?

Author response:
2. If we understand the reviewer's question correctly, they seem to refer to the aggregation of time series for non-deep learning methods. We have chosen to do this once per stay as we seem to get reasonable results for this aggregation. However, one can change this to multiple times per stay by customizing the preprocessing pipeline (appendix D3). For the general preprocessing to attain the "harmonized cohorts," we refer to Appendix B. If the answer refers to the aggregation per hour, we provide details in Appendix B as well; the main reason is that MIMIC provides measurements once per hour. We hope this answers the question, but invite the reviewer to ask follow-up questions.

Reviewer comment (Evaluation):
In Table 4, it would be helpful to also include the range or the units of KF/LoS, or additionally show the MAEs as % relative to the valid ranges of KF/LoS (absolute MAEs do not provide much information).

Author response:
3. We acknowledge that absolute MAE might not mean without context, although one can still compare the performance of each method relative to each other. It is still informative to have the absolute MAE, as it is a universally recognized metric. Table 14 contains the average and IQR for each task. To increase the interpretability of the results and accommodate the reviewers' feedback, we have included the units of both Kidney function and Length of Stay in Table 4 and provide a footnote to refer to Table 14 (where one can find the average and IQR for each dataset).

Reviewer comment (Writing):
A minor typo, a space is missing between "repository" and "to" near the bottom on Page 4.

Author response:
4. Thank you for pointing out the typo. It has been corrected in the updated manuscript.

Reviewer comment (Experiments):
I acknolwedge the motivation of this paper and appreciate considerable effort invested in establishing such a benchmark dataset. However, while the paper's central claim is the adaptability of the YAIB benchmark to other datasets, tasks, and models, this assertion necessitates a comprehensive experience with the benchmark, which is challenging to evaluate during the brief review period.

Author response:
We sincerely appreciate the time and effort invested by the reviewer in evaluating our manuscript. We would like to address the points mentioned by reviewer VQQN. 

  

**Adaptability of YAIB**

The reviewer argues it is hard to assess the adaptability to other datasets, tasks, and models. Firstly, we would like to stress that YAIB is not a benchmark dataset but rather a framework that standardizes the process of benchmarking new methods on ICU/EHR data. 

Furthermore, the YAIB pipeline has helped us to produce reproducible results quickly and provides the required extensibility for our purposes. We are in touch with some researchers who have used YAIB to date and have helped improve the framework. We additionally refer to the work “Closing Gaps: An Imputation Analysis of ICU Vital Signs” accepted to the 1st Workshop on Deep Generative Models for Health at NeurIPS 2023 in the supplemental material. The authors of this work used YAIB as a bedrock for implementing imputation methods and are in the process of extending this to more methods and downstream tasks. For concrete examples of how to extend YAIB, for other authors, we refer to Appendix D and the wiki documentation we added to the supplemental materials. 

  

Therefore, we would argue that our work provides solid evidence of the adaptability the reviewer has pointed out. We further elaborate on the design of YAIB in our common response and Appendix F.1. We hope this addresses possible uncertainty regarding adaptability.

Reviewer comment (Experiments):
Datasets like MIMIC and eICU are highly complex and heterogeneous. The proposed YAIB benchmark only supports 52 clinical features (mainly vital signs and lab tests). A vast amount of other available features are ignored (like diagnosis, prescriptions, clinical notes, x-rays). This limited feature support might hinder future model advancements (which can be seen in Tables 3 & 4 where the performance between classic ML and DL models are very similar).

Author response:
**Heterogeneity of datasets and support of features**

The reviewer argues that the choice of features might limit usability. We chose the 52 most common clinical features shared by all datasets. They were readily available in all benchmarked datasets, demonstrating YAIB's adaptability. Our work focuses on the interoperability of datasets and the opportunity for experiments with a.o. transfer learning and domain adaption. We strongly believe there is the most value in providing a modular setup where the user can add or remove features to suit their needs better and, most importantly, do so reproducibly. 

Nevertheless, several medications for eICU and MIMIC-IV are readily available; the ricu package (https://github.com/eth-mds/ricu/blob/main/inst/extdata/config/concept-dict.json) maintains a complete list of the currently available native concepts which are available. Complex concepts, dependent on several native concepts, such as SOFA scores, are additionally available. Each concept that is available in ricu can be readily used in YAIB. Some medications that are already implemented, such as antibiotics and vasopressors, are used in the definition of the complex Sepsis endpoint. Therefore, we decided to leave those out to have the same features for each task.

We note, additionally, that it is straightforward to implement new concepts in our pipeline; Appendix E.2 describes the addition of Potassium Chloride to the ICU harmonization package ricu. A similar process can be followed for adding new medications, which immediately improves the usability of YAIB. Moreover, we are actively working on integrating more features, including comorbidities and medications. We want to note that many features are not available in each dataset; this does not mean they can not be valuable in clinical prediction tasks. The reviewer additionally argues we might support clinical notes and X-rays as well. Unfortunately, these are not available across all datasets, which was the focus of this work, and require very different (non-timeseries oriented) models. Additionally, we see this as a feature that is rarely used by existing ICU prediction models. 

Given the reasoning above, we disagree with the reviewer’s suggestion that future model advancements are hindered by the chosen features. However, we see this as a natural extension of YAIB when moving towards general EHR data. To accommodate the reviewer, we have added this point to the discussion. 

Additionally, we would like to point out that YAIB's end-to-end pipeline is designed as a solid starting point for 1) clinicians looking for external validation to employ ML in practice, 2) dataset creators looking for a solid platform to facilitate widespread use, and 3) the ML community to contribute novel prediction models. They can use a mature and externally developed framework, which adds to the credibility of any experiment results. Adding new feature concepts for their datasets can also increase the adoption of their datasets. They are likely domain experts for their respective datasets, meaning fewer errors are made in this process. This process will improve the usability of YAIB as an end-to-end benchmarking tool and improve the confidence of health experts in clinical ML.

**Similar performance ML and DL**

Furthermore, we disagree with the reviewer that the similar performance of our experiments can solely be explained by little features. Our preprocessing pipeline produces processed datasets suitable to both ML and DL; the effect is that both types of models can perform similarly. The use of “regular” ML is still widespread in the medical field as it does not require large processing power and provides explainability. Additionally, we refer to the HiRID benchmark [1], which has similar differences between ML and DL while including most of the features specific to HiRID. They provide feature generation that allows the LGBM model even to outperform deep learning models. 

We address the choice of features in the above response, in the general response (Choice of Features), and in Appendix F.2.

*[1] Hugo Yèche, Rita Kuznetsova, Marc Zimmermann, Matthias Hüser, Xinrui Lyu, Martin Faltys, and Gunnar Rätsch. HiRID-ICU-Benchmark – A Comprehensive Machine Learning Benchmark on High-resolution ICU Data. arXiv:2111.08536 [cs], January 2022*

**Proposed actions:** 

- Added a wiki to the supplementary materials to show users how to adapt datasets, tasks, preprocessing, and models
    
- Added an anonymized version of the work “Closing Gaps: An Imputation Analysis of ICU Vital Signs” accepted to the 1st Workshop on Deep Generative Models for Health at NeurIPS 2023 to the supplemental material. The work uses YAIB as an extendable base to show the performance of several imputation methods on ICU vital signs.  
    
- Added an explanation in the experiments section of the manuscript to indicate that one is not limited to the used features but 1) can use more features for some datasets and 2) use our extensive guide in Appendix E or our wiki.
    
- Added Appendix F.1 and F.2 to provide information on extensibility and reproducibility and the choice of features. 
    
- Added clinical notes and medical images as future extensions in the discussion.
    

We hope this rebuttal addresses the reviewer's points. Additionally, we invite them to respond with additional points that could be used to improve our work.