Paper: GlucoBench: Curated List of Continuous Glucose Monitoring Datasets with Prediction Benchmarks (ICLR-2024)

Reviewer comment (Novelty):
I am concerned about the level of contribution of this work. Many of the conclusions of the paper are not novel. For example, people should already know “Healthy subjects demonstrate markedly smaller errors” and “using simpler shallow models when data is limited”.

Author response:
Medical facts may not be immediate for ML practitioners and vice versa for medical researchers. Therefore, part of the novelty of our work is in empirically establishing intuitive conclusions (e.g. errors are smaller for healthy subjects and shallow models are better for small data) and debunking popular misconceptions in the specific context of CGM. For example, it is common to assume that for large datasets deep learning must be a default choice. However, we show in Section 5.1 on the example of Colas (2nd largest dataset) that when there is less 7 days of data per patient, the deep learning models are unable to close the performance gap with the shallow -- the fact that has never been established in the literature before.

Reviewer comment (Evaluation):
What is the motivation for some analysis you did in the paper? For example, what is the motivation for exploring the impact of time of day on accuracy? Why should researchers care about the analysis you did in Section 5, especially under the context of CGM?

Author response:
Our motivation for writing Section 5 is to help guide intuition for future model development and benchmarking based on the empirical facts. In each subsection of the analysis, our goal was to highlight a major shortcoming of the current models and their evaluation methods and contribute a discussion on the future steps. For example, in Section 5.1, we highlight that a patient's stochastic behavior poses a major challenge to all forecasting models. At the same time, we observe that each patient tends to adhere to a schedule with repetitive pronounced states, such as active / rest cycles, which affect the models' accuracy. Therefore, when designing new experiments, we need to control data splitting to account for different times of the day.

Reviewer comment (Novelty):
How is this work different from other review papers mentioned in related works?

Author response:
Previous works can be divided into review [2, 3, 4, 5] and benchmarking papers [6, 7]. Review papers do not explicitly compare the models and do not have an associated code base, but rather offer a survey of the literature with a summary of existing methods. Previous benchmarking papers [6, 7] have focused on a single dataset, excluded hybrid models from the analysis, and compared the model solely based on their accuracy.

Reviewer comment (Experiments):
Are there any potential biases in the selection of the public datasets, and how might they impact the generalizability of the results? Authors should discuss about the quality of the selected CGM datasets assessed, and what criteria were used to determine data quality.

Author response:
> Are there any potential biases in the selection of the public datasets, and how might they impact the generalizability of the results?

We recognize that using publicly available datasets may introduce biases into the analysis arising from (1) selection bias, e.g., vulnerable sub-populations incapable of accepting public information sharing agreement are excluded (2) missing information due to privacy concerns, e.g., demographic information or medical history may be partially concealed (3) conflict of interest, e.g., researchers may be funded by external sources to collect and publish data that supports certain claims etc. Previous works such as [1, 2] that have focused on developing benchmarks have only included analyzed a single dataset. Therefore, to help offset the possible biases, we have included 5 datasets collected by medical professionals from different countries in our benchmark: Broll (USA), Colas (Spain), Dubosson (France), Hall (USA), and Weinstock (USA). At the same time, we note that our efforts in developing a more representative benchmark are ongoing. We welcome contributions and will make our code available online at github.com/XXX.

> Authors should discuss about the quality of the selected CGM datasets assessed, and what criteria were used to determine data quality.

We thank the reviewer from bringing this point up! Data quality is crucial in devising benchmarks, and we have paid meticulous attention to ensuring it. On CGM curve level, we ensure that the measurements remain in range (between 20 mg/dL and 400 mg/dL), do not exhibit drastic fluctuations > 40 mg/dL in 5 minutes), and are not constant. On the patient level, we check that each patient has enough data (> 16 hours of continuous recording). Finally, on the dataset level, we ensure that there are enough patients to allow for in- / out-of-distribution splits (> 5 patients). We emphasize these details in Section 3.1 and highlight the changes in red.

Reviewer comment (Evaluation):
The paper discusses the impact of covariates on model performance. Can you provide insights into which specific covariates had the most influence on the predictions and whether their quality was uniform across all datasets? And, how does the decision to omit certain models due to their lack of support for covariates affect the overall model comparison?

Author response:
> The paper discusses the impact of covariates on model performance. Can you provide insights into which specific covariates had the most influence on the predictions and whether their quality was uniform across all datasets?
    
We provide summary of the covariates for each dataset in Table 8 in the Supplement and include it here for reference. Besides demographic information, each dataset is unique in the set of provided covariates. We ensure the quality by checking each covariate distribution, where we check for range and any possible data corruption.

Out of 5 models that support covariates, only XGBoost consistently (on 3 out of 5 datasets) benefits from their inclusion. For XGBoost, we report the covariate importance in Table 11 in the Supplement. In general, we observe that:

- Among features available for all datasets, time features such as hour and day of the week, consistently appear in the top 3 important features across all datasets, indicating that patients possibly tend to adhere to daily routines that affect their glucose patterns. 
- Dynamic physical activity features such as heart rate and blood pressure are only available for Dubosson and appear to be all selected as highly important.
- Demographic and medical record information is not available for Broll and Dubosson. For the rest of the datatsets, we observe medication (e.g. Vitamin D, Lisinopril for Weinstock), disease indicators (e.g. Diabetes T2 for Colas, Osteoporosis for Weinstock), health summary metrics (Body Mass Index for Colas), as well as indices derived from CGM measurements (e.g. J-index) being selected as highly important.

> How does the decision to omit certain models due to their lack of support for covariates affect the overall model comparison?

We compare models with and without covariates separately and do not exclude the latter from the overall analysis.

Reviewer comment (Experiments):
The paper mentions not considering additional model-specific tuning for benchmark models. Could such optimizations significantly enhance the models' performance, and why were they excluded from the evaluation?

Author response:
> The paper mentions not considering additional model-specific tuning for benchmark models. Could such optimizations significantly enhance the models' performance, and why were they excluded from the evaluation?

In tuning the models, we have followed the original authors' suggestions on the best set of tuning parameters. We summarize the tuning parameters for each model in Table 16 in the Supplement. However, we do not go further than the author suggested best practices on tuning the models. As an example, for the Transformer, we use the vanilla implementation with encoder-decoder structure as proposed in [3], and do not investigate further possible developments that enhance input length capabilities or expressiveness of the model as proposed in [4, 5, 6]. Such developments indeed may boost the base model performance but should be included as a separate entry. Indeed, we are actively adding new base models and hope that publication will add more light to our effort and bring attention to this crucial task.

Reviewer comment (Experiments):
How diverse were the patient populations in the selected datasets in terms of demographics, disease severity, or other relevant factors, and how might this diversity affect the results and generalization?

Author response:
> How diverse were the patient populations in the selected datasets in terms of demographics, disease severity, or other relevant factors, and how might this diversity affect the results and generalization?

One of the main motivations of our paper was to establish benchmarks on a diverse set of datasets covering multiple subpopulations. We summarize each dataset demographic composition in Table 2 in the main paper and provide it here for reference. With an exception of Hall, all patients were recruited during routine clinic visits: diabetes clinics for Dubosson (France), Weinstock (USA), and Broll (USA), and hypertension and vascular risk clinic for Colas (Spain). For Hall (USA), human subjects were recruited via newspapers and community outreach. Based on the table and recruiting patterns, we observe good coverage across various age, gender, and disease status groups. Therefore, we believe our study is reasonably randomized and representative. However, we note that our efforts are ongoing and we encourage and solicit the addition of new datasets.

Reviewer comment (Experiments):
Cross validation results will be preferred than simple train/test/validation splitting in benchmark model performance comparison.

Author response:
We agree that a simple train/validation/test split is not sufficient for accurately evaluating the model. In fact, we note in Section 5.1, how different splits of the data based on the time of the day may drastically affect the model performance. However, we found that full cross-validation, e.g. 10-fold cross-validation, is too computationally demanding for some methods. Thus, as a compromise, we use 2 random splits (rather than just one split) to evaluate the performance, which we describe in Section 4.3. We report the standard deviation of the errors for both tasks across random data splits in the Tables 12-15 in the Supplement.

Reviewer comment (Experiments):
Automated machine learning (AutoML) could also help in benchmarking performance across all datasets with more ML models and pre-processing pipelines.

Author response:
Thank you for the suggestion! Currently, we are using Optuna [1] to automate model hyper-parameter optimization. As a future step, we recognize that incorporating a fully automated pre-processing, training, and evaluation pipeline would significantly simplify the use of our benchmark and increase its adoption.

Reviewer comment (Reproducibility):
How could other researchers contribute to the repository?

Author response:
This is an excellent question! There are 3 ways to contribute: (1) adding a new dataset, (2) adding a new model, (3) helping build out the code base.  The code will be available online upon the publication of the paper at github.com/XXX. We welcome any contribution and are soliciting help from researchers working on CGM. In the revised paper, we add a link to the online repository to the Abstract and Section 6, where we discuss future directions.