The hopes and hazards of using personal health technologies in the diagnosis and prognosis of infections
Authors: Jennifer M Radin, Giorgio Quer, Marwa Jalili, Dina Hamideh, Steven R Steinhubl
Journal: The Lancet Digital Health (2021)

Work from numerous groups has shown the potential of using data from wearable devices to characterise each individual's unique baseline, identify deviations from that baseline suggestive of a viral infection, and to aggregate that data to better inform population surveillance trends. With a growing global population of connected wearable users, this could potentially help improve the earlier diagnosis and management of infectious individuals and improve timeliness and precision of tracking infectious disease outbreaks. However, despite these possibilities, there are important considerations when interpreting wearable data, including generalisability of user populations, sensor accuracy and comparability, and overfitting of models. Additionally, before deploying such tools as a global health solution, the proactive integration of community engagement through a user-centred design is, at a minimum, required to begin mitigating the risk of digital health illiteracy, structural inequality, and social marginality. Regional frameworks establishing transparent standards for participant data protection as data privacy, data use, and data rights are required to truly protect participant's rights, empower participants, and minimise risk of harm. As these sensors continue to evolve, standardised data and reporting, as well as collaboration and data sharing across study and technology groups will also be necessary. Personal health technologies—technologies (consumer or medical grade) used by an individual that generate data from that individual—offer an unprecedented opportunity to completely alter how we currently detect and manage infectious diseases at an individual as well as population level. Currently, management decisions in the setting of possible infection are made based on symptoms and objective physiological measurements, whether the person primarily making those decisions is the patient or a health-care provider. Because symptoms are often non-specific, the recognition of abnormalities in physiological parameters have an important role in diagnosing and guiding therapeutic interventions. However, abnormality has historically been defined by what is normal for a healthy population, rather than what is normal for an individual. For example, an elevated temperature, which has been recognised as a hallmark of infection since the beginning of recorded history, is commonly (but not consistently) classified as a temperature of 38°C or higher.1Atkins E Fever: its history, cause, and function.Yale J Biol Med. 1982; 55: 283-289PubMed Google Scholar Similarly, a respiratory rate of more than 20 breaths per min or a heart rate of more than 100 beats per min in an adult would also be considered abnormal and might indicate a person who requires expedited, or a higher level of care. However, what is actually a person's "normal" varies substantially between individuals. For oral temperature, that range falls between 35·7°C and 37·4°C,2Geneva II Cuzzo B Fazili T Javaid W Normal body temperature: a systematic review.Open Forum Infect Dis. 2019; 6ofz032Crossref PubMed Scopus (49) Google Scholar respiratory rate between 12 and 20 breaths per min,3Chourpiliadis C Bhardwaj A Physiology. Respiratory Rate.https://www.ncbi.nlm.nih.gov/books/NBK537306/Date: 2019Date accessed: May 6, 2021Google Scholar and resting pulse rate between 40 and 109 beats per min.4Quer G Gouda P Galarnyk M Topol EJ Steinhubl SR Inter- and intraindividual variability in daily resting heart rate and its associations with age, sex, sleep, BMI, and time of year: retrospective, longitudinal cohort study of 92,457 adults.PLoS One. 2020; 15e0227709Crossref PubMed Scopus (22) Google Scholar Therefore, these broad population ranges are imprecise when applied to a specific individual, especially for identifying an early change in their physiological status. Individuals experience daily, weekly, and seasonal fluctuations unique to them in a range of physiological parameters and activities.4Quer G Gouda P Galarnyk M Topol EJ Steinhubl SR Inter- and intraindividual variability in daily resting heart rate and its associations with age, sex, sleep, BMI, and time of year: retrospective, longitudinal cohort study of 92,457 adults.PLoS One. 2020; 15e0227709Crossref PubMed Scopus (22) Google Scholar, 5Jaiswal SJ Quer G Galarnyk M Steinhubl SR Topol EJ Owens RL Association of sleep duration and variability with body mass index: sleep measurements in a large US population of wearable sensor users.JAMA Intern Med. 2020; 1801694Crossref PubMed Scopus (5) Google Scholar Only by knowing what is normal for an individual when they are well is it possible to identify the earliest possible deviations from that normal. This is what personal health technologies make possible, in the real world outside of a health-care setting, and in a nearly passive manner. Because of the emergence of COVID-19, numerous studies examining personal sensor data have found digital signals supporting the potential benefit of using these technologies to identify and track viral illness. Continuously evolving personal health technologies can be harnessed to collect unique baseline data on individuals and populations, which allows for earlier and more precise detection of viral illness; but for these tools to be successful, biases related to health inequity, loss to follow-up, an absence of data harmonisation, and others need to be addressed. It is relatively recent that personal health technologies have become available that enable investigators to even consider addressing the potential value of identifying individual changes in physiological parameters in large populations. Wearable sensors available to individuals can now track not only activity, but also body position, heart rate and rhythm, skin temperature, oxygen saturation, an electrocardiogram, and electrodermal activity and sound (figure 1). From these metrics respiratory rate, heart rate variability, heart arrhythmias, sleep, and sleep stages are derived. Other medical-grade wearables, not yet designed for a large consumer market, can also continuously track blood pressure as well as derive systemic vascular resistance and cardiac output.6Nachman D Gepner Y Goldstein N et al.Comparing blood pressure measurements between a photoplethysmography-based and a standard cuff-based manometry device.Sci Rep. 2020; 1016116Crossref PubMed Scopus (13) Google Scholar Wearable sensor technologies are available in a wide range of form factors beyond the most commonly used wrist-worn sensors. There are options for rings, arm bands, earbuds, adhesive patches, and clothing that all are capable of tracking multiple physiological parameters. Each offers certain advantages and disadvantages, but the most effective will be the one that provides the most reliable information that the intended user will be willing and able to use as frequently and for as long as needed. There are little data available on long-term use of any of these wearables, with one survey7Macridis S Johnston N Johnson S Vallance JK Consumer physical activity tracking device ownership and use among a population-based sample of adults.PLoS One. 2018; 13e0189298Crossref PubMed Scopus (25) Google Scholar in Canada finding that of people who purchased their own activity tracker, just 55% were still using it, but those that were still wearing it wore it an average of 23 days in the previous month. In another study8Galarnyk M Quer G McLaughlin K Ariniello L Steinhubl SR Usability of a wrist-worn smartwatch in a direct-to-participant randomized pragmatic clinical trial.Digit Biomark. 2019; 3: 176-184Crossref PubMed Scopus (2) Google Scholar that gave participants a wrist wearable, approximately 25% wore the sensor for the majority of the 4-month requested monitoring period.8Galarnyk M Quer G McLaughlin K Ariniello L Steinhubl SR Usability of a wrist-worn smartwatch in a direct-to-participant randomized pragmatic clinical trial.Digit Biomark. 2019; 3: 176-184Crossref PubMed Scopus (2) Google Scholar A study of nearly 4·5 million insurance plan members who were offered financial incentives for achieving activity goals found that only 1·2% activated a device, but of those that did, 80% were still using the device at 6 months.9Patel MS Foschini L Kurtzman GW et al.Using wearable devices and smartphones to track physical activity: initial activation, sustained use, and step counts across sociodemographic characteristics in a national sample.Ann Intern Med. 2017; 167: 755-757Crossref PubMed Scopus (39) Google Scholar A meta-analysis10Pratap A Neto EC Snyder P et al.Indicators of retention in remote digital health studies: a cross-study evaluation of 100,000 participants.NPJ Digit Med. 2020; 3: 21Crossref PubMed Scopus (41) Google Scholar found that median participant retention across eight studies was only 5·5 days, and that most studies did not include a population representative of the ethnicities and diversity of the USA. For many, a non-wearable, passive multiparametric sensor can provide the best option for longitudinal data. For example, under-mattress pads that can monitor heart rate, respiratory rate, and various sleep parameters have been found in preliminary work to identify early physiological decompensation.11Bennett MK Shao M Gorodeski EZ Home monitoring of heart failure patients at risk for hospital readmission using a novel under-the-mattress piezoelectric sensor: a preliminary single centre experience.J Telemed Telecare. 2017; 23: 60-67Crossref PubMed Scopus (7) Google Scholar A wide range of contactless sensors for in-home use that use computer-vision, infrared thermography, radar, and audio can track individual vital signs, as well as cough frequency and quality, in some cases, even when in different rooms.12Adib F Mao H Kabelac Z Katabi D Miller RC Smart homes that monitor breathing and heart rate.in: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. Association for Computing Machinery, Seoul, Republic of Korea2015: 837-846Crossref Scopus (366) Google Scholar, 13Negishi T Abe S Matsui T et al.Contactless vital signs measurement system using RGB-thermal image sensors and its clinical screening test on patients with seasonal influenza.Sensors. 2020; 20E2171Crossref PubMed Scopus (20) Google Scholar, 14Antink CH Lyra S Paul M Yu X Leonhardt S A broader look: camera-based vital sign estimation across the spectrum.Yearb Med Inform. 2019; 28: 102-114Crossref PubMed Scopus (22) Google Scholar, 15Hall JI Lozano M Estrada-Petrocelli L Birring S Turner R The present and future of cough counting tools.J Thorac Dis. 2020; 12: 5207-5223Crossref PubMed Scopus (5) Google Scholar As remarkable as progress has been over the last several years in the availability of consumer technologies that monitor and report physiological variables, it is important to recognise that the accuracy of the information provided is variable and should not be considered clinically dependable unless appropriate validation evidence exists, and ideally regulatory agency approval.16Manta C Jain SS Coravos A Mendelsohn D Izmailova ES An evaluation of biometric monitoring technologies for vital signs in the era of COVID-19.Clin Transl Sci. 2020; 13: 1034-1044Crossref PubMed Scopus (13) Google Scholar For example, there have been recent concerns raised of the accuracy of oxygen saturation determined not only from the wrist with consumer sensors,17Bent B Goldstein BA Kibbe WA Dunn JP Investigating sources of inaccuracy in wearable optical heart rate sensors.NPJ Digit Med. 2020; 3: 18Crossref PubMed Scopus (74) Google Scholar but also from hospital-based finger-tip devices due to skin pigmentation differences.18Perry TS Should you trust Apple's new blood oxygen sensor? View from the valley. IEEE Spectrum.https://spectrum.ieee.org/view-from-the-valley/biomedical/devices/should-you-trust-apples-new-blood-oxygen-sensorDate accessed: May 6, 2021Google Scholar, 19Sjoding MW Dickson RP Iwashyna TJ Gay SE Valley TS Racial bias in pulse oximetry measurement.N Engl J Med. 2020; 383: 2477-2478Crossref PubMed Scopus (69) Google Scholar To aggregate findings from multiple sensors it will also be important to understand how sensor-specific and usually proprietary algorithms for calculating metrics such as daily resting heart rate and respiration rate can differ across devices. There is typically a delay from the time someone gets sick to when they develop symptoms, seek care, get tested, and finally receive a test result for COVID-19 and other viral infections. It then takes an additional 1–3 weeks before test results are collected and aggregated into a central surveillance system, which often relies on outdated reporting methods such as fax machines.20Kliff S Sanger-Katz M Bottleneck for U.S. Coronavirus response: the fax machine. The New York Times, July 13, 2020https://www.nytimes.com/2020/07/13/upshot/coronavirus-response-fax-machines.htmlDate accessed: May 6, 2021Google Scholar When wearable data is aggregated for a population, it is possible to so called nowcast or track viral activity in real time. Previous studies have shown that identifying resting heart rate and sleep data outside of an individual's normal levels, can be used to improve real-time predictions for influenza-like illness at the state level.21Quer G Radin JM Gadaleta M et al.Wearable sensor data and self-reported symptoms for COVID-19 detection.Nat Med. 2021; 27: 73-77Crossref PubMed Scopus (61) Google Scholar Similarly, models to predict COVID-19 anomalies in China and south-central Europe using wearable data from 1·3 million users who wore Huami smartwatches (Hefei, China) also showed promise (table 1).32Zhu G Li J Meng Z et al.Learning from large-scale wearable device data for predicting epidemics trend of COVID-19.Discrete Dyn Nat Soc. 2020; 20206152041Crossref Scopus (28) Google Scholar The Robert Koch institute, Berlin, Germany, has launched a similar fever trend tracker based on resting heart rate and activity data from more than 500 000 participants.31Corona DatenspendeNews.https://corona-datenspende.de/science/en/Date: Dec 27, 2020Date accessed: December 27, 2020Google Scholar Kinsa smart thermometers (San Francisco, CA, USA) have also shown utility in predicting influenza-like illness activity29Miller AC Singh I Koehler E Polgreen PM A smartphone-driven thermometer application for real-time population- and individual-level influenza surveillance.Clin Infect Dis. 2018; 67: 388-397Crossref PubMed Scopus (19) Google Scholar, 33Ackley SF Pilewski S Petrovic VS Worden L Murray E Porco TC Assessing the utility of a smart thermometer and mobile application as a surveillance tool for influenza and influenza-like illness.Health Informatics J. 2020; 26: 2148-2158Crossref PubMed Scopus (7) Google Scholar, 34Miller AC Peterson RA Singh I Pilewski S Polgreen PM Improving state-level influenza surveillance by incorporating real-time smartphone-connected thermometer readings across different geographic domains.Open Forum Infect Dis. 2019; 6ofz455Crossref Google Scholar and potentially COVID-19.30Chamberlain SD Singh I Ariza CA Daitch AL Philips PB Dalziel BD Real-time detection of COVID-19 epicenters within the United States using a network of smart thermometers.medRxiv. 2020; (published online April 10.) (preprint).https://doi.org/10.1101/2020.04.06.20039909Google Scholar Novel data streams from sensors can offer key insight into trends, timing of outbreaks, and identifying specific geographical hotspots of infection. They might prove especially useful when integrated with both traditional clinical and laboratory surveillance and other novel surveillance data, including those from wastewater, internet search terms, social media, and mobility data (figure 2).Table 1Observational studies of wearables in the prediction of viral illnessesParameters analysedWearable sensors includedStudy populationKey findingTotal participants, NCOVID-19 positive participants, nIndividual-level studiesTemPredict22Smarr BL Aschbacher K Fisher SM et al.Feasibility of continuous fever monitoring using wearable devices.Sci Rep. 2020; 1021640Crossref PubMed Scopus (13) Google ScholarSkin temperature, heart rate, respiratory rate, heart rate variabilityOura ring sensor5050Peripheral temperature elevations can be captured by wearable sensors and correlate with self-reported feverStanford consumer smartwatches23Mishra T Wang M Metwally AA et al.Pre-symptomatic detection of COVID-19 from smartwatch data.Nat Biomed Eng. 2020; 4: 1208-1220Crossref PubMed Scopus (59) Google ScholarHeart rate, sleep, physical activityFitbit, Apple Watch, Garmin, and others52623226 (81%) COVID-19 individuals had changes in their heart rate, steps, or sleep; retrospectively, 20 (63%) of COVID-19 cases could be detected pre-symptoms onset using extreme elevations in resting heart rateFitbit Study24Natarajan A Su HW Heneghan C Assessment of physiological signs associated with COVID-19 measured using wearable devices.NPJ Digit Med. 2020; 3: 156Crossref PubMed Scopus (36) Google ScholarResting heart rate, activity, respiration rate, heart rate variability, sleepFitbit187 5732745 (PCR) and 1117 (serology)Physiological data could predict illness on a specific day with an AUC of 0·77DETECT21Quer G Radin JM Gadaleta M et al.Wearable sensor data and self-reported symptoms for COVID-19 detection.Nat Med. 2021; 27: 73-77Crossref PubMed Scopus (61) Google ScholarResting heart rate, sleep, physical activityData from Fitbits and any sensors connected with HealthKit or GoogleFit30 52954Wearable sensors data can significantly improve symptom only based models to distinguish COVID-19 positive versus negative symptomatic infections (AUC 0·80 [95% CI 0·73–0·86])WHOOP system25Miller DJ Capodilupo JV Lastella M et al.Analyzing changes in respiratory rate to predict the risk of COVID-19 infection.PLoS One. 2020; 15e0243693Crossref PubMed Scopus (18) Google ScholarRespiratory rate, resting heart rate, heart rate variabilityWHOOP; wrist-worn strap27181Model identified two (20%) of COVID-19 positive cases 2 days before symptom onset and eight (80%) positive cases by day 3 of symptomsEvidation26Shapiro A Marinsek N Clay I et al.Characterizing COVID-19 and influenza illnesses in the real world via person-generated health data.Patterns. 2021; 2100188Summary Full Text Full Text PDF PubMed Scopus (6) Google ScholarResting heart rate, physical activity, and sleepFitbit6926230Wearable sensor data showed similar magnitudes in daily changes of steps and heart rate measurements for both influenza and COVID-19 cohortsWarrior Watch Study27Hirten RP Danieletto M Tomalin L et al.Use of physiological data from a wearable device to identify SARS-CoV-2 infection and symptoms and predict COVID-19 diagnosis: observational study.J Med Internet Res. 2021; 23e26107Crossref PubMed Scopus (17) Google ScholarHeart rate variabilityApple Watch29761The collected heart rate varability metrics can identify the diagnosis of COVID-19 and related symptomsPopulation-level studiesScripps' Fitbit study28Radin JM Wineinger NE Topol EJ Steinhubl SR Harnessing wearable device data to improve state-level real-time surveillance of influenza-like illness in the USA: a population-based study.Lancet Digit Health. 2020; 2: e85-e93Summary Full Text Full Text PDF PubMed Scopus (84) Google ScholarResting heart rate and sleepFitbit47 249..Inclusion of Fitbit data significantly improved Centers for Disease Control and Prevention models of current influenza-like illnessKinsa29Miller AC Singh I Koehler E Polgreen PM A smartphone-driven thermometer application for real-time population- and individual-level influenza surveillance.Clin Infect Dis. 2018; 67: 388-397Crossref PubMed Scopus (19) Google Scholar, 30Chamberlain SD Singh I Ariza CA Daitch AL Philips PB Dalziel BD Real-time detection of COVID-19 epicenters within the United States using a network of smart thermometers.medRxiv. 2020; (published online April 10.) (preprint).https://doi.org/10.1101/2020.04.06.20039909Google ScholarTemperatureKinsa smart thermometers1321 counties..Fever anomalies are significantly correlated (r=0·54–0·55) with COVID-19 case counts at the county and state level, respectively, and with national influenza-like illness activity (r>0·95) in the USACorona Data Donation App31Corona DatenspendeNews.https://corona-datenspende.de/science/en/Date: Dec 27, 2020Date accessed: December 27, 2020Google ScholarResting heart rate, physical activityWearable fitness sensors535 298*As of Jan 2, 2020...Sensor data might predict fever anomalies in GermanyHuami users32Zhu G Li J Meng Z et al.Learning from large-scale wearable device data for predicting epidemics trend of COVID-19.Discrete Dyn Nat Soc. 2020; 20206152041Crossref Scopus (28) Google ScholarResting heart rate, sleepHuami wearables1·3 million..Physiological anomaly rate correlates with COVID-19 case counts in Chinese cities (average ρ=0·68)AUC=area under the curve. Some data not available because specific numbers of COVID-19 positive individuals were not reported. Studies needed to have a peer-reviewed, preprint article or post supporting data on their website.* As of Jan 2, 2020. Open table in a new tab AUC=area under the curve. Some data not available because specific numbers of COVID-19 positive individuals were not reported. Studies needed to have a peer-reviewed, preprint article or post supporting data on their website. Individual-level diagnosis using continuously collected data from wearables has many benefits, including improved COVID-19 screening, especially when routine diagnostic tests are not readily available at scale. During infections, continuously collected heart rate data has shown that an individual's heart rate tends to increase about 8–10 beats per min for every 1°C rise in fever,35Davies P Maconochie I The relationship between body temperature, heart rate and respiratory rate in children.Emerg Med J. 2009; 26: 641-643Crossref PubMed Scopus (59) Google Scholar, 36Karjalainen J Viitasalo M Fever and cardiac rhythm.Arch Intern Med. 1986; 146: 1169-1171Crossref PubMed Scopus (56) Google Scholar and data from animal models suggested that changes in diurnal heart rate can be detected 4 days before fever.37Milechin L Davis S Patel T et al.Detecting pathogen exposure during the non-symptomatic incubation period using physiological data.bioRxiv. 2017; (published online Nov 13.) (preprint).https://doi.org/10.1101/218818Google Scholar Wearables have also shown early promise to improve models to differentiate symptomatic individuals who have COVID-19 versus those who have other infections. App-based collection of self-reported symptom data has identified key symptoms predictive of COVID-19 compared with other infections, with loss of smell and taste as the main predictor.38Menni C Valdes AM Freidin MB et al.Real-time tracking of self-reported symptoms to predict potential COVID-19.Nat Med. 2020; 26: 1037-1040Crossref PubMed Scopus (489) Google Scholar The addition of wearable data into these models has shown potential to further improve the discriminative ability of these models, with an area under the curve increasing from 0·71 (95% CI 0·63–0·79; considering symptoms only) to 0·80 (0·73–0·86; using also wearable data; table 1).21Quer G Radin JM Gadaleta M et al.Wearable sensor data and self-reported symptoms for COVID-19 detection.Nat Med. 2021; 27: 73-77Crossref PubMed Scopus (61) Google Scholar Individual sensor data also has shown promise to identify pre-symptomatic infection,39Li X Dunn J Salins D et al.Digital health: tracking physiomes and activity using wearable biosensors reveals useful health-related information.PLoS Biol. 2017; 15e2001402Crossref PubMed Scopus (158) Google Scholar including for individuals who tested positive for COVID-19,23Mishra T Wang M Metwally AA et al.Pre-symptomatic detection of COVID-19 from smartwatch data.Nat Biomed Eng. 2020; 4: 1208-1220Crossref PubMed Scopus (59) Google Scholar which if confirmed would be especially valuable. Researchers have also looked at identifying the need for hospitalisation looking at symptoms alone,24Natarajan A Su HW Heneghan C Assessment of physiological signs associated with COVID-19 measured using wearable devices.NPJ Digit Med. 2020; 3: 156Crossref PubMed Scopus (36) Google Scholar how elevations in peripheral temperature correlate with self-reported fever,22Smarr BL Aschbacher K Fisher SM et al.Feasibility of continuous fever monitoring using wearable devices.Sci Rep. 2020; 1021640Crossref PubMed Scopus (13) Google Scholar and how symptoms and physiological changes are more severe for COVID-19 positive individuals than for influenza positive individuals.26Shapiro A Marinsek N Clay I et al.Characterizing COVID-19 and influenza illnesses in the real world via person-generated health data.Patterns. 2021; 2100188Summary Full Text Full Text PDF PubMed Scopus (6) Google Scholar As new metrics are added to sensors, substantially greater research is needed to better understand wearable changes for different infections, asymptomatic infections, non-infectious insults, and tracking long-term consequences, such as with post-acute sequelae of SARS-CoV-2 infection. Identifying early signs of decompensation can be especially useful for early initiation of antivirals, monoclonal antibodies, supportive care, and closer individual monitoring. Additionally, as vaccines are rolled out, sensor-based monitoring might prove useful for identifying immune response to vaccination, and potential side-effects and improve our ability to track infections and quantify vaccine effectiveness. The true value of the growing availability of personal, multiparametric, continuous sensor technologies is dependent on the development of meaningful detection or prediction models, which should follow existing standards before being implemented in the clinic40Vasey B Clifton DA Collins GS et al.DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence.Nat Med. 2021; 27: 186-187Crossref PubMed Scopus (17) Google Scholar to ensure their reproducibility.41Beam AL Manrai AK Ghassemi M Challenges to the reproducibility of machine learning models in health care.JAMA. 2020; 323: 305-306Crossref PubMed Scopus (59) Google Scholar, 42McDermott MBA Wang S Marinsek N Ranganath R Ghassemi M Foschini L Reproducibility in machine learning for health.https://arxiv.org/abs/1907.01463Date: 2019Date accessed: May 6, 2021Google Scholar Experience with early diagnostic and prognostic models in COVID-19 suspected or confirmed individuals has highlighted several challenges and limitations. Most importantly, they are at high risk of bias, due to (1) bias on the participant domain, when the participants enrolled might not be representative of the general population; (2) bias in the predictor domain, when the predictors are not available at the intended time and can be influenced by the measurement outcome; (3) bias in the outcome measurement, when it is measured subjectively; and (4) bias in the analysis domain (overfitting) due to small sample size and complex modelling strategies, which can be avoided only with a strict separation between training, validation, and test sets (table 2).43Wynants L Van Calster B Collins GS et al.Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal.BMJ. 2020; 369m1328Crossref PubMed Scopus (896) Google Scholar There are often issues in reporting important parameters such as the length of the follow-up or the concurrent prevalence of COVID-19 and other viral infections such as influenza in the considered population, although different models report performance with different statistical measures, making it more difficult to compare them for the same prediction task. Among the studies overviewed in the work of Wynants and colleagues,43Wynants L Van Calster B Collins GS et al.Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal.BMJ. 2020; 369m1328Crossref PubMed Scopus (896) Google Scholar all reported moderate to excellent prediction performance, but because of the high risk of bias due to both poor reporting and methodological issues, the results could be overly optimistic and not represent reality. This bias is a serious issue which was underscored in analyses after the publication of a mortality prediction model for COVID-19 patients that was based on several blood sample biomarkers and found a very high predictive accuracy of 90%.44Yan L Zhang H-T Goncalves J et al.An interpretable mortality prediction model for COVID-19 patients.Nat Mach Intell. 2020; 2: 283-288Crossref Scopus (0) Google Scholar Subsequent studies45Barish M Bolourani S Lau LF Shah S Zanos TP External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19.Nat Mach Intell. 2020; (published online Nov 12.) (preprint).https://doi.org/10.1038/s42256-020-00254-2Google Scholar, 46Dupuis C De Montmollin E Neuville M Mourvillier B Ruckly S Timsit JF Limited applicability of a COVID-19 specific mortality prediction rule to the intensive care setting.Nat Mach Intell. 2020; 3: 20-22Crossref Scopus (5) Google Scholar, 47Quanjel MJR van Holten TC Gunst-van der Vliet PC et al.Replication of a mortality prediction model in Dutch patients with COVID-19.Nat Mach Intell. 2020; 3: 23-24Crossref Scopus (6) Google Scholar, 48Reeve JLV Twomey PJ Consider laboratory aspects in developing patient prediction models.Nat Mach Intell. 2021; 3: 18Crossref Scopus (4) Google Scholar, 49Giacobbe DR Clinical interpretation of an interpretable prognostic model for patients with COVID-19.Nat Mach Intell. 2021; 3: 16Crossref Scopus (5) Google Scholar and perspect