TO THE EDITOR
The widespread use of machine learning has created the possibility of generating robust prediction models for individual patients; however, caution is needed in their use for heterogeneous critically ill populations.(11 Huang CY, Grandas FG, Flechet M, Meyfroidt G. Clinical prediction models for acute kidney injury in the intensive care unit: A systematic review. Rev Bras Ter Intensiva. 2020;32(1):123-32.) Recent literature has demonstrated major advances in the field of acute kidney injury prediction and the need for renal replacement therapy (RRT).(22 Ramos FJ, França AM, Salluh JI. Subphenotyping of critical illness: where protocolized and personalized intensive care medicine meet. Rev Bras Ter Intensiva. 2022;34(3):316-8.) In a large multicenter cohort, we evaluated how a previously published model(33 França AR, Rocha E, Bastos LS, Bozza FA, Kurtz P, Maccariello E, et al. Development and validation of a machine learning model to predict the use of renal replacement therapy in 14,374 patients with COVID-19. J Crit Care. 2023;80:154480.) that predicts the need for RRT in coronavirus disease 2019 (COVID-19) intensive care unit (ICU) patients would perform in a general ICU patient.
Recently, using a data-driven methodology in a multicenter cohort of 14,374 critically ill COVID-19 patients, we developed and validated a machine learning prediction model to predict the use of RRT (the "COVID-19-RRT Model").(33 França AR, Rocha E, Bastos LS, Bozza FA, Kurtz P, Maccariello E, et al. Development and validation of a machine learning model to predict the use of renal replacement therapy in 14,374 patients with COVID-19. J Crit Care. 2023;80:154480.) In the present study, we performed an external validation of the "COVID-19-RRT Model" in a cohort of non-COVID-19 adult patients admitted to 126 ICUs in 2022 in a Brazilian private hospital network. The data were acquired using a solution used for quality assessment (Epimed Monitor).(44 Zampieri FG, Soares M, Borges LP, Salluh JI, Ranzani OT. The Epimed Monitor ICU Database®: A cloud-based national registry for adult intensive care unit patients in Brazil. Rev Bras Ter Intensiva. 2017;29(4):418-26.) The study was approved by the Institutional Review Board after providing informed consent (Instituto D'Or de Pesquisa e Ensino [IDOR], CAAE:17079119.7.0000.5249). The prediction performance was evaluated in terms of calibration (plots and Brier's score) and discrimination (area under the ROC curve [AU-ROC]). A description of the materials and methods used are provided in the Supplementary Material (Table 1S, 2S and Figure 1S).
In 2022, 8,735 adult ICU patients without COVID-19 needed early respiratory support. Of these, 770 (8.8%) patients underwent RRT, a lower percentage than that in the development cohort (12%) (Table 1). Patients in the non-COVID-19 external validation cohort were older (median age 72 versus 56 years), more frequently female (54% versus 36%) and more frequently frail (43% versus 16%) than were those in the model development cohort. The median ICU stay was shorter (6 versus 10 days), and ICU mortality was lower compared to the development group (18% versus 22%). In the non-COVID-19 cohort, the model's AUC-ROC curve was 0.82 (95% confidence interval [95%CI]: 0.80 - 0.83), which was greater than that in the internal validation cohort (0.79; 95%CI: 0.78 - 0.82). Brier's score was comparable between the external validation dataset and the interval validation dataset; however, the calibration plots showed an overestimation of the predicted RRT probabilities, especially for patients at low risk (Figure 1).
Clinical characteristics and outcomes of critically ill general intensive care unit patients who needed respiratory support (within the first 24 hours after admission) and who received renal replacement therapy
Despite the good discrimination, the COVID-19-RRT Model overestimated the probability of needing RRT, especially in the "low-risk" strata.(55 Kurtz P, Bastos LS, Dantas LF, Zampieri FG, Soares M, Hamacher S, et al. Evolving changes in mortality of 13,301 critically ill adult patients with COVID-19 over 8 months. Intensive Care Med. 2021;47(5):538-48.) This may be explained by differences in the baseline severity of illness between COVID-19 patients and general ICU patients: the former had a lower severity at baseline, but the proportion of RRT use was greater than that in general ICU patients. Otherwise, a general ICU patient with a low disease severity at baseline will seldom require RRT. Therefore, despite good general performance, this model has limited clinical use for a mixed ICU population. Our study supports the need for models with better generalizability for the prediction of RRT and acute kidney injury in mixed ICU populations. Moreover, these findings should be interpreted with caution when translating the use of models developed for a specific population to a general group of critically ill patients.
DECLARATIONS
-
FundingThis study was supported by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) - Finance Code 001, Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ), the Pontifícia Universidade Católica do Rio de Janeiro, and departmental funds from the Instituto D'Or de Pesquisa e Ensino. All the authors carried out the research independently of the funding bodies. The findings and conclusions in this manuscript reflect the opinions of the authors alone.
-
Code availabilityThe programming code for the data analysis is available in the GitHub repository (https://github.com/lslbastos/covid_rrt_ml_model).
-
Publisher's Note
Availability of data and material
The data supporting this study's findings are available from the corresponding author upon reasonable request.
REFERENCES
-
1Huang CY, Grandas FG, Flechet M, Meyfroidt G. Clinical prediction models for acute kidney injury in the intensive care unit: A systematic review. Rev Bras Ter Intensiva. 2020;32(1):123-32.
-
2Ramos FJ, França AM, Salluh JI. Subphenotyping of critical illness: where protocolized and personalized intensive care medicine meet. Rev Bras Ter Intensiva. 2022;34(3):316-8.
-
3França AR, Rocha E, Bastos LS, Bozza FA, Kurtz P, Maccariello E, et al. Development and validation of a machine learning model to predict the use of renal replacement therapy in 14,374 patients with COVID-19. J Crit Care. 2023;80:154480.
-
4Zampieri FG, Soares M, Borges LP, Salluh JI, Ranzani OT. The Epimed Monitor ICU Database®: A cloud-based national registry for adult intensive care unit patients in Brazil. Rev Bras Ter Intensiva. 2017;29(4):418-26.
-
5Kurtz P, Bastos LS, Dantas LF, Zampieri FG, Soares M, Hamacher S, et al. Evolving changes in mortality of 13,301 critically ill adult patients with COVID-19 over 8 months. Intensive Care Med. 2021;47(5):538-48.
Edited by
Publication Dates
-
Publication in this collection
22 Apr 2024 -
Date of issue
2024
History
-
Received
24 Nov 2023 -
Accepted
02 Dec 2023
The external validation dataset included 8,735 patients with general intensive care unit admissions who needed respiratory support upon admission (1st hour), 770 of whom needed renal replacement therapy in 2022. (A) Discrimination was evaluated using receiver operating characteristic curves and area under the curve curves. (B) The calibration belt evaluates the concordance between predicted and observed values: shaded areas represent the confidence bands (belt) of 80% and 95%, and the red line is the bisector line (observed = predicted). When the confidence bands significantly deviate from the bisector line, the corresponding quantiles of under- or overestimation are displayed in the lower right part of the plots. The p value represents a statistical test for the null hypothesis of calibration.