Comparison of Machine Learning Tools for the Prediction of ICU Admission in COVID-19 Hospitalized Patients

Since December 2019, the coronavirus disease 2019 (COVID-19) has been spreading aggressively worldwide, and the health systems have encountered an increasing shortage of critical care resources, such as personal protective equipment, intensive care unit (ICU) beds, and ventilators (1). So far, this virus has affected almost all countries, with more than two million deaths worldwide (2). Approximately 20% of COVID-19 patients need hospitalization, and 20%-30% of COVID-19 in-hospital patients are admitted to ICU (3). In Iran, the ICU admission rate is estimated to be 32% of hospitalized patients, and the ICU death rate is about 39% (4). The ICU resources are currently restricted, and more than 50% of ICU beds are occupied under normal conditions (5).

The unexpected outbreak, rapid transmission, ambiguous disease course, and prognosis, as well as emerging new mutations of COVID-19, along with health systems being not prepared for large-scale epidemic responses, created a situation in which the management of intensive care resources is crucial (6). Under this condition, it is urgent to construct and test effective clinical risk prediction tools for appropriately triaging critical patients (7, 8). Accordingly, predicting the individual disease courses and outcomes is essential for triaging patients, customized care service provision, utilizing life-saving resources in the best possible way, and directing them toward vulnerable and at-risk sub-groups deteriorating to critical COVID-19. Furthermore, many problems resulting from the shortage of hospital resources can be overcome by predicting the risk of patient deterioration, determining the length of stay, using hospital resources efficiently, and managing bed turnover (9, 10).

Innovative approaches for the early identification and triaging of patients at the time of admission will be greatly beneficial and effective in estimating which patients are at the high risk of clinical deterioration and have poor outcomes requiring ICU admission (8, 9). In such conditions, the design and implementation of clinical decision support systems equipped with machine learning (ML)-based prediction models will be critical for the optimal use of limited hospital resources and supporting clinical decisions. The ML, as a sub-form of artificial intelligence technologies, provides new insight or knowledge through extracting functional patterns and applicable rules from the large raw datasets (7, 10).

In the previous researches, a large number of ML algorithms were trained for the prediction of COVID-19 disease progression, patient condition deterioration (9, 10), ICU hospitalization (8, 9, 11-13), and death (8, 11, 14-19).

The present study aimed to develop and validate a data-driven framework using five ML techniques to predict the patients who need transfer to ICU and find out the contributing clinical predictors by analyzing the available data at the time of admission.

This retrospective single-center cross-sectional study was conducted in 2021 to predict admission to ICU based on selected data-driven ML techniques.

3.1. Dataset and Participants

In this study, a COVID-19 hospital-based registry database from Taleghani Hospital, Abadan, Iran, was reviewed retrospectively. Only hospitalized confirmed COVID-19 patients aged ≥18 years and admitted during January 9, 2020 - January 20, 2021, met our inclusion criteria. During this period, a total of 13885 suspected cases with COVID-19 were referred to Taleghani Hospital, 3350 of which were confirmed as COVID-19 by RT-PCR test. The exclusion criteria entailed non-COVID-19 cases, non-hospitalized COVID-19 patients, cases with unknown disposition, patients under 18 years old, incomplete case records (missing more than 70%), and admission time before January 9, 2020, or after January 20, 2021. Following applying the exclusion criteria, finally, 1225 records were entered in the study (Figure 1).

Figure 1.

Flowchart describing patient selection

3.4. Study Roadmap and Experiment Environment

All experiments on the classification algorithms described in this study were implemented using Python version 3.7.7. The Python experiment environment offers a well-defined framework for researchers and developers to run and assess their ML models. The roadmap of the proposed prediction model in this work is depicted in Figure 2.

Figure 2.

Block diagram of the proposed system for predicting ICU admission in patients with COVID-19

Given the heterogeneity of the clinical manifestations of COVID-19, it is critical to develop models for predicting the likelihood of ICU admission by ML techniques. This study created five ML-based models using the most relevant variables in determining the risk of ICU admission derived from a correlation coefficient analysis. The techniques used herein included ANN, DT, KNN, SVM, and RF, which were trained through the most significant predictors from 1225 laboratory-confirmed COVID-19 patients at the time of admission. Finally, based on our analysis of selected algorithms, we found that RF with a mean accuracy of 99.5%, a mean specificity of 99.7%, and a mean sensitivity of 99.4% have better performance than other ML algorithms in predicting the probability of ICU transfer after hospital admission.

In the ICU, the need for informed decision-making is critical, especially in crisis circumstances, such as the current COVID-19 pandemic, where the healthcare systems encountered an increasing surge of patients and severe shortage in hospital resources (21, 22). The models developed in this study could be simply computerized as an alternative to manual and subjective clinical assessment methods. To correctly extract clinical predictors for estimating the potential need for ICU services, we evaluated clinical features at the time of admission and not at the progressive/severe course of the disease. In addition, the critical patients at admission time were discarded from the analysis. Thus, if validated, these features could be applied for predicting the likelihood to enter in ICU at the first hospitalization. For this purpose, Feature selection is a significant step to prepare the data before entering into the model (23). Hence, we identified the most important variables (n = 11) through correlation coefficient at P-value < 0.05. The most significant predictors of ICU admission were older age, high creatinine, leukocytosis, increased BUN, elevated ASP/ALT, augmented LDH, dry cough, hypertension, cardiovascular disorders, diabetes, dyspnea, decreased SPO₂, pneumonia, and high C-reactive protein.

Many studies have focused on determining the key risk factors for ICU admission (8, 9, 11, 13, 24, 25). The ten top clinical variables predicting ICU risk in reviewed studies encompassed age (older age), body temperature (high), oxygen saturation (decreased), neutrophil count and lymphocyte count (raised), C-reactive protein (elevated), D-dimer (increased), ALT and/or AST (augmented), LDH (elevated), loss of consciousness, and hypertension/cardiovascular diseases. In general, high compliance was observed between the results of reviewed studies and the most common variables in the current study.

In general, the developed ML algorithms in this study, similar to those reported in the previous studies (26), have achieved optimum results with an accuracy range of 86.4% - 90.37%. In particular, the experimental findings showed that RF had the best performance compared to the other four ML techniques with the mean accuracy of 99.5%, mean specificity of 99.7%, mean sensitivity of 99.4%, Kappa metric of 95.7%, and RMSE of 0.015. According to the results of the previous studies, the ANN and RF techniques have the most remarkable performance in predicting COVID-19 outcomes, which is consistent with the present study.

As a screening instrument for the development of severe disease, model developed in our study has several opportunities for clinical use. These models decrease the existing uncertainty and ambiguity in COVID-19 clinical practice by presenting measurable, non-subjective, and evidence-based approaches (12, 18). Accurate ICU admission prediction can support the sharing of limited hospital resources and improve the quality of care along with patient survival chance (12). The timely identification of at-risk patients could diminish the need for imminent ICU beds and invasive mechanical ventilators. Moreover, using proposed model in present study can surge the tolls of timely ICU transfers, resulting in reduced mortality and shorter lengths of ICU stay. Designing a scientific and valid ML-based prediction model would assist in early detection and effective supportive intervention to improve patient outcomes, the quality of care, and ultimately a reduction in the mortality rate of COVID-19 patients. Ambiguity declines due to offering quantitative, objective, and evidence-based models for risk stratification, prediction, and care planning (9, 10).

This study had several limitations. First, we retrospectively analyzed a dataset without control over data fields or incomplete data. Second, the dataset was extracted from a single hospital with a low sample size of 1225, making the results ungeneralizable. Third, this study only included 11 clinical features at admission to the hospital. It does not mean that these should be the only criteria for determining ICU admission. Longitudinal changes in these clinical features need to be investigated. Moreover, we only used five ML algorithms for prediction analyses. Finally, the selected dataset lacked some critical clinical variables, such as radiological indicators. In the future, the performance accuracy of our model and its generalizability will be enhanced if we test more ML techniques for larger, multicenter, and prospective datasets equipped with more qualitative and validated data.