Validation of Pediatric Index of Mortality 3 in a Single Referral Pediatric Intensive Care Unit in Iran

1. Background

Different models such as the pediatric risk of mortality (PRISM) and pediatric index of mortality (PIM) and their derivatives have been developed to predict mortality among pediatric intensive care unit (PICU) patients and evaluate the quality of intensive care provided to these patients (1, 2). These models are objectively used to assess the severity of illness, make inter-and intra-unit comparisons based on time, evaluate and compare outcomes, survival rate, quality, and cost-efficacy of measures, randomization, and stratification of patients to clinical trials, and make decisions in clinical settings. The PRISM III is a common instrument to assess the severity of illness in terms of physiological variables and their ranges (1). De Leon et al. (1) assessed the association of the PRISM III score with outcomes in infants admitted to the PICU. A total of 170 infants who were consecutively admitted to the PICU were enrolled in their prospective cohort study. With a sensitivity of 0.71 and specificity of 0.64, the PRISM III was found as a mortality predictor. Such scores have been obtained from particular study populations during particular periods. Several factors such as primary diagnosis, baseline health status, laboratory data, and severity of the illness can influence the risk of death in an individual patient.

Mofid Children’s Hospital is a referral children’s hospital in Tehran, Iran, which exclusively presents tertiary and quaternary services, particularly for young infants. The Intensive Care Unit (ICU) in this hospital has a yearly patient flow of 1000 to 1500 young infants.

The first version of the PIM model was derived by analyzing data gathered from 5,695 datasheets of eight PICUs, including seven in Australia and one in the United Kingdom (3). The second version (PIM-2) was designed by utilizing data obtained from 20,787 pediatric patients treated under intensive care in Australia, New Zealand, and the United Kingdom during 1997-1999 (4). The relationships between mortality and the predictors have changed since the development of the PIM-2 model (5) and recent applications of PIM-2 for other study populations have shown mixed results (6-10). As a valid scoring system, PIM3 provides reliable estimates of mortality risk among children admitted to the ICU (11).

To the best of the authors’ knowledge, the performance of these predictive models varies, and it is still questioned that which scoring model is better for predicting outcomes in critically ill children (12).

2. Objectives

Considering the special patient population in our PICU and the above-mentioned explanations, the present study aimed to investigate the generalized validity of PIM-3 as a PICU mortality risk assessment model in Mofid Children’s Hospital using recent data.

3. Methods

The data were collected in the Pediatric Intensive Care Unit (PICU) of Mofid Children’s Hospital from October 2017 to February 2018.

3.1. Data Collection

The obtained data from consecutive admissions to the ICU included information about the patients’ age, sex, weight, height, nutritional status, and diagnosis, as well as PIM-3-related variables such as blood pressure and pupillary light reflex. In the current study, different variables were assessed, such as partial oxygen tension (PaO₂) and the fraction of inspired oxygen (FiO₂), if oxygen was supplied by different ventilatory support methods, especially during the first hour of PICU admission, blood oxygenation status, and acid-base status assessment in arterial blood gas analysis, elective or non-elective admission status, recovery from surgery or ICU admission related to the surgery, and the presence of low-risk, high-risk, and very high-risk diagnosis. Variable definition and scoring methods were outlined according to PIM-3 developers’ guidelines (11).

3.2. Inclusion and Exclusion Criteria

Data were collected in a way slightly similar to the previous studies of PIM (PIM-2 and PIM-3) (3, 4, 11) using datasheets according to the general instruction in appendix 1. All consecutive patients older than 28 days and younger than 14 years admitted to the PICU were included in this study. Readmissions were considered as new admissions. On the other hand, patients suffering from end-stage diseases such as cardiopulmonary arrest before admission, as well as cardiac, hepatic, or renal failure and neurodegenerative and hemato-oncological diseases were excluded from the study. Patients transferred to other hospitals or those who died in the first 24 hours of admission were excluded.

3.3. Model Development and Selection

As one of the several severity-scoring systems, PIM-3 is used for predicting the outcome of pediatric patients admitted to the PICU based on data collected within the first hour of admission. It is a simple, inexpensive system consisting of 10 freely available variables that predict mortality before any additional diagnosis or therapies received. The variables in the formula of predicted mortality of the PIM-3 scoring system compared to the other scoring formula are more accessible. The PIM-3 is an updated version of PIM-2 for the comparison of risk-adjusted mortality among children admitted to the ICU. The scores were calculated using the PIM-3 calculator application. Further, the obtained data were entered in SPSS version 20.0 software, and the association between the PIM-3 score and mortality was analyzed using this software.

Different risks of mortality were calibrated for PIM-3 by the Hosmer-Lemeshow test. The standardized mortality ratio (SMR) is the ratio of observed deaths in the study group to expected deaths in the general population. Using the SMR, the calibration of the PIM-3 model was done by dividing the observed mortality rate by the expected rate. In addition, the chi-square test was run using the ∑ (O-E)²/E formula, where O refers to Observed and E to Expected, for the survivors and non-survivors in each interval group. The Hosmer-Lemeshow test was used to test the goodness-of-fit based on the PIM-3 score groups. Diagnoses with SMRs, which were significantly different from each other at a 95% confidence interval, were calculated for categorizing into different conditions of low or high-risk weighting.

Similarly, the discriminatory evaluation was done using the area under the ROC curve to show how well PIM distinguished between the patients who survived and those who died. Eventually, the predicted mortality was compared with the observed mortality using the AUC of the ROC of sensitivity by 1-specificity to assess the discrimination between death and survival (13). The discrimination measured by AUC was considered to be very good, good, and fair for ROC > 0.9, 0.80 - 0.90, and 0.70 - 0.80, respectively.

4. Results

Data from 365 patients admitted to the PICU were collected based on the inclusion and exclusion criteria during six months, among whom 38 (10.4%) cases died based on the reports. The patients` median age was 49 months, and the 10-29-month-old group (34.32%) accounted for the highest mortality number. Male patients comprised the majority of the sample (60.2%).

In this study, the range of PIM-3 scores was 0.06% to 2.37% (95% Confidence Interval), with a mean score of 1.45%. In addition, most patients received scores of less than 1%. The lower level of PIM-3 score was correlated with higher mortality probability. The mean PIM-3 score in non-survivors was 4.16%, while that of the survivors was 1.14% (Table 1).

Furthermore, according to our results, non-survivors’ PIM-3 scores ranged from 0.0% to 27.0%, with a mean of 4.16%, whereas the mean PIM-3 score was 1.14% for survivors, ranging from 0.0% to 22.0%. Most patients obtained scores that were less than 1%, and the lower level of PIM-3 score indicated a higher probability of mortality. Additionally, the results of the Mann-Whitney U test showed that non-survivors` median PIM-3 score was significantly higher than that in the survivors (P = 0.001).

Table 2 demonstrates the SMR calibration of the PIM-3 model based on four intervals (e.g., < 1%, 1-5%, 5 - 20%, and > 20%). The overall SMR was estimated at 7.18. Further, the expected mortality rate calculated with the PIM-3 scoring chart was 1.45%, which was less than the actual observed mortality rate (10.4%).

Table 3 and Figure 1 show the ROC analysis. The Area Under the Curve (AUC) of PIM-3 score in this study was 0.711 (95% CI 0.63 to 0.80), which was considered accurate for predicting death probability.

ROC Curve Analysis for Pediatric Index of Mortality 3 Scores with the Area Under the Curve (AUC)

Figure 1.

ROC Curve Analysis for Pediatric Index of Mortality 3 Scores with the Area Under the Curve (AUC)

5. Discussion

The Pediatric Index of Mortality (PIM) model, as an available method for assessing a child’s death probability in the ICU, is based on data gathered during the early hours of admission. The PIM-3, as an updated version of this model, was developed using large registries in ICUs in different countries all over the world. Data for PIM are collected during the early hours of admission, avoiding the probable bias of therapeutic and drug effects during hospitalization, which may propose to assign high-risk young infants to intensive care shortly after admission.

Our study investigated 365 PICU patients at Mofid hospital, Tehran, Iran, for six months to evaluate the discrimination ability of PIM-3. The overall discrimination ability for the final PIM-3 model was slightly similar to that of the PIM-2 model (0.71 vs. 0.90) (11). Additionally, the prevalence of mortality in our PICU was 10.4%, which was less than that of the study by Honna et al. (45.7%) (14), Gandi et al. (46.2%) (15), and Qureshi et al. (28.7%) (12). The results of our study are in line with another study conducted in Iran (15%) (16) and a study in Egypt (8.5%) (17).

The calculated mortality risk was lower than observed rates at other PICUs where the prognostic scores were validated. The probable factors affecting the PICU outcome include demographic and clinical characteristics, health status, human sources (i.e., nurse to patient ratio, human and organizational factors, subjective factors in calculating PIM-3 score, physician's standards of care, as well as nursing skill in collecting arterial blood specimens), and the validity of the laboratory measurements reported (18).

The SMR for all groups was > 1, ranging from 2.86 - 34.43, with mean actual mortality of 7.18 times the expected rate, and this rate was significantly higher in the 1 - 5% interval group (34.43 times). Multiple factors such as a poor referral system and, somehow, delayed initial therapy or surgical/medical complications may affect these results.

The ability of PIM-3 to predict the death rate was 1.45%, which was lower than the observed rate of 10.4%. The overall SMR was 7.18, meaning that the PIM-3 model underpredicted the mortality rate in our study unit. It will be better explained if the standards of care in different PICUs are studied. Likewise, death probability was 7.12 times higher in Mofid Children's Hospital. Other studies in Egypt, Pakistan, and India reported SMRs from PIM-2 scores of 1.57, 1.92, and 3.3, respectively (12, 16, 17). On the other hand, a study in Japan reported PIM-2 SMR < 1 (0.77), representing that the score showed an overprediction of mortality (8).

Our study has several limitations. First, it was a single-center study conducted at a tertiary hospital. The findings may, therefore, not be generalizable to the entire pediatric population of Iran. Multicenter studies that include neonatal, surgical and cardiac ICU patients in primary and secondary hospitals are needed. The second limitation was the retrospective data collection. Although the authors made every effort to validate the data thoroughly, some patients were excluded from the study because of incomplete data. Prospective studies on this topic in the future will be helpful.

Finally, the AUC was measured as 0.711 (95%CI: 0.626 to 0.821), which was less than the AUC in more related studies. However, AUCs in developed countries, including Scotland, the United Kingdom, New Zealand, and Australia, were found to be 0.84 - 0.86, 0.85, 0.90 - 0.93, and 0.91, respectively (6).

5.1. Conclusion

Overall, the results indicated that PIM-3 underpredicted the risk of mortality in young infants admitted to our ICU in 2017. In other words, the prediction was weak among low-risk patients. In this regard, the PIM-3 score can be administrated in our PICU to calculate the probable mortality rate by correcting with multiplying the calculated PIM3 score by 7.12.