Validation of Pediatric Index of Mortality 3 in a Single Referral Pediatric Intensive Care Unit in Iran

authors:

avatar Hossein Saidi 1 , avatar Hamed Basir Ghafouri 2 , avatar Hamed Aghdam 2 , * , avatar Ghamartaj Khanbabaei 3 , avatar Narges Ahmadizadeh 1 , avatar Atefeh Ahmadi 3

Mofid Children's Hospital, Shahid Beheshti University of Medical Sciences, Tehran, Iran
Trauma and Injury Research Center, Iran University of Medical Sciences, Tehran, Iran
PICU of Mofid Children's Hospital, Shahid Beheshti University of Medical Sciences, Tehran, Iran

how to cite: Saidi H , Basir Ghafouri H, Aghdam H, Khanbabaei G , Ahmadizadeh N, et al. Validation of Pediatric Index of Mortality 3 in a Single Referral Pediatric Intensive Care Unit in Iran. Arch Pediatr Infect Dis. 2021;9(3):e104428. https://doi.org/10.5812/pedinfect.104428.

Abstract

Objectives:

The research aimed to evaluate the Pediatric Index of Mortality 3 (PIM-3) for determining the risk of mortality among pediatric intensive care unit patients.

Methods:

A retrospective analysis was conducted on case records, as well as patient data from all admissions to the PICU of Mofid Children’s Hospital, Tehran, from October 2017 to February 2018. Employing an android calculator application, the PIM-3 score was estimated early within the first PICU admission. Then, the PIM-3 score and mortality rate were analyzed using the Mann-Whitney U test. In addition, calibration and discrimination were assessed by the Hosmer-Lemeshow goodness-of-fit test and a receiver operating characteristic curve method, respectively. Finally, the Standardized Mortality Ratio (SMR) was calculated.

Results:

In this study, 365 young infants, ranging from 10 to 29-months-old, were included. The overall mortality rate was 10.4%. Further, the patients’ PIM-3 scores ranged from 0.06% to 2.37% (95% confidence interval), with a mean of 1.45% (4.16% in non-survivors and 1.14% in survivors). The SMR was estimated at 7.18, demonstrating the underprediction of the death rate. The AUC of 0.714 (95% CI: 0.626 to 0.801) demonstrated a fair to good discrimination power of PIM-3 as an international standard risk-adjusted mortality indicator. Moreover, this score underpredicted the risk of mortality in young infants admitted to our ICU in 2017. Generally, the prediction was weak among low-risk patients. Therefore, the Pediatric Index of Mortality-3 score has the potential to be implemented in our PICU by modifying the expected probability of death by multiplying the original PIM-3 score by 7.12.

1. Background

Different models such as the pediatric risk of mortality (PRISM) and pediatric index of mortality (PIM) and their derivatives have been developed to predict mortality among pediatric intensive care unit (PICU) patients and evaluate the quality of intensive care provided to these patients (1, 2). These models are objectively used to assess the severity of illness, make inter-and intra-unit comparisons based on time, evaluate and compare outcomes, survival rate, quality, and cost-efficacy of measures, randomization, and stratification of patients to clinical trials, and make decisions in clinical settings. The PRISM III is a common instrument to assess the severity of illness in terms of physiological variables and their ranges (1). De Leon et al. (1) assessed the association of the PRISM III score with outcomes in infants admitted to the PICU. A total of 170 infants who were consecutively admitted to the PICU were enrolled in their prospective cohort study. With a sensitivity of 0.71 and specificity of 0.64, the PRISM III was found as a mortality predictor. Such scores have been obtained from particular study populations during particular periods. Several factors such as primary diagnosis, baseline health status, laboratory data, and severity of the illness can influence the risk of death in an individual patient.

Mofid Children’s Hospital is a referral children’s hospital in Tehran, Iran, which exclusively presents tertiary and quaternary services, particularly for young infants. The Intensive Care Unit (ICU) in this hospital has a yearly patient flow of 1000 to 1500 young infants.

The first version of the PIM model was derived by analyzing data gathered from 5,695 datasheets of eight PICUs, including seven in Australia and one in the United Kingdom (3). The second version (PIM-2) was designed by utilizing data obtained from 20,787 pediatric patients treated under intensive care in Australia, New Zealand, and the United Kingdom during 1997-1999 (4). The relationships between mortality and the predictors have changed since the development of the PIM-2 model (5) and recent applications of PIM-2 for other study populations have shown mixed results (6-10). As a valid scoring system, PIM3 provides reliable estimates of mortality risk among children admitted to the ICU (11).

To the best of the authors’ knowledge, the performance of these predictive models varies, and it is still questioned that which scoring model is better for predicting outcomes in critically ill children (12).

2. Objectives

Considering the special patient population in our PICU and the above-mentioned explanations, the present study aimed to investigate the generalized validity of PIM-3 as a PICU mortality risk assessment model in Mofid Children’s Hospital using recent data.

3. Methods

The data were collected in the Pediatric Intensive Care Unit (PICU) of Mofid Children’s Hospital from October 2017 to February 2018.

3.1. Data Collection

The obtained data from consecutive admissions to the ICU included information about the patients’ age, sex, weight, height, nutritional status, and diagnosis, as well as PIM-3-related variables such as blood pressure and pupillary light reflex. In the current study, different variables were assessed, such as partial oxygen tension (PaO2) and the fraction of inspired oxygen (FiO2), if oxygen was supplied by different ventilatory support methods, especially during the first hour of PICU admission, blood oxygenation status, and acid-base status assessment in arterial blood gas analysis, elective or non-elective admission status, recovery from surgery or ICU admission related to the surgery, and the presence of low-risk, high-risk, and very high-risk diagnosis. Variable definition and scoring methods were outlined according to PIM-3 developers’ guidelines (11).

3.2. Inclusion and Exclusion Criteria

Data were collected in a way slightly similar to the previous studies of PIM (PIM-2 and PIM-3) (3, 4, 11) using datasheets according to the general instruction in appendix 1. All consecutive patients older than 28 days and younger than 14 years admitted to the PICU were included in this study. Readmissions were considered as new admissions. On the other hand, patients suffering from end-stage diseases such as cardiopulmonary arrest before admission, as well as cardiac, hepatic, or renal failure and neurodegenerative and hemato-oncological diseases were excluded from the study. Patients transferred to other hospitals or those who died in the first 24 hours of admission were excluded.

3.3. Model Development and Selection

As one of the several severity-scoring systems, PIM-3 is used for predicting the outcome of pediatric patients admitted to the PICU based on data collected within the first hour of admission. It is a simple, inexpensive system consisting of 10 freely available variables that predict mortality before any additional diagnosis or therapies received. The variables in the formula of predicted mortality of the PIM-3 scoring system compared to the other scoring formula are more accessible. The PIM-3 is an updated version of PIM-2 for the comparison of risk-adjusted mortality among children admitted to the ICU. The scores were calculated using the PIM-3 calculator application. Further, the obtained data were entered in SPSS version 20.0 software, and the association between the PIM-3 score and mortality was analyzed using this software.

Different risks of mortality were calibrated for PIM-3 by the Hosmer-Lemeshow test. The standardized mortality ratio (SMR) is the ratio of observed deaths in the study group to expected deaths in the general population. Using the SMR, the calibration of the PIM-3 model was done by dividing the observed mortality rate by the expected rate. In addition, the chi-square test was run using the ∑ (O-E)2/E formula, where O refers to Observed and E to Expected, for the survivors and non-survivors in each interval group. The Hosmer-Lemeshow test was used to test the goodness-of-fit based on the PIM-3 score groups. Diagnoses with SMRs, which were significantly different from each other at a 95% confidence interval, were calculated for categorizing into different conditions of low or high-risk weighting.

Similarly, the discriminatory evaluation was done using the area under the ROC curve to show how well PIM distinguished between the patients who survived and those who died. Eventually, the predicted mortality was compared with the observed mortality using the AUC of the ROC of sensitivity by 1-specificity to assess the discrimination between death and survival (13). The discrimination measured by AUC was considered to be very good, good, and fair for ROC > 0.9, 0.80 - 0.90, and 0.70 - 0.80, respectively.

4. Results

Data from 365 patients admitted to the PICU were collected based on the inclusion and exclusion criteria during six months, among whom 38 (10.4%) cases died based on the reports. The patients` median age was 49 months, and the 10-29-month-old group (34.32%) accounted for the highest mortality number. Male patients comprised the majority of the sample (60.2%).

In this study, the range of PIM-3 scores was 0.06% to 2.37% (95% Confidence Interval), with a mean score of 1.45%. In addition, most patients received scores of less than 1%. The lower level of PIM-3 score was correlated with higher mortality probability. The mean PIM-3 score in non-survivors was 4.16%, while that of the survivors was 1.14% (Table 1).

Table 1.

Pediatric Index of Mortality 3 Score and Patient Outcome

OutcomeNMean PIM-3 Score (Range), %P Value
Non-survivors384.16 (0.0 - 27.0)0.001
Survivors3271.14 (0.0 - 22.0)

Furthermore, according to our results, non-survivors’ PIM-3 scores ranged from 0.0% to 27.0%, with a mean of 4.16%, whereas the mean PIM-3 score was 1.14% for survivors, ranging from 0.0% to 22.0%. Most patients obtained scores that were less than 1%, and the lower level of PIM-3 score indicated a higher probability of mortality. Additionally, the results of the Mann-Whitney U test showed that non-survivors` median PIM-3 score was significantly higher than that in the survivors (P = 0.001).

Table 2 demonstrates the SMR calibration of the PIM-3 model based on four intervals (e.g., < 1%, 1-5%, 5 - 20%, and > 20%). The overall SMR was estimated at 7.18. Further, the expected mortality rate calculated with the PIM-3 scoring chart was 1.45%, which was less than the actual observed mortality rate (10.4%).

Table 2.

Pediatric Index of Mortality 3 Score Intervals and Observed and Expected Mortality

PIM 3 Score Interval by GroupLess than 1%1 - 5%5 - 20%More than 20%Total
Mean PIM 3 score, %0.222.719.1424.021.45
N27860234365
Non-survivors
Observed, n2186338
Expected, n (%)0.61 (0.21)1.63 (2.71)2.10 (9.14)0.96 (28.10)5.29 (1.74)
Survivors
Observed, n25752171327
Expected, n (%)277.42 (99.79)58.37 (97.29)20.90 (90.86)2.88 (71.9)358.65 (98.26)
Chi-square value of Hosmer Lemeshow goodness-of-fit0.0000.0000.0000.0010.000
SMR34.434.882.863.1267.18

Table 3 and Figure 1 show the ROC analysis. The Area Under the Curve (AUC) of PIM-3 score in this study was 0.711 (95% CI 0.63 to 0.80), which was considered accurate for predicting death probability.

Table 3.

Discrimination of Pediatric Index of Mortality 3 Concerning Mortality Rate

VariablesAUCSE95% CI
PIM-3 score0.7110.0450.626 - 0.801
ROC Curve Analysis for Pediatric Index of Mortality 3 Scores with the Area Under the Curve (AUC)
ROC Curve Analysis for Pediatric Index of Mortality 3 Scores with the Area Under the Curve (AUC)

5. Discussion

The Pediatric Index of Mortality (PIM) model, as an available method for assessing a child’s death probability in the ICU, is based on data gathered during the early hours of admission. The PIM-3, as an updated version of this model, was developed using large registries in ICUs in different countries all over the world. Data for PIM are collected during the early hours of admission, avoiding the probable bias of therapeutic and drug effects during hospitalization, which may propose to assign high-risk young infants to intensive care shortly after admission.

Our study investigated 365 PICU patients at Mofid hospital, Tehran, Iran, for six months to evaluate the discrimination ability of PIM-3. The overall discrimination ability for the final PIM-3 model was slightly similar to that of the PIM-2 model (0.71 vs. 0.90) (11). Additionally, the prevalence of mortality in our PICU was 10.4%, which was less than that of the study by Honna et al. (45.7%) (14), Gandi et al. (46.2%) (15), and Qureshi et al. (28.7%) (12). The results of our study are in line with another study conducted in Iran (15%) (16) and a study in Egypt (8.5%) (17).

The calculated mortality risk was lower than observed rates at other PICUs where the prognostic scores were validated. The probable factors affecting the PICU outcome include demographic and clinical characteristics, health status, human sources (i.e., nurse to patient ratio, human and organizational factors, subjective factors in calculating PIM-3 score, physician's standards of care, as well as nursing skill in collecting arterial blood specimens), and the validity of the laboratory measurements reported (18).

The SMR for all groups was > 1, ranging from 2.86 - 34.43, with mean actual mortality of 7.18 times the expected rate, and this rate was significantly higher in the 1 - 5% interval group (34.43 times). Multiple factors such as a poor referral system and, somehow, delayed initial therapy or surgical/medical complications may affect these results.

The ability of PIM-3 to predict the death rate was 1.45%, which was lower than the observed rate of 10.4%. The overall SMR was 7.18, meaning that the PIM-3 model underpredicted the mortality rate in our study unit. It will be better explained if the standards of care in different PICUs are studied. Likewise, death probability was 7.12 times higher in Mofid Children's Hospital. Other studies in Egypt, Pakistan, and India reported SMRs from PIM-2 scores of 1.57, 1.92, and 3.3, respectively (12, 16, 17). On the other hand, a study in Japan reported PIM-2 SMR < 1 (0.77), representing that the score showed an overprediction of mortality (8).

Our study has several limitations. First, it was a single-center study conducted at a tertiary hospital. The findings may, therefore, not be generalizable to the entire pediatric population of Iran. Multicenter studies that include neonatal, surgical and cardiac ICU patients in primary and secondary hospitals are needed. The second limitation was the retrospective data collection. Although the authors made every effort to validate the data thoroughly, some patients were excluded from the study because of incomplete data. Prospective studies on this topic in the future will be helpful.

Finally, the AUC was measured as 0.711 (95%CI: 0.626 to 0.821), which was less than the AUC in more related studies. However, AUCs in developed countries, including Scotland, the United Kingdom, New Zealand, and Australia, were found to be 0.84 - 0.86, 0.85, 0.90 - 0.93, and 0.91, respectively (6).

5.1. Conclusion

Overall, the results indicated that PIM-3 underpredicted the risk of mortality in young infants admitted to our ICU in 2017. In other words, the prediction was weak among low-risk patients. In this regard, the PIM-3 score can be administrated in our PICU to calculate the probable mortality rate by correcting with multiplying the calculated PIM3 score by 7.12.

References

  • 1.

    Slater A, Shann F; Anzics Paediatric Study Group. The suitability of the Pediatric Index of Mortality (PIM), PIM2, the Pediatric Risk of Mortality (PRISM), and PRISM III for monitoring the quality of pediatric intensive care in Australia and New Zealand. Pediatr Crit Care Med. 2004;5(5):447-54. [PubMed ID: 15329160]. https://doi.org/10.1097/01.PCC.0000138557.31831.65.

  • 2.

    Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: Hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006;34(5):1297-310. [PubMed ID: 16540951]. https://doi.org/10.1097/01.CCM.0000215112.84523.F0.

  • 3.

    Shann F, Pearson G, Slater A, Wilkinson K. Paediatric index of mortality (PIM): A mortality prediction model for children in intensive care. Intensive Care Med. 1997;23(2):201-7. [PubMed ID: 9069007]. https://doi.org/10.1007/s001340050317.

  • 4.

    Slater A, Shann F, Pearson G; Paediatric Index of Mortality Study Group. PIM2: A revised version of the Paediatric Index of Mortality. Intensive Care Med. 2003;29(2):278-85. [PubMed ID: 12541154]. https://doi.org/10.1007/s00134-002-1601-2.

  • 5.

    Pearson GA, Stickley J, Shann F. Calibration of the paediatric index of mortality in UK paediatric intensive care units. Arch Dis Child. 2001;84(2):125-8. [PubMed ID: 11159286]. [PubMed Central ID: PMC1718658]. https://doi.org/10.1136/adc.84.2.125.

  • 6.

    Eulmesekian PG, Perez A, Minces PG, Ferrero H. Validation of pediatric index of mortality 2 (PIM2) in a single pediatric intensive care unit of Argentina. Pediatr Crit Care Med. 2007;8(1):54-7. [PubMed ID: 17251883]. https://doi.org/10.1097/01.pcc.0000256619.78382.93.

  • 7.

    Ng DK, Miu TY, Chiu WK, Hui NT, Chan CH. Validation of Pediatric Index of Mortality 2 in three pediatric intensive care units in Hong Kong. Indian J Pediatr. 2011;78(12):1491-4. [PubMed ID: 21617907]. https://doi.org/10.1007/s12098-011-0443-8.

  • 8.

    Imamura T, Nakagawa S, Goldman RD, Fujiwara T. Validation of pediatric index of mortality 2 (PIM2) in a single pediatric intensive care unit in Japan. Intensive Care Med. 2012;38(4):649-54. [PubMed ID: 22270470]. https://doi.org/10.1007/s00134-011-2460-5.

  • 9.

    Taori RN, Lahiri KR, Tullu MS. Performance of PRISM (Pediatric Risk of Mortality) score and PIM (Pediatric Index of Mortality) score in a tertiary care pediatric ICU. Indian J Pediatr. 2010;77(3):267-71. [PubMed ID: 20177831]. https://doi.org/10.1007/s12098-010-0031-3.

  • 10.

    Czaja AS, Scanlon MC, Kuhn EM, Jeffries HE. Performance of the Pediatric Index of Mortality 2 for pediatric cardiac surgery patients. Pediatr Crit Care Med. 2011;12(2):184-9. [PubMed ID: 20581732]. https://doi.org/10.1097/PCC.0b013e3181e89694.

  • 11.

    Straney L, Clements A, Parslow RC, Pearson G, Shann F, Alexander J, et al. Paediatric index of mortality 3: an updated model for predicting mortality in pediatric intensive care. Pediatr Crit Care Med. 2013;14(7):673-81. [PubMed ID: 23863821]. https://doi.org/10.1097/PCC.0b013e31829760cf.

  • 12.

    Qureshi AU, Ali AS, Ahmad TM. Comparison of three prognostic scores (PRISM, PELOD and PIM 2) at pediatric intensive care unit under Pakistani circumstances. J Ayub Med Coll Abbottabad. 2007;19(2):49-53. [PubMed ID: 18183720].

  • 13.

    Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148(3):839-43. [PubMed ID: 6878708]. https://doi.org/10.1148/radiology.148.3.6878708.

  • 14.

    Honna L, Triratna S, Triwani T, Theodorus T. Use of pediatric logistic organ dysfunction in determining prognostic among pediatric intensive care unIt patIents. Paediatr Indones. 2016;50(6):347. https://doi.org/10.14238/pi50.6.2010.347-50.

  • 15.

    Gandhi J, Sangareddi S, Varadarajan P, Suresh S. Pediatric index of mortality 2 score as an outcome predictor in pediatric Intensive Care Unit in India. Indian J Crit Care Med. 2013;17(5):288-91. [PubMed ID: 24339640]. [PubMed Central ID: PMC3841491]. https://doi.org/10.4103/0972-5229.120320.

  • 16.

    Salamati P, Talaee S, Eghbalkhah A, Chaman R, Mokhtari Z, Azarshahin M. Validation of pediatric index of mortality-2 scoring system in a single pediatric intensive care unit in Iran. Iran J Pediatr. 2012;22(4):481-6. [PubMed ID: 23429969]. [PubMed Central ID: PMC3533148].

  • 17.

    Rene labib Youssef M, Mosleh H, Rene Labib J. Assessment of the performance of the Pediatric Index of Mortality 2 (PIM2) among Egyptian pediatric patients admitted to the intensive care. Gaz Egypt Paediatr Assoc. 2014;62(3-4):65-71. https://doi.org/10.1016/j.epag.2014.10.002.

  • 18.

    Zimmerman JE. Measuring intensive care unit performance: a way to move forward. Crit Care Med. 2002;30(9):2149-50. [PubMed ID: 12352059]. https://doi.org/10.1097/00003246-200209000-00036.