Abstract
Background:
The pediatric risk of mortality (PRISM III), pediatric index of mortality (PIM3), and pediatric logistic organ dysfunction (PELOD-2) are of the most used predictive models in predicting the risk of mortality in the pediatric intensive care unit (PICU).Objectives:
The current study aimed at comparing the predictive ability of these three modes in medical/surgical ICUs (MICU/SICU).Methods:
A total of 90 consecutive patients, aged ≤ 18 years, admitted to MICUs or SICUs were enrolled in the current observational, prospective study. The PRISM III, PIM3, and PELOD-2 as well as demographic characteristics of the subjects were recorded on admission. A receive operator characteristic (ROC) curve, logistic regression, and the Hosmer-Lemeshow goodness-of-fit test were used for statistical analyses [95% confidence interval (CI)].Results:
Data analysis showed a significant difference in PRISM III, PIM3, and PELOD-2 scores between survivors and nonsurvivors (P < 0.001, P < 0.001, P < 0.001, respectively). The discrimination power was moderate for PRISM III (area under ROC curve (AUC): 77.3%; standard error (SE): 6.0%), and good for PIM3 and PELOD-2 (AUC: 82.4%, SE: 5.5% and AUC: 80.3%, SE: 4.9%, respectively). All the three models were well calibrated (χ2 = 4.73, P = 0.79; χ2 = 3.09, P = 0.93; and χ2 = 5.01, P = 0.66, respectively).Conclusions:
PRISM III, PIM3, and PELOD-2 had good performance in predicting outcomes in children admitted to MICUs or SICUs. Further studies on different ICUs may provide more conclusive results with greater generalization of the validity of these predictive models.Keywords
Child Mortality Intensive Care Unit Nonsurvivors Pediatrics Severity of Illness Index Survivors
1. Background
Predictive models such as the pediatric risk of mortality (PRISM III), pediatric index of mortality (PIM3), and pediatric logistic organ dysfunction (PELOD-2) are objectively used to assess the severity of illness, inter- and intra-unit comparisons based on time, evaluation and comparison of outcomes, survival rate, quality, and cost-efficacy of measures, randomization and stratification of patients to clinical trials, and decision making in clinical settings.
The PRISM III is a common instrument to assess severity of illness in terms of physiological variables and their ranges (1). The PIM3 is also a scoring system provides reliable estimates of mortality risk among children admitted to ICU (2). Also, the PELOD-2 is a valid scoring system allows assessment of severity of cases of multi organ dysfunction in the pediatric ICU (PICU) and can be used as an outcome measure tool in clinical trials (3). Since each ICU serves a different patient population, a specific predictive model should be tailored for each hospital to certify the applicability of the model (4-6). Before applying a predictive model in a particular population, its external validity should be evaluated for the generalizability of the results (7).
Leteurtre et al. (8) in a prospective, multicenter, cohort study, developed the PELOD-2 by multivariable logistic regressions and bootstrap process. A total of 3671 consecutive patients, with the mean age of 15.5 months (interquartile range (IQR) = 2.2 - 70.7) were enrolled in their study. The discrimination (areas under the receiver operating characteristic curve (AUC) = 0.934) and the calibration (chi-square test for goodness-of-fit (GOF) = 9.31, P = 0.317) powers of the PELOD-2 score were reported as good.
De Leon et al. (9) assessed the association of the PRISM III score with the outcomes in infants admitted to PICU. A total of 170 infants who were consecutively admitted to PICU were enrolled in their prospective cohort study. With a sensitivity of 0.71 and a specificity of 0.64, the PRISM III was found as a mortality predictor.
To the best of authors’ knowledge, the performance of these predictive models varies and it is still questioned that which scoring model is better in predicting outcomes in critically ill children (10-12).
2. Objectives
It is suggested that regular recalibration of predictive models should be undertaken to provide a well-validated one (13). To the best of authors’ knowledge, no studies thus far assessed the predictive value of PRISM III, PIM3, and PELOD-2 in terms of the outcomes of critically ill children admitted to PICU. The current study aimed at comparing the predictive ability of these three models in medical/ surgical ICUs (MICU or SICU).
3. Methods
The current prospective, observational cohort study was conducted on 90 consecutive pediatric patients admitted to the mixed adults/pediatric university hospital ICUs (separated units) from July 2014 to October 2015. The inclusion criterion was age ≤ 18 years and exclusion criteria were less than 24 hours ICU length of stay (LOS) and diagnosis of brain death on admission.
To find the predictive value of PRISM III, PIM3, and PELOD-2 for mortality, logistic regression test was used. In a two-tailed hypothesis with a predetermined effect size of 0.50, a significant alpha of 0.05, and a statistical power of 0.80, the desired sample size was 76 (14); however, it was increased to 90.
Demographic data (i.e., age and gender) were collected and the PIM3 on admission, the PELOD-2 within the first 24 hours of admission, and the PRISM III during the first eight hours of admission were measured for each subject and recorded by well-trained and qualified nurse. The PRISM III was developed by Pollack et al. in 1996 as a reliable and objective predictive model with 17 physiological variables (1). The PIM3 score provides an international standard based on a big contemporary dataset to compare the risk-adjusted mortality rates among children admitted to ICUs; it consists of eight variables subdivided into 26 ranges (15). The PELODS-2 evaluates dysfunctions in six organs based on 12 variables. The scores were recorded daily and the most abnormal values were used to compute daily PELOD-2 score (5).
All data were recorded initially in a standardized data sheet for PRISM III, PIM3, and PELOD-2 and then, transferred to SPSS version 22 (IBM Corp., Released 2013, IBM SPSS Statistics for Windows, Armonk, NY). The PRISM III, PIM3, and PELOD-2 scores for each patient as well as the relationship between scores and patient's outcomes were assessed. The primary outcomes of the study were attributed to survivors and nonsurvivors. Patients’ information was published anonymous to maintain confidentiality of data. There was no intervention in the study.
Based on the inclusion criteria, the patients who died or diagnosed with brain dead 24 hours after admission were assigned to nonsurvivors group and the ones transferred from MICU or SICU to other wards of the hospitals were considered as survivors. In contrast to the criteria used in previous studies for PIM3, which included only the patients aged < 16 years, the current study included patients aged < 18 years. After encoding data with SPSS, the characteristics of the study population were summarized using simple descriptive statistics. The frequencies and percentages were used for categorical data; the means with standard deviations were used for continuous variables and then, the correlation between PRISM III, PIM3, and PELOD-2 scores, and patients’ outcomes was assessed using logistic regression. Since PRISM III, PIM3, and PELOD-2 scores were independent continuous variables, P value < 0.05 was considered significant. To evaluate the predictive value of the models, standard tests to measure discrimination and calibration powers were performed. The discrimination power of a predictive model reflects the power of distinguishing between survivors and nonsurvivors, which can be obtained by calculating the AUC. An AUC of 0.5 indicates a random chance (a diagonal line); AUC > 0.7 shows a moderate prognostic value, and > 0.8 (a bulbous curve) good prognostic model (16). To generate risk estimates in accordance with the observed outcomes at different classes of risk, model calibration should be assessed; in other words, calibration power of the model represents the agreement between individual probabilities and actual outcomes. Using the Hosmer-Lemeshow goodness of fit (GOF), P value > 0.05 indicates a well-calibrated model (17). Also, the STROBE checklist was used to guide the reporting of the study.
4. Results
Overall, 90 pediatric patients admitted to MICU or SICU were enrolled in the current study. The mean age of the subjects was 7.80 ± 4.43 years (ranged 2 - 18), of which 73.3% (n = 66) were male and 26.7% (n = 24) female. The overall mortality rate was 17.8% (15). The clinical and general characteristics of the study population are shown in Table 1.
| Characteristic | Total (N = 90) | Survivors (N = 74) | Nonsurvivors (N = 16) | P Value |
|---|---|---|---|---|
| Age, y | 7.80 ± 4.43 | 8.57 ± 4.29 | 4.25 ± 3.30 | < 0.001 |
| Medical | 50 | 48 | 2 | |
| Trauma and surgical | 32 | 23 | 9 | |
| Post-resuscitation care | 8 | 3 | 5 | |
| Gender | 0.87 | |||
| Male | 66 (73.33) | 54 (72.97) | 12 (30.77) | |
| Female | 24 (26.67) | 20 (27.03) | 4 (25.48) | |
| ICU length of stay, d | 3.65 ± 3.95 | 3.06 ± 2.25 | 6.38 ± 7.65 | 0.10 |
| PRISM III | 10.88 ± 5.13 | 10.01 ± 4.93 | 14.87 ± 4.11 | < 0.001 |
| PIM3 | 1.97 ± 1.30 | 1.67 ± 1.03 | 3.35 ± 1.55 | < 0.001 |
| PELOD-2 | 9.49 ± 3.65 | 8.81 ± 3.55 | 12.62 ± 2.25 | < 0.001 |
For the entire cohort of patients, PRISM III, PIM3, and PELOD-2 scores were significantly different between the nonsurvivors and survivors. The survivors showed significantly lower values for all the three scores than nonsurvivors (P < 0.001, P < 0.001, and P < 0.001, respectively).
The model performance can be quantified with respect to calibration and discrimination. Calibration power, or reliability, refers to the degree of agreement between predicted probabilities and observed outcome frequencies. The discrimination power refers to the model ability to distinguish patients with different outcomes. The performance of the two models is compared in Table 2.
Comparison of the Studied Predictive Models Between Survivors and Nonsurvivors
| Variable | Cutoff Point | Sensitivity, % | Specificity, % | Positive Predictive Value, % | Negative Predictive Value, % | Accuracy, % | AUC | SE | P Value |
|---|---|---|---|---|---|---|---|---|---|
| PRISM III | 11.5 | 81.3 | 59.5 | 30.23 | 93.62 | 63.33 | 0.773 | 0.060 | 0.001 |
| PIM3 | 2.71 | 68.8 | 87.8 | 55.00 | 92.86 | 84.44 | 0.824 | 0.055 | < 001 |
| PELOD-2 | 10.5 | 87.5 | 66.2 | 35.90 | 96.08 | 70.00 | 0.803 | 0.049 | < 001 |
Analyzing the AUCs showed that the discrimination power of PIM3 and PELOD-2 was good (AUC = 0.824 and 0.803, respectively); this value for PRISM III was moderate. The Youden index (sensitivity + specificity -1) was used to calculate the optimal cutoff point for the three models. By a cutoff point 11.5, PRISM III predicted the mortality risk with a sensitivity of 81%, specificity of 60%, and accuracy of 63%; for PIM3, a cutoff point of 2.71 showed a sensitivity of 69%, specificity of 88%, and accuracy of 84%; for PELOD-2 with a cutoff point of 10.5, sensitivity was 88%, specificity 66%, and accuracy 70% (Table 2).
The Hosmer-Lemeshow chi-square test showed that the GOF of PRISM III, PIM3, and PELOD-2 were good (χ2 = 4.73, P = 0.79; χ2 = 3.09, P = 0.93; and χ2 = 5.01, P = 0.66, respectively). To access the predictive value of the three models, the ROC curve was drawn (Figure 1). To compare the discrimination power of the models, three AUCs were compared based on the model of DeLong et al. (18); the three predictive models had similar accuracies in predicting the prognosis of pediatric patients' outcomes. The discrimination power of PELOD-2 and PIM3 was good, but it was moderate for PRISM III. In terms of gender and ICU LOS, the survivors and nonsurvivors were homogeneous. The survivors were older than nonsurvivors and the difference was statistically significant (P < 0.001) (Table 1).
The ROC curves for pediatric risk of mortality III, pediatric index of mortality III, and pediatric logistic organ dysfunction II on the first 24 hours of admission at medical or surgical intensive care units. The AUCs respectively were 0.773, 0.824, and 0.803.
Under the logistic regression model, each point increase in the PRISM III score was accompanied by 1.25 times increase in the odds of mortality rate in ICU (odds ratio (OR): 1.251, 95% confidence interval (CI): 1.088 - 1.441; P = 0.002); also, each point increase in PIM3 and PELOD-2 scores were respectively associated with 2.86 and 1.41 times increase in the odds of mortality rate (OR: 2.859, 95% CI: 1.646 - 4.965; P < 0.001; OR: 1.412, 95% CI: 1.157 - 1.724; P = 0.001, respectively). The relationships between all the three scores and mortality rate were even unchanged after adjusting for age and gender; thus, these models were significant predictors for pediatric patients’ outcomes in the MICUs/SICUs.
5. Discussion
In the current study, the predictive ability of PRISM III, PIM3, and PELOD-2 models were evaluated in the MICUs/SICUs. The mean scores of PRISM III, PIM3, and PELOD-2 were significantly higher in nonsurvivors compared with survivors (P < 0.001, P < 0.001, and P < 0.001, respectively). Also, analysis of AUCs showed that both PIM3 and PELOD-2 were good at discriminating survivors and nonsurvivors (AUC = 0.824, P < 0.001 and AUC = 0.803, P < 0.001, respectively), and discrimination power of PRISM III was moderate (AUC = 0.773, P = 0.001). The Hosmer-Lemeshow chi-square test showed a good calibration for PRISM III, PIM3, and PELOD-2 scores (χ2 = 4.73, P = 0.79; χ2 = 3.08, P = 0.93; and χ2 = 5.01, P = 0.66, respectively). It indicated the good performance of these three scoring models in the pediatric population. Also, the survivors were significantly older than nonsurvivors (P < 0.001).
The optimal cutoff point based on the Youden index was 11.5, 2.71, and 10.5 for PRISM III, PIM3, and PELOD-2, respectively. The optimal cutoff point for PRISM III was 5.5, 7, 7.5, and 8 in four studies, with a sensitivity range of 68% - 83% and a specificity range of 82% - 87.5% (19-22). The cutoff point reported in other studies were lower than those of the current study, but the sensitivities and specificities were similar to those of the present study. The optimal cutoff point for PIM3 and PELOD-2 was not reported in other studies.
Findings of several studies were consistent with those of the current study indicating that lower scores of PRISM III, PIM3, and PELOD-2 were significantly associated with a higher mortality rate (23-26).
Inconsistent with the current study findings, several studies noted the good performance of the three predictive models in terms of discrimination and calibration power. Goncalves et al. (11) in a study compared PRISM III and PELOD-2 for the prediction of mortality in a PICU in a Portuguese population. A total of 556 patients consecutively admitted to PICU, with the mean age of 65 months (range: one month to 17 years), the male to female ratio of 1:5, and the median PICU LOS of three days were enrolled in their study. Both models had good discrimination; the AUC for the PRISM III was 0.92 and for the PELOD-2 was 0.94. The calibration power was good just for PRISM III (PRISM III: χ2 = 3.820, P = 0.282; PELOD-2: χ2 = 9.576, P = 0.022). Unlike the current study findings, they concluded that the PELOD-2 needs recalibration to be a more reliable predictor.
Straney et al. (15) in an international, multicenter, prospective cohort study evaluated the predictive ability of PIM3 for mortality risk among children admitted to an ICU. Sixty ICUs admitting pediatrics in Australia, New Zealand, Ireland, and the United Kingdom were selected. A total of 53,112 children, under 18 years old, admitted to ICU from 2010 to 2011 were enrolled in their study. Children who transferred to another ICU were not included. The PIM3 model discriminated well (AUC, 0.88, 0.88 - 0.89); however, the performance of the model in Australia and New Zealand was superior to those of the United Kingdom and Ireland (AUC 0.91, 95% CI: 0.90 - 0.93 and 0.85, 95% CI: 0.84 - 0.86, respectively).
In the current study, the age was a predictor of mortality as survivors were significantly older than nonsurvivors (P < 0.001). Unlike to the current study findings, the result of the retrospective study by Campbell et al. (23) on 83 children aged 1 to 18 years showed that the important predictors for mortality were the younger age at the time of injury, higher PRISM III score, and lower GCS score. The overall mortality rate in the present study was 17.8%, which was 3.9% and 28.7% in the studies by Wolfler et al. (27) and Qureshi et al. respectively (12).
By increasing the potential use of the scoring models through education, and standardization of assessments across different ICUs, and customizing an appropriate model, it is hoped that the role of predictive models is maintained in clinical practices and research in the future (28, 29). There are several limitations to the current study that should be addressed in further research: the first was the substantial influence of sample size on model calibration. The second was different contexts (case mix), quality of care, and policies that might cause bias. Ethical considerations were considered in the study.
5.1. Conclusions
The performance of the three predictive models was good. In terms of discrimination power, the performance of PIM3 and PELOD-2 was slightly better than that of the PRISM III. Further recalibration of predictive models in different contexts, multicenter studies, on larger sample sizes would enable the generalizability of the most validated ICU scoring systems.
References
-
1.
Pollack MM, Patel KM, Ruttimann UE. PRISM III: An updated pediatric risk of mortality score. Crit Care Med. 1996;24(5):743-52. [PubMed ID: 8706448].
-
2.
Straney L, Clements A, Parslow RC, Pearson G, Shann F, Alexander J, et al. Paediatric index of mortality 3: An updated model for predicting mortality in pediatric intensive care*. Pediatr Crit Care Med. 2013;14(7):673-81. [PubMed ID: 23863821]. https://doi.org/10.1097/PCC.0b013e31829760cf.
-
3.
Leteurtre S, Duhamel A, Salleron J, Grandbastien B, Lacroix J, Leclerc F, et al. PELOD-2: An update of the pediatric logistic organ dysfunction score. Crit Care Med. 2013;41(7):1761-73. [PubMed ID: 23685639]. https://doi.org/10.1097/CCM.0b013e31828a2bbd.
-
4.
Unertl K, Kottler BM. [Prognostic scores in intensive care]. Anaesthesist. 1997;46(6):471-80. German. [PubMed ID: 9297377]. https://doi.org/10.1007/s001010050426.
-
5.
Gulla KM, Sachdev A. Illness severity and organ dysfunction scoring in Pediatric Intensive Care Unit. Indian J Crit Care Med. 2016;20(1):27-35. [PubMed ID: 26955214]. [PubMed Central ID: PMC4759990]. https://doi.org/10.4103/0972-5229.173685.
-
6.
Ramazani J, Hosseini M. Prediction of ICU mortality in critically ill children: Comparison of SOFA, GCS, and FOUR score. Med Klin Intensivmed Notfmed. 2018. [PubMed ID: 30276565]. https://doi.org/10.1007/s00063-018-0484-0.
-
7.
Stessel B, Fiddelers AA, Marcus MA, van Kuijk SM, Joosten EA, Peters ML, et al. External validation and modification of a predictive model for acute postsurgical pain at home after day surgery. Clin J Pain. 2017;33(5):405-13. [PubMed ID: 27428546]. [PubMed Central ID: PMC5638419]. https://doi.org/10.1097/AJP.0000000000000413.
-
8.
Leteurtre S, Duhamel A, Salleron J, Grandbastien B, Lacroix J, Leclerc F, et al. PELOD-2: An update of the pediatric logistic organ dysfunction score. Crit Care Med. 2013;41(7):1761-73. [PubMed ID: 23685639]. https://doi.org/10.1097/CCM.0b013e31828a2bbd.
-
9.
De Leon AL, Romero-Gutierrez G, Valenzuela CA, Gonzalez-Bravo FE. Simplified PRISM III score and outcome in the pediatric intensive care unit. Pediatr Int. 2005;47(1):80-3. [PubMed ID: 15693872]. https://doi.org/10.1111/j.1442-200x.2004.01997.x.
-
10.
Oto A, Erdogan S, Bosnak M. Oxygen therapy via high flow nasal cannula in pediatric intensive care unit. Turk J Pediatr. 2016;58(4):377-82. [PubMed ID: 28276209].
-
11.
Goncalves JP, Severo M, Rocha C, Jardim J, Mota T, Ribeiro A. Performance of PRISM III and PELOD-2 scores in a pediatric intensive care unit. Eur J Pediatr. 2015;174(10):1305-10. [PubMed ID: 25875250]. https://doi.org/10.1007/s00431-015-2533-5.
-
12.
Qureshi AU, Ali AS, Ahmad TM. Comparison of three prognostic scores (PRISM, PELOD and PIM 2) at pediatric intensive care unit under Pakistani circumstances. J Ayub Med Coll Abbottabad. 2007;19(2):49-53. [PubMed ID: 18183720].
-
13.
Hosseini M, Ramazani J. Evaluation of acute physiology and chronic health evaluation II and sequential organ failure assessment scoring systems for prognostication of outcomes among Intensive Care Unit's patients. Saudi J Anaesth. 2016;10(2):168-73. [PubMed ID: 27051367]. [PubMed Central ID: PMC4799608]. https://doi.org/10.4103/1658-354X.168817.
-
14.
Cohen J. A power primer. Psychol Bull. 1992;112(1):155-9. [PubMed ID: 19565683].
-
15.
Straney L, Clements A, Parslow RC, Pearson G, Shann F, Alexander J, et al. Paediatric index of mortality 3: An updated model for predicting mortality in pediatric intensive care*. Pediatr Crit Care Med. 2013;14(7):673-81. [PubMed ID: 23863821]. https://doi.org/10.1097/PCC.0b013e31829760cf.
-
16.
Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8(4):283-98. [PubMed ID: 112681].
-
17.
Hosseini M, Ramazani J. Comparison of acute physiology and chronic health evaluation II and Glasgow Coma Score in predicting the outcomes of Post Anesthesia Care Unit's patients. Saudi J Anaesth. 2015;9(2):136-41. [PubMed ID: 25829900]. [PubMed Central ID: PMC4374217]. https://doi.org/10.4103/1658-354X.152839.
-
18.
DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics. 1988;44(3):837-45. [PubMed ID: 3203132].
-
19.
Li Y, Wang J, Bai Z, Chen J, Wang X, Pan J, et al. Early fluid overload is associated with acute kidney injury and PICU mortality in critically ill children. Eur J Pediatr. 2016;175(1):39-48. [PubMed ID: 26206387]. https://doi.org/10.1007/s00431-015-2592-7.
-
20.
Ibrahim Rady H. Prediction of stress related gastrointestinal bleeding in critically III children using PRISM III score. J Anesth Crit Care. 2014;1(4). https://doi.org/10.15406/jaccoa.2014.01.00023.
-
21.
Bai Z, Zhu X, Li M, Hua J, Li Y, Pan J, et al. Effectiveness of predicting in-hospital mortality in critically ill children by assessing blood lactate levels at admission. BMC Pediatr. 2014;14:83. [PubMed ID: 24673817]. [PubMed Central ID: PMC3976355]. https://doi.org/10.1186/1471-2431-14-83.
-
22.
Tan GH, Tan TH, Goh DY, Yap HK. Risk factors for predicting mortality in a paediatric intensive care unit. Ann Acad Med Singapore. 1998;27(6):813-8. [PubMed ID: 10101556].
-
23.
Campbell CG, Kuehn SM, Richards PM, Ventureyra E, Hutchison JS. Medical and cognitive outcome in children with traumatic brain injury. Can J Neurol Sci. 2004;31(2):213-9. [PubMed ID: 15198446].
-
24.
Leteurtre S, Duhamel A, Deken V, Lacroix J, Leclerc F; Groupe Francophone de Réanimation et Urgences Pédiatriques. Daily estimation of the severity of organ dysfunctions in critically ill children by using the PELOD-2 score. Crit Care. 2015;19:324. [PubMed ID: 26369662]. [PubMed Central ID: PMC4570178]. https://doi.org/10.1186/s13054-015-1054-y.
-
25.
Lee OJ, Jung M, Kim M, Yang HK, Cho J. Validation of the pediatric index of mortality 3 in a single pediatric intensive care unit in Korea. J Korean Med Sci. 2017;32(2):365-70. [PubMed ID: 28049251]. [PubMed Central ID: PMC5220006]. https://doi.org/10.3346/jkms.2017.32.2.365.
-
26.
Choi KM, Ng DK, Wong SF, Kwok KL, Chow PY, Chan CH, et al. Assessment of the pediatric index of mortality (PIM) and the pediatric risk of mortality (PRISM) III score for prediction of mortality in a paediatric intensive care unit in Hong Kong. Hong Kong Med J. 2005;11(2):97-103. [PubMed ID: 15815062].
-
27.
Wolfler A, Osello R, Gualino J, Calderini E, Vigna G, Santuz P, et al. The importance of mortality risk assessment: Validation of the pediatric index of mortality 3 score. Pediatr Crit Care Med. 2016;17(3):251-6. [PubMed ID: 26825046]. https://doi.org/10.1097/PCC.0000000000000657.
-
28.
Teasdale G, Maas A, Lecky F, Manley G, Stocchetti N, Murray G. The Glasgow Coma Scale at 40 years: Standing the test of time. Lancet Neurol. 2014;13(8):844-54. [PubMed ID: 25030516]. https://doi.org/10.1016/S1474-4422(14)70120-6.
-
29.
Reith FC, Van den Brande R, Synnot A, Gruen R, Maas AI. The reliability of the Glasgow Coma Scale: A systematic review. Intensive Care Med. 2016;42(1):3-15. [PubMed ID: 26564211]. https://doi.org/10.1007/s00134-015-4124-3.