https://doi.org/10.5812/ans.89993

Mortality Related to Intubation in Adult General ICUs: A Systematic Review and Meta-Analysis

authors:

avatar Zahid Hussain Khan ORCID 1 , avatar Mohammed AbdulZahra Sasaa 2 , avatar Mostafa Mohammadi ORCID 3 , * , avatar Abbas Alipour ORCID 4 , avatar Asghar Hajipour 1

Anesthesiology Department, Tehran University of Medical Sciences, Tehran, Iran
Allied Medical School, International Campus, Tehran University of Medical Sciences, Tehran, Iran
Anesthesiology and Intensive Care Department, Tehran University of Medical Sciences, Tehran, Iran
Epidemiology Department, Mazandaran University of Medical Sciences, Sari, Iran
Archives of Neuroscience: Vol.7, issue 3; e89993
published online: July 14, 2020
article type: Systematic Review
received: October 23, 2019
revised: April 5, 2020
accepted: May 2, 2020

how to cite: Hussain Khan Z , Sasaa M A, Mohammadi M , Alipour A , Hajipour A . Mortality Related to Intubation in Adult General ICUs: A Systematic Review and Meta-Analysis. Arch Neurosci. 2020;7(3):e89993. https://doi.org/10.5812/ans.89993.

Abstract

Context:

Mortality related to intubation occurs as a result of multiple factors such as patient's condition, operator's skills, equipment use, intubation time, duration of laryngoscopy and intubation, and drugs and dosage used for endotracheal intubation (ETI).

Objectives:

This systematic review and meta-analysis aimed to determine mortality related to intubation and the overall intensive care unit (ICU) mortality rate in adult general ICUs.

Methods:

We performed a systematic review and meta-analysis on randomized clinical trials and cohort and cross-sectional research from three electronic databases with hand searching. The studies reported mortality related to intubation and the overall ICU mortality rate in adult general ICUs. Our search resulted in 28 published articles without any restriction on date and language. The systematic review and meta-analysis was performed to examine mortality related to intubation and the overall ICU mortality rate.

Results:

We found 7,866 articles in the literature review from the three databases based on our keywords, of which 28 studies were eligible to include in the study. We observed that mortality related to intubation and the overall ICU mortality rate in intubated patients were 1% and 30%, respectively.

Conclusions:

This was the first comprehensive systematic review on mortality related to intubation and the overall ICU mortality rate in adult general ICUs, which showed the current care of ETI. However, it was associated with increased complications, which may increase mortality.

1. Context

Endotracheal intubation (ETI) is an indispensable and common procedure in intensive care units (ICUs). It protects the patient's airway or provides invasive mechanical ventilation to treat respiratory failure as a common problem in critically ill patients (1-7), which is correlated with numerous complications including mortality. This procedure will be performed in different locations for critically or non-critically ill patients. For example, tracheal intubation is performed before surgery in the operating room (OR). Orotracheal intubation is usually performed electively in OR in a comfortable condition. However, it needs to be immediately performed in ICU during emergency conditions under troubling circumstances to prevent aspiration because patients are generally not in a fasting situation (8). Some methods are used for tracheal intubation in critically ill patients such as fiber-optic, video laryngoscopy, and direct laryngoscopy (DL). However, the standard of care and the most common method is DL with Macintosh blade, and rarely, Miller blade. DL has a high success rate in expert hands; however, it decreases the chance of first successful intubation in emergency conditions with a poor patient's condition and a non-expert operator. It also causes frequent intubation attempts due to an excessive danger of complications and a low operator's experience level (3, 8, 9).

Different pharmacologic agents such as propofol, ketamine, etomidate, benzodiazepine, and opioids, as well as neuromuscular blocking agents are used to provide perfect intubation conditions. However, these drugs are associated with significant complications in hemodynamically unstable ICU patients, especially when used by those with low training and experience levels (10). Some serious complications might occur due to intubation, which may be related to patients, operators, instruments, drugs, and emergency conditions (11, 12).

2. Objectives

Few studies have been conducted on mortality related to intubation in general ICU patients and the management of these extremely vulnerable patients. Thus, we carried out a systematic review and meta-analysis about adult general ICUs. The main objective of our study was to determine mortality related to intubation in adult general ICUs.

3. Data Sources

Four dependent search strategies were used in this systematic review and meta-analysis: (1) data sources, (2) study selection, (3) data extraction, and (4) statistical evaluation and quality assessment.

Studies on mortality related to intubation and the overall ICU mortality rate due to ETI published up to the end of 2018 were included. Mortality related to intubation is considered as mortality during and 60 minutes after intubation, and the overall ICU mortality rate is defined as the mortality of intubated patients during admission and the period in the general ICU. PubMed, Embase, and Google Scholar were systematically reviewed using appropriate textual content phrases and dictionary expressions for articles related to the following terms: any phrases including mortality rate and their synonyms or acronyms, intensive care unit and its synonyms, intubation and its synonyms or acronyms, and age above 18 years. However, there was no limitation on date, language, and geographical sites for selecting articles. Searches and evaluations were undertaken in the reference lists of the selected articles by hand searching. All citations were downloaded into EndNote.

4. Study Selection

Studies with narrative and descriptive assessment, studies with unknown case definitions, studies in pediatric and obstetric ICUs, at the emergency department (ED), OR, inwards, and outside hospitals, and studies mixing different departments without separated results or data were eliminated from the study. Potentially applicable articles were recognized according to the above-mentioned inclusion and exclusion criteria, and replications were omitted. The remaining articles were assessed for enrollment, first based on title and then based on abstract, before downloading their full texts. The subjects were arranged based on the following criteria: data and type of study; authors; year of publication; participants’ age range; age standard deviation (age SD); the male/female percentage; quality assessment; sample size, and number of events.

5. Data Extraction

Data were extracted from studies with the exact rate of mortality related to intubation immediately after the procedure until 60 minutes in elective or emergency intubation and difficult intubation, specifically in adult general ICUs. The studies used usual care, direct laryngoscope for ETI, and simple face masks for preoxygenation without using video laryngoscope or fibro optic, high flow nasal cannula (HFNC), and non-invasive positive pressure ventilation (NIPPV), and also, without loading fluid or using vasopressor prior to intubation. The main focus of the studies was on mortality related to intubation in general ICUs.

Afterward, we excluded articles with a quality assessment score of less than 0.6. Finally, 28 articles were included in the study, mentioning the exact number of death cases related to intubation and the overall ICU mortality rate. After three (four if necessary) experts on epidemiology independently assessed the full texts of the selected articles, statistics were entered into an excel database, specifically designed for this review, and analyzed using STATA version 14.1. Figure 1 shows a diagram describing the study design process.

The diagram of the study design process
The diagram of the study design process
Figure 1.

The diagram of the study design process

5.1. Data Analysis and Quality Assessment

We used a metaprop command using STATA version 12.1 (StataCorp, College Station, TX, USA) for pooling proportions. We estimated 95% confidence intervals using the score statistic and the exact binomial method and Freeman-Tukey double arcsine transformation of proportions via ftt option. The heterogeneity of the prevalence estimates between the studies was decided using the I2 index. A random influence model was used unless the index was < 25%; In that case, a constant outcome model would be used (10). The generation of a funnel plot and the Egger and Begg P value permitted the determination of potential publication bias in the selected studies. The influence of the study characteristics (such as age, date, and quality level) was explored using meta-regression. Moreover, the methodology was assessed according to the STROBE checklist for the cohort and cross-sectional studies and according to Jaded tools for the randomized clinical trials. Each study with a 0.6 score higher than the maximum score was considered as a high-quality study.

6. Results

Of the 7,866 studies obtained from the three databases, 28 were eligible to be included in this systematic review and meta-analysis. Thirteen articles notified results about the rate of mortality related to intubation in general ICUs. The studies included in this meta-analysis were five cross-sectional studies, five cohort studies, and three randomized clinical trials. After the exclusion of two articles with a low-quality score (below 0.6), of 4754 individuals in the studies, 34 cases were dead, with the pooled estimate of the mortality rate being 1% (95% CI: 0% - 1%) (Figure 2). Data analysis confirmed that there was no meaningful relationship between the mortality rate with age (P = 0.27) and year of study (P = 0.88), although it was marginally significantly correlated with the quality score (P = 0. 09). The evidence of publication bias was provided by a funnel plat. The egger test (P = 0.3%) and the Begg test (P = 0.4%) were statistically significant for publication bias. Fifteen articles reported results about the rate of mortality related to intubation in general ICUs. The articles included in this meta-analysis were two cross-sectional studies, 11 cohort studies, and two randomized clinical trials. After the exclusion of two studies with a low-quality score (below 0.6), of 2054 individuals in the studies, 604 cases were dead, with the pooled estimate of the mortality rate being 30% (95% CI: 23% - 36%) (Figure 3). Data analysis showed that there was no significant relationship between the ICU mortality rate with age (P = 0.89), year of study (P = 0.97), and the quality score (P = 0.18). The evidence of publication bias was provided by a funnel plat. The egger test (P = 11%) and the Begg test (P = 2%) were not statistically significant for publication bias.

The overall rate of mortality related to intubation in adult general ICUs
The overall rate of mortality related to intubation in adult general ICUs
Figure 2.

The overall rate of mortality related to intubation in adult general ICUs

Mortality related to intubation in adults general ICUs
Mortality related to intubation in adults general ICUs
Figure 3.

Mortality related to intubation in adults general ICUs

7. Discussion

Tracheal intubation is a routine and lifesaving technique in ICUs. Life threatening complications may occur in a significant number of intubations, but those complication are less frequent when intubation were done after complete preparation (2-4, 6, 11). The present study was the first systematic review and meta-analysis that evaluated the rate of mortality related to intubation, specifically in general ICUs.

This study described mortality related to intubation occurring during or 60 minutes after the procedure. Of the 4754 intubations performed, 34 cases died (1%). The ICU mortality of intubated patients was defined as the death of intubated patients during ICU stay. Of 2054 individuals in studies selected for the purpose of the current review, 604 cases died (30%). Petrini et al. and Ma et al. (12, 13) reported that the rate of mortality during and 60 minutes after intubation was 3%, which is higher than that in our study. Schwartz et al.'s study (6) showed the rate of mortality related to intubation to be 3%, which is higher than that in the current review. Schwartz's study was not included in our review because it reported the mortality rate outside ICUs.

In our review, 13 articles mentioned the rate of mortality related to intubation to range between 0% - 4%. Moreover, the reported mortality rate completely differed in three of the articles compared to the others. Accordingly, the rate was 4% in Audrey et al.’s study, 3% in Ma et al.’s study, and 2.8% in Semler et al.’s study. Other studies reported that the mortality rate 30 - 60 minutes after intubation ranged between 0% - 1% (14-16).

The most popular complication of tracheal intubation in ICU is hypoxemia (4, 17), which is a potent risk factor for periprocedural cardiac arrest and death (5). Semler et al. (14) reported that the mortality rate was lower in apneic oxygenation via high-flow nasal cannula oxygen preoxygenation during larngoscope (0%), as compared with usual care (2.8%).

Audrey et al. (18), in their study, found that the mortality rate was high in post-intubation cardiac arrest (29%) compared to intubation without cardiac arrest (0%). Post intubation cardiac arrest is more frequent during emergency situations and may increase death risk due to time shortage to perform oxygenation and stabilize hemodynamics. Hypoxemia, hemodynamic failure before intubation, lack of oxygenation, and age above 75 years are potential high-risk factors that increase cardiac arrest and death related to intubation.

Acute respiratory failure occurs due to dramatic changes in the lung and high shunt fraction, which may lead to disturbance in gas exchange. Patients with acute respiratory distress syndrome are at a higher risk of hypoxia. Thus, the intubation of these patients should be performed with adequate preparation (3, 19, 20).

Baillard et al. (2) realized that the use of non-invasive ventilation (NIV) for pre-oxygenation in general ICUs led to a lower overall ICU mortality rate (30%), as compared with usual care (50%). They also mentioned that decreased SPO2 less than 80% was higher with usual care (46%) compared with NIV (7%).

Audrey et al. (21) reported that the incidence of death was a little higher in difficult intubation (0.5%) compared with non-difficult intubation (0%). Moreover, De Jong et al. (22) reported that the incidence of death was high with difficult intubation (4%) compared with non-difficult intubation (0%).

Emergency ETI in ICU is a complex process with numerous possible failure points. Complications of the emergency process mostly include acute hypoxemia, hypotension, or death. Critically ill patients undergoing emergent intubation have a higher than expected rate of post-intubation hypotension. Green et al. (23) reported that ICU mortality was higher with post-intubation hypotension (37%) compared with non-hypotension after intubation (28%).

DL intubation was associated with a higher rate of mortality (3%) compared with Bronchoscopic intubation (0%) used by Petrini et al. (12). Further, Griesdale et al. (4) reported that the use of DL was associated with a higher rate of overall ICU mortality (25%) compared with video laryngoscope (VL) (15%). In addition, The ICU mortality rate was a little higher in primary intubation (35.5%) compared with failed NIPPV (29.5%) in the study of Mosier and colleagues (24).

The use of intravenous anesthetic drugs and neuromuscular block drugs is associated with lower mortality. However, a majority of ICU patients are hemodynamically unstable, which may increase the rate of complications and mortality following the administration of anesthetic, hypnotic, and analgesic drugs. Mort et al. reported that the hemodynamic effects of pharmacologic agents used for induction in emergency ETI in ICU patients were common. Wilcox et al. (25) showed that the use of muscle relaxant drugs caused a lower prevalence of hypoxemia and procedure-related complications and improved tracheal intubation conditions. They also reported that the use of muscle relaxant drugs diminished the procedure-related complications when used with sufficient training and experience.

Audrey et al. divided their patients into two groups with easy and difficult intubation. They reported that the mortality rate was 0% in the first group and 4% in the second group (the difficult intubation group). Thus, one of the most significant reasons for a higher rate of mortality related to intubation is difficult intubation.

Moreover, Ma et al. had two groups in their study. The mortality rate was 0% in the first group (intubation by bronchoscopy), while it was 3% in the second group (DL intubation), showing the low rate of first successful intubation in case of using DL.

Semler et al. mentioned that the mortality rate was 2.8% higher while supplementing oxygen during laryngoscopy compared with non-supplementation (0%).

A higher overall ICU mortality related to intubation was observed in three articles, 50% mortality was referred using preoxygenation three minutes prior to intubation with non-rebreather bag-valve masks, compared with 30% in NIV group, Christophe Baillard. Schmidt et al. reported 47% mortality in ICUs that manage with non- attending supervision (26). Peter Luedike et al. mentioned 43% mortality related to intubation failure by non-anesthesiologist ICU trainees (27).The standard of care for ETI is used in ICUs, but it is associated with many complications that increase mortality. Moreover, intensive care specialists believe that additional procedures added to the standard of care will improve the outcome and decrease the rate of post-intubation mortality.

Insertion of ETI in ICUs is often decisive, and in addition to considering airway management and equipment, we should also take into account hemodynamic stability, gas exchange, neurologic protection, and drug use.

7.1. Limitations

Our study has some limitations. Different protocols for intubation were used in different ICUs, and the patients’ conditions were not clearly defined, or even, were not mentioned in some of the papers. Like with all previous meta-analyses, the presented review was subject to information bias. The second limitation of the study was related to the accuracy of data summation as the experts had possibly different skills or training levels.

7.2. Conclusions

Mortality related to intubation in adult general ICUs ranged between 0% - 4%. There was no correlation between the mortality rate with age and year of publication. Difficult intubation, use of direct laryngoscopy, lack of preoxygenation or oxygen therapy during intubation, and attending or non-attending physician may be involved in mortality related to intubation, and thus, should be taken into account while performing tracheal intubation in ICU patients.

References

  • 1.

    Roux D, Reignier J, Thiery G, Boyer A, Hayon J, Souweine B, et al. Acquiring procedural skills in ICUs: a prospective multicenter study*. Crit Care Med. 2014;42(4):886-95. [PubMed ID: 24275511]. https://doi.org/10.1097/CCM.0000000000000049.

  • 2.

    Baillard C, Fosse JP, Sebbane M, Chanques G, Vincent F, Courouble P, et al. Noninvasive ventilation improves preoxygenation before intubation of hypoxic patients. Am J Respir Crit Care Med. 2006;174(2):171-7. [PubMed ID: 16627862]. https://doi.org/10.1164/rccm.200509-1507OC.

  • 3.

    Jaber S, Amraoui J, Lefrant JY, Arich C, Cohendy R, Landreau L, et al. Clinical practice and risk factors for immediate complications of endotracheal intubation in the intensive care unit: a prospective, multiple-center study. Crit Care Med. 2006;34(9):2355-61. [PubMed ID: 16850003]. https://doi.org/10.1097/01.CCM.0000233879.58720.87.

  • 4.

    Griesdale DE, Bosma TL, Kurth T, Isac G, Chittock DR. Complications of endotracheal intubation in the critically ill. Intensive Care Med. 2008;34(10):1835-42. [PubMed ID: 18604519]. https://doi.org/10.1007/s00134-008-1205-6.

  • 5.

    Adnet F, Borron SW, Racine SX, Clemessy JL, Fournier JL, Plaisance P, et al. The intubation difficulty scale (IDS): proposal and evaluation of a new score characterizing the complexity of endotracheal intubation. Anesthesiology. 1997;87(6):1290-7. [PubMed ID: 9416711]. https://doi.org/10.1097/00000542-199712000-00005.

  • 6.

    Schwartz DE, Matthay MA, Cohen NH. Death and other complications of emergency airway management in critically ill adults. A prospective investigation of 297 tracheal intubations. Anesthesiology. 1995;82(2):367-76. [PubMed ID: 7856895]. https://doi.org/10.1097/00000542-199502000-00007.

  • 7.

    Jaber S, Jung B, Corne P, Sebbane M, Muller L, Chanques G, et al. An intervention to decrease complications related to endotracheal intubation in the intensive care unit: a prospective, multiple-center study. Intensive Care Med. 2010;36(2):248-55. [PubMed ID: 19921148]. https://doi.org/10.1007/s00134-009-1717-8.

  • 8.

    Healy DW, Maties O, Hovord D, Kheterpal S. A systematic review of the role of videolaryngoscopy in successful orotracheal intubation. BMC Anesthesiol. 2012;12:32. [PubMed ID: 23241277]. [PubMed Central ID: PMC3562270]. https://doi.org/10.1186/1471-2253-12-32.

  • 9.

    Griesdale DE, Liu D, McKinney J, Choi PT. Glidescope(R) video-laryngoscopy versus direct laryngoscopy for endotracheal intubation: a systematic review and meta-analysis. Can J Anaesth. 2012;59(1):41-52. [PubMed ID: 22042705]. [PubMed Central ID: PMC3246588]. https://doi.org/10.1007/s12630-011-9620-5.

  • 10.

    Ried K. Interpreting and understanding meta-analysis graphs--a practical guide. Aust Fam Physician. 2006;35(8):635-8. [PubMed ID: 16894442].

  • 11.

    Petrini F, Accorsi A, Adrario E, Agro F, Amicucci G, Antonelli M, et al. Recommendations for airway control and difficult airway management. Minerva Anestesiol. 2005;71(11):617-57. [PubMed ID: 16278626].

  • 12.

    Ma KC, Chung A, Aronson KI, Krishnan JK, Barjaktarevic IZ, Berlin DA, et al. Bronchoscopic intubation is an effective airway strategy in critically ill patients. J Crit Care. 2017;38:92-6. [PubMed ID: 27875775]. https://doi.org/10.1016/j.jcrc.2016.10.022.

  • 13.

    Semler MW, Janz DR, Lentz RJ, Matthews DT, Norman BC, Assad TR, et al. Randomized Trial of Apneic Oxygenation during Endotracheal Intubation of the Critically Ill. Am J Respir Crit Care Med. 2016;193(3):273-80. [PubMed ID: 26426458]. [PubMed Central ID: PMC4803060]. https://doi.org/10.1164/rccm.201507-1294OC.

  • 14.

    Lakticova V, Koenig SJ, Narasimhan M, Mayo PH. Video laryngoscopy is associated with increased first pass success and decreased rate of esophageal intubations during urgent endotracheal intubation in a medical intensive care unit when compared to direct laryngoscopy. J Intensive Care Med. 2015;30(1):44-8. [PubMed ID: 23771876]. https://doi.org/10.1177/0885066613492641.

  • 15.

    Koenig SJ, Lakticova V, Narasimhan M, Doelken P, Mayo PH. Safety of Propofol as an Induction Agent for Urgent Endotracheal Intubation in the Medical Intensive Care Unit. J Intensive Care Med. 2015;30(8):499-504. [PubMed ID: 24536033]. https://doi.org/10.1177/0885066614523100.

  • 16.

    Badia M, Montserrat N, Servia L, Baeza I, Bello G, Vilanova J, et al. [Severe complications of orotracheal intubation in the Intensive Care Unit: An observational study and analysis of risk factors]. Med Intensiva. 2015;39(1):26-33. [PubMed ID: 24612759]. https://doi.org/10.1016/j.medin.2014.01.003.

  • 17.

    Simpson GD, Ross MJ, McKeown DW, Ray DC. Tracheal intubation in the critically ill: a multi-centre national study of practice and complications. Br J Anaesth. 2012;108(5):792-9. [PubMed ID: 22315326]. https://doi.org/10.1093/bja/aer504.

  • 18.

    De Jong A, Rolle A, Molinari N, Paugam-Burtz C, Constantin JM, Lefrant JY, et al. Cardiac Arrest and Mortality Related to Intubation Procedure in Critically Ill Adult Patients: A Multicenter Cohort Study. Crit Care Med. 2018;46(4):532-9. [PubMed ID: 29261566]. https://doi.org/10.1097/CCM.0000000000002925.

  • 19.

    Mort TC. Emergency tracheal intubation: complications associated with repeated laryngoscopic attempts. Anesth Analg. 2004;99(2):607-13. table of contents. [PubMed ID: 15271750]. https://doi.org/10.1213/01.ANE.0000122825.04923.15.

  • 20.

    Davis DP, Dunford JV, Poste JC, Ochs M, Holbrook T, Fortlage D, et al. The impact of hypoxia and hyperventilation on outcome after paramedic rapid sequence intubation of severely head-injured patients. J Trauma. 2004;57(1):1-8. discussion 8-10. [PubMed ID: 15284540]. https://doi.org/10.1097/01.ta.0000135503.71684.c8.

  • 21.

    De Jong A, Molinari N, Terzi N, Mongardon N, Arnal JM, Guitton C, et al. Early identification of patients at risk for difficult intubation in the intensive care unit: development and validation of the MACOCHA score in a multicenter cohort study. Am J Respir Crit Care Med. 2013;187(8):832-9. [PubMed ID: 23348979]. https://doi.org/10.1164/rccm.201210-1851OC.

  • 22.

    De Jong A, Molinari N, Pouzeratte Y, Verzilli D, Chanques G, Jung B, et al. Difficult intubation in obese patients: incidence, risk factors, and complications in the operating theatre and in intensive care units. Br J Anaesth. 2015;114(2):297-306. [PubMed ID: 25431308]. https://doi.org/10.1093/bja/aeu373.

  • 23.

    Green RS, Turgeon AF, McIntyre LA, Fox-Robichaud AE, Fergusson DA, Doucette S, et al. Postintubation hypotension in intensive care unit patients: A multicenter cohort study. J Crit Care. 2015;30(5):1055-60. [PubMed ID: 26117220]. https://doi.org/10.1016/j.jcrc.2015.06.007.

  • 24.

    Mosier JM, Sakles JC, Whitmore SP, Hypes CD, Hallett DK, Hawbaker KE, et al. Failed noninvasive positive-pressure ventilation is associated with an increased risk of intubation-related complications. Ann Intensive Care. 2015;5:4. [PubMed ID: 25852964]. [PubMed Central ID: PMC4385202]. https://doi.org/10.1186/s13613-015-0044-1.

  • 25.

    Smischney NJ, Seisa MO, Heise KJ, Wiegand RA, Busack KD, Deangelis JL, et al. Predictors of hemodynamic derangement during intubation in the critically ill: A nested case-control study of hemodynamic management-Part II. J Crit Care. 2018;44:179-84. [PubMed ID: 29132057]. https://doi.org/10.1016/j.jcrc.2017.10.018.

  • 26.

    Schmidt UH, Kumwilaisak K, Bittner E, George E, Hess D. Effects of supervision by attending anesthesiologists on complications of emergency tracheal intubation. Anesthesiology. 2008;109(6):973-7. [PubMed ID: 19034093]. https://doi.org/10.1097/ALN.0b013e31818ddb90.

  • 27.

    Luedike P, Totzeck M, Rammos C, Kindgen-Milles D, Kelm M, Rassaf T. The MACOCHA score is feasible to predict intubation failure of nonanesthesiologist intensive care unit trainees. J Crit Care. 2015;30(5):876-80. [PubMed ID: 26002431]. https://doi.org/10.1016/j.jcrc.2015.04.118.