1. Background
Cardiopulmonary arrest is one of the primary and essential reasons for urgent care in the emergency department (1, 2). Every year, about 700,000 people suffer from cardiopulmonary arrest in Europe and the United States (3). Cardiovascular disease is Iran’s leading cause of death, accounting for 46% of all deaths and 20 - 23% of the disease burden (4).
When CPR is performed correctly, quickly, and on time, the patient's chance of survival doubles to three times (5). According to the American Heart Association report, starting CPR in the first 3 - 5 minutes after cardiopulmonary arrest and increasing the chances of survival reduces complications after cardiopulmonary arrest (6, 7). Correct CPR prevents about 90% of cardiac deaths in the emergency room (8). Studies in Iran have reported that the rate of standard CPR is equal to 30 - 50% (4).
When CPR is performed incorrectly, the chance of patients being discharged alive from the hospital is less than 7%, and this issue causes negative consequences for patients and society (9, 10). However, when CPR is principled and standard, it saves patients from death (11). The treatment staff's need for more knowledge and awareness regarding the CPR process is one of the most important causes of CPR failure (12). For this reason, in-service training courses are vital in improving nurses’ knowledge and clinical skills regarding the CPR process (13). Therefore, the CPR training course is the most critical in-service training program for nursing (14). Published reports have shown that the rate of successful CPR in Iran is lower than the standard (15). This problem shows that training alone is not enough, and it is essential to check the effectiveness of training (16).
Educational evaluation is one of the most important programs of any organization, which provides good information on the design and revision of each system to the executors of each training course (17). Effectiveness is achieved when there is a change in the learner's performance in the work environment and the transfer of their learning to the real environment (14). Evaluating educational effectiveness means determining to what extent the training leads to the creation of skills the organization needs in a practical way (18).
There are many models and patterns to determine the effectiveness of training courses, but Kirkpatrick's evaluation model is one of the most important (19, 20). This model evaluates in-service training in four levels: Reaction, learning, performance, and finding (19, 21). At the first level, the reaction is the minimum expectation from a training program. (16, 22). The second level of learning involves assessing learners' mastery of educational goals and pre-and post-tests (18, 23). The third level is performance, which is the learners' use of training in learning environments (19, 22, 24). The finding of the fourth level is the degree of realization of educational goals (25). At this level, the effectiveness of educational programs is evaluated in terms of reducing mortality and infection and increasing patient satisfaction, production, and productivity (22).
However, the evaluation is simple at the Kirkpatrick model's first and second levels in most educational courses in Iran. Few educational programs are comprehensively evaluated this subject. In addition, the educational effects decrease as we approach the third and fourth levels of the model (18). Considering the importance of CPR in reducing death, the CPR training program for medical staff, especially nurses, is regarded as one of the essential job skills (26). The complete mastery of nurses and other workers in correctly applying CPR skills brings patients back to life (14). In addition, the Kirkpatrick model is a comprehensive, simple, and practical model for evaluating clinical courses (15).
2. Objectives
Based on the Kirkpatrick model, this study evaluated the effectiveness of an in-service training course (ISTC) on adult cardiopulmonary resuscitation (CPR).
3. Methods
3.1. Study Design
This cross-sectional descriptive study was conducted on 50 emergency department nursing staff of 5th Azar Hospital in Gorgan, Iran, in 2022. The inclusion criteria were a bachelor’s degree in nursing, at least one year of work experience, and not taking sick leave in the six months before and after participating in the desired training course. The title of the in-service training course was based on the latest cardiopulmonary resuscitation guidelines of the American Heart Association (27) and the new guidelines of the Iranian Ministry of Health (24-26, 28). The content presented in the course included the topics of dealing with cardiac arrest patients, basic resuscitation (BLS: Basic life support), advanced resuscitation (ACLS: Advanced cardiac life support), working with an electroshock device, and post-resuscitation care (29).
The training course content was approved by the hospital's emergency medicine specialist, cardiologist, and educational supervisor with the scientific degree of a master's degree in intensive care nursing and a master's degree in internal surgical nursing. Then, the study participants were trained in two groups of 25 people for two days in the implementation phase of this training. A teacher of the training course explained the purpose of the training course before it was implemented to study participants. In addition, the training course teacher emphasized that the pre-test and post-test scores and evaluations aim only to empower the personnel, and all evaluations remain confidential.
3.2. Implementation of the Kirkpatrick Model
The Kirkpatrick model evaluated the training course in four levels: Reaction, learning, performance, and findings. In the first level (reaction), a 12-question questionnaire was used to evaluate the level of the learners to the program, its enjoyment, and the satisfaction level of the participants, which included several factors related to the content, instructor, and facilities, respectively. The participants completed this questionnaire at the end of each course for each group. This questionnaire had a 5-item Likert scale, where "1" was assigned to the "very poor" option and "5" was assigned to the "very good" option. The final score was divided by the total number of questions, and the score for each person was reported as a number between 1 and 5. Finally, the score obtained for each person was calculated based on the number 100 to equalize the scores throughout the research, and a weighting factor equal to "1" was considered for this level.
The second level (learning) was conducted to determine the level of skill and techniques learned during the training course. A questionnaire of 20 questions was designed based on the title and objectives of the training course to evaluate the second level. The content validity of the questionnaire was confirmed using the "content validity index" and "content validity ratio" by three experts in nursing education and two experts in emergency medicine. The value of the "content validity index" and "content validity ratio coefficient" was obtained as 0.8 and 0.99, respectively, which indicates a high level of experts' agreement. The reliability of this questionnaire was evaluated through Cronbach's alpha, and the alpha level for the entire questionnaire was 0.85. Finally, the score obtained for each individual was calculated based on the number 100 to equalize the scores throughout the research, and a weighting factor equal to "2" was considered for this level.
The third level (performance) was conducted to evaluate nurses' behavioral and functional changes. At this stage, the behavioral and functional changes of the nurses in the real and natural environment were reviewed by the supervisors and management of the resuscitation team using a questionnaire. This questionnaire included 14 two-choice questions ("yes" and "no"). The validity of this tool was confirmed by the content method, and its reliability was confirmed by Cronbach's alpha method, the value of which coefficient was equal to 0.78. Finally, the score obtained for each person was calculated based on the number 100 to equalize the scores throughout the research, and a weighting factor equal to "3" was considered for this level.
The fourth level (finding) can be evaluated in four areas: Cost avoidance, saving, benefit, and strategic findings to achieve training findings. This study used a questionnaire to determine the strategic effects so stakeholders could assess how well the training course achieved the intended findings. The officials of the resuscitation team in the hospital completed this questionnaire. This questionnaire included seven five-choice questions that educational supervisors and clinical supervisors completed. The content validity of the questionnaire was confirmed using the "content validity index" and "content validity ratio" by three experts in nursing education and two experts in emergency medicine. The reliability of this questionnaire was evaluated by determining Cronbach's alpha, and the alpha level for the entire questionnaire was 0.75. At this level of the model, the CPR indicators in the six months before the training course were compared with the six months after it to evaluate the effectiveness of the training course.
3.3. Statistical Analysis
The obtained findings were analyzed using SPSS version 21 statistical software using descriptive statistics (including frequency, mean and standard deviation) and inferential statistics (including Paired Sample t-test and Independent Samples t-test) in a significance level of α = 0.05.
4. Results
The findings of the first level of the Kirkpatrick model (reaction) showed that the satisfaction level of the research participants from the CPR training course was equal to 57.88 ± 3.66, and this satisfaction level based on the number "100" was equivalent to 46 ± 6.11 96 was estimated (Tables 1 and 2). The second-level evaluation (learning) findings showed that the participants' average CPR skill learning score before the intervention was 11.56 ± 2.28 and 14.26 ± 2.21 after the intervention. The findings of the Paired Sample t-test indicated significant differences before and after the intervention (P < 0.01). The post-test score related to CPR skills based on the number "100" was estimated as 71.3 ± 11.05. In addition, the findings of the third level (performance) revealed that the average performance score of the participants was 13.48 ± 0.93, and this value was 96.28 ± 6.65 based on the number "100". The average of the fourth level (effectiveness) was equal to 4.78 ± 1.64, which was obtained based on the number "100" equal to 53.24 ± 1.44 (Table 2). The findings comparing the cardiopulmonary resuscitation status in the first six months with the second six months are presented in Table 3 based on some demographic variables of the patients.
Table 1.Participants' Satisfaction with the Cardiopulmonary Resuscitation Training Course a
| Variables | Amount |
|---|---|
| Content of the training course | 19.38 ± 1.25 |
| The ability and literacy level of the teacher | 19.36 ± 1.2 |
| The quantity and quality of the necessary facilities to hold the training course | 19.14 ± 1.22 |
| The level of overall satisfaction based on the score obtained from the questionnaire | 57.88 ± 3.66 |
| Overall satisfaction based on the number "100" | 96.46 ± 6.11 |
a Values are presented as mean ± SD.
Table 2.The Effectiveness of the Kirkpatrick Model on the Cardiopulmonary Resuscitation Training Course a
| Evaluation Levels | Based on the Score Obtained from the Questionnaire | Based on the Score "100" | Weighting Coefficient |
|---|---|---|---|
| First level (reaction) | 57.88 ± 3.66 | 96.46 ± 6.11 | 1 |
| Second level (learning) | 14.26 ± 2.21 | 71.30 ± 11.05 | 2 |
| Third level (performance) | 48.58 ± 13.10 | 95.97 ± 13.10 | 3 |
| Fourth level (results) | 4.78 ± 1.64 | 53.24 ± 1.44 | 4 |
| Overall index | 88.23 % | ||
a Values are presented as mean ± SD.
Table 3.Comparison of Cardiopulmonary Resuscitation Status in the First Six Months with the Second Six Months
| Variables | The First Six Months | The Second Six Months | P-Value |
|---|---|---|---|
| Age of patients | 59.91 ± 17.4 | 59.61 ± 18.3 | 0.91 |
| Recovery time | 33.42 ± 17.3 | 36.8 ± 7.18 | 0.16 |
| Gender of patients | 0.4 | ||
| Female | 37 (36) | 37 (30) | |
| Male | 66 (64) | 74 (70) | |
| Result of recovery | 0.01 | ||
| Successful | 25 (24) | 59 (53) | |
| Unsuccessful | 78 (76) | 52 (47) |
a Values are presented as No. (%) and mean ± SD.
5. Discussion
The findings indicated the effectiveness of the in-service training course for emergency nursing personnel regarding the cardiopulmonary resuscitation (CPR) process. The chance of the training mentioned above course being successful was estimated to be 88.23%. In addition, participants' satisfaction with the course content, the instructor's teaching style, and the course facilities was relatively high according to their responses on the first level of the model (reaction). The findings of some similar studies in the past are consistent with the present study. Hojjati et al. and Dehghani et al. showed that nurses have a high level of satisfaction with the way of holding the CPR training course (14, 18). Often, this level of evaluation is conducted by a questionnaire about the conditions of holding the course and the instructor's skills, and its findings play an essential role in improving the implementation quality of the CPR training course (25). There are disagreements about evaluating this level because the review of the first level (reaction) does not play an important role in examining educational goals, and mainly this category of reaction is subject to interest and motivation (30). Therefore, for this reason, researchers consider this level (reaction) with a lower coefficient to evaluate the effectiveness of the CPR training course.
The findings at the second level of Patrick's model indicated that the post-test scores increased and the learning rate of the participants in the CPR training course enhanced. The findings of most studies showed that in-service training increases learning and clinical skills, which increases the care and protection of patients (30). Motaarefi et al. reported that the learning rate of nursing students increased significantly before and after the CPR training course (28). Akbari et al. found that the CPR training course increased the learning level of nurses and paramedics in the internal, surgical, and emergency departments (26). Al-Hadid and Suleiman concluded that the average scores after the intervention have a significant difference and increased the knowledge and skills of nurses (31). Hojjati et al. and Dehghani et al. evaluated the effectiveness of in-service training courses for nurses based on Patrick's model, which was also consistent with the findings of the present study in the learning phase (14, 18). Bakhshi-Hajikhajeloo and Sattari also achieved similar findings in examining the effectiveness of brain CPR training workshops in the clinical environment based on the Kirkpatrick model (32).
In the present study, the finding of the third level (performance) of nurses and the fourth level (findings) also indicated the achievement of the majority of the goals of the CPR training course, and in general, the training course was practical. In line with the present study’s findings, Khaledi et al. showed that the pre-organizer training method significantly increased learning and sustainable learning in the training course participants (33). In this regard, Chang et al. investigated the effect of triage start training with face-to-face training on the level of awareness of personnel and indicated that triage training with the face-to-face method could significantly increase awareness in this field. In addition, personnel errors in prioritizing the injured were reduced and increased the ability to perform correct triage (34).
Holding CPR courses for nurses positively increased their efficiency. In general, the findings of this study were consistent with those of Aoki et al.. The findings of the mentioned study showed that implementing the CPR training program in all three levels of the Kirkpatrick model was desirable and had positive effectiveness (35). Omar et al. also confirmed the usefulness of the assessments conducted on middle managers of health in different provinces of Iran based on the Kirkpatrick model (36).
5.1. Limitations
One of the limitations of this research was the lack of a control group for comparison, which can limit the generalizability of the study to some extent. In addition, this study was conducted only on nurses of a teaching hospital, and only the effectiveness of a training course was evaluated. Therefore, it is necessary to be careful in generalizing its findings to other medical centers and training courses.
5.2. Conclusions
Based on the findings, the cardiopulmonary resuscitation training course increases nursing skills, knowledge, and efficiency. In addition, based on the findings of this study, it can be said that the course is practical for nurses. The findings showed that the Kirkpatrick model could be a suitable method for evaluating the effectiveness of in-service training in health. Since the practical and clinical skills of nurses are complex activities and the correct implementation of many skills guarantees the life of patients. Therefore, it is suggested to evaluate nurses' learning and functional skills continuously and regularly. In addition, it is recommended to use up-to-date and efficient models in the field of health sciences for proper evaluation.
