Abstract
Background:
Evaluating the effectiveness of educational courses is essential for ensuring high-quality healthcare.Objectives:
This study assessed the effectiveness of current in-service training courses on infection prevention and control for nurses, using the Kirkpatrick evaluation model.Methods:
This evaluative study was conducted at Shiraz Army Hospital in 2024, involving 40 nurses and 10 supervisors. The educational program consisted of interactive workshops held over two days, covering topics such as standard precautions, isolation procedures, and hand hygiene. The evaluation followed Kirkpatrick’s four levels: Reaction, learning, behavior, and impact. Nurses' reactions and knowledge were measured using validated questionnaires, while supervisors assessed behavioral changes. Data were analyzed with SPSS 26, using descriptive statistics and a one-sample t-test.Results:
All four levels of the Kirkpatrick model were evaluated. Nurses reported a mean reaction score of 3.73 (SD = 0.80), reflecting positive feedback on the training. Knowledge scores significantly increased from a pre-test mean of 2.39 (SD = 0.74) to a post-test mean of 3.72 (SD = 0.74) (P < 0.001). Supervisors observed a behavioral improvement, with scores increasing from 2.34 (SD = 0.94) to 3.72 (SD = 0.74) (P = 0.004). Furthermore, the nosocomial infection index decreased from 0.7 to 0.5 (P = 0.002) following the training.Conclusions:
The findings demonstrate the effectiveness of current in-service training courses on infection prevention and control for nurses. The Kirkpatrick model proved to be a valuable evaluation tool, underscoring the importance of ongoing assessment of nurses’ competencies to enhance infection prevention practices.Keywords
1. Background
Education is fundamental to human resource development, and ensuring its quality poses a substantial challenge for universities worldwide. Effective education is especially critical within universities, where employee training drives organizational growth and dynamism (1). In-service training, as a core component of efforts to enhance knowledge, awareness, technical skills, and professional competencies, prepares individuals for optimal job performance and responsibility (2). Currently, short-term in-service training courses are widely available programs that significantly contribute to the growth and improvement of employees' job skills within organizations (3). However, true development cannot be achieved without effective education that enhances quality (4).
Various factors impact the success of training programs in achieving their intended outcomes, and thorough evaluation is a key aspect of program implementation (5). Education is beneficial only when it is evaluated, and effective evaluation encompasses not only learners' self-assessment but also the assessment of the educational process (including educational programs, evaluation methods, and facilities) and participants' behaviors (6). Evaluating past programs is crucial for healthcare workers to understand strengths and weaknesses, guiding future improvements. Nurses, as essential members of healthcare teams, possess complex practical and clinical skills that are often critical for patient survival. Therefore, continuous and systematic evaluation of nurses' learning and functional skills is of paramount importance (2).
Various models exist for educational evaluation, including those by Kirkpatrick, Sullivan, London Business School, as well as capability-based and performance evaluation-based models. Among these, the Kirkpatrick model is widely recognized and utilized (1). Developed by Donald Kirkpatrick, this model has been applied for over three decades, especially within medical science education institutions (7). The Kirkpatrick model evaluates in-service training at four levels: Reaction, learning, behavior, and results (8). Three primary reasons motivate this evaluation: Determining the program's relevance and alignment with the unit's philosophy and mission, deciding whether to continue the training program, and assessing the program’s effectiveness and areas for improvement (9, 10). A training program proves valuable when it demonstrates a positive impact on learners' behavior and performance through documented and reliable evidence (11). Ultimately, effectiveness is realized when learners apply their acquired knowledge effectively in real-world scenarios, leading to positive changes in their work performance (12).
A crucial aspect of implementing training programs is accurately evaluating their outcomes, which requires selecting an appropriate assessment method. One such method is Kirkpatrick’s pyramid. The model’s characteristics include process clarity, limited variable assessment, simplicity in evaluation criteria, and no requirement to gather participants' past performance. These features make it an effective model for evaluating educational programs. Although every model has its limitations, Kirkpatrick’s model demonstrates acceptable performance in assessing educational programs, according to evaluations (3).
In our country, however, training course evaluations often concentrate on the first or, at most, the second level of the Kirkpatrick model, highlighting early-stage successes and effectiveness in the educational process. Yet, as evaluations extend to the third and fourth levels, the measurable educational impact typically lessens (13).
2. Objectives
This study aimed to evaluate the effectiveness of existing in-service training courses on infection prevention and control among nurses at Shiraz Army Hospital, using Kirkpatrick's evaluation model.
3. Methods
3.1. Study Design
This evaluative study aimed to assess the impact of existing in-service training courses on infection prevention and control workshops among nurses at Shiraz Army Hospital, utilizing Kirkpatrick's evaluation model. The inclusion criteria were participation in the infection control training course and completion of the evaluation test. Exclusion criteria included non-participation in the training course and unwillingness to complete the test.
3.2. Educational Program
The educational program comprised a series of interactive workshops focused on infection prevention and control practices. Conducted over two days, these workshops included lectures, group discussions, and hands-on training sessions. Topics covered encompassed standard precautions, isolation procedures, and the significance of hand hygiene. Participants were encouraged to engage in discussions and share experiences related to infection control in their clinical settings.
3.3. Setting
The study population included all nurses and supervisors at Shiraz Army Hospital. A convenience sampling method was used, encompassing all nurses and nursing supervisors who had participated in the in-service training courses on infection prevention and control. In total, 40 nurses and 10 supervisors were included in the study.
3.4. Instrument
The data collection instrument used was Kirkpatrick's standard evaluation questionnaire, previously validated and proven reliable by Filizadeh in 2016 (14). Experts in medical and nursing education reviewed the questionnaire for validation, and adjustments were made based on their feedback. The reliability of the questionnaire was evaluated using Cronbach's alpha coefficient, yielding an overall reliability of 96% for the staff questionnaire and 97% for the supervisor questionnaire.
The learners' questionnaire included:
- 17 questions for the reaction level, rated on a 5-point Likert Scale from 1 (strongly disagree) to 5 (strongly agree), yielding a possible score range of 17 to 85.
- 8 questions for the knowledge level, rated similarly, with a possible score range of 8 to 40.
The officials' questionnaire comprised:
- 15 questions each for the behavior and results levels, rated on the same scale, with a possible score range of 15 to 75 for each level.
The questionnaires were distributed in printed form and completed by participants. Ethical approval was obtained from the ethics committee, and informed consent was secured from all participants. They were assured of confidentiality regarding their information and their right to withdraw from the study at any time.
To assess the effectiveness of the training courses, level one (reaction) and level two (knowledge) questionnaires were administered to nurses one year after completing the training. Level three (behavior) questionnaires were completed by nursing officials to evaluate behavioral changes. Additionally, infection control indicators—including rates of hospital-acquired infections, compliance with hygiene protocols, and staff adherence to infection control practices—were measured before and after the training course to assess the fourth level of Kirkpatrick's evaluation model.
3.5. Sample Size
The sample size for this study was calculated to detect a significant improvement in nurses' knowledge scores following the in-service training on infection prevention and control. Assuming a medium effect size (Cohen's d = 0.5), a significance level (α) of 0.05, and a power (1 − β) of 80%, the required sample size was determined using the formula for a paired t-test:
Where: Zα/2 is the critical value of the standard normal distribution at α/2 (for a two-tailed test at α = 0.05, Zα/2 = 1.96), Zβ is the critical value at the desired power level (for 80% power, Zβ = 0.84), d represents the effect size (Cohen’s d = 0.5).
Based on these parameters, the minimum required sample size is approximately 32 nurses. To account for potential dropouts and to further enhance the study's power, we included 40 nurses in the study, exceeding the minimum requirement. This ensured sufficient statistical power to detect a meaningful difference in knowledge scores before and after the training.
3.6. Data Analysis
Statistical analysis included the use of frequency and percentage to describe qualitative data, while mean, standard deviation, median, and range were used for quantitative data. One-sample t-tests were conducted to compare the average scores against hypothetical scores. All analyses were performed using SPSS software version 26, with a P-value of less than 0.05 considered statistically significant.
4. Results
The study gathered feedback from a sample of 40 nurses and 10 supervisors concerning the study's objectives. Among the nurses, 12 (30%) were male, and 28 (70%) were female. In the supervisor group, 4 (40%) were male, and 6 (60%) were female. Of the nurses, 36 (90%) held undergraduate degrees, while 4 (10%) held postgraduate qualifications. Among the supervisors, 7 (70%) had bachelor’s degrees, and 3 (30%) had postgraduate qualifications.
The results indicate that, among the nurses in the sample, the average responses to questions assessing the impact of training courses in generating a positive reaction were distributed as follows: 0% very low, 9% low, 25% somewhat, 49% high, and 16% very high. Moreover, the average positive reaction score (3.73) was significantly higher than the hypothetical average score of 3, with a significance level of 0.05 (P-value < 0.001) (Table 1).
The Survey Results Depicting the Learners' Reactions to the Educational Program (N = 50)
| Questions | Mean ± SD |
|---|---|
| To what extent have the held training courses been related to the field of your activities and job characteristics? | 3.9 ± 0.81 |
| To what extent have the held training courses been useful in your job? | 3.95 ± 0.78 |
| To what extent were the goals set in the courses logically organized? | 3.65 ± 0.77 |
| To what extent has participation in training courses met your anticipated expectations? | 3.68 ± 0.76 |
| In your case, to what extent have the goals set in the courses been achieved? | 3.65 ± 0.83 |
| To what extent has the content of the training courses increased your interest in training in your job? | 3.82 ± 0.87 |
| How up-to-date is the content of the courses and educational materials? | 3.75 ± 0.78 |
| How suitable were the resources and educational pamphlets used for the courses? | 3.75 ± 0.78 |
| To what extent were the audiovisual facilities and the use of educational aids appropriate? | 3.35 ± 0.7 |
| How appropriate was the way of organizing and planning and executive management of the courses? | 3.62 ± 0.81 |
| To what extent was the quality of the space, equipment and venue of the courses (in terms of: Light, ventilation, capacity, distance from noise) suitable? | 3.4 ± 0.74 |
| How appropriate is the quality of welfare services? | 3 ± 0.72 |
| To what extent did the lecturers master the subjects and content of the courses? | 3.97 ± 0.83 |
| How skilled were the teachers in teaching methods? | 3.97 ± 0.83 |
| To what extent did the lecturers respect the order and continuity of the materials? | 3.83 ± 0.87 |
| To what extent were the lecturers able to answer your questions? | 3.93 ± 0.89 |
| How appropriate is the behavior of the teachers? | 4.1 ± 0.9 |
| Total | 3.72 ± 0.80 |
The results of the second level of Kirkpatrick's pyramid, which assesses the knowledge gained during the "Prevention and Control of Infection" course, showed that the average knowledge score of the nurses increased from 2.39 out of 5 before the training program to 3.72 after completing the program. This indicates a substantial improvement in the nurses' knowledge following the training (Table 2).
Evaluation of the Second and Third Levels of Kirkpatrick's Pyramid of the Training Course (One Sample t-test)
| Evaluation of Supervisors | Before the Educational Program | After the Educational Program | Difference of Averages | P-Value |
|---|---|---|---|---|
| Knowledge | 2.93 ± 0.74 | 3.72 ± 0.74 | 2.98 | 0.001 |
| Observed behavior | 2.34 ± 0.94 | 3.72 ± 0.74 | 1.38 | 0.004 |
The third level of Kirkpatrick's model evaluates behavioral or functional changes. At this level, supervisors assessed the nurses' behavioral changes in their actual work environment using a 5-point Likert Scale, based on the training content and objectives. The initial performance score of the nurses was 2.34 out of 5, which significantly increased to 3.72 post-training (Table 2).
For the fourth level of Kirkpatrick's model, a comparison was made between the hospital infection control indicators from the previous year and the year following the training program. In terms of the hospital infection reduction indicator (normal range: Close to zero), the nosocomial infection rate decreased from 0.7 before the study to 0.5 after the study (P-value = 0.002), indicating a 0.2% reduction. Additionally, the hand hygiene compliance rate improved from 55% to 75% (P-value = 0.005), marking a 20% increase in compliance (normal range: 100%).
5. Discussion
The research results indicate that the "Infection Prevention and Control" training course was assessed as effective by the participating nurses and supervisors. Based on findings from the fourth level of Kirkpatrick's pyramid, it can be asserted that the course positively impacted the hospital's infection control indicators, leading to improvements in these measures.
The analysis of nurses' responses to questions about generating a positive reaction revealed that participants highly appreciated the teaching method's effectiveness in eliciting a favorable response. Nurses expressed the highest satisfaction with the instructors' behavior and approach, followed by their expertise in the subject matter and teaching skills. Overall, the majority of participants found the course to be beneficial and effective. These findings align with another study, which also reported positive feedback from nurses regarding instructors, course content, and facilities (13).
The level of learning was measured by comparing participants' knowledge before and one month after the training using the same questionnaire. Results indicated an increase in nurses' learning levels post-training. This aligns with a study by Al-Hadid and Suleiman, which also demonstrated a significant improvement in nurses' knowledge and skills following an intervention (15).
The results regarding nurses' behavior and outcomes indicate the successful achievement of most course objectives, supporting the overall effectiveness of the training. Consistent with our findings, Khaledi et al.'s study (16) demonstrated the efficacy of the pre-organizer training method in enhancing learning and promoting long-term knowledge retention among participants.
Kirkpatrick's evaluation model has been widely applied to assess the effectiveness of both nursing and non-nursing programs. Studies such as D'Alessandro et al. highlight the importance of evaluating nurses in performing their responsibilities, as inadequate knowledge of infection control standards can result in non-compliance (17). Numerous nursing studies have employed the Kirkpatrick model, including Huang et al.’s research, which examined the model's impact on innovation training for clinical nurses, focusing on the first three levels of the model (18).
During the COVID-19 pandemic, Li et al.'s study used the Kirkpatrick model to evaluate clinical nurses’ training and their response to the pandemic, concentrating on the first two levels (19). Similar to our study, nurses showed relatively high levels of satisfaction, and there was a statistically significant improvement in theoretical and operational scores pre- and post-training. Notably, the Kirkpatrick model emphasizes the importance of evaluating learners after training; without such evaluation, the model is incomplete. However, the referenced study evaluated only nurses' reactions and learning levels, lacking assessment at the third and fourth levels, which is an important consideration.
In Suresh et al.'s study, which assessed the pre-deployment training of army nurses and doctors, most participants demonstrated adequate training levels, although their crisis care capabilities were moderate (20). This finding underscores the importance of evaluating performance in high-stakes situations, highlighting the need to assess behavior and results within nursing education.
Zarparvar et al.'s internal study demonstrated that participation in in-service training courses led to nurses’ and supervisors’ improved job skills, enhanced work discipline, greater cooperation and collaboration, and increased job satisfaction (21). These findings align with our research. Additionally, Dorri et al.'s study indicated that training courses effectively enhanced learners’ knowledge of cardiopulmonary resuscitation, subsequently improving their performance (7).
However, a common trend in studies employing the Kirkpatrick model to evaluate effectiveness, both in nursing and other fields, is an emphasis on the first two levels, with limited focus on the higher levels, particularly in our country. Therefore, more comprehensive evaluations covering all levels of the Kirkpatrick model are necessary. This study aimed to assess all levels of the model in the context of nursing education, and the positive results indicate the model’s effectiveness in enhancing the educational experience for participants. Further studies with larger sample sizes are recommended to substantiate these findings. It is hoped that this research contributes to the advancement of nursing education quality, ultimately benefiting patients in need.
5.1. Limitation
This study was limited to nurses from a single teaching hospital and evaluated only one specific time period. Additional limitations included a small sample size and the use of purposive sampling, which may affect generalizability. Finally, the measurement of the fourth level of the Kirkpatrick model was not feasible in this study.
5.2. Suggestions for Further Studies
It is recommended to continuously and regularly evaluate the learning and practical skills of nurses. Additionally, adopting current and effective models in health sciences for thorough evaluation is advised.
5.3. Conclusions
The results confirm the effectiveness of the "Infection Prevention and Control" course for nurses across all four levels: Reaction, learning, behavior, and results. The Kirkpatrick model proves to be a suitable method for assessing the impact of in-service training in healthcare. Since nurses’ practical and clinical skills involve complex activities that can directly affect patient outcomes, ongoing evaluation of their knowledge and skills is essential. Furthermore, the adoption of updated and efficient evaluation models in healthcare is highly encouraged.
Acknowledgements
References
-
1.
Heydari MR, Taghva F, Amini M, Delavari S. Using Kirkpatrick's model to measure the effect of a new teaching and learning methods workshop for health care staff. BMC Res Notes. 2019;12(1):388. [PubMed ID: 31292006]. [PubMed Central ID: PMC6617554]. https://doi.org/10.1186/s13104-019-4421-y.
-
2.
Dennick R. Long-term retention of teaching skills after attending the Teaching Improvement Project: a longitudinal, self-evaluation study. Med Teach. 2003;25(3):314-8. [PubMed ID: 12881057]. https://doi.org/10.1080/0142159031000100436.
-
3.
Bates R. A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence. Eval Program Plan. 2004;27(3):341-7.
-
4.
Hasani H, Bahrami M, Malekpour A, Dehghani M, Allahyary E, Amini M, et al. Evaluation of Teaching Methods in Mass CPCR Training in Different Groups of the Society, an Observational Study. Med (Baltimore). 2015;94(21). e859. [PubMed ID: 26020392]. [PubMed Central ID: PMC4616420]. https://doi.org/10.1097/MD.0000000000000859.
-
5.
Patel SR, Margolies PJ, Covell NH, Lipscomb C, Dixon LB. Using Instructional Design, Analyze, Design, Develop, Implement, and Evaluate, to Develop e-Learning Modules to Disseminate Supported Employment for Community Behavioral Health Treatment Programs in New York State. Front Public Health. 2018;6:113. [PubMed ID: 29868533]. [PubMed Central ID: PMC5949337]. https://doi.org/10.3389/fpubh.2018.00113.
-
6.
Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE guide no. 67. Med Teach. 2012;34(5):e288-99. [PubMed ID: 22515309]. https://doi.org/10.3109/0142159X.2012.668637.
-
7.
Dorri S, Akbari M, Dorri Sedeh M. Kirkpatrick evaluation model for in-service training on cardiopulmonary resuscitation. Iran J Nurs Midwifery Res. 2016;21(5):493-7. [PubMed ID: 27904633]. [PubMed Central ID: PMC5114794]. https://doi.org/10.4103/1735-9066.193396.
-
8.
Vaitkaitis D. EMS systems in Lithuania. Resuscitation. 2008;76(3):329-32. [PubMed ID: 17822828]. https://doi.org/10.1016/j.resuscitation.2007.07.028.
-
9.
Smidt A, Balandin S, Sigafoos J, Reed VA. The Kirkpatrick model: A useful tool for evaluating training outcomes. J Intellect Dev Disabil. 2009;34(3):266-74. [PubMed ID: 19681007]. https://doi.org/10.1080/13668250903093125.
-
10.
Mbagwu F, Nwachukwu C. Staff training and development, enhancement of job performance: Case of Federal University of Technology Library, Owerri. Inform Technol (The). 2011;7(2). https://doi.org/10.4314/ict.v7i2.65644.
-
11.
Farjad S. The Evaluation Effectiveness of Training Courses in University by Kirkpatrick Model (Case Study: Islamshahr University). Procedia-Soc Behav Sci. 2012;46:2837-41. https://doi.org/10.1016/j.sbspro.2012.05.573.
-
12.
Salehi A, Harris N, Lotfi F, Hashemi N, Kojouri J, Amini M. Reform in medical and health sciences educational system: a Delphi study of faculty members' views at Shiraz University of Medical Sciences. East Mediterr Health J. 2014;20(3):151-61. [PubMed ID: 24950072].
-
13.
Dagnew Gebrehiwot G, Elantheraiyan P. A study on the effect of training on employee performance in the case of Mekelle City, Tigray, Ethiopia. Soc Sci Humanities Open. 2023;8(1). https://doi.org/10.1016/j.ssaho.2023.100567.
-
14.
Filizadeh SH. [Evaluation of the effectiveness of in-service training courses for Shiraz Namazi Hospital laboratory staff based on the Kirkpatrick model [dissertation]]. Shiraz, Iran: Shiraz University of Medical Sciences; 2016. FA.
-
15.
Al Hadid L, Suleiman KH. Effect of boost simulated session on CPR competency among nursing students: A pilot study. J Educ Practice. 2012;3(16):186-93.
-
16.
Khaledi S, Moridi G, Shafieian M, Gharibi F. [The effect of a combination of lecture and participatory on the stable learning of nursing students]. J Nurs Edu. 2010;20(4):76-80. FA.
-
17.
D'Alessandro D, Agodi A, Auxilia F, Brusaferro S, Calligaris L, Ferrante M, et al. Prevention of healthcare associated infections: medical and nursing students' knowledge in Italy. Nurse Educ Today. 2014;34(2):191-5. [PubMed ID: 23725909]. https://doi.org/10.1016/j.nedt.2013.05.005.
-
18.
Huang X, Wang R, Chen J, Gao C, Wang B, Dong Y, et al. Kirkpatrick's evaluation of the effect of a nursing innovation team training for clinical nurses. J Nurs Manag. 2022;30(7):2165-75. [PubMed ID: 34747090]. https://doi.org/10.1111/jonm.13504.
-
19.
Li Z, Cheng J, Zhou T, Wang S, Huang S, Wang H. Evaluating a Nurse Training Program in the Emergency Surgery Department Based on the Kirkpatrick's Model and Clinical Demand During the COVID-19 Pandemic. Telemed J E Health. 2020;26(8):985-91. [PubMed ID: 32357086]. https://doi.org/10.1089/tmj.2020.0089.
-
20.
Suresh MR, Valdez-Delgado KK, Staudt AM, Trevino JD, Mann-Salinas EA, VanFosson CA. An Assessment of Pre-deployment Training for Army Nurses and Medics. Mil Med. 2021;186(1-2):203-11. [PubMed ID: 33007065]. https://doi.org/10.1093/milmed/usaa291.
-
21.
Zarparvar S, Hassani M, Kabiri A, Sameri M. [Study in service training effectiveness in nurses group of imam reza hospital]. Nurs Midwifery J. 2013;11(9):0. FA.