1. Background
Establishing and maintaining a suitable airway for ventilation is essential in general anesthesia, and this is achieved by endotracheal intubation. Endotracheal intubation is a vital, critical and at the same time complicated procedure, the purpose of which is to place a flexible tube inside the trachea for the purpose of respiratory support for a patient who does not breathe spontaneously (1). Nurse anesthetists are usually the first healthcare providers to be with the patients in critical situations to help them. If they do not have sufficient knowledge and skills and cannot perform the necessary interventions to maintain the airway before the anesthesiologist arrives, the gold time to save the patient's life may be lost (2, 3).
One of the educational methods that can be used in this regard is education based on evaluation (4). The ability of students to recognize their strengths and weaknesses is of paramount importance. Self-evaluation encourages learners to play a more active role in the learning process and facilitates continuous learning (5). Visual self-evaluation is at the level of self-regulation, and is one of the most effective types of feedback given by filming the person during the task and showing the recorded video to them after completion of the task (4).
2. Objectives
The present study investigated the effect of visual self-evaluation training on the endotracheal intubation skill of nurse anesthesia students. It was hypothesized that visual self-evaluation has a more profound effect on promoting the nurse anesthesia students' learning and satisfaction compared to conventional methods.
3. Methods
3.1. Ethics Statement
The present study was approved by the Ethics Committee of Ahvaz Jundishapur University of Medical Sciences (AJUMS) (U-02004) (IR.AJUMS.REC.1402.020) and was carried out in accordance with the provisions of the 2013 Declaration of Helsinki. The aims, procedures, and conditions of the study were fully explained to the potential participants. Written informed consent was obtained from all participating students. Also, the confidentiality of the data and the anonymity of the students during the entire study process were guaranteed.
3.2. Study Design
This randomized controlled trial was in compliance with the CONSORT (Consolidated Standards of Reporting Trials) statement.
3.3. Setting
This study was conducted in April and May of 2023. The training of students in intervention and control groups was done in the clinical skill center of the Anesthesiology Department at the School of Allied Medical Sciences of AJUMS. Pre-test and post-test were conducted in the real work conditions of the participants in the future, namely the operating room of Imam Khomeini Hospital of Ahvaz affiliated to AJUMS.
3.4. Participants
Sixty-three nurse anesthesia students, including 17 (27%) men and 46 (73%) women, with an average age of 21.53 ± 3.65, were included in the study. They were selected by the census method from among the nursing anesthesia students who met the inclusion criteria. Second- or third-year nurse anesthesia students who were willing to participate in the research were eligible to enter the study. The exclusion criteria were: (1) Students' withdrawal from cooperation at any time and for any reason, (2) students' absence in training sessions or (3) students' absence in pre-test or post-test sessions. The data of students who had at least one of the exclusion criteria were not analyzed statistically.
3.5. Sample Size
We performed this study on all second- and third-year nurse anesthesia students at AJUMS. Besides, to define the minimum sample size, it was calculated using HeinrichHeine-Universität Düsseldorf G*Power version 3.1.9.2, based on the independent sample Student t-test, a 2-tailed alpha of 0.05 and power (1-β) of 0.95, based on a previous investigation by Khalafi et al. (6). They noted that mean endotracheal intubation skill scores in the intervention and control groups were 65.36 ± 5.66 and 51.90 ± 3.9 respectively. Therefore, 6 students in each group with 20% droprate were indicated as the minimum sample size.
3.6. Assignment
In this study, the students were allocated to the intervention and control groups according to the academic year, using stratified random sampling. Each student was randomly assigned a code, and the codes were then placed in two boxes according to the academic year. The first code extracted from each box was assigned to the intervention group, while the second code indicated allocation to the control group. This process continued until all students were selected.
3.7. Interventions
The first stage: Upon arrival to the study, the participants completed a demographic questionnaire. Then the pre-test was administered in order to evaluate the endotracheal intubation skill of all participants. This evaluation was done based on structured observation method, using a checklist (Appendix 1 in Supplementary File), on a real patient. In the next step, the students were divided into intervention and control groups.
The second stage: Endotracheal intubation training was offered in two sessions. The first session was based on lecture method and included training in airway anatomy, endotracheal intubation devices and equipment, and the procedures before, during, and after intubation. The first session was held similarly in intervention and control groups. In the second session, each participant in the intervention group was given 15 minutes to perform the endotracheal intubation procedure on the airway management mannequin. After completion of the task and video recording, the students' were shown their own video on screen in the clinical skills center. Also, the endotracheal intubation skill assessment checklist was provided to each participant so that they could complete it immediately after watching their video. The checklist was used to record the students' feedback on their performance so that this feedback could be observed and re-examined. Then the video and the checklist of each participant were delivered to them. After the end of the two sessions, the participants in the intervention group completed a questionnaire to evaluate their level of satisfaction with the visual self-evaluation method (Appendix 2 in Supplementary File). The second session in the control group involved 15 minutes of endotracheal intubation practice on the airway management mannequin by each student in the presence of the instructor who gave the necessary feedback to the students in real time according to the checklist. It should be noted that the checklist used in the training was the same checklist used in the pre-test and post-test stages.
The third stage: After the end of the intervention, a post-test similar to the pre-test stage was administered for all participants. The research data was recorded by a blinded rater who had to have at least 2 years of clinical experience and 1 year of educational experience.
3.8. Outcomes
The outcomes of this study included the following: Primary outcome was measuring skill in performing endotracheal intubation in the intervention and control groups, before and after training based on visual self-evaluation and conventional methods and secondary outcome was measuring satisfaction of students in the intervention group, after training by visual self-evaluation method.
3.9. Data Sources/ Measurement
The data collection tool in this study included 3 sections:
(1) Demographic information questionnaire: Age, gender and academic year
(2) Endotracheal intubation skill evaluation checklist (Appendix 1 in Supplementary File): This checklist was developed and used by Hart et al. in 2016. The validity and reliability of this checklist has been examined and confirmed by its developers in the main article. In the reference study, the inter-rater reliability of the instrument was confirmed by obtaining a correlation coefficient of 0.94 (7). The checklist has 20 items which are scored dichotomously: Yes: 1 and or no: 0. Its focus is on the procedures before (4 items), during (11 items) and after endotracheal intubation (5 items). Also, a part of the checklist is dedicated to errors, which has 7 items and includes error (4 items), critical error (2 items) and critical fail (1 item). Error is defined as an error that does not lead to significant morbidity or mortality. Critical error refers to an error that probably leads to significant morbidity or mortality. Critical fail is a situation in which the evaluator intervenes in order to save the patient's life or prevent injury. Error, critical error and critical fail are reverse scored and calculated as: Error (-1), critical error (-2) and critical fail (-3). In cases where the rater cannot visualize the item or due to equipment limitations, the item cannot be rated, it is removed from the calculation of scores. The range of scores is from the highest score (20) to the lowest score (-11).
(3) Questionnaire of satisfaction with training by visual self-evaluation method (Appendix 2 in Supplementary File): This questionnaire was developed by Pourfakhr et al. in 2021 in Iran (8). This questionnaire includes 10 items that are scored based on a 5-point Likert scale (I completely agree: 5, I agree to a certain extent: 4, I have no opinion: 3, I disagree to a certain extent: 2 and I completely disagree: 1, and the range of scores is from the highest score (50) to the lowest score (9).
3.10. Bias and Blinding
The only rater in this study was blind to group allocation in order to avoid any bias in data entry. Also, prior to commencement of the study, a short meeting was held for the rater to be introduced to the checklist and learn how to use it. For practice, 3 cases of endotracheal intubation were scored by the rater using the checklist so that the necessary feedback could be given by the researcher to eliminate any possible problems.
3.11. Statistical Methods
IBM SPSS ver. 25.0 (IBM Crop.) was used for data analysis. Data analysis included descriptive and inferential statistics. The latter involved comparing the mean of quantitative variables in the 2 groups using the independent t-test and paired t-test, and if the assumptions of the test were not established, its non-parametric equivalent (i.e., the Mann-Whitney test), was used. The significance level was set at P < 0.05.
4. Results
4.1. Participants
All 63 nurse anesthesia students, including 17 men (27%) and 46 women (73%) with a mean age of 21.53 ± 3.65 years, participated in this study (Table 1). Based on the chi-square test, the intervention and control groups displayed no statistically significant difference in terms of sex (P = 0.55) or academic year (P = 0.91). Furthermore, based on the independent t-test results, no statistically significant difference was present between the groups in terms of age (P = 0.13). The study flow diagram is presented in Figure 1.
Table 1.
Demographic Characteristics of Students in Intervention and Control Groups a
| Characteristic | Total (n = 63) | Group | P-Value | |
|---|---|---|---|---|
| Intervention (n = 32) | Control (n = 31) | |||
| Sex | 0.55 b | |||
| Male | 17 (27.00) | 10 (58.80) | 7 (41.20) | |
| Female | 46 (73.00) | 22 (47.80) | 24 (52.20) | |
| Year of study | 0.91 b | |||
| Second year | 32 (50.80) | 16 (50.00) | 16 (51.60) | |
| Third year | 31 (49.20) | 16 (50.00) | 15 (48.40) | |
| Age | 21.53 ± 3.65 | 22.40 ± 4.93 | 20.63 ± 0.89 | 0.13 c |
a Values are expressed as number (%) or mean ± standard deviation.
b P-value obtained with the chi-square test.
c P-value obtained with the independent t-test.
Figure 1.
The study flow diagram
4.2. Main Results
4.2.1. Endotracheal Intubation Skill of Nurse Anesthesia Students
The mean scores of students in the post-test compared to the pre-test in both intervention (23.56 ± 2.68 at baseline and increased to 28.87 ± 4.80; P < 0.001; 95% CI, 4.36 to 6.26) and control (24.03 ± 2.22 at baseline and increased to 25.19 ± 2.46; P = 0.001; 95% CI, 0.65 to 2.1) groups increased significantly (Table 2). As far as the procedures before endotracheal intubation were concerned, only the changes in the control group were significant, with the mean score of the pre-test being significantly higher than that of the post-test (P = 0.001; 95% CI, -0.89 to -0.26). With regard to procedures during intubation, the changes were not significant in both intervention (P = 0.174; 95% CI, -0.69 to 0.13) and control (P = 0.161; 95% CI, -0.62 to 0.1) groups. However, positive and significant changes were observed in both groups in terms of the procedures after intubation (P < 0.001).
Table 2.
Comparison of the Mean and Standard Deviation of the Pre-test and Post-test Scores for Endotracheal Intubation Skill and its Subscales in the Intervention and Control Groups (Paired t-Test and Independent t-Test) a
| Skill and Group | Pre-test | Post-test | t-Value | P-Value b |
|---|---|---|---|---|
| Endotracheal intubation | ||||
| Intervention | 23.56 ± 2.68 | 28.87 ± 4.80 | 11.415 | < 0.001 |
| Control | 24.03 ± 2.22 | 25.19 ± 2.46 | 3.830 | 0.001 |
| t-value | - 0.754 | 3.807 | ||
| P-value c | 0.454 | < 0.001 | ||
| Before endotracheal intubation | ||||
| Intervention | 4.09 ± 0.29 | 4.31 ± 0.73 | 1.487 | 0.147 |
| Control | 4.77 ± 0.88 | 4.19 ± 0.83 | - 3.815 | 0.001 |
| t-value | - 4.125 | 0.600 | ||
| P-value c | < 0.001 | 0.551 | ||
| During endotracheal intubation | ||||
| Intervention | 14.50 ± 2.60 | 14.78 ± 2.81 | 1.392 | 0.174 |
| Control | 13.51 ± 1.78 | 13.77 ± 1.83 | 1.438 | 0.161 |
| t-value | 1.724 | 1.525 | ||
| P-value c | 0.090 | 0.040 | ||
| After endotracheal intubation | ||||
| Intervention | 4.34 ± 0.48 | 6.93 ± 0.50 | 19.407 | < 0.001 |
| Control | 4.77 ± 0.71 | 5.61 ± 0.76 | 6.789 | < 0.001 |
| t-value | - 2.804 | 8.173 | ||
| P-value c | 0.007 | < 0.001 |
a Values are expressed as mean ± SD.
b P-value obtained with the paired t-test.
c P-value obtained with the independent t-test
Also, in the pre-test, there was no statistically significant difference between the mean scores of the intervention and control groups (P = 0.454; Hedges’ g = 0.46; 95% CI, -1.7 to 0.77). However, in the post-test, the mean score of the intervention group was significantly higher compared with the control group (P < 0.001; Hedges’ g = 3.68; 95% CI, 1.7 to 5.6) (Table 2).
Satisfaction with visual self-evaluation method
The intervention group students' satisfaction with this training method was measured. The mean satisfaction score in the intervention group was 40.79 ± 1.22 out of a maximum of 50.
5. Discussion
5.1. Key Results
This study investigated the effects of visual self-evaluation training method versus conventional methods on the improvement of endotracheal intubation skill of nurse anesthesia students. According to the results, the mean score of the learning outcome in the intervention group increased significantly compared with the control group. In addition, students' satisfaction with visual self-evaluation was very high.
5.2. Interpretation
According to the results of this study, visual self-evaluation method as opposed to traditional methods can lead to self-regulated learning, skill mastery, and increased independence among students. In the present study, the changes in skill scores in both groups were positive and significant, but the improvement of the scores of the intervention group was significantly higher compared with the control group. Given the self-regulated nature of learning in education through visual self-evaluation method, the use of this method can further improve the clinical skills of nurse anesthesia students and accelerate their achievement of sufficient mastery and independence in these skills. Visual self-evaluation helped students to better identify their strengths and weaknesses and work on the modification and enhancement of their performance. Also, visual self-evaluation made students better prepared to enter the hospital and have a more efficient performance. Students were also highly satisfied with this method. This lends support to the importance of using self-evaluation methods to better educate students and help them acquire practical skills, develop professional responsibility, and shift from dependent and supervised practice to fully independent and autonomous practice.
5.3. Comparison with Previous Studies
Most of the previous studies addressing self-evaluation have investigated its accuracy and improvement. For example study by Warehime et al. was a clinical trial in Canada investigating visual self-evaluation accuracy among senior surgical residents. The subjects evaluated their performance after performing laparoscopy and after observing their video-recorded performance. Correlations between evaluation made by experts and surgical residents' self-evaluation were initially moderate but increased significantly after the residents reviewed their self-recorded performance. In the end, the researchers concluded that observing one's recorded performance can improve their ability in self-evaluation (10). Netter et al. in London also conducted an intervention study aimed at investigating the ability of surgeons to evaluate their performance for training and self-regulation. The obtained results showed that there was a strong correlation between the experts' rating of technical skills and self-evaluation. It was concluded that senior surgical residents are accurate in their evaluation of technical skills (9). Very limited studies have used self-evaluation as an educational method and measured the learning outcome. A case in point was Gicolini et al. examined the effect of visual self-evaluation training on the knowledge and performance of nurses in performing basic cardiopulmonary resuscitation in Iran. The results of this study showed that visual self-evaluation training affects nurses' performance in cardiopulmonary resuscitation, but no significant difference was observed in the knowledge scores of the two groups (11), which is in line with the present study. Also, Azadi et al. conducted a study in Iran focusing on the effect of visual self-evaluation training on the knowledge, self-esteem and performance of operating room staff in performing advanced cardiopulmonary resuscitation. Their results showed that visual self-evaluation promoted the knowledge, self-esteem and performance of employees (4), which is consistent with the results of the present study.
5.4. Limitations
In the current study, due to observance of patient privacy and research ethics, filming was not possible in a real clinical setting. Therefore, visual self-evaluation was done in the clinical skill center.
5.5. Generalizability
The results of this study can be useful for the development and improvement of the clinical skills of students in all fields of medical sciences around the world.
5.6. Suggestions
Future studies are recommended to compare the effectiveness of visual self-evaluation with other existing educational methods. Also, training based on visual self-evaluation method should be repeated at regular intervals to consolidate learning, and for this purpose, it is recommended to use this method in continuous education.
5.7. Conclusions
The use of visual self-evaluation method can be considered as a low-cost and effective way to improve clinical skills, to identify student strengths and weaknesses, and to improve performance, along with other educational methods.
