Enhancing Professionalism in Nurse Anesthesia Education: A Randomized Controlled Trial of Multi-Source Feedback

1. Background

Professionalism is defined as a set of knowledge, skills, moral values, and personal and group identity that affect the quality of providing care to patients and dealing with medical claims (1). Professionalism is the foundation of medical and nursing care and an important indicator of the commitment of the treatment staff to the patients, the community, and their profession (2). Given the alarming increase in ethical and professional issues within medical professions, it is imperative that clinical caregivers and students internalize professional values in order to develop and maintain their professional identity (3).

Professionalism in anesthesia nurses is a comprehensive set of principles and behaviors (e.g., patient-centered care, support for patient safety, and responsibility for the quality of care) that guide their performance (4). Nurse anesthetists are trained to evaluate patients before surgery, monitor them during surgery, manage possible complications during anesthesia, and generally provide effective and safe intraoperative care (5). Hospital educational environments, where most of this training takes place for students, have limitations that can affect their professional development, and anesthesia nurses who lack professionalism are a real threat to the safety of patients (6).

Based on previous studies, one of the ways to enhance the professionalism level of students and medical practitioners such as doctors and nurses is multi-source feedback (MSF) or 360-degree evaluation, which has been used to reliably evaluate communication skills and the level of professionalism of doctors and residents in various medical environments (7). MSF is a questionnaire-based process that allows for gathering feedback from different stakeholders, such as hospital officials, residents, nurses, and patients, in a systematic way (8). It allows students to gain a comprehensive insight into their performance from the perspective of different people present in the learning environments with whom they interact, and this helps them to know their strengths and weaknesses and to improve themselves accordingly (9).

2. Objectives

The adoption of MSF as one of the effective strategies in the professional development of doctors has been promising (10, 11), but the effect of this type of feedback on the performance of nurses, especially nurse anesthetists who work cooperatively with other health professionals, remains unknown. Also, the absence of any comprehensive system to evaluate and improve the professionalism level of nurse anesthesia students is a fundamental challenge (12), which has led to the failure of a large number of these students to achieve an ideal professionalism level after graduation. This problem, which is one of the serious defects of the Iranian medical education system (13), reduces the efficiency of treatment and care services. Therefore, the purpose of this study was to see whether MSF has any effect on the level of professionalism of nurse anesthesia students or not.

3. Methods

3.1. Study Design

This was a randomized controlled trial adopting a pre-test/post-test design.

3.2. Setting

The MSF process was carried out from September to November 2023 in three educational and treatment centers affiliated with Shahid Sadoughi University of Medical Sciences in Yazd, Iran. These included Shahid Sadoughi Hospital, Shahid Rahnamoun Hospital, and Afshar Hospital, which served as a real internship environment.

3.3. Participants

This study was performed on all 48 third and fourth-year nurse anesthesia students (aged 20 - 24, 59% women) studying at Shahid Sadougi University of Medical Sciences, Yazd, who were selected using the census method from among those who met the inclusion criteria. Inclusion criteria were: (1) Being a third- or fourth-year nurse anesthesia student; and (2) willingness to participate in research. The exclusion criteria were: (1) withdrew from the study at any stage and for any reason; (2) missed internship sessions more than the allowed limit set by the faculty rules; or (3) had clinical work experience. Besides, to define the minimum sample size, we used G-power software, and we indicated the sample size in the MSF group based on a previous investigation by Asmara and Santoso. (14). They noted that the mean pre and post-test scores were 106.33 ± 10.85 and 120.15 ± 8.55, respectively. G-power: t-tests - Means: Difference between two dependent means (matched pairs), Analysis: A priori: Compute required sample size, Input: Tail(s) = Two, Effect size dz = 1.3947794, α err prob = 0.05, Power (1-β err prob) = 0.95, Output: Noncentrality parameter δ = 4.1843382, Critical t = 2.3060041, Df = 8, Total sample size = 9, Actual power = 0.9540680.

The students were randomly classified based on academic year (third year and fourth year) and allocated into intervention and control groups. Each student was assigned a unique code that was randomly assigned, and these codes were placed in two boxes representing the academic year. The first code drawn from each box indicated allocation to the intervention group, whereas the second code was assigned to the control group. The same procedure was repeated until all participants were assigned to a group (Figure 1).

The diagram of the study

Figure 1.

The diagram of the study

In this study, no blinding was done. The statistical consultant was responsible for the random student allocation, and the study outcome evaluators and the students themselves were aware of the group allocation.

3.4. Intervention

3.4.1. First Stage: Sampling

After the necessary permits were obtained from the Research Vice-Chancellor of AJUMS (U-02150), sampling was started. First of all, a 2-hour briefing session about the general objectives of the research and implementation of the MSF process was held in the presence of students and faculty members of the Anesthesia Nursing Department, and the students were informed about the general process of the research. After informed written consent was obtained from the participants, they were allocated into intervention and control groups.

3.4.2. The Second Stage: Selection of Evaluators

At this stage, three evaluators, including an anesthesiologist, a nurse anesthetist, and a nurse anesthesia instructor, were selected for an MSF evaluation in each teaching hospital. The inclusion criterion for the anesthesiologist and the nurse anesthetist was to have at least 5 years of clinical work experience, and the inclusion criterion for the nurse anesthesia instructor was to have at least 2 years of experience in nursing anesthesia teaching in addition to clinical work experience. These individuals were invited to a face-to-face meeting with the research team and were fully briefed on the study objectives and method and on how to complete the checklists.

3.4.3. The Third Stage: Implementation of MSF

The MSF evaluation process was carried out in a 3-month period of the academic semester. To this aim, at the end of each month, the evaluators, who included an anesthesiologist, a nurse anesthetist, and a nurse anesthesia instructor, monitored and evaluated the student's performance from the perspective of professionalism in both intervention and control groups, using the researcher-made checklist. In addition to these evaluators, the students themselves performed self-evaluation as another source of MSF evaluation using a valid and reliable tool. Therefore, there were four evaluation sources in total. At the end of each month, based on the results obtained from the evaluation, feedback was provided online to the students in the intervention group. In this way, the important points about students' strengths and weaknesses were highlighted on the checklists completed by 3 external evaluators for each student. Also, appropriate solutions were proposed through text and voice messages sent exclusively to them through Telegram messenger. In addition, in case the students received lower scores, effective solutions were proposed by one of the faculty members of the Nursing Anesthesia Department in person to overcome the student's weaknesses and encourage them to improve their performance in the internship. Also, the mean professionalism scores of the entire group of students were announced to them at each stage so that each student could compare his or her score with the maximum, minimum, and mean scores of the group. It is worth mentioning that in the mentioned time period, the evaluations of all students were analyzed by the research team, and lists of the common problems of professionalism were prepared and provided to the students at each of the 3 stages so that they could solve their problems with more appropriate knowledge. After each feedback, the students in the intervention group could raise any possible problems or ambiguities with the research team through messengers or in person and receive answers. On the other hand, the students of the control group did not receive feedback in this way, but they received routine training and verbal feedback in person, as they used to during their internship. The allocation of students into groups was concealed from the students until the end of the first month. The blinding of the evaluators continued until the end of the study.

3.5. Instruments

The data collection tool in this study had two sections:

(1) MSF checklist: Since there was no appropriate tool for evaluating the level of professionalism of nursing anesthesia students by others, the research team developed a tool for this purpose. This tool included 25 items scored based on a 5-point Likert scale (1: Very low, 2: Low, 3: Moderate, 4: high, and 5: Very high) with a minimum score of 25 and a maximum score of 125. To develop this tool, several reference books and articles addressing professionalism in anesthesia were reviewed (2, 15-17). Then, a draft, including a set of items, was prepared. To evaluate content validity, the tool was sent to 10 experts in this field, including 3 anesthesiologists, 4 nurse anesthetists, and 3 faculty members of the Nursing Anesthesia Department. They assessed how relevant each item was. Afterward, the content validity index (CVI) and content validity rate (CVR) were calculated for each item. The CVI obtained for all questions was 1. As with CVR, the value obtained for all items was greater than the number in Lawshe's table (0.62). Therefore, all questions remained intact, indicating that all the questions were fully valid. Also, to ensure face validity, 10 experts were asked to check the wording, structure, and format of the questions. The reliability of the tool was checked using the test-retest method. To this aim, after a ten-day interval, all evaluators evaluated the performance of one student using this tool, and the process was video recorded. According to the results obtained, the evaluation of all evaluators had good reliability. That is, the reliability value for the anesthesiologist evaluator was 0.71, that of the nurse anesthetist was 0.77, and the reliability value obtained for the nurse anesthesia instructor was 0.70. The reliability of the tool was acceptable for all evaluators since all these values were greater than 0.65. It should be noted that this tool had several open-ended questions in the final section in order to guide the type of feedback given to the students. These questions include: How do you evaluate the overall professional performance of the student? What are the main strengths and weaknesses of a student as far as professionalism is concerned? What advice do you have to strengthen the student's professional performance?

(2) Self-evaluation tool: In this study, students used a valid and reliable tool called the Professional Self-description Form (PSDF) for professional self-evaluation. This tool contains 21 items organized in 4 dimensions (Professionalism, Trait of character, Scientific knowledge, and Empathy) scored based on a 7-point Likert scale (1 = absolutely worst of all to 7 = rarely incomparable with others). The minimum and maximum scores are 21 and 147, respectively. A study conducted in Sweden to evaluate the professionalism of nurse anesthetists obtained a Cronbach's alpha of 0.96 for PSDF (18). Also, the use of PSDF has been confirmed in studies on nurses in the United States (19). In our study, in order to achieve content validity, the necessity of each question was checked by the content validity ratio (CVR). After the objectives of the tool were explained to a group of experts consisting of anesthesiology faculty members, anesthesiologists, and medical education experts (n = 16), they were asked to rate each question using a three-point Likert scale: "Necessary," "useful but not necessary," and "not necessary. “Then, the CVR was calculated. The content validity index (CVI) was used to ensure the relevance and clarity of the questions. To calculate the CVI, a panel of experts was asked to rate each question on a four-point Likert scale (irrelevant, need for fundamental revision, relevant but need for revision, and completely relevant). After the faculty members' views were collected, using the formula, the CVR and CVI of each question were obtained. The CVR and CVI for the overall tool were 0.81 and 0.95, respectively. Also, after content validity evaluation, the tool's face validity was checked by giving it to 20 students eligible to enter the study but not among the study participants. The research team sought the students' viewpoints regarding item difficulty and relevance, the relationship between items and the main objective, item ambiguity and misinterpretations, and/or incomprehensibility of the meaning of words. The tool's reliability was measured using test-retest and Cronbach's alpha methods. The tool was filled out twice by the same 20 students with a one-week interval, and the obtained Pearson correlation coefficient and Cronbach’s alpha were 0.87 and 0.91, respectively.

3.6. Data Analysis

Data were collected in Excel 2019 and analyzed using SPSS ver. 22. Mean and standard deviation were used to report quantitative data, whereas qualitative data were described using frequency and frequency percentages. The homogeneity of demographic variables in both groups was examined using the chi-square test and Fisher's exact test. To examine the effect of multi-source assessment on the level of professionalism of students over time, repeated measures of MANOVA were employed. P-Values < 0.05 were considered statistically significant.

3.7. Ethical Consideration

Ethical approval for this study was received from the Ethics Committee of Ahvaz Jundishapur University of Medical Sciences (AJUMS) (IR.AJUMS.REC.1402.256). The study was conducted according to the provisions of the 2013 Declaration of Helsinki. The potential participants were briefed on the study objectives, procedures, and conditions. All participating students signed written informed consent forms. Data confidentiality and anonymity of the students were guaranteed throughout the study process.

4. Results

4.1. Participants’ Characteristics

During the initial eligibility evaluation, 2 students declined to participate. Finally, 46 students completed the study. They were allocated to either the intervention (n = 23) or control (n = 23) groups. Most students in both groups were in the age range of 20 – 21 (~82%) years and others were in the age range of 22 - 24 (18%). About 59% of the participants were women, and nearly 94% were not married. According to the demographic information obtained from the participants, including age, gender, marital status, and academic term, there was no statistically significant difference between the intervention and control groups (P > 0.05) (Table 1).

4.2. Changes in Professionalism Status

At each measurement stage, the mean score of professionalism in the intervention group was consistently higher compared with the control group. The lowest mean and standard deviation of the total professionalism score was related to the first evaluation stage (259.26 ± 19.70) in the control group, and the highest was related to the third evaluation stage (361.04 ± 23.43) in the intervention group. The lowest score of professionalism was obtained from the anesthesiologist as an evaluator (58.95 ± 4.51) in the control group at the first evaluation stage, and the highest score was related to self-evaluation in the intervention group at the third evaluation stage (101.82 ± 5.59) (Table 2).

In the next step, a statistical analysis was performed to determine the significance of this difference. The obtained results showed the significant effect of time on the student professionalism score obtained based on MSF (P < 0.001). Notably, the MSF score demonstrated an upward trend over time. The relationship between time and group (intervention and control) yielded significant results across all evaluators (P < 0.001). Put simply, the pattern or trajectory of changes in the MSF score over time exhibited notable differences between the intervention and control groups (P < 0.001). Specifically, the two groups experienced an upward trend in the MSF score, yet the control group displayed a slight increase, while the intervention group demonstrated a significantly steeper rise. Intergroup effect tests consistently revealed that across all evaluators, the intervention group outperformed the control group (P < 0.001) (Table 3).

5. Discussion

This study investigated the effect of MSF on the level of professionalism of nurse anesthesia students. The results showed a significantly greater improvement in the level of professionalism in the intervention group as opposed to the control group. According to the students' scores, this type of evaluation helped them to improve their professional level, become better aware of their strengths and weaknesses, and be able to work on them accordingly. The 360-degree evaluation and multi-source feedback provide students with a broader view of their professional performance and enable them to make the most of the real clinical environment during their internship (20).

In the present study, evaluation of students' professional performance was done objectively and formatively under different conditions by pre-determined evaluators, and the appropriate feedback was provided to the intervention group led to a further improvement in the level of professionalism compared to those in the control group. This finding reveals the importance of using formative assessments in the form of MSF for better training of students in clinical environments, helping them to acquire practical skills, improving their professional responsibility, and establishing appropriate professional communication with others. Since the quality of student professionalism education is usually measured based on performance results (21), policymakers and administrators who are involved in the clinical education of students are required to adopt more appropriate strategies for better education and evaluation. Therefore, the use of innovative models that are based on the workplace (e.g., 360-degree evaluation) and providing MSF are necessary for teaching and assessing professionalism. This is because regular assessment of students' professionalism is closely related to the acquisition of necessary skills related to their profession (22).

Our results are consistent with those of studies conducted in the United States investigating the use of MSF as a method to evaluate the level of professionalism of anesthesiology residents, where the evaluation scores of the intervention group grew more significantly compared with the control group (23, 24). It has also been suggested that the detailed design of the MSF method can strengthen standardized assessment, exerting a positive effect on students' communication skills and professional behavior (23). Also, the results of the present study are consistent with those of a previous study conducted on nurses (25), but to the best of our literature review, we could not find any study examining the effect of the MSF method on nurse anesthetists and students of nurse anesthesia.

Adoption of the MSF method should not be limited only to the evaluation of student professionalism within the internship environment. Rather, this method could be used to influence other important indicators, such as professional communication in academic and medical environments, and to measure the degree of student's mastery of procedural anesthesia skills used in clinical training environments (8). On the other hand, one of the reasons for the low acceptance of the MSF method is the difficulty of coordination between evaluators due to the changing conditions in clinical environments, which leads to the limited use of this method in internship environments (26). However, by making well-considered decisions, this shortcoming could be alleviated, and MSF could be implemented in an ideal form in internship environments.

Also, according to the results of this study, the MSF method can be combined with other evaluation methods such as OSCE, DOPS, etc. Internship and university environments can be improved in this way, and future studies can investigate the integration of the mentioned evaluation methods and their impact on various indicators (e.g., socialization) in nursing anesthesia and other medical fields.

Another important consideration regarding the use of the MSF is the length of time it takes to be implemented (20). Due to time constraints, this study involved only 3 months of student evaluation, but if this method is implemented over a longer period of time, we will probably observe a better improvement in the level of student professionalism and performance, and this can be investigated in future studies. Another ambiguity of the MSF method is the number of evaluation sources (25). In this study, we used 4 evaluation sources. To clarify the uncertainties surrounding this method, new evaluation sources such as patients as individuals receiving medical services, the operating room manager, and other staff can be investigated in future studies. Of course, it is clear that with the increase in the number of evaluation resources, new challenges such as difficult coordination between different people, lack of suitable tools for others to evaluate, and bias in the study process will be created.

Also, in this study, the students were used as evaluation sources. This was done because the students had a better understanding of themselves, and their general views on different indicators could be elicited. This makes it possible to use their information for curriculum planning and enhancing the quality of student internships in clinical environments. In addition, it is possible to use students' self-evaluation opinions as a basis for grading their performance, which may increase their motivation for better training in an internship. Of course, it should be acknowledged that the students might overestimate their performance in their self-evaluation, and they might give themselves a higher score than their actual score.

5.1. Conclusions

The results showed that the MSF evaluation process had a statistically significant effect on the level of professionalism of nurse anesthesia students. It is possible to benefit from integrating this method with other evaluation methods in future studies, but it should be noted that the accurate implementation of this method requires proper coordination between evaluators and students, which makes the method difficult to implement.