1. Background
The core competencies defined by the Accreditation Council for Graduate Medical Education (ACGME) need an arsenal of medical education methods to thrive, especially in such stress loaded fields like anesthesiology (1-4). The “simulated-patient scenarios and role-playing” method is among the many non-traditional education methods with variable results in different clinical settings (5-7).
On the other hand, the Objective Structured Clinical Examination (OSCE) is a modern objective assessment technique in medical education with a systematic methodology and variable results. Although both “simulated-patient scenarios and role-playing” and OSCE methods are usually objective assessments, there is a paucity of data about any potential correlation in-between. In other words, there are not enough studies to demonstrate the correlation between the results of these two important methods (8-11). However, if we would find a significant correlation, this correlation between the results of the two assessment methods could provide evidence regarding the criterion validity (either in the form of concurrent validity if the two exams are rather simultaneous, or predictive validity if the gold standard is conducted later on).
In addition, some have expressed concerns regarding the validity and reliability of role-playing as an assessment method instead of a real situation (12-17). Considering the standards and international accord regarding the validity and reliability of OSCE, we sought to demonstrate if there is any objective relationship between the results of these two assessment methods. Thus, this study was designed and implemented to assess any potential relationship between these two objective assessment methods, considering that each of them would assess some ACGME core competencies (many in common). If such a hypothesis would be proven, the two methods can be used interchangeably after taking considerations regarding other aspects of validity, such as content validity.
2. Objectives
This observational analytic study was designed and performed to assess the correlation between the results of “simulated-patient scenarios and role-playing” and OSCE methods among senior clinical anesthesiology residents, with a special focus on four of the six ACGME core competencies (i.e., medical knowledge, practice-based learning and improvement, interpersonal and communication skills, and professionalism) in the Anesthesiology Department of Shahid Beheshti University of Medical Sciences (SBMU), Tehran, Iran. The four selected competencies were much more relevant in this study.
3. Methods
The study protocol was approved by the Institutional Review Board (IRB), Research Deputy, Shahid Beheshti University of Medical Sciences, Tehran, Iran. All the participants (i.e., residents who were assessed in the role-playing arm of the study) were informed that they were going to be assessed using this method. In addition, if any of them was reluctant to continue the study, his/her results were withdrawn from the study. Besides, none of the results was assessed in a personal manner. On the other hand, all of the faculty members who took part in the study filled out their scoring checklists anonymously, and they were free whether to take part in the study or not.
This cross-sectional study was designed and performed to evaluate the potential correlation between OSCE and role-playing results concerning the performance of anesthesiology residents. During two years (2018 and 2019 educational years), a general clinical case discussion class was weekly held each Monday afternoon. The senior anesthesiology residents (CA-4 or CA-3) were subject to “simulated patient scenario and role-playing” sessions. A real case scenario based on previous real patients was presented by a junior resident (either CA-1 or CA-2) that was supervised by an attending anesthesiologist who was in charge of the patient in the real clinical setting. The selection of the clinical topics was agreed upon the list of more common daily clinical challenges for the residents. The study sample included all CA-3 and CA-4 residents; indeed, there was no sampling, and all the residents were included in the study.
The role of the patient in the scenarios was played by two faculty members (a man and a woman), or if needed, one of the relatives of the patient, e.g., a parent or guardian of a child or a patient unable to talk directly. A checklist was prepared to assess the performance of senior anesthesiology residents (CA-4 or CA-3), which was standardized through the following steps (Table 1):
1. A primary draft was prepared based on previous studies, professionalism criteria, and interpersonal communication skills.
2. The face and content validity of the checklist was assessed by five faculty members of the Anesthesiology Department of SBMU.
3. A list with 15 items was finalized.
4. A five-point Likert scale was used for rating the 15 items, including “strongly agree = 5”, “agree = 4”, “neutral = 3”, “disagree = 2”, and “strongly disagree = 1”.
5. To test the questionnaire’s reliability, Cronbach’s alpha was calculated at the end of the study.
6. To improve the quality of the rating, the checklists were filled out anonymously. Although the faculty member knew the examinees and observed them during the test, they did not mention their names under the assessment checklist or did not sign it to compensate for one of the potential sources of bias.
Table 1.Study Questionnaire Translated from Persian to English
| The Stem of The Question | Strongly Agree | Agree | Neutral | Disagree | Strongly Disagree | |
|---|---|---|---|---|---|---|
| Q1 | The resident adhered strictly to ethical principles while talking to the patient (or patient companion). | |||||
| Q2 | The resident showed enough self-esteem in dealing with the patient (or patient companion) | |||||
| Q3 | The resident spent enough time to take the history of the disease. | |||||
| Q4 | The resident performed necessary examinations during the patient visit. | |||||
| Q5 | The resident delayed the patient more than usual before performing anesthesia on the operating room bed. | |||||
| Q6 | The resident induced unnecessary excessive stress to the patient (or patient companion) | |||||
| Q7 | The resident correctly presented his/her professional and scientific status to the patient (or patient companion). | |||||
| Q8 | The resident communicated the information calmly and realistically when declaring the patient's clinical condition. | |||||
| Q9 | The resident conveyed to the patient (or patient companion) sufficient information about anesthesia. | |||||
| Q10 | The resident provided the patient (or patient companion) with information about the procedure. | |||||
| Q11 | Medical explanations provided by the resident to the patient (or patient companion) were unnecessary. | |||||
| Q12 | The resident provided the patient (or patient companion) with adequate explanations of possible complications and risks in the perioperative period. | |||||
| Q13 | The resident recorded informed consent in the patient's file. | |||||
| Q14 | The resident coordinated with the responsible anesthesiologist for anesthesia. | |||||
| Q15 | The resident introduced the anesthesia team (including responsible attending, other anesthesia residents, the anesthesia nurse) to the patient (or patient companion). |
The performance of each senior anesthesiology resident (CA-4 or CA-3) was rated as follows:
1. The simulated patient scenario was presented by a CA-1 or CA-2 resident supervised by a faculty member.
2. One of the faculty members played the role of the simulated patient who was a candidate for an elective or emergent anesthesia plan to undergo a surgical procedure.
3. Another faculty member played the role of “a patient’s relative” or “a guardian of a child” in the sessions that were to assess pediatric patients.
4. The senior anesthesiology resident (CA-4 or CA-3) interviewed and handled the patient.
5. Those faculty members who observed the session rated the senior anesthesiology resident (CA-4 or CA-3) objectively and anonymously, based on the finalized checklist (Table 1).
6. At least seven faculty members of the Anesthesiology Department of SBMU took part in each of these rating sessions. There was no upper limit for the observer faculty members to rate the checklist, as, in one of the sessions, a ceiling of 19 faculty members was touched.
Meanwhile, an ordered pattern of OSCE was defined by the technical team of the Education Deputy, Anesthesiology Department, SBMU, to cover the four competencies, i.e., medical knowledge, system-based practice, interpersonal and communication skills, and professionalism. This OSCE was prepared in 10 stations, focusing on a range of clinical common challenges, covering the daily challenges of anesthesiology residents. The OSCE was performed exactly in the midterm of the study, i.e., April 2019.
The study outcomes and variables are as follows:
● Study outcomes: The relationship between OSCE scores and the scores of faculties on the role-playing checklist.
● Exposure factors: In this study, several exposure factors were defined including the exposure variable that was the time duration for the study, which could affect the outcome, since, with time, the knowledge of the residents may have been affected; this might have effects on the outcome (assessment results). Besides, the personal attitudes of the faculty members (i.e., their attitudes) towards each trainee could be the exposure factor. However, since there was a global assessment of “role-playing” scores and they were then compared with the OSCE results, the effects of the above factors were possibly minimized, resulting in the alleviation of both non-differential and differential measurement errors (18).
● Predictors: The general performance of each resident in previous formal assessments could be considered a predictor for the results of OSCE and role-playing (19).
● Potential confounders and effect modifiers: They included the atmosphere of the role-playing session.
The mean and standard deviation of each of the 15 items in the checklist were calculated. All checklist scoring results were also accumulated to calculate the correlation between the checklist assessments and the semester-length scores. Another correlation was calculated between the scores on the “simulated patient scenario and role-playing” checklists gained by each resident and the individual OSCE exam results; in other words, the results of each resident on “simulated patient scenario and role-playing” were compared with his/her results on the departmental OSCE exam. To compare the results, both the results of the OSCE exam and the scores of the “simulated patient scenario and role-playing” checklist were calculated based on “hundred scores”. Usually, the assessment ranks are performed with different scoring scales; however, we aimed to compare the results of the two different assessment methods (i.e., OSCE scores and the scores of “simulated patient scenario and role-playing” checklist) with each other using different scales. Therefore, using a mathematical calculation, we adjusted the results of the two assessment methods to make the comparisons easier.
4. Results
From 2018 to 2019, a total of 40 senior anesthesiology residents (CA-4 or CA-3) entered the study (CA-4 = 19 residents, CA-3 = 21 residents). Besides, 344 assessments done by faculty members were collected in 40 sessions, with a minimum of 7 and a maximum of 19 faculty members who took part in rating the senior residents, yielding a mean of 8.6 raters per each senior resident (i.e., 8.6 faculty members rated in each session). The questionnaire’s Cronbach’s alpha reliability was 0.74.
There was no statistically significant difference between the results of the aggregated OSCE exam (71 ± 17) and the scores of the “simulated patient scenario and role-playing” checklist (74 ± 16) using the independent t-test as cumulative results (P value < 0.01). Of course, as mentioned in the methods section, both the results of the OSCE exam and the scores of the “simulated patient scenario and role-playing” checklist were modified and calculated on the basis of “hundred scores” to facilitate the comparisons.
In addition, for each of the senior anesthesiology residents (CA-4 or CA-3), there was an individual significant correlation between the results of the OSCE exam and the scores of the “simulated patient scenario and role-playing” checklist. In addition, there was a similar correlation for the cumulative results of the two methods (two-tailed correlation coefficient = 0.886; N = 40; P value < 0.001).
5. Discussion
This study demonstrated that the “simulated patient scenario and role-playing” method in a departmental educational session, when done in a regular pattern, is in strong concordance with OSCE results of the same population. Considering this finding and keeping the role of OSCE in mind, we may hypothesize that simulated patient scenario and role-playing sessions could improve the performance of the anesthesiology residents regarding the ACGME core competencies, a hypothesis mandating a different methodology and another study (such as a pre- and post-design study). However, for gaining such a degree of generalizability (external validity) of the study results, more complementary studies are mandatory.
In addition, there was a good correlation between the OSCE examination scores and the scores of simulated patient scenarios and role-playing sessions. This is while both of these assessments were done using objective measures. The questionnaire reliability (Cronbach’s alpha) was at an appropriate level, indicating an acceptable reliability level of the rating checklist.
The six core competencies of ACGME include patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. Among the six core competencies, the four latter competencies were much more relevant in the study, and this is why we chose them (2, 3, 20). Throughout the clinical practice, these competencies that focus far beyond just medical knowledge should be built up using a systematic training approach. In addition, they should be measured in a formative assessment and robust evaluation model. The latter approach would lead to an appropriate process of “clinical thriving” in medical education of residency programs. These studies are in favor of our research; however, some studies mandate further assessments to evaluate the psychomotor and behavioral skills of trainees (7, 21).
However, in some anesthesiology residency programs, a try and error pattern is used instead of a systematic approach, which would not thrive the capacities needed for trainees, including professionalism, especially considering the relatively high stress loaded fields of clinical anesthesiology (1-3, 7, 22). This is why competency-based medical education is much more appreciated with its special focus on outcomes, capabilities, and learner-centeredness, emphasizing the final outcome of medical education. This is a specific feature that was assessed in our study and has been the focus of some previous researches, being in favor of our findings (23-27).
Although several simulation methods have been developed, the “simulated patient scenario and role-playing” method has been described as a cost-effective method with fruitful results in training. This notion supports our findings in this study, although some aspects of our study need more attention (5, 6, 28-30).
Teaching medical emergencies using a simulation-based model has been demonstrated as an effective method in improving students’ knowledge, experience, and confidence, for the management of critical events (7, 31, 32). Role-playing is one of those techniques benefiting from simulation-based models, leading to improvement in several aspects of ACGME core competencies, especially those affected by a load of stress in anesthesiology (7, 32-34). As pointed out by Dias et al., “Role-play strategies can provide high psychological fidelity for simulation-based training” (35). Besides, there are practical clinical outcomes in patient care, leading to tangible results (28, 29), although some controversies exist (30). In addition, Lambert et al. demonstrated that the Emotional Quotient (EI)-related capabilities of anesthesiology residents through simulation-based medical education would improve important aspects of ACGME competencies, promoting patient outcomes in perioperative care (7).
5.1. Study Limitations
Among the study limitations, the following could be mentioned as the main limitations:
1. Simulated patient scenarios and role-playing are techniques that can be used in teaching and learning activities or assessments. They are not stand-alone methods by themselves. In addition, role-playing and patient scenarios could be moved to an OSCE station. However, in this study, we compared the two methods that are inherently different; indeed, what we performed in this study was the comparison between the results of an assessment performed through single observation of one examinee by several examiners who scored based on a pre-designated checklist and the results of an OSCE. The former is very close to a “long case” exam, but on a simulated patient, it is not as a workplace-based assessment. Considering the settings of our study, there are some similarities between our research and an objective structured clinical viva examination (OSVE).
2. The so-called “scenario-based simulation and role-playing” method was conducted over a span of about two years, while each resident did not take part in the process repeatedly. The knowledge and skills of the examinees were bound to change over this rather long period. Each resident was evaluated only once (using the simulation model). Thus, it could have been very early or very late in the study period, and they could not be assessed in a more sophisticated way; however, the goal of the research was to perform a global and collective movement involving all residents, which may partly compensate for this gap.
3. Only four of the six ACGME core competencies were assessed; however, this was due to the design of the study. Future studies should focus on other aspects of ACGME core competencies.
4. Role-playing was the only strategy used in this research; however, other strategies and modalities used for experiential learning were not considered in this study, which should be focused on in future studies to cover the gap.
In conclusion, though there are many studies considering the simulated patient scenario and role-playing approach, scant studies have assessed the relationship between “simulated patient scenario and role-playing” strategies and the results of OSCE exams using an observer-based rating method. Our study could demonstrate an objective relationship between the results of these two assessment methods. Thus, based on our findings, these two methods could be used as surrogates for each other in the assessment of the core clinical competencies of senior anesthesiology residents.
