A Comprehensive Analysis of Graduated Residents’ and Faculty Members’ Attitudes on Iranian Radiation Oncology Board Examination: A Cross-sectional Study

1. Background

In one way or another, the assessment of residents’ knowledge and competence is a necessity before entering practice (1, 2). To that end, policymakers have tried to improve the assessment tools worldwide. For instance, the Accreditation Council for Graduate Medical Education (ACGME) introduced six criteria (i.e., the core competencies) to shape and evaluate the education of residents as follows: (1) practice-based learning and improvement; (2) patient care and procedural skills; (3) systems-based practice; (4) medical knowledge; (5) interpersonal and communication skills; and (6) professionalism (3, 4). In this regard, several education assessment methods have been developed for both undergraduate and postgraduate residents, which can be classified into four categories as following: (1) written exercises [e.g., multiple-choice questions (MCQ) examination and script concordance test (SCT)]; (2) assessment by supervising clinician [e.g., mini clinical evaluation exercise (mini-CEX)]; (3) clinical simulations [e.g., objective structured clinical examination (OSCE)]; and (4) multisource assessment (e.g., peer and patient assessments). All aforementioned methods have intrinsic strengths and limitations (1). Therefore, applying multiple assessment methods can partially compensate for limitations in any method (5, 6). Accordingly, the Iranian radiation oncology board examination (IROBE) contains two components: (1) MCQ, and (2) OSCE. The current structure of IROBE was initially held by the secretariat of the Council on Medical Education in 2000. Although the simplified nonstandard blueprint for MCQ and OSCE have been used until recent years, active monitoring is a necessity to find the strength and flaws and enhance their quality. As far as the researchers investigated, this is the first study to evaluate the strengths and flaws of IROBE from both board exam constructors’ and participants’ views to provide a basis for the IROBE and improve its quality.

2. Methods

2.1. Ethical Considerations

Before commencing the study, ethical clearance was obtained from the Institutional Review Board. In this regard, all surveys were completed anonymously over all phases of data collection, analysis, interpretation, and presentation. Also, no personal health information or personally identifiable information was collected.

2.2. Participants

Two groups were included: (1) faculty members (FMs) of radiation oncology from seven medical universities of Iran who participated in designing the questions of IROBE (at least one time) between 2015 and 2019; and (2) graduated residents (GRs) of radiation oncology from six medical universities of Iran who participated in IROBE (at least one time) between 2015 and 2019. By considering the level of confidence (z) equal to 95%, the margin of error (E) equal to 5%, variance of the population (p) equal to 50%, and population size of 35 (for FMs) and 107 (for GRs), the sample sizes (n) for FMs and GRs groups were estimated as 32 and 84, respectively, using the following formula (7, 8):

2.3. Instruments

After a comprehensive review of the literature, two preliminary questionnaires were designed (by AA) containing 22 and 29 questions to evaluate the perspectives of FMs and GRs towards IROBE, respectively. The questionnaire was modified by three FMs in medical education (SA, MAL, and AH), and the questions were categorized into eight (for faculties) and nine (for examinees) domains to better evaluate the attitudes towards the IROBE. While the first six questions aimed at defining the general characteristics (e.g., gender, age, university of origin, etc.), other ones addressed other domains of interest, including: (1) how much IROBE is efficient in evaluating clinical abilities; (2) what is the indexed component of IROBE; (3) how is the quality of IROBE; (4) how the university of origin and examinees’ age or gender affect the IROBE results; (5) how effective is having a board certification (or ranking) in faculties’ attitude for faculty recruitment of GRs; (6) GRs’ personal attitudes on IROBE; and (7) the shortcomings of IROBE. A final open-ended question allowed participants to offer feedback on issues not covered in the survey.

2.4. Data Collection

Considering the lockdown situation due to the novel coronavirus disease (COVID-19 pandemic) during the time of conducting the study, we developed an online questionnaire and shared it with the candidates using the WhatsApp© application (Copyright 2020 WhatsApp Inc., Menlo Park, CA, USA) through cellular phones. Continuous weekly reminders were sent via the same application to ensure optimal participation.

2.5. Endpoints

The main outcome of interest was a comprehensive evaluation of both GRs’ and FMs’ attitudes towards IROBE. In this regard, the comparative analysis is provided for the common questions. Also, we analyzed the GRs’ performance on the IROBE according to their age, gender, and marital status.

2.6. Statistical Analysis

To summarize the data, we used frequencies (percentages) and means (standard deviation and ranges) for categorical and continuous variables, respectively. To find an association between the study endpoints and ordinal (or nominal) variables, we applied the Mann-Whitney U test (or Pearson's chi-square test). In addition, we used the Kolmogorov Smirnov test to determine the normal distribution of variables. A P-value of ≤ 0.05 was considered significant. All analyses were performed using IBM® SPSS® Statistics (version 26.0).

3. Results

The survey was carried out in several days between August 1, 2020, and August 16, 2020. Out of a total of 35 FMs and 107 GRs, 16 (45%) FMs and 42 (39%) GRs completed the survey. Participants were excluded from the analysis if they chose the ‘prefer not to say’ choice for the attitude questions. In this regard, no participant was excluded from the study. Overall, 18.7% of FMs and 59.5% of GRs were females. The age of FMs ranged from 38 to 60 years with a mean age of 53.5 ± 7.6 years, and the age of GRs ranged from 30 to 41 years with a mean age of 33.4 ± 2.8 years. The majority of FMs were from Shahid Beheshti University of Medical Sciences (SBMU) and Shiraz University of Medical Sciences (SUMS) (25.0 and 18.75%, respectively). Meanwhile, the majority of GRs were from SBMU and Tehran University of Medical Sciences (TUMS) (50.0 and 16.66%, respectively). The remaining demographic data of participants are detailed in Table 1.

There were five questions regarding clinical abilities [How would you rate the IROBE in evaluating (1) clinical competence, (2) clinical knowledge, (3) clinical performance, (4) medical ethics, and (5) responsibility?]. The FMs’ and GRs’ opinions on the capacity of IROBE for the aforementioned topics (based on 4-point Likert scale questions) were almost similar, except for the item of clinical performance (Table 2, Category A). In this regard, the rate of FMs and GRs who believed that IROBE has an average to poor quality was 62.5 vs. 76.1% in clinical competence (P = 0.07), 62.5 vs. 71.4% in clinical knowledge (P = 0.19), 68.8 vs. 88.3% in clinical performance (P = 0.02), 100 vs. 92.9% in medical ethics (P = 0.21), and 93.7 vs. 95.3% in responsibility (P = 0.15), respectively. The Importance of IROBE components (MCQ or OSCE) in evaluating clinical skills was asked from both groups (which IROBE component has a greater role in evaluating knowledge/performance?). For evaluating clinical knowledge, 25% of FMs and 28.5% of GRs voted for MCQ, and 25% of FMs and 23.8% of GR voted for OSCE. One-third of GRs believed that no component of IROBE can evaluate clinical knowledge, while just 6.25% of FMs were in agreement with them (P = 0.05). In this regard, 43.7% of FMs believed that MCQ and OSCE have the equal capability in evaluating clinical knowledge. Regarding clinical performance, there was also disagreement between FMs and GRs, so that 68.7% of FMs chose OSCE as a better test for evaluating clinical performance while 54.7% of GRs chose neither MCQ nor OSCE to be able to evaluate the skill (P = 0.03) (Table 2, Category B). Then, participants evaluated the necessity for improving the general quality of the IROBE (How much IROBE-MCQ/OSCE requires improvement?). Most of the FMs and GRs were in agreement that IROBE requires high to extreme improvement in both MCQ (75 vs. 59.5%) and OSCE (87.5 vs. 90.4%) components (Table 2, Category C). The last common question evaluated the importance of some potential confounding factors on examinees’ results in IROBE, including university, gender, and age. In case of the university of origin (how much examinee’s university affects the pass in IROBE-MCQ/OSCE/Rank?), most of FMs and GRs agreed that it may have a medium to extreme effect on the IROBE results (62.5 vs. 78.5% pass in MCQ, 68.7 vs. 83.3% pass in OSCE, and 62.5 vs. 73.7% obtaining high rank, respectively) (Table 2, Category D). In contrast, most of the FMs and GRs acknowledged that gender has a negligible to low effect on these items (87.5 vs. 80.9% pass in MCQ, 87.5 vs. 76.1% pass in OSCE, 81.2 vs. 76.1% obtaining high rank, respectively) (Table 2, Category E). Although more than half of the FMs considered GRs’ age to have a medium to high effect on the pass probability and obtain high-rank, most of GRs did not believe so (Table 2, Category F).

Table 3 summarizes the FMs’ responses to four specific questions in two domains. The first question considered the effect of having a board certification on the FMs’ decision for recruitment of GRs as a faculty of a university (Table 3, Categories A3-A1). All FMs except for 2 (87.5%) stated that it has a medium to extreme influence on their decision. The results of Pearson's chi-square test demonstrated no association between the FM’s decision and their gender (P = 0.82), experience (P = 0.94), academic degree (P = 0.18), and university of origin (P = 0.98). Likewise, most (81.25%) FMs believed that a high rank in IROBE has a medium to extreme effect on their decision for recruiting GRs as a faculty (Table 3, Categories A3-A2). Based on association analysis, FMs’ gender was associated with this decision (P = 0.004), so that 3 (100%) female and 5 (38.4%) male FMs stated that high-ranking in IROBE has a high to extreme effect on their decision to employ GRs as a faculty. The other two specific questions from the FMs dealt with the shortcomings of IROBE over the last five years. In FMs’ opinion, the major limitations of the MCQ component of IROBE were as follows: (1) lack of motivation (75.0%); (2) lack of familiarity (62.5%); (3) lack of experience (56.25%) for designing high-quality questions; and (4) lack of centralized question design committee (31.25%). In addition, in the case of OSCE examination, the major limitations were as follows: (1) lack of familiarity (81.25%); (2) lack of motivation (62.5%); (3) lack of executive facilities (43.75%); and (4) lack of experience (31.25%).

The GRs’ opinions regarding the 11 specific questions in the three domains are presented in Table 4. The first domain evaluated the importance of IROBE in their personal lives. Most of the participants stated that passing the board examination (92.8%) and obtaining a high rank (66.7%) had medium to extreme importance for them. In line with these, 73.8% of GRs declared that passing the IROBE caused medium to extreme worry in them, and 85.7% believed that having a board certification has a medium to extreme effect on their future career opportunities. These beliefs had incremental impacts over the residency period in half of the GRs (Table 4, Category A). The next domain dealt with the IROBE shortcomings. According to the GRs’ opinions, the major limitations of the MCQ component of IROBE were as follows: (1) high rate of impractical questions (54.7%); (2) the involvement of faculty’s personal comment in answers (52.3%); and (3) poor-quality and equivocal questions (each 40.4%). Concerning the OSCE examination, the major limitations were designing descriptive (rather than OSCE-standardized) questions (69.0%) and inability to evaluate the clinical performance (61.9%), medical ethics (50.0%), and responsibility (50.0%) (Table 4, Category B). About 69.0, 76.1, and 76.1% of GRs declared that the number of FMs’ of the GRs’ universities - participating in the board examination - had a medium to extreme effect on their results in MCQ, OSCE, and final ranking, respectively (Table 4, Category C).

Finally, we evaluated the association of GRs’ demographic information in their success in IROBE. The characteristics were similar between the residents who had passed the IROBE and those who had not passed it; however, male participants were more likely to pass the IROBE (100 vs. 76%, P = 0.02). In addition, no association was found between obtaining a high rank in IROBE and residents’ gender (P = 0.44), age (P = 0.32), university of origin (P = 0.47), and marital status (P = 0.74) (Table 5).

4. Discussion

In this survey, most FMs and GRs agreed that IROBE still has average to poor quality in evaluating the clinical competence, clinical knowledge, medical ethics, and responsibility, and requires major improvement in both MCQ and OSCE components. While GRs did not realize either IROBE components to have enough capacity to evaluate clinical knowledge and performance, most FMs had a contrasting opinion. Both groups believed that the university has major effects and gender has minor effects on success in IROBE. However, in contrast to the GRs, the majority of FMs considered age as a determining factor in success in both MCQ and OSCE exams. Interestingly, most GRs acknowledged the participation of their university’s FMs as a contributing factor for their success in IROBE.

Most GRs highlighted the poor quality of both MCQ and OSCE examinations for including many impractical and equivocal questions. This is possibly due to the lack of motivation, familiarity, and experience to design questions that were common among FMs’ responses to the question regarding IROBE limitations. More than half of the GRs stated that the IROBE-MCQ contained many impractical questions, and about 60% of GRs complained about the inability of IROBE-OSCE in evaluating clinical performance. To cope with these limitations, about one-third of GRs proposed to increase the number of case-study-based questions in the MCQ exam, and about 70% of them agreed to switch the descriptive questions into OSCE-standardized ones. These changes can potentially enhance the GRs’ clinical reasoning to cope with the possible future clinical dilemmas, as stated by Albert Einstein: “Education is not learning the facts but training the mind to think” (9). To that end, developing the comprehensive test blueprint with weightings that emphasize clinical radiation oncology is suggested (10). Moreover, the questions of medical physics and radiation biology can be designed in line with the ACGME core competencies (10). On the other hand, approximately 20% of GRs complained about the short time for answering the OSCE questions. This might stem from their lack of preparation, and most probably shortcomings of OSCE component of IROBE in comparison with standard OSCE. Running mock OSCE or its alternatives (such as peer-led multi-role practice OSCEs) during residency could help FMs for designing high-quality OSCE and prepare GRs for the exam (11, 12).

While a larger proportion of FMs compared to GRs (31.2 vs. 11.9%) believed that IROBE-OSCE could effectively evaluate the clinical performance, most participants of both groups believed that the existing IROBE could not efficiently evaluate all aspects of clinical abilities (including clinical competence, knowledge, medical ethics, and responsibility). Hence, emergency action is needed to improve the quality of IROBE.

Most GRs declared that IROBE has majorly affected their personal lives and caused so much concern for them. This condition has also been reported in overseas radiation oncology residents (13). It has been shown that the stress experienced by medical students might induce high rates of burnout and depression (14). The residents’ concern might stem from the FMs’ attitudes that having a board certification, as well as a high rank in IROBE, have great importance for employing new faculties. This level of stress emphasizes an emergency request for improving the quality of IROBE again.

The current study had some limitations. Firstly, the findings may have been confounded by the small number of participants. Secondly, participants were inquired about their experience for constructing or participating in IROBE, which was related to few months to five years earlier. This may subject the answers and comments to recall bias. Although these limitations are important, the current study is the first to seek the strengths and flaws of IROBE.

4.1. Conclusion

The results of the present study revealed that the current structure of IROBE has several major drawbacks and requires a comprehensive revision in both MCQ and OSCE components. Both FMs and GRs had somehow similar ideas in this regard. The IROBE and Education Deputy of the Ministry of Health can consider these findings to enhance the board examination purpose, shape, and experience for both faculties and residents.