A Comprehensive Analysis of Graduated Residents’ and Faculty Members’ Attitudes on Iranian Radiation Oncology Board Examination: A Cross-sectional Study

authors:

avatar Soleiman Ahmady ORCID 1 , avatar Abtin Heidarzadeh ORCID 2 , avatar Ainaz Sourati 3 , avatar Maryam Akbarilakeh ORCID 4 , avatar Ahmad Ameri 5 , *

Department of Medical Education, Virtual School of Medical Education, Tehran, Iran
Medical Education Research Center, Education Development Center, Guilan University of Medical Sciences, Rasht, Iran
Department of Radiation Oncology, Guilan University of Medical Sciences, Rasht, Iran
Center for Education Research in Medical Sciences (CERMS), Department of Medical Education, School of Medicine, Iran University of Medical Sciences, Tehran, Iran
Department of Clinical Oncology, Shahid Beheshti University of Medical Sciences, Tehran, Iran

How To Cite Ahmady S, Heidarzadeh A, Sourati A, Akbarilakeh M, Ameri A. A Comprehensive Analysis of Graduated Residents’ and Faculty Members’ Attitudes on Iranian Radiation Oncology Board Examination: A Cross-sectional Study. Rep Radiother Oncol. 2020;7(1):e114502. https://doi.org/10.5812/rro.114502.

Abstract

Background:

The Iranian radiation oncology board examination (IROBE) includes a multiple-choice question (MCQ) examination and an objective structured clinical examination (OSCE). The current mission of IROBE is to qualify the graduated residents (GRs) in terms of clinical competence. However, lack of active monitoring of its construction and administration can impair the evaluation of GR competencies.

Objectives:

To determine the strengths and limitations of IROBE.

Methods:

This cross-sectional online survey involved faculty members (FMs) and GRs as constructors and participants of the IROBE, respectively. The target time window for inclusion in the study was set between 2015 and 2019. To evaluate the strengths and limitations of IROBE, a 22-item questionnaire was distributed among FMs and a 29-item questionnaire among GRs. The Mann-Whitney U test and Pearson's chi-square test were applied to find the association between the ordinal and nominal variables, respectively.

Results:

The surveys were sent to 35 FMs and 107 GRs. A total of 16 (45%) FMs and 42 (39%) GRs completed the survey. Overall, the majority of FMs and GRs stated that IROBE has average to poor quality in evaluating all aspects of clinical abilities (62.5 vs. 76.1% in clinical competence, P = 0.07; 62.5 vs. 71.4% in clinical knowledge, P = 0.19; 100 vs. 92.9% in medical ethics, P = 0.21; and 93.7 vs. 95.3% in responsibility, P = 0.15). However, FMs assigned a higher score to OSCE in the assessment of clinical performance compared to GRs (P = 0.02). Most FMs and GRs declared that IROBE requires extreme to high improvement in both MCQ and OSCE components (75 vs. 59.52% and 87.5 vs. 90.47%, respectively).

Conclusions:

According to the participants, the present structure of IROBE has several drawbacks in both MCQ and OSCE components. Considering the highlighted strengths and limitations can help the IROBE constructors to improve its quality.

1. Background

In one way or another, the assessment of residents’ knowledge and competence is a necessity before entering practice (1, 2). To that end, policymakers have tried to improve the assessment tools worldwide. For instance, the Accreditation Council for Graduate Medical Education (ACGME) introduced six criteria (i.e., the core competencies) to shape and evaluate the education of residents as follows: (1) practice-based learning and improvement; (2) patient care and procedural skills; (3) systems-based practice; (4) medical knowledge; (5) interpersonal and communication skills; and (6) professionalism (3, 4). In this regard, several education assessment methods have been developed for both undergraduate and postgraduate residents, which can be classified into four categories as following: (1) written exercises [e.g., multiple-choice questions (MCQ) examination and script concordance test (SCT)]; (2) assessment by supervising clinician [e.g., mini clinical evaluation exercise (mini-CEX)]; (3) clinical simulations [e.g., objective structured clinical examination (OSCE)]; and (4) multisource assessment (e.g., peer and patient assessments). All aforementioned methods have intrinsic strengths and limitations (1). Therefore, applying multiple assessment methods can partially compensate for limitations in any method (5, 6). Accordingly, the Iranian radiation oncology board examination (IROBE) contains two components: (1) MCQ, and (2) OSCE. The current structure of IROBE was initially held by the secretariat of the Council on Medical Education in 2000. Although the simplified nonstandard blueprint for MCQ and OSCE have been used until recent years, active monitoring is a necessity to find the strength and flaws and enhance their quality. As far as the researchers investigated, this is the first study to evaluate the strengths and flaws of IROBE from both board exam constructors’ and participants’ views to provide a basis for the IROBE and improve its quality.

2. Methods

2.1. Ethical Considerations

Before commencing the study, ethical clearance was obtained from the Institutional Review Board. In this regard, all surveys were completed anonymously over all phases of data collection, analysis, interpretation, and presentation. Also, no personal health information or personally identifiable information was collected.

2.2. Participants

Two groups were included: (1) faculty members (FMs) of radiation oncology from seven medical universities of Iran who participated in designing the questions of IROBE (at least one time) between 2015 and 2019; and (2) graduated residents (GRs) of radiation oncology from six medical universities of Iran who participated in IROBE (at least one time) between 2015 and 2019. By considering the level of confidence (z) equal to 95%, the margin of error (E) equal to 5%, variance of the population (p) equal to 50%, and population size of 35 (for FMs) and 107 (for GRs), the sample sizes (n) for FMs and GRs groups were estimated as 32 and 84, respectively, using the following formula (7, 8):

n=p 100-p z2E2

2.3. Instruments

After a comprehensive review of the literature, two preliminary questionnaires were designed (by AA) containing 22 and 29 questions to evaluate the perspectives of FMs and GRs towards IROBE, respectively. The questionnaire was modified by three FMs in medical education (SA, MAL, and AH), and the questions were categorized into eight (for faculties) and nine (for examinees) domains to better evaluate the attitudes towards the IROBE. While the first six questions aimed at defining the general characteristics (e.g., gender, age, university of origin, etc.), other ones addressed other domains of interest, including: (1) how much IROBE is efficient in evaluating clinical abilities; (2) what is the indexed component of IROBE; (3) how is the quality of IROBE; (4) how the university of origin and examinees’ age or gender affect the IROBE results; (5) how effective is having a board certification (or ranking) in faculties’ attitude for faculty recruitment of GRs; (6) GRs’ personal attitudes on IROBE; and (7) the shortcomings of IROBE. A final open-ended question allowed participants to offer feedback on issues not covered in the survey.

2.4. Data Collection

Considering the lockdown situation due to the novel coronavirus disease (COVID-19 pandemic) during the time of conducting the study, we developed an online questionnaire and shared it with the candidates using the WhatsApp© application (Copyright 2020 WhatsApp Inc., Menlo Park, CA, USA) through cellular phones. Continuous weekly reminders were sent via the same application to ensure optimal participation.

2.5. Endpoints

The main outcome of interest was a comprehensive evaluation of both GRs’ and FMs’ attitudes towards IROBE. In this regard, the comparative analysis is provided for the common questions. Also, we analyzed the GRs’ performance on the IROBE according to their age, gender, and marital status.

2.6. Statistical Analysis

To summarize the data, we used frequencies (percentages) and means (standard deviation and ranges) for categorical and continuous variables, respectively. To find an association between the study endpoints and ordinal (or nominal) variables, we applied the Mann-Whitney U test (or Pearson's chi-square test). In addition, we used the Kolmogorov Smirnov test to determine the normal distribution of variables. A P-value of ≤ 0.05 was considered significant. All analyses were performed using IBM® SPSS® Statistics (version 26.0).

3. Results

The survey was carried out in several days between August 1, 2020, and August 16, 2020. Out of a total of 35 FMs and 107 GRs, 16 (45%) FMs and 42 (39%) GRs completed the survey. Participants were excluded from the analysis if they chose the ‘prefer not to say’ choice for the attitude questions. In this regard, no participant was excluded from the study. Overall, 18.7% of FMs and 59.5% of GRs were females. The age of FMs ranged from 38 to 60 years with a mean age of 53.5 ± 7.6 years, and the age of GRs ranged from 30 to 41 years with a mean age of 33.4 ± 2.8 years. The majority of FMs were from Shahid Beheshti University of Medical Sciences (SBMU) and Shiraz University of Medical Sciences (SUMS) (25.0 and 18.75%, respectively). Meanwhile, the majority of GRs were from SBMU and Tehran University of Medical Sciences (TUMS) (50.0 and 16.66%, respectively). The remaining demographic data of participants are detailed in Table 1.

Table 1.

Demographic Characteristics of Study Respondents a

CharacteristicsValues
Faculty Members
Gender
Male13 (81.25)
Female3 (18.75)
Age (y), mean ± SD (range)53.5 ± 7.6 (38 - 60)
University of origin
SBMU4 (25.0)
TUMS1 (6.25)
AJUMS2 (12.5)
MUMS2 (12.5)
SUMS3 (18.75)
MUI2 (12.5)
IUMS2 (12.5)
Academic degree
Professor6 (37.5)
Associate professor7 (43.75)
Assistant professor3 (18.75)
Years as faculty member, mean ± SD (range)19 ± 7.54 (10 - 30)
Years as board member (between 2015 - 2019)
11 (6.25)
27 (43.75)
34 (25.0)
43 (18.75)
51 (6.25)
Graduated Residents
Gender
Male17 (40.48)
Female25 (59.52)
Age (y), mean ± SD (range)b33.4 ± 2.8 (30 - 41)
University of origin
SBMU21 (50.0)
TUMS7 (16.66)
AJUMS5 (11.9)
MUMS4 (9.52)
SUMS2 (4.76)
MUI3 (7.14)
IUMS0
Marital status b
Married26 (61.9)
Single16 (38.1)
Age of youngest child (y)b
< 13 (7.14)
1 - 32 (4.76)
3 - 65 (11.9)
> 65 (11.9)
None27 (64.28)
Leisure time to prepare for IROBE (mon)
0 - 322 (52.38)
3 - 620 (47.62)
Participation in IROBE (times)
133 (78.58)
26 (14.28)
33 (7.14)

There were five questions regarding clinical abilities [How would you rate the IROBE in evaluating (1) clinical competence, (2) clinical knowledge, (3) clinical performance, (4) medical ethics, and (5) responsibility?]. The FMs’ and GRs’ opinions on the capacity of IROBE for the aforementioned topics (based on 4-point Likert scale questions) were almost similar, except for the item of clinical performance (Table 2, Category A). In this regard, the rate of FMs and GRs who believed that IROBE has an average to poor quality was 62.5 vs. 76.1% in clinical competence (P = 0.07), 62.5 vs. 71.4% in clinical knowledge (P = 0.19), 68.8 vs. 88.3% in clinical performance (P = 0.02), 100 vs. 92.9% in medical ethics (P = 0.21), and 93.7 vs. 95.3% in responsibility (P = 0.15), respectively. The Importance of IROBE components (MCQ or OSCE) in evaluating clinical skills was asked from both groups (which IROBE component has a greater role in evaluating knowledge/performance?). For evaluating clinical knowledge, 25% of FMs and 28.5% of GRs voted for MCQ, and 25% of FMs and 23.8% of GR voted for OSCE. One-third of GRs believed that no component of IROBE can evaluate clinical knowledge, while just 6.25% of FMs were in agreement with them (P = 0.05). In this regard, 43.7% of FMs believed that MCQ and OSCE have the equal capability in evaluating clinical knowledge. Regarding clinical performance, there was also disagreement between FMs and GRs, so that 68.7% of FMs chose OSCE as a better test for evaluating clinical performance while 54.7% of GRs chose neither MCQ nor OSCE to be able to evaluate the skill (P = 0.03) (Table 2, Category B). Then, participants evaluated the necessity for improving the general quality of the IROBE (How much IROBE-MCQ/OSCE requires improvement?). Most of the FMs and GRs were in agreement that IROBE requires high to extreme improvement in both MCQ (75 vs. 59.5%) and OSCE (87.5 vs. 90.4%) components (Table 2, Category C). The last common question evaluated the importance of some potential confounding factors on examinees’ results in IROBE, including university, gender, and age. In case of the university of origin (how much examinee’s university affects the pass in IROBE-MCQ/OSCE/Rank?), most of FMs and GRs agreed that it may have a medium to extreme effect on the IROBE results (62.5 vs. 78.5% pass in MCQ, 68.7 vs. 83.3% pass in OSCE, and 62.5 vs. 73.7% obtaining high rank, respectively) (Table 2, Category D). In contrast, most of the FMs and GRs acknowledged that gender has a negligible to low effect on these items (87.5 vs. 80.9% pass in MCQ, 87.5 vs. 76.1% pass in OSCE, 81.2 vs. 76.1% obtaining high rank, respectively) (Table 2, Category E). Although more than half of the FMs considered GRs’ age to have a medium to high effect on the pass probability and obtain high-rank, most of GRs did not believe so (Table 2, Category F).

Table 2.

The Attitudes of Faculty Members and Graduated Residents to the Common Questions

CategoriesMain QuestionsOutlineAnswersFacultiesExamineesP-Value
A. Evaluation of clinical skills
A1How would you rate the IROBE in evaluating clinical competence?Clinical competenceExcellent04 (9.52)0.07
Good6 (37.5)6 (14.28)
Average10 (62.5)20 (47.62)
Poor012 (28.57)
A2How would you rate the IROBE in evaluating clinical knowledge?Clinical knowledgeExcellent1 (6.25)3 (7.14)0.19
Good5 (31.25)9 (21.42)
Average10 (62.5)22 (52.38)
Poor08 (19.04)
A3How would you rate the IROBE in evaluating clinical performance?Clinical performanceExcellent01 (2.38)0.02
Good5 (31.25)4 (9.52)
Average9 (56.25)20 (47.62)
Poor2 (12.5)17 (40.47)
A4How would you rate the IROBE in evaluating medical ethics?Clinical ethicsExcellent000.21
Good03 (7.14)
Average7 (43.75)7 (16.66)
Poor9 (56.25)32 (76.19)
A5How would you rate the IROBE in evaluating responsibility?ResponsibilityExcellent000.15
Good1 (6.25)2 (4.76)
Average6 (37.5)8 (19.04)
Poor9 (56.25)32 (76.19)
B. Indexed component
B1Which IROBE component has a greater role in evaluating clinical knowledge?Index section for clinical knowledgeMCQ4 (25.0)12 (28.57)0.05
OSCE4 (25.0)10 (23.80)
Equally7 (43.75)6 (14.28)
None1 (6.25)14 (33.33)
B2Which IROBE component has a greater role in evaluating clinical performance?Index section for clinical performanceMCQ02 (4.76)0.03
OSCE11 (68.75)14 (33.33)
Equally2 (12.5)3 (7.14)
None3 (18.75)23 (54.76)
C. IROBE status
C1How much IROBE-MCQ requires improvement?MCQ qualityExtreme4 (25.0)12 (28.57)0.42
High8 (50.0)13 (30.95)
Medium4 (25.0)10 (23.80)
Low01 (2.38)
Negligible06 (14.28)
C2How much IROBE-OSCE requires improvement?OSCE qualityExtreme8 (50.0)21 (50.0)0.93
High6 (37.5)17 (40.47)
Medium2 (12.5)3 (7.14)
Low01 (2.38)
Negligible00
D. University bias effect
D1How much does the examinee’s university affect the pass in IROBE-MCQ?University effect on MCQExtreme2 (12.5)8 (19.04)0.07
High2 (12.5)11 (26.19)
Medium6 (37.5)14 (33.33)
Low2 (12.5)3 (7.14)
Negligible4 (25.0)6 (14.28)
D2How much does the examinee’s university affect the pass in IROBE-OSCE?University effect on OSCEExtreme3 (18.75)9 (21.42)0.23
High3 (18.75)15 (35.71)
Medium5 (31.25)11 (26.19)
Low3 (18.75)4 (9.52)
Negligible2 (12.5)3 (7.14)
D3How much does the examinee’s university affect the rank in IROBE?University effect on IROBE rankingExtreme4 (25.0)14 (33.33)0.79
High5 (31.25)8 (19.04)
Medium1 (6.25)9 (21.42)
Low4 (25.0)3 (7.14)
Negligible2 (12.5)8 (19.04)
E. Gender bias effect
E1.1How much does the examinee’s gender affect the pass in IROBE-MCQ?Gender’s effect on MCQExtreme02 (4.76)0.37
High1 (6.25)1 (2.38)
Medium1 (6.25)5 (11.90)
Low5 (31.25)3 (7.14)
Negligible9 (56.25)31 (73.80)
E1.2If yes, which one?Male2 (12.5)3 (7.14)0.99
Female1 (6.25)5 (11.90)
E2.1How much examinee’s gender affects the pass in IROBE-OSCE?Gender’s effect on OSCEExtreme02 (4.76)0.63
High2 (12.5)1 (2.38)
Medium07 (16.66)
Low2 (12.5)3 (7.14)
Negligible12 (75.0)29 (69.04)
E2.2If yes, which one?Male2 (12.5)2 (4.76)0.38
Female1 (6.25)7 (16.66)
E3.1How much does the examinee’s gender affect the rank in IROBE?Gender’s effect on IROBE rankingExtreme03 (7.14)0.69
High03 (7.14)
Medium3 (18.75)4 (9.52)
Low1 (6.25)1 (2.38)
Negligible12 (75.0)31 (73.80)
E3.2If yes, which one?Male2 (12.5)2 (4.76)0.97
Female2 (12.5)8 (19.04)
F. Age bias effect
F1How much does the examinee’s age affect the pass in IROBE-MCQ?Age’s effect on MCQExtreme01 (2.38)0.16
High8 (50.0)8 (19.04)
Medium2 (12.5)9 (21.42)
Low1 (6.25)7 (16.66)
Negligible5 (31.25)17 (40.47)
F2How much does the examinee’s age affect the pass in IROBE-OSCE?Age’s effect on OSCEExtreme01 (2.38)0.26
High7 (43.75)8 (19.04)
Medium3 (18.75)11 (26.19)
Low1 (6.25)7 (16.66)
Negligible5 (31.25)15 (35.71)
F3How much does the examinee’s age affect the rank in IROBE?Age’s effect on IROBE rankingExtreme02 (4.76)0.45
High5 (31.25)9 (21.42)
Medium4 (25.0)6 (14.28)
Low2 (12.5)7 (16.66)
Negligible5 (31.25)18 (42.85)

Table 3 summarizes the FMs’ responses to four specific questions in two domains. The first question considered the effect of having a board certification on the FMs’ decision for recruitment of GRs as a faculty of a university (Table 3, Categories A3-A1). All FMs except for 2 (87.5%) stated that it has a medium to extreme influence on their decision. The results of Pearson's chi-square test demonstrated no association between the FM’s decision and their gender (P = 0.82), experience (P = 0.94), academic degree (P = 0.18), and university of origin (P = 0.98). Likewise, most (81.25%) FMs believed that a high rank in IROBE has a medium to extreme effect on their decision for recruiting GRs as a faculty (Table 3, Categories A3-A2). Based on association analysis, FMs’ gender was associated with this decision (P = 0.004), so that 3 (100%) female and 5 (38.4%) male FMs stated that high-ranking in IROBE has a high to extreme effect on their decision to employ GRs as a faculty. The other two specific questions from the FMs dealt with the shortcomings of IROBE over the last five years. In FMs’ opinion, the major limitations of the MCQ component of IROBE were as follows: (1) lack of motivation (75.0%); (2) lack of familiarity (62.5%); (3) lack of experience (56.25%) for designing high-quality questions; and (4) lack of centralized question design committee (31.25%). In addition, in the case of OSCE examination, the major limitations were as follows: (1) lack of familiarity (81.25%); (2) lack of motivation (62.5%); (3) lack of executive facilities (43.75%); and (4) lack of experience (31.25%).

Table 3.

The Attitudes of Faculty Members to the Specific Questions

CategoriesMain QuestionsOutlineAnswersFacultiesP-Value
A. Faculty attraction
A1How effective is having a board certification in your personal opinion for faculty attraction?Board certification for faculty attractionExtreme5 (31.25)Gender: 0.82; Experience: 0.94; Ac. Degree: 0.18; University: 0.98
High5 (31.25)
Medium4 (25.0)
Low2 (12.5)
Negligible0
A2How effective is ranking on board examination in your personal opinion for faculty attraction?Board ranking for faculty attractionExtreme3 (18.75)Gender: 0.004; Experience: 0.92; Ac. Degree: 0.66; University: 0.83
High5 (31.25)
Medium5 (31.25)
Low3 (18.75)
Negligible0
B. IROBE shortcomings
B1What are the shortcomings (if any) of IROBE-MCQ?MCQ shortcomingsLack of motivation12 (75.0)-
Lack of familiarity10 (62.5)
Lack of executive facilities2 (12.5)
Lack of centralized question design committee5 (31.25)
Insufficient wage4 (25.0)
Lack of informative blueprint1 (6.25)
Lack of question bank2 (12.5)
Lack of experience9 (56.25)
Old age1 (6.25)
None of the above1 (6.25)
B2What are the shortcomings (if any) of IROBE-OSCE?OSCE shortcomingsLack of motivation10 (62.5)-
Lack of familiarity13 (81.25)
Lack of executive facilities7 (43.75)
Lack of centralized question design committee4 (25.0)
Insufficient wage3 (18.75)
Lack of informative blueprint2 (12.5)
Lack of appropriate location2 (12.5)
Lack of experience5 (31.25)
Old age0
None of the above1 (6.25)

The GRs’ opinions regarding the 11 specific questions in the three domains are presented in Table 4. The first domain evaluated the importance of IROBE in their personal lives. Most of the participants stated that passing the board examination (92.8%) and obtaining a high rank (66.7%) had medium to extreme importance for them. In line with these, 73.8% of GRs declared that passing the IROBE caused medium to extreme worry in them, and 85.7% believed that having a board certification has a medium to extreme effect on their future career opportunities. These beliefs had incremental impacts over the residency period in half of the GRs (Table 4, Category A). The next domain dealt with the IROBE shortcomings. According to the GRs’ opinions, the major limitations of the MCQ component of IROBE were as follows: (1) high rate of impractical questions (54.7%); (2) the involvement of faculty’s personal comment in answers (52.3%); and (3) poor-quality and equivocal questions (each 40.4%). Concerning the OSCE examination, the major limitations were designing descriptive (rather than OSCE-standardized) questions (69.0%) and inability to evaluate the clinical performance (61.9%), medical ethics (50.0%), and responsibility (50.0%) (Table 4, Category B). About 69.0, 76.1, and 76.1% of GRs declared that the number of FMs’ of the GRs’ universities - participating in the board examination - had a medium to extreme effect on their results in MCQ, OSCE, and final ranking, respectively (Table 4, Category C).

Table 4.

The Attitudes of Graduated Residents to the Specific Questions

CategoriesMain QuestionsOutlineAnswersExamineesP-Value (for Variables)
A. Perspective
A1How important was acceptance in IROBE?Importance of passing the IROBEExtreme25 (59.52)Gender: 0.52; Age: 0.55; University: 0.77; Marital status: 0.85; Paternal/maternal status: 0.10; Board certified: 0.49; High-ranked: 0.90
High10 (23.8)
Medium4 (9.52)
Low1 (2.38)
Negligible2 (4.76)
A2How important was rank in IROBE?Importance of rank in IROBEExtreme14 (33.33)Gender: 0.24; Age: 0.59; University: 0.91; Marital status: 0.54; Paternal/maternal status: 0.70; Board certified: 0.26; High-ranked: 0.74
High3 (7.14)
Medium11 (26.19)
Low5 (11.9)
Negligible9 (21.42)
A3How worried were you about not passing the IROBE?Concern about IROBEExtreme17 (40.47)Gender: 0.42; Age: 0.30; University: 0.78; Marital status: 0.56; Paternal/maternal status: 0.79; Board certified: 0.12; High-ranked: 0.09
High6 (14.28)
Medium8 (19.04)
Low6 (14.28)
Negligible5 (11.9)
A4How much did you consider that IROBE certification can affect your future career opportunities?Importance of IROBE in career opportunitiesExtreme17 (40.47)Gender: 0.21; Age: 0.13; University: 0.40; Marital status: 0.48; Paternal/maternal status: 0.25; Board certified: 0.89; High-ranked: 0.47
High5 (11.9)
Medium14 (33.33)
Low5 (11.9)
Negligible1 (2.38)
A5How did your attitude on the importance of passing the IROBE change over the residency course?Importance of pass in IROBE over the timeIncreasing21 (50.0)Gender: 0.91; Age: 0.57; University: 0.25; Marital status: 0.33; Paternal/maternal status: 0.56; Board certified: 0.93; High-ranked: 0.93
Constant12 (28.57)
Decreasing7 (16.66)
No comment2 (4.76)
A6How did your attitude on the importance of rank in IROBE change over the residency course?Importance of rank in IROBE over the timeIncreasing21 (50.0)Gender: 0.94; Age: 0.43; University: 0.59; Marital status: 0.48; Paternal/maternal status: 0.56; Board certified: 0.98; High-ranked: 0.37
Constant13 (30.95)
Decreasing7 (16.66)
No comment1 (2.38)
B. IROBE shortcomings
B1What are the shortcomings (if any) of IROBE-MCQ?MCQ shortcomingsLow quality questions17 (40.47)-
Unfit number of questions for rate of illnesses13 (30.95)
Equivocal questions17 (40.47)
Relative short time for answering3 (7.14)
Repetitious questions0
The involvement of faculty’s personal comment in answers22 (52.38)
Impractical questions23 (54.76)
Few case study-based questions12 (28.57)
Unjustifiable increase in scores upon protests9 (21.42)
None of the above5 (11.9)
B2What are the shortcomings (if any) of IROBE-OSCE?OSCE shortcomingsLow quality questions15 (35.71)-
Number of questions are unfit for rate of illnesses9 (21.42)
Lack of appropriate location7 (16.66)
Relative short time for answering8 (19.04)
The questions are more descriptive than OSCE-standardized questions29 (69.04)
The presence of the faculties in the session creates anxiety9 (21.42)
It cannot evaluate medical performance26 (61.90)
It cannot evaluate medical ethics21 (50.0)
It cannot evaluate responsibility21 (50.0)
None of the above1 (2.38)
C. Faculty bias effect
C1How much does the number of faculties from your university involved in IROBE-MCQ affect your results?The effect of familiar faculties on MCQExtreme12 (28.57)Gender: 0.29; Age: 0.47; University: 0.25; Marital status: 0.99; Paternal/maternal status: 0.97; Board certified: 0.58; High-ranked: 0.54
High8 (19.04)
Medium9 (21.42)
Low2 (4.76)
Negligible11 (26.19)
C2How much does the number of faculties from your university involved in IROBE-OSCE affect your results?The effect of familiar faculties on OSCEExtreme16 (38.09)Gender: 0.42; Age: 0.21; University: 0.22; Marital status: 0.58; Paternal/maternal status: 0.32; Board certified: 0.65; High-ranked: 0.72
High9 (21.42)
Medium7 (16.66)
Low2 (4.76)
Negligible8 (19.04)
C3How much does the number of faculties from your university involved in IROBE-OSCE affect your rank in IROBE?The effect of familiar faculties on IROBE rankingExtreme18 (42.85)Gender: 0.99; Age: 0.15; University: 0.14; Marital status: 0.12; Paternal/maternal status: 0.63; Board certified: 0.86; High-ranked: 0.49
High12 (28.57)
Medium2 (4.76)
Low2 (4.76)
Negligible8 (19.04)

Finally, we evaluated the association of GRs’ demographic information in their success in IROBE. The characteristics were similar between the residents who had passed the IROBE and those who had not passed it; however, male participants were more likely to pass the IROBE (100 vs. 76%, P = 0.02). In addition, no association was found between obtaining a high rank in IROBE and residents’ gender (P = 0.44), age (P = 0.32), university of origin (P = 0.47), and marital status (P = 0.74) (Table 5).

Table 5.

The Association Between Board Certification and High-Rank in IROBE with the Residents’ Characteristics

CharacteristicsDescriptive Analysis (% Within Group)P-Value
Board-Certification
Gender0.02
Male100
Female76.0
Age0.06
30 - 32100
33 - 3576.5
36 - 3850.0
39 - 41100
University0.24
SBMU90.5
TUMS85.7
AJUMS60.0
MUMS100
SUMS50.0
MUI100
Marital status0.18
Married92.3
Single75
Having children39.5
Yes93.3
No81.5
High-Ranked
Gender0.44
Male29.4
Female16.0
Age0.32
30 - 3229.4
33 - 3517.6
36 - 380.0
39 - 4125.0
University0.47
SBMU19
TUMS14.3
AJUMS40.0
MUMS50.0
SUMS0.0
MUI0.0
Marital status0.74
Married23.1
Single18.8
Having children0.34
Yes13.3
No25.9

4. Discussion

In this survey, most FMs and GRs agreed that IROBE still has average to poor quality in evaluating the clinical competence, clinical knowledge, medical ethics, and responsibility, and requires major improvement in both MCQ and OSCE components. While GRs did not realize either IROBE components to have enough capacity to evaluate clinical knowledge and performance, most FMs had a contrasting opinion. Both groups believed that the university has major effects and gender has minor effects on success in IROBE. However, in contrast to the GRs, the majority of FMs considered age as a determining factor in success in both MCQ and OSCE exams. Interestingly, most GRs acknowledged the participation of their university’s FMs as a contributing factor for their success in IROBE.

Most GRs highlighted the poor quality of both MCQ and OSCE examinations for including many impractical and equivocal questions. This is possibly due to the lack of motivation, familiarity, and experience to design questions that were common among FMs’ responses to the question regarding IROBE limitations. More than half of the GRs stated that the IROBE-MCQ contained many impractical questions, and about 60% of GRs complained about the inability of IROBE-OSCE in evaluating clinical performance. To cope with these limitations, about one-third of GRs proposed to increase the number of case-study-based questions in the MCQ exam, and about 70% of them agreed to switch the descriptive questions into OSCE-standardized ones. These changes can potentially enhance the GRs’ clinical reasoning to cope with the possible future clinical dilemmas, as stated by Albert Einstein: “Education is not learning the facts but training the mind to think” (9). To that end, developing the comprehensive test blueprint with weightings that emphasize clinical radiation oncology is suggested (10). Moreover, the questions of medical physics and radiation biology can be designed in line with the ACGME core competencies (10). On the other hand, approximately 20% of GRs complained about the short time for answering the OSCE questions. This might stem from their lack of preparation, and most probably shortcomings of OSCE component of IROBE in comparison with standard OSCE. Running mock OSCE or its alternatives (such as peer-led multi-role practice OSCEs) during residency could help FMs for designing high-quality OSCE and prepare GRs for the exam (11, 12).

While a larger proportion of FMs compared to GRs (31.2 vs. 11.9%) believed that IROBE-OSCE could effectively evaluate the clinical performance, most participants of both groups believed that the existing IROBE could not efficiently evaluate all aspects of clinical abilities (including clinical competence, knowledge, medical ethics, and responsibility). Hence, emergency action is needed to improve the quality of IROBE.

Most GRs declared that IROBE has majorly affected their personal lives and caused so much concern for them. This condition has also been reported in overseas radiation oncology residents (13). It has been shown that the stress experienced by medical students might induce high rates of burnout and depression (14). The residents’ concern might stem from the FMs’ attitudes that having a board certification, as well as a high rank in IROBE, have great importance for employing new faculties. This level of stress emphasizes an emergency request for improving the quality of IROBE again.

The current study had some limitations. Firstly, the findings may have been confounded by the small number of participants. Secondly, participants were inquired about their experience for constructing or participating in IROBE, which was related to few months to five years earlier. This may subject the answers and comments to recall bias. Although these limitations are important, the current study is the first to seek the strengths and flaws of IROBE.

4.1. Conclusion

The results of the present study revealed that the current structure of IROBE has several major drawbacks and requires a comprehensive revision in both MCQ and OSCE components. Both FMs and GRs had somehow similar ideas in this regard. The IROBE and Education Deputy of the Ministry of Health can consider these findings to enhance the board examination purpose, shape, and experience for both faculties and residents.

References

  • 1.

    Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387-96. [PubMed ID: 17251535]. https://doi.org/10.1056/NEJMra054784.

  • 2.

    Emadzadeh A, Ravanshad Y, Makarem A, Azarfar A, Ravanshad S, Aval SB, et al. Challenges of OSCE national board exam in Iran from participants' perspective. Electron Physician. 2017;9(4):4195-201. [PubMed ID: 28607655]. [PubMed Central ID: PMC5459292]. https://doi.org/10.19082/4195.

  • 3.

    Accreditation Council for Graduate Medical Education. ACGME common program requirements. Illinois, United States: Accreditation Council for Graduate Medical Education; 2018.

  • 4.

    Batalden P, Leach D, Swing S, Dreyfus H, Dreyfus S. General competencies and accreditation in graduate medical education. Health Aff (Millwood). 2002;21(5):103-11. [PubMed ID: 12224871]. https://doi.org/10.1377/hlthaff.21.5.103.

  • 5.

    Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287(2):226-35. [PubMed ID: 11779266]. https://doi.org/10.1001/jama.287.2.226.

  • 6.

    Wass V, Van der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet. 2001;357(9260):945-9. https://doi.org/10.1016/s0140-6736(00)04221-5.

  • 7.

    Taherdoost H. Determining sample size; how to calculate survey sample size. Int J Econ Manag. 2017;2:237-9.

  • 8.

    Gill J, Johnson P. Research methods for managers. California, United States: SAGE Publications; 2002.

  • 9.

    Butt A. Education is not the learning of facts but the training of the mind to think–Albert Einstein. Glob Manag J Acad Corp Stud. 2019;9(2).

  • 10.

    Lee WR, Amdur RJ. A call for change in the ABR initial certification examination in radiation oncology. Int J Radiat Oncol Biol Phys. 2019;104(1):17-20. [PubMed ID: 30967225]. https://doi.org/10.1016/j.ijrobp.2018.12.046.

  • 11.

    Harden RM, Lilley P, Patricio M. The definitive guide to the OSCE: The objective structured clinical examination as a performance assessment. Amsterdam, Germany: Elsevier Health Sciences; 2015.

  • 12.

    Bevan J, Russell B, Marshall B. A new approach to OSCE preparation - PrOSCEs. BMC Med Educ. 2019;19(1):126. [PubMed ID: 31046773]. [PubMed Central ID: PMC6498564]. https://doi.org/10.1186/s12909-019-1571-5.

  • 13.

    Ramey SJ, Ahmed AA, Takita C, Wilson LD, Thomas CJ, Yechieli R. Burnout evaluation of radiation residents nationwide: Results of a survey of United States residents. Int J Radiat Oncol Biol Phys. 2017;99(3):530-8. [PubMed ID: 29280446]. https://doi.org/10.1016/j.ijrobp.2017.06.014.

  • 14.

    Prober CG, Kolars JC, First LR, Melnick DE. A plea to reassess the role of United States medical licensing examination step 1 scores in residency selection. Acad Med. 2016;91(1):12-5. [PubMed ID: 26244259]. https://doi.org/10.1097/ACM.0000000000000855.