Performance Prediction of the Family Medicine Applicants of Oman Medical Specialty Board in Multiple Mini-interviews Using the International Foundations of Medicine Exam and Situational Judgment Test

1. Background

Using valid and reliable methods in selecting candidates for medical specialty training should be of high priority to any academic institute (1). A concise analysis of what is needed for the job and its purpose should determine the selection method and the candidates most likely to perform well during training. Therefore, combined with academic and clinical competencies, an assessment of the applicant’s personal attributes is essential to the selection process (2). Moreover, to avoid any selection bias, the chosen selection method should be tailored and sensitive to diversity issues (1, 2).

At Oman Medical Specialty Board (OMSB), the regulatory body responsible for overseeing postgraduate medical education in Oman, all residency training applicants are required to pass the International Foundations of Medicine (IFOM) examination. Regardless of their chosen specialty, applicants who have passed the IFOM examination subsequently undergo three panels of multiple mini-interviews (MMIs) over four days from 8:00 am to 5:00 pm, which is highly connected with interviewers’ motivation to improve their profession and future patient care (3). Approximately 80 - 100 candidates apply yearly to the OMSB Family Medicine Residency Program as their first or second choice, representing more applicants than other programs. For the 2021 academic year, the OMSB Education Committee piloted an additional selection measure (a situational judgment test (SJT)) to assess attributes other than knowledge alone.

Tests assessing situational judgment seek to evaluate the applicant’s responses to professional dilemmas likely to be encountered in clinical practice (4-6). Worldwide, SJTs are widely used in the selection of high-stakes industry personnel, as well as medical students in the UK, Belgium, Canada, Australia, and Latin America. In the UK, 50% of the total scores of final-year medical students come from the SJT (6-11). In addition, SJTs were useful for teaching empathy, assertiveness, inter-professional communication, and ethics to pharmacy students (12).

During SJTs, candidates are required to respond to questions assessing both behavioral tendency (i.e., how to behave in a given specific situation) and knowledge tendency (i.e., suitable and effective responses to a given situation). According to McDaniel et al., the portion of SJTs testing knowledge correlates with general cognitive ability, while the section testing behavior correlates with positive or desirable personality attributes (13). Furthermore, Lievens et al. found that SJT and cognitive tests completed each other in selecting doctors (14). Ahmed et al. revealed that SJT overcomes clinical problem-solving in predicting selection center performance (R² = 0.51 versus 0.35) (15). However, combining the two tests maximized the predictive ability of performance, explaining 57% of the variance in overall mean scores (15). In addition, Husbands et al. found the reliability of one particular SJT to be high during the undergraduate selection process (Cronbach’s alpha coefficient = 0.64), with significant associations observed between SJT scores, integrity, other desirable personality traits, and MMI scores (16).

Recently, SJTs have been incorporated into the postgraduate selection process for the UK Foundation Programme (2, 10, 17). However, the evidence supporting the predictive validity of SJTs is lacking in the postgraduate trainees’ performance and selection (4). In particular, Prideaux et al. highlighted the difficulties faced by selection process designers due to the lack of research on this topic (1). A systematic review and meta-analysis conducted by Webster et al. identified 26 studies reporting correlation coefficients related to the validity of SJTs, with a pooled estimate of 0.32 (95% confidence interval (CI): 0.26 - 0.39; P < 0.001) (18). In a well-designed selection system, the researchers concluded that SJTs could be a useful selection tool, with studies of postgraduate medical applicants reporting larger validity coefficients in contrast to those conducted in undergraduate settings (β = 0.21, 95% CI: 0.11 - 0.31; P < 0.001) (16). However, another meta-analysis found that the internal reliability of SJTs varied widely (Cronbach’s alpha coefficients: 0.43 - 0.94) (19). Furthermore, Juster et al. found that including an SJT in the selection process will result in expanding access to medical education in terms of gender, racial, and ethnic diversity, increasing the variation of future physicians and thereby helping in addressing the needs of a changing patient population (20).

2. Objectives

This study aimed to test the validity of both SJT and the IFOM examination in predicting performance during MMIs among applicants to the OMSB Family Medicine Residency Program. In addition, the study evaluated the complementary role of an SJT in shortening the list of applicants undertaking the MMI panel. Implementing valid, cost-effective measures to shortlist applicants for postgraduate residency training programs will save time and effort. It is beneficial both on the part of interviewers, many of whom are required to interview a large number of applicants under the current selection system, as well as on the part of unsuitable applicants who would otherwise have to undergo unnecessary lengthy interviews and stressful waiting periods before finding out they have been rejected. Moreover, a streamlined and more efficient selection process will positively impact the future applicant’s performance during residency, as well as ensure that the best possible candidates are admitted into residency training programs.

3. Methods

This retrospective cross-sectional study was conducted during the OMSB residency application process for the academic year 2021 and included 80 applicants who applied for the OMSB Family Medicine Residency Program as either their first or second choice. Various information was collected about the applicants, including their age, gender, graduating institute, work experience, current position, and grade point average (GPA) during their undergraduate medical degree. All data were obtained from the relevant OMSB administration and examination sections.

All candidates included in the study had passed the IFOM examination as a prerequisite for their residency application. Subsequently, they underwent a two-hour, computer-based SJT under invigilation conditions. The test consisted of 50 questions in nine attributes, including a commitment to professionalism, coping with pressure, effective communication, working effectively as part of a team, learning and professional development, organization and planning, patient focus, problem-solving and decision making, and self-awareness and insight. It had two formats of best answer or multiple responses of the three most appropriate options (Table 1). The internal reliability of the SJT used during the selection process was high (Cronbach’s alpha coefficient = 0.84). Afterwards, all applicants participated in three MMI panels, each consisting of two interviewers. Each panel lasted 10 min, during which the applicant had 5 min to answer one question posed by each interviewer. Questions comprised clinical and non-clinical elements, including those related to the applicant’s personality, attitude, history of community service, and level of enthusiasm related to their chosen specialty. Each interviewer then rated the applicant’s responses and their proficiency in spoken English (scored from 0 to 4) for a cumulative score out of 22.

The frequency of categorized variables was obtained using the Statistical Package for the Social Sciences (SPSS) version 23 (IBM Corp., Armonk, NY, USA). Pearson’s correlation coefficient was used to evaluate the correlations of MMI scores with SJT and IFOM examination scores. In addition, a multiple hierarchical regression analysis was conducted to assess the predictors of the MMI score. A receiver operating characteristic (ROC) curve analysis was also performed to calculate the area under the curve (AUC) and determine the ability of SJT to categorize applicants as recommended or not recommended for the OMSB Family Medicine Residency Program based on their MMI scores. Overall, the AUC values of ≤ 0.5, 0.6 - 0.7, 0.7 - 0.8, 0.8 - 0.9, and 0.9 - 1 indicated no, poor, acceptable, excellent, and outstanding accuracy in selecting candidates, respectively (21). Ethical approval for this study was obtained from the OMSB Institutional Research and Ethics Committee.

4. Results

A total of 67 applicants completed the entire selection process (i.e., underwent the SJT and appeared for the MMI panels) and were included in the study. Of these, five (7.5%) were male, and 62 (92.5%) were female. The majority aged 25 - 29 years (n = 46; 68.7%), with less than five years of experience since graduation (n = 50; 74.6%). All, except one of the applicants, were of Omani nationality (n = 66; 98.5%), and most had graduated from institutions based within Oman (n = 61; 76.1%). The applicants were currently working in various types of job, such as general practitioner (n = 23; 34.3%), enrolled in a general foundation program (n = 27; 40.3%), or undertaking their internship year (n = 17; 25.4%). A total of 47 applicants (70.1%) had chosen family medicine as their first choice of specialty. More than half of the applicants had scored < 60 in the IFOM examination (n = 35; 52.2%), while almost three - quarters (n = 50; 74.6%) had scored > 70 in the SJT. A total of 18 applicants (26.9%) scored ≥ 15 in the MMI panels and were, therefore, recommended for entry into the OMSB Family Medicine Residency Program (Table 2).

Overall, the SJT score was significantly correlated with the MMI score (r = 0.367; P = 0.002), whereas IFOM score (r = 0.105; P = 0.396), age (r = -0.010; P = 0.936), years of experience since graduation (r = 0.056; P = 0.651), and GPA (r = 0.036; P = 0.772) were not significantly correlated (Table 3). According to multiple hierarchical regression analysis, model 1 resulted in an R² value of 0.134, indicating that the SJT score accounted for 13.4% of the variance in the MMI score. Model 2 resulted in an R² value of 0.138, showing that the two variables together (the SJT score and IFOM examination score) accounted for 13.8% of the variance in the MMI score. The R² change between the two models was 0.004, signifying that the addition of the IFOM examination score caused a 0.4% additional variance to the MMI score (P = 0.586). Consequently, the inclusion of an additional variable (the IFOM examination score) in model 2 did not result in a statistically significant increase in the variance of the outcome variable (MMI score) compared to model 1 (P = 0.586 versus 0.002) (Table 4).

The regression model validated the ability of the SJT score to predict the MMI score according to the following formula: MMI score = 1.512 + 0.152 × SJT score (Table 5). However, upon calculating the ROC curve, the AUC was found to be 0.633 (95% CI: 0.482 - 0.785), indicating that the SJT score had a poor accuracy in categorizing individuals as either recommended or not recommended for the residency program (Figure 1). No significant differences in the mean MMI scores were observed between candidates based on the graduating institution (P = 0.260) or current position (P = 0.460) (Table 6).

Receiver-operating characteristic curve of situational judgment test scores as a selection method to predict multiple mini-interviews scores among applicants to the Oman Medical Specialty Board Family Medicine Residency Program (n = 67)

Figure 1.

Receiver-operating characteristic curve of situational judgment test scores as a selection method to predict multiple mini-interviews scores among applicants to the Oman Medical Specialty Board Family Medicine Residency Program (n = 67)

5. Discussion

To date, most literature related to the application of SJTs has evaluated the predictive potential of this type of test on the performance of applicants in the end-of-training assessments (22), with few studies highlighting the use of SJTs as a selection method for entry into postgraduate training (4, 23). This gap in the existing research, along with introducing an SJT in the selection process for entry to the OMSB Family Medicine Residency Program, was the source of inspiration for the present study. Consequently, the SJT was performed for the first time in Oman, and the OMSB Education Committee of Family Medicine program decided not to use SJT scores as a shortlisting mechanism prior to the MMI panels for 2021 applicants. However, this decision may be amended in the future once the validity of this SJT as a selection criterion is definitively confirmed.

The current study found a non-significant correlation between the IFOM examination and MMI scores, whereas there was a significant correlation between SJT and MMI scores. This can be explained by the fact that the IFOM examination concentrates on applied knowledge, while the MMI panels gauge various non-academic attributes and behaviors, similar to SJTs (23). Therefore, it is reasonable that SJTs would be a better predictor of MMI scores compared to the IFOM examination. Patterson et al. found that both SJT and MMI scores had strong correlations with the objective structured clinical examination (OSCE) scores (r = 0.44 and r = 0.46, respectively; P < 0.001 for both) during an end-of-training assessment. The researchers attributed these findings to the similarities between these methods because both extend beyond applied knowledge (23, 24). For the same reasons, Patterson et al. and Tiffin et al. found SJT scores to be a strong predictor of practical assessment of clinical examination skills scores during an end-of-training assessment, especially with construct-driven SJT (23, 25). In addition, McDaniel et al., in a meta-analysis, indicated that behavioral tendency questions within SJTs resulted in the largest incremental validity compared to knowledge tendency questions (0.05 versus 0.03) (13).

In the present study, the SJT scores were positively correlated with the MMI scores and were significantly predictive of performance in the MMI panels (r = 0.367; P = 0.002). Therefore, it can be concluded that the SJT used in the present study may be utilized as a prerequisite to shortlisting applicants to the OMSB Family Medicine Residency Program prior to the interview phase. This would be especially beneficial considering the high number of applicants to this specialty compared to other programs, which results in a considerable administrative burden. Patterson et al. similarly concluded that SJTs might offer an additional, standardized benchmark by which program directors could shortlist applicants before the interview phase (26). Husbands et al. also reported statistically significant correlations between the SJT and MMI scores and multiple aspects of the HEXACO Personality Inventory (16). These findings further support the reliability and validity of SJTs and strengthen their utilization as a selection method, either as a pre-interview screening tool or as a partial replacement for MMI panels altogether (16). In another study, Patterson et al. confirmed the reliability of SJTs in medical selection, predicting performance in different medical education settings (undergraduate and postgraduate) and their incremental validity over other selection methods (9). Furthermore, the variations in SJT validity, reliability, and diversity benefits are attributed to the cultural differences between Western and Asian contexts concerning professionalism, values, and communication style (27).

Overall, the SJT scores were found to explain 13.4% of the variability in the MMI scores in the current study, which is considerably lower than the previous research, showing that this selection method explained 22% of the variation in selection center performance when used as a shortlisting tool during selection for specialty training (15). Patterson et al. reported that machine-marked SJT scores added incremental validity (R² = 0.143) when evaluating non-clinical attributes for the postgraduate medical selection compared to a clinical problem-solving test alone (26). Moreover, the researchers found that SJT was the most influential independent predictor of selection center scores (adjusted R² = 0.213; P < 0.001) compared to two other shortlisting methods (a clinical problem-solving test and structured application form questions) (28). Lievens et al. reported that SJT showed criterion-related validity and incremental validity above cognitive ability and personality tests in predicting job performance (29). In contrast, Lievens and Patterson revealed that SJT resulted in an additional variance of 5.9% over a clinical problem-solving test regarding job performance (30).

In Oman, there is yet no previous data regarding the use of SJTs as a part of the selection process in the postgraduate medical education settings. A recent study by Al Hashmi and Klassen focused on developing an SJT to determine admission into initial teacher education programs (ITEPs) in Oman (31). The results demonstrated that the test had good internal consistency (α = 0.75). However, while SJT scores were significantly correlated with GPA as well as conscientiousness and agreeableness, out of the five personality measures, no correlation was found with interview scores. The researchers recommended using SJT as a promising additional selection method for ITEPs in Oman (31). Ongoing quality assurance of the selection process as a whole is imperative in postgraduate medical education to ensure improved competency in healthcare delivery, thereby enhancing future patient care. Moreover, adding an SJT in the residency selection process would reduce the resources needed for assessment, as SJTs are relatively cheap to administer to candidates compared to other methods (16).

5.1. Conclusions

The findings of this study indicated that an SJT is a valid selection method for entry into residency training alongside the traditional MMI panel. To the best of the authors’ knowledge, this study represents the first analysis of SJTs in the medical setting in Oman and adds to existing international research regarding the predictive validity of this type of test in postgraduate medical settings. Furthermore, it opens the door for further longitudinal research evaluating the predictive validity of such selection methods during the end-of-training assessment in Oman, especially in comparison with practical assessments, such as the OSCE or simulated surgery.

5.2. Limitations

This study is limited by its small sample size compared to previous investigations conducted in the same field. Furthermore, the scope of this study was limited to the ability of an SJT to predict MMI score as a selection method for entry into the residency program. Further research is required to determine whether SJTs are suitable for evaluating long-term clinical training or work performance. Future studies assessing this aspect will provide additional evidence demonstrating the utility of SJTs as a shortlisting selection method prior to the MMI panel.