1. Background
Assessment of trainees is one of the main steps in the medical education and training process, both for undergraduate and postgraduate trainees (1-4). Clinical faculty members are the cornerstone of trainee assessment, and multiple-choice questions (MCQ’s) are one of the most common methods of assessment; however, MCQ’s are not always designed in a valid and reliable method (5-9).
Multiple-choice questions (MCQs) are used commonly to evaluate medical health students. MCQs can assess recall, comprehension, and application of science (10-12). These types of testing are chosen because you can test a large number of learners at the same time, ability to cover a wide area of subjects, and also have better accuracy and consistency than subjective forms (10, 11, 13).
Other evaluation methods, such as OSCEs, patient management problems (PMPs), roleplaying, etc., are proposed as an alternative or complementary form for evaluation (14, 15). On the other hand, instructors do not follow the general principle regarding the increasing quality of MCQs (13, 16). It has been shown that training workshops for one full week could increase the quality of questions designed by instructors (17). Most novice educators tend to create poor quality, flawed, and low-cognitive-level questions. Therefore, there is a need for educating the instructors to maximize the quality of MCQs and evaluations.
2. Objectives
This study was designed to assess the role of “Faculty MCQ Education Workshop on knowledge of the faculty who are involved in the Anesthesiology Departmental Evaluation Committee: ADEC”, Department of Anesthesiology and Critical Care (DACC), Shahid Beheshti University of Medical Sciences (SBMU), Tehran, Iran. So, the primary goal of the current study is to evaluate the effect of a one-day MCQ workshop on anesthesiology faculty members. The participants’ reactions to the workshop, knowledge, and their capability for evaluating questions were assessed. We hypothesized that a four-hour workshop would improve the attitude of instructors toward creating high-quality questions.
3. Methods
The study was approved by the Research Ethics Committee, Vice-Chancellor of Research Affairs of Shahid Beheshti University of Medical Sciences, Tehran, Iran (Ethics code: IR.SBMU.RETECH.REC.1399.780). In our educational department of anesthesiology, we are conducting monthly exams containing 120 MCQ questions. The questions are created by more than 30 faculty members. In an interventional medical education study, all members of ADEC, Anesthesiology Department, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran who had designed these exams were invited for a one-day workshop.
A single lecturer guided the one-day workshop. Among the 64 members of the ADEC, 17 subjects participated in the workshop. Two questionnaires were handed out among the participant; both before the workshop and just after termination of the discussions. This questionnaire was designed to assess the knowledge of the participants regarding MCQ tests.
The first questionnaire was about their knowledge about MCQ quality indexes and also about MCQ general principles (Pre-test). In this questionnaire, participants were asked to name three indicators of grading MCQs, and also four true/false questions were asked about their knowledge of common flaws. Each indicator and each question were given one point if answered correctly.
To achieve the workshop objectives, the instructor, who was also a faculty member of the Emergency Department of our university and had experience in this field, different teaching methods were applied. The first session consists of a short, interactive lecture, following that examples of common flaws were discussed. After the break, participants were asked to correct the flaws to improve the quality of the questions. At last, indicators of grading MCQs such as difficulty index (P), discriminating index (DI), non-functional distractors (NFDs), item writing flaws (IWF), and Bloom’s taxonomy level were described and discussed.
The participants were again asked about the questions which they had in the pre-test as their post-test and were questioned about their expectations and the influence of this workshop.
4. Results
Of the total sixty-four faculty members, 17 subjects attended the MCQ workshop. Only 6 of them did not take the MCQ course before this workshop. Members with more than 20-year history of membership have participated in MCQ training courses at least once. Members with less than 5-year history of membership did not take any course regarding designing MCQ questions (Table 1). The others have passed training courses once or twice.
| Variables | Values |
|---|---|
| Gender | |
| Male | 10 |
| Female | 7 |
| Educational rank | |
| Full professor | 1 |
| Associated professor | 10 |
| Assistant professor | 6 |
| Years of experience in teaching | |
| > 15 | 2 |
| 5 - 10 | 11 |
| < 5 | 4 |
| The experienced MCQ training course | |
| 0 | 5 |
| 1 | 7 |
| > 1 | 5 |
Characteristics of the Participants
The participants were asked about their motivation for passing this workshop, their responses are listed in Table 2. Also, at the end of the workshop, the participants declared that their expectations were fulfilled (9.4 ± 0.6 out of 10), and the course was practical (9.7 ± 0.7 out of 10).
| Expectations From Workshop | Values |
|---|---|
| How to design an MCQ? | 5 (29.4) |
| What are the standards for the MCQ examination? | 3 (17.6) |
| What are the principles of designing MCQ? | 2 (11.7) |
| What does taxonomy mean? | 1 (5.8) |
Participants Expectations From the MCQ Workshopa
The participants were asked to write indicators for question assessment before and after the workshop. Before the workshop, only 12.5% of the participants knew these indicators. This number increased to 41% after the workshop (P < 0.05). Moreover, they were questioned about Millman’s criteria for the MCQ examination. Participants’ correct answers were increased from 2.75 to 3.05 out of four (P < 0.05) (Table 3).
| Pre-Test | Post-Test | Significance | |
|---|---|---|---|
| Knowledge about indicators, % | 12.5 | 41.1 | P < 0.05 |
| Millman’s checklist | 2.75 of 4 | 3.05 of4 | P < 0.05 |
Pre-Test and Pot-Test Results
A linear regression analysis was done to check whether a history of attending MCQ training workshops or years of experience correlates with improvement in pre-test and post-test answers. The analysis revealed an R square of 30%, and significant F was reported more than 5%.
5. Discussion
In this study, the impact of a short workshop for MCQs were assessed by pre-test and post-test examination. The participants’ expectations were fulfilled in concordance with previous studies regarding training programs and faculty development workshops. Although this satisfaction is not enough to change the behavior of examinees, this will be fundamental (13, 18, 19).
The participants’ knowledge and skills were also assessed and showed improvements in short-term evaluation. This would be promising to regard short training for MCQ as effective. Although previous participation in MCQ training courses did not demonstrate an increase in knowledge and attitude, it could be theorized that short-term repetition would yield better results (20). In other words, our results suggest that the one-day approach workshop seems effective considering the time-limits of faculty members, a finding in concordance with similar experiences (21, 22).
The assessment of trainees is considered the main step in their thriving process, which could be both summative and/or formative; this process yielded improved results in previous experiences (4, 23, 24). Meanwhile, improving the faculty members in their “assessment and feedback” capacities has been considered an essential task and an important method in improving the educational quality of the residency program (25, 26). If we are going to decrease the distance between the desired curriculum and the experienced curriculum, we have to improve the faculty members through methods that are both feasible and effective (27).
5.1. Conclusions
The one-day short workshop for MCQs improves the faculty members’ capacities while it is feasible for the faculty. Although previous participation in MCQ training courses did not demonstrate an increase in knowledge and attitude, it could be theorized that short-term repetition would yield better results.
