1. Background
The National Medical Education Assessment Center (NMEAC) in Iran, part of the Ministry of Health and Medical Education (MoHME), oversees medical exams, from question design to results analysis. To enhance efficiency, NMEAC recognized the importance of integrating electronic item banking and information technology in test development. Despite challenges such as low-quality software and concerns about item confidentiality, adopting IT was crucial for streamlining test processes. The NMEAC aimed to implement a system that adheres to both technical and non-technical standards, ensuring efficient and secure test management (1).
Question banking stores exam questions with relevant metadata such as content area, learning outcomes, difficulty, and source. Questions are organized using a catalog system, updated after use, and arranged by objectives to create exams. Key benefits include efficient test construction, quality assurance, transparency, and performance monitoring. Item banks support uniform teaching and assessment standards, while metadata aids administration. Exams are created through manual selection or computerized randomization, which helps prevent plagiarism but raises concerns about comparability and reliability, addressed by aligning questions with consistent standards (1).
An item in testing includes a question, instructions, response processing, and feedback, defined by IMS QTI as the smallest assessment unit. With the rise of e-assessment, diverse item types require item banks to manage complexity. Research focuses on item design for specific domains, addressing evolving needs in complex assessments. The system’s key stakeholders — item designers, authors, administrators, translators, project managers, and psychometricians — handle item creation, management, and quality control. Essential use cases include creating, editing, and managing items. Nonfunctional requirements such as reliability, security, and scalability guide the system architecture to meet stakeholders’ needs effectively.
In the literature related to item banks, numerous studies have focused on creating high-quality items, item selection methods, test construction, and security issues. However, there is less research providing information on item bank design and how items are collected (2). We seek strategies for effective item design and item bank utilization. Research indicates that item banks enhance test security and ensure consistent difficulty distribution, improving item performance and test design through streamlined writing, reviewing, and automated tools that incorporate statistical information.
The term "Item Bank", coined in the mid-1960s, was used to describe a group of test items "organized, classified, and cataloged like books available in a library" (3). Subsequently, Bruce Choppin and others interested in item banks based on the item-response theory (IRT) have attempted to distinguish between an "item bank" (a set of questions calibrated with an IRT measurement system and equal to a common scale) and an "item pool" (a set of items grouped by content but not calibrated). This distinction has not been widely welcomed, and the terms "item bank" and "item pool" are often used interchangeably today (4).
The distinction between an item bank and an item pool is crucial in medical education assessments. An item bank allows for standardized test development, precise difficulty adjustments, and improved measurement accuracy. In contrast, an item pool may lack the consistency needed for high-stakes testing. Given the sensitivity of medical assessments, a structured item bank ensures fairness, reliability, and alignment with educational standards. According to AMEE Guide No. 71, all test systems require an item bank and an administrator responsible for maintaining and classifying items according to the test blueprint. The administrator ensures the item bank remains current by communicating with the scientific committee about required question types and quantities. Factors affecting item bank size include test repetition, item usage, reusability policies, and whether answer keys are provided after tests (5).
Using a blueprint for test construction ensures alignment with educational objectives and allows for varying item difficulty levels. By employing a predetermined blueprint, tests maintain validity and reliability while covering essential concepts fairly. This approach prevents the assessment of irrelevant knowledge and avoids bias towards specific topics. A study by Sclater et al. found item banks typically contain between 500 and 1,025 items, with their size increasing over time, primarily influenced by financial resources rather than the item bank’s age. This strategy optimizes resource use in the assessment process (6).
Random extraction from item banks often fails in medicine and health sciences due to rapid developments, leading to outdated items. Additional quality control is necessary to ensure appropriate item combinations and maintain the relevance of test questions (7). Examples of popular commercial software that provides comprehensive test construction services and includes question banks are FastTEST Web (8), Perception, Random Test Generator PRO, and Test Creator; also, TAO (9) is an example of an open-source question bank. The IMS International Question and Test Interoperability (QTI) standard for including metadata items is not met for some available item banks. These item banks face the challenge of interoperability with other item banks or online test systems. Some systems used pre-testing to calibrate items (10, 11), while others used the specialized title and subject of each item to label items with appropriate metadata (2).
A study conducted in 2004 on item banks showed that most of the studied item banks were implemented in various ways using SQL databases and XML technologies (10, 12). McAlpine et al. developed an outline of the elements that constituted an item banking system, and the workflow and initial diagrams provided an overview of the overall system and user interaction (13). Chituc et al. developed an item bank for computer-based assessments, highlighting its role in service-oriented architecture (SOA). Their project encompassed item design, test construction, and analysis of reports, using unified modeling languages (UML) to represent key components of the system (14).
Nuntiyagul et al. developed a tool for categorizing items in an item banking system using feature selection. This tool classifies items lacking predefined keywords based on item text, and user satisfaction evaluations demonstrate the system’s acceptable accuracy (15). Nguyen et al. proposed using the particle swarm optimization (PSO) algorithm to create tests that achieve specific difficulty levels. This method generates tests from item banks with varying difficulties, aligning item difficulty with user assessments from prior tests, demonstrating superior performance across most criteria (16).
Janpla and Wanapiron studied the requirements for designing a question-banking and intelligent test system, identifying five key modules: Ser management, question management, examination management, evaluation management, and scoring management, forming the system framework. The intelligent question banking system integrates five modules within a cloud computing framework, designed by the researcher to enhance functionality. Machine learning is applied to examination management, question management, and evaluation management modules. This enables automatic test classification, selection of appropriate items for test-takers, and automated assessment with feedback provided to examinees based on their results (17).
Item banks are crucial for educational assessments, especially in medical sciences, but research on optimizing item design strategies is limited. Existing studies focus on technical aspects, like item categorization, validation, and the integration of advanced technologies, with insufficient exploration of adapting these strategies to evolving medical education needs. In Iran, implementing a question bank for medical assessments requires addressing security, scalability, question quality, and cost management, alongside local challenges such as limited financial resources and slow updates to reflect new medical knowledge. A tailored framework for Iran, aligning with both global best practices and local needs, is essential for success.
2. Objectives
How can strategies be effectively applied in the design of items within the framework of an item bank?
3. Methods
3.1. Study Design
This study employed a developmental research design to create, implement, and refine a framework for integrating item banks into national medical science examinations. A cyclical process of analysis, design, development, and iterative evaluation ensured alignment with the specific needs of medical assessments.
3.2. Document Study
Diverse sources — including textbooks, peer-reviewed articles, academic theses, NMEAC internal reports, and international standards — were analyzed. Sources were selected based on relevance, credibility, and alignment with study objectives, and the extracted information guided framework design and development.
3.3. Framework Development
Requirements were analyzed, and essential features were identified for integrating the question bank with the examination system. Automated and AI-based solutions were explored, and multiple modules were implemented to facilitate national exams. Evaluation considered question accuracy, diversity, difficulty, and participant feedback. Multi-stage expert reviews, iterative feedback, and structured validation procedures minimized bias and ensured consistency, accuracy, and relevance across all components.
4. Results
Key components of the test development cycle encompass building the item bank, designing tests, delivering tests, collating test data, analyzing test data, and importing statistics into the item bank (18). The Faradid web-based software was selected and tailored to meet the specific needs of the NMEAC after a thorough review of existing software. This software is based on a multi-topic shared model (19). Implementing Sadaf et al.’s step-by-step approach (20) ensured the development of a high-quality item bank that meets specific criteria for the assessment center. Modifications were made to adapt the process to medical science examinations.
This research utilizes components from Figure 1 to establish an item bank in both Web-based Item Banking Tools (WIT) and a quarantine environment. The web-based software supports various question types, including multiple-choice questions (MCQs) with 2 to 9 options, extended matching questions (EMQs), key feature (KF)/key feature problem (KFP) questions, true/false questions, and patient management problems (PMPs), with features for entering questions via an embedded word processor or importing from a Word file, along with plugins for complex formulas and diagrams. User log files are also recorded for retrieval.
Figure 1.
Architectures of an item bank system for national medical sciences examinations
4.1. Step 1: Input Users
- Various roles are defined, including administrators, typists, subject-matter experts (SMEs), reviewers, editors, and security users.
- Administrator responsibilities include defining users, access levels, and test-related resources.
- Typists handle corrections and modifications during the final stage of testing.
4.1.1. Preparing Comprehensive Blueprint and Test Blueprint
Comprehensive blueprints and test blueprints specify concepts for evaluation based on Bloom’s taxonomy. Questions are gathered according to the comprehensive blueprint, and a table is utilized to allocate items for each test.
4.1.2. Uploading Resources’ File and Users’ Information
The system embeds the latest PDF files of required resources for professors, and users’ information can be uploaded in groups using Excel format.
4.1.3. Exam Description
The exam management module allows adding, editing, deleting, and displaying exams, requiring the addition of majors, subjects, topics, and objectives beforehand.
4.2. Step 2: Item Management and Improvement
In the item management process, SMEs submit questions for review. First-level reviewers evaluate submissions and provide feedback if necessary. Approved items are sent to editors for a final quality check, ensuring alignment with educational goals and suitability for the target audience. Each question is assigned a score based on committee feedback, which influences its selection for the item bank, ensuring high-quality assessment items.
4.3. Step 3: Test Management
4.3.1. Selecting Questions for the Test
This step is performed in the quarantine environment with the presence of editors. Questions are selected based on the test blueprint and the determined proportions of the degree of difficulty. In the test booklet-making module, it is possible to randomize the items and questions. If the test is implemented electronically, on the test day, the selected set of questions is easily transferred to the test software in a safe environment. If it is used for a written test, it only needs to be printed and copied.
4.3.2. Detecting Similarities Between Questions and Mini Decision Support System
After final item design, software was created to detect question similarities in the booklet-making phase. Users can set similarity thresholds and apply filters. The system also checks for issues like option similarity and unusual question lengths, alerting users to approve or reject items that do not meet criteria.
4.4. Step 4: Testing Questions’ Answer Key
The answer key construction for the questions is separate from and independent of the test. The test, after completion, is visible, can be printed, and can be downloaded as a file.
4.4.1. Item Analysis
This step involves reviewing questions and generating statistical information, such as difficulty and discrimination indices. Classical analysis and IRT are used to evaluate test psychometrics. If issues are identified, the problematic items are suspended until reviewed. After each use, the indices are updated in the item bank.
4.5. Step 5: Output
- Testing reports and item analysis reports are generated, providing detailed insights into the performance of the test and individual questions.
- Item writers receive feedback on their performance based on the analysis reports, helping them understand areas for improvement and enhancing the quality of future question designs.
4.6. Challenges
Challenges during the project included ensuring question confidentiality. To address this, the process was divided into online and quarantine phases: The online phase allowed submissions, while the quarantine phase safeguarded confidentiality during selection. Additionally, many end-users were clinical specialists with limited computer skills, necessitating substantial helpdesk support.
4.7. General Specifications
- Security: All information is stored in an encrypted database, with no direct user access. Transactions occur through the system, which defines access levels based on user roles. Users can log in with a username, password, and one-time password (OTP) to access specific item bank sections and add new items.
- Usability: In designing the system, easy operation and graphic design were considered. Users could easily find out the number of required and designed questions for each lesson and each field on the page related to their desktop.
- Item writers training: A virtual modular training course consisting of 7 modules was prepared and implemented at the Smart University of Medical Sciences of Iran. This course aimed to familiarize participants with designing and assessing various types of questions in the field of medical sciences (21-23).
- Supporting: Virtual support of 24/7 ensured continuous communication and expert assistance, maintaining professor collaboration and effective implementation of the design process. The system registered 900 professors and received over 8,000 questions, with 6,000 reviewed for test booklets. Professors’ departmental needs were addressed through meetings, leading to revisions like adding supervision levels in dentistry. An expert committee ensured quality assurance, promoting effective collaboration and maintaining high design standards for the medical test system.
This study presents a framework that not only enhances the quality of medical exam questions but also incorporates unique features such as high security, scalability, and clinical reasoning assessment. While previous studies (24, 25) have primarily focused on the technical design of item banks and summative testing models at the undergraduate level, this study emphasizes the specific needs of medical exams and offers continuous support for question designers through virtual training courses and 24/7 assistance. Compared to the I-MIB model for undergraduate evaluations (26), the proposed framework offers greater capabilities for more accurate and effective assessments. Considering these factors, item banks can significantly improve the quality, efficiency, and educational value of high-stakes national medical science exams. The considerations examined in this study are outlined in Table 1.
Table 1.Key Considerations for Integrating Item Banks into National Medical Science Examinations
| Considerations | Details |
|---|---|
| Quality and validity | Item analysis, regular item rotation, maintaining psychometric properties |
| Development and maintenance | Investment in development, collaborative efforts to reduce costs |
| Technological integration | Computer-based testing, security measures |
| Educational impact | Focus on conceptual knowledge, use summative assessments |
| Practical considerations | Guidelines, review processes |
5. Discussion
This study examined the implementation of a computer-based item bank to enhance national medical examinations. Implementation involved question collection, multi-stage expert reviews, and final test assembly within secure online and quarantined environments. Collaboration among administrative leadership, technical support, and ongoing assistance for question designers was key to success. Technical challenges included Persian language processing, item version control, and data security, partly managed through parallel manual processes. Limitations involve limited generalizability to non-medical disciplines, scalability challenges for diverse content types, and the need to strengthen security with tools such as biometric authentication and digital signatures. Despite these challenges, integrating item banks provides clear benefits, including precise knowledge assessment and flexible test design. While well-suited for medical sciences, application to other fields requires contextual adaptation. Future research should focus on improving scalability, enhancing security, and exploring broader applicability. These findings align with previous research on item bank development in high-stakes medical assessments, emphasizing security, scalability, and contextual adaptability (14, 15). Similar to the I-MIB model (26), this framework incorporates multi-stage expert reviews but differs in its explicit focus on clinical reasoning and bilingual interface support, highlighting that core item bank principles are transferable, yet domain-specific adaptations are essential for validity and reliability.
