Experiences of Medical Students About Computer-based Testing: A Qualitative Study

1. Background

In the modern era, the use of technology is expanding rapidly, and the area of assessment and measurement has also been affected as an integral part of various technologies (1). Exams play a key role in student assessment as a practical tool, and the methods applied to evaluate students may have a significant impact on students’ learning (2).

With the advent of technology, the assessment paradigm has changed from paper testing to computer testing, which is regarded as a proper alternative to the former (2, 3). Computers have been used in medical science assessments since 1960. Since 1990, computer tests have been used more frequently widely. Some of the common approaches in this regard include computer-assisted testing, computer-based assessment, computerized assessment, computer-based testing (CBT), computer-aided assessment, online assessment, and web-based assessment (3).

The key advantages of using computer tests include saving paper and maintaining the green space. In this regard, Piaw emphasizes the pivotal role of universities and educational institutions in saving paper and believes that the replacement of paper-based testing with computer tests could save tons of paper, while remarkably decreasing greenhouse gas emissions and energy consumption (4). Among the other advantages of computer tests are the ability to evaluate efficiently and simplify the process of exam management (5) and to make correction and interpretation more reasonable and cost-effective than paper tests. Paper-pencil tests require printing questions and should be transported and stored, while computer tests eliminate these factors and save on finances and manpower. Furthermore, paper-pencil tests may lead to errors such as question inaccuracies and errors in the manual scoring, while these problems are eliminated in computer tests. Therefore, computerized testing may be a valuable and standard approach as a replacement for common evaluation methods (6).

According to Rudland et al. (7), students believe that computer testing has advantages such as ease of use, flexibility, and usability at any given time and place. In the mentioned study, some students stated that computer tests could effectively provide immediate feedback. The researchers also considered the possibility of cheating in this method due to freedom and availability of resources, and high flexibility was reported to be the main weakness of computer tests as students might not take these tests seriously (7).

2. Objectives

Given the growing use of CBT and the role of students the most important stockholders in this regard, the present study aimed to describe the experiences of medical students about CBT.

3. Methods

This qualitative research was conducted using conventional content analysis during 2018 - 2019. The participants included 10 medical students of Isfahan University of Medical Sciences and Kashan University of Medical Sciences, Iran who had experience in the pre-internship electronic exam. The exam encompasses 200 multiple-choice questions and is implemented every year at the end of the internship period in September and March. The reason for choosing the pre-internship electronic exam was that first the seventh amayeshzone (including the medical universities of Isfahan, Kashan, Shahrekord, and Yazd) was the first region holding the pre-internship exam electronically. Another reason was that the pre-internship exam has high stakes as it is paramount in the future and judgment of graduate students in terms of their medical career, and changes in the way of the exam from paper-based testing to electronic methods may affect their performance and destiny.

The participants were selected via purposive sampling, and data were collected via individual semi-structured interviews with the students willing to participate. Method of approaching the selection of participants was by phone. The interviews were conducted either face-to-face (n = 2) or via phone (n = 8). The location of the interviews was determined by the participants. The interviewer was a female PhD candidate in medicine who had obtained a master’s degree in medical education as well during 2011 - 2014. In addition, she had nine years of experience in medical education and six years of experience in qualitative research. After introducing herself, the interviewer explained the research objectives to the participants and obtained their written informed consent for the interviews; the duration of each interview was 15 - 30 minutes. The participants were initially asked to describe their experiences about the pre-internship electronic test. Based on their responses, probing questions were asked (What were the strengths and weaknesses of the test?; In which way would you prefer to take your exams?; Would you like to have your other tests done as such?). The interviews continued until no new themes emerged and data saturation was achieved.

The interviews were recorded using an audio recorder, and the text of the interviews was also transcribed word-for-word. Afterwards, the interviews were analyzed in MAXQDA 10. Initially, the researcher listened to the interviews and matched them with the transcriptions, and data analysis was performed using conventional qualitative content analysis. After reviewing the interview texts and immersion, the researcher attempted to obtain a general understanding. At the next stage, the meaning of each unit was determined and assigned with a code. The codes were compared based on similarities and differences, and those with similar meanings formed a category. After evaluation, new codes were placed in the previous categories, and new categories were formed as the interviews progressed. The initial categories were compared and merged to create common categories, and the researcher gained an overview of the phenomenon at the end of this stage. Finally, the categories and subcategories were titled based on their code and content.

3.1. Researcher’s Role

The main responsibilities of the researcher were designing the interview questions, selecting the participants, determining the approach and methodology of the study, performing the process of coding, and analyzing the accuracy of the data. Prior to the study, the researcher was unfamiliar with the participants. In order to show reflexivity in the research, the researcher attempted to immerse herself in the results at each stage of the research. Reflexivity was also achieved through analysis and writing, along with recording, discussion, and the review of the assumptions, she sought to preserve the memories of her reflection.

3.2. Trustworthiness

Lincoln and Guba criteria were used to ensure the rigor of the study (8), and peer debriefing and member checking were also employed to confirm the credibility of the research. To accomplish dependability, an expert in qualitative research was recruited. To confirm the confirmability an experienced external observer confirmed the study processes from implementation to data analysis. In terms of transferability, the stages of the study were described in detail.

4. Results

The participants included 10 students who were selected from Isfahan University of Medical Sciences and Kashan University of Medical Sciences and had prior experience in the computerized pre-internship exam. The participants consisted of seven female and three male students. They all consented to partake in the interview and completed the study. The mean age of the participants was 25 years. After data analysis, 150 initial codes were obtained; after eliminating the duplicate codes and merging the similar codes, 66 final codes were extracted and subdivided into two themes of examinee-related components and test-related components with multiple categories and subcategories (Table 1).

4.1. Examinee-related Components

4.1.1. Experience Aspect

According to the participants, CBT may cause fatigue in students.

“My eyes were tired of the paper maybe because we were not used to it. The computer screen completely exhausts the eyes” (p3).

Another student had a different view in this regard:

“In this test, we no longer had to bend our head over the question paper, which made us less physically tired” (p4).

The participants also believed that CBT reduced their concentration.

“The advantages of writing tests to me were that I felt I could focus more on the questions” (p9).

“Because it was actually the first experience, I did not have much focus. I am more used to paper and pencil; for instance, you can underline something or mark any questions you suspect” (p6).

Another participant had a contrary opinion in this regard.

“Our focus on this test was higher because on a paper test, matters such as an observer coming over to you or commuting may make you more sensitive and decrease your focus” (p3).

The students also believed that their speed and accuracy were more favorable in CBT.

“We were finally looking at the answer sheet, and the number of the questions we had not answered became clear to us. This is very good because it increase the accuracy. It is also notable that it was not if you selected the options by shifting, the whole test would be affected, because you chose each question separately, it would send you a reply and record it in the answer sheet” (p4).

“I was very satisfied because I always counted my score but it was very different of what was coming, but in CBT, what I counted was the same because I had no stress to make the right choice, and the test accuracy was very high” (p10).

Some students noted the simultaneous visibility of the answer sheet and questionnaire as a reason for their speed and accuracy.

“I think the speed was very high, and because you quickly see your question, there is no need to fill in the questionnaire, which is a time-consuming task. We quickly marked the question as soon as we saw it” (p3).

Since some of our participants had their first experience with CBT, they considered this factor as the reason for their stress during the test.

“The test was serious and we had no experience of electronic testing. Most of the students were somehow stressed that this is just a system now and we could not work with it” (p10).

Some participants attributed their stress to technical problems, such as difficulty entering the test page.

“We entered the username but did not enter the test page. It took a while, and stress was subconsciously rising” (p3).

Some of the participants also stated that at the beginning of the test, their stress increased and gradually decreased.

“Before the test, I had the stress that the test may be too complicated, I may get confused or the computer may shut down in the middle of the test, but I found it very easy and without stress” (p6).

4.1.2. Preferred Aspect

The participants believed that their experience with CBT changed their paper testing habits.

“We have taken tests on sheets of paper for several years and used to experiment since childhood, such as negative crosses, ignoring the questions that we could not answer, and easily turning over the questionnaire. Overall, we had a quick overview from the beginning to the end. But the electronic test was not like the paper test and, it was time consuming” (p4).

Another issue was the acceptability of the test. Some of the participants considered the CBT experience to be positive and were willing to try it again.

“All over the world, tests are conducted in this way, but it was an experience for the residency exam or the exams they give to enter foreign universities. How good it would be for universities to give all exams like this” (p3).

Another participant disagreed:

“Comprehensive tests such as the residency exam should not be implemented in this way because that would be costly” (p4).

4.2. Test-related Components

4.2.1. Technical Aspect

In this subcategory, the participants mentioned items such as software, hardware, and feedback. Furthermore, some of the participants referred to test software and its quality, question marking capability, and better CBT management.

“The test software was so that you would be sure or doubt your question. There were two choices, and either you did not answer a question or if you did not, the system itself would say that you did not answer the questions. There was an option that made it red, and it made skeptical questions orange” (p3).

“A good thing is the placement of the option that has to be re-examined, which means that you will re-examine your skeptical question after the exam” (p5).

In this regard, another participant stated that the test was not user-friendly.

“The test software did not seem to interact with me, maybe because I was not familiar with it and worked with the software for the first time” (p4).

Another participant pointed out the inappropriate contrast of the software screen.

“The test page was completely white, and the questions were completely black. It bothered my eyes” (p3).

Exiting the test hardware was another issue mentioned by the participants.

“The systems they used were outdated or very slow. For example, the mouse did not work or it was hard to choose an option. It took a lot of time” (p7).

Some of the participants considered feedback in terms of the test results.

“Test feedback was also good, and I think that the key questions came on the same day, not even a week” (p3).

4.2.2. Financial Aspect

In this subcategory, the participants mentioned no use of paper and the cost-effectiveness of CBT, as well as the need for paper in the CBT.

“In my opinion, electronic testing is going forward as an action by reducing paper consumption and saving it” (p8).

Another participant opposed this view.

“In my opinion, the problem with the exam was that some questions needed to be resolved, such as acid-base kidney or the questions regarding ophthalmology. I kept repeating I wanted a sheet, but they did not provide, and I had to solve the question on the back of a meeting card” (p7).

4.2.3. Executive Aspect

CBT Infrastructure and facilities were discussed as the participants needed to take the test.

“I agree that the computer test is good, but only if the necessary facilities and infrastructure are available” (p8).

“We had trouble logging in the system. The problem was central, and the systems were all interconnected. The university system was not capable of supporting many of these systems” (p2).

The same participant mentioned the possibility of a power outage during CBT.

“There was a power outage because all the systems were switched on at the same time.”

Discipline in the implementation of CBT was also highlighted in the comments of the participants.

“Discipline in a computer test can be an important factor. There is no problem with the test implementation. The authorities also guided us very well, and everything was really good, and they answered every question” (p3).

Some of the participants mentioned the discussion of justice in scoring.

“In the comprehensive exam we took in the 5th semester, any question that was improper were eliminated, and its scores was added to everyone. But now the scores were such that one of student said that two marks added to me, again he said that four marks were deducted from my score. They added 7 - 8 marks to some student, which means that the test was not corrected justly” (p10).

Participants pointed to the possibly of cheating in correcting computerized test score.

“In no society I don’t think this is the way to pass electronic test and this answer sheet we filled out earlier was a document for us to follow but there is the stress of being cheated and they can manipulated” (p4).

Another participant noted that the test session was safe, and it was not possible to cheat on the CBT.

“The exam session was safe, and it was not possible to cheat. The order of the questions was also different, and there was no cheating at all” (p3).

5. Discussion

The present study aimed to describe the experiences of medical students about CBT. Fatigue was one of the issues mentioned by participants, and some of the students believed that looking at the computer screen for a long time caused fatigue. This is in line with the results obtained by Mourant et al. In the mentioned study, the students partaking in a computer test also experienced more fatigue while reading on a computer screen (9). The other students in our study considered the fatigue of the computer test to be less than the paper test since there was no need to bend over the test booklet. This is consistent with the study by Alabi et al. (10), in which fatigue was reported to be an important limitation of paper tests versus CBT.

In the present study, the participants believed that reading the questions from a computer screen increased their focus on each question and the entire test, which was different from their experience in paper-based tests. According to Ozalp-Yaman and Cagiltay (2), the focus on computer test is higher than paper-based testing due to reading the questions on the computer screen, which increased focus on the questions separately. In the current research, the other participants reported less focus during the CBT. In the study by Boeve et al. (11), among the reasons for the decreased concentration of the students were technical problems and disquiet in the test environment.

The speed and accuracy of test performance were also mentioned by our participants, and the reason was claimed to be the speed of action in registering the correct option and no need to fill in with a pencil, as well as accuracy in registering the correct option, not moving the option, and low visual error. This finding is in line with the results obtained by Bodmann and Robinson (12). In the mentioned study, students also performed faster on the CBT compared to paper-based testing (12).

In the current research, the participants mentioned increased stress with CBT due to the novelty of this method, lack of experience, and possible software problems during the test. This is consistent with the findings of Hochlehnert et al. (13), which denoted the fear of computer errors and problems as a cause of stress during CBT. According to some of our participants, the computerized test method caused them to abandon their paper-based testing habits. In this regard, the study by Hochlehnert et al. (13) indicated that the elimination of their paper-based testing habit was an important reason for students to disagree with CBT.

The participants in the current research stated that CBT was more acceptable than paper-based testing and were willing to extend this technique to other academic tests. In this regard, the results obtained by Hariri-Akbari et al. (14) are in line with this finding. In the mentioned study, students were motivated to improve self-regulation and self-efficacy by CBT, and the technique was considered preferable to paper-based testing. On the other hand, the other participants preferred the paper test method as they believed that highly sensitive tests (e.g., residency exams) should not be conducted as such. The findings of Boeve et al. (11) also indicated that students preferred paper-based testing to CBT; although their scores on CBT were similar to paper tests, the students needed preparation beforehands. In the present study, the participants considered pretest preparatory sessions as a strategy for the better acceptance of CBT.

In our research, the participants noted some issues with regard to the test software, such as problems in showing the questions with a figure/shape content, which caused the loss of the score. Among the other issues in this regard were the lack of user-friendly software, incompatibility of the question color and background, and eye fatigue. The study conducted by Mahfira et al. (15) showed inconsistent results in investigating the views of students toward the aspects of media use (color, size, font, text, videos, images, and tables). In addition, the mentioned research indicated that the software features were desirable for students, and they preferred this method to paper tests.

Another issue with CBT was associated with the use of hardware, and the participants mentioned issues such as hardware disruptions, outdated computers, sensitivity to computers, and slow mouse movements. This is in line with the study by Jimoh et al. (16), which indicated that problems such as mouse abnormalities, display speed, scene graphics, and resolution could the CBT performance of examinees. Furthermore, our participants pointed out faster feedback on computerized test scores than paper tests, which was considered as an important strength of CBT. In this regard, Hochlehnert et al. (13) also addressed the benefits of computerized testing, such as fast and automatic feedback, scoring system, and detailed statistical analysis.

In the view of the participants in the current research, not using paper and saving this commodity made CBT more economical, and these were considered as the important advantages of this testing method. According to Piaw Chua (17), CBT as a policy to conserve the green space could reduce paper consumption. If educational institutions replace paper-based testing with CBT, the desirable outcomes will be the reduction of paper consumption and greenhouse gas emissions, as well as energy saving. Notably, some of our participants stated the occasional need for paper during CBT for computational questions.

Infrastructure and facilities are one of the most important discussions in CBT. The finding of Suryadi (18) regarding the challenges and opportunities of CBT have indicated that this technique is associated with multiple technical challenges and problems. In addition, the study by Khatib Zanjani et al. (19) demonstrated that for the successful implementation of CBT, several factors must be considered in regards to the infrastructures of the university, including technological, human, pedagogical, cultural, social, economic, managerial, administrative, and supportive infrastructures.

In the present study, justice in scoring was stated by the participants as another issue with CBT. They believed that the computerized test was not fair to the test correction and scoring, and there was the possibility of cheating. In this regard, the findings of Dingler (20) have also indicated that the comparison of paper and computer tests is necessary to ensure fairness in scoring. Cheating was among the other issues that our participants found less likely to be assessed on CBT. In this regard, the findings of Al-Qdah and Ababneh (3) are consistent with the present study. Accordingly, a high percentage of students acknowledged that despite its cost management and implementation, CBT could reduce and eliminate cheating.

In the present study, the participants were more likely to be fraudulent in correcting computerized test answer sheet. The results of the study by Alabi et al. regarding the limitations associated with paper tests indicated that subjective scoring was likely to be manipulated and delayed in announcing the results, which is inconsistent with our findings (10). The results of the present study generally indicated the high acceptance of CBT.

5.1. Limitations of the Study

One of the limitations of the present study was the small sample size and selection of the participants from only two universities in Isfahan and Kashan. Since administrators, planners, and technical experts could be considered as the stakeholders of the assessment system and examinations in addition to the students, the non-consideration of their opinions may be another limitation of our study.

5.2. Conclusions

Currently, the student assessment system in Iran is on the path of development and innovation. The importance of this issue is to the extent that one of the evolutionary packages in education discusses tests and their promotion as a step toward achieving the goals of this package. As CBTs have grown exponentially in recent years, it is essential to evaluate the opinions and experiences of the stakeholders of these tests. In this study, the participants stated their experiences regarding CBT from different perspectives, and their experiences generally indicated that CBT is more acceptable than paper-based testing. Furthermore, the participants stated that CBT could be a proper alternative to paper-based testing if the weaknesses were resolved. Therefore, it is suggested that other CBTs be evaluated similar to the present study to investigate the views of all the stakeholders involved in the assessment system.