Opportunities and Challenges of Online Take-Home Exams in Medical Education

1. Context

Student assessments are being transferred to technology-based assessment methods, e.g., computer-based tests (CBT). The CBT is used by the Medical Education Assessment Center to holding international GRE and TOEFL assessments and national exams such as the Ministry of Health Language Exam (MHLE) and the subspecialty assistant admission exams since 2017 in Iran.

The CBT is managed automatically by computers, or similar technologies, and these exams have the following advantages over traditional methods of examination: (1) reduced human errors, (2) rapid scoring, and (3) feedback (1). But, the most important disadvantage of CBT over traditional exams is the distortion of the exams’ credibility as a result of students' unethical conduct (2). This means that the main assessment method is still the in-class exam (ICE) in some universities.

Given the circumstances in recent months and the COVID-19 pandemic in many countries, including Iran, and the need to minimize traveling, prevent gatherings, and observe social distancing, and on the other hand, the need to hold university and national exams to continue the process of higher education despite the recent pandemic, taking exams has changed to online take-home exams. Online take-home exams are similar to online computer exams, except that, in the former, the examinees are able to take the exam in a place of their own choice and use their own computers (3). Despite the benefits of online take-home exams, the risk of cheating increases.

The present study was conducted to investigate the conditions for holding online take-home exams, challenges, and strategic plans to adopt for their implementation and to answer the following questions:

- What are the advantages and disadvantages of online take-home exams, and what are the risks of their application in non-competitive medical exams?

- Can the challenges of these examination methods be resolved to ensure their more extensive use in situations of crisis in Iran?

2. Evidence Acquisition

In the present review article, a search was carried out in Scopus, Google Scholar, and Google’s general search engine using the following keywords and search strategies: "Take-home exam", OR "Take-home assessment", OR "Online exam", OR "Online assessment", AND "Higher education". Given the volume of the material and study objectives, the results retrieved were confined to English and Farsi articles published between 2009 and 2020. A total of 35 documented content, including articles, books, and web pages relevant to the subject of online take-home exams, were selected and assessed, and after analysis and integration, the results were compiled in a sequence to meet the study objectives.

2.1. Research Background

Tao & Li (2012) conducted an experimental take-home computer exam on the students of nutrition and assessed its advantages and challenges. Their results showed that this method brings peace of mind to both the teacher and the examinee although the examinees’ cheating remained the biggest challenge (4). In a study conducted at Virginia Military Institute, Frein (2011) investigated three types of experiments, including pen-and-paper and in-class and take-home computer exams in terms of method and examinees' performance. The study showed that the pen-and-paper method was the preferred method for most examinees. Despite these results, cheating remained a concern when the individual chose both the exam's time and place. Since the participants were from a military school with highly critical codes of ethics that had to be observed and could pose a risk of expulsion for the student if violated, the codes were explained by the teachers at the start of each course (5).

Kelwyn & Denise (2017) developed a conceptual framework for detecting cheating in online take-home exams, which can be useful for teachers switching from traditional in-class to online take-home exams without invigilators (6). The Cross River University of Technology, Calabar, Nigeria, developed a system called Crutech Online Examination System by a group of computer scientists and others (7). The features of this program include the random arrangement of questions and answers, timing the exam for each candidate using a timer, the use of a preparation system to manage saving the questions, and the allocation of exam ID and exam planning. Encryption was used to verify the student's identities before the exam. Bedford et al. (2011) conducted a pilot study to assess the use of distant observers for online lessons. The distant monitoring software was needed to verify the candidates' identity and visually recognize them. This device had a USB port and contained a fingerprint scanner and a 360-degree camera. The scanner and camera confirmed the candidates’ ID, and the camera monitored the candidates’ activities during the exam (8).

The Institute of Company Secretaries of India introduced a compulsory pre-exam as a prerequisite for enrolling in CS exams. Once the payment was confirmed, an invitation message was sent to the candidates with information, including their username, password, and pre-exam date. Then, the candidates could log into the application by entering their own information and join the online portal for giving the pre-examination test (9). McGee et al. (2013) investigated the nature of misconduct in online exams and the ways to reduce it. According to the authors, teachers and designers could promote ethical conduct, on the one hand, and reduce the attraction and ease of cheating and plagiarism, on the other hand, by careful planning of curricula. According to the results, the five forms of misconduct include collusion (helping others or using cellphones and headphones), deception (using someone else's assignments and articles or copying their answers without their permission), plagiarism, manipulation of technology (cutting off the internet or intentionally creating problems in the system), and providing incorrect information (the issue of identification) (10).

Institute CECE supports the design, development, and delivery of virtual learning courses at the University of Central Oklahoma (UCO). This institute investigated the reduction of cheating in online courses. According to the results, cheating prevention methods include the randomization of questions and response choices for each examinee, locking the examinee's system, and locking the right-click function on the mouse, print, and internet search functions (11).

Several studies have listed various types of cheating methods used by examinees, including plagiarism, planning to access the questions and answers before the exam, having someone else take the exam instead of oneself, and illegally accessing the exam bank or exam guide booklet. According to these studies, the ways to minimize cheating in online exams include implementing invigilated final exam, using descriptive questions, using short-answer or comprehension questions, fixing the exam time, use of randomized questions and answers, using question banks, generating different versions of question books for the examinees, and limiting the number of times an examinee can take the same exam (11-14).

The Purdue University has developed an online education and learning system and proposed the use of software called Blackboard, which uses different assessment tools to prevent plagiarism (15). The features of this software for the prevention of cheating and plagiarism include: (1) creating a question bank; (2) activation of blackboard plagiarism detection software; (3) explaining details and policies of plagiarism on blackboard; (4) randomly changing the answer key; (5) considering online monitoring solutions such as Proctortrack.

Zhao Qiao-Fang and Li Yong-Fei (2008) presented a model for online exams by combining client-side and server-side technologies. Their model uses the JSP architecture, with the browser on the client-side and the main logic on the server-side. This model has the following operational functions: Managing questions, producing exam booklets, and online examination (16). Abass et al. (2017) introduced a web-based exam system that composed of three phases (17): (1) display phase, provides the user with a user interface; (2) logic phase, processes and serves the users' requests; (3) database or question bank phase, serves as a pool of questions and answers.

In a new endeavor, the ETS Institute proposed conducting TOEFL home edition due to the outbreak of COVID-19. Students can take this exam on their own computers, and it is invigilated online by a human observer using ProctorU. During the exam, an invigilator monitors the candidate and his computer page through a video camera to ensure all the exam stages are properly carried out. The entire exam session and the candidate's image throughout the exam are recorded and then shared with institutions as part of the TOEFL IBT score report.

The main issues concerning the exam setting generally include: (1) the candidate must be alone in an adequately lit, quiet room; (2) the desk and its surroundings should be visible; (3) the use of earphones and headphones is entirely prohibited; (4) men's ears should be visible at all times and not be covered with hair, hat, or other items; (5) the use of any kind of recording device is prohibited; (6) the use of sunglasses or other accessories that make the candidate's face invisible is prohibited; (7) the use of accessories such as jewelry, tie pins, cufflinks, cosmetic pins, French combs, headbands, and other hair accessories is prohibited; (8) the computer in use should be a desktop or laptop, and using smart cellphones or tablets is prohibited; (9) the use of Macintosh computers and IOS systems is prohibited, and only Windows-7 or higher versions are allowed; (10) the camera should be an integrated part of the computer or a separate webcam and should be movable to enable the 360-degree vision of the room before the exam, including the area above the desk; (11) the candidates should sit on a standard chair (not on a sofa or bed) and should not be lying down or seated on the floor.

Compliance with the following rules is essential during the exam: (1) the candidate should make sure that the invigilator can see him on the camera. Suspicious moves can invalidate the exam; (2) using unauthorized equipment, including cellphones, prewritten notes, or course books, is prohibited; (3) after the listening part, the candidate is given 10 minutes of rest in which he can leave his chair. If he fails to return to his chair on time, the exam will be canceled, and the fee will not be reimbursed; (4) the candidates can view their reading and listening scores at the end of the exam (18).

Given the outbreak of COVID-19 and the quarantine orders, in addition to TOEFL tests, all official IELTS centers have also been closed, and IELTS Indicator has been set up instead as the official and temporary replacement for the original IELTS. The applicants of this test should enter their details in the registration form to receive their username and password. This test merely provides an IELTS-equivalent score for students admitted to universities, and they should register and take the original test and submit the original test result to the university when the situation has abated. Nonetheless, some universities do not accept this temporary certificate. Those who need to take the test for immigration and visa purposes cannot use this tool and should wait until a return to normal conditions to take the original test.

The checklist for taking the alternative IELTS includes: (1) a quiet and comfortable place for full concentration during the test; (2) laptop or desktop computers (cellphones and tablets cannot be used to take this test); (3) proper internet connection and speed; (4) proper quality headphones for the listening part (wireless headphones are allowed); (5) a webcam; (6) downloading and installation of ZOOM; (7) ability to download and install Safe Exam Browser (SEB) (19).

3. Results

Concerning the implementation of online take-home exams in Iran and given the sample studies conducted, the global research, and past experiences of experts at the Medical Education Assessment Center on this subject, the challenges in this method of examination are detailed in Table 1.

The candidates must be connected to the internet to take online tests, and it should be noted that some people in remote and rural areas have no or little access to the internet or have a low internet speed. Considering that the examinee should remain online throughout the exam to be fully monitored, power outages in the area where the candidate lives are also a potential risk.

Monitoring the candidate's performance during the exam is another issue for which strategies have been proposed; however, legal and belief impediments prevent their implementation. For example, to ensure that the candidate is alone in the exam room, the surrounding area should be fully inspected with the camera throughout the exam, which may cause legal problems. For instance, someone in the data center might abuse these videos or be accused of their abuse. Moreover, regarding academic exams, the teacher may abuse the candidate in the future depending on the conditions of his surroundings and financial status.

Another issue is the costs imposed on the candidate, who needs at least one camera or webcam to ensure his proper identification and the assessment of his settings, an internet connection with a proper speed, and also a computer system while some people may not have access to these facilities. Meanwhile, cheating remains the biggest challenge in this area because the individual decides the exam setting himself and can create a setting that facilitates cheating. To prevent these incidents, the use of an online monitoring system is proposed that assesses the candidate's exam setting and conduct; however, this measure itself poses further challenges, including: (1) the candidate may display a prerecorded image; (2) there may be someone in the room on the blind spot of the camera; (3) a person masked with makeup may take the exam instead of the candidate; (4) the candidate must be monitored throughout his breaks from the exam as well, which are necessary for longer exams; (5) examinees often use Bluetooth devices that are small and undetectable and may use notes or write notes on their palms; (6) it is easier for female examinees to use unauthorized hands-free devices due to their hijab.

It is also imperative to give candidates the support of trained technical human resources in educational institutions for dealing with any technical issues. Moreover, taking images of private living spaces is associated with ethical constraints, and sometimes other family members are forced to leave the house due to their limited physical space.

Regarding accessibility to the internet for the online exam, as well as the costs imposed on the candidate for authentication and environmental assessment, the candidates must check the internet connection status at home and the service of the internet provider beforehand. If the conditions are not suitable, they must use an internet provider center such as internet cafes. Also, due to the physical limitation of the home environment, if the conditions are not suitable, you should use centers as internet cafes. Training of technical staff is one of the infrastructures of online exams and should be on the agenda of exam centers. Also, regarding the legal constraints to the monitoring of the candidate’s performance during the exam, as well as the mentioned practical-ethical constraints, a letter of commitment to maintain the security and confidentiality of the information should be prepared by the exam organizers and notified to the examiners (20).

Previous studies have proposed solutions for preventing such incidents, which are divided into academic and technical solutions for reducing misconduct, as shown in Boxes 1 and 2. Box 1 refers to the use of random questions from the question bank to limit cheating. This method is practical when there is no legal prohibition on the use of different questions. Moreover, the candidates can misuse the sequence of questions for cheating purposes. The likelihood of cheating is reduced if the questions are mixed up and random questions are chosen from a question bank, even in the case of contact with the other candidates. Moreover, it is suggested that the exam commencement time be the same for all candidates, or it is accessible only within a specific interval of time. It is also advised that the duration of the online exam be short (between 15 and 30 minutes).

Also, when a limited number of questions are shown (one question per page, or a maximum of five questions per page), only these limited questions can be transmitted to others at each time. The possibility of cheating is also reduced if questions that require a better understanding of the concepts instead of their memorization are used or if the conceptual and memorizing questions are mixed up. In these cases, even if the candidate has access to resources, he still needs more time to find the answers.

Regarding software capabilities, the system should be able to identify the candidate's device types and operating system so that the exam session can be deactivated and canceled if the candidate uses Linux, IOS, or Android operating systems. Moreover, the software should be able to provide the candidate with a disposable username and password once his documents and certificates have been obtained and then send the assigned username and password to the central server with the candidate's full details. It also should be able to sign out of the assigned username and password if the internet connection is lost. The use of SEB has also been discussed, which has features including not displaying the URL and disabling the reload, back/forward, and print functions.

Software and hardware considerations in the implementation of online take-home exams:

- Items that should be considered by the exam administrator: (1) appropriate personal internet bandwidth to respond to the users’ concurrent connection and receive audio, video, and images; (2) a server with proper technical specifications; (3) choosing suitable software; (4) trained and coordinated technical workforce as the support team for providing urgent guidance, responses, and support to the candidates; (5) maintaining data security and confidentiality.

- Items that the candidate is required to observe: (1) having a suitable computer; (2) being alone in an adequately lit and quiet room; (3) not using unauthorized electronic devices during the exam (based on exam guidelines); (4) checking the internet connection before the exam and ensuring sufficient bandwidth; (5) complying with the important issues announced by the administrator.

One of the ways to increase security is the use of online monitoring tools. The concurrent use of automatic monitoring devices and an observer can prevent the use of digital tools for cheating, such as smart cellphones and watches. These tools can store data related to movements of the head and eyes and also hands when clicking on the keyboard. They can also store the candidate's voice for its detection if heard during the exam (30). Table 2 lists five online monitoring tools developed by various individuals and universities.

A suitable online monitoring tool can be selected based on the features of each of these systems and the number of examinees supported (34). Although using these systems increases the security of online take-home exams, their implementation is time-consuming.

4. Conclusions

This study examined the opportunities and challenges of online take-home exams. The biggest advantage of such exams includes reduced anxiety for the students, reduced human errors, quick scoring, and the opportunity to assess higher cognitive skills according to Bloom's taxonomy. The main challenge of these exams, as repeatedly discussed, is the evident risk of unethical conduct from students. Students no longer use the old cheating techniques but rather employ electronic devices and methods for cheating in the modern-day. The present study investigated various cheating techniques adopted by students and the strategies presented in previous studies, which have been briefly examined in the results section.

Given the COVID-19 pandemic and the situation developed in the country, the unclear state of the pandemic based on scientific evidence, and the uncertainty about when the situation will become stable and when the country's educational institutions can again hold exams, prompt measures need to be taken for the development of online take-home exams. The use of combined methods has a significant positive effect on the authenticity of these exams. One of the suggestions is to hold one-question-per-minute exams, in which return to a question and changing the answer is not an option. Other strategies include using question banks and also giving equivalent random questions to different candidates which are not necessarily similar, and mixing up the questions and answers, and combining different types of questions (multiple-choice, right/wrong, analytical/problem-solving questions, verbal statements, etc.). Combining these methods requires more time to answer the questions and can constitute a means of preventing or limiting cheating. Although these measures do not ensure 100% security, the exam health can be improved a lot with their application.

In addition to exam-holding security methods, statistical analysis based on available conceptual frameworks is essential for detecting cheating in non-invigilated exams. For example, if a person's score in an academic progress exam is different by more than two standard deviations from his overall score in that knowledge field, the educational institution should be able to re-administer his exam in person or at home. Reviewing and updating policies at educational institutions are necessary for ensuring that exams are properly held, and the authorities’ and students’ concerns are abated. Meanwhile, organizations need to have a strong mechanism of punishment for violation of their rules and refusals to comply with the ethical rules of exams.

Given the above practical considerations, to improve exams’ security, it is necessary to ensure a safe assessment method compatible with the educational objectives that guarantee that cheating students cannot obtain undeserved qualifications. It is, therefore, necessary to use a combination of different assessment methods and frequently monitor them at all stages to ensure the validity of assessments and preserve the credibility of universities.