1. Background
The rapid advancement in sciences and comprehensive access to the internet technology have led to changes in the information system and education. E-learning has emerged as a new paradigm in modern education, and education is no longer limited to the concept of face-to-face classes (1). E-learning is a subcategory of distance learning, which was introduced in the mid-20th century. With the global expansion of the internet infrastructure, e-learning has also grown rapidly and is on the agenda of numerous universities and educational institutions across the world (1, 2).
E-learning involves the use of information and communication technologies, such as the internet and multimedia systems, to improve the quality of learning and create a network for the transfer of course contents, communication between faculty and students, and facilitating the learning process through access to a wide range of resources and services. This educational system meets the educational needs of the individuals who are eager to learn, especially since time and place are completely optional in this educational approach, which saves tremendous time and money (1).
With the outbreak of COVID-19 in the world and given the capability of the infrastructure of universities and educational institutions, the importance of e-learning has become prominent and significant. Although e-learning is useful and effective for teaching various courses in the field of medicine, the success of e-learning programs requires appropriate principles of planning, design, assessment, and the continuous evaluation of these educational systems (1, 3). The structure of e-learning is rather complex, and several components (e.g., infrastructure capabilities and e-learning technology) play a pivotal role in achieving distance learning goals (4).
Today, human-machine interface analysis is used to evaluate various systems. This approach examines various aspects of a product, including applicability and user experience in using the product. The results of these analyses are applied to modify and upgrade the product, redesign the product, attract customers, and compete with other products (5).
User experience is a concept that has emerged with the recent advancement in technology, expressing the memory that is imprinted in the mind of the customer or user by using a product, system, or service. The international standard for human-computer interaction ergonomics defines user experience as an individual's perception, reaction, and expectation in using a product, system, or service (6). The formation of the concept of user experience has become particularly important with the growing competition between companies to attract customers and users. In this context, the applicability of a product alone is not enough to attract customers and users. In the decision-making process to purchase or use a product, the effect of the product or service on the customer or user is of the essence (7). Therefore, the user experience component plays a key role in using NAVID e-learning system to effectively transfer educational content from academic staff and lecturers to students and learners.
2. Objectives
Given the current state of the Iranian educational system and the need to use e-learning to transfer educational content and prevent the further spread of COVID-19, the present study aimed to evaluate the quality of NAVID e-learning system in terms of applicability and user experience. Our findings will help authorities take an effective step toward improving the quality of these courses and adopt appropriate policies and strategies to promote the level of e-learning in the future.
3. Methods
This cross-sectional study aimed to evaluate the status of NAVID e-learning system in the School of Nutritional Sciences and Food Technology of Kermanshah University of Medical Sciences (KUMS), Iran in 2020. The study protocol was approved by the Ethics Committee of KUMS, and the study was conducted in accordance with the Declaration of Helsinki (IR.KUMS.REC.1399.352). Participants were recruited from the faculty members and students of the School of Nutrition and Food Technology of KUMS. The faculty members and students who were willing to participate and used NAVID e-learning system were enrolled in the study.
The international user experience questionnaire (UEQ) was used to evaluate the users’ expectations of NAVID e-learning system in six domains, including attractiveness, perspicuity, efficiency, dependability, motivation, and novelty. The items in the UEQ are scored from -3 (negative extreme) to +3 (positive extreme). Laugwitz et al. have confirmed the consistency (Cronbach’s alpha) and construct validity of the UEQ domains by several usability tests (8). The attractiveness domain of the UEQ consists of satisfactory/annoying, bad/good, pleasant/unpleasant, undesirable/desirable, unattractive/attractive, and unfavorable/favorable questions with the Cronbach's alpha coefficient of 0.86. The perspicuity domain consists of intelligible/incomprehensible, hard learning/easy learning, easy/complex, and ambiguity/clarity questions with the Cronbach's alpha coefficient of 0.71. The efficiency domain consists of slow/fast, sufficient/inadequacy, practical/non-practical, and cluttered/organized questions with the Cronbach's alpha coefficient of 0.79. The dependability domain consists of predictable/unpredictable, supportive/disruptive, insecure/safe, and unexpected/expected questions with the Cronbach's alpha coefficient of 0.69. The motivation domain consists of undesirable/valuable, passionate/boring, attractive/unattractive, and lack of motivation/motivation questions with the Cronbach's alpha coefficient of 0.88. The novelty domain consists of useless/creative, traditional/new, technological/ordinary, and innovative/conservative questions with the Cronbach's alpha coefficient of 0.84 (8).
The UEQ has been translated to Persian with acceptable consistency (Cronbach's alpha coefficient: 0.81) (9). Based on the results obtained from the evaluation of 452 used products by 20,190 users, four cut-off points and five quality levels have been defined for each domain of this scale (Table 1) (10). The goal of this classification is to help UX practitioners interpret the scale results based on UEQ evaluations.
| Domains | Attractiveness | Perspicuity | Efficiency | Dependability | Motivation | Novelty |
|---|---|---|---|---|---|---|
| Excellent | < 1.8 | > 2.03 | > 1.9 | > 1.7 | > 1.7 | > 1.61 |
| Good | 1.68 - 1.8 | 1.77 - 2.03 | 1.5 - 1.9 | 1.47 - 1.7 | 1.38 - 1.7 | 1.14 - 1.61 |
| Above average | 1.19 - 1.8 | 1.25 - 1.77 | 1.06 - 1.5 | 1.15 - 1.47 | 1.01 - 1.35 | 0.75 - 1.04 |
| Below average | 0.7 - 1.19 | 0.75 - 1.25 | 0.6 - 1.06 | 0.75 - 1.15 | 0.5 - 1.01 | 0.25 - 0.75 |
| Poor | < 0.7 | < 0.75 | < 0.6 | < 0.75 | < 0.5 | < 0.25 |
Classification of User Experience Domains Based on General Benchmark
The participants were selected via convenience sampling from the faculty members and students who volunteered to participate in the research. Given the health protocols that should be properly observed during the outbreak of COVID-19, the questionnaires were electronically provided to the faculty members and students who used NAVID e-learning system in the School of Nutrition and Food Technology of KUMS.
The UEQ has a useful tool developed in Excel (Microsoft Corp.) to interpret and compare the results with the results of previous studies in the same field. This Excel tool has a tab (Sample_Size), which estimates the number of the required data to reach a specific precision in the estimation of the scale means. Using this tab, we estimated the sample size of the study to be 55 subjects based on the standard deviation (σ) of 1.1, precision (d) of 0.3, and the two-tailed significance level (α) of 0.05.
The UEQ Excel and the UEQ Handbook are available free-of-charge at www.ueq-online.org. In the present study, the quantitative data were expressed as mean and standard deviation, and the qualitative data were expressed as frequency and percentage. In order to examine the agreement of the participants and determine the lack of different interpretations of the questions, the Cronbach's alpha coefficient was used for each domain of the UEQ.
4. Results
Among 88 eligible students, including 55 bachelor’s degree and 33 master’s degree students, 48 students were enrolled in the research. Among the bachelor’s degree students, 25 subjects studied food sciences, and among the master’s degree students, 10 subjects studied nutrition sciences, and 13 subjects studied food sciences. As for the faculty members (n = 12), six were in the field of nutrition sciences, and six were selected from the food technology field. In total, 48 students (nutrition sciences = 10, food technology = 38) were enrolled. All the participants had used NAVID e-learning system for one year. A total of 60 participants completed the UEQ of NAVID e-learning and were included in the final analysis. Figure 1 depicts the distribution of the participants' responses to each item of the UEQ of NAVID e-learning system.
Distribution of responses to each item of UEQ of NAVID E-learning system
Fifty percent of the items in the UEQ had a negative score, and the lowest score belonged to the easy/complicated item (mean score: -1.13 ± 1.53). The other items (50%) had a positive score, with the highest score belonging to the motivation/demotivation item (mean score: 1.06 ± 1.57) (Figure 2). The mean score of the attractiveness domain was 0.161 ± 1.350, the mean score of the efficiency domain was -0.188 ± 1.169, the mean score of the perspicuity domain was -0.321 ± 1.305, the mean score of the dependability domain was -0.404 ± 0.999, the mean score of the motivation domain was 0.417 ± 1.248, and the mean score of the novelty domain was -1.101 ± 1.097 (Figure 3). According to the findings, NAVID e-learning system had poor quality in all the domains of user experience. The Cronbach's alpha for the attractiveness domain was 0.96, while it was estimated at 0.83 for the perspicuity domain, 0.75 for the efficiency domain, 0.62 for the dependability domain, 0.81 for the motivation domain, and 0.74 for the novelty domain, which confirmed the reliability and agreement of the users in responding to the items of the scale.
Mean score of each UEQ domain of NAVID E-learning system
Quality assessment of NAVID E-learning system in user experience domains
5. Discussion
The main finding of the current research was a negative user experience with NAVID e-learning system in all the domains, including attractiveness, perspicuity, efficiency, dependability, motivation, and novelty. Although a previous survey in this regard indicated the applicability of NAVID e-learning system in the provision of medical courses during the COVID-19 epidemic, it is important to consider the pragmatic and hedonic qualities of NAVID e-learning system (11). Success in distance learning during the current pandemic requires NAVID e-learning system to meet user expectations and create a positive user experience. To the best of our knowledge, this is the first study to evaluate the user experience with NAVID e-learning system as a well-known educational pathway in medical universities during the COVID-19 epidemic.
The hardware and software constructions in e-learning systems are the most important factor from the perspective of e-learning users (4). The study conducted by Ghasemi et al. at Shahid Beheshti University of Medical Sciences in Tehran (Iran) also indicated that the weakness of the technological and technical aspect of distance educational systems is the major disadvantage of e-learning systems in the field of medical sciences (12). On the other hand, the results of our study showed the weakness of NAVID e-learning system mainly in the attractiveness domain (overall impression), which indicated that the users did not like working with this system as the method could not create a positive e-learning experience. Our findings also demonstrated the weakness of NAVID e-learning system in terms of perspicuity, highlighting the fact that this system is difficult for the users to become familiar with and understand. Another weakness of NAVID e-learning system was observed in the efficiency domain, indicating that the users of this system needed unnecessary efforts to fulfill their tasks, and the system could not react fast enough to the users’ input. NAVID e-learning system was also weak in terms of dependability, which indicated that the users could not predict the system’s behavior and did not feel safe while working with the system.
According to the current research, another weakness of NAVID e-learning system was in the motivation domain, which indicated that the system was not appealing to the users. Our findings also showed that NAVID e-learning system was weak in terms of novelty, which is associated with the lack of innovation and creativity within the system and the inability of the system to capture the users’ attention.
In line with our findings, the user experience of eLisa e-learning system in Gadjah Mada University showed the poor quality of all the domains, with the Cronbach's alpha coefficient reported to be more than 0.7, indicating the sufficient consistency of the scale (13). Furthermore, the user experience of the student-centered e-learning environment, which is a Moodle-based learning management system, obtained by the UEQ from 213 computer science students at a university in Indonesia indicated the poor quality of the novelty domain, while the quality of the attractiveness and motivation domains was reported to be below average. In the mentioned study, the UEQ domains had sufficient scale consistency (14). On the other hand, Edmodo e-learning, which is a free e-learning system accessible on http://www.edmodo.com, has been reported to be high-quality in all the domains (15).
The results of the present study confirmed the reliability and agreement of the users on the quality level of NAVID e-learning system in the domains of user experience, including attractiveness, perspicuity, efficiency, motivation, and novelty. Only the score of the dependability domain was rather low (Cronbach's alpha coefficient: 0.62), which might be due to the fact that this feature is not essential to the user experience of a learning platform or due to problems associated with the interpretation of the items in this domain. In line with our findings, the user experience of the student-centered e-learning environment has been reported to have a low Cronbach's alpha coefficient (0.58) (14).
One of the strengths of our study was the simultaneous evaluation of the quality of the e-learning system and the agreement of the participants in answering the questions. Although the sample size of this study was sufficient, it was not possible to perform sensitivity tests based on the type of users (faculty members and students) and type of the offered courses (practical and theoretical) due to the limitations of the sample population. A previous study in this regard has shown that differences in gender and academic degree contributed to the participants' technology acceptance and their attitude toward medical education through NAVID e-learning system (11). Therefore, it is suggested that similar studies be performed in various medical universities in Iran and the data be analyzed as a set based on the type of users and type of courses, so that the results could be used to improve the quality of e-learning. Notably, the participants in the current research had used NAVID e-learning system for one year, and the user experience of NAVID e-learning system may improve over time. It is recommended that the user experience of NAVID e-learning system be enhanced by a novel and attractive design and increasing the capabilities of the system, such as interaction and data exchange with other information software (e.g., WhatsApp).
5.1. Conclusions
Success in e-learning during the COVID-19 pandemic requires special attention to the users’ expectations and experiences with various e-learning systems. According to the results, NAVID e-learning system had poor quality in the attractiveness, pragmatic (perspicuity, efficiency, and dependability), and hedonic aspects (motivation and novelty). Therefore, it is proposed that similar evaluations be performed in other medical universities in Iran.
