1. Background
Educational institutions globally have increasingly adopted electronic learning (e-learning) tools to meet growing educational demands (1). This approach makes education more accessible by allowing learners to engage from any location and at any time, saving both time and money. Consequently, e-learning systems have extended educational opportunities to a global audience (2). Failure to identify the factors that influence the acceptance of e-learning systems could render substantial investments in this area ineffective, preventing students from achieving the desired learning outcomes (3).
One of the most widely used and comprehensive systems in Iranian medical universities is the NAVID e-learning system. Despite the robust virtual education infrastructure of the NAVID system, several challenges have been identified (4). Addressing these shortcomings is essential to improving the quality of virtual education. Neglecting usability concerns may result in poor system adaptation and a failure to meet user expectations (5).
A systematic approach to evaluating the structure of virtual education systems involves assessing their usability. Usability in a system reflects its capacity to operate effectively and efficiently while ensuring user satisfaction (6). Various methods have been developed to assess the usability of e-learning systems (7). One expert-based method is the cognitive approach, which involves analyzing the user’s thought process when interacting with the system (8). This cognitive approach involves a task-based evaluation that identifies problems through simulation. It measures all three dimensions — effectiveness, efficiency, and satisfaction — and reveals usability issues (9). This technique is based on the assumption that evaluators can adopt the user’s viewpoint and apply this perspective to specific task scenarios to pinpoint design flaws.
2. Objectives
Given the extensive and nationwide use of the NAVID system and reports concerning its effectiveness, efficiency, and user satisfaction, this study aimed to evaluate the system's usability using a walkthrough cognitive method. The goal was to identify areas for improvement and enhance the overall user experience of the NAVID e-learning system.
3. Methods
This study evaluated the usability of the NAVID national e-learning system from the perspectives of its users. The population consisted of 29 professors from the basic and clinical science departments of the faculty of medicine, 30 medical informatics graduate students, and 2 information technology experts from Tabriz University of Medical Sciences in the first semester of the 2022 - 2023 academic year. To ensure consistency, all participants were asked to complete their evaluations on the university's internet network using a desktop computer connected to the wireless network.
3.1. Validation of Checklists
Custom checklists were developed by reviewing the literature and an expert panel. For the assessment of face validity, the preliminary questionnaire was evaluated by eight medical education professionals. These participants assessed the difficulty, generality, and ambiguity of the items. The content validity of the questionnaire was evaluated using both qualitative and quantitative methods. Eight medical education professionals were asked to comment on the grammar, vocabulary, item placement, and scoring. The content validity ratio (CVR) and Content Validity Index (CVI) were calculated. The participants rated each item using a 3-point Likert scale (essential, useful but not essential, not essential). The CVI was assessed by the same ten experts for "simplicity", "relevance", and "clarity" using a 4-point Likert scale. Items with a CVR above 0.85 and a CVI above 0.70 were considered valid. The Cronbach’s alpha coefficient was calculated. After confirming validity and reliability, the checklists were finalized. The three checklists are presented in Appendix 1 in Supplementary File.
3.2. Usability Evaluation
Participants evaluated the NAVID system by performing specific tasks. Effectiveness (completeness) was measured by the degree to which goals were achieved, efficiency was assessed by the time taken to complete tasks, and satisfaction was gauged using a 7-point Likert scale.
3.3. Statistical Analysis
The effectiveness of tasks was calculated using the completion rate formula, which is defined as the number of completed tasks divided by the total number of tasks. The effectiveness was calculated by: Effectiveness = (Number of tasks completed successfully/total number of tasks undertaken) × 100. Efficiency was calculated using a time-based efficiency formula, considering the number of users, the success rate of each task, and the time taken to complete tasks. Time-based efficiency is calculated as:
Where N: The total number of tasks; R: The number of users; nij: The result of task i by user j; if the user successfully completes the task, then nij = 1, if not, then nij = 0; tij: The time spent by user j to complete task i. If the task is not successfully completed, then time was measured until the moment the user quits the task.
Satisfaction was assessed using a seven-point Likert scale, where participants indicated their level of satisfaction by selecting a response that ranged from "very dissatisfied" (assigned a score of 0) to "very satisfied" (assigned a score of 6). Then the overall satisfaction score was reported as a percentage.
Spearman's correlation analysis was performed using SPSS 16. A significance level of P < 0.05 was set.
4. Results
4.1. Checklist Validation
After validation, the checklists were finalized with a CVR of 0.86 and a CVI of 0.83. Reliability was confirmed with a Cronbach's alpha of 0.80. Out of 30 students, 20 were female and 10 were male. Among the 29 professors, there were 5 female and 24 male professors. In the expert group, there was 1 female and 1 male expert. The mean age of the participating students was 24.2 ± 2.7 years. For the professors, the mean age was 48.5 ± 8.4 years, while the experts had a mean age of 36.1 ± 5.6 years. Students were in their mean semester of 3.1 ± 1.4. The mean length of professional experience was 21.3 ± 5.9 years for the professors, and 9.1 ± 1.4 years for the experts.
4.2. Usability Evaluation by Professors
Professors reported an overall task completion rate of 87.5%. Tasks 9 (viewing the list of students of a course) and 11 (editing lesson information and curriculum plan) had the lowest effectiveness. The overall efficiency was 85%, with the lowest efficiency related to the same tasks due to difficulties in task navigation and icon placement. Satisfaction was also lower for these tasks, with an overall satisfaction score of 75.2% (Table 1). Suggestions for improvement included enhancing interaction with students, improving system performance during exams, and addressing technical issues like microphone quality.
| Task Number | List of System Tasks | Achieving The Desired Goal | Time-Based Efficiency | Satisfaction |
|---|---|---|---|---|
| 1 | Viewing the list of courses for the current semester | 29 (100) | 0.095 | 100 |
| 2 | Selecting one of the lessons and adding a new source from the computer | 29 (100) | 0.062 | 100 |
| 3 | Choosing one of the lessons and adding a source from Arman | 29 (100) | 0.063 | 100 |
| 4 | Creating a new assignment with a deadline of 10 days | 29 (100) | 0.047 | 60 |
| Creating a quiz for tomorrow at 10 a.m. | 29 (100) | 0.041 | 100 | |
| 5 | Creating a new forum for discussion and problem-solving with no time limit | 29 (100) | 0.050 | 100 |
| 6 | Creating a virtual class and its link | 29 (100) | 0.046 | 60 |
| 7 | Sending a message to all students with information about the final exam | 29 (100) | 0.052 | 100 |
| 8 | Viewing the list of students of a course | 0 (0) | 0.039 | 0 |
| 9 | Copying two resources from another lesson into the selected lesson | 29 (100) | 0.066 | 60 |
| 10 | Editing lesson information | 0 (0) | 0.035 | 0 |
| 11 | Viewing assignment answers and giving feedback | 29 (100) | 0.050 | 100 |
| 12 | Removing a resource from the selected lesson | 29 (100) | 0.055 | 100 |
| 13 | Logging in to the report page and seeing the activity of students in doing homework | 29 (100) | 0.087 | 100 |
| 14 | Viewing previously uploaded files in the repository | 29 (100) | 0.088 | 59 |
| 15 | Activating/deactivating the content of one of the lesson sessions | 29 (100) | 0.075 | 64 |
The Results of the Professors Regarding the Usability of the NAVID System a
For the professors, Spearman's correlation test showed a strong positive correlation between effectiveness and satisfaction scores (rs = 0.63, P = 0.008). However, there was no correlation between efficiency and satisfaction scores (rs = 0.31, P = 0.23). There was a moderate positive correlation between effectiveness and efficiency (rs = 0.57, P = 0.02).
4.3. Usability Evaluation by Students
Students reported an overall task completion rate of 83.3%. Effectiveness was lower for tasks related to password recovery and logging in through the University’s education management system (SAMA), primarily due to issues with phone number and email registration. Efficiency was also lower for these tasks, with an overall efficiency score of 85%. The overall satisfaction score was 81.1%, with lower satisfaction for tasks related to system support and password management (Table 2). Students suggested improvements such as better support communication, increased file upload limits, and system notifications for assignments.
| Task Number | List of System Tasks | Achieving the Desired Goal | Time-Based Efficiency | Satisfaction |
|---|---|---|---|---|
| 1 | Viewing the answer to one of the questions through the frequently asked questions section | 30 (100) | 0.15 | 100 |
| 2 | Viewing the support contact information of the site through the support section | 30 (100) | 0.12 | 4.5 |
| 3 | Recovering your password through the forgotten password link | 3 (10) | 0.021 | 34 |
| 4 | Entering your NAVID account by the login button through “SAMA” | 4 (13) | 0.035 | 25.5 |
| Searching for one of your current lessons, through the search button | 30 (100) | 0.11 | 100 | |
| 5 | Opening the guide to work with the NAVID system through the help button | 30 (100) | 0.11 | 100 |
| 6 | Changing your password in the profile section | 3 (10) | 0.046 | 34 |
| 7 | Receiving the student activity report in all courses in Excel, through the reports section | 30 (100) | 0.067 | 100 |
| 8 | Receiving the report of all course assignments in Excel, through the reports section | 30 (100) | 0.071 | 100 |
| 9 | Getting the report of grades in all course tests in Excel, through the reports section | 30 (100) | 0.070 | 100 |
| 10 | Receiving the report on all lesson discussions in Excel, through the reports section | 30 (100) | 0.071 | 100 |
| 11 | Opening one of the contents and confirming the reading of the resource | 30 (100) | 0.075 | 100 |
| 12 | Viewing one of the assignments of your courses and respond by sending text and files, through the assignments section | 30 (100) | 0.057 | 100 |
| 13 | Seeing the guide for participating in the exam, through the self-exams section | 30 (100) | 0.090 | 100 |
| 14 | Sending a question to your classmate, through the messages section | 30 (100) | 0.066 | 100 |
| 15 | Sending a thank you message to the course teacher, through the messages section | 30 (100) | 0.061 | 100 |
The Results of the Usability Assessment of the NAVID System by Students a
For the students, Spearman's correlation test showed a strong positive correlation between effectiveness and satisfaction scores (rs = 0.72, P = 0.002). However, there was no correlation between efficiency and satisfaction scores (rs = 0.33, P = 0.22). There was a strong positive correlation between effectiveness and efficiency (rs = 0.69, P = 0.004).
4.4. Usability Evaluation by Experts
Experts reported 100% task completion but noted that many tasks were time-consuming due to difficulties in accessing task menus and unclear icon text. The overall efficiency and satisfaction scores were 100% and 81.3%, respectively (Table 3). Suggestions included improving the user interface and providing up-to-date user guides.
| Task Number | List of System Tasks | Achieving the Desired Goal | Time-Based Efficiency | Satisfaction |
|---|---|---|---|---|
| 1 | Entering the system management page | 2 (100) | 0.10 | 100 |
| 2 | Viewing the system users (teacher and student) | 2 (100) | 0.05 | 50 |
| 3 | Accessing the new user creation form | 2 (100) | 0.20 | 100 |
| 4 | Creating a new user as a professor | 2 (100) | 0.02 | 100 |
| Creating a new degree | 2 (100) | 0.10 | 100 | |
| 5 | Viewing the defined semester | 2 (100) | 0.11 | 100 |
| 6 | Extracting the total number of students for the current semester | 2 (100) | 0.03 | 100 |
| 7 | Adding a new course to the system | 2 (100) | 0.02 | 100 |
| 8 | Choosing a course in the master's degree and extracting a report of the professor's activity in that course | 2 (100) | 0.01 | 25 |
| 9 | Extracting the students' activity in the above-selected course | 2 (100) | 0.01 | 50 |
| 10 | Extracting a comprehensive report on the status of a particular professor's courses | 2 (100) | 0.01 | 100 |
| 11 | Analyzing the status of submitted assignments in nursing school | 2 (100) | 0.02 | 50 |
| 12 | Correcting the “contact us” information in the system | 2 (100) | 0.05 | 100 |
| 13 | Adding a new teaching assistant to one of the selected courses of the rehabilitation faculty | 2 (100) | 0.01 | 100 |
| 14 | Adding a new field of study to the system and define a student for it | 2 (100) | 0.01 | 100 |
| 15 | Preparing the activity log of one of the courses from the first of the semester to the end of the year | 2 (100) | 0.01 | 25 |
The Results of the Experts Regarding the Usability of the NAVID System a
For the experts, Spearman's correlation test showed a strong positive correlation between effectiveness and satisfaction scores (rs = 0.66, P = 0.01). Additionally, there was a moderate positive correlation between efficiency and satisfaction scores (rs = 0.52, P = 0.05). There was a moderate positive correlation between effectiveness and efficiency (rs = 0.53, P = 0.05).
5. Discussion
The effectiveness of the NAVID system, as evidenced by the task completion rates, was generally high. This suggests that the system can support users in achieving their objectives. However, the lower effectiveness rates for specific tasks, particularly for students, indicate areas where the system's design does not fully support user needs. Tasks such as recovering forgotten passwords or logging in through SAMA presented significant challenges. Since accessing a user account is foundational to system use, the observed difficulties in these areas could have broader implications, potentially deterring consistent use and reducing overall system engagement. To address these issues, more robust integration between NAVID and SAMA is essential, alongside a more intuitive password recovery process that does not burden users with prerequisites they might not have fulfilled.
The findings of the study align with previous research on the usability of e-learning systems. A study found that while students generally view e-learning systems positively, there are significant issues with system features, including spontaneous and unwarranted log-outs (10).
Efficiency reflects how quickly and effortlessly users can complete tasks. While the system performed well for simple tasks, such as viewing the list of current courses, more complex tasks, like editing course information or presenting lesson plans, showed significant inefficiencies. These tasks were not only time-consuming but also required navigating through less intuitive parts of the system, which likely contributed to the lower satisfaction scores observed for these tasks. The inefficiencies in these areas can be linked to poor information architecture, where essential functions are buried under multiple layers, making them less accessible to users (11). This indicates a need for a redesign of the system's user interface, prioritizing ease of access to frequently used functions and reducing the cognitive load on users (12).
User satisfaction is perhaps the most subjective yet critical dimension of usability, as it encapsulates the overall user experience and willingness to continue using the system. The satisfaction levels, particularly among professors and students, were relatively low for specific tasks, such as managing course information and interacting with students during online sessions. These findings align with prior studies, which suggest that e-learning systems often fall short in providing seamless interaction and robust evaluation tools, especially in environments that heavily rely on offline methods (13). The dissatisfaction with the system's exam and evaluation capabilities, frequent disconnections, and limited interaction options during non-office hours point to significant areas needing improvement. Enhancing the system's reliability, particularly during high-stakes tasks like exams, and expanding support options to include real-time assistance through multiple channels, would likely increase overall user satisfaction (14).
In the present study, the reason mentioned for the low satisfaction with tasks 9 and 11 was the difficulty in finding the location and the inappropriate design of the icons. Ennam indicated that the lack of success in online learning is primarily due to inadequate web accessibility and affordability and insufficient training in distance education (15). For both students and professors, a strong positive correlation was found between effectiveness and satisfaction, and between effectiveness and efficiency, underscoring the importance of task success in shaping both their satisfaction and perception of system usability. However, no significant correlation was observed between efficiency and satisfaction, which may reflect their tolerance for slower system interactions if tasks are ultimately completed successfully. However, among experts, correlations were found between efficiency and satisfaction. These results indicate that for experts, who may have higher expectations and more system knowledge, task performance influences both perceived efficiency and satisfaction.
The correlation observed between effectiveness, efficiency, and satisfaction in this study underscores the interdependence of these usability dimensions, suggesting that when users can complete tasks effectively, their satisfaction with the system increases (16). Conversely, when tasks are inefficient or complex, effectiveness and satisfaction decline, as seen with tasks involving password recovery or system login. This interdependence suggests that improvements in one area, such as making key tasks more efficient, are likely to yield broader gains in overall system usability (17).
The Online Learning Consortium considers student satisfaction with online learning in higher education to be an essential element for measuring the quality of online courses (18). However, satisfaction is multidimensional, and different factors influence learner satisfaction, such as their digital literacy levels, social and professional engagements, perceived stress levels, and the course learning design (19).
Moreover, the frustration students expressed with limited support options during exams—especially when issues arise outside of office hours — suggests a need for more robust, round-the-clock support mechanisms (20). This could include automated solutions like AI-driven chat bots for common issues or an expanded support team available during peak usage times. A study by Kim et al. showed a strong correlation between student satisfaction, the acceptance of online learning, and the effectiveness of online support services for both American and Korean students (21).
Future research could focus on integrating more advanced technologies, such as AI-driven support, to further elevate the system’s usability. This study involved various groups of e-learning stakeholders, including students, professors, and administrators, enriching the research by providing multiple perspectives. Although the participants were students from diverse backgrounds, cultures, and cities attending one university in Iran, the validity and reliability of the model could be further improved by surveying participants from different universities across the country. Additionally, as technology and e-learning continue to evolve, conducting longitudinal research to explore how the usability factors identified in this study change over time could yield further interesting insights.
5.1. Conclusions
While the system demonstrates overall acceptable usability, particularly in task completion rates, the findings also reveal critical areas that require significant enhancement. The challenges faced by users, particularly in accessing and interacting with essential system functions such as password recovery and course management, indicate a need for a more user-friendly design. Furthermore, relationships were observed between different usability dimensions, suggesting that improvements in one aspect are likely to lead to overall improvements in system usability.
