Smart University of Medical Sciences Virtual Summer Semester from the Perspective of Its Stakeholders: A Multi-methods Cross-sectional Study

1. Background

With the advent of information and communication technology (ICT), fundamental changes have occurred in higher education's processes, methods, and products. Many universities have adopted e-learning to deliver their courses and are attempting to increase the efficacy of emerging technologies in educational activities (1). In the fourth and fifth generations of distance education, e-learning has enabled lifelong learning in any subject, for anybody, at any time, and in any location. There is a growing interest in using virtual and online learning in medical education, beginning with the undergraduate years and extending through residency, fellowship, and continuing medical education (2). In the meantime, with the rise of the phenomena and crisis of COVID-19, an extraordinary and revolutionary change occurred in remote learning, resulting in a paradigm shift toward online learning as an alternative educational approach as the only feasible solution (3, 4).

Studies show that COVID-19 did not only lead to a paradigm shift towards e-learning, but this pandemic has brought good opportunities and achievements for medical education, such as improvements in attitudes to e-learning and adaptation, preventing students' separation from the educational environments, self-learning control, documentation, and monitoring training, increasing perceived usefulness (5).

In response to this global emergency, universities, medical schools, and other higher education institutions have been forced to close university campuses and switch to online learning to avoid the suspension of education. If they have a virtual education system, they should work to grow and enhance it (6).

Studies show that students prefer virtual education formats because of their flexibility and because they facilitate learning by providing opportunities to reflect on course material (7). In addition, studies indicate that e-learning, by increasing students' engagement (8) and self-directed learning opportunities (9), may enhance their learning.

In this regard, e-learning approaches in universities and medical institutions have spread in various ways, with the virtual summer semester course in Iran being one of its manifestations. Virtual summer courses offer opportunities to learn or continue education for students who cannot participate in face-to-face summer courses, or in general it is not possible to design and implement face-to-face summer semester courses as a result of crises such as COVID-19 (10).

Despite the fact that teachers and administrators frequently have a negative attitude toward summer courses and believe that these courses are less effective than regular semesters (11-13), research indicates that online summer semester courses are an effective strategy for creating educational equity and participation among underprivileged students, maintaining student graduation rates, and decreasing dropout rates (14).

In addition, Stephenson-Hunter et al. (15) demonstrated in a mixed-method evaluative study that the participation of undergraduate students in a summer virtual mentorship course during the COVID-19 pandemic significantly strengthened non-cognitive skills, particularly self-efficacy.

In general, the virtual summer semester course was designed and implemented as a large-scale national project for all medical science universities across Iran. While this course has certain advantages like other online courses, its success rate, usefulness, strengths, and weaknesses are particularly important from the point of view of its stakeholders, and the results of this research can be the basis for policy-making and decision-making regarding the implementation of its next courses. Therefore, in this research, the national virtual summer semester course was investigated using a multimethod approach and from the perspective of its numerous stakeholders.

2. Objectives

What are the strengths and weaknesses of the virtual summer semester from the point of view of teachers and heads of departments?

How can the virtual summer semester be improved?

To what extent has the virtual summer semester been desirable and appropriate from the perspective of students?

To what extent has the virtual summer semester been successful from the point of view of various stakeholders?

3. Methods

3.1. Setting and Course Description

Undergraduate medical education in Iran has two pre-clinical courses (including two stages of basic science and semiology or physiopathology) and a clinical course (including two phases of clinical clerkship and internship). The virtual summer semester course in the academic year of 2022 was designed and implemented for 125 lessons by 18 departments at Smart University of Medical Sciences for students of medical sciences universities across Iran.

3.2. Multiple Methods and Data Sources (Triangulation)

While mixed methods research refers to combining at least one quantitative and one qualitative method, the multi-methods research approach is not limited solely (and entirely) to combining quantitative and qualitative methods. Multi-methods research integrates several quantitative methods, several qualitative methods, or a combination of both methods (16) to collect richer data. Teddlie and Tashakkori (17) define the multiple-method approach as a sort of design in which more than one method or more than one worldview is utilized to collect data. Considering that the virtual summer semester course includes several stakeholders (such as students, teachers, heads of departments, managers, and an executive staff) with conflicting interests, the use of several methods and several data sources, in addition to helping to eliminate blind spots in the collected data also balance the interests of different stakeholders (16). Hence, this study employed a combination of approaches (including remote qualitative interviews, a mini literature review, a questionnaire-based survey, and a checklist-based survey) to collect.

3.2.1. Remote Qualitative Interviews (Web-based and Telephone)

To investigate of the virtual summer semester course and suggestions for its improvement from the point of view of teachers and heads of departments, remote qualitative interviews (web-based and telephone) were used. In this phase, the open-ended questions were uploaded to the Porsline, and after its final editing, its link was shared with the teachers and chairpersons of departments via email. In addition, a number of teachers and heads of departments were interviewed by phone. Twelve teachers and heads of departments participated in remote qualitative interviews using the intensity sampling method. The second qualified researcher conducted all interviews. In the telephone interviews, an initial question was asked to establish rapport with the participants, followed by four key questions (Appendix 1 in the Supplementary File). The average telephone interview time for each participant was 25 minutes. A matrix was created in Microsoft Word according to the qualitative interview questions (web and telephone), and the answers were categorized based on thematic review. Strategies such as prolonged engagement with data (three of the authors were directly engaged in the design and implementation of the course), member checking, coding, and categorization based on the matrix, devoting adequate time to data collection and analysis, audit trail, quality assessment of data by two medical education and e-learning specialists, were used for data trustworthiness

3.2.2. Mini Literature Review

In the second phase, a mini-review was conducted to review the literature related to evaluating online and e-learning courses. Persian and English articles from the end of 2019 to the end of 2022 were searched in Google Scholar and PubMed databases. The keywords “Desirability,” “Success,” “Attitude to e-learning”, “e-Learning”, “Online learning”, “Online course”, “Virtual course”, “Distance education”, “Summer course” and terms related to each of these keywords were searched. This phase aimed to develop the desirability and success indicators of online learning courses using questionnaires and checklists.

3.2.3. Survey 1 (Questionnaire)

Based on the second phase, a questionnaire with 8 questions about demographic characteristics and 35 Likert scale questions (very low/1 to very high/5) was designed. Face and content validity was used to determine the questionnaire's validity. Face validity was qualitatively conducted in a 6-member panel consisting of faculty members of Smart University of Medical Sciences with specialties (medical education, e-learning, educational technology, and instructional design). This way, the level of difficulty, the degree of disproportion, the ambiguity of the questions, or the presence of insufficiency in the meanings of the words were checked, and their opinions were applied in the form of minor changes in the questionnaire. The content validity was evaluated quantitatively based on the opinion of experts and by calculating two indices, CVR and CVI. To check these two indicators, the questionnaire was sent to 10 experts (medical education, e-learning, educational technology, and instructional design). The CVR results for 32 questions of this questionnaire (out of 35 questions of the initial questionnaire) were equal to or greater than Lawshe's table number (0.62), and as a result of this process, 3 questions were removed. Therefore, the final questionnaire was compiled with 32 questions (Appendix 2 in the Supplementary File). The results of CVI indicated that all questions except questions 5, 9, and 16 had a score higher than 0.79, and these three questions were revised. Finally, Cronbach's alpha index was used to estimate the internal consistency of this questionnaire, and its value was calculated and confirmed as 0.84 for the total instrument. This questionnaire was designed with the aim of investigating students' attitudes towards the desirability of the course. The desirability of questionnaire items was determined by summing the products of each option's frequency score divided by the total number of responders. For each question, a score between 1 and 2.33 was deemed unfavorable, a score between 2.34 and 3.66 was deemed relatively favorable, and a score between 3.67 and 5 was deemed favorable.

3.2.4. Survey 2 (Checklist)

Based on the second stage, a checklist was designed, including 6 demographic questions and 90 questions with three options (relative success, complete success, and unrelated). Face and content validity were employed to determine its validity. Qualitative face validity was also performed for this checklist, and the necessary changes were applied based on the opinions of the experts present in the panel. Content validity was evaluated quantitatively, based on the opinion of experts (10 people), and by calculating the CVR and CVI indexes for this checklist as well. The CVR results for 82 questions of this checklist (out of 90 questions of the initial checklist) were equal to or greater than Lawshe's table number (0.62), and as a result of this process, 8 questions were removed. Therefore, the final checklist was developed with 82 questions (Appendix 3 in the Supplementary File). The results of CVI indicated that all questions had a score higher than 0.79. Finally, Cronbach's alpha index was used to estimate the internal consistency of this checklist, and its value was calculated and confirmed as 0.82 for the total instrument. The purpose of this checklist was to evaluate the degree of success of the course from the perspectives of various stakeholders (students, teachers, heads of departments, managers, and executive staff of the course).

3.3. Participants, Samples, and Sample Size

In order to select the sample and participants, three types of sampling methods were used.

Table 1 shows the participants/samples and the sampling methods.

In order to collect qualitative data from teachers and heads of departments, purposeful sampling and the method with intensity sampling were used. In fact, in this method, people who showed the phenomenon to a large extent were selected, and their participation rate was at the highest level because of the information they had. In the questionnaire-based survey phase, a multi-stage cluster sampling method was used to select students. In other words, the total population of students participating in the course was considered in terms of departments (clusters), and groups with the largest population were selected; then, among the students who had the most logins in the LMS system in these departments, the study sample was selected. Finally, in the phase of a checklist-based survey, in order to select the sample from among different stakeholders (such as students, teachers, heads of Departments, managers, and executive staff of the course), a relatively stratified sampling method was used. In other words, the sample was selected according to the size of the population of these groups.

Table 2 shows the number of participants and samples separately for each group.

As shown in Table 2, data was collected quantitatively and qualitatively from a total of 194 individuals. Sampling in the qualitative phase continued until the theoretical saturation of the data. In the quantitative phase (surveys 1 and 2), in order to estimate the sample size, the study of Motte-Signoret et al. (18) was used as the basis of the work. According to the study by Motte-Signoret et al. (18), only 41% of students and 30% of teachers believed that the e-learning course was successful and that they received or provided virtual training with the level and quality as expected. Furthermore, only one-third of students and teachers have indicated that online education will continue after the end of the COVID-19 crisis. In general, based on this study, the P ratio (0.2) was calculated. By placing this ratio in the sample size formula for prevalence and cross-sectional studies (19, 20), the sample size of 246 people was estimated (d = 0.05, z = 1.96, P = 0.2, q = 0.8).

In total, 82 students participated in (survey 1), 74 students in (survey 2), 18 teachers and heads of departments in (survey 2), and 8 members of the executive staff in (survey 2). Therefore, the response rate was about 74%.

3.4. Data Analysis

The data of this study were analyzed in two quantitative and qualitative levels. The thematic review method was used to analyze the qualitative data. Quantitative data analysis was also performed at two descriptive and inferential levels. To use parametric tests (one-sample T and ANOVA), the assumption of normality of the data and homogeneity of variances of the variables were checked and confirmed.

4. Results

Through multi-methods, study (remote qualitative interview, mini literature review, questionnaire-based survey, and checklist-based survey) and data source (students, teachers, heads of departments, managers, and executive staff of the course), the attitudes and viewpoints of different stakeholders about the virtual summer semester were presented in the form of quantitative and qualitative results.

4.1. Descriptive of Participants/Sample

Table 3 presents the demographic characteristics of the study participants separately for an interview, questionnaire-based survey, and checklist. As can be seen based on the collected data, two of the web-based interview participants did not declare demographic characteristics.

4.2. Qualitative Findings

Using web and telephone interviews, the virtual summer semester course was investigated from the perspective of teachers and heads of departments. Based on the thematic review of remote qualitative interviews and the created matrix, strengths, weaknesses, and suggestions for improving this course were categorized (see Table 4).

Based on the data analysis of remote qualitative interviews, the responses of teachers and department heads were presented in three categories: the course's strengths, its weaknesses, and improvement ideas.

4.3. Quantitative Findings

Based on the data collected from the questionnaire-based survey, students' attitudes toward the course were placed on a three-spectrum continuum (unfavorable, relatively favorable, and favorable). Table 5 shows an example of questions with descriptive statistics.

In general, based on the results of the questionnaire-based survey, approximately 91% of the students have a positive attitude towards this course. The students evaluated the virtual summer semester course as favorable in terms of the comprehensiveness of the course content. In addition, most students agreed with the importance of continuing the course and occasionally changing the schedule for subsequent courses. In order to study the statistical significance level of the course's favorability from the student's point of view, the one-sample T-test was used. The results of the one-sample T-test indicate that the sample mean is greater than the population mean (P = 0.048).

Another objective of this research was to investigate the course's success based on the checklist tool from the perspective of different stakeholders. In this regard, the descriptive findings showed that all stakeholders had evaluated the course as successful in terms of the organizational capacity of Smart University of Medical Sciences, pedagogy and educational effectiveness, and human resources engaged in the course. Thus, the ANOVA test was used to compare the averages of these three components.

The statistics presented in Table 6 show that the average score of organizational capacity of Smart University of Medical Sciences (P = 0.004), pedagogy and educational effectiveness (P = 0.035), and human resources engaged in the course (P = 0.043) is different based on multiple stakeholders.

5. Discussion

The virtual summer semester course for the academic year of 2022 was designed and implemented for 125 lessons by 18 departments at Smart University of Medical Sciences for students of medical sciences universities across Iran. This evaluative study evaluated this course with a multimethod approach and from the perspective of its various stakeholders. In order to evaluate this course from the point of view of teachers, heads of departments, students, and executive staff, three tools, including remote qualitative interviews and a Likert scale questionnaire and checklist, were developed.

Based on the thematic analysis of the responses of the teachers and heads of departments to the remote qualitative interviews (web-based and telephone), the features of the virtual summer semester were categorized into three themes: strengths, weaknesses, and suggestions for improvement.

In addition, based on the data collected from the questionnaire-based survey, the status of the virtual summer semester from the student's point of view was scored on a continuum (unfavorable, relatively favorable, and favorable). Based on the analysis of the collected data, approximately 91% of the students participating in the research have a positive attitude toward this course. In this regard, the average score of the students in all the questions related to the favorability of the course was higher than the hypothetical average (P = 0.048).

Finally, based on the data collected from the checklist tool, all the stakeholders evaluated the virtual summer semester course favorably in terms of the organizational capacity of the Smart Medical Sciences University (allocation of sufficient materials and resources, commitment and support to provide a quality course, etc.), pedagogy and educational effectiveness (clear design of the course, creating diverse learning opportunities, constructive alignment between goals, teaching-learning methods, assessment, etc.) and sufficient and capable human resources (teachers, managers and executive staff). In this regard, the average score of the course's success components differed according to the perspective of different stakeholders for the organizational capacity (P = 0.004), for the pedagogy and educational effectiveness (P = 0.035), and sufficient and capable human resources components (P = 0.043).

In general, the COVID-19 pandemic has accelerated the implementation of e-learning, and e-learning in education and higher education, including universities and schools of medical sciences, has become a double necessity in this era. These trainings, which are typically delivered in a blended format, can be utilized by students in clinical, basic sciences, and post-graduate courses and provide both theoretical and clinical training. They offer several benefits, including the ability for students to access the training at the desired time, lower costs, up-to-date educational materials, flexible learning, more freedom for students, and the ability to keep their jobs and earn money (21).

Although the virtual summer semester was introduced as one kind of virtual education before the coronavirus pandemic, it was created and implemented with increased emphasis and significance in this era because of the global paradigm shift toward virtual education and e-learning. Yet, the relevance of evaluating this course is no less than its design and implementation. Hence, the focus of this study was on evaluating this course. Evaluating various training courses is a prerequisite for improving and enhancing their quality and should be considered using scientific and systematic methods (22).

Concerning the qualitative findings of this study, some of the emergent results support the findings in the literature. For example, participants in this study cited flexibility as one of the course's strengths, which is consistent with the findings of Zheng et al. (23), Amir et al. (24), and Dost et al. (25). The participants also mentioned educational equity and compensating students' academic backwardness as one of the course's strengths, which aligns with the findings of Bentley's study (14). Similarly, in our study, participants identified the lack of teacher-student interaction as one of the virtual summer semester course's disadvantages, which is consistent with the findings of studies by Wang et al. (26), Zheng et al. (23), Abbasi et al. (27) and Amir et al. (24).

In summary, even though the qualitative findings indicated shortcomings in the virtual summer semester course, the results of the questionnaire-based survey among various stakeholders reflect the course's success. Meanwhile, the findings from the checklist-based survey among students (as the main stakeholders) show that a very high percentage of them expressed satisfaction with this course and evaluated it from relatively favorable to favorable, which is consistent with some studies in the literature. For example, Zheng et al. (23) investigated students' perceived effectiveness of online courses during the summer quarter of 2020. Also, in this study, the scores of students exposed to the online course during the summer quarter of 2020 were compared with the control group who had taken the same course face-to-face during the summer quarter of 2019 before the COVID-19 pandemic. Their study showed that students generally had a positive attitude about online learning during the pandemic, and most wanted to continue learning online after the pandemic. In addition, according to the findings of this study, the online group received as many or more a grades during the summer quarter of 2020 compared to the identical face-to-face group during the summer quarter of 2019. Meanwhile, Fischer et al. (10) concluded that students' performance in online courses was slightly lower compared to face-to-face courses, which is contrary to the results of our research.

Furthermore, similar to our findings, Goppert & Pfost's (28) research revealed that students experienced less worry and more enjoyment during online summer semester courses. Much research has been conducted in the Corona era regarding the satisfaction, attitude, and effectiveness of online courses (excluding summer semester virtual courses). The results of our research are consistent with those of AlQhtani et al. (29), Sadid-Zadeh et al. (30), Rad et al. (31), Agarwal & Kaushik (32), Khalil et al. (33), and partially aligns with the study of Hameed et al. (34).

This study has implications for practice and policy based on findings such as a positive attitude toward the course, satisfaction with the course, and the success and effectiveness of the course obtained from the qualitative interview, questionnaire, and checklist. Among them, policymakers and planners can expand virtual education and e-learning to long-term courses and develop regular and annual term courses in blended learning. Secondly, based on the implementation of this course in a wide geographic distribution (throughout the country) and its effectiveness, it can be stated that these short-term and intensive e-learning courses can accommodate large groups of students with diverse individual and cultural backgrounds. Hence, policymakers can be somewhat relieved in this regard. Based on this experience and the lessons gained, they can extend development-focused short-term e-learning courses at the level of continuous education. Finally, according to the results of this study, to sustainably improve the virtual summer semester course, policymakers can integrate it with a blended learning approach into the undergraduate curriculum.

There were some limitations in our research. First, because the geographical distribution of the participants and the study sample was so extensive and spanned the entire country, we did not have direct access to them; thus, the response rate in some universities was slow and poor. Second, our study's sample and participants came from different groups of stakeholders, making data collecting challenging and time-consuming. The other limitation is that two data components were collected via a survey (questionnaire and checklist) and self-report. Hence, it is likely that participants selected the answer that was most advantageous to them, compromising the data's reliability. In conclusion, we recommend that future studies examine students' performance, academic success, and grade comparisons in virtual summer semester courses to assess their usefulness. In addition, longitudinal studies with a concentration on one area of the educational course, such as the quality of instruction, assessment, and the learning of students etc., can provide the basis for future research.

5.1. Conclusions

In conclusion, although the results of this study revealed weaknesses associated with the virtual summer semester course, the findings indicated that students have a positive attitude toward this course, and various stakeholders have deemed the implementation of this course a success. Thus, based on the results and the conditions of the COVID-19 pandemic and its impact on medical education, such short-term and intensive, structured e-learning and virtual courses can be expanded and developed into regular semesters, and the course's sustainability can be ensured through continuous improvement.