Evaluating Educational Performance of Postgraduate Students Based on the Tennessee Academic Audit Model

1. Background

A country’s economic and social life depends on scientific growth, especially in higher education, which is one of the driving forces of societies today. On the other hand, the increasing demand in societies for higher and specialized education in recent decades has led to the foundation of an increasing number of universities and, subsequently, a higher number of students worldwide (1). Meanwhile, the recent focus on the accountability of universities has made them pay more attention to enhancing the quality of education (2, 3). The use of proper evaluation in educational institutions is one of the factors affecting the achievement of quality and promotion of performance (4). Accordingly, the need to design a framework and an instrument for evaluating performance is very important (5). Different approaches and educational evaluation models have been developed and implemented in higher education (6).

The evidence on this subject suggests that continuous improvement in the quality of higher education requires the use of coherent and systematic approaches and programs, such as auditing, to focus all personnel on quality-related topics (7, 8) since auditing is a valid and reliable approach to monitor and evaluate the quality of different services and care (9). The audit is a systematic and reliable approach based on credible and evidence-based standards. It ensures the organizations that the quality of services is based on the latest and most authoritative knowledge and is continuously upgraded (10). Similarly, the academic audit is a process by which the quality and value of programs, processes, and outcomes of an educational institution are carefully evaluated by assessing the extent to which the goals have been achieved (4). It also addresses the accountability of educational institutions and the quality of planning, performance, and outputs (5).

The application of audit and its significant effects are not limited to health, services, and industry. A glimpse at scientific literature demonstrates increased audit use in education, especially in medical education. For example, an educational audit has proved its effectiveness in creating professional attitudes and practices in medical students, and subsequently improving the quality and safety of the services they provide in their professional lives (11), and highlighted the need for an ongoing professional development strategy for medical professionals (12). Although audit has significant potential for expanding the quality and performance of an educational institution’s activities by taking the best measures in delivering services and improving organizational outcomes, many areas need more attention to realize the potential and minimization of resource waste (8, 13). This is because the design and implementation phases require teamwork skills, problem identification, process review, appropriate data collection, problem-solving, changing management skills, and appropriate training for quality improvement (14).

Evidence suggests that academic audits are available at various levels, such as university, college or institute, academic department, curriculum, or training course, and various models have been developed for them. Although most of these models have been developed at the college level, academic assessments and audits are often carried out at the academic department level in Iran (5). The prevailing mechanism of academic assessment in Iran, especially in medical universities, is called the internal assessment of academic departments. The standards of this method are presented in eight axes: mission, organizational goals, and status, educational programs, faculty, student, learning strategies, teaching facilities and equipment, dissertations, and graduates (15-17). Although this evaluation mechanism is a relatively new strategy to improve the country’s educational system quality, it was unsuccessful in achieving its goals. Moreover, its major shortcomings led to the creation of a new mechanism called the accreditation of educational systems. The main shortcomings in the internal evaluation are the weaknesses in its evaluation standards, the non-scientific process defined for its implementation, and the inadequate knowledge and experience of the evaluators. The main weakness of this mechanism is the focus of its evaluation standards on learning inputs and negligence in process-outcome standards (18).

Since the audits carried out in Iranian medical universities are focused on the academic departments, and due to the existence of remarkable deficiencies in their evaluation instruments, many important functional aspects, especially in the process and outcome dimension, are not addressed. Therefore, the use of an effective evaluation model to conduct an educational audit and, subsequently, to improve the quality of education in Iran is essential because many countries and educational institutions in this field have not developed an audit model, and they merely localized the existing models and adapted them to their needs (6).

2. Objectives

The present study aimed to conduct an educational audit of the Tabriz Faculty of Management and Medical Informatics using the model developed by the Tennessee Higher Education Commission. This study can improve the students’ abilities to provide quality services and subsequently improve the level of health and community satisfaction.

3. Methods

3.1. Participants

This cross-sectional study was conducted in the Faculty of Management and Medical Informatics of a Medical School (located in northwest Iran) in 2019. The study comprised two phases: Self-assessment and external evaluation. In the self-assessment phase, all faculty members with at least one year of experience were enrolled in the study, 21 (out of 24) of whom answered the questionnaire. Due to the low number of faculty members, data was collected by a census method, and all eligible faculty members were included in the study. The external evaluation phase was conducted with the participation of four departmental managers, including healthcare management, health economics, health information technology, medical librarians, and the dean and faculty of the academic department. The experts in localizing the instruments included the managers of the medical education development center, the medical education research center, the vice chancellor for education, the education deputy of other faculties, and the medical school faculty members. Having at least five years of experience in these positions was the study’s inclusion criterion.

3.2. Instruments

The researchers translated and localized the checklist developed by Tennessee Higher Education Commission, which is a valid and internationally recognized model. The model consists of 14 dimensions, including educational objectives (four questions), curriculum and co-curriculum (three questions), teaching and learning processes (seven questions), student learning assessment (four questions), quality assurance (five questions), research outcomes (two questions), research environment (five questions), synergy with education (five questions), supported programs (two questions), quality and productivity indicators (three questions), contributions to the program, departmental and university goals (three questions), overall assessment (four questions), support (five questions), and follow-up of previous academic audits (three questions). In addition, 52 standards and dozens of guidance points have been defined to measure the dimensions objectively (19).

To determine the validity and localization of the instrument in terms of adapting its dimensions to the educational system in Iran, the standards of this instrument were examined by 15 experts based on the criteria of necessity, importance, transparency, simplicity, and measurement capability. According to the statistical principles, the mean of the necessity index (content validity ratio (CVR)) was examined, and if the question was confirmed in this index, the score of the other four factors (content validity index (CVI)) was examined. Regarding the reviews of 15 experts, obtaining a minimum score of 65% was considered valid (20, 21). In this study, the instrument was approved with 87% CVR and 91% CVI. Face validity was assessed by obtaining and applying the expert’s qualitative viewpoints regarding the writing style and the language used in the questionnaire. The tool’s reliability was confirmed by examining its internal consistency and obtaining a score of 0.89 for Cronbach’s alpha. To answer the questionnaire questions, a three-point Likert scale was defined (complete compliance, partial compliance, and non-compliance).

3.3. Data Collection and Analysis

The self-assessment phase was done by physically distributing the questionnaires among the faculty members and returning the completed form. The external evaluation was conducted by interviewing the faculty executives, reviewing their documentation, observing the activities, and interviewing the students (with the participation of seven experts as the external evaluation team). In the analytical section, data were reviewed descriptively, and the results were reported as frequency (percentage) for qualitative variables and mean ± standard deviation (SD) for quantitative variables. Regarding the ten scales of this instrument, the final score of the performance was between 0 (worst performance) and 10 (best performance). Analysis of variance (ANOVA) and Tukey’s post hoc tests were used to assess the significance of the difference between the academic performance of departments. t-test was used to evaluate the difference between the performance scores in the two self-assessment and external evaluation phases. The Statistical Package for the Social Sciences (SPSS) software version 19 was used, and a P-value < 0.05 was considered significant in all analyses.

3.4. Ethical Considerations

To observe ethical principles, the researchers committed themselves to observe principles such as the freedom of the participants to accept or decline to participate in the study, obtaining informed consent from them, respecting the privacy and character of the participants, paying attention to the anonymity of the participants, and assuring them of the fact that the results of the study are in line with the goals defined. Also, the Ethics Committee of Tabriz University of Medical Sciences, Tabriz, Iran, granted permission to conduct this research.

4. Results

4.1. Internal Evaluation (Self-assessment) Results

According to our results, among the four dimensions of the Tennessee Academic Audit Model for assessing the status of the higher education system at the postgraduate level, the Faculty of Management and Medical Informatics scored 5.94 in the dimension of “comprehensive valuation” and 4.31 in the dimension of “follow-up of previous audits,” indicating the best and worst performance scores, respectively. The total score gained in the self-assessment phase was 5.32 (Table 1 and Figure 1).

Comparison of the performance of the Faculty of Management and Medical Informatics in different dimensions of educational audit based on self-assessment results

Figure 1.

Comparison of the performance of the Faculty of Management and Medical Informatics in different dimensions of educational audit based on self-assessment results

Comparing the self-assessment results of the academic departments of the Faculty of Management and Medical Informatics regarding the observance of auditing standards demonstrated that the Department of health economics (score: 6.34) and the Department of Management and Medical Informatics (score: 4.12) obtained the highest and lowest scores, respectively. The overall performance of the academic departments did not show a significant difference (P > 0.05). There was a significant difference between the academic departments only in the dimension of “cooperation in achieving goals of programs, college, and university” (P > 0.05) (Table 2).

According to the results of post hoc tests to check the significance of the difference between academic departments in the dimension of “cooperation in achieving goals of programs, college, and university,” we witnessed that the score of the Department of health economics was significantly higher than the score of the department of healthcare services (P < 0.05) (Table 3).

We also stratified the functional status of the Faculty of Management and Medical Informatics in education. Based on the self-assessment scores, from the viewpoint of faculty members, the highest frequency was assigned to moderate, good, and weak levels, respectively (Figure 2). However, in the external evaluation, the Faculty of Management and Medical Informatics and all its departments gained a weak score due to the score of 20-40% in their educational performance at the postgraduate level.

The overall performance of the Faculty of Management and Medical Informatics in education from the perspective of faculty members based on self-assessment (%)

Figure 2.

The overall performance of the Faculty of Management and Medical Informatics in education from the perspective of faculty members based on self-assessment (%)

4.2. External Evaluation Results

The results obtained during the external evaluation phase indicated that the dimensions of “collaborating in achieving the goals of the programs, college, and university” and “follow-ups of previous audits” had the best (3.83) and the worst (1.08) performance scores, respectively. The total score of the Faculty of Management and Medical Informatics was also estimated as 2.75. Also, comparing the results obtained in the self-assessment and external evaluation phases showed that in all dimensions of the audit and, in general, the scores obtained in the external evaluation phase were significantly lower than those obtained in the self-assessment phase (P < 0.05) (Table 4 and Figure 3).

Comparison of the performance of the Faculty of Management and Medical Informatics in different dimensions of academic audit based on external evaluation results

Figure 3.

Comparison of the performance of the Faculty of Management and Medical Informatics in different dimensions of academic audit based on external evaluation results

A comparison of external evaluation results in academic departments showed that the Department of Health Services Management (score: 3.22) and the health information technology department (score: 2.40) had the highest and lowest scores, respectively. However, due to the allocation of only one score to the academic departments in the external evaluation and the lack of multiple scores (and consequently the possibility of comparing the differences in the means), it was not possible to examine the significance of the difference in scores obtained using statistical tests (Table 5).

5. Discussion

The performance of the School of Management and Medical Informatics at postgraduate level in the self-assessment phase was moderate (score: 5.32), but in the external evaluation phase, it was weak (score: 2.75). Similar to our results, the results obtained by Imani and Gharibi (22) revealed that the performance score at undergraduate level in self-assessment and external evaluation was moderate (4.98) and weak (2.80), respectively. Yarmohammadian and Kalbasi evaluated the academic departments of the Faculty of Management and Medical Informatics of Isfahan University of Medical Sciences (17). They reported that the overall status and all dimensions were relatively favorable. The only exception in this regard was related to the dimension of “students,” which was unsatisfactory (17). In the study by Abedini et al. conducted at the Faculty of Nursing of Qom University of Medical Sciences (15) and the study by Parsa Yekta et al. conducted at the Faculty of Nursing of Tehran University of Medical Sciences (23), the status of the study departments was desirable.

Ghane Pour et al. evaluated the Public Health Department of the Faculty of the Health of Semnan University of Medical Sciences (24). They reported the results as satisfactory (obtaining 62% of the total score). The faculty had a high level of educational programs; in the dimensions of mission and goals of education, faculty members, and assessment, it was favorable; had a more than enough satisfying level in the dimensions of management and organization; had a satisfying level in students dimension; in the dimensions of educational resources and research activities were at the border level; and finally, at the graduate level was unsatisfactory (24). In a similar study, Farzianpour et al. evaluated the Department of Epidemiology and Biostatistics at Tehran University of Medical Sciences (25). The means score for all dimensions was 59.8%, where the dimensions of “goals and mission” and “graduates” had the worst (37%) and best (72.6%) performance scores, respectively (25). In another study, Zarrabian et al. evaluated the endodontic department of the Faculty of Dentistry of Tehran University of Medical Sciences; this department’ mean total score was 73.7%, and all nine study dimensions had a relatively favorable situation (26). Also, the status of the department was declared to be at a satisfactory level in the dimensions of the “faculty members” and “learners” and was undesirable in the dimensions of “educational spaces” and “educational equipment” (26).

The low scores of the external evaluation phase in the present study compared to the previously mentioned studies can be attributed to the bias and negligence of the academic departments in assessing their performance due to their preferred interests and lack of establishment of quality higher education in Iran. The lower relative scores obtained in the self-assessment phase of the present study compared to the above studies (which are entirely self-assessing) can also be found in the standards of the Tennessee model, which are more stringent than the current model of internal evaluation of medical universities in Iran (22).

Based on our results, the researchers present the following suggestions to improve the academic performance of the Faculty of Management and Medical Informatics: defining a specific process for educational needs assessment and objective setting by stakeholders’ participation; revising the educational curriculum based on upgraded objectives; optimal and purposeful use of active teaching methods; developing a written plan for encouraging students’ socialization in accordance with professional and educational principles; developing key indicators to assess the achievement of educational objectives; having a written plan for effective use of the results of educational evaluations; enhancing the commitment to continually improve the quality of training programs; creating a commitment to provide top research activities and outcomes compared to similar colleges; enhancing the synergy of education and research; developing a plan to ensure that the outcomes of the research match the needs of the departments, college, and university; providing and allocating adequate funds to meet educational needs; and designing an effective framework to purposeful use of educational audits results. These suggestions have been emphasized in many other similar studies (22, 27-29).

The main strength of the present study is that it used an educational audit process based on a valid international model for the first time in Iran. We also performed an external evaluation for the first time at the level of the medical sciences universities in Iran. One of the limitations of this study was the unfamiliarity of educational managers and faculty members with the Tennessee Educational Audit Model, which was solved by training them. Another limitation was the lack of similar studies to compare and discuss the results.

5.1. Conclusions

The present study demonstrated that according to international standards and measures, the Tabriz Faculty of Management and Medical Informatics of a Medical School had major deficiencies. Also, there were major challenges in improving the quality and educational performance of this faculty, which may be due to the following reasons: The lack of systemic activities to improve the quality and educational performance, severe weakness in the documentation of activities related to educational audits, poor adaptation between educational objectives, educational curriculum, teaching methods, and educational assessment, non-purposeful use of evaluation results, and paying little attention to objective evidence, best actions, and stakeholders’ participation in the activities. The researchers hope that the results of this study, in particular the depiction of the actual performance of the faculty and the applied promotional suggestions provided by them, could guide the managers and instructors to improve the quality of education.