Educational process (EP) refers to all learning and teaching activities that increase the quality and outcomes of education. Annually, EPs performed by the faculty members of the medical sciences universities are first assessed at the university, and then at the Deputy for Education of Ministry of Health and Medical Education of Iran on the basis of the Glassick’s criteria. Based on these assessments, superior EPs are selected at two levels of university and country in six domains.
This year (2018), at Kermanshah University of Medical Sciences (KUMS), 10 EPs registered in the relevant system for participation in the university festival. Each EP was sent to two referees and the comments of the referees were notified to the process owner (PO). After correcting them by PO each 10 EPs were scored (range: 0 - 100) in an evaluation session, with standard forms by eight referees. The mean of all referees’ scores (from 32.8 to 61.4) for each EP (from 30.25 to 71.14) was determined (Table 1). Three EPs did not find complete conditions for scoring, and two EPs did not get the required score (30.25 and 31.88), and eventually 5 EPs (42.5 - 71.14) were selected for 5 domains.
Table 1.
The scores of 8 Referees in the Evaluation of Educational Processes (0-100 Scores) at KUMS in 2018
| No. of referee | Scores Range | Mean ± SD |
|---|---|---|
| 1 | 0 - 59 | 32.8 ± 26.3 |
| 2 | 20 - 63 | 37.3 ± 14.5 |
| 3 | 33 - 72 | 45.8 ± 14.4 |
| 4 | 14 - 100 | 52.4 ± 31.3 |
| 5 | 37 - 83 | 56.9 ± 14.3 |
| 6 | 41 - 85 | 59.0 ± 17.6 |
| 7 | 38 - 93 | 60.6 ± 21.8 |
| 8 | 30 - 96 | 61.4 ± 24.8 |
| Total | 0 - 100 | 50.78 ± 20.63 |
Among the referees, there was a combination of different ages, experiences, genders, and etc. It seems that the above variable 'being the PO' and the domain of the EPs play a role in the scoring by the referees. In order to avoid the bias of the POs' referees, they did not have the right to vote at the time of scoring their own process and they left the evaluation session. The scores of all referees for all EPs (7 items) were entered into the SPSS-16 software and analyzed using independent t-test, Friedman test, Pearson and Spearman correlation coefficient.
Friedman test showed a significant difference between the referees’ scores (P = 0.016). Experienced referees compared to less experienced ones gave lower scores to the EPs. Independent t-test did not show any significant difference between the scores of both referees gender (P = 0.593) and with and without EPs (P = 0.055). However, the scores of referees with EP (65.1 ± 8.5) were more than referees without EP (41.5 ± 14.4). Correlation coefficient of the scores of 5 referees with mean total score of referees (0.827 - 0.936) was significant but for the scores of 3 other referees (0.522 - 0.630) was not significant. Also, the scores of two referees did not show any significant relationship with any of the other referees and even with themselves. This means that the opinions of the referees did not have the necessary coordination.
Finally, the highest scores were assigned to the processes conducted in the “compiling and revising educational programs”, “contributing to the design and production of educational products” and “Management and educational leadership” domains, and the lower scores were given to the “teaching and learning” and “evaluation” domains. There was no EP in the “e-learning” domain.
It is recommended that in the future evaluations, if possible, the PO’ referees do not participate in the EPs evaluation, so that there is no doubt about the selection bias. It is also suggested that to perform the final selection of the EPs after analyzing the referees’ scores, so that the referee score or referees who are likely to have a bias or a low correlation with the total of scores can be eliminated, or even the scores of this referees can be weighted less. Therefore, the final selection of processes is fairer. I would like to acknowledge Dr. Vida Sepahi, Khansa Rezaei, all referees and POs for participation in this study.
