Validating “Students’ Opinion Questionnaire” and “Student’s Evaluation of Educational Quality Questionnaire” in Relation to Teacher Evaluation Using Criterion Method

1. Background

Evaluation of teachers by students, which has been done in universities and higher educational centers, is one of the common evaluation methods designed to evaluate teaching quality and educational performance of teachers (1). The aim of this evaluation was to improve teaching quality and increase educational promotion in the universities (2). Therefore, the process of evaluating teachers, part of which is done by students, provides feedback that improves the teaching and learning system. In this regard, using a valid evaluation form with appropriate reliability and validity and that results in the identification of teaching and learning challenges is important. This process enables an academic system to reinforce its strengths and correct weakness, and provides a rich application for the personal growth and empowerment of the faculty members (3). In his study, Tazakori et al. (2), noted some deficiencies and shortcomings in teachers’ evaluation, such as the dissatisfaction rate with the validity and reliability of existing evaluation tools and students’ dissatisfaction with the evaluation forms in evaluating the teaching quality. Raoofi et al. (4), believed that the teachers’ evaluation by students is a valuable resource for improving teaching quality. Yamani et al. (3), showed that students’ evaluations are not valid and reliable and that they consisted only of the personal characteristics of the teacher. Besides providing teachers with feedback on students’ attitudes toward them, teachers’ evaluation by students also allows teachers to make decisions on the job (5). Therefore a valid and comprehensive evaluation system which represents the evaluation score of teachers’ educational performance, whose data has the necessary reliability and validity and is not affected by the students personal opinions, is essential (6). Results of some studies have also shown that teaching quality has a multi-dimensional structure. Therefore, tools designed to evaluate teaching effectiveness should show this multi-dimension (7). Researches have shown that each time these tools were used, students would distinguish the differences between the various teaching dimensions (5). In this regard we suggest using comprehensive and multi-dimensional questionnaires that have effective variables in educational process and are more general and generalizable. The student evaluation of educational quality questionnaire (SEEQ) was prepared in Sydney university and its differences have been proven as a useful tool for improving the universities teacher education in various research (7-9). Thirty one questions of this tool cover nine dimensions (factors) of efficient teaching: learning, being an interested teacher, organizing content, group interactions, teacher and student relationship, integrity of content, exams and grading, assignments (homework), and overall evaluation (3). Each dimension on this scale could affect efficient teaching and improve teaching quality. Also, the SEEQ answers in relation with the types of the criteria were validated. The SEEQ rating is a function of the teacher teaching the lesson and not of evaluating the lesson (7). The efficiency that exists in the teacher evaluation by the present methods in one hand, and the SEEQ property such as evaluating the various teaching dimension, confirming the validation of its structure in more than 30 research with analytical factor methods (7); on the other hand, controlling the probable bias in various study which couldn’t explain more than 15% of the variance rating in SEEQ of teaching quality (10) and also the remarkable validation coefficient of this questionnaire (88 - 97%) (8), make the use of this tool in SEEQ important. The medical sciences universities in the country use different evaluation forms that are different for the evaluated item and scoring (1-5). In Kermanshah university of medical sciences (KUMS) a student’s opinion form has been used to evaluate the teacher. This form includes components such as: personal characteristics, educational method and teacher’s scientific ability. The aim of this study was to determine the validity and reliability of the SEEQ and SOQ tools in teacher teaching evaluation in KUMS. The results of various research showed that the teaching quality is multi-dimensional but the teaching quality dimension in various studies is reported differently (7).

2. Objectives

This research tried to identify factors of efficient teaching quality performance by analyzing the exploratory factor, and by comparing the two questionnaires to determine which of the questionnaires was a suitable tool for evaluating teaching quality.

3. Methods

This was a cross-sectional study done in the second educational semester of 2011 - 2012. All students in the health faculty participated in the study (251 persons). The students who were in the internship semester were removed from the study. Data was collected using the Persian versions of the SEEQ and SOQ in relation with the teacher’s teaching evaluation of KUMS based on the presented course of each major in the mentioned semester. Two weeks before the final exam the project partners were present in the class and explained the importance of the research and explained to the students how to complete the questionnaires. In total, 762 evaluation forms were collected. For comparing the teaching dimension quality in evaluated groups, first the sources obtained from each SEEQ and SOQ were calculated separately for each faculty’s members of the departments; then the obtained score from each dimension for all the faculty’s members of the department were added and the average obtained was the quality criterion of that dimension after teaching on that department. To compare scores in the two tools, the scores was calculated by percentage and equalized. SEEQ contains 31 questions and 9 dimensions (factors) of efficient teaching: learning, being interested teacher, organizing content, group interactions, teacher–student relationship, integrity of content, exams and grading, assignment (homework), and overall evaluation (3).

The SEEQ was ranked on a 5-point Likert scale (very good = 5 to very weak = 1). The score for each dimension was obtained by adding its ballots score and eliminating the effect of the number of ballots. The score range for each dimension was 1 to 5. Taleapasand et al. (7), examined the psychometric properties of the translated version of SEEQ. In his study, 339 students from 5 faculties of Semnan university completed the SEEQ questionnaire for their professors. To validate this questionnaire, Cronbach’s alpha coefficient was used to calculate the internal coordination of each dimension of the tool. All dimensions, with the exception of the total dimension are rated as acceptable. The range of reliability coefficient of dimension is from 0.63 (overall assessment) to 0.86 (group engagement) and the predictive reliability was calculated by computing the total scores of the “teaching quality assessment questionnaire” with the academic achievement scores of the students in the relevant lesson and at the end of the semester. The results showed a significant correlation (γ = 0.50). The SOQ in relation with teacher’s teaching evaluation at KUMS had 16 questions and three dimensions: teacher’s personal characteristic, educational method and teacher’s scientific ability. Data were analyzed to determine the validity of the questionnaire using the Pearson and Spearman coefficient, and Cronbach’s alpha coefficient was used to estimate the internal consistency of the questionnaire the total component. To determine the numbers of the teaching quality’s basic factors in both questionnaires, the exploratory factor analysis with the principal component analysis (PCA) method with varimax rotation was conducted on all the samples data. The questionnaires were encoded to have confidential data which related to the teachers.

4. Results

In this research, students filled out 762 evaluation forms. 19.7% of the students were male and 77.3% female; 3% did not mention their gender. 7.9% of the forms were filled out by associated students, 86.2% of it by undergraduates students, and 5.9% by postgraduates. Also 131 evaluation forms (17.2%) were filled out by occupational health students, 198 (26%) by public health students, and 433 (56.8%) for by environmental health students. Cronbach’s alpha coefficient was used to calculate the reliability (internal consistency) of each SEEQ dimension of teaching quality, and the SOQ in relation with teaching evaluation of public health faculty’s member in KUMS (Tables 1 and 2).

The results of the exploratory factors analysis in relation with SEEQ questionnaire showed that Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy is excellent (KMO = 0.975). Also, finding showed questions 1 - 4 are for learning; 5, 6 and 8 are about the professor’s interest; 9 - 12 are for articles organizing; 21 - 23 are about the contents comprehensiveness; and question 31 the overall evaluation of factor 1 (the mentioned questions have high correlation that in total are one factor). Questions 25 - 27 relate to exams, 28 and 29 relate to assignment, and question 30 to overall evaluation: these questions formed factor 2. The factor 3 constitutes questions 13 - 16 relate to group interactions, and question 24 to contents comprehensiveness. Question 7 is from the professor’s interest and questions 17 - 20 relate to professor–student relationship aspect; these formed factor 4. Therefore, the factor structure of this questionnaire is a multi-dimensional tool with four factors which have the sufficient validity. The factor structure of this questionnaire are: (1) the professor’s interest to teach and organizing the contents, (2) exams and assignments, (3) group’s interaction, and (4) professor-students relationship.

The results of the exploratory factors analysis in relation with SOQ questionnaire showed that Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy is excellent (KMO = 0.953). Also, finding shwed in the SOQ questions 1, 2 and 6 are factor (2) and the other questions formed factor (1). Therefore the SOQ tool has two factors that have sufficient validity. Structure of this questionnaire include two parameters: factor (1) is the instruction method and professor’s scientific ability, factor (2) is the professor’s discipline.

5. Discussion

The results of this research showed that all dimensions in both questionnaires have acceptable reliability coefficient and both questionnaires have proper reliability. In Taleapasand et al. study (7) all the SEEQ dimensions (except the overall evaluation) have acceptable validity coefficient. For determining the number of the basic factor and evaluating the property of 31 questions in evaluating the teaching quality questionnaire and the common questionnaire, the exploratory factor analysis was conducted by the principal component method on all samples data. The results showed that the number of the basic factors of teaching quality in common questionnaire of KUMS decreased from three factors to two. According to the types of questions, the teacher’s discipline was named factor 1 and the teacher’s scientific ability and other factors were factor 2. While the factor’s analysis on the SEEQ in this research showed that it is a multi-dimensional tool with four factors, and the number of basic factors in teaching decreased from nine factors to four factors. The results of Taleapasand et al. study (7) that was carried out in Semnan on the validity of the Persian version of the questionare showed that teaching quality was the sixth factor. In Germain-Rutherford study (11) in Greece, the evaluation of SEEQ by 1264 students was studied. The results provided solid evidence of the applicability of the Greek version of SEEQ, by confirming the factor structure of the tool and reassuring the multidimensionality of the teaching effectiveness construct. In Lidice and Saglam study (2013), a total of 20 students were evaluated for SEEQ evaluation. The results indicated that efforts should be made to strengthen English language development (12). Although the results of this study in relation with SEEQ that has factor four, it is consistent with the ninth factor which Marsh clamied but it was in line with Marsh opinion about multi-structure’s teaching quality (7). Based on the Taleapasand (7) and Kulik (13) study, teaching quality is a multi-dimension structure. Therefore, tools designed to evaluate the efficency of teaching should show this multi-diminsion. When these designing tools were used the research showed that the students distinguished the differences between the various aspect (7). On the other hand, Taleapasand research cited to the studies that SEEQ questionnaire used for teaching evaluation of undergraduate, postgraduate and various types of the educational major and result showed role of the same factors in teaching quality. This study is an extensive generalization of SEEQ factors during a period of time (7).

5.1. Conclusions

The aim of this study was to select an approprate tool for evaluating efficent teaching. Because KUMS’s common questionaire evaluates only two factors of teaching quality, it appears that using the SEEQ questionaire whose teaching quality dimesion has four aspects is more efficient in showing the weaknesses and strengths of the teachers’ teaching quality. The most important and appropriate aim of the evaluation in the field of education and education system is the awareness of the present condition and the level of its distance from evaluated proper condition. Therefore it is important to choose tools that separate the teaching quality dimension better and help us in achieving the goal.