Preparing Medical Students to Become Effective Tutors: A Reaction, Learning and Behavior Evaluation Study

1. Background

Problem-based learning (PBL) is a case-based, cooperative teaching and learning method using tutors, and is highly recommended for learning facilitation (1, 2). Whether faculty members or students assume the role of tutors in PBL, one of the most important factors that influence the quality of this strategy would be tutor and tutoring. This important matter and its major effects on learning facilitation are generally neglected by researchers, especially in Iran (1, 3, 4).

In order to effectively guide the discussion, tutors should be well-equipped with content expertise, facilitation skills and active learning stimulating abilities (4, 5). Despite the importance of this issue and its major role in promoting the quality of PBL, little attention has been paid to preparing students to take on the role of tutors in PBL sessions. Moreover, few studies have addressed the differences in tutors’ performance based on gender or education level in our context. Furthermore, the focus of studies, most often, has been on comparing the tutors’ performance based on mastering the content (5, 6).

A study by Groves et al. at Queensland School of Medicine in Australia suggested that subject matter expertise and process facilitation skills are required for training PBL tutors (7). Moreover, Baroffio et al. reported that facilitation skills and knowledge about content should be taught when students are tutors. They designed, implemented and evaluated a workshop for training tutors. It was reported that individual needs of tutors and practical training should be considered in designing tutor training program (TTP) (8). Dolmans et al. analyzed the studies conducted in the field of tutoring and reported that content mastery exerted different effects on students’ academic achievement. Their results also showed that facilitation skills and mastery of content are required for the tutors to be successful (9).

Liew et al. assessed the effects of near peer tutoring program. Their findings showed that assuming the role of tutor increased the students’ knowledge about the content and made them interested in education (10). Further, Burgess et al. performed a review study and found that using students as tutors helped them become interested in education (5).

In the study by Hajihosseini et al., peer teaching was used to teach the procedures to the nursing students of Mazandaran University of Medical Sciences, School of Nursing. The findings showed that peer teaching could be as effective as the teacher education (11). Although peer teaching was used in this study, it was not mentioned whether they had been trained in tutoring skills or not.

“The role of basic sciences in medicine” workshop is one of the workshops of “transition course from high school to university” at Tehran University of Medical Sciences (TUMS), School of Medicine. The transition course is held annually at the beginning of the academic year for the first-year medical students as a set of educational-recreational-cultural programs, aimed at preparing students for this transition. This workshop requires the use of senior students to facilitate learning in PBL groups.

Although the value of preparing tutors of PBL to perform their role appropriately is well-established, little attention has been paid to the evaluation of the performance of tutors in PBL sessions using the perspectives of the tutees and field notes in Iran using the Kirkpatrick model of evaluation.

2. Objectives

We investigated three specific research purposes: (1) To assess the performance of tutors in guiding PBL groups after participating the TTP; (2) to examine the differences between tutor performances in related factors, including the tutors’ gender and education level, in an Iranian undergraduate medical education setting; (3) to determine tutors’ reaction to and learning from toward TTP.

3. Methods

This mixed method, concurrent triangulation study was carried out after approval from TUMS, school of medicine. The participants in the TTP were the medical students of basic sciences (second year), physiopathology and clerkship. The participants included 22 students of medicine as tutor and 240 newly admitted students as tutee.

The inclusion criteria were appropriate academic performance and recognition of the participants as active and capable students in communication and teamwork skills. The student’s capability in communication and teamwork skills was determined based on the student’s performance profile which is available at the school of medicine- education development office. Finally, after announcement, 30 students were registered and invited to the TTP.

Based on the review of related literature the draft of the TTP was developed and finalized in a meeting with three experts in the field of medical education. First, a half-day workshop was designed to introduce participants to the principles of PBL, and the role of the PBL tutor followed by hands-on practice, feedback and discussion. Next, three two-hour meetings were held to review the clinical scenarios (designed by “basic science integration to clinical medicine” team). Finally, to help students play the role of tutor, a facilitation guide was developed about how to start the discussion, how to ask questions, how to guide group work, and how to deal with group discussion challenges.

Kirkpatrick model, a well-established model to evaluate the educational programs, was considered as the framework. According to this model, four levels of reaction, learning, behavior and results of an educational program could be evaluated (12, 13). In this study, the reaction, learning and behavior were evaluated.

Three researcher-made scales were utilized, including the TTP evaluation questionnaire, peer evaluation, and field note taking form. The validity and reliability of these scales were assessed by five experts in medical education who were familiar with the concept of tutoring/PBL. To this end, the drafts of tools, which were developed according to the objectives of the program, were emailed to them. Based on their comments, some items were combined and some others were revised. The reliability of the TTP evaluation questionnaire and peer evaluation form was reported to be 0.71 and 0.78 using Cronbach’s alpha.

3.1. Level 1 and 2- Tutors’ Reaction and Learning

Tutors’ reaction to TTP as well as their learning was evaluated by a 9-item questionnaire on a Likert scale ranging from 1 to 5. In this scale, the views of tutors about the program, and their perceptions about what they learned were asked. For the latter purpose, one open-ended item was provided for the comments to be expressed.

3.2. Level 3- Performance

Evaluation of tutors’ performance in PBL session comprised of two methods of evaluation: from the perspective of tutees and observation records. At the end of the PBL session, students evaluated their tutor’s performance using a peer evaluation form with 5-point Likert scale (from completely agree to completely disagree) the form items were: (1) Strategies of opening the discussion; (2) tutor’s ability in managing the session and guiding the group discussion; (3) providing an opportunity for students’ discussion. Tutor’s performance was graded ranging from 3 to 15. Higher scores signify a higher level of performance.

3.2.1. Note Taking During Observation Phase

A field note record form was used to capture the performance of tutors by an expert external observer. Data were collected as field notes in a short span of time, immediately after the observation, and were then analyzed using conventional content analyses to extract codes and themes (14).

3.2.2. Problem-Based Learning Session (Workshop on the Role of Basic Sciences in Medicine)

Each tutor was assigned to a group of 10-members (a total of 22 groups) and played their role for 16 hours. Six clinical scenarios which were used in the PBL session were related to the complications of blood hemostasis, immune, cardiovascular, respiratory, endocrine and gastroenterology systems, with items related to basic sciences. A sample of scenarios is presented in Box 1.

The seven-step PBL model was performed, including the statement of problem, precise determination of problem, analysis of problem, thematic classification and formation of hypotheses, determination of the learning objectives, collection of data and presentation of PBL report. After a written statement and study of the problem, the students were asked to brainstorm on the scenarios and write down their questions and ambiguities, form hypotheses and determine their learning objectives. Then, they were given two hours to find answers to their questions and achieve the given objectives through books, library and internet sources. Next, they summed up the results and shared their findings (15). Finally, the representatives of each team presented a summary of their scenarios, responses they had come up with and interesting points they had found through PBL process.

3.3. Data Analysis

The quantitative data were analyzed by SPSS-22 software. Quantitative variables are reported by mean (SD)/median (Q25th, Q75th) and qualitative variables are presented through frequencies (percentages) where appropriate. Furthermore, differences in performance of tutors between male and female tutors were analyzed with the Mann-Whitney test. Differences between basic sciences, physiopathology and stager tutors’ performance were analyzed with Kruskal-Wallis test. Content analysis was performed to analyze the field notes.

4. Results

Out of 30 students registered, 22 participated in the TTP. Some of them quit the study due to interference of hospital classes. Out of 240 newly admitted students, 153 completed the evaluation questionnaires (63.75% response rate). Seventy six (50%) were female, and students’ mean age was 18.05 ± 1.2 years. Participants’ demographics are presented in Table 1.

4.1. Levels 1 and 2- Tutors’ Reaction and Learning

All the tutors (100%) believed that the TTP persuaded them to guide a small group. Also, all of them reported they could get a clear understanding of facilitation skills, 20 (90%) stated they were able to perform tutor role appropriately and 14 (63.6%) stated they were able to recognize and deal with the tutoring challenges. Students commented that “The program was excellent”, “My knowledge and perceptions about facilitation and guiding a group were changed compared with the past”, “Now I am more aware of the PBL benefits”, “It was interesting”, and “Facilitating a group learning was exciting, I learned how to facilitate PBL”.

4.2. Level 3- Tutors’ Performance and Comparison Based on Gender and Education Level

The total mean of performance was 13.32 (0.72). There was no evidence of difference in tutors’ performance between men and women. The median was 13.37 (13.16 - 13.90) for men and 13.40 (12.37 - 13.48) for women, P = 0.89 (Table 2).

We found no difference in tutors’ performance based on the level of education using the Kruskal- Wallis test (χ² = 1.84, DF = 2, P = 0.39) (Table 3).

Content analysis of the field notes showed four main themes of tutors’ performance in PBL session “helping to learn how to apply basic science”, “deep learning enhancement”, “group dynamics creation”, and “interest in the field of education”.

Some students stated that tutors helped them to learn how to apply basic science and made them interested in studying basic science. In their final group presentation, one student mentioned:

“I want to thank our tutor. He helped us to understand the importance of basic science, now I am more interested in studying it”.

Another student said:

“Today, I learned that seniors can help us to learn the application of basic science. She taught us that without a mastery of basic science, we are not able to be a professional physician”.

Deep learning enhancement was expressed in the following way:

Group 04 tutor at the rest time stated: “I requested my group members to draw concept maps”

Group 05 tutor at the rest time spoke excitedly: “My group members extracted 58 questions from one scenario”.

One student in their final group presentation showed a drawing of the heart anatomy, presented his group answers and mentioned that it was the result of their group effort to understand the anatomy of the heart to solve the problem.

This was also observed in a situation where one student showed a cycle from the kidney physiology and the effects of Lupus on the physiologic function and explained how different interventions can reduce the symptoms. His suggestions were interesting for the internist who was the director of the workshop.

Group dynamics creation and interest in the field of education themes were reflected in the following observations:

“Student tutor 02 asked her group members at the start of the discussion to introduce themselves with three trustful and one false personalities. She used an ice breaker to prepare them for the rest of the discussion”.

Group 04 tutor at the rest time stated: “I requested my group members to draw concept maps” “During group discussion 09, one of the group members tried to force his own views upon the group four times. Three students seemed to be uninterested. One student asked to go out for a rest, and the tutor reminded them that they should appreciate each other’s views and should listen to one another. The tutor asked student X to do some research on their idea and present the results to their teammates”.

5. Discussion

The present study was performed to assess the performance of tutors in guiding PBL groups after participating TTP, compare their performance based on gender and education level, and determine their reaction to and learning from TTP. Hence, based on Kirkpatrick model, three levels of the effects of the TTP (reaction, learning and behavior) were evaluated.

The tutors believed that they were prepared to play their role as tutor appropriately. Studies have shown that tutors’ knowledge about the small group facilitation process along with familiarity with content can promote their qualifications in guiding discussions (5, 8, 16).

The results of this study indicated that tutors’ performance was satisfactory. Studies have shown that the presence of a tutor during the discussion, especially when they are a senior student, can help students express their ideas more easily and provide an enjoyable learning experience (5). Effects of solving challenging cases with the help of a peer can be explained by Vygotsky’s social development theory and scaffolding concept. According to this theory, when students solve the learning problems with the help of someone who is more capable, they gain more confidence in solving the problem and achieve higher levels of development. A short-term effect of scaffolding is the creation of a sense of support in learning, which can increase satisfaction with peer teaching (5, 16-18).

Our results suggested that there was no difference between men and women in tutors’ performance. This finding is in accordance with that of Groves et al., who reported that although the tutor’s gender did not correlate with their total performance, there was a positive relationship between tutors’ gender and students (7).

Our findings revealed that the performance of tutors did not differ in basic science, pathophysiology or clerkship levels. One of the major debates in tutoring of PBL is whether mastering the content has a significant effect on the performance of tutors and their effectiveness. This issue is a widely investigated subject in the literature. However, maybe expertise in basic science is considered valuable in PBL tutoring as a qualification in the clinical/content knowledge. An alternative explanation is that the main purpose in our PBL session was learning the basic science concepts; therefore, the tutors had to use their knowledge of basic science. In other words, the level of expertise (on basic science), was perhaps at the same level. In a study by Groves et al., no significant differences were observed between clinicians and non-clinicians in terms of overall effectiveness of tutors (7), which is in line with our finding. Alternatively, and perhaps most likely, failure to find significant differences may be attributed to the small sample size.

Deep learning enhancement was one of our most important findings. One of the main goals of using active learning methods is achieving higher levels of learning, analysis and synthesis. The students’ analysis of the cases and proposing creative suggestions for solving patients’ problems, based on the knowledge of basic sciences, were indicative of their understanding, analysis and synthesis of the knowledge of basic sciences, which was clear in their presentations. The students asserted that tutors played a pivotal role in creating this deep understanding. This was in line with the results of Ward and Lee's study, which showed that the PBL method was a useful strategy to develop higher levels of thinking in students (19).

Group dynamics creation was the other main theme in this study. It appears that tutors were competent in making students engage in group work and had a major role in increasing the group dynamicity. In fact, students were able to construct knowledge by self-study and support of a senior student, which consequently encouraged them to participate in group discussions (19-24). More intervention studies should be designed to compare the dynamicity between teams with and without tutors.

The tutors’ use of teaching techniques like ice breakers, drawing concept maps and diagrams were other observations in the current study. It appears that tutors had studied the educational techniques well beyond what they were taught in the TTP, and had focused their efforts on guiding the discussions. These efforts, which could be an indication of creating interest in the field of medical education, were similar to the results of Burgess et al. study, which showed tutors expressed interest in education (5). Moreover, Baroffio et al. and Liew et al. reported the same results (8, 10). More studies should be done to explore the long-term effects of this result, i.e. whether the tutors will continue their activity in the field of medical education in the future or not.

A strength of the current study was the attention paid to developing facilitation skills among medical students before using them in the context of PBL. Moreover, an attempt was made in this study to use the field notes and to evaluate the tutors’ performance from the viewpoint of both participants and an external observer. Finally, and the most important strength of the study, was the assessment of the tutors’ performance based on Kirkpatrick model. Three levels of the effects of the TTP (reaction, learning and behavior) were evaluated while other studies are limited just to one or two levels.

One of the limitations of this study was that tutors’ performance was evaluated in a short interval. Future studies are recommended to examine the effects of the TTP on their long-term performance in various situations. The tutors were the top students, which might have affected the results. The observer was one of the research team members and it may have resulted in bias. A small sample size was another limitation of this study. Sample size was limited due to concerns about the quality of education and time-consuming practical exercises. The short duration of TTP and evaluation of tutors’ performance was related to the educational schedule in the medical school.

5.1. Conclusions

Tutors’ reaction to and learning from TTP was positive and satisfactory. The results suggested no differences between men and women in terms of tutors’ performance. Furthermore, our findings revealed that the performance of tutors did not differ based on education level (basic science, pathophysiology or clerkship). Four main themes of tutors’ performance in PBL session were “Helping to learn how to apply basic science”, “Deep learning enhancement”, “Group dynamics creation” and “Interest in the field of education”. However, further research must be conducted with a large sample size and on other aspects of tutoring, i.e. sense of support in learning, relationship between tutors’ gender and students learning, dynamicity between teams with and without tutors, whether the tutors will continue their activity in the field of medical education in the future or not.