Self-assessment and Teacher-Assessment During an OSCE in Undergraduate Dental Students-Application of a Visual Metaphor

1. Background

Teaching in undergraduate dental education is challenging due to the vast array of competencies required for graduates to practice autonomously immediately after their studies (1). In this context, self-assessment and self-directed learning are critical skills for dental students and practitioners, making these topics essential in dental curricula (1, 2). Self-assessment has been shown to enhance motivation, learning, and academic performance (3). Recent systematic reviews emphasize the importance of self-assessment and strategies to support it by increasing attention to self-assessment and developing and testing relevant approaches (4, 5).

In addition to students' self-assessment, teacher and peer assessments are also crucial in dental education. Peer assessment has been identified as a valuable tool to support the training progress of dental students (6). Specifically, in objective structured clinical exams (OSCE), the combination of self- and peer-assessment has been found to be beneficial for dental students (7, 8). Furthermore, understanding the relationship between teacher-assessment and students’ self-assessment is of significant interest in both dental and overall medical education. A recent study from Japan indicated that while the assessments by students and teachers in medical bedside teaching were generally consistent, some students tended to over- or underestimate their performance (9). This variation in perceptions between teacher-assessment and self-assessment is crucial for providing substantive feedback in dental education (10). It also impacts the student-teacher relationship, a key element in creating a positive learning environment and facilitating academic progress (11). Thus, examining the relationship between students' self-assessment and teacher-assessment is valuable in the educational context.

In this context, various approaches can be used to perform and visualize self-assessment and teacher assessment. Tools such as rubrics, digital methods, or scanners have been reported to support self-assessment (6, 9, 12, 13). In addition to these strategies, a novel visual tool has recently been introduced to illustrate the learning progress of dental students. The pictorial representation of illness and self-measure (PRISM) is a visual metaphor that has already been used in dental education to visualize students' self-perceived competencies (14). Pictorial representation of illness and self-measure has been found to be superior to numeric scales for self-assessment (15), suggesting it could be a promising approach for depicting both self-assessment and teacher-assessment in a dental education context.

2. Objectives

In summary, this study was designed to visualize self-assessment and teacher-assessment within a dental education setting. To achieve this, different practical tasks within an OSCE were utilized. Additionally, students' estimated self-assessment from the teacher's perspective (how teachers would expect students to rate their own performance) and the estimated teacher-assessment from the students' perspective (how students would expect teachers to rate their performance) were evaluated using PRISM. The study aimed to compare students' self-assessment with teacher-assessment, as well as with estimated self-assessment and estimated teacher-assessment using PRISM. Furthermore, self-assessment and teacher-assessment were compared with OSCE scores. It was hypothesized that self-assessment and teacher-assessment would correlate with each other.

Additionally, it was hypothesized that the estimated self-assessment (from the teacher's perspective) and the estimated teacher-assessment (from the students' perspective) would show limited correlations with actual self-assessment and teacher-assessment. The third hypothesis assumed that both self-assessment and teacher-assessment would correlate with the OSCE results, as reflected by the sum score.

3. Methods

3.1. Study Design

This observational cohort study included a group of fourth-year undergraduate dental students before the beginning of their first clinical patient course. The ethics committee of the Medical Faculty of the University of Leipzig, Germany, reviewed and approved the protocol (No: 117/20-ek). Written informed consent was obtained from all participants to ensure voluntary participation.

3.2. Participants

Prior to the beginning of the clinical patient treatment course in October 2022 for fourth-year dental students, an OSCE was conducted. The participants of this OSCE, a cohort of fourth-year undergraduate dental students, were invited to participate voluntarily and were subsequently included in the study. Only students who were undergoing the OSCE for the first time were considered for participation. No further inclusion or exclusion criteria were applied.

3.3. Objective Structured Clinical Examination Tasks

The OSCE comprised a series of tasks. For the current study, three tasks were selected (Figure 1). First, students performed an oral examination on a dental model featuring healthy, carious, and restored teeth, and were required to record the findings accurately. In the second task, students selected and placed a suitable matrix for an occlusal-distal defect. In the third task, students evaluated a dental radiograph of endodontically treated teeth to assess the adequacy of the endodontic therapy. After completing each OSCE task, students self-assessed their performance as good, moderate, or poor. Additionally, an independent rater assigned an OSCE score based on a set of criteria and points, using a provided protocol to ensure consistent and reproducible ratings.

Tasks within the objective structured clinical examination (OSCE), which were used in the current study. Task 1 included a dental examination at a model. Task 2 was the correct choice and application of a matrix for an oral-distal cavity. Task 3 was the assessment and evaluation of an endodontic radiograph.

Figure 1.

Tasks within the objective structured clinical examination (OSCE), which were used in the current study. Task 1 included a dental examination at a model. Task 2 was the correct choice and application of a matrix for an oral-distal cavity. Task 3 was the assessment and evaluation of an endodontic radiograph.

3.4. Pictorial Representation of Illness And Self-Measure (PRISM) Tasks

The general approach for applying PRISM in dental education has been previously described in detail (14). Originally developed in the field of psychology/psychosomatic medicine (16), PRISM consists of a white metal board (297 × 210 mm; "context") with a fixed yellow circle (7 cm in diameter) located in the bottom right corner ("subject"). Different colored discs (5 cm in diameter, "objects") are placed relative to the yellow circle to represent their relationships.

In the current study, PRISM was adapted to visualize both the students' self-assessment and the teacher's assessment, as well as their perceptions of each other’s assessments. Both the teacher and the students used their own PRISM boards. After completing each task, teachers placed two objects on the board: Their evaluation of the student's performance (teacher-assessment) and their estimate of the students' self-assessment (how they thought students would rate their own performance) (Figure 2). Independently, students placed two objects: Their evaluation of their own performance (student’s self-assessment) and their estimate of the teacher's assessment (how they thought teachers would rate their performance) (Figure 3). Teachers and students performed the PRISM evaluation independently and without knowledge of each other’s placements. PRISM evaluations were conducted for each of the three tasks separately. Distances between the centers of the objects and the subject were measured in centimeters using a template (ranging from 0 to 25 cm).

Principle of pictorial representation of illness and self-measure (PRISM) as used in the current study. The teacher and the student had an own PRISM board each. This figure shows the teacher board, whereby teacher-assessment and estimated self-assessment of the student were evaluated for each task.

Figure 2.

Principle of pictorial representation of illness and self-measure (PRISM) as used in the current study. The teacher and the student had an own PRISM board each. This figure shows the teacher board, whereby teacher-assessment and estimated self-assessment of the student were evaluated for each task.

Principle of pictorial representation of illness and self-measure (PRISM) as used in the current study. The teacher and the student had an own PRISM board each. This figure shows the student board, whereby student´s self-assessment and estimated teacher-assessment were evaluated for each task.

Figure 3.

Principle of pictorial representation of illness and self-measure (PRISM) as used in the current study. The teacher and the student had an own PRISM board each. This figure shows the student board, whereby student´s self-assessment and estimated teacher-assessment were evaluated for each task.

3.5. Study Flow

The flow of the current study is depicted in Figure 4. Due to organizational reasons, the OSCE was conducted over three consecutive days. The entire group of 44 students was randomly assigned to the respective days. Students from each group completed all three OSCE tasks on the same day.

Study flow; at three different days, the randomly assigned students underwent all of the three objective structured clinical examination (OSCE) tasks and performed pictorial representation of illness and self-measure (PRISM) with a respective teacher. An independent rater provided the OSCE score for the respective task.

Figure 4.

Study flow; at three different days, the randomly assigned students underwent all of the three objective structured clinical examination (OSCE) tasks and performed pictorial representation of illness and self-measure (PRISM) with a respective teacher. An independent rater provided the OSCE score for the respective task.

Students first underwent the OSCE tasks. After completing these tasks, both students and teachers performed the PRISM evaluation independently. Students then assessed their own performance (good, moderate, poor) and received their OSCE scores from an independent rater. Three calibrated and trained teachers conducted the PRISM evaluations. Calibration was achieved by performing repeated independent PRISM evaluations of constructed cases until there was at least 80% agreement among the values. To minimize potential teacher effects on the results, teachers rotated among the three tasks throughout the days. Experienced dentists acted as raters for the OSCE tasks and used standardized rating protocols for this purpose. The detailed study flow can be seen in Figure 4.

3.6. Statistical Analysis

The basis for statistical analysis was the distance (in centimeters) between the objects representing the student’s self-assessment, estimated teacher-assessment, teacher-assessment, and estimated self-assessment, and the subject circle at the bottom right-hand corner of the PRISM board. Statistical analysis was performed using SPSS for Windows, version 25.0 (SPSS Inc., U.S.A.).

First, normal distribution was assessed using the Shapiro-Wilk test. Correlations between the different assessments were evaluated using the Pearson correlation coefficient. The following interpretation was used: 0.1 - 0.39 = weak correlation, 0.4 - 0.69 = moderate correlation, 0.7 - 0.89 = strong correlation, and 0.9–1 = very strong correlation (17). A potential influence of the teacher on the different assessments was tested using linear regression. Two-sided significance testing was conducted for all values, with the significance level set at P < 0.05.

4. Results

4.1. Participants

In the current study, 44 fourth-year undergraduate dental students, with a mean age of 23.5 ± 2.6 years and comprising 64% female participants, were included. The OSCE results revealed that students achieved the lowest mean score in the first task, which involved an oral examination at a model. The OSCE results are presented in Table 1. The distances in PRISM for the different assessments and tasks are detailed in Appendix 1.

4.2. Correlation Between Students’ Self-Assessment and Teacher-Assessment Using Pictorial Representation Of Illness And Self-Measure (PRISM)

In Task 1, a moderate correlation between students' self-assessment and estimated self-assessment was found (r = 0.44, P < 0.01). In Task 2, moderate correlations were observed between self-assessment and teacher-assessment, estimated teacher-assessment and teacher-assessment, as well as between self-assessment and estimated self-assessment (P ≤ 0.01). In Task 3, a moderate correlation was found between self-assessment and teacher-assessment, as well as between self-assessment and estimated self-assessment (P < 0.01, Table 2).

4.3. Correlation Between Pictorial Representation of Illness And Self-Measure-Based Assessment and Objective Structured Clinical Examination Results

The self-assessment in PRISM and the OSCE self-assessment (good, moderate, or poor) were significantly correlated across all three tasks (P < 0.01). Notably, in Task 3, a strong correlation was observed (r = 0.72, P < 0.01). A moderate negative correlation between self-assessment and the OSCE score was found only in Task 2 (r = -0.41, P = 0.01). For teacher-assessment in PRISM and the OSCE score, moderate negative correlations were found in all three tasks (P < 0.01; Table 2).

4.4. Influence of the Teacher

In the regression analysis, the teacher was found to be an influential factor on estimated self-assessment (β = 0.26, P < 0.01; Appendix 2). When using self-assessment and estimated teacher-assessment as the dependent variables, no significant associations were found (Appendix 2).

5. Discussion

This current study used PRISM, a visual metaphor, to depict students' self-assessment and teacher-assessment in an OSCE setting. The underlying method originated from the field of psychology and was conceived to visualize suffering, and was subsequently applied to several other fields in psychology and medicine (18). Previous studies successfully modified the original PRISM task for application in dental education, showing reasonable reproducibility, sensitivity to changes, and a perceived superiority over numeric scales for self-reflection (14, 15). This methodological approach has now been applied in the context of the current study to test three different hypotheses.

First, it was hypothesized that self-assessment and teacher-assessment depicted with PRISM would correlate with each other. As the results show, a moderate positive correlation was found in this respect, although this was only observed for task 2 (matrix placement) and task 3 (endodontics). The OSCE scores revealed that the average points for task 1 were the lowest among the three tasks, potentially indicating that task 1 (oral examination at a model) was the most challenging, which might have influenced the difference between students' self-assessment and the teacher-assessment provided by the respective teacher. A Japanese study, which compared self-assessment and teacher-assessment in medical interview performance using an assessment tool, showed good agreement between self-assessment and the teachers' ratings (9).

In contrast, another hospital-based study of medical graduates showed a wide range between self-assessment and assessment by an expert (19). Interestingly, this range increased in cases of inadequate performance (19). This could explain the lack of correlation in task 1, which appeared to be the most challenging task with the largest number of students with inappropriate results. Another study, which compared mini clinical evaluation exercises of clinical students with their supervisors in gynecology, internal medicine, pediatrics, psychiatry, and surgery, found correlations between student-assessment and teacher-assessment ranging from 0.29 to 0.51, depending on the task (20). The scale of correlation was similar to that in the current study and supports the assumption that the match between self-assessment and teacher-assessment depends on the task and possibly on how challenging it is for the student. Besides the tasks on which students and teachers assess performance, several influential factors appear relevant in the context of the overlap or discrepancy between self-assessment and teacher-assessment. It has been reported that students' strategies to self-assess their performance are complex and influenced by both internal and external factors (21). Potential issues include the purpose of assessment, students' self-efficacy, the context in which self-assessment is performed, and gender (21-24). Therefore, this study also considered the estimated assessment of the teacher (estimated self-assessment and estimated teacher-assessment) to gain a deeper understanding of the results, which led to the second hypothesis of this study.

This second hypothesis assumed that the estimated self-assessment (teacher perspective) and the estimated teacher-assessment (student perspective) would show limited correlations with the real self-assessment and teacher-assessment. Therefore, PRISM was used as a novel approach to depict the physical distance as a visual metaphor. For the relation between estimated teacher-assessment and real teacher-assessment, which was again task-dependent, almost weak or no correlations were found. In contrast, a moderate correlation was found between estimated self-assessment and the real self-assessment across all tasks. This is interesting, as it suggests that students' estimates of how the teacher would evaluate their work differ from the teacher's perspective on the student's self-assessment. On the one hand, this finding supports a type of strategic self-assessment, as mentioned previously (21). Students appear to anticipate a certain evaluation from the teacher, which might influence their self-assessment. In this context, the allocation of roles between the teacher and the student, along with a related imbalance of power and potential sources of conflict, has been a topic in pedagogy for several decades (25, 26). This role constellation and the associated internal or social tensions might influence how students expect to be rated by the teacher, who are often seen more as adversaries than as supporters.

Based on the current study's findings, PRISM could serve as a basis to discuss the discrepancy between students' and teachers' ratings, potentially helping to resolve this perceived mismatch. A strength of PRISM is that the visual metaphor can facilitate discussions about the problem using the metal board, thereby transferring the subject of discussion. This allows students and teachers to perceive the problem more objectively, which might support their collaborative validation. Given the significant impact of a good student-teacher relationship in medical teaching (11), this might be a novel and promising approach. However, this study did not include a discussion of the results between students and teachers based on the PRISM findings; this is planned for a subsequent project to confirm the potential benefits of using PRISM in this respect. Additionally, regression analysis showed that the estimated self-assessment was somewhat teacher-dependent. Therefore, teachers should be aware of how and under what circumstances they assess their students.

The third and last hypothesis of this study assumed that both self-assessment and teacher-assessment would correlate with the OSCE results, as reflected by a sum score. This was primarily evaluated to assess whether PRISM would be suitable for depicting a situation in accordance with objective scoring criteria. The teacher-assessment showed, as expected, a moderate negative correlation with the OSCE score of an independent rater. Since the principle of PRISM indicates that a lower distance reflects a better result, the negative correlation is plausible. These results support previous findings that PRISM can quantify competencies (14). In addition, students’ self-assessment in PRISM correlated with their assessment of the OSCE task as good, moderate, or poor, further supporting the potential of PRISM to facilitate self-reflection and learning progress. Interestingly, students’ self-assessment showed a task-dependent relationship with the OSCE score of an independent rater.

Thereby, no correlation (task 1), a weak negative correlation (task 3), and a moderate negative correlation (task 2) were found. These results indicate a mismatch between the students’ self-assessment and their actual performance, supporting the discussion of the second hypothesis. In principle, the ability to self-reflect on competencies appears essential in medical education, as it is related to performance, skills, and empathy (1, 2, 27, 28). Age and study progress are relevant influential factors on the self-assessment abilities of students (29). The current sample of beginning fourth-year students appears to have some limitations in their self-assessment skills. This might be due to their limited experience in a clinical context, as they start treating real patients in the fourth year. The visualization of this discrepancy using PRISM could be helpful, especially since a previously performed study showed that PRISM was perceived as a good tool for self-reflection among fourth-year dental students (15).

5.1. Strengths and limitations

Overall, this study employs a promising approach by using PRISM as a visual metaphor to illustrate students’ and teachers’ perspectives in an OSCE setting. Objective structured clinical examinations have long been recognized as reliable and valid procedures in dental education (30). Despite their widespread use and evaluation, there remains a continued need for improvement (30-32). While the reliability and validity of OSCE procedures are subjects of debate, this study sets the scoring of the OSCE and its ability to depict students’ competencies in relation to PRISM-based reflection. The results suggest that PRISM has the potential to enhance student-teacher communication, their relationship, and cooperation, which aligns with the idea of teachers acting more as coaches than judges (33, 34).

However, there are several limitations that need to be considered. Firstly, the sample consists of only 44 volunteers from a single center and was studied on a one-time basis. This sample size is too small to draw generalizable conclusions. Future studies should adopt longitudinal approaches with a larger sample size, possibly in a multicentric setting, to increase the robustness of the results. While the study execution appears robust, with measures such as independent OSCE raters, teacher rotation, independent and blinded PRISM tasks, varied tasks, and reproducible settings, this stringent setting limited the more practical application of PRISM. It has been documented that self-assessment should be combined with feedback or discussion of results (35). This will be a necessary step for future research in this field. The chosen approach represents a novel use of PRISM and thus has a pilot character. Further studies are needed to provide more evidence regarding the effects of PRISM in dental and medical education.

5.2. Conclusions

Within the limitations of this study, PRISM successfully visualized students' self-assessment, the corresponding teacher-assessment, and their relationship to the OSCE results for fourth-year undergraduate dental students. The estimated teacher-assessment and the real teacher-assessment often differed, potentially reflecting individual perspectives on performance and varying expectations within the teaching context. Using PRISM to illustrate these conflicting views could provide a solid foundation for addressing problems, defining learning objectives, and thereby enhancing the quality of feedback in dental and medical education.

5.3. Lay Summary

This study examined how dental students assess their own performance compared to their teachers during practical exams, using a visual tool called the PRISM. Forty-four fourth-year dental students participated in OSCEs involving three tasks. The findings revealed that students' self-assessments generally aligned with teacher assessments, although variations were noted, particularly in more challenging tasks. The study suggests that using PRISM can enhance communication about performance between students and teachers, potentially improving educational outcomes in dental training.

5.4. Highlights

Enhancing self-assessment skills: The study emphasizes the importance of self-assessment in dental education, suggesting that integrating tools like PRISM can help students better understand their competencies and improve their learning outcomes.

Improving teacher-student communication: By visualizing assessments, PRISM fosters clearer communication between students and teachers, allowing for more effective feedback and discussions about performance.

Modernizing assessment methods: The findings advocate for the adoption of innovative assessment methods in medical education, highlighting the need to move beyond traditional numeric scales to more engaging and informative visual tools.