1. Background
Breaking bad news (BBN) is a complex communication task that is a common occurrence for clinicians working with oncology patients. Being aware of how bad news is shared between providers and patients is essential in the examination. Consequently, different formats of assessment tools (e.g. detailed checklist and global rating scales) have been developed, as well as the assessor (e.g. standardized patients and independent raters) (1).
Moreover, the need for valid assessment of medical students’ preparation in communication skills has been emphasized in evidence. This issue is particularly important for the high-stakes exam, in which decisions of students’ Pass/Fail are made (2). It was proposed that Standardized patients (SPs) play a vital role in ensuring more objective means for assessment, particularly in the field of CS (3). Therefore, it is worthwhile to consider whether the use of SPs assessment has been also applied in North American medical schools and Licensure examinations (4, 5). At this point, certain studies have employed SPs for assessing CS and BBN (6-8).
Some studies indicated that adequately-trained SPs could be satisfactory alternatives to faculty raters (9, 10). On the contrary, other studies indicated less agreement between SPs’ and instructors’ ratings (11-13). Based on the previous research, intensive rater training is one of the most key points in achieving the consistency of scoring (14).
2. Objectives
A common concern regarding educators, who adopt the use of SPs for performance assessment, is to train SPs reliably and consistently to re-enact scenarios and to assess the student. Despite the widespread use of SPs as an assessment tool, there is a paucity of literature in the oncology profession, seeking to determine the quality assurance, the accuracy (validity), and consistency (reliability) of the SPs as the tools used for the assessment of BBN. For this reason, this study assesses the psychometric characteristic of a well-trained and unannounced SP as the evaluator in the clinic.
3. Methods
This is a cross-sectional study to describe the process of creating valid SPs for assessing fellows’ performance-related BBN. The study was conducted in the following steps:
3.1. Scenario Development
Four various clinical scenarios depicting cancer cases (chronic myeloid leukemia, chronic lymphocytic leukemia, lung cancer, and stomach cancer) were developed by consensus among the panel of experts, consisting of medical educationalists, and physicians specialized in oncology. Each scenario included the key points of standardized information related to BBN to be presented in the real encounter. To facilitate the scenario writing process, a case template was developed, in which the experts could fill out some detailed information regarding the case content consisting of social and demographic, lifestyle, symptoms, medical history, family history, findings on physical examination, laboratory and imaging. The content validity of the scenarios was ensured through consensus in an expert panel including 10 experts from the Medical Education and Department of Oncology at Tehran University of Medical Sciences (TUMS).
3.2. Selection of SPs
Eight SPs volunteers from the simulated patient pool at TUMS were invited to take part in this study.
3.3. Training the Standardized Patient
SPs received a 3-day training focusing on their realistic portrayal of each clinical scenario (authenticity), and SPs’ practice rating the BBN’s criteria on a checklist to record the practice of the performance of oncology correctly. The case developers and faculty members, who have experience in working with SPs, undertook the training of all SPs.
All SPs were trained by working as SP trainers.
Based on the SP training method (15), the following steps have been accomplished:
1) Familiarizing SPs with the clinical scenario reading the case materials;
2) Ensuring that SPs have learned trainer checklists;
3) Performing SPs role play of the cases with feedback from the trainers and peers;
4) Allowing SPs to dress rehearsal to enhance the authenticity of their performance;
5) Promoting SPs’ ability of accurate and consistent checklist scoring skills.
During training (rating checklists and SPs’ portrayals), the SPs read detailed written instructions describing the case, case-specific observational checklist, and watched videos regarding BBN, and also receive training protocols and complete assessment instruments.
Moreover, SPs play their roles with other SPs and faculty members under the supervision of an experienced faculty. In addition, the training process involved SPs-visit in the outpatient oncology setting. During these unannounced visits and interactions with real patients, the SPs became more familiar with all aspects of the scenario; besides, they learned how to handle certain situations. After the encounter in the oncology outpatient setting, SPs practiced all aspects of their roles as cancer patients.
Throughout the study, SPs participated in weekly meetings for further training in their roles, discussing common SP errors (rating checklist & portrayals), how to avoid them, and receiving feedback from both peers and faculty on their performances.
3.4. Validation of SPs’ Portrayals
Based on different scenarios, case-specific and observational checklists for measuring SP role-plays were developed via an expert team consisting of oncologists and medical education experts. The validity of the scale was determined by agreement among the panel of experts. Each key relevant item that was noted to the SPs in training was included in the scale, comprising 5 items for verbal and 5 for nonverbal expressions. Each item was graded on a Likert scale from 1 to 3 (1 = poor, 2 = mild, and 3 = good” or “excellent). The final score was calculated as the mean score of the ten items.
One week after the training, the performance of each of the 8 SPs during the interactions with an oncologist as the doctor was assessed by other oncologists and a medical education expert, using case-specific scales. The SPs portrayed 2 cases with the same oncologist. Each of the 4 scenarios was performed by 2 SPs. The SPs’ portrayal would have an acceptable accuracy if it reaches the cut-off score of 90% or above. To determine the reliability of SPs’ portrayals, the test-retest approach across the first and second role-play was employed.
3.5. Validation of SPs’ Completed the Checklist
BBN scale was selected as the tool for measuring the BBN skill of oncology fellows in the outpatient setting. The BBN’s validity and reliability were published in a previous study in Iran (16).
The BBN checklist had 16 items, measuring 7 variety domains of BBN skills, including the setting interview (3 items), strategy (2 items), planning (2 items), professionalism (1 item), empathy (2 items), knowledge (4 items), and invitation (2 items). This examination tool was to be completed by the SP after the consultation with the fellow.
For testing the concurrent validity of the SPs’ ability to complete the checklists, each SP played their role with 3 oncologists (who were not involved in the study) as a clinician in a simulated medical environment. Afterward, SPs rated the clinician’s performance. Following the SP-clinician encounter, other oncology faculty, who had experience in working as SP-based performance assessments independently, rated the SP-faculty interaction immediately. Accordingly, the agreement of the SP in grading the BBN scale and again scale-rating by an independent expert (oncologist faculty members) as a gold standard supported the concurrent validity of the SP-based performance assessment. Additionally, the inter-rater reliability of 2 SPs on the same condition was assessed for indicating the reliability of the SPs rating checklist.
3.6. Data Analysis
We analyzed the data using SPSS. To determine the correlation, the kappa (k) coefficient was computed.
3.7. Ethical Considerations
The study was approved by the Ethics Committee of Tehran University of Medical Sciences (Reference: IR.TUMS.REC.1394.1621).
4. Results
The mean age range of the SPs was 46.5 ± 9. There were 2 men and 6 women, who had 9.5 ± 2.5 years of work experience. The mean score for measuring the SPs’ portrayal validity was 2.97 (ranging: 2.90 - 3.00). The assessment of the portrayal’s reliability indicated a mean of 89% (range: 82.6% - 96%) equal responses to the items on the observational rating scale. The mean K for the validity of SPs’ filled-out checklists was 0.82 (ranging: 0.694 - 0.985). The mean K for the reliability of SPs’ completed checklists was 0.86% (ranging: 0.725 - 1.000). The scores and individual test results for the 8 SPs are shown in Table 1.
| Standard Patient (SP) | Observational Rating Scale | Checklists | ||
|---|---|---|---|---|
| Total Mean of Raters’ Validity of SPs’ Portrayal | Percentage of Inter-rater Reliability of SPs’ Portrayal | K Coefficient (validity of Completed Checklists) | K Coefficient (Reliability of Completed Checklists) | |
| 1st SP | 3.000 | 93.8 | 0.848 | 0.921 |
| 2nd SP | 2.96 | 96 | 0.794 | 1.000 |
| 3rd SP | 2.98 | 79.8 | 0.784 | 0.725 |
| 4th SP | 3.000 | 94 | 0.985 | 0.883 |
| 5th SP | 3.000 | 90 | 0.784 | 0.788 |
| 6th SP | 2.98 | 82.6 | 0.694 | 0.882 |
| 7th SP | 2.90 | 83. | 0.789 | 0.785 |
| 8th SP | 2.95 | 92.8 | 0.890 | 0.898 |
Mean Scores of Validity of SPs’ Portrayal and Inter-Rater Reliability of SPs’ Portrayal as Well as the Validity and Reliability of Completed Checklists by Each SP
5. Discussion
Since outcome-based education and early clinical encounter is introduced in medical education, a great deal has been accomplished in developing more standardized, objective, and structured methods of assessment. To address this need, SP-based assessment has become widespread in undergraduate and graduate medical education (17).
Although how SPs portray the case efficiently is identified as SP error in SP-based assessment, quality assurance is indicated to significantly reduce the SP performance error (18, 19). In addition, literature has noted that SPs should be measured for accuracy and reliability of the portrayal ahead of being employed in performance assessment (20, 21).
In the present study, we evaluated the accuracy (validity) and consistency (reliability) of the SP-based performance assessment to estimate oncology fellows’ performance regarding BBN skills, when SPs serve as unannounced patients in the clinical practice. According to evidence, the cut-off score of 85% or higher rate of agreement with the “expert” that developed the case for the SPs’ portrayal is considered to have strong accuracy (22). Therefore, our results implied that SPs can present different scenarios of cancer patients with a high level of accuracy (cut-off score = 95% for all SPs). Our results were in line with the study representing 91% and 89% of accuracy for SP portrayal in the prenatal and cancer cases (23).
In our opinion, our finding is rather related to the development of a valid case-specific observational checklist to evaluate the SP’s authenticity by the oncologist in defined task components. This also guided the oncologists to utilize such scale as an instructional tool to provide the SPs with constructive feedback.
On top of that, the current study indicated that the reliability of performance of all SPs portraying each scenario across two role-playings was acceptable. This might be due to the best selection of experienced SPs and systematic training of them, which is in concordance with the findings of an early study (24).
Regarding the validity and reliability of rating the fellows’ performance on BBN checklists by SPs, in the current study, SPs’ ratings were found to be highly consistent. Furthermore, there was a high percentage of agreement (80% - 100%) between the SPs’ completing the checklists and experts’ judgments. It should be noted that the SP stake part in the training session learning how to rate BBN skill consistently. In addition, literature suggested that the reliability of the assessment by SPs increases using case-specific checklists and careful rater training (25). As noted in a previous study, providing rater training is of great value for achieving more reliable and valid results. However, one of the consequences of this experiment is that training not only ensures that all raters interpret the content of an item’s description similarly, but also they apply similar standards to students’ performances (13).
Overall, our results demonstrate that well-trained incognito SP can serve as a standardized assessment tool for assessing oncology fellows’ performances regarding BBN skills in the clinical field. Although the small sample size of our study limits the generalizability of the results, quality assurance processes for SP-based performance assessment of health care providers were outlined. It would be valuable for future research to report this process of training and validation of SPs in other areas of medicine.
