1. Context
Chemotherapy is an effective but potentially dangerous treatment (1) and presents risks to the health of patients and healthcare team (2). Chemotherapy drugs are considered as high-risk medications due to their cytotoxic properties. These drugs have limited therapeutic index and have to be administered based on complicated protocols and programs (3). Maintaining the patient’s safety during the administration of chemotherapy drugs is an important goal for health institutions (4). Nurses’ knowledge and skill play a vital role in managing chemotherapy and preventing complications, so nurses should have adequate knowledge in this area and follow evidence-based recommendations and clinical guidelines (5, 6).
Clinical guidelines include advice and recommendations that are systematically developed so they can help the care providers with decision making in specific clinical conditions (7). In fact, clinical guidelines include a set of care recommendations developed for enhancing the effectiveness and safety of interventions, based on national needs and conditions (8). A good clinical guideline should be scientifically valid, practical, and consistent with clinical situations as well as it should eventually improve the patient’s condition (9). However, methods of the developing clinical guidelines are different in every organization, and developers are not always faithful to applying the best evidence while formulating guidelines. Accordingly some guidelines are often below the basic standards, which have raised concerns about the quality of these methods (10, 11). The purpose of quality of clinical guidelines is to ensure that potential biases in the process of developing guidelines are well prevented, the internal and external validity of recommendations are met, and the recommendations are clinically applicable (12). Increase in the number, complexity, and heterogeneity of clinical guidelines and concerns about their quality has led to an increased need to develop internationally-recognized criteria for determining the quality of clinical guidelines (11, 13). The appraisal of guidelines for research & evaluation (AGREE) instrument is a valid evaluation tool that has been approved by the leading critics and authors of international clinical guidelines (14) and is the only clinical guideline assessment tool that has been formally developed and internationally validated. It has been approved by several organizations, including the advisory committee on health research of World Health Organization (WHO) (15) and provides a framework for evaluating the quality of clinical guidelines (16). So far, no study has been conducted in order to assess the quality of clinical guidelines for the administration of chemotherapy drugs. Therefore, this study aimed to assess the quality of clinical guidelines for the administration of chemotherapy drugs by using AGREE II tool in order to clinically apply them for the patients with cancer with the aim of improving evidence-based decisions and increasing the safety of the patient and the health care team.
2. Evidence Acquisition
This study is part of the nursing PhD dissertation entitled “Designing Clinical Guideline for Preparation and Administration of Injectable Chemotherapy Drugs and Prescription Care in Adult Cancer Patients for Nurses Working in Iranian Oncology Centers” with the confirmation of the vice-chancellor in research affairs (code: 395553) and using the adaptation method.
2.1. Selection Criteria
Inclusion criteria were: (1) clinical guidelines for the administration of chemotherapy drugs; (2) the target population consisting of adult cancer patients (above 18 years old); (3) guidelines with English language; (4) clinical guidelines that were developed by institutions, associations, and groups associated with cancer; (5) clinical guidelines which were developed based on evidence-based and systematic reviews (the quality of evidence was examined). The latest updated version of the guideline for assessment was also selected and for the guidelines which were published in several forms, the one with the most details about the development methodology was selected.
Exclusion criteria were: (1) clinical guidelines without accessible full text; (2) service packs, care plans, systematic reviews, patients’ guides, and books.
2.2. Search Strategy and Screening of Guidelines
In order to choose clinical guides, English language articles which were published from 2007 to August 2018 were reviewed. We searched through robust databases such as Scopus, PubMed, ProQuest, Cochrane Library, MEDLINE, Web of Science, and clinical guideline development centers including the Guidelines International Network (GIN), the National Institute for Clinical Excellence (NICE ), The Cancer Care Ontario (CCO), the National Guidelines Clearinghouse (NGC), the Scottish Intercollegiate Guidelines Network (SIGN), and also Iranian article databases such as Irandoc, SID, and Magiran with a combination of keywords: clinical guideline, administration, chemotherapy, and cancer.
After a systematic search, related clinical guidelines to the search keywords were found our research team which was included a nursing PhD student, and two academic members in cancer nursing and familiar with the guideline development, evaluated the obtained clinical relevant guidelines based on the title, abstract, and text, as well as the inclusion and exclusion criteria after excluding repetitive titles. In case of disagreement, the research team reached a consensus by more discussions.
2.3. Appraising Screened CPGs by Applying AGREE II Instrument
Selected clinical guidelines were presented to the relevant experts to be evaluated with AGREE II. The experts reviewed the clinical guidelines individually and separately, in terms of publication year, developer institution, target population, quality of evidence, and main instructions using AGREE II tool. The researcher contacted the health system staff via phone or in-person and persuaded them to co-operate and participate in the appraisal of clinical guidelines. Then, carrying an invitation letter for co-operation containing a brief description of goals and methodology of the research, she visited their offices or workplaces and after obtaining informed consent forms from the appraisers, they were provided with a CD containing 4 clinical guides along with the AGREE II tool, its completion guide and necessary explanations on how to evaluate the guidelines, and their questions were answered. According to the recommendations in AGREE II manual, each guideline is evaluated by at least 2 and preferably 4 appraisers, as increasing the number of appraisers will increase the reliability of the assessment (17). Therefore, in this study, each guideline was appraised by 5 health care workers from different disciplines in order to improve the reliability of the assessment. The review and the appraisal of clinical guidelines took from June to late October 2018.
AGREE II is a standard tool for appraisal of clinical guidelines and helps health care providers to assess a clinical guideline before following its recommendations. The AGREE II tool is a general tool. It can be used to evaluate guidelines in regard to each aspect of the disease, including diagnostic, health promotion, treatment, and intervention guidelines. This tool is proper for assessing provided guidelines in paper or electronic format (software).
AGREE II tool consists of 23 key classified items in the 6 domains of Scope and Purpose (items 1 - 3), Stakeholder Involvement (items 4 - 6), Rigor of Development (items 7 - 14), Clarity of Presentation (items 15 - 17), Applicability (items 18 - 21), and Editorial Independence (items 22 - 23) besides 2 global rating items. Each section addresses one aspect of the guideline’s quality. After completing the 23 items, verified appraisers will answer the 2 global items. The overall assessment requires that the evaluators judge the overall quality of the clinical guideline according to the criteria considered in the appraisal process. They should also state whether they recommend its clinical implementation or not. In this tool, each item is rated and assigned a score of 1 (strongly disagree) to 7 (strongly agree). The score depends on the precision and the quality of the report and is independent for each of the 6 domains. The results were analyzed by standardizing the obtained score for each domain by the following formula, as well as calculating the overall mean score in each domain.
In addition, the scores of the domains should not be added as a single score. In the final evaluation, the overall quality of the guidelines and the appraisers’ final recommendation on their clinical implementation are also presented (18).
The agreement between appraisers was assessed using the intraclass correlation coefficient (ICC) and was defined as slight (0.20 ≥), fair (0.21 - 0.40), moderate (0.41 - 0.60), strong (0.61 - 0.80), or almost perfect agreement (0.81 - 1). According to the previous studies, the overall quality of each clinical guideline was calculated by using a threshold of 60% for the final score of each domain (19, 20). The guidelines with scores above 60% for 5 or more domains were defined as high quality guidelines, for 3 or 4 domains as moderate quality, and for 2 or fewer domains as poor quality. In addition, the overall quality has been measured as (mean ± standard deviation). Recommendations for the clinical implementation of the guidelines have been expressed as recommended, recommend with modifications, and not recommended.
According to the previous articles, the scores of the domains were classified as good (> 80%), acceptable (60% - 79%), low (40% - 59%), and very low (< 40%) (21, 22).
The AGREE II tool is widely used to evaluate the quality standard of clinical guidelines to assess methodological rigor and transparency of guideline development. This tool has been validated and tested for high reliability with detailed framework to assess the quality of guidelines in 6 standardized domains but also provides a methodological strategy for guideline development and content (18, 23, 24). Terrace validated this tool in a study as an international assessment tool for evaluating the quality of clinical guidelines in which 95% of evaluators considered the instrument useful for the appraisal of clinical guidelines. Moreover, the reliability of its mentioned parts was acceptable with the score of 64% - 88% (19). In Iran, Rashidian and Yousefi-Nooraie translated the AGREE tool into Persian, and its validity was confirmed by the joint cooperation committee of Tehran University of Medical Sciences and the Ministry of Health and Medical Education. In addition, the reliability of Persian version of the instrument and its English version was not found to be significantly different after being compared with each other (25).
3. Results
3.1. Selecting the Clinical Guidelines
After performing a systematic search, 51 clinical guidelines associated with search keywords were found, a total of 36 clinical guidelines remained after excluded repetitive titles. Then the abstracts of the found clinical guidelines were investigated by the research team and 21 of them were excluded. Next, the full texts of the 15 remaining guidelines were evaluated based on the inclusion and exclusion criteria, and 11 of them were removed. Finally, 4 guidelines on the administration of chemotherapy drugs were presented to the relevant experts in order to be appraised using AGREE II tool. The flow chart for selecting the clinical guidelines is shown in Figure 1.
Figure 1.
The flow chart of the identification process for guideline selection
3.2. The Characteristics of Clinical Guidelines
An Australian clinical guideline developed by the Clinical Oncology Society of Australia aims the safe prescribing, dispensing, providing, and administration of cancer treatment drugs (26). A Canadian guideline, developed by the Cancer Care Ontario Institute, seeks to provide guidance on the processes, technologies, and instruments of administering chemotherapy drugs (27). Another guideline is developed by the London Cancer Alliance from the UK, with the purpose of prescribing cytotoxic drugs (28). The other clinical guideline is from Ireland, created by Northern Ireland Cancer Network, which focuses on administering anti-cancer drugs (29). Only in one of the clinical guidelines, the conflict of interest has been mentioned (27). All 4 clinical guidelines are evidence-based (26-29). The updated versions of 2 guidelines are not available, but are current (27, 28). The updated version of one guideline is for 2017 (26) and another one has been planned for 2019 (29). Only one guideline has mentioned its funding source (27). The Characteristics of Clinical Guidelines are presented in Table 1.
Table 1.Characteristics of Clinical Guidelines
| Guideline Title | Publication Year | Country | Institution | Purpose | Conflicts of Interest | Evidence-Based Guideline | Update | Funding | Size of Complete Guideline (Pages) |
|---|---|---|---|---|---|---|---|---|---|
| 1. COSA guidelines for the safe prescribing, dispensing, and administration of systemic cancer therapy | 2010 | Australia | Clinical Oncological Society of Australia (COSA) | Safe prescribing, dispensing, and administration of cancer therapy | Not available | Yes | June 2017 | Not disclose | 247 |
| 2. Safe Administration of Systemic Cancer Therapy. Part 2: Administration of Chemotherapy and Management of Preventable Adverse Events | 2014 | Canada | Cancer Care Ontario | Provide guidance on processes, technologies, and devices for the administration of chemotherapy | Statement about conflicts of interest of group members present | Yes | Not available | Ontario Ministry of Health and Long-Term Care | 83 |
| 3. London Integrated Care Systems (ICSs), Guidelines for Safe Prescribing, Handling, and Administration of Systemic Anti-Cancer Therapy | 2015 | England | London Cancer Alliance | Administration of cytotoxic drugs | Not available | Yes | Not available | Not disclose | 98 |
| 4. Guidelines for the safe prescribing, handling and administration of Systemic Anti-Cancer Therapies | 2017 | Ireland | Northern Ireland Cancer Network | Administration of Systemic Anti-Cancer Therapies | Not available | Yes | May 2019 | Not disclose | 92 |
3.3. The Appraisal of the Clinical Guidelines Using AGREE II
A total of 4 clinical guidelines were reviewed and evaluated by 5 expert appraisers and analyzed after standardizing the scores obtained for each domain, as well as calculating the overall mean score of each. In addition, the overall quality of the clinical guidelines and the final opinions of appraisers about the clinical practice of the guidelines are presented in Table 2.
Table 2.Summary of the Average of Domain Scores of Guidelines According to AGREE IIa
| Guideline Title | Domain 1 (Scope and Purpose) | Domain 2 (Stakeholder Involvement) | Domain 3 (Rigor of Development) | Domain4 (Clarity of Presentation) | Domain 5 (Applicability) | Domain 6 (Editorial Independence) | Total Score Mean | Overall Quality | Overall Recommendation (to Use Guideline) |
|---|---|---|---|---|---|---|---|---|---|
| 1. COSA guidelines for the safe prescribing, dispensing and administration of systemic cancer therapy | 100% (good)b | 77.78% (acceptable)b | 86.25% (good)b | 96.67% (good)b | 80% (good)b | 61.67% (acceptable)b | 80.58% | High | Yes (4 reviewers) yes, with modifications (1 reviewer) |
| 2. Safe Administration of Systemic Cancer Therapy. Part 2: Administration of Chemotherapy and Management of Preventable Adverse Events | 95.56% (good)b | 93.33% (good)b | 81.25% (good)b | 88.89% (good)b | 89.17% (good)b | 98.33% (good)b | 80.87% | High | Yes (2 reviewers) yes, with modifications (3 reviewer) |
| 3. London Integrated Care Systems (ICSs), Guidelines for Safe Prescribing, Handling and, Administration of Systemic Anti-Cancer Therapy | 92.22% (good)b | 91.11% (good)b | 83.75% (good)b | 95.56% (good)b | 92.5% (good)b | 91.67% (good)b | 82.46% | High | Yes (3 reviewers) yes, with modifications (2 reviewer) |
| 4. Guidelines for the safe prescribing, handling, and administration of Systemic Anti-Cancer Therapies | 93.33% (good)b | 93.33% (good)b | 88.33% (good)b | 93.33% (good)b | 94.17% (good)b | 95% (good)b | 84.49% | High | Yes (4 reviewers) yes, with modifications (1 reviewer) |
| Total domain score; mean | 95.27% (good) | 88.88% (good) | 84.89 (good) | 93.61% (good) | 88.96% (good) | 86.66% (good) |
aDomain scores ≥ 80% = good, 60% - 79% = acceptable, 40% - 59% = low; < 40% = very low.
bTotal score of domain > 60%.
All 4 clinical guidelines had a high level of quality, with scores above 60% in at least 5 domains. Among them, the guidelines for the safe prescribing, handling, and administration of systemic anti-cancer therapies (29) earned the highest score (84.49%). The range of the scores of the domains falls between 61.67% (the lowest score which belonged to the domain 6 (editorial independence) in COSA guidelines for the safe prescribing, dispensing and administration of systemic cancer therapy (26)), and 100% (the highest which belonged to the domain1 (score and purpose) in COSA guidelines for safe prescribing, dispensing, and administration of systemic cancer therapy (26)). By comparing the mean scores of each domain among the guidelines, it is observed that domain 1 had obtained the highest scores (95.27%), while domain 3 (rigor of development) had the lowest mean score (84.89%).
In domain 1 with the highest mean score (95.27%), COSA guidelines for the safe prescribing, dispensing, and administration of systemic cancer therapy had, the highest score (26) (100% = good) and the London Integrated Care Systems (ICSs) guidelines for safe prescribing, handling, and administration of systemic anti-cancer therapy (28)had the lowest score (92.22% = good).
In domain 2 (stakeholder involvement) with a mean score of 88.88%, the 2nd and the 4th guidelines had the highest scores (93.3% = good) and guideline 1 had the lowest score (77.78% = acceptable). In domain 3 (rigor of development) with the lowest mean score (84.89%), the 4th guideline had the highest score (83.33% = good) and the 2nd guideline had the lowest score (81.25% = good). In domain 4 (clarity of presentation), with a mean score of 93.61%, guideline 1 had the highest score (96.67% = good) and guideline 2 had the lowest one (88.89% = good).
In domain 5 (applicability), with a mean score of 88.96%, the 4th guideline had the highest score (94.17% = good) and guideline 1 had the lowest one (80% = good). In domain 6 (editorial independence), with a mean score of 86.66%, guideline 2 had the highest score (98.33% = good) and guideline 1 had the lowest (61.67% = acceptable).
The highest agreement among appraisers was belonged to domain 1 (scope and purpose) (0.95 = very good) and the lowest one was belonged to domain 3 (rigor of development) (0.84 = very good).
4. Conclusions
The findings of this study showed the appraisal of common clinical guidelines for the administration of chemotherapy drugs using AGREE II instrument is acceptable and the overall quality of all four clinical guidelines is high. All domains had a score above 60%, and the scores of 6 domains in all guidelines varied from good to acceptable. These results may reflect the improving trend in clinical guidelines methodology over the past decade. The criteria used to appraise clinical guidelines have also been enhanced.
The obtained results are somehow steady considering the scores of the domains. In addition, domain 1 (scope and purpose) achieved good scores with the least variability in all guidelines. Furthermore, this domain has been well explained in all the evaluated guidelines.
Domain 1 (scope and purpose) and domain 4 (clarity of presentation) have obtained the highest scores, both of which with a score above 90%. This finding was consistent with the results of the previous studies on the appraisal of clinical guidelines using AGREE II tool, which covered various clinical topics (19, 21, 22, 30-33). Such high scores are a result of full explanations of these domains presented by clinical guidelines developers. These domains also consist of items such as the general objectives of the guidelines, the clarity and characteristics of recommendations, population and health questions, the clarity of health management items, and key recommendations that cannot be easily excluded.
Domain 3 (rigor of development) had the lowest score among the domains. This was in line with the study conducted by Gavriilidis et al. that they assessed the clinical guidelines for the resection of hepatocellular carcinoma (15). In other studies, this domain had also the lowest scores (14, 34), which may be a result of the lack of methodological expertise and insufficient information on the methods of developing clinical guidelines and the fact that two of these clinical guidelines lacked the updated versions. Therefore, it is recommended to use expert librarians in the process of developing clinical guidelines, to explicitly describe the information on search strategies, to explore databases, the process of selecting articles, the method of evidence quality assessment, and recommendations rating.
Four of the appraisers recommended the clinical implementation of COSA guidelines for the safe prescribing, dispensing and administration of systemic cancer therapy, as well as for safe prescribing, handling, and administration of systemic anti-cancer therapies. However, studies have shown that the implementation of evidence-based information in many healthcare settings remains a challenge, and it is often difficult to appraise the clinical applicability and performance of clinical guidelines. It takes almost 5 years for a clinical guideline to be clinically used as a routine. Even most of the approved guidelines are not fully pursued by staff (35, 36).
Most of the domains had high reliability. Therefore, the degree of agreement between the appraisers showed a strong correlation (0.84 - 0.95). The highest degree of agreement among the evaluators was found in domain 1 (scope and purpose) (0.95 = very good) and the lowest, in domain 3 (rigor of development) (0.84 = very good) this domain has 8 items and this is the most important reason for the ICC value being less compared to other domains.
Clinical guidelines are known as the main means of knowledge transfer, which require expertise and teamwork for their successful and sustainable implementation. The application and implementation of clinical guidelines is also dependent on the staff’s complete trust in their quality and credibility. Since nurses are responsible for the clinical administration of drugs and play a major role in chemotherapy, and due to the high overall quality of common clinical guidelines for prescribing chemotherapy drugs, their clinical use is recommended to nurses and all treatment team members who work in oncology wards in order to improve the safety of the patients and the treatment team and also to prevent complications during chemotherapy.
The present study had some limitations. We performed an inclusive, accurate and complete search although some clinical guidelines and their updated versions may have been overlooked. Another limitation was that only the guidelines in English language were appraised, due to the research team members did not fluent in other languages. The advantage of this study was using AGREE II tool, which has a high reliability and validity. The appraisers used instructions on how to use the tool, and all the guidelines were independently evaluated.
