Abstract
Objective:
The quality of reporting Randomized Controlled Trials (RCTs) in the most prestigious scientific medical journals was investigated to show that what extent the items in the Consolidated Standard of Reporting Trials (CONSORT) 2010 checklist are addressed.Methods:
In this cross-sectional study five the most prestigious scientific medical journals that they had high impact factor (IF) were selected including: Lancet, New England Journal of Medicine, British Medical Journal, Journal of the American Medical Association, and Canadian Medical Association Journal. Ten randomized controlled trials in 2011 and 2012 were selected randomly from each journal.Results:
The percentage of items in the CONSORT checklist for each study was investigated. The total percentage of items addressed by these studies was 74.06 (95%CI: 71.21, 76.90).Conclusions:
We concluded that reporting of RCTs published in the top and the most prestigious scientific medical journals are not desirable and not enough yet.Keywords
CONSORT Randomized Controlled Trials Interventional Study Reporting
1. Introduction
Randomized controlled trials have an important role for identifying adverse effects of therapy that are relatively common and occur relatively soon after therapy has been initiated (1). This type of studies are considered the gold standard in clinical medicine and public health, for evaluating the efficacy and effectiveness a side effect of new therapeutic or preventive interventions (2, 3). RCTs, as the best research design, are widely accepted because they distribute both known and unknown confounding factors between intervention groups by the play of chance thereby minimizing the possibility that any treatment effect is due to bias or confounding, and providing the basis for valid statistical comparison .To assess a randomized trial accurately, readers and reviewers of published RCTs need complete, clearly written, and transparent information on a study’s methodology and findings (3). Also completing reporting concerning the design, conduct, analysis, and generalizability of the RCTs should be conveyed because these studies can have a powerful and immediate impact on patient care and accurate (4).
According to previous studies, randomized controlled trials with a low-quality reporting tend to overestimate the effect of the evaluated intervention (5, 6). To improve the reporting of interventional research such as, trials of herbal interventions, non-inferiority and equivalence, cluster randomized designs, reporting of abstracts, data on harms, and of non-pharmacological interventions, a group of experts such as epidemiological researchers, statisticians, methodologist, and clinical researchers developed a checklist of items known as CONsolidated Standard of Reporting Trials (CONSORT) 2010 statement (7). Previous surveys suggest that the use of CONSORT items is associated with improvements in the quality of RCTs being reported (8, 9). The original CONSORT statement first published in 1996 (4) and then revised in 2001, 2007 and published its latest revision of the statement in 2010 (10, 11).
The CONSORT statement supports researcher to improve the reporting of different types of health research and to improve the quality of research used in decision-making in healthcare. On the other hand, it is useful for editors in considering such manuscript for publication and critical appraisal (10, 12). Because results of randomized controlled trials are enough for assessing the strengths and weaknesses of the evidence as well as interpretation of RCT results becomes difficult, if not impossible, with inadequate reports causing biased results to receive false reliability (13).
The aim of present study was to determine whether the use of the CONSORT 2010 statement is associated with improved quality of reporting of RCTs published in journals or not, also, to indicate to what extent the items in CONSORT 2010 checklist are noticed by both authors and publishers.
2. Materials and Methods
In this cross-sectional study, we selected five most prestigious scientific medical journals with high impact factor. All of these journals were indexed in international databases. These journals included Lancet (IF: 38.278), New England Journal of Medicine (N Engl J Med) (IF: 53.298), British Medical Journal (BMJ) (IF: 14.093), Journal of the American Medical Association (JAMA) (IF: 30.026) and Canadian Medical Association Journal (Can Med Assoc J) (IF: 8.217). For this assessment, we randomly selected 10 randomized controlled trials that published in each of the five most prestigious scientific medical journals in 2011 and 2012. After doing sorting process for the articles from newly published to the old, respectively, we looked for RCTs to find 10 eligible articles.
On the whole, we could enroll 50 randomized controlled trials from five prestigious scientific medical journals and then, randomly assigned them to three reviewers. The reviewers were independent for making decisions on the number of each item that exist in CONSORT checklist, which were addressed in the selected studies. In this study didn’t set up blinding process to the names of the studies’ authors and journals.
The statement provides guidance for reporting three type of randomized controlled trials with a focus on individually randomized, two groups, parallel trials (3). The CONSORT statement included a checklist of 25 items. Three answers including: “Not applicable”, “Reported” and “Not reported” were assigned for each item indicating whether the author had reported it. The CONSORT score of each RCT was calculated by adding the correctly reported domains of the CONSORT checklist and the percentage of each item addressed in the selected studies, was estimated. The total percentage for all items was reported separately. All domains had the same weight.
All analyses were performed using statistical software Stata 11 (Stata Corp, College Station, TX, USA).
3. Results
In this cross-sectional study, 50 randomized controlled trials were selected from the five prestigious scientific medical journals including: 10 of Lancet, 10 of New England Journal of Medicine, 10 of British Medical Journal, 10 of Journal of the American Medical Association and 10 of Canadian Medical Association Journal.
The percentage of each items, addressed by these RCTs are summarized in Table 1. Sample size (44%), blinding (44%) and follow up (44%) items have been poorly reported.
The items were not applicable in 8.23% (95% CI: 5.38, 11.07), were not reported in 17.71% (95% CI: 14.86, 20.55), and were reported in 74.06% (95% CI: 71.21, 76.90) of the RCTs.
Percentage of Items in Consorta2010 Checklist Which Were Addressed in Reports of a Randomized Controlled Trials Published in Five Top Scientific Medical Journals In 2011 And 2012
| Item | Recommendation | Not applicable | Not reported | Reported | ||
|---|---|---|---|---|---|---|
| Title and abstract | ||||||
| 1a | Identification as a randomized trial in the title\abstract | 0(0.0) | 0(0.0) | 50(100.0) | ||
| 1b | Structured summary of trial design, methods, results, and conclusions | 0(0.0) | 0(0.0) | 50(100.0) | ||
| Introduction | ||||||
| 2a | Scientific background and explanation of rationale | 0(0.0) | 4(8.0) | 46(92.0) | ||
| 2b | Specific objectives or hypotheses | 0(0.0) | 2(4.0) | 48(96.0) | ||
| Methods | ||||||
| 3a | Description of trial design (such as parallel, factorial) including allocation ratio | 0(0.0) | 3(6.0) | 47(94.0) | ||
| 3b | Important changes to methods after trial commencement (such as eligibility criteria), with reasons | 3(6.0) | 13(26.0) | 34(68.0) | ||
| 4a | Eligibility criteria for participants | 0(0.0) | 7(14.0) | |||
| 4b | Settings and locations where the data were collected | 0(0.0) | 2(4.0) | 48(96.0) | ||
| 5 | The interventions for each group including how and when they were actually administered | 0(0.0) | 0(0.0) | 50(100.0) | ||
| 6a | Completely defined pre-specified primary and secondary outcome measures | 2(4.0) | 5(10.0) | 43(86.0) | ||
| 6b | Any changes to trial outcomes after the trial commenced, with reasons | 39(78.0) | 6(12.0) | 5(10.0) | ||
| 7a | How sample size was determined | 12(24.0) | 7(14.0) | 31(62.0) | ||
| 7b | When applicable, explanation of any interim analyses and stopping guidelines | 21(42) | 6(12.0) | 23(46.0) | ||
| 8a | Method used to generate the random allocation sequence | 0(0.0) | 4(8.0) | 46(92.0) | ||
| 8b | Type of randomization; details of any restriction (such as blocking and block size) | 0(0.0) | 9(18.0) | 41(82.0) | ||
| 9 | Mechanism used to implement the random allocation sequence | 0(0.0) | 6(12.0) | 44(88.0) | ||
| 10 | Who generated the random allocation sequence, enrolled, and assigned participants to interventions | 0(0.0) | 8(16.0) | 42(84.0) | ||
| 11a | If done, who was blinded after assignment to interventions | 5(10.0) | 23(46.0) | 22(44.0) | ||
| 11b | If relevant, description of the similarity of interventions | - | 21(42.0) | 29(58.0) | ||
| 12a | Statistical methods used to compare groups | 0(0.0) | 0(0.0) | 50(100.0) | ||
| 12b | Methods for additional analyses, such as subgroup analyses and adjusted analyses | 26(52.0) | 18(36.0) | 6(12.0) | ||
| Results | ||||||
| 13a | For each group, number of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome | 0(0.0) | 0(0.0) | 50(100.0) | ||
| 13b | For each group, losses and exclusions after randomization | 0(0.0) | 1(2.0) | 49(98.0) | ||
| 14a | Dates defining the periods of recruitment and follow-up | 12(24.0) | 16(32.0) | 22(44.0) | ||
| 14b | Why the trial ended or was stopped | 13(26.0) | 31(62.0) | |||
| 15 | A table showing baseline data for each group | 0(0.0) | 0(0.0) | 50(100.0) | ||
| 16 | For each group, number of participants that analyzed | 0(0.0) | 0(0.0) | 50(100.0) | ||
| 17a | results for each group, and the estimated effect size and its precision | 0(0.0) | 0(0.0) | 50(100.0) | ||
| 17b | presentation of both absolute and relative effect sizes | 0(0.0) | 16(32.0) | 34(68.0) | ||
| 18 | Results of any other analyses performed | 18(36.0) | 23(46.0) | 9(18.0) | ||
| 19 | All important harms or unintended effects in each group | 0(0.0) | 43(86.0) | 7(14.0) | ||
| Discussion | ||||||
| 20 | Trial limitations | |||||
| 21 | Generalisability of the trial findings | 0(0.0) | 0(0.0) | 50(100.0) | ||
| 22 | Interpretation consistent with results | 0(0.0) | 0(0.0) | 50(100.0) | ||
| Other information | ||||||
| 23 | Registration number and name of trial registry | 0(0.0) | 12(24) | 38(76.0) | ||
| 24 | Where the full trial protocol can be accessed | 0(0.0) | 42(84.0) | 8(16.0) | ||
| 25 | Sources of funding and other support | 0(0.0) | 6(12.0) | 44(88.0) | ||
| 1-25 | Total | (8.23) | (17.71) | (74.06) | ||
3. Discussion
We carried out a descriptive cross-sectional study to assess the quality of reporting of randomized controlled trials from five high-impact general medical journals published in the year 2011 and 2012, that is, after the release of the CONSORT statement in 2010 (10).
Randomized controlled trials are the gold standard (2, 3) intervention for clinical knowledge, and the reporting of RCTs are often ambiguous and incomplete and are not always well reported (14, 15). The CONSORT 2010 statement seeks recommendations for reporting RCTs evidence and also, to improve the reporting of RCTs in journal articles (10). As a major result, two year after revising in CONSORT 2010 statement, almost 74.06% of the items in this checklist were addressed by RCTs published in five top scientific medical journals. Our results are consistent with many studies which have assessed the quality of reporting of RCTs published in medical journals (14, 16-19). All of these studies showed there is poor quality of reporting in RCTs. Unfortunately, there is same results based on previous reporting of observational studies (20, 21).
Turner et al conducted a systematic review in order to assess whether journal endorsement of the 1996 and 2001 CONSORT checklists influences the completeness of reporting of RCTs published in medical journals. They reported that evidence has accumulated to suggest that the reporting of RCTs remains sub-optimal (22). Another systematic review has reported similar result. This study reported from 72 applicable checklist items, 42% were generally reported adequately and 25% inadequately (23).
Ziogas et al assessed the reporting quality of published RCTs concerning myeloid hematologic malignancies according to the CONSORT statement in 2009. They reported that only 13 of the 24 items of CONSORT statement were addressed in 75% or more of the studies (24).
A study which was conducted in 2011 in Germany reported that the CONSORT scores increased from 66.7 ± 12.5% in the pre-CONSORT period to 87.4 ± 6.9% in the post-CONSORT era (25).
Several interventional studies have documented how inadequate randomizations, double blinding, concealment of allocation, and differential losses to follow-up or dropouts per treatment group may affect the observed treatment effects (9, 26). Accordingly our study has reported a higher percentage for mentioned items.
We found that 17.71% of domains of the CONSORT checklist were “Not reported”. This may reflect the inadequate reporting of study methods and procedures. In some cases the assessment of “Not reported” resulted from poor reporting at the individual study level (25). While reporting may improve for more recent studies as journals and authors adopt the CONSORT guidelines (11). According to the this statement, Rezaeian et al assessed the quality of reporting in cohort studies published in journals with high and low impact factor and its association with journals impact factor (IF). They reported that a significant difference was found in the percentage of STROBE items reported (89.5% vs. 81.7%, respectively, P < 0.001) in the journals with high IF compared to the journals with low IF (27).
Abstract reporting is one of the important items to space restriction and journal formats, which may lead to difference between both of full paper results and abstract results. However, a previous study showed that with a word limit of 250–300 words, the checklist items can easily be incorporated (28). In this study the items related to abstract and title were mentioned in 100%.
Sinha et al assessed the quality of reporting of trial methodology and adverse events in a sample of general surgical RCTs published in high-quality surgical journals using the criteria specified in the CONSORT statements in 2009. This study confirmed our finding of a high rate of inadequate reporting of adverse effects (29).
This result obtained from the RCTs that published in the most prestigious scientific medical journals that they generally accept the well-written and well-done studies. On the other hand, if this survey had been set up to assess reporting RCTs that published in less prestigious scientific medical journals, the estimated result would be much poorer and undesirable than what we have been estimate in the present survey. As compared to other studies (30), reporting of CONSORT checklist items is higher but still suboptimal in the current study.
The present survey had a number of limitations. First, randomly selection of RCTs from a five prestigious scientific medical journals may increase the possibility of occurrence of selection bias. Second, in the present survey, we limited the number of RCTs for assessment and this issue may increase the possibility of occurrence of random error (Chance).
We concluded that reporting of RCTs published in the top and the most prestigious scientific medical journals are not desirable enough for assessment of the rigor of RCTs and we had expect that this percentage be more than this estimated in the present study. A similar survey which was carried out previously obtained same results (31, 32). Therefore we have to focus on both authors’ and editors’ when reporting and/or reviewing the reports of RCTs. And both of them should reinforce the use of the CONSORT 2010 Statement in the reporting and reviewing of trials.
4. Conclusions
We concluded that reporting of RCTs published in the top and the most prestigious scientific medical journals are not desirable and not enough yet. Therefore, journal editors, reviewers and authors should be encouraged to adhere to the CONSORT statement in order to ensure high-quality trials. Researchers also need to design research with full understanding of the CONSORT reporting guidelines and full consideration of items whose reporting quality is low.
References
-
1.
Greenland S. Clinial Epidemiology. In: Rothman kJ, Greenland S, Last TL, editors. Modern Epidemiology. Philadelphia: Lippincott Williams & Wilkins; 2008. p. 646-8.
-
2.
Gordis L. Assessing the Efficacy of Preventive and Therapeutic Measures: Randomized Trials. In: 4th, editor. Epidemiology. Philadelphia: Saunders; 2008.
-
3.
Schulz KF, Altman DG, Moher D, CONSORT G. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 2010;18:8.
-
4.
Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996;276:637-9.
-
5.
Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet. 1998;352(9128):609-13.
-
6.
Savović J, Jones H, Altman D, Harris R, Jűni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies. Health Technol Assess. 2012;16(35):1-82.
-
7.
Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390. doi: 10.1136.
-
8.
Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C, et al. Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust. 2006;185(5):263-7.
-
9.
Jüni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ. 2001;323(7303):42-6.
-
10.
Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. Int J Surg. 2012;10(1):28-55.
-
11.
Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869. doi: 10.1136.
-
12.
Turner L, Shamseer L, Altman DG, Schulz KF, Moher D. Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane reviewa. Syst Rev. 2012;1(1):60 [Epub ahead of print].
-
13.
Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. Ann Intern Med. 2001;134(8):657-62.
-
14.
Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005;365:1159-62.
-
15.
Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne DG, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134:663-94.
-
16.
Lai TY, Wong VW, Lam RF, Cheng AC, Lam DS, Leung GM. Quality of reporting of key methodological items of randomized controlled trials in clinical ophthalmic journals. Ophthalmic Epidemiol. 2007;14(6):390-8.
-
17.
Paul JE, Arya A, Hurlburt L, Cheng J, Thabane L, Tidy A, et al. Femoral nerve block improves analgesia outcomes after total knee arthroplasty: a meta-analysis of randomized controlled trials. Anesthesiology. 2010;113(5):1144-62.
-
18.
Dong W, Li C, Xiang C, Shi Z, Tian W. Quality assessment on reports of randomized controlled trials of oral and maxillofacial surgery in China. Hua Xi Kou Qiang Yi Xue Za Zhi. 2012;30(5):505-8.
-
19.
Richards D. Quality of reporting randomised controlled trials in dental public health. Evid Based Dent. 2011;12(2):54. doi: 10.1038.
-
20.
Poorolajal J, Cheraghi Z, Doosti Irani A, Rezaeian S. Quality of Cohort Studies Reporting Post the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement. Epidemiology and Health. 2011;33. https://doi.org/10.4178/epih/e2011005.
-
21.
Kim MR, Kim MY, Kim SY, Hwang IH, Yoon YJ. The quality of reporting of cohort, case-control studies in the korean journal of family medicine. Korean J Fam Med. 2012;33(2):79-88.
-
22.
Turner L, Shamseer L, Altman DG, Weeks L, Peters J, Kober T, et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev. 2012;11:MR000030. doi: 10.1002/14651858. https://doi.org/10.1002/14651858.
-
23.
Strech D, Soltmann B, Weikert B, Bauer M, Pfennig A. Quality of reporting of randomized controlled trials of pharmacologic treatment of bipolar disorders: a systematic review. J Clin Psychiatry. 2011;72(9):1214-21.
-
24.
Ziogas DC, Zintzaras E. Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement. Ann Epidemiol. 2009;19(7):494-500.
-
25.
Herdan A, Roth R, Grass D, Klimek M, Will S, Schauf B, et al. Improvement of quality of reporting in randomised controlled trials to prevent hypotension after spinal anaesthesia for caesarean section. Gynecol Surg. 2011;8(2):121-7.
-
26.
Schulz KF. Assessing allocation concealment and blinding in randomised controlled trials: why bother? Evid Based Nurs. 2001;4(1):4-6.
-
27.
Rezaeian S. Impact Factor and its Association with Quality of Cohort Studies Reporting. MedicReS World Congress. Vienna. 63 p.
-
28.
Hopewell S, Clarke M, Moher D, Wager E, Middleton P, Altman DG, et al. CONSORT for reporting randomised trials in journal and conference abstracts. Lancet. 2008;371(9609):281-3.
-
29.
Sinha S, Sinha S, Ashby E, Jayaram R, Grocott MPW. Quality of reporting in randomized trials published in high-quality surgical journals. J Am Coll Surg. 2009;209(5):565-71.
-
30.
Süt N, Senocak M, Uysal O, Köksalan H. Assessing the quality of randomized controlled trials from two leading cancer journals using the CONSORT statement. Hematol Oncol Stem Cell Ther. 2008;1(1):38-43.
-
31.
Borg Debono V, Zhang S, Ye C, Paul J, Arya A, Hurlburt L, et al. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiol. 2012;12:13. https://doi.org/10.1186/1471-2253-12-13.
-
32.
Ma B, Ke FY, Chen ZM, Qi GQ, Li HM, Liu WJ, et al. Does the reporting of randomized clinical trials published in Chinese pediatrics journals improve after the CONSORT Statement is adopted? Contemp Clin Trials. 2012;33(5):889-94.