1. Context
1.1. Quality Assessment of Randomized Controlled Trials
In randomized controlled trials (RCTs), the participants are allocated to two groups of intervention and control for the comparison of some outcomes between them (1). RCTs are considered the most valuable method to assess the efficacy of treatments whether or not the result of a comparison is statistically significant. Thus, it is important that they be conducted and reported with the highest possible quality so as to enable the readers to judge which results are internally valid and bias-free. Moreover, it is significant to differentiate between assessing the quality of a trial and the quality of its reporting. The quality of a trial is defined as the confidence that the design, conduct, and analysis of the trial have minimized or avoided biases in its treatment comparisons. This definition focuses on design quality. The quality of a report can be defined as the provision of information about the design, conduct, and analysis of the trial. A biased but well-reported trial can receive a high score of quality. Inversely, a well-conducted but weakly reported trial can receive a low score of quality (2).
Assessing the quality of RCTs is a relatively important development and is usually performed via three tools of component, checklist, and scale. The component tool evaluates some aspects of a trial, whereas checklists and scales involve lists of items for the assessment of its quality. Scales provide a numeric score of quality which can be formally included into a systematic review study (2).
Meta-analyses of RCTs are being published increasingly (2, 3); there is, therefore, great interest in the quality assessment of the RCTs in such analyses (4-9). If safety and efficacy results of a meta-analysis are significantly affected by the quality of the original trials, then its results may be less meaningful if quality is not assessed formally (10).
There has also been a rise in the number of journals publishing RCTs. The past decade (i.e. 2000 - 2014) has witnessed an increase in the number of Iranian RCTs published in PubMed journals. Searching PubMed using the “randomized controlled trial” keyword in the [Title/Abstract] reveals an increase over time, i.e. from 15 records between 2000 and 2005 to 92 records between 2006 and 2010 and 373 records between 2011 and 2014. Consequently, it is important for the authors, reviewers, and editors of journals to pay special attention to reporting quality assessment.
1.2. Nephrology Urology Monthly Journal
Nephrology Urology Monthly (NUM) is a clinical open-access Iranian journal indexed in PubMed with an average number of published articles per year of 63. NUM is an authoritative clinical source devoted to selected compilations of the latest worldwide and interdisciplinary research and reviews in the field of basic and clinical urology and nephrology. The journal’s main focus is on the efficacy in improving clinically relevant outcomes such as mortality, morbidity, and quality of life. NUM accepts all kinds of manuscripts and other scientific communications, including original manuscripts, meta-analyses and reviews, health economic papers, debates, and consensus statements of clinical relevance of nephrological and urological fields. The impact factor and rejection rate of this journal in 2012 were 0.3 and 32%, respectively.
1.3. Objectives
Our primary focus was to evaluate the reporting quality of the RCTs published in NUM. As a secondary aim, we examined whether there was a change over time in the reporting quality.
2. Evidence Acquisition
This study is a journal-based assessment. The inclusion criterion was trials on humans with control groups published from 2012 to 2014 in NUM. We extracted descriptive information such as the year and location of the study, number and gender of patients, condition under investigation (diseases), outcomes, intervention and comparison groups, and ethical approval.
We also completed a comprehensive quality assessment of each report using the Consolidated Standards of Reporting Trials (CONSORT) and Jadad methods. The CONSORT, first conducted in 1996, comprises randomization, allocation concealment, sample size, statistical analysis, blinding, and primary and secondary outcomes (1). The objective of the CONSORT is to provide guidance to authors and reviewers about how to improve the quality of reporting. The CONSORT has been revised and published as the CONSORT 2010 statement checklist. This checklist contains multiple modified items listed separately (25 items and 37 sub-items) about title, abstract, introduction, methods, results, discussion, and additional information. Each item is reported as “yes” if the author has reported it. The CONSORT 2010 Statement and its website (www.consort-statement.org) are helpful in enhancing the reporting quality of RCTs. In the present assessment, we employed only 7 important items of the 37 sub-items of the CONSORT 2010 Statement: 1a) title and abstract: identification as a randomized trial in the title; 4a) methods (participants): eligibility criteria for the participants; 4b) methods (participants): Settings and Locations where the data were collected; 5) methods (interventions): interventions for each group with sufficient details to allow replication, including how and when they were actually administered; 6a) methods (outcomes): completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed; 17a) results (outcomes and estimation): for each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval); and 23) Registration Number and the name of the trial registry.
The Jadad scale, comprised of 5 items of randomization, method of randomization, blinding, method of blinding, and dropouts and withdrawals, is used to assess quality. Each question contains a “yes” or “no” response option. In total, 5 points can be awarded, with higher points indicating superior quality. Although this scale is conducted primarily to assess the quality of the reports on pain studies, it has been used in other areas as well (8).
In the present study, two reviewers conducted all the assessments. We performed a prior training to evaluate the quality of the RCTs via the two methods. We compared the number of the checklist criteria (7 items of 37 sub-items) that were reported appropriately as specified in the CONSORT 2010 Statement checklist. We also assessed separately the items and the total quality score obtained from the Jadad scale. The mean of the number of appropriate reported items according to the CONSORT 2010 Statement checklist and the mean Jadad score over time were compared.
3. Results
Our database searching on PubMed identified 188 published articles from 2012 to 2014 in NUM. Of these, 170 non-trial or non-human trials were excluded and 22 trials remained. Four reports had no control group and failed to meet our eligibility criteria. Finally, 18 RCTs were selected (Figure 1). The descriptive and quality assessment information on the RCTs is depicted in Tables 1 and 2.
Figure 1.
Tracking and Enrollment of the Randomized Controlled Trials
Table 1.
Quality Assessment via the Jadad Score for the Randomized Controlled Trials Published in Nephrology Urology Monthly From 2012 to 2014 a
| Reference | Randomization | Blinding | Dropouts/Withdrawals | Total Score | ||
|---|---|---|---|---|---|---|
| Randomized | Method of Randomization | Blinding | Method of Blinding | |||
| (11) | 0, NR | 0, NR | 0, NR | 0, NR | 1 | 1 |
| (12) | 1 | 0, NR | 0, NR | 0, NR | 0, NA | 1 |
| (13) | 1 | 0, NR | 1 | 1 | 1 | 4 |
| (14) | 1 | 1 | 0, NR | 0, NR | 1 | 3 |
| (15) | 1 | 0, NR | 0, NR | 0, NR | 0, NA | 1 |
| (16) b | 1 | 0, NR | 1 | 1 | 0, NA | 3 |
| (17) | 1 | 0, NR | 0, NR | 0, NR | 1 | 2 |
| (18) b | 1 | 0, NR | 1 | 1 | 1 | 3 |
| (19) | 1 | 0, NR | 1 | 0, NR | 0, NA | 2 |
| (20) | 1 | 0, NR | 1 | 1 | 1 | 4 |
| (21) | 1 | 0, NR | 0, NR | 0, NR | 1 | 2 |
| (22) | 1 | 0, NR | 0, NR | 0, NR | 1 | 2 |
| (23) | 1 | 0, NR | 0, NR | 0, NR | 1 | 2 |
| (24) b | 1 | 1 | 1 | 1 | 1 | 5 |
| (25) b | 1 | 0, NR | 0, NR | 0, NR | 1 | 2 |
| (26) | 1 | 0, NR | 1 | 1 | 1 | 5 |
| (27) | 1 | 1 | 1 | 1 | 1 | 5 |
| (27) | 1 | 0, NR | 0, NR | 0, NR | 1 | 2 |
a Abbreviations: NA: not appropriate; and NR, not reported.
b First author or corresponding author was affiliated to Baqiyatallah university of medical sciences.
Table 2.
Randomized Controlled Trials Published in Nephrology Urology Monthly From 2012 to 2014 a
| Reference | Year | Country | Area | Sex | Sample Size | Interventions | Comparison Group | Outcome | Ethical Approval | Quality Score (Jadad) | CONSORT 2010 Statement Checklist Important Items No. b | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1a | 4a | 4b | 5 | 6a | 23 | 17a | No.Yes Items | |||||||||||
| (11) | 2012 | Egypt | Infertility | M | 20 + 20 | Loupe-assisted sub-inguinal varicocelectomy | Sub-inguinal varicocelectomy | Sperm parameters | N | 1 | N | Y | N | Y | Y | N | Y | 4 |
| (12) | 2012 | India | Analgesics for prostate biopsy | M | 20 + 20 + 20 | Diclofenac patch, periprostatic nerve block | No analgesic | Pain scores | N | 1 | N | Y | Y | Y | Y | N | Y | 5 |
| (13) | 2012 | Iran | Helicobacter pylori eradication in hemodialysis patients | MF | 20 + 17 | Omeprazole, amoxicillin, clarithromycin | Omeprazole, amoxicillin, azithromycin | UBT and the HBsAg test | N | 4 | Y | Y | Y | Y | Y | Y | Y | 7 |
| (14) | 2012 | Iran | Renal stone | MF | 20 + 20 | Prone supine percutaneous nephrolithotomy | supine percutaneous nephrolithotomy | Electrolyte, hemodynamic and metabolic changes | N | 3 | Y | Y | N | Y | Y | N | Y | 5 |
| (15) | 2012 | Iran | Renal stone | MF | 32 + 30 | Transurethral lithotripsy | Shock-wave lithotripsy | Renal stone management success | N | 1 | N | Y | N | Y | Y | N | Y | 4 |
| (16) c | 2012 | Iran | Renal transplantation complications | MF | 112 + 101 | Cyclosporine | Cyclosporine and calcium channel blockers | Gingival Index of McGaw and others, and Packet Index of Turesky–Gilmore–Glickman | Y | 3 | N | Y | N | Y | Y | N | Y | 4 |
| (17) | 2012 | Iran | Teaching hypertension | - | 28 + 23 | Cooperation lecture | Planned lecture | Long-term learning quality | N | 2 | N | Y | Y | NA | NA | N | NA | 2 |
| (18) c | 2013 | Iran | Hyperparathyroidism . | MF | 37 + 39 | 250 mg vitamin C | Placebo saline | Serum PTH | Y | 3 | Y | Y | Y | Y | Y | Y | Y | 7 |
| (19) | 2013 | Iran | Benign prostatic hyperplasia | M | 100+ 100 | Modified transurethral resection of the prostate | Standard Transurethral resection of the prostate | Symptom scoring, post-micturating residual volume, uroflowmetry urine examination, bacterial count and assessment for late complications, International Index of Erectile Function and quality of life, uroflowmetry test | N | 3 | N | Y | Y | Y | Y | N | Y | 5 |
| (20) | 2013 | Iran | Dialysis complications | MF | 90 + 90 | 80 mg/day Aspirin | Placebo | Catheter efficacy | N | 4 | N | Y | Y | Y | NA | N | Y | 4 |
| (21) | 2013 | Iran | Dialysis | 30 + 30 | Side-to-side (STS) anastomosis | End-to-side (ETS) anastomosis | Arteriovenous fistulae efficacy | N | 2 | Y | Y | Y | Y | Y | N | NA | 5 | |
| (22) | 2013 | Iran | Functional iron deficiency in patients under hemodialysis | MF | 20 + 20 | Intravenous iron | Intravenous ascorbic acid | Hb and iron metabolism indices | Y | 2 | Y | Y | Y | Y | Y | Y | N | 6 |
| (23) | 2014 | Iran | Voiding dysfunction | MF | 42 + 42 | Midazolam | - | Disorders of the urinary tract and voiding dysfunction | N | 2 | Y | N | Y | Y | NA | Y | Y | 5 |
| (24) c | 2014 | Iran | Hemodialysis | MF | 55 + 55 + 31 | Vitamin C supplementation | 1: Saline / 2: no intervention | CRP level | N | 5 | Y | Y | Y | Y | Y | Y | N | 6 |
| (25) c | 2014 | Iran | Urethral stricture | F | 86 | On-demand dilatation | Intermittent dilatation | Effectiveness of urethral stricture treatment | N | 2 | Y | Y | Y | Y | Y | Y | N | 6 |
| (26) | 2014 | Iran | Renal stone | MF | 52 + 50 | Tamsulosin | Placebo | Success rate of ureteroscopic lithotripsy | Y | 5 | Y | Y | Y | Y | Y | Y | N | 6 |
| (27) | 2014 | Iran | Diabetic nephropathy | MF | 30 + 30 | Spironolactone + placebo | Spironolactone + losartan | Diabetic nephropathy treatment success rate | Y | 5 | Y | Y | Y | Y | Y | Y | Y | 7 |
| (28) | 2014 | Malaysia | Hypovitaminosis D | MF | 25 + 25 | Oral calcitriol + calcium carbonate | Calcium carbonate alone | Renal function | Y | 2 | Y | Y | NA | Y | Y | Y | Y | 6 |
a Abbreviations: NA, not appropriate; N, no; and Y, yes.
b Mentioned items are as followings: 1a, title and abstract; identification as a randomized trial in the title; 4a, methods (participants); eligibility criteria for the participants; 4b, methods (participants); settings and locations where the data were collected; 5, methods (interventions); interventions for each group with sufficient details to allow replication, including how and when they were actually administered; 6a, methods (outcomes); completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed; 17a, results (outcomes and estimation); for each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval); and 23, registration number and name of trial registry.
c First author or corresponding author was affiliated to Baqiyatallah university of medical sciences.
Fifteen (83%) reports were submitted from Iran. In 11 (61%) studies, the first or the corresponding author was affiliated to Baqiyatallah university of medical sciences. There were 12 (66%) reports involving both male and female genders. In 7 (38.8%) reports, hemodialysis and renal transplantation patients were the target group. The measured outcome in 6 (33%) reports was lab data, and the intervention methods were surgery and drug therapy in 7 (38.8%) and 10 (55.5%) reports, respectively (Table 3).
Table 3.
Randomized Controlled Trials Published in Nephrology Urology Monthlya
| Status | Values a |
|---|---|
| Country | |
| Iran b | 4 (22.22) |
| Iran | 11 (61.11) |
| Other | 3 (16.67) |
| Outcome | |
| Lab | 6 (33.33) |
| Other (pain scores, success rate, quality of life, etc.) | 1 (5.56) |
| Gender, male and female | 12 (66.67) |
| Interventions | |
| Surgery | 7 (38.89) |
| Drug | 10 (55.56) |
| Teaching method | 1 (5.56) |
| Patients | |
| Infertility | 1 (5.56) |
| Prostate | 2 (11.11) |
| Renal stone | 3 (16.67) |
| Hemodialysis and renal transplantation | 7 (38.89) |
| Voiding dysfunction | 1 (5.56) |
| Teaching hypertension | 1 (5.56) |
| Urethral stricture | 1 (5.56) |
| Diabetic nephropathy | 1 (5.56) |
| Hypovitaminosis D | 1 (5.56) |
| Consort statement items c | |
| 1a | 11 (61.11) |
| 4a | 17 (94.44) |
| 4b | 12 (66.67) |
| 5 | 17 (94.44) |
| 6a | 15 (83.33) |
| 17a | 16 (88.89) |
| 23 | 5 (27.78) |
| Jadad score | |
| Randomization | 17 (94.4) |
| Method of randomization | 3 (16.6) |
| Blinding | 8 (44.4) |
| Method of blinding | 7 (38.8) |
| Dropout | 14 (77.7) |
a Data are presented as No. (%).
b First or corresponding author was affiliated to Baqiyatallah university of medical sciences.
c Mentioned items are as followings: 1a, title and abstract; identification as a randomized trial in the title; 4a, methods (participants); eligibility criteria for the participants; 4b, methods (participants); settings and locations where the data were collected; 5, methods (interventions); interventions for each group with sufficient details to allow replication, including how and when they were actually administered; 6a, methods (outcomes); completely defined pre-specified primary and secondary outcome measures, including how and when they were assessed; 17a, results (outcomes and estimation); for each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval); and 23, registration number and name of trial registry.
The items of 1a, 4a, 4b, 5, 6a, and 17a and 23 items of the CONSORT 2010 Statement checklist were reported in 61.1%, 94.4%, 66.6%, 94.4%, 83.3%, 88.8%, and 27.7% of the RCTs, correspondingly (Table 3). Accordingly, the best reports were related to the eligibility criteria for the participants and the pre-specified primary and secondary outcome measures. We defined the number of the appropriate reported items as a total score for this checklist. Three studies obtained full points (7 items). The quality of reporting (by CONSORT 2010 Statement) was 4.4, 5.4, and 6 in the years 2012, 2013, and 2014, respectively, showing a trend of improvement over time. The quality of reporting in the RCTs performed by Baqiyatallah university of medical sciences, other Iranian universities, and other countries was 3.25, 3, and 1.2, respectively. Thus, the highest reporting quality in the RCTs was achieved by Baqiyatallah university of medical sciences (Table 4).
The mean score of the Jadad scale was 2.72 ± 1.36 (54% of the maximum possible total score). The items of randomization, method of randomization, blinding, method of blinding, and dropouts/withdrawals of the Jadad scale were reported in 94.4%, 16.6%, 44.4%, 38.8%, and 77.7% of the RCTs, respectively. This score was 2.7, 2.8, and 3.5 in the years 2012, 2013, and 2014, respectively, which shows an increase in the quality of reporting over time (Table 4).
Table 4.
Randomized Controlled Trials Published in Nephrology Urology Monthly According to the Year of Publication
| Year | Country | ||||||
|---|---|---|---|---|---|---|---|
| 2012 | 2013 | 2014 | Total | Iran a | Iran | Other | |
| n | 7 | 5 | 6 | 18 | |||
| Jadad scale b | 2.7 | 2.8 | 3.5 | 2.72 | 3.25 | 3 | 1.3 |
| CONSORT 2010 Statement c | 4.42 | 5.4 | 6 | 5.27 | 5.75 | 5.09 | 5 |
a First or corresponding author was affiliated to Baqiyatallah university of medical science.
b Mean of score.
cMean of the number of appropriate reported items.
4. Discussion
Some potential biases in RCTs may never be minimized. For example, double blinding is questionable ethically and scientifically during surgical trials (3). One way to improve the reporting quality of RCTs is to use the CONSORT 2010 Statement. The journals that adhere to this checklist in their publication of RCTs have higher reporting quality than the ones that do not (8).
Proper reporting is required to generate unbiased comparison groups in controlled trials. Nonetheless, the reports of Iranian researchers in some journals usually provide inadequate or unacceptable information on critical points. Indeed, there are weaknesses in the reporting quality of the RCTs published by Iranian researchers in both Persian and English languages. A previous research suggested, however, that the quality of the non-English RCTs was similar to that in the English ones (9).
In 2012, (29), utilizing the CONSORT 2010 Statement, assessed 314 RCTs indexed in PubMed with affiliation to Tehran university of medical sciences and Iran university of medical sciences and showed that only the intervention used in the two groups was presented completely (100%) in the abstracts. In our assessment, the best report was related to defining eligibility criteria for the participants and defining intervention for each group. The other items in the Amanollahi et al. (29) study regarding the method of randomization, method of blinding, identification as a randomized trial in the title (item 1a), eligibility criteria for the participants (item 4a), and settings and locations where the data were collected (item 4b) were reported weakly and seen in 5.4%, 50.3%, 37.6%, 66.4%, and 19.4% of the reports, respectively. These percentages are comparable to those in our assessment. We showed that the Method of Randomization (16.6%), Eligibility criteria (94.4%), and Settings and Locations where the data were collected (66.6%) were reported with higher quality in the RCTs included in our study.
In 2013, Ghujazadeh et al. (30), drawing on the CONSORT 2010 Statement, assessed the reporting quality of 141 RCTs published by Iranian researchers in obstetrics and gynecology level-1 journals and showed that the weaknesses were chiefly in the Methods and Material, where out of 17 items, sample size determination, method of randomization, details of any kind of randomization (e.g. categorization and block formation), and blinding method accounted for the most notable shortcomings. The authors found that the method of randomization and method of blinding were reported in 39% and 50.4% of the RCTs; these percentages are higher than those in the present study. In 2014, Faizi et al. (31) in the quality assessment of the RCTs on applied psychotherapy for chronic pains in Iran showed that the mean score of Jadad was 1.53 ± 1.37, while this score had a higher mean in our assessment. The authors reported that the items (5 items) of the Jadad score were appropriately reported in 41.2%, 64.7%, 11.8%, 5.9%, and 29.4% of the RCTs, respectively, while these percentages in our study were 94.4%, 16.6%, 44.4%, 38.8%, and 77.7% of the RCTs, correspondingly, which indicates a higher quality of reports in our assessment with the exception of the method of randomization item.
Moher et al. (3) evaluated the reporting quality of RCTs on pediatric alternative medicine and showed that the studies achieved approximately 40% of their maximum possible total Jadad score. This result in our study was 54.4%.
According to the SCImago Journal Rank (SJR) indicator (http://www.scimagojr.com/journalsearch.php?q=19700201343&tip=sid&clean=0), our findings demonstrated that the rank of NUM in 2013 was higher than that in 2012 (SJR2012 = 0.118 vs. SJR2013 = 0.126). This shows that the reporting quality score can be considered a supplement to the ranking indices.
In general, meticulous reporting is required to generate an unbiased comparison of groups in RCTs. Our reporting quality assessment of the RCTs published in NUM, however, revealed low quality scores. It is possible that the Iranian investigators in this field have conducted few RCTs and therefore, not very experienced. Training courses for researchers, utilizing necessary reporting standard tools such as the CONSORT 2010 Statement by the editors of medical journals, and consultation with methodologists can improve the quality of RCTs.
