1. Background
Evidence-based medicine with clear methodology guides clinical practitioners on how to access scientific evidence, make it available, and ultimately make the most appropriate decision for treatment. In addition, the number of citations is increasingly being used as performance indicators in research policy and research evaluation systems. Research has shown that the impact of clinical research on decision making by specialists on the patient’s bedside is directly related to the type and design of the study. Accordingly, the pyramid of evidence is based on the type and quality of the study method. A well-designed randomized controlled clinical trial (RCT) that can confirm the research hypothesis has long been recognized as the strongest method and technique for gathering valid evidence on the effects of therapeutic interventions (1). According to the Levels of Evidence (LoE) by Oxford Centre for Evidence-based Medicine, medical studies are divided into five levels based on the areas of treatment, diagnosis, prognosis and health economics. The highest level of evidence in the field of therapy, prevention, aetiology and harm is related to randomized clinical trials and their systematic reviews and meta-analysis. At the second level, there are cohort studies. The third level relates to case-control studies and their systematic review and the fourth level relates to case-series and poor quality cohort and case-control studies. Finally, the fifth level includes the expert opinion without an explicit critical appraisal or based on physiology, bench research (2).
Much of the new information reaches the scientific community through the publication of research findings in journals. The impact of a published article on scientists and experts can be accessed on the basis of its use (3). The number of citations of an article over a given period of time may indicate the importance of its findings by active members of the medical community (4). Whether or not to cite research findings based on study characteristics and whether their results are positive or negative is called citation bias (5). Understanding the citation bias in articles is important because this type of bias can influence the dissemination of information in primary and secondary literature and the attitude of the medical community. In some studies examining the citation of scientific articles in the medical and health sciences, higher levels of evidence have been associated with positive citation bias (4, 6-8). Jamshidi Orak et al. examined the level of evidence of theses and dissertations of nursing dissertations of Iran University of Medical Sciences from 1991 to 2010. Their study showed that 60.7% of Master’s theses and 92.6% of doctoral theses were in the lowest level of evidence (descriptive and qualitative studies) (9).
Giuffrida and Brown examined the relationship between levels of evidence and the number of citations in veterinary articles over a five-year period. They extracted the scientific and non-scientific properties of original research articles published in five clinical veterinary journals in 2004. The papers were ranked based on one of the levels of evidence-based classification schemes and their citations were extracted over a five-year period. Researchers reported no correlation between the number of citations and the levels of evidence or method of conducting the studies. In this study, it was reported that veterinary specialties and the position of a journal can influence the rate of citations (10).
Amiri et al. in a systematic study examined all articles published in the five spinal research journals in 2010 in terms of the relationship between the level of evidence of the articles and the journals’ impact factor (JIF). From 1,658 published articles, they categorized 703 clinical papers based on the table of Oxford levels of evidence between levels I and IV. 66.3% of articles were in the field of treatment, 14.9% were prognostic, 11.9% were diagnosed, 5.3% were differential diagnosis and 1.6% were economic/decision analysis. 4.7% of articles were classified as level I, 23.2% were level II, 12.5% were level III and 59.6% were classified as level IV. 71.8% of treatment articles were related to level IV. There was no significant difference in the number of articles in levels I and II in five spine journals. Researchers reported that journals with a higher JIF had a higher proportion of evidence-level papers I & II (11).
Lopez et al. investigated the predictors of citation rates in plastic surgery articles over a five-year period. The subject area, the conflict of interest, the number of authors, and the journal influenced the number of citations. The study showed that high levels of evidence or method of study were not correlated with the number of citations (12).
The inclusion of evidence levels in clinical papers helps develop evidence-based medicine and facilitates the selection of appropriate articles for clinical decision making for specialists. In evidence-based medicine, articles with high levels of evidence have less bias and error, and are, therefore, more suitable for clinical decision making and are more commonly used. Studies have reported inconsistent results regarding the impact of the level of evidence and the number of citations to various journal articles or issues.
This study aims to evaluate factors affecting the number of citations to clinical therapeutic articles mentioning the level of evidence, which indexed in PubMed and Web of Science databases.
2. Methods
This study is a cross-sectional study that deploys bibliometric techniques. The databases used in this study were PubMed, Web of Science, and the Journal Citation Reports. The searches were made in July 2019. Using the Advanced Search Tool of PubMed Database, a suitable query was formulated to retrieve research articles related to the treatment of human diseases published from 2011 to 2013 that mentioned in their abstract the evidence level (Table 1).
Table 1.Stages of the Retrieval of Clinical Articles Related to the Treatment of Human Diseases from the PubMed Database that Have Specified the Level of Clinical Evidence in the Abstract
| Search | Query | Items found |
|---|---|---|
| #1 | Search (“therapeutic uses”[MeSH Terms] OR “therapeutics”[MeSH Terms] OR “therapeutic use”[MeSH Subheading] OR “therapy”[MeSH Subheading] OR “treatment outcome”[MeSH Terms] OR “Therapeutic”[Title/Abstract]) | 10424181 |
| #2 | Search (“adaptive clinical trial”[Publication Type] OR “clinical study”[Publication Type] OR “clinical trial protocol”[Publication Type] OR “clinical trial”[Publication Type] OR ... OR “comparative study”[Publication Type] OR “controlled clinical trial”[Publication Type] OR “meta analysis”[Publication Type] OR “multicenter study”[Publication Type] OR “observational study”[Publication Type] OR “randomized controlled trial”[Publication Type] OR “systematic review”[Publication Type] OR “systematic review”[Title/Abstract] OR “meta analysis as topic”[MeSH Terms] OR “meta analysis”[Title/Abstract] OR “randomized controlled trials as topic”[MeSH Terms] OR “double blind method”[MeSH Terms] OR “cohort studies”[MeSH Terms] OR “follow up studies”[MeSH Terms] OR “prospective studies”[MeSH Terms] OR “retrospective studies”[MeSH Terms] OR “case control studies”[MeSH Terms] OR “case series”[Title/Abstract]) | 4465298 |
| #3 | Search (#1 AND #2) | 2595999 |
| #4 | Search (#3) AND (“humans”[MeSH Terms] AND hasabstract[text]) | 2138459 |
| #5 | Search (#4) AND (“2011/01/01”[EPDAT] : “2013/12/31”[EPDAT]) | 301745 |
| #6 | Search (#5) AND (“level of evidence”[title/abstract] OR “levels of evidence”[title/abstract] OR “ebm level”[title/abstract] OR “level of proof”[title/abstract]) | 5145 |
| #7 | Screening of (#6) | 3904 |
| #8 | Search (#7) in Web of Science | 3832 |
| #9 | Screening of (#8) | 3683 |
After the retrieval, abstracts were reviewed to select the articles related to treatment that specified in their abstracts the level of evidence. Then a search query was developed to retrieve bibliographic and citation information of the selected articles from the Web of Science database. Level of evidence, number of references, number of authors, number of title words, number of pages, article type, and subject category were extracted from the information fields using formulas in MS Excel. The study design was determined based on the information contained in the abstract and other information sections of the articles. The electronic publication date of the articles was considered from the beginning of 2011 until the end of 2013, and the number of citations received was extracted from the Web of Science database over a five-year period following their publication. JIF and quartile raking of the journals were also extracted from the Journal Citation Reports database based on the year before publication.
The quantitative data of the study were analysed in two parts: descriptive and inferential statistics. The descriptive section included tables and graphs of absolute and relative frequency distribution, mean and standard deviation, median and Interquartile range. Independent samples T-test, one-way ANOVA and Pearson correlation test were used to analyse the relationship between variables by SPSS 16 software. Stepwise linear regression analysis was used to determine the factors affecting the number of five-year citations and the effect and contribution of each of the independent variables in explaining the number of five-year citations received was evaluated. VIF index was used to determine the effect of linearity between the independent variables. Adjusted R2 index was used to determine the variance explained by the independent variables. This method also adjusted the effect of the number of variables entered in the regression model. All interpretations are reported in a linear regression model based on a logarithmic criterion. P value < 0.05 was considered significant.
3. Results
Between 2011 and 2013, 3,683 clinical articles on the treatment of human diseases that specified the level of evidence in their abstract were published and indexed in PubMed and Web of Science databases. The number of articles based on the year of electronic publication from 2011 to 2013 respectively was 907 (24.6%), 1214 (33%) and 1562 (42.4%). Descriptive indicators of variables including number of authors, number of title words, number of references, page counts of articles, JIF and 5-year citations count are reported in Table 2.
Table 2.Descriptive Statistics of Variables of Articles in the Field of Treatment (N = 3683)
| Variables | Mean ± SD | Median | [Q1 - Q3] | Min | Max |
|---|---|---|---|---|---|
| Author count | 5.31 ± 2.27 | 5 | [4 - 6] | 1 | 27 |
| Title words count | 14.15 ± 4.69 | 14 | [11 - 17] | 2 | 36 |
| References count | 29.77 ± 13.64 | 28 | [20 - 37] | 1 | 146 |
| Pages count | 7.44 ± 2.24 | 7 | [6 - 9] | 2 | 23 |
| IF 2 years | 2.50 ± 1.08 | 2.21 | [1.57 - 3.31] | 0.30 | 8.25 |
| IF 2 years (2018) | 3.82 ± 1.63 | 3.61 | [2.37 - 4.87] | 0.48 | 9.80 |
| 5 years citation count | 19.15 ± 20.00 | 13 | [7 - 24] | 0 | 215 |
One article was in French and the rest were in English language. More than 95% of articles were journal articles (3,297) and conference papers (234). The articles were published in 80 different journals. About 22% of articles were published in the American Journal of Sports Medicine; and Knee Surgery, Sports Traumatology, Arthroscopy. More than half of the articles were published in top 25% of journals on each topic and the share of Lippincott Williams & Wilkins was greater than Elsevier, Springer and other publishers.
A total of 14,237 authors from over 3,000 scientific institutions contributed to the publication of these articles. The University of California with 120 articles, the University of Texas with 117 articles and Harvard University with 111 articles contributed the most. The authors were from 70 countries, with the United States accounting for more than 40% (1,485) of the articles. There were 14 articles in which Iranian authors contributed.
About 64% of the articles were orthopedic, according to the first subject group on the Web of Science. Frequency distribution of the articles by the level of evidence showed that about 50% of the articles are classified in the fourth level of evidence and the average number of five-year citations decreased from high levels of evidence to low levels of evidence (Table 3). Investigating JIF in the year prior to the publication of the article shows that articles with high-level evidence were published in journals with a higher average JIF.
Table 3.Comparison of the Mean Number of Citations Received by Articles According to Different Variables Based on Logarithmic Normalized Data (N = 3635)a
| Variables | Frequency (%) | Citation, Mean ± SD | P Value |
|---|---|---|---|
| Published year | < 0.001 | ||
| 2011 | 899 (24.73) | 2.65A ± 0.95 | |
| 2012 | 1200 (33.01) | 2.48B ± 0.96 | |
| 2013 | 1536 (42.26) | 2.53B ± 0.93 | |
| Subject categories | < 0.001 | ||
| Orthopedics | 2328 (64.04) | 2.60AB ± 0.94 | |
| Surgery | 531 (14.61) | 2.34B ± 0.99 | |
| Obstetrics & gynecology | 195 (5.36) | 2.80A ± 0.88 | |
| Critical care medicine | 180 (4.95) | 2.42B ± 0.95 | |
| Clinical neurology | 151 (4.15) | 2.57AB ± 0.94 | |
| Urology & nephrology | 123 (3.38) | 2.57AB ± 0.78 | |
| Medicine, research & experimental | 85 (2.34) | 2.30B ± 0.85 | |
| Others | 42 (1.16) | 1.88C ± 1.01 | |
| Level of Evidence | < 0.001 | ||
| I | 288 (7.92) | 3.01A ± 0.93 | |
| II | 618 (17.00) | 2.72B ± 0.94 | |
| III | 858 (23.60) | 2.61BC ± 0.87 | |
| IV | 1799 (49.49) | 2.41C ± 0.94 | |
| V | 72 (1.98) | 1.91D ± 1.15 | |
| Document types | < 0.001 | ||
| Systematic review & meta-analysis | 210 (5.78) | 3.13A ± 0.78 | |
| RCT & clinical trial | 691 (19.01) | 2.73B ± 1.01 | |
| Cohort study | 531 (14.61) | 2.67BC ± 0.93 | |
| Case-control study | 161 (4.43) | 2.45D ± 0.92 | |
| Case series study | 861 (23.69) | 2.49CD ± 0.96 | |
| Other | 1181 (32.49) | 2.33D ± 0.87 | |
| Corresponding author address | < 0.001 | ||
| USA | 1474 (40.55) | 2.65AB ± 0.93 | |
| South Korea | 264 (7.26) | 2.46BC ± 0.89 | |
| France | 225 (6.19) | 2.30C ± 0.88 | |
| China | 168 (4.62) | 2.26C ± 0.98 | |
| Germany | 162 (4.46) | 2.71B ± 0.78 | |
| Italy | 142 (3.91) | 2.90A ± 0.93 | |
| Other | 1200 (33.01) | 2.46BC± 0.98 | |
| Journals | < 0.001 | ||
| Am. J. Sports Med. | 412 (11.33) | 3.24A ± 0.81 | |
| Knee Surg. Sports Traumatol. Arthrosc. | 389 (10.70) | 2.45D ± 0.91 | |
| Plast. Reconstr. Surg. | 356 (9.79) | 2.63C ± 0.91 | |
| Arthroscopy | 301 (8.28) | 2.99B ± 0.83 | |
| Other | 2177 (59.89) | 2.35D ± 0.92 | |
| Journal quartile | < 0.001 | ||
| Q1 | 1942 (53.43) | 2.82A ± 0.90 | |
| Q2 | 857 (23.58) | 2.35B ± 0.91 | |
| Q3 | 623 (17.14) | 2.04C ± 0.85 | |
| Q4 | 213 (5.86) | 2.29B ± 0.92 |
aA, B, C & D show posthoc comparisons using Tukey’s HSD. The mean difference is significant at the 0.05 level. (A > B > C > D)
The results of comparing the mean number of citations received by the articles showed a significant difference between all the subgroups of the variables studied (P < 0.001). The Tukey test for each of the variables showed that the articles were published in 2011 received more citations than in 2012 and 2013. Articles on Obstetrics & Gynecology received more citations than other subject categories. The number of citations decreased from level I to level V, and the articles with high levels of evidence received more citations than those with low levels of evidence. Based on the type of study, systematic review and meta-analysis articles received the most and the case-control studies received the lowest number of citations. Also, the articles written by the Japanese corresponding authors were the least and the Italian articles were the most cited. Articles published in Q1 journals received the most citations in a five-year period (P < 0.001).
The results of Pearson correlation coefficient test to calculate the relationship between the variable of the JIF and the number of 5-year citations of articles indicated a positive and significant relationship between these two variables (r = 0.36 , P < 0.001). There was also a positive and significant relationship between the number of references and the pages count of articles (r = 0.53, P < 0.001) and between the references count and the number of authors of articles (r = 0.36, P < 0.01). The correlation matrix of variables showed in Table 4.
Table 4.Correlation Matrix of the Variables Studied (N = 3683)
aP<0.001
bP<0.01
Stepwise linear regression analysis was used to investigate and determine the factors affecting the number of citations. The variables were entered into the models in order of importance. Variables predictor accounted for about 25% of the variation in the number of citations of therapeutic articles (F = 77.04, P < 0.001). Factors affecting the number of citations are reported in Table 5.
Table 5.Predicting the Factors Affecting the Number of Citation Received by Clinical Articles Based on Multiple Linear Regressiona
| Variables | B | SE | Beta | P Value | VIF |
|---|---|---|---|---|---|
| Level of Evidenceb | -0.13 | 0.02 | -0.14 | < 0.001 | 1.21 |
| IF 5 Years | 0.20 | 0.01 | 0.35 | < 0.001 | 1.22 |
| References Count | 0.01 | 0.001 | 0.13 | < 0.001 | 1.51 |
| Authors Count | 0.03 | 0.006 | 0.08 | < 0.001 | 1.09 |
| Title Word Count | 0.003 | 0.003 | 0.01 | 0.341 | 1.10 |
| Page Count | 0.05 | 0.01 | 0.11 | < 0.001 | 1.63 |
| Type of Articlesc | |||||
| Proceeding | 0.25 | 0.06 | 0.07 | < 0.001 | 1.10 |
| Review | 0.28 | 0.07 | 0.06 | < 0.001 | 1.07 |
| Editorial | 0.04 | 0.31 | 0.002 | 0.900 | 1.02 |
| Subject Categoryd | |||||
| Surgery | -0.16 | 0.04 | -0.06 | < 0.001 | 1.22 |
| Obst. & Gen. | -0.33 | 0.07 | -0.08 | < 0.001 | 1.31 |
| Critical Care | -0.06 | 0.07 | -0.01 | 0.349 | 1.11 |
| Clinical Neurology | -0.06 | 0.07 | -0.01 | 0.428 | 1.05 |
| Urology & Neph. | 0.05 | 0.08 | 0.01 | 0.552 | 1.06 |
| Medicine Res. & Exp. | 0.14 | 0.09 | 0.02 | 0.143 | 1.05 |
| Other | -0.34 | 0.13 | -0.04 | 0.01 | 1.04 |
aR2adj = 25.1, P value ≤ 0.001, F = 77.04
bReference variable was Level I
cReference variable was original article
dReference variable was orthopedic category
Five-year JIF, level of evidence, references count, page count, authors count and other variables that are shown in Table 5 significantly predict the number of citations. For each unit of evidence-level change, from level I to V, the number of citations drops by 0.14 standard deviations. It is also expected that by one unit increase in the score of five-year JIF, the number of citations will be increased by 0.35 standard deviation.
4. Discussion
Research has shown that the impact of clinical research on the decision making of specialists on the patient’s bedside is directly related to the type and design of the study. Accordingly, the system of evidence classification is based on the type and quality of the research method (1). Although the level of evidence is not an absolute criterion for the quality of articles, studies of high-level evidence are more reproducible and the results derived from the design of such studies will have a more reliable clinical application (13). Moreover, the purpose of medical research is to uncover information that can advance understanding of disease processes and patient care. Physicians often pay attention to the credibility of the journal for reading related articles in their field of expertise. The number of times an article is cited over a given period may indicate the relative importance of its findings by active research members of the medical community. Besides, the amount of citations an article is important for authors, journals, publishers, and readers; and citation is increasingly used as a measure of impact and quality in research policy and evaluation systems. This reported the results of a comprehensive study on the distribution of appropriate clinical papers for evidence-based medical practice in the field of human disease treatment between 2011 and 2013. The 3,683 articles that specified the level of evidence were published in 80 journal titles. Fourteen journals published more than 85% of these articles.
Usually, the number of citations is affected by the research and its quality (14). Examination of the number of citations received for five-year clinical trials related to disease treatment showed that high-level evidence (level I & II) received more citations than low-level evidence (level III & IV). The results of past studies on the relationship between the level of evidence and citations are not conclusive or consistent. While some previous research found no relationship (10, 15-18), some other studies found positive relationships between the two (14, 19). The inconsistency could be due to different subject areas studied, the definition of citation impact, and sampling. In our study, we included all levels of evidence, something that was lacking in some past studies.
Systematic reviews and meta-analysis articles received more citations than articles with other types of designs. Also, randomized clinical trials and cohort studies received more citations than case-control and case-series studies. In this research, the type of study design was extracted from the information contained in the abstracts of the articles and retrospective and prospective studies and follow-up studies were considered as other types of study design. Similarly, some past studies have shown that research method and study design and some other factors such as sample size and industrial financial support influence the number of citations (12, 20, 21)
Subject difference in the number of citations received has been shown in past studies (22). Our study showed that the majority of articles with a level of evidence mentioned in the abstract were in the orthopedic subject area (64%), and surgery (14%). Statistical tests showed a significant difference between the mean numbers of citations for different topics. Articles on Obstetrics & Gynecology had the highest citations.
The correlation between the different variables was positive. The correlation between the number of five-year citations and JIF, the number of references, article length, number of authors, and number of title words were weak. Multiple linear regression analysis showed that the evidence levels alone accounted for 4.2% of citation variation and the evidence levels along with the five-year JIF accounted for 18.4% of the variation in the number of citations received in clinical articles. Also adding other variables and modeling them across the seven models showed that fitting the variables of the levels of evidence, the five-year JIF, the number of references, the number of authors, the number of pages, the types of articles, and the subject influenced the number of citations received and explains 25.1% of these changes (Table 5). Similarly, Antoniou et al. found that topic, study design, paper length could predict citation rate (20). Another study also that factors such as sample size, JIF and study design could influence the citation rate (6).
The findings showed that more than 78 percent of these articles were on orthopedics and surgery. The study showed that the articles on gynecology have received more citation than others.
4.1. Conclusions
Number of citations was significantly correlated with JIF, level of evidence, number of references, number of authors, number of title words, length of article, subject, type of study design, geographical area of corresponding author, journal and publisher. Statistical tests predicted that JIF, the levels of evidence, the number of references, and the number of authors were more effective than the other factors on changes in the number of citations received. Clinical papers with high LoE (levels I & II) received more citations over a five-year period than those with lower LoE (levels III & IV).
