Negative Studies Have a Higher Burden of Proof

Dear Editor,

Science involves controversy, conflicting results and thoughtful analysis. Clinical trials finding a significant efficacy for a new investigational therapy require confirmatory trials, but often subsequent trials result in a negative finding (equivalence or non-significant efficacy). These negative trials are important and, in the past, have had trouble getting published. Publication bias is common, as studies showing a significant efficacy are typically published, while negative studies have been largely ignored. While this trend is slowly reversing, the onerous task for authors in overcoming this bias necessities a higher burden for quality writing and more detailed analysis than required for studies with significant positive findings. It is imperative that scientists explore all the possible explanations for the finding of non-significance efficacy. It may be that the investigated therapy is truly not effective, but it also may be due to insufficient power due to small study size, or the choice of a low dose, or differential attrition, or other types of bias.

The field of probiotics for the treatment and prevention of various diseases is replete with both positive (showing significant therapeutic effect) and negative (showing equivalence or non-significant results) clinical trials. This is not surprising, as the efficacy of probiotics is both strain-specific and disease-specific. Not all probiotic strains work for all diseases or even within one disease indication. A recent paper by dashti et al. In this journal reports finding no significant difference in the incidence of necrotizing entercolitis (NEC) in neonates treated with a probiotic mixture compared to placebo (1).

NEC is an important cause of morbidity and mortality in low birth weight neonates and currently there are no modalities to prevent NEC from occurring. The use of probiotics has been suggested by a recent significant finding of pooled efficacy for NEC from 11 randomized trials in a meta-analysis by Deshpande et al. (2). Dashti et al. tested a probiotic mixture of eight bacterial strains, but did not find a significant reduction in NEC when compared to their control group (1). However, when we try to place this result into context, it is difficult to determine why or how this result may have occurred. The authors in this paper failed to provide readers with sufficient details on the study to fully assess the reasons why they did not find a significant efficacy of this probiotic mixture to prevent NEC. The major limitation is that the authors fail to present the identity of the two groups (designated “Group A” and “Group B”), so it is impossible to determine which is the group treated with the probiotic and which is the control group. Blinding a trial is commendable during its operation, but maintaining the blinding when presenting the results is counterproductive.

The recommended use of a CONSORT figure would have provided valuable insight into the numbers of infants screened versus number enrolled. No presentation of the rates and reasons for attrition were presented, so it cannot be determined if loss-to-follow-up may have played a role in the negative results. Another common reason for negative efficacy findings in that the trial is too small and lacks the power to detect a significant difference. the authors did not report if they calculated the required sample size in the methods sections, indeed, the power of this study is extremely low (only 4%).

Careful description of the investigational treatment is paramount in clinical trials. The investigational mixture was not fully described (missing bacterial strain for one of the Bifidobacterium strains) and the authors failed to indicate that this product contains 990 mg of a prebiotic (FOC) per sachet.

The authors discuss some possible reasons for their negative findings (low dose, low rate of NEC or the lack of efficacy of the tested probiotic product). Low dose may be an explanation, in that the neonates most at risk (birth weight < 1500 mg) only received 10⁸ organisms/d and most trials in neonates (BW < 1500 g) at risk for NEC have received a daily dose of 10⁹/d (2). it would have been useful for the authors to present the rate of NEC by the two groups, stratified on birthweight, as 43% of the enrolled neonates were not at high risk for NEC (birthweight > 1500 g). Their rate of NEC is not low (12-14%) compared to other studies (1-10% in various probiotic groups and 6-16% in control groups) (3-5).

In conclusion, the results reported by Dashti et al. may not have found a significant efficacy for this specific probiotic mixture in preventing NEC, but insufficient study methods and data presentation limits the interpretation of their data. this study exemplifies the requirement for a higher burden of analysis for possible reasons for non-significant findings.