The Inherent Problems With Randomized Controlled Trials (RCTs)

The Chiro.Org Blog

SOURCE:   J Manip Physiol Ther 2003 (Sep); 26 (7): 460–467

Anthony Rosner, PhD

Foundation for Chiropractic Research and Education,
1330 Beacon Street, Suite 315,
Brookline, MA 02446, USA.

For 50 years, the accepted standard by which the usefulness of a therapeutic treatment is judged has been the randomized controlled trial (RCT), building from Hippocrates’ premise 2000 years ago that experience combined with reason was the therapy of choice for patients; that is, any treatment plan should both seem reasonable in theory and then be tested experimentally. Assuming that threats to both internal and external validity could be ruled out, the RCT became what is commonly regarded as the highest quality of clinical outcome study that could be mounted to allow inferences about cause and effect relationships to be drawn. The thinking was that the more rigorous and fastidious the design, the more credibility could be attached to the conclusions drawn from the outcomes of the study and the more likely the intervention was thought to have brought about those outcomes. [1] One of the strongest proponents of the RCT through the 1950s and 1960s was the British epidemiologist Archie Cochrane, who held that this type of experimental approach was essential for upgrading the quality of medical evidence. [2] In common hierarchical schemes of clinical experimental design, the RCT has been ranked the highest in rigor, as shown in Table 1.3 Even greater rigor has been presumed to occur with the statistical combination and weighting of the results of multiple RCTs in a meta-analysis to generate a more conclusive estimate of effect size. [4-5]

Table 1.   Hierarchy of Experimental Designs [3]

  1. Control group outcomes study (including RCTs).
  2. Single-subject experiment, replicated single-subject experiments.
  3. Single-group outcome study.
  4. Systematic case study.
  5. Anecdotal case report.

Designs are presented in descending order of rigor.

RCT, Randomized control trial.

From the point of view of clinical practice, however, especially in areas in which physical treatments are applied, the principles of fastidious treatments and blinding begin to wear thin and in a few recent examples regarding spinal manipulation, appear to have fallen apart completely. This difficulty is by no means confined to physical treatments, as the literature pertaining to the use of medications has also suggested that the inexperienced use and/or uncritical acceptance of the results of RCTs can lead to confusion. In this presentation, a few representative samples will be introduced as 7 case studies, which ironically would be ranked among the lowest in experimental rigor by the aforementioned hierarchy of clinical evidence. [3]

1. Reduction of Meta-analyses To Subjective Value Scales

In their efforts to compare 2 different preparations of heparin for their respective abilities to prevent postoperative thrombosis, Juni et al [6] have demonstrated that diametrically opposing results can be obtained in different meta-analyses, depending on which of 25 scales is used to distinguish between high-quality and low-quality RCTs. The root of the problem is evident from the variability of weights given to 3 prominent features of RCTs (randomization, blinding, and withdrawals), as shown in Table 2 by the 25 studies which have compared the 2 therapeutic agents. In 1 study, a third of the total weighting of the quality of the trial is afforded to both randomization and blinding, whereas in another study cited in the article, none of the quality scoring is derived from these 2 features. Widely skewed intermediate values for the 3 aspects of RCTs under discussion are apparent from the 23 other scales presented. The astute reader will immediately suspect that sharply conflicting conclusions might be drawn from these different studies, and these are amply borne out by the statistical plots shown in Figure 1. Here, each of the meta-analyses listed resolve the 17 studies they have reviewed into high-quality and low-quality strata, based on each of their scoring systems. It can be seen that 10 of the studies selected show a statistically superior effect of 1 heparin preparation, low-molecular weight heparin (LMWH), over the other but only for the low-quality studies. Seven other studies reveal precisely the opposite effect, in which the high-quality but not the low-quality studies display a statistically significant superiority of LMWH.

2. Occult “Salami” Publications

SOURCE:   Read the rest of this Full Text article now!