The Obstacles and Barriers to CAM or Alt-Med Research

The Obstacles and Barriers to CAM or Alt-Med Research

The Chiro.Org Blog

SOURCE:   Testimony to the White House Commission On Complementary and Alternative Medicine Policy

By Anthony L. Rosner, Ph.D.

FCER Director of Research and Education

Introduction:

Until 25 years ago, chiropractic research was vastly underdeveloped and appeared to some as an oxymoron. In 1975, a conference at the National Institute of Neurological Disorders and Stroke (NINDS) and the National Institutes of Health (NIH) concluded that “There are little scientific data of significance to evaluate this (chiropractic’s) clinical approach to health and to the treatment of disease.” [1] From that time onward, both clinical and basic research have advanced to the point at which:

The efforts to launch and develop a National Center for Complementary and Alternative Medicine (NCCAM) within the framework of the NIH are indeed admirable, taking the Center from a humble $2 million annual budget in 1991 to one that approaches $70 million today. This has taken place despite the comments of highly visible and influential individuals within the medical community to discredit alternative medicine in virtually any shape or form, a topic that I shall return to momentarily.

The following are what I believe to be the most significant barriers to research efforts in alternative medicine, the barriers having either remained in place or only recently having been removed.

1. Collaborative Arrangements:

Dating from the first RFA in March 1993, the Office of Alternative Medicine (OAM) required that researchers in alternative medicine collaborate with people from an orthodox medical background, described as “individuals familiar with conventional research methodologies.” [13] The implication was that researchers in alternative medicine, having fewer resources and shorter bibliographies than their allopathic counterparts to begin with, were somehow less qualified to pursue research questions of any kind. With the lack of exposure to either the theory or practice of alternative medicine modalities, potential allopathic medical collaborators had to overcome both gaps in knowledge and professional prejudices in order to become allied with alternative medical researchers, clearly delaying their efforts to launch research programs fundable from a federal point of view.

Furthermore, directing grant funding and their associated indirect costs toward allopathic medical centers rather than specific institutions in alternative medicine served to delay the building of the research infrastructure specifically within alternative medicine. NCCAM’s establishment of specific research centers (including the chiropractic center at Palmer University) and its recent provision of RO1 programs (in which individual researchers in CAM may step forward as the PI on a fundable grant application) are major steps in overcoming this obstacle.

2. Domestic Institutions:

A number of major milestones of research that have significantly lowered barriers to both the practice and research of chiropractic have been accomplished abroad. The low-back studies of Meade (Great Britain), [14, 15] the cervicogenic and tension headache studies of Nilsson (Denmark), [16, 17] the first randomized clinical trial addressing colic — possibly a nonmusculoskeletal condition in infants — [Denmark], [18] and numerous asthma case reports [19] and a pilot for a randomized clinical trial from Australia [20] to lay the foundation for future clinical trials directed at this condition, are but a few outstanding examples. Thus the requirement of many past federal programs restricting grants to domestic institutions only represents a major impediment to the accomplishments and potential of research in alternative medicine — which recognizes no national boundaries and which has clearly benefitted from the additional resources available beyond American borders.

3. Composition and Proceedings of Institutional Review Boards:

Undoubtedly institutional review boards (IRB) are an indispensable component of insuring patient safety and knowledgeability in a clinical trial. [21] From this writer’s firsthand experience, however, there have been instances in which a proposed randomized clinical trial within a major medical center have been rejected out of hand from what was probably the harboring of anti-chiropractic sentiments by the head of the IRB, who among other transgressions referred to this treatment alternative as “chiropracty.” While implementing panels to monitor the behavior of IRBs may appear excessive, the issue does bear further scrutiny in the event that viable and safe alternatives in the patients’ interest fall victim to prejudice within an IRB.

4. Composition of Study Sections:

For many of the same reasons as in the previous section, there must be an equitable number of individuals within each study section of a grant proposal who are familiar with and sensitive to the conduct of the therapeutic regimens to be tested. Common sense dictates that as eloquent and sympathetic a presentation of the therapies to be studied be made to the study section as a whole. This writer recalls with dismay an egregious violation of this principle in the first round of alternative medicine applications reviewed by the OAM in 1993, in which only a single member among the eight different 17-member study sections drafted by the NIH was a chiropractor (and he was unfortunately undistinguished as a researcher and lacked the necessary background to provide constructive critiques of grant proposals in this field). Adding insult to this injury was the fact that the OAM had been provided with the names of over 15 highly-qualified and accomplished chiropractic researchers. Completing this sorry state of affairs was the fact that these sections were charged with reviewing over 400 applications, nearly half of which pertained to chiropractic. Providing properly balanced and enlightened study sections reviewing grant applications is clearly an absolute requisite for reducing significant barriers to the conduct of CAM research.

5. Publication Bias and Quotas:

Examples can be introduced of editorial bias and quotas that have prevented the most robust of chiropractic research from reaching necessary audiences. A headache study by Boline, [22] for instance, rated the highest in quality by two independently conducted systematic literature reviews, [23, 24] was denied publication at The New England Journal of Medicine, Headache, and Cephalalgia before finally appearing in the Journal of Manipulative and Physiological Therapeutics two years later. Examples exist in which editorial comments to the principal authors of studies have clearly indicated that obtaining negative results for spinal manipulation was the criterion for acceptability for publication. [25, 26] Thus it is with dismay that this writer finds two inferior and widely-publicized studies in chiropractic which did get published in the New England Journal of Medicine [27, 28] (extensive rebuttals of which have been published elsewhere [29–32]). Statements by two previous editors of the New England Journal of Medicine offer little encouragement, as they have been patently biased with little qualification in their negative assessments of alternative medicine. [33, 34]

Publication quotas likewise impede the dissemination, and therefore the incentive, to perform CAM research. The American Journal of Public Health, for example, allows the publication of but one chiropractic study per year based upon current membership. While subsidization of publication costs through membership is entirely appropriate, restricting access of information for a modality that has been experienced by 37% of the American population at some point in their lifetime [35] appears arbitrary and Draconian.

6. Distortion of Research Results in the Press and in the Journals:

The crippling effects of bias and editorial policy of certain medical journals just discussed have ramifications in what is actually stated in papers and subsequently in the lay press. One study published in the New England Journal of Medicine (NEJM), for instance, stated a conclusion that was far beyond anything supported by the data. Specifically, the study discouraged the routine referral of patients to chiropractic: “Given the limited benefits and high costs, it seems unwise to refer patients with low back pain for chiropractic or McKenzie therapy.” [27] As egregiously out-of-bounds as a statement such as this is for a scientific journal, the lay press (to which the NEJM reportedly controls half of what health news we hear) only made matters worse. Such scare headlines as “Study Targets Worth of Chiropractic” [36] and “Chiropractic Care Blasted in Two Studies” [37] only poisoned the atmosphere, inhibiting further research efforts and inducing third party payors to deny reimbursements for chiropractic services in which the outcomes have yet to be definitively disproved. News releases such as these need to be actively discouraged, and the public needs to be further enlightened as to the research and potential of multiple modes of alternative therapy — not just chiropractic.

7. Mainstream Versus Alternative Status of Chiropractic:

Primarily due to the aforementioned research accomplishments regarding spinal manipulation and low-back pain, [2–7, 14, 15] chiropractic intervention has often been regarded as “mainstream” rather than alternative in the management of low-back pain [8–12] — and possibly at least some types of headache as well. [16, 17, 22–24]

However, in the treatment of:

asthma, [19, 20, 28, 32]
otitis media, [40–42]
attention deficit disorder/hyperactivity, [59]
infantile colic, [18, 43]
enuresis, [44]
dysmenorrhea [49–51]
premenstrual syndrome, [52–54]
chronic pelvic pain, [55]
repetitive stress disorders, [45–48]
scoliosis, [38, 39]
GI dysfunctions, [56–58]

chiropractic should still be regarded as an alternative therapeutic approach.

As a hybrid, therefore, chiropractic should be regarded as having important attributes of alternative medicine. Accordingly, it should not be dismissed as only a mainstream specialty ineligible for funding from sources that are supporting research in alternative medicine.

8. The Origins of Mainstream Medicine:

As suggested in the preamble to the 5-year Strategic Plan of NCCAM, “As CAM practices once considered unorthodox . . . are proven safe and effective by rigorous scientific investigation, they become part of mainstream healthcare.” This is certainly the way one hopes to transform good research into practice and clearly represents the mission of both NCCAM and the Foundation for Chiropractic Education and Research. However, since only 15% of medical procedures have been reported to be supported by any documentation [60] and only 1% is considered to be scientifically sound, [61] it is presumptuous to assume that what is currently accepted in standard medical procedures is intrinsically robust from a scientific point of view. Have large-scale clinical trials supported the use of every variation of catheter used in angioplasty, for instance? One need only consult the Merck Index of 100 years ago to realize that the following treatments — now woefully inadequate, outdated, and even dangerous — were unflinchingly accepted into the mainstream as de rigeur [62] within at least some of our lifetimes:

1. Formaldehyde for the common cold,
2. Arsenic or ammonia for baldness,
3. Opium and morphine for typhoid fever,
4 Blood-letting and chloroform for streptococcal infections; and
5. Strychnine, ice and lemon juice for diphtheria.

Thus, it is my belief that the idealism expressed regarding the origins of mainstream practices (in medicine or elsewhere) has to be tempered with realism. It is simply unreasonable to expect that every procedure and variation in healthcare delivery will be supported by a randomized clinical trial (RCT).

9. Intervention Paradigms:

Perhaps most flagrantly illustrated by both studies that were published in the New England Journal of Medicine, [27, 28] chiropractic must never be confused with merely high-velocity thrusting of the spine. Such is to reduce it to a one-dimensional specialty of cracking joints. Indeed, low-force contact procedures that have been incorrectly classified as placebos (shams) [28] have actually been shown to produce major improvements in both lung functional tests and patient symptoms with regard to asthma. [63]

According to the preamble of the charter for the Council of Chiropractic Education, chiropractors are fully trained as a portal of entry primary care health service capable of complete diagnosis. The Council of Chiropractic Education has accrediting status with the U.S. Department of Education (since 1974) and the Council on Postsecondary Accreditation (since 1976). Among the therapeutic regimens that chiropractors are licensed to administer are the use of hot and cold packs, electrical stimulation, soft tissue procedures (including trigger point therapy), and nutritional counseling. [64]

Building upon preliminary studies appearing just this year, [65, 66] more attention needs to be paid to long-term outcomes and supportive care. These attributes may differ somewhat from allopathic medicine’s historical approach to disease management. The point is to emphasize the effects of interventions over the long term, which have the potential of forestalling or preventing far more invasive and expensive procedures which are substantially riskier for the patient. In terms of overall cost control and offering the possibility of reducing both the costs and morbidity of medication error-related deaths, [67, 68] the implications of earlier intervention by alternative medical procedures are enormous.

10. The Role of the RCT:

There is no doubt that the RCT remains an important piece of the mosaic of evidence that needs to be assembled to substantiate a clinical procedure. However, it is certainly not the only piece and in many instances in my experience has been overrated:

• • In a highly publicized randomized clinical trial regarding the use of chiropractic in managing asthma, [28] both the use of a highly invasive sham procedure (an inappropriate placebo) and the possibility of small sample sizes obscuring possible effects by a Type II error have led to misleading conclusions, let alone interpretation by the lay press. [36, 37]

• • Another highly visible clinical trial comparing three interventions in the management of acute low-back pain [27] suffered from poor design [30] and inappropriate statistical procedures [31] Worse, it implied that a single intervention represented chiropractic care such that its clinical relevance was highly questionable. Indeed, the Royal College of General Practitioners in a very recent systematic review of the literature designed to update the CSAG Guidelines of the United Kingdom [8] has concluded that this trial neither adds nor detracts from the evidence base regarding appropriate interventions for low-back pain. [69]

• • A meta-analysis has shown that contrasting interpretations can be obtained, depending upon which of 25 scales used to distinguish between high- and low-quality trials is actually employed. [70]

• • A review of clinical trials comparing two antifungal agents has indicated that the apparent advantages of one of the instruments could have been obtained by manipulations of the design of most of the trials, in which the competing agent was inappropriately administered. [71]

• • The weight of evidence produced by clinical trials may be over-calculated due to the fact that the clinical trials are over-represented as duplicate, “sausage” publications by the same authors. [72–75]
• Methodological scores attached to clinical trials create a misleading profile of high- and low-quality studies if they place too much emphasis on sham procedures that we already know will seriously compromise controlled studies involving physical methods such as spinal manipulation if they are not true placebos. In other instances, the mere utterance of such terms as “blinded” or “randomized” in the title of the paper cited may be sufficient to glean points in the rating of clinical trials — even though such terms are never defined or qualified. The proper remedy in this instance would be to demote the trial ratings if such terms are inappropriately used. [70]

The point to realize here is that RCTs are subject to misinterpretation and outright abuse. Their generalization from a fastidious, defined laboratory setting is problematical. It is sometimes forgotten that the source of randomized clinical trials remains the sound, well-documented observations in the clinical setting. This has led no less an epidemiologist than David Sackett to conclude that there are essentially two pillars of sound clinical evidence, only one of which is experimentally derived from the RCT: [76]

“External clinical evidence can inform, but can never replace, individual clinical expertise, and it is this expertise that decides whether the external evidence applies to the individual patient at all and, if so, how it should be integrated into a clinical decision.”

In light of these many arguments, I would maintain that the White House Commission on Complementary and Alternative Medicine Policy (WHCCAMP) should place far greater emphasis upon cohort studies and case series in its research goals rather than assume categorically that they provide inferior guidance to clinical decision-making than RCTs. It should be quite clear from this discussion that a well-crafted cohort or case series is far more informative than a flawed or corrupted RCT.

Solutions:

In light of the foregoing discussion, I would recommend that the WHCCAMP pursue the following:

• Encourage all qualified researchers in alternative medicine to apply for federal and private grant support, taking into account refusals of attempted publication in peer-reviewed medical journals.
• Encourage researchers from around the world rather than only from American soil to apply for such support;
• Institute an appeals process if a research team believes that an Institutional Review Board has turned down a research proposal unfairly;
• Ensure that all study sections include a sufficient number of individuals who are both familiar with and sensitive to the therapeutic regimens described within the proposal under review;
• Frame and encourage legislation that encourages better communication of both researchers and practitioners of alternative medicine with the media, limiting the contacts with those journals found to harbor unjustifiable biases against alternative medical procedures.
• Ensure that chiropractic is recognized as both a mainstream and an alternative intervention, depending upon the condition for which therapy is indicated. Accordingly, ensure that chiropractic is excluded from neither category in terms of grant eligibility and collaboration.
• Encourage research directed at long-term outcomes and supportive care, areas which have commonly been neglected in allopathic medical care and which offer the possibility of low-cost, preventive health management.
• Ensure that chiropractic is not limited to referral-only specialty care restricted to the back, based upon current accreditation, licensure, and research. In this regard, it is to be appreciated as a direct portal of entry for patient care — appreciating the ability of chiropractors to diagnose and apply treatments that are broader in scope than merely high-velocity thrusts.
• Appreciate the limitations of randomized clinical trials, admitting well-designed and well-executed cohort and case studies into the evidence base supporting a given intervention.
• Frame and encourage legislation that does not permit third party payors to restrict reimbursements beyond the scope of practice currently stipulated by licensure laws within the states.
• Coordinate activities with NCCAM and avoid duplication of efforts wherever possible.
• Encourage private sources to invest in all types of alternative and mainstream medical research with adequate oversight. This would have the twofold benefit of offsetting the pharmaceutical industry’s virtual monopoly of private support of medical research, as well as offering a variety of measures to reduce the chances of having research quality compromised.

REFERENCES:

1. Goldstein M [ed]:
   Monograph No. 15. The research status of spinal manipulation, U.S. Department of Health, Education, and Welfare,
   Washington, DC, February 3-4, 1975.
2. Koes BW, Assendelft WJJ, van der Heijden GJMG, Bouter LM.
   Spinal manipulation for low-back pain: A updated systematic review of randomized clinical trials.
   Spine 21(24): 2860-2871.
3. van Tulder M, Koes BW, Bouter LM.
   Conservative treatment of acute and chronic nonspecific low back pain: A systematic review of randomized controlled trials of the most common interventions.
   Spine 1997; 22(18): 2128-2156.
4. Anderson R, Meeker WC, Wirick BE, Mootz RD, Kirk DH, Adams A.
   A meta-analysis of clinical trials of spinal manipulation.
   Journal of Manipulative and Physiological Therapeutics 1992; 15(3): 181-194.
5. Shekelle PG, Adams AH, Chassin MR, Hurwitz EL, Brook RH.
   Spinal manipulation for low-back pain.
   Annals of Internal Medicine 1992; 117(9): 590-598.
6. Bigos S, Bowyer O, Braen G, et al.
   Acute low back pain in adults. Clinical practice guideline No. 14. AHCPR Publication No. 95-0642.
   Rockville, MD: Agency for Health Care Policy and Research, Public Health Service,
   U.S. Department of Health and Human Services. December 1994.
7. Manga P, Angus D, Papadopoulos C, Swan W.
   The effectiveness and cost-effectiveness of chiropractic management of low-back pain.
   Ottawa, Ontario, CANADA: Pran Manga & Associates, Inc.,
   University of Ottawa, 1993, pp 65-70.
8. Rosen M.
   Back pain. Report of a Clinical Standards Advisory Group Committee on back pain.
   May 1994, London: HMSO.
9. Commission on Alternative Medicine, Social Departementete, Legitimization for Vissa Kiropraktorer,
   Stockholm, SOU [English Summary] 1987; 12: 13-16.
10. Danish Institute for Health Technology Assessment:
    Low-back pain, frequency, management, and prevention from an HTA perspective.
    Danish Health Technology Assessment 1999; 1(1).
11. Thompson CJ. Second Report, Medicare Benefits Review Committee, Canberra, AUSTRALIA:
    Commonwealth Government Printer, June 1986, Chapter 10 [Chiropractic].
12. Hasselberg PD.
    Chiropractic in New Zealand, Report of A Commission of Inquiry.
    Wellington, NEW ZEALAND: Government Printer, 1979.
13. NIH Guide, March 26, 1993; 22(12).
14. Meade TW, Dyer S, Browne W, Townsend J, Frank AO.
    Low back pain of mechanical origin: Randomised comparison of chiropractic and hospital outpatient treatment.
    British Medical Journal 1990; 300: 1431-1437.
15. Meade TW, Dyer S, Browne W, Townsend J, Frank AO.
    Randomised comparison of chiropractic and hospital outpatient management for low back pain: Results from extended follow-up.
    British Medical Journal 1995; 311: 349-351.
16. Nilsson N, Christensen HW, Hartvigsen J.
    The effect of spinal manipulation in the treatment of cervicogenic headache.
    Journal of Manipulative and Physiological Therapeutics 1997; 20(5): 326-330.
17. Bove G, Nilsson N.
    Spinal manipulation in the treatment of episodic tension-type headache.
    Journal of the American Medical Association 1998; 280(18): 1576-1579.
18. Wiberg JMM, Nordsteen J, Nilsson N.
    The short-term effect of spinal manipulation in the treatment of infantile colic: A randomized controlled trial with a blinded observer.
    Journal of Manipulative and Physiological Therapeutics 1999; 22(8): 517-522.
19. Jamison JR, McEwen AP, Thomas SJ.
    Chiropractic adjustment in the management of visceral conditions: a critical appraisal.
    Journal of Manipulative and Physiological Therapeutics 1992; 15(3): 171-180.
20. Hayek R, Ali S, Brice C.
    Asthma and chiropractic: An Australian trial [single blind cross over study].
    Proceedings of the 1998 International Conference on Spinal Manipulation, Vancouver,
    British Columbia, July 16-19, 1998, pp. 72-74.
21. The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research.
    Institutional Review Boards. Washington, DC:
    U.S. Government Printing Office, 1978.
22. Boline P, Kassak K, Bronfort G, Nelson C, Anderson AV.
    Spinal manipulation vs. amitriptyline for the treatment of chronic tension-type headaches: A randomized clinical trial.
    Journal of Manipulative and Physiological Therapeutics 1995; 18(3): 148-154.
23. Hurwitz EL, Aker PD, Adams AH, Meeker WC, Shekelle PG.
    Manipulation and mobilization of the cervical spine: A systematic review of the literature.
    Spine 21(15): 1746-1760.
24. Kjellman GV, Skagren EI, Oberg BE.
    A critical analysis of randomised clinical trials on neck pain and treatment efficacy: A review of the literature.
    Scandinavian Journal of Rehabilitative Medicine 1999; 31: 139-152.
25. Brennan P. Personal communication, 1992.
26. Nilsson N.
    Publication bias in the medical journals: An n of 1 study.
    Presentation at the 2000 International Conference on Spinal Manipulation,
    Bloomington, MN, September 24, 2000.
27. Cherkin DC, Deyo RA, Battie M, Street J, Barlow W.
    Comparison of physical therapy, chiropractic manipulation, and provision of an educational booklet for the treatment of patients with low back pain.
    New England Journal of Medicine 1998; 339(14): 1021-1029.
28. Balon J, Aker PD, Crowther ER, Danielson C, Cox PG, O’Shaugnessy D, Walker C, Goldsmith CH, Duku E, Sears MR.
    A comparison of active and simulated chiropractic manipulation as adjunctive treatment for childhood asthma.
    New England Journal of Medicine 1998; 339(15): 1013-1020.
29. Rosner AL. Response to the Cherkin article in The New England Journal of Medicine.
    Dynamic Chiropractic November 2, 1998; 16(23).
30. Chapman-Smith D.
    Back pain, science, politics and money.
    The Chiropractic Report November 1998; 12(6).
31. Freeman MD.
    A critical evaluation of the methodology of a low-back pain clinical trial.
    Journal of Manipulative and Physiological Therapeutics 2000; 23(5): 363-364.
32. Blum CL.
    Chiropractic and sacro-occiptal technique in asthma treatment.
    Chiropractic Technique 1999; 11(4): 174-180.
33. Bunk S.
    Is integrative medicine in the future? Debate between Andrew Weil, M.D. and Arnold Relman, M.D.
    The Scientist 1999; 13(10): 1,10-11.
34. Angell M, Kassirer, JP.
    Alternative medicine – The risks of untested and unregulated remedies.
    New England Journal of Medicine 1998; 339(12): 839-841.
35. www.intersurvey.com, July 2000.
36. Haney DC. Associated Press, October 8, 1998.
37. Des Moines Register, October 8, 1998, p. 1.
38. Plaugher, G, Cremata EE, Phillips, R.
    A retrospective consecutive case analysis of pretreatment and comparative static radiological parameters following chiropractic adjustments.
    Journal of Manipulative and Physiological Therapeutics 1990; 13(9): 498-506.
39. Tarola GA.
    Manipulation for the control of back pain and curve progression in patients with skeletally mature idiopathic scoliosis: two case studies.
    Journal of Manipulative and Physiological Therapeutics 1994; 17(4): 253-257.
40. Froehle RM.
    Ear infection: A retrospective study examining improvement from chiropractic care and analyzing for influencing factors.
    Journal of Manipulative and Physiological Therapeutics 1996; 19(3): 169-177.
41. Fallon J.
    The role of chiropractic adjustment in the care and treatment of 332 children with otitis media.
    Journal of Clinical Chiropractic Pediatrics 1997; 2(2): 167-183.
42. Degenhardt BF, Kuchera ML.
    Efficacy of osteopathic evaluation and manipulative treatment in reducing the morbidity of otitis media in children.
    Journal of the American Osteopathic Association 1994; 94(8): 673.
43. Klougart N, Nilsson N, Jacobsen J.
    Infantile colic treated by chiropractors: a prospective study of 316 cases.
    Journal of Manipulative and Physiological Therapeutics 1989; 12(4): 281-288.
44. Reed WR, Beavers S, Reddy SK, Kern G.
    Chiropractic management of primary nocturnal enuresis.
    Journal of Manipulative and Physiological Therapeutics 1994; 17(9): 596-600.
45. Sucher B.
    Palpatory diagnosis and manipulative management of carpal tunnel syndrome.
    Journal of the American Osteopathic Association 1994; 94(8): 647-663.
46. Strait BW, Kuchera ML.
    Osteopathic manipulation for patients with confirmed mild, modest and moderate carpal tunnel syndrome.
    Journal of the American Osteopathic Association 1994; 94(8): 673.
47. Davis PT, Hulbert JR, Kassak KM, Meyer JJ.
    Comparative efficacy of conservative medical and chiropractic treatments for carpal tunnel syndrome: A randomized clinical trial.
    Journal of Manipulative and Physiological Therapeutics 1998; 21(5): 317-326.
48. Bersten G, McCarthy K.
    Conservative chiropractic approaches to carpal tunnel syndrome.
    Topics in Clinical Chiropractic 1999; 6(4): 62-72.
49. Liebl NA, Butler LM.
    A chiropractic approach to the treatment of dysmenorrhea.
    Journal of Manipulative and Physiological Therapeutics 1990; 13(3): 101-106.
50. Boesler D, Warner M, Alpers A, Finnerty EP, Kilmore MA.
    Efficacy of high-velocity low-amplitude manipulative technique in subjects with low back pain during menstrual cramping.
    Journal of the American Osteopathic Association 1993; 93(2): 203-214.
51. Kokjohn K, Schmid DM, Triano JJ, Brennan PC.
    The effect of spinal manipulation on pain and prostaglandin levels in women with primary dysmenorrhea.
    Journal of Manipulative and Physiological Therapeutics 1992; 15(5): 279-285.
52. Stude DE.
    The management of symptoms associated with premenstrual syndrome.
    Journal of Manipulative and Physiological Therapeutics 1991; 14(3): 209-216.
53. Walsh MJ, Chandraraj S, Polus BI.
    The efficacy of chiropractic therapy on premenstrual syndrome: a case series study.
    Chiropractic Journal of Australia 1994; 24(4): 122-126.
54. Walsh MJ, Polus BI.
    A randomized, placebo-controlled clinical trial on the efficacy of chiropractic therapy on premenstrual syndrome.
    Journal of Manipulative and Physiological Therapeutics 1999; 22(9): 582-585.
55. Hawk C, Long C, Azad A.
    Chiropractic care for women with chronic pelvic pain: A prospective single group intervention study.
    Journal of Manipulative and Physiological Therapeutics 1997; 20(2): 73-79.
56. Falk JW.
    Bowel and bladder dysfunction secondary to lumbar dysfunctional syndrome.
    Chiropractic Technique 1990; 2(2): 45-48.
57. Wagner T, Owen J, Malone E, Mann K.
    Irritable bowel syndrome and spinal manipulation: a case report.
    Chiropractic Technique 7(4): 139-140, 1995.
58. Pikalov A, Kharin VV.
    Use of spinal manipulative therapy in the treatment of duodenal ulcer: A pilot study.
    Journal of Manipulative and Physiological Therapeutics 1994; 17(5): 310-313.
59. Giesen JM, Center DB, Leach RA.
    An evaluation of chiropractic manipulation as a treatment of hyperactivity in children.
    Journal of Manipulative and Physiological Therapeutics 1989; 12(5): 353-363.
60. Smith R.
    Where is the wisdom: The poverty of medical evidence.
    British Medical Journal 1991; 303: 798-799.
61. Rachlis N, Kuschner C.
    Second opinion: What’s wrong with Canada’s health care system and how to fix it. Toronto: Collins, 1989.
    Merck’s 1899 Manual, or the Materia Medica. New York, NY: Merck & Co., 1899.
62. Field T, Henteleff T, Hernandez-Reif M, Martinez E, Mavunda K, Kuhn C, Schanberg S.
    Children with asthma have improved pulmonary functions after massage therapy.
    Journal of Pediatrics 1998; 132(5): 854-858.
63. Haldeman S, Chapman-Smith D, Peterson, DM Jr.
    Guidelines for chiropractic quality assurance and practice parameters.
    Proceedings of a consensus conference commissioned by the Congress of Chiropractic State Associations, held at the Mercy Conference Center, Burlingame, CA, January 25-30, 1992.
    Gaithersburg, MD: Aspen, 1993.
64. Rupert RL.
    A survey of practice patterns and the health promotion and prevention attitudes of U.S. chiropractors: Maintenance care: Part I.
    Journal of Manipulative and Physiological Therapeutics 2000; 23(1):1-9
65. Rupert RL, Manello D, Sandefur R.
    Maintenance care: Health promotion services administered to U.S. chiropractic patients age 65 and older: Part II.
    Journal of Manipulative and Physiological Therapeutics 2000; 23(1): 10-19.
66. Lazarou J, Pomeranz B, Corep P.
    Incidence of adverse drug reactions in hospitalized patients.
    Journal of the American Medical Association 1998; 279(15): 1200-1205.
67. Johnson JA, Bootman JL.
    Drug-related morbidity and mortality: A cost-of-illness model.
    Archives of Internal Medicine 1995; 155: 1949-1956.
68. Royal College of General Practitioners, unpublished update of CSAG Guidelines [reference 8], 1999.
69. Juni P, Witschi A, Bloch R, Egger M.
    The hazards of scoring the quality of clinical trials for meta-analysis.
    Journal of the American Medical Association 1999; 282(11): 1054-1060.
70. Johansen HK, Gotzsche PC,
    Problems in the design and reporting of trials of antifungal agents encountered during meta-analysis.
    Journal of the American Medical Association 1999; 282(18): 1752-1759.
71. Rennie D.
    Fair conduct and fair reporting of clinical trials.
    Journal of the American Medical Association 1999; 282(18): 1766-1768.
72. Gotzsche PC.
    Multiple publication of reports of drug trials.
    European Journal of Clinical Pharmacology 1989; 36: 429-432.
73. Huston P, Moher D.
    Redudancy, disaggregation, and the integrity of medical research.
    Lancet 1996; 347: 1024-1026.
74. Tramer MR, Reynolds DJM, Moore RA, McQuay HJ.
    Impact of covert duplicate publication on meta-analysis: A case study.
    British Medical Journal 1997; 315: 635-640
75. Sackett DL.
    Editorial: Evidence-based medicine.
    Spine 1998; 23(10): 1085-1086.