Chapter 8 Research in Natural Medicine
What Is Natural Medicine?
There have been a great many published reviews of natural medicine practices, especially of its substances and approach to diseases, including the present volume.1–3 Nevertheless, many of its practices have not borne the level of scrutiny of standard modern medicine. Some within CAM ranks have resisted the usual scientific evaluation as reductionistic and an inappropriate means to the understanding of healing.* This chapter addresses needs in the development of relevant original data in natural medicine.
Purpose of Research
Research tools such as descriptive statistics and hypothesis testing are epidemiologic techniques; they lead to generalizations about populations. They have limitations in clinical practice, in which the focus is on one patient at a time. Current biomedical research increasingly focuses on strategy developed from molecular (lately genetics and the “-omics”) and physical (surgical and prosthetic) models. In research practice, hypothesis testing is usually most sensitive to a selected outcome and most internally valid (able to answer the precise question it set for itself) when it is most reductionistic. The more discrete variables of chemistry and mechanics lend themselves to more reductionistic hypothesis testing. In turn, the validity of the reductionistic approach depends on the precision (narrowness) of the question. Natural medicine is ecologic and holistic in its orientation, with numerous broad analog inputs and outputs with what might appear to be intuitive discriminations. The common models in research that emphasize transferability, internal validity, replicability, and generalizability in molecular and physical systems may conflict with model validity when studying natural medicine systems.4 Model validity ensures that the study design reflects the structure and logic of the practices of interest. Thus, one might ask whether alternative medicine calls for alternative science. It may be so; some philosophers of science and mathematicians are working toward methods that may be better suited to the evaluation of natural medicine systems.5,6 Although the development of accepted research methods that incorporate more complex and subtle biomedical quanta are awaited, much can be done with the currently available biomedical research tools.
Methodological Problems and Strategies
What does replicability mean for a medicine that may offer individualized treatment to every patient? How does individualized treatment map onto an evaluative system in which outcomes typically refer to populations? Better tools are needed to understand and evaluate these practices, particularly informational tools that go beyond the more commonly used physiologic, psychological, pharmaceutical, and epidemiologic methods.5 Many accommodations within clinical study methodology can be made that will reduce the apparent contradictions of the different medical value systems. Research methods from other fields like psychology, ecology, and anthropology can also be used. When performing research studies in natural medicine that might lead to improved practice, issues that are different from or need more emphasis than those in conventional biomedicine will need to be addressed. This chapter touches on some of the methodological issues for clinical trials, observational studies, and basic science studies in natural medicine.
Clinical Trials
Properly performed clinical research in conventional medicine is a demanding discipline that encompasses study design; determination of sample size and population characteristics; ethics; clinical care; definition and sensitive measurement of diseases and outcomes; project management, recruitment, and participant adherence and retention; and data management and statistics, analysis, and interpretation. All of these are secondary to establishing the research questions—the purposes—of the trial which will differ, for example, in explanatory (development and discovery: Could this work in controlled circumstances? How?) versus pragmatic (application and policy: Does this work in everyday practice? Which treatment is better?) study.7 Different designs can be usefully applied in research approaches to different aspects of natural medicine practice. For any type of clinical trial of natural medicine, the following issues bear additional attention beyond those required for pharmaceutical trials:
2. Individualization of treatment: nosology and indications
3. Combination therapies: effect size and safety
Standardization
The organized natural medical professions have variably definitive articulations of philosophy and clinical strategies. What constitutes a competent practitioner in a given discipline often remains uncertain,8 and many natural medicine approaches, like those of indigenous practices, are unlikely ever to be codified. Even with the licensed practices (e.g., naturopathic medicine, acupuncture, and chiropractic), health insurance coverage with regulation by practitioner-review panels is recent. It is only in the last two decades that coding systems for alternative practices have begun to describe the array of interventions available in the United States.9 The current emphasis on research in CAM practices calls for speeding up the process of standardization toward replicability.
Communication about natural medicine research issues can be problematic when using standard clinical practice and biomedical research metaphors. One way to understand the problem is to say that natural medicine’s critical practices lack articulation and operationalization of accepted and, within the respective practitioner groups, widely understood memes. Memes are collective concepts: ideas, behaviors, or skills that are transferred from one person to another by imitation and replicable on a population basis (see Richard Dawkins’ book The Selfish Gene, from 1976)9a. A meme, as a phrase, invokes meaning beyond those of the words themselves, such as in the principle vis medicatrix naturae (the healing power of nature). Dictionaries of memes may be needed for the collective concepts of natural medicine practice that may be identified as the practices are investigated. The development of memes through which to express the syncretic concepts of natural medicine and which may have broad applicability is one of the benefits of researching the practices.5 The dearth of articulation of natural medicine’s concepts accounts in part for its remarkable variation among practitioners. Its transfer from mentor to student may be imprecise or misinterpreted even before practitioner preferences and personalities are brought into play. This is not to say that the concepts of natural medicine practice are not reliably transmitted from mentor to student, only that they are not yet always articulated in ways that have been operationalized for clinical trials.
Standardization of natural remedies is becoming better understood. To perform a single-agent controlled trial of herbs in a specific disease, numerous choices should be made about the intervention. The Policy Announcement on the Quality of Natural Products of the National Institutes of Health (NIH) National Center for Complementary and Alternative Medicine (NCCAM) has brought some clarity to the botanical standards for NIH sponsored trials (http://www.nccam.nih.gov/research/policies/naturalproducts.htm). Studies should begin with verification of plant species used, growth and harvesting conditions, and the stability of purported active compounds. Selections should be made among plant parts, various crude extracts, or specific chemical constituents that may be concentrated in various ways and to varying degrees of purity. Crude fresh extracts, which traditional herbalists prefer, are highly susceptible to deterioration. In more sophisticated systems of botanical medicine preparation, a product is standardized to guarantee the minimum or maximum concentration of a number of ingredients for a given period, for example, EGb 761 (Schwabe GmbH, Karlsruhe, Germany), the Ginkgo biloba, which has been the most researched botanical and is standardized on four constituents. Standardizing on particular constituents has its challenges. Active ingredients in plants are often classes of molecules (e.g., polysaccharides, saponins, terpenes) that are difficult to distinguish in biological activity. Different compounds in a single species may have similar, possibly complementary effects, such as the polysaccharides and isobutyramides in Echinacea species. During in vitro assays, which guide fractionation of the crude extract toward a single active molecule, it is not uncommon for activity to increase but then diminish as greater purity of an identified molecular species is reached, as was the case of the terpenes of Andrographis paniculata (AndroVir, Paracelsian, Inc., Ithaca, NY) in cell signaling. A few botanicals are standardized on an in vitro biological activity. The industry standard, porously applied, is chemical standardization on actives and certified Good Manufacturing Practices.
“Body, mind, spirit” is a leitmotif of natural medicine. The interplay of psychodynamic and spiritual phenomena and physical health are only partially encompassed by the behaviorist approach of most health psychology research, but this has been greatly augmented by recent studies in “mind–body” medicine with numerous reviews in various specialties in publication. Advances have been made in the neuroscience of expectancies in the placebo effect, with the size of the placebo effect recognized as different in different conditions, but despite the ubiquitous use of placebo in practice,10 operationalization and manipulation of placebo effects remain fraught methodologically and ethically. Placebo use in medical encounters continues to be explored, but raises more questions than answers;11 however, hope remains for advances in future studies.12
While they represent a small proportion of all health studies, questions about spirituality remain of investigative interest, although with few authoritative and widely used research methods. Operationalization of spiritual experience is likely to be idiosyncratic or culture specific. The 4th edition of the Diagnostic and Statistical Manual (DSM-IV, 1995) reinstated the possibility of a religious or spiritual problem. Definitions of spirituality in the medical literature may refer to hope and meaning or to a personal relationship with God, serenity, or connectedness, all perhaps related to states of the patient’s consciousness.13,14 A number of thinkers continue to bemoan the lack of definition in the area.15 Although efforts to present cogent, broadly acceptable definitions have been made, they have not been successful.16,17 Better approaches to specifying spiritual interventions may be needed before they will be accessible to replicable research. Recent reviews of the health effects of intercessory prayer, an observable phenomenon, have been equivocal.18
Individualization of Treatment
Some natural medicine concepts may no longer have or may never have had a biomedical equivalent. An example is the concept of “constitution” used in naturopathy, homeopathy, and Asian medical systems. This is a patient’s given biological potential, tendencies, and patterns of long-standing psychophysical strengths and weaknesses that are genetically and embryologically determined. Others are the “biological terrain” (the background physical health and individual context for the immediate medical problem) and the “vital force” (the motive plan or spirit animating mind and body expressed as physiologic and psychological functionality and adaptability). Biomedical equivalents exist in some parameters for system control concepts; however, they are therapeutically exploited more thoroughly by holistic practitioners. Such concepts are balance (as in the immune system, among microbial symbionts, hormones, and neurotransmitters); deficiency (not just nutritional but organ deficiencies, such as hypochlorhydria and hypothyroidism); functional reserves; endogenous and exogenous toxicities; and dysmetabolisms (e.g., syndrome X). An important concept is that a disease syndrome may be an attempt by the body to adapt to ecologic stress and so should not be unnecessarily suppressed. This is of interest not only because it is an independent variable (baseline factor), but also because it influences the measurement of the dependent variable (outcome).
Nonspecific Healing Effects
An important confounder that may be integral to treatment is intentionality. (See Chapters 6, Placebo and the Power to Heal and 7, Positive Mental Attitude.) The potential therapeutic action of pure intention, which is not mediated by language or any well-known material force, might be espoused explicitly by only a minority of holistic practitioners, although actually it might be widely practiced among them. This may be called “psychic healing” or simply “healing”. If the intent is to determine whether these techniques have effect on disease, surprisingly, there may be no particular difficulties in conducting studies than those already addressed here. Trials of intercessory prayer may provide a model.19 Design differences may be called for if the therapists are special (gifted) versus ordinary people, or if the healing energy is directed or willed versus invoked (as from God or spirits). Little has been written on the capacity of intention to influence physical outcomes; there is implicit rejection of the idea based on cartesian mind–body duality. Nevertheless, a number of studies suggest its existence.20,21 Studies of the effect of prayer or other spiritual practices also may be relevant here and raise the possibility that specific kinds of intentionality may make a difference.22
Health care is practiced within a culture whereby practitioners and patients have roles and expectations that reinforce belief in expected outcomes. Some studies have indicated that good practitioner–patient communication results in better health outcomes. Thus, some of the magnitude of effect may depend on the setting in which care is offered. Differences in outcome between medical practices may also hinge on the difference between research and nonresearch settings. For example, it is possible that research participants who refuse random assignment have a psychological orientation that may work synergistically with the physical effects of the practices of a medical culture that they prefer or to which they are native. If this is true, the evaluation of systems of practice relative to each other would preclude simple randomization. Trials that should be of real interest to policymakers are those that determine not simply whether a practice works for anyone to whom it is applied, but whether it works (and is cost effective) among those who choose it. A study design that could determine the added value of the availability of some form of natural health care in those who choose it would be one that randomly assigns patients to either a group that undergoes further randomization to strict assignment of alternative or conventional care or a group offering them a choice of alternative or conventional care.23 Such a four-group trial including randomization with choice would compare the effect of the different practices among those patients who discriminatively select among them versus those who are simply assigned to them.
Outcomes and Measures
Because natural medicine addresses the holistic, it is expected by practitioners to have broad benefit (e.g., correcting deficiencies, strengthening and balancing metabolic processes, and “removing the obstacles to cure”). If a natural medicine intervention for a particular disease is shaped and delivered, a benefit is likely to affect more than one disease condition at a time. Randomized clinical trials (RCTs) are typically designed to provide higher validity on a few measures and tend to overlook effects on diseases other than the target disease, neglecting effects on overall well-being. (See discussion on “Cure and Suppression” in Chapter 5, Philosophy of Naturopathic Medicine.) Clients, who often use CAM for overall health and wellness,22 expect to see positive results not only in objective disease parameters but also in general body functions (e.g., fatigue and energy, pain, inflammation, digestion, mental and emotional function) at a perceptible level. Thus, methods to assess the efficacy of the medicine should use holistic measures accounting for effects throughout the body systems and over the life cycle, if possible, and in an intervention’s influence on social roles, productivity, and cost. Beyond the effectiveness of medicine for the individual patient, measures of an entire holistic practice may reasonably be expected to address public health as well as environmental, social, and economic outcomes.
Strides have been made in the development of measures that assess individual health globally.24 There has been progress with the development and wide adoption of instruments, beginning with the Medical Outcomes Study SF-36 and its successors, and of measures such as “quality-adjusted years of life saved.” However, there remains a lack of sensitivity to changes in disease at higher levels of function. The lack of sensitivity relates to the general absence of health measures (e.g., productivity) as opposed to disease measures in clinical trials. Gentle natural treatments may show smaller, more gradual improvements than may be expected from pharmaceutical and surgical interventions, and instruments sensitive to change are needed if trials are to be relatively short. Ultimately, the possibility of the superiority of natural health care in a few areas, such as longevity, the incidence of chronic disease in aging, and the incidence and prevalence of disease in progeny, ideally will require multidecade evaluation. Online tools available and in development for patient-reported outcomes hold potential for large, long-term studies with whole person measures.24a
Whole Practice Models
Given that individualized combinations of modalities and remedies generally are the rule in natural medicine and that their selection is guided by a health system’s own principles and guidelines, evaluation of models of whole practices is most likely to reveal their potential health benefit. The main difficulty that arises is in attaining replicability of the intervention. (See Chapter 2, Functional Medicine, for a presentation of a methodology for systematizing individualization.) Although they remain both innovative and challenging, a number of intervention models are possible in approaching the ideal.
The simplest is a fixed combination of remedies, although this approach abandons individualization. If it is applied in trials with specific inclusion and exclusion criteria, however, the loss of individualization may be tolerable. A slightly more ambitious approach to whole practice is a menu of choices applied to several possible etiologies in a symptom manifestation (disease) under study. As alternative medicines have been studied in recent years, pragmatic trials have been explored using a practitioner of given qualifications as the unit of intervention, treating the practitioner and his discipline as a “black box.” A further step in sophistication is the intervention driven by guidelines or algorithms from the principles and body of knowledge of the discipline. Such guidelines may be derived through expert practitioner panels. The design and replicability of best practice protocols derived by experts from discipline principles are being studied in naturopathic25 and Chinese medicines.26 The complex models that have resulted may be facilitated operationally by manualization for the guidance of practitioners, who must be well trained to assess patients and deliver a standardized intervention flexibly.27 A new question that would arise in such studies of manualized practice is the degree of adherence by the provider to the protocol. Another method is artificial intelligence programs that seek to emulate the decision-making processes of clinicians so that personalization can be standardized according to the algorithms practitioners use, often unconsciously.
Observational Studies
There are essentially two components to medical research: observation and experimentation. Observation documents the natural history of clinical conditions, the approach to health, and the outcomes of treatment. In observational studies, ideas of association are formed and hypotheses are generated. Experimentation in medicine evaluates the outcomes while varying practice. Experimentation tests hypotheses generated in a variety of ways, including from observational study. In CAM, where many variants of health care of unknown safety and effectiveness are already in practice, observational study holds an important place. In medicine, research questions often involve pursuing a desired outcome by manipulating the input variables. The input variables are discovered ultimately through observation, as are the models in which to study them. Both observation and experimentation are needed to validly evaluate medical practices. With an appreciation for the appropriate place of observational study, we should not expect observational studies to provide the information nor meet the methods desiderata for experiments. In the relatively little examined practices of natural medicine and CAM, “until we know more about the practice, scope of practice, patient characteristics, utilization rates, and patient numbers, it is difficult to design appropriate studies or to even know which issues are worth studying.”6
The challenge with results observed in evaluation of clinical practice is determining whether the observed association reflects a causative relationship. Although newer statistical tools, such as propensity scores and principal component analysis, may help, observational studies have been considered low on the hierarchy of validity in evaluation of medical practices. Effect sizes found in observational studies are often considered inflated, yet those from clinical trials and observational studies appear to be similar.28,29
Observational outcomes studies have some unique advantages in the study of natural medical practice. Outcomes research involves cohort studies wherein repeated health status evaluations in ordinary care settings compare outcomes of spontaneously arising variations of care, such as standard care and alternatives to it. Variations may be specific remedies, treatment approaches, or even whole professional practices. Advantages include the lack of ethical conflicts about withholding conventional treatment because the patients choose their practitioner. The effect of intention on the part of patient and practitioner is not abrogated. The medicine implemented is fully individualized with no constraint on the combinations considered best for a patient. The cost of research per case is a lesser constraint than in randomized trials if patients (or third parties) cover the cost of care. Costs can be included as an outcome. Mixed methods—quantitative and qualitative—in observational studies are usual and useful. More early observational studies will help to guide future trials. Best-case series, a retrospective review of selected cases, are having an effect on research directions in CAM.30
The care of patients currently using natural medicine is a valuable source of data for evaluating and monitoring care characteristics in different forms of practice. The growing coverage of CAM practitioners by third-party payers in health delivery systems over the past 20 years has generated much data on use and cost for currently licensed and insurance-covered natural medicine practitioners that can be used for comparison with outcomes of standard therapy.31 The study by Cherkin et al,32 designed on the model of the National Ambulatory Care survey, was remarkable in ascertaining the content of practice of state-licensed naturopaths, acupuncturists, chiropractors, and massage therapists.
The complexities of natural medicine (and any whole practice), with multiple inputs (patient characteristics and combination individualized treatments), broad measures, standardization data, and complex control conditions (e.g., “usual” care), both descriptive and analytical, all imply worthwhile data for exploration. For some CAM professions, the adoption of electronic medical records and the increased integration of CAM trained practitioners into conventional settings is generating rapidly growing databases in which clinical epidemiology may become common. To plan studies (including single-agent trials) and to understand results, the disciplines benefit from improving infrastructures for design, data collection, management, analysis, and interpretation. The NIH NCCAM has recognized the need and value of developing infrastructure at CAM institutions where there is clinical expertise in the practices.33 There is a growing cadre of cross-trained and productive research CAM practitioners to engage in high priority studies with interdisciplinary expertise.
Basic Science Studies
Data from the basic sciences ensure the causal chain linking a happy clinical observation and an intervention. They help the wider community understand and use the natural medicine professions’ experience. All of the natural medicine disciplines would benefit from greater exposure to the array of laboratory pharmacology and might, in turn, inform pharmacology of their own useful strategies. In the exploration of natural medical theory and in refining remedies, it would be useful to have access to a range of basic science techniques from analytical chemistry to classical microbiology to advanced molecular biology. Thousands of natural products have been screened for numerous laboratory outcomes. However, new understandings of human biology and advances in technology have made many more types of studies possible in areas such as combinatory pharmacodynamics, gene expression, proteomics, metabolomics, glycobiology, and immunology. Practitioners could guide the refinement of study questions from their best clinical judgment for future investigation.
Criteria for a Prioritized Research Program
The entire array of research design has a place in natural medicine research. There is a wide range of methodological discussion in publications and many currently ongoing studies spurred by substantial funding from the NIH. Methodologists have covered many problems in evaluation of different expressions of natural medicine, and the approaches and methods have been well summarized.34 However, resources are always limited; therefore, research should be prioritized. Some major criteria to consider in developing programs of research to be considered are the following:
1. Focus on diseases with the highest burden of human suffering and the greatest public health significance as well as those in which standard therapy falls short of cure and effective management.
2. Evaluate first the treatments that have the greatest likelihood of reducing the burden of suffering—the low-hanging fruit among natural medicine therapeutics and preventives. Two sources for targeting decisions in this regard are the evidence already present in the scientific literature and the opinions and data of experienced natural medicine practitioners.
3. Attend to methodological evolution and the necessary sequencing of studies to achieve long range goals and to explore for and validate positive clinical experiences. Some research questions require methodological innovation because we do not yet know exactly how to answer them (e.g., in evaluating the whole-systems approach to disease, long-term safety, ultradilutional and combinatorial effects, spiritual practices, and even measures of overall health). Important questions may call for complex designs that combine observational and experimental research. Clinical research usually requires preclinical preparation. The confidence of investigators and funders and the clarity of their direction improve with preliminary pilot and observational data.
4. Attend to building a practice improvement and discovery infrastructure for the future in those practice disciplines prepared for it.
1. Jonas W., Levin J. Essentials of complementary and alternative medicine. Philadelphia: Williams & Wilkins, 1999.
2. Micozzi M., ed. Fundamentals of complementary and alternative medicine. St. Louis: Saunders/Elsevier, 2010.
3. Natural medicines: comprehensive database, 5th ed, Stockton, CA: Therapeutic Research Faculty, 2003.
4. Jonas W.B., Linde K. Conducting and evaluating clinical research on complementary and alternative medicine. In: Gallin I., ed. Principles and practice of clinical research. San Diego: Academic Press, 2002.
5. Ahn A., Nahin R., Calabrese C., et al. Applying principles from complex systems to studying the efficacy of CAM therapies. J Altern Comple Med. 2010 Sep;16(9):1015–1022.
6. Coulter I.D., Khorsan R. Is health services research the Holy Grail of complementary and alternative medicine research? Altern Ther Health Med. 2008;14(4):40–45.
7. Thorpe K.E., Zwarenstein M., Oxman A.D., et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62(5):464–475.
8. Eisenberg D.M., Cohen M.H., Hrbek A., et al. Credentialing complementary and alternative medical providers. Ann Intern Med. 2002;137:965–973.
9. Dumoff A. Current Procedural Terminology (CPT) coding and the practice of integrative medicine. Seminars in Integrative Medicine. 2005 June;3(2):61–63.
9a. Dawkins R. The Selfish Gene. New York: Oxford University Press; 1976.
10. Tilburt J.C., Emanuel E.J., Kaptchuk T.J., et al. Prescribing “placebo treatments”: results of national survey of US internists and rheumatologists. BMJ. 2008;337:a1938.
11. Lewith G., Barlow F., Eyles C., et al. The context and meaning of placebos for complementary medicine. Forsch Komplementmed. 2009 Dec;16(6):404–412.
12. Finniss D.G., Kaptchuk T.J., Miller F., Benedetti F. Biological, clinical, and ethical advances of placebo effects. Lancet. 2010;375(9715):686–695.
13. Swanson C.S. A spirit-focused conceptual model of nursing for the advanced practice nurse. Issues Compr Pediatr Nurs. 1995;18:267–275.
14. Roberts K.T., Whall A. Serenity as a goal for nursing practice. Image J Nurs Sch. 1996;28:359–364.
15. Levin J. “And let us make us a name”: reflections on the future of the religion and health field. J Relig Health. 2009;48(2):125–145.
16. Dyson J., Cobb M., Forman D. The meaning of spirituality: a literature review. J Adv Nurs. 1997;26:1183–1188.
17. Pehler S.R. Children’s spiritual response: validation of the nursing diagnosis spiritual distress. Nurs Diagn. 1997;8:55–66.
18. Roberts L., Ahmed I., Hall S., Davison A. Intercessory prayer for the alleviation of ill health. Cochrane Database Syst Rev, 2009;2:CD000368..
19. Harris W.S., Gowda M., Kolb J.W., et al. A randomized, controlled trial of the effects of remote, intercessory prayer on outcomes in patients admitted to the coronary care unit. Arch Intern Med. 1999;159:2273–2278.
20. Benor D.J. Healing research: holistic energy medicine and spirituality. Munich: Helix Verlag; 1993.
21. Schlitz M., Braud W. Distant intentionality and healing: assessing the evidence. Altern Ther Health Med. 1997;3:62–73.
22. Targ E. Research methodologies for studies of prayer and distant healing. In: Lewith G., Jonas W.B., Walach H. Clinical research in complementary therapies: principles, problems and solutions. New York: Churchill Livingstone, 2002.
23. Institute of Medicine. National Academy of Sciences, Committee on the Use of Complementary and Alternative Medicine Use by the American Public. Complementary and Alternative Medicine in the United States. Washington, DC: National Academies Press; 2005.
24. Verhoef M.J., Vanderheyden L.C., Dryden T., et al. Evaluating complementary and alternative medicine interventions: in search of appropriate patient-centered outcome measures. BMC Complement Altern Med. 2006;6:38.
24a. Eton D.T., Bauer B.A., Sood A., Yost K.J., Sloan J.A. Patient-reported outcomes in studies of complementary and alternative medicine: problems, solutions, and future directions. Explore (NY). 2011 Sep-Oct;7(5):314–319.
25. Cooley K., Szczurko O., Perri D., et al. Naturopathic care for anxiety: a randomized controlled trial ISRCTN78958974. PLoS One. 2009;4(8):e6628.
26. Ritenbaugh C., Hammerschlag R., Calabrese C., et al. A pilot whole systems clinical trial of traditional Chinese medicine and naturopathic medicine for the treatment of temporomandibular disorders. J Altern Complement Med. 2008;14(5):475–487.
27. Schnyer R., Allen J.J. Bridging the gap in complementary and alternative medicine research: manualization as a means of promoting standardization and flexibility of treatment in clinical trials of acupuncture. J Altern Complement Med. 2002;8:623–634.
28. Benson K., Hartz A.J. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342:1878–1886.
29. Concato J., Shah N., Horwitz R.I. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342:1887–1892.
30. Jacobson J.S., Grann V.R., Gnatt M.A., et al. Cancer outcomes at the Hufeland (complementary/alternative medicine) klinik: a best-case series review. Integr Cancer Ther. 2005;4(2):156–167.
31. Lafferty W.E., Tyree P.T., Devlin S.M., et al. CAM provider use and expenditures by cancer treatment phase. Am J Manag Care. 2008;14(5):326–334.
32. Cherkin D.C., Deyo R.A., Sherman K.J., et al. Characteristics of visits to licensed acupuncturists, chiropractors, massage therapists, and naturopathic physicians. J Am Board Fam Pract. 2002;15:463–472.
33. NCCAM (National Center for Complementary and Alternative Medicine, NIH). Draft Stategic Plan, 2010. http://nccam.nih.gov/about/plans. Accessed 9/18/10
34. Lewith G., Jonas W.B., Walach H. Clinical research in complementary therapies: principles, problems and solutions. New York: Churchill Livingstone; 2002.