Shawn D. St. Peter

‘In God we trust; all others must bring data.’

—W. EDWARDS DEMING, PHYSICIST AND QUALITY IMPROVEMENT PIONEER

History

The initial groundwork forming the framework for EBM can be considered the earliest scientists who pursued explanatory truth instead of accepting beliefs. The process of discovering truth became replicable for aspiring scientists when the scientific method was developed. No one individual can be credited for creating the scientific method because it is the result of the progressive recognition for a natural process of acquiring facts. However, the earliest publication alluding to the steps of current scientific methodology may be found in Book of Optics published in 1021 by Ibn al-Haytham (Alhazen).² His investigations were based on experimental evidence. Furthermore, his experiments were systematic and repeatable as he demonstrated that rays of light are emitted from objects rather than from the eyes. In Western literature, Roger Bacon wrote about a repeating cycle of observation, hypothesis, and experimentation in the 1200s. Influenced by the contributions of many scientists and philosophers, Francis Bacon delineated a recognizable form of the scientific method in the 1620 publication of Novum Organum Scientificum.³ He suggested that mastery of the world in which man lives is dependent on careful understanding. Moreover, this understanding is based entirely on the facts of this world and not, as the ancients portrayed it, in philosophy. Nearly 400 years later, we find ourselves coming to the same conclusions in the practice of medicine and surgery. We now understand that facts and truths transcend experience, and that facts about optimal care can be gained through experimentation more reliably than from beliefs generated through experience.

The establishment of the scientific method in the pursuit of proven truths was fundamental to the development of EBM as an entity. The current scientific method consists of several steps that are outlined in Box 78-1. However, the concepts of EBM are not simply encompassed by the application of these steps to attain facts, but also address the ability to understand the value of the results generated from experimentation. Under the auspices of EBM, investigators are burdened with the first five steps, from inquiry to experimentation, while all caregivers must develop a deep understanding of experimental methodology, analysis of data, and how conclusions are drawn to be able to place appropriate value on published studies to influence their practice.

The movement urging physicians to utilize proven facts in the development of decision-making algorithms began in 1972 with the publication of the revolutionary book Effectiveness and Efficiency: Random Reflections on Health Services.⁴ The author, Archie Cochrane, a Scottish epidemiologist working in the UK National Health Service, has likely had the greatest influence on the development of an organized means of guiding care through data and results. The book demonstrates disdain for the scientific establishment, devalues expert opinion, and shows that physicians should systematically question what is the best care for the patient. Most impressively, Cochrane calls for an international registry of randomized controlled trials and for explicit quality criteria for appraising published research. At the time of his passing in 1988, these aspirations had not fully materialized. However, the field of medicine is fortunate that Cochrane’s prophetic ideas have precipitated the maturation of centers of EBM research that make up the global not-for-profit organization called the Cochrane Collaboration. The product of this collaboration is The Cochrane Library, which is a collection of seven databases that contain high-quality, independent evidence to inform health care decision-making.⁵ The most clinically utilized database is the Cochrane Database of Systematic Reviews containing reviews on the highest level of evidence on which to base clinical treatment decisions. There are now over 550,000 entries.

After Cochrane and other proponents of EBM showed the importance of comparative data, it took over a decade for the amorphous clouds of these concepts to solidify into tangible terms and usable methods. The methodology of specifically qualifying data and judging the relative merits of available studies did not begin to appear in the literature until the early 1990s. In a paper published in 1990 by David Eddy on the role of guidelines in medical decision-making, the term evidence-based first appeared in the literature.⁶ Much of the framework currently used to determine best available evidence was established by David Sackett and Gordon Guyatt at the McMaster University-based research group called the Evidence-Based Medicine Working Group. In 1992, JAMA published the group’s landmark paper titled, ‘Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine,’ and the term evidence-based medicine was born.⁷ A cementing moment in the paradigm shift occurred when the Centre for Evidence-Based Medicine was established in Oxford, England, in 1995 as the first of several centers. Over the past decade, centers or departments focusing on clinical research and EBM have been developed at universities and hospitals around the world, including our own (www.cmhclinicaltrials.com). EBM principles are now represented in the core curriculum that is integral to three of the six general competencies outlined by the Accreditation Council for Graduate Medical Education that oversees the accredited residency training programs in the United States. The concept of EBM is no longer a movement of progressive physicians, but rather a basic guiding principle of medical training and practice.

Levels of Evidence

Utilization of evidence in guiding health care decision-making requires an understanding of the merits of the evidence and the ability to decipher whether a given course of treatment has been proven superior. The quality of evidence specifically indicates the extent to which one can be confident that an estimate of effect is correct. Systems to stratify evidence by quality have been developed by several sources. Although there are specific differences in published rankings, the generally accepted levels in a broad sense are defined as follows:

The quality of data that is conveyed within each level and study type is clearly a wide spectrum. The complete delineation in the levels of evidence as defined by the Oxford Centre for Evidence-Based Medicine is outlined in Table 78-1.

As can be seen by the levels listed in Table 78-1, the strength of evidence improves significantly by the application of prospective data collection. In clinical medicine, and particularly in the practice of surgery, many aspects of trial design are not feasible such as blinding, placebo treatments, independent follow-up evaluation, and others. However, if one accepts these limitations and conducts a trial with prospective evaluation, the results remain markedly more meaningful than a retrospective case-control comparative series that compares surgeons and/or timeframes against one another.

The review of several studies can gain strength over an individual study, which is valid in many models and fields of medicine. However, one should be cautioned about the real strengths of such a meta-analysis before considering it to have a high level of evidence. The strength of these combined reviews is derived from the strength of the individual trials providing the numbers for the analysis. In the best scenario, such a combined review is composed of multiple prospective trials with similar design that each compares the effect of two treatments on a specific outcome. However, in the field of pediatric surgery, multiple prospective trials with similar designs that address the same disease with the same interventions are nonexistent. Also, surgeons should not overvalue the influence of combined reviews derived from retrospective studies. Such reviews should be interpreted as a mosaic of the individual studies. Any attempt by authors to combine a number of retrospective studies to harness statistical power is fraught with hazard.

Grades of Recommendation

The quality of the evidence as defined in Tables 78-1 and 78-2 applies a score of strength for each individual contribution in the literature. However, on many topics, particularly common clinical scenarios, there is an abundance of published studies such that the appropriate care cannot be guided by a single study. The total body of available information places caregivers in the difficult position of evaluating the published principles from the multiple sources that make up practice guidelines. The caregiver applying these clinical practice guidelines and other recommendations needs to know how much confidence can be placed in the recommendations from a conglomerate of citations. Strength of recommendation scales were born from this need.^8–12 Given that the level of evidence indicates the extent to which one can be confident that an estimate of effect is correct, the strength or grade of recommendation indicates the extent to which one can be confident that adherence to the recommendation will do more good than harm.¹³ As with the levels of evidence, there are multiple published grading scales. The grading format used by the Oxford Centre for Evidence-Based Medicine is outlined in Table 78-2.

The levels of evidence are more easily assessed because each contribution falls into a given level based on study design. However, establishing the grade of recommendation can be more complex given the fact that multiple levels of evidence from different timeframes invariably exist on any given clinical topic. This is further confusing because many disease processes have multiple outcome measures. Also, each treatment option can affect different outcomes in independent ways that must be balanced against the risk or toxicity of each treatment. A working group has outlined a process for establishing grade based on the following sequential steps:¹⁴

At this time, an accepted standard for respecting the level of evidence and assessing grades of recommendation is lacking in the pediatric surgery field.

Landscape of Pediatric Surgery

Progress in the field of pediatric surgery regarding EBM has been hindered by its relatively young age and the small size of its membership. As opposed to our adult general surgery colleagues who operate on a foundation built through centuries of work, the first American textbook in pediatric surgery, Abdominal Surgery of Infancy and Childhood by Robert Gross and William Ladd, was published in 1941. These men trained the initial cohort of pediatric surgeons who then developed other training centers around the country. The total number of training sites has remained relatively small. As a result, the philosophies of only a few people have been taught to the entire practicing population of pediatric surgeons. The practice of pediatric surgery has generally progressed without extensive critical analysis. This phenomenon is pronounced in the pediatric surgery literature, which has been replete with retrospective case-control studies and case series (levels 3 and 4). There are few publications in the pediatric surgery literature that entail comparative analysis, and most of these are retrospective in nature.

The future of care being directed by comparative research is brightening. There has historically been a great focus on basic science research when resource allocation is considered. However, despite the fact that government funding for basic science has far exceeded clinical research, the impact on clinical care relative to the investment has been low. The National Institutes of Health (NIH) is beginning to place greater emphasis on comparative effectiveness research versus classic basic science investigations. The Obama administration has recently approved $400M of funds to the NIH, while the Agency for Healthcare Research and Quality (AHRQ) received $300M from the Recovery Act legislation in addition to $400M that can be used at the discretion of the Secretary of Health for comparativeness effectiveness research.¹⁵ Comparative effective research is defined by AHRQ as research designed to inform health care decisions by providing evidence on the effectiveness, benefits, and harms of different treatment options. The evidence is generated from research studies that compare drugs, medical devices, tests, operations, or ways to deliver health care.

Retrospective Studies

As our field is overly represented with retrospective comparisons, it is important to recognize the natural flaws introduced by retrospective analysis. At least one of two confounding factors is present when an institution reviews and compares different therapies retrospectively. When an institution is concurrently using different treatment plans, usually the comparison between treatment plans also compares caregivers. The other possibility is that the center has changed treatment plans and then retrospectively investigates the effect of this change. Although this sounds attractive, it becomes a comparison between timeframes. Historical comparison groups are difficult to use with confidence because of the rapid ongoing changes in hospital systems. Many institutions have quality assurance programs that intend to make all patient care more consistent and efficient. Such change will impact outcomes regardless of the surgical treatment. A universal concern with retrospective studies is the assurance of reviewing the entire population and the entire dataset. It can be very difficult to accurately identify all the data desired when retrospectively collecting information. As a result, datasets are often incomplete, which decreases scientific confidence. Another issue, rarely acknowledged in retrospective studies, is the difficulty with capturing the entire population intended for study. There are many coding nuances within an institution that may exclude patients from lists when the database is searched by ICD-9 code. A specific type of patient, such as a treatment failure, can go undetected, because of being hidden under a different diagnosis code. This can create tremendous error in the published data.

Considerations for Prospective Trials in Pediatric Surgery

Although it is clear that prospective trials offer the highest scientific integrity and the best vehicle for determining superior efficacy among options, they have been underutilized in the field of surgery, and specifically, pediatric surgery. Critics have postulated that consented prospective trials would have a limited role in pediatric surgery because parents would be unwilling to enroll their children into such studies. Studies should not be conducted in any human population when there is substantial risk of harm balanced against questionable benefit. A well-designed study should offer patients the potential for a better outcome than the standard means of treatment outside the study. However, there is no example in the published literature to support the speculation that studies should not be attempted in children. Moreover, there are now several prospective, randomized trials being conducted and emerging in the pediatric surgery literature that would suggest otherwise.16–24

The specific details of trial development are beyond the scope of this chapter, but there are general considerations that can allow pediatric surgeons to identify situations for which a trial can be employed. Specifically, these include an understanding of equipoise or stepwise progression, the need for patient volume, an understanding of population characteristics, and the possibility of influencing practice habits.

Outcomes Research

Outcomes research is the analysis of outcomes and their predictors at different levels in the health care delivery system.²⁷ The critical difference between clinical trials research and outcomes or database research is the scope of view as depicted in Figure 78-1.²⁸ Clinical trials set inclusion criteria to create a desired population most likely to answer the question of interest. Thus, the patients are defined from the outset. Outcomes research uses all patients in the database. Clinical trials control for patient differences by randomization. In outcomes research, however, one can control for differences only in the analysis models. While some may argue outcomes research provides a better answer that is generalizable to the practicing community, the inability to sort out patients with confounding variables makes outcomes research more practical in addressing national, regional, and hospital trends where the view of individual patient management becomes blurred. Clinical trials provide more a clear picture to answer specific management questions. However, outcomes research can be utilized to benchmark quality of care, compare therapeutic effectiveness, refine management strategies, provide patient education, and produce important marketing information.²⁸

Resources

The application of EBM practices requires a functional knowledge of how to identify, interpret, and implement the information available into care plans and practice guidelines. Although many institutions have entire departments focusing on EBM utilization, not all physicians have ready access to the information. In the field of surgery, there are many examples of heterogeneous care plans within institutions and group practices despite convincing, high-level evidence. Also, there are many reasons for these often conflicting care plans. One is that personal bias for some providers is stronger than their ability to interpret data. Thus, they continue working in their comfort zone and are skeptical of sound data. Some of these surgeons completed training in the apprentice era, and they have continued to follow what their chief told them. Also, some physicians who understand the principles of EBM simply do not have wide access to the evolving body of EBM. To overcome some of these hurdles, several web-based support sites have been developed for both EBM tutorial and review of available studies. A few examples are listed in Table 78-3.

References