Primary Research

Single-Study Results—Critically Appraised Topics

Determining which studies provide information useful in the care of patients is largely a question of deciding whether a study is valid and, if so, can its results be applied to the patients in question. One format for appraising individual studies is the critically appraised topic (CAT) format that has been popularized as part of EBM. The purpose of the CAT is to evaluate a given study or set of studies using a standardized approach. Studies that address diagnosis, prognosis, etiology, therapy, and cost-effectiveness all have a separate CAT format.³ An example is shown in Box 227-1 for studies that address therapy. The CAT format for studies on therapy asks several questions intended to address the issues of validity and clinical utility. Studies that fail to achieve these measures are not generally useful, although studies do not necessarily have to fulfill every criterion, depending on the nature of the topic. For example, a study that examined the effect of walking once a day for the prevention of stroke would not be expected to include a detailed examination of side effects or a cost-effectiveness analysis. However, a study comparing streptokinase to placebo for treatment of stroke would likely be required to include a detailed examination of side effects and a cost-effectiveness analysis because of the excessive risks and costs associated with such therapy. Similarly, blinding may not always be possible, and the effects of the investigators being unblinded can be minimized by separating them from the clinicians making the treatment decisions or by establishing standard treatment protocols that are applied equally to both the study and control groups. Alternatively, a study would be “fatally flawed” if it failed in terms of randomization or was not analyzed as “intention to treat.” There are a number of other useful tools for assessing study design and for quantifying effect size and cost-effectiveness. In general, these are the tools of epidemiology and biostatistics, and their discussion is beyond the scope of this chapter. A basic primer and glossary of terms is included in Table 227-2.

Systematic Reviews of Multiple Studies

A systematic literature review combines the results of multiple studies through the systematic search, assembly, and appraisal of existing primary research on a given subject. Meta-analysis is a type of systematic review that incorporates a quantitative summary of the data, which combines actual data from several small although high-quality studies. Criteria for reviews to be systematic as opposed to narrative (see later) are quite explicit. All systematic reviews should start with a four-part (three-part when applicable) question, as described previously. Both the search criteria and inclusion and exclusion criteria should be predefined. The review should combine only RCTs or discuss how and why it is combining different types of evidence. Additionally, the methods section should provide search terms and key words, thus establishing some degree of reproducibility.

The advantages of systematic reviews are that by pooling many studies, the power to find a true effect is increased. This is particularly important when many well-done but small and inconclusive studies have attempted to answer a particular question. Systematic reviews often represent an exhaustive effort to find all related information in a given area. In this regard, they provide an excellent summary of the literature up to the date of the review.

The disadvantage of systematic reviews is that they are only as good as the studies they include and can only be interpreted if all the criteria just mentioned have been met. Unfortunately, there is considerable variability in the quality and comprehensiveness of available systematic reviews. Much of this dilemma stems from a lack of commonly accepted methodology for conducting and writing systematic reviews. For example, there are no standard exclusion criteria for studies in systematic reviews. Each author establishes the criteria, which the reader must assess to determine the quality and utility of the review to answer his clinical question. In addition, there is publication bias. Popular search techniques to identify studies are inherently limited by the fact that unpublished studies are unaccounted for in any review. Issues such as these have led authors to propose the development and maintenance of study registries where all RCTs are registered irrespective of their publication status.¹¹ This would enable review of smaller studies and those studies published in journals not listed in cumulative Index Medicus, MEDLINE, and other popular databases in systematic reviews.

Narrative Reviews of Multiple Studies

The most common system of non–peer-reviewed pooling of study results is the familiar “review” article or collection of reviews. This textbook is an example of the latter. Articles or chapters combine information from several primary articles, sometimes a few hundred, in a way that is digestible by the average reader. Reviews may be focused on recent advances, or they may provide a complete tutorial on a given subject. In either case, in the traditional method known as the narrative review, the methodology is the same: an author, presumably someone knowledgeable of the subject matter, reviews the existing literature in some way, formulates an opinion, and disseminates this opinion along with references to support each argument. This approach is also used in the discussion section of most original articles, in which the authors attempt to discuss their findings in the context of the existing literature.

The advantage of narrative reviews is that they provide a detailed qualitative discussion, usually by an expert with years of experience. However, they do have several limitations. The most important of these is that evidence used to support the author’s positions is not collected, evaluated, and compared in an organized and reproducible manner. That information is complete or that it is judged in an unbiased manner cannot be assured. Journal articles are often peer reviewed, which provides some limited oversight for completeness and lack of bias, but this is far from perfect. Furthermore, review articles and textbook chapters are not generally subject to vigorous review and therefore may be the least reliable sources of information, particularly current information. For example, by 1988, fifteen studies had been reported on the use of prophylactic lidocaine in acute myocardial infarction. While no single study was definitive, pooled data from the nearly 9000 patients showed that the practice was useless at best. Nonetheless, by 1990 there were still more recommendations for its use than against it appearing in textbooks and review articles.¹²

Bedside Decision Making

The goal of EBM is to facilitate bedside decision making by placing evidence in the context of clinical judgment and the preferences of the patient.¹⁵ There is often sufficient medical evidence to influence a number of daily decisions. Therefore, the clinician should always ask, “Is my patient receiving the best level of care as indicated by the evidence in the literature? Are there any study protocols or results that could be applied to this patient that currently are not?” Clinicians should also recognize knowledge deficits and be alert for opportunities to formulate EBM questions during daily rounds or routine patient care. Once the evidence is found and deemed useful, it must be judiciously applied. Clinicians must use their knowledge and experience to understand how to apply the results of studies to individual patients. Some cases, owing to patient- or environment-specific circumstances, may be sufficiently unique to render even good evidence inappropriate. Individual patient or family values and expectations could also direct therapy in one direction when medical evidence and physician judgment would have led it in another.

Guidelines and Protocols

Perhaps a natural extension of EBM is the desire to standardize care when evidence can be found for treatments or diagnostic procedures that are cost-effective. When such therapeutic or diagnostic strategies exist, they should be widely applied. A convenient way to ensure this is to develop a protocol or guideline. Protocols and guidelines are especially useful for common illnesses and procedures and have the advantage of allowing an institution to implement EBM even in the presence of physician lack of expertise in EBM. However, developing and maintaining protocols and guidelines is extremely labor intensive because the EBM criteria for guideline validity are explicit. Sackett states, “We should think of [a guideline] as having two distinct components: first the evidence summary, and second, the detailed instructions for applying that evidence to our patient.”¹³ The evidence summary consists of a recent review of the literature both for and against the guideline.³ The applicability of the guideline in each clinical situation with particular patient and institutional characteristics is assessed in the same manner as other evidence.

Generating Evidence

It is impossible to practice EBM without a body of evidence in the literature. Until recently, there was little strong evidence supporting particular care paradigms in the critically ill. There are now a large number of studies guiding a wide set of critical care problems,^16–21 whereas other elements of care remain largely empirical. Why has our field had such difficulty conducting clinical trials? There are a number of reasons. First, critical illness occurs in a heterogeneous group of patients in whom treatment effects may be small. Narrow selection criteria may introduce bias, and smaller sample sizes may not show an effect. Second, investigators must ensure the novel therapy is tested against “current best methods of care.” Since a study will be interpreted in the light of likely treatment patterns at the completion of a trial rather than the initiation, recent strong evidence should be promoted in both arms of a trial. But the large number of recent critical care trials combined with the financial and practical difficulties of implementing all of the changes has made “current best methods of care” an evolving process that remains a constantly moving target. Third, the choice of appropriate outcomes continues to be debated in critical care. The historic choice of 28-day (or 30-day) mortality rate, which has been used as the primary outcome in most critical care trials,²² has been criticized as arbitrary and incomplete. There is growing recognition that clinical research has to define and focus on the outcomes most meaningful to patients and society, including quality of life, functional status, freedom from pain and other symptoms, and satisfaction with medical care.⁸

Reporting Results

Another threat to the validity of EBM is the accessibility to evidence as a function of study results reporting. Randomized trials can yield biased results if they lack methodological rigor, and it may be difficult to determine their flaws if they are not reported accurately. Unfortunately, authors of many trial reports neglect to provide lucid and complete descriptions of critical information needed to judge the methodological rigor and hence the validity of the results. In response to this problem, a series of Consolidated Standards of Reporting Trials (CONSORT) statements were published beginning in 1996.²³ Most recently, these statements have been updated with CONSORT 2010.²⁴ Figure 227-1 shows a flow diagram for reporting information on research subjects in a parallel randomized trial of two groups. CONSORT 2010 also provides a 25-point checklist for information to include when reporting a randomized trial. The hope is that by improving and standardizing trial reporting, evidence appraisal will be more objective and overall evidence quality will improve.

Figure 227-1 Flow diagram of progress through phases of a parallel randomized trial of two groups (i.e., enrollment, intervention allocation, follow-up, and data analysis).

(From Schulz KF, Altman DG, Moher D; for the CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Ann Intern Med 2010;152:726-32.)

Practicing When Evidence Is Lacking

Although the application of EBM has produced very useful information to guide therapy²⁵ and further research,²⁶ it has also generated considerable controversy.^27–29 The disagreement is not over recommendation of practices based on sound evidence, but instead whether these practices should be avoided when evidence is lacking. Thus clinicians are weary of being told they and their patients cannot pursue diagnostic and therapeutic choices because there is no evidence these practices work. In this regard, it is important to note one of the basic principles of EBM: “not finding an effect is not the same as finding no effect.” Stated differently, the lack of evidence that something works is not evidence that it does not work. This issue is particularly relevant to critical illness where, by definition, patients are seriously ill and often do not respond to therapy. Should treatment that is possibly effective be withheld from patients with otherwise lethal conditions on grounds that it is unproved?

For new therapies, there are already evidence-based standards in place for evaluation and approval.³⁰ However, numerous therapies are in use in the ICU today without proven efficacy, and many others, for which there may be proof in one patient population, are being prescribed in another. Unfortunately, there may be significant barriers to obtaining evidence for these practices. For example, funding agencies and corporations may be unwilling to study therapies that are no longer patented. Furthermore, placebo-controlled studies are often impossible to conduct because clinicians find it unethical to withhold “standard” therapies. Efforts to use “lack of evidence” to justify withholding these therapies should be tempered by these and the following considerations:

Clinicians routinely grapple with these issues even for therapies that have proven to be effective. The risk-benefit ratio for any therapy is patient specific, and the clinician must judge the probability for benefit or harm to each individual patient. Evidence-based guidelines can be useful in helping clinicians and patients make these decisions, but they cannot take the place of clinical judgment. Treatments that are proven to be useless or even harmful should be avoided unless compelling evidence exists for their use in a specific patient. However, restrictions on existing therapy on the grounds that this therapy is unproven must be developed with great caution.

Barriers to Applying Evidence

When we see patients, we are responsible for applying EBM in the management of their clinical problems. But even when the evidence is strong and the patient is in agreement with the plan, powerful impediments often bar our way.³¹ As has been experienced by other specialties, critical care medicine has many logistical barriers to implementing EBM at the level of the clinician or the institution and regionally/nationally.

At the level of the clinician, there are a number of potential barriers. First, each step of EBM practice is difficult. For example, generating specific, patient-centered questions is difficult when patients suffer from poorly defined conditions with unknown underlying pathophysiology and uncertain outcome. Because of the relative paucity of available evidence, searching for the right article can be something akin to searching for a needle in a haystack. Second, there are time pressures in clinical practice. This is true both for finding and appraising evidence and for developing the skills of practicing EBM. Whereas this may be true in some circumstances, many non-urgent decisions (e.g., when to restart feeds, ventilator weaning, ulcer prophylaxis) are made in ICUs every day. All such decisions could benefit from thoughtful consideration of the evidence. And after an emergent situation, the clinician (in training) could identify any questions about the course of action taken and review later what evidence exists in such a circumstance. Fortunately, electronic databases are increasingly making this concern less of an issue. Last, clinicians are largely responsible for implementing EBM on their own initiative. If they lack the skills or confidence to apply best evidence to their patient care, nothing will change.

At the institutional level, commitment to practicing EBM requires both philosophical and financial support. Regular revision of institutional guidelines and protocols is time consuming to conduct, expensive to implement, and full implementation of best care practices may require changes in the array of clinical services available to patients. Purchase of new medical technology or establishing new clinical units can be very expensive. In the current healthcare environment, many hospitals are likely to be hesitant about such expenses, especially if the evidence is relatively young and the practice not firmly established.

Regionally and/or nationally, implementing EBM requires enormous resources in the effort to continually educate physicians and insurers and to update policies. State and federal governments have to consider medical education requirements and how compliance with policies and guidelines will be defined and enforced. Effective strategies to communicate policy changes and updates in guidelines must also be developed. Because regional and national systems are responsible for socially and geographically diverse healthcare environments, the aforementioned issues must be adaptable to local needs and circumstances.

Although the obstacles are significant, we are not without resources to overcome them. Paralleling the evolution of EBM has been research into how evidence can be implemented into practice—so-called implementation research.³² Understanding how individuals and institutions absorb evidence and implement change has, in select cases, translated into fundamental improvements in health care. We have also learned much about barriers to research transfer through our failed attempts to modify behavior. Success in modifying a discrete aspect of medical practice has invariably been achieved through multidisciplinary strategies that meld concepts and techniques from epidemiology, education, marketing, psychology, sociology, and economics.