The context

This excerpt from an editorial by Davidoff et al. in the BMJ in 1995 still holds true today. EBM comprises the latest information on the most effective or least harmful management for patients (Davidoff et al., 1995). The key processes in EBM are:

Critical appraisal is ‘the process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision’ (Mark, 2008). It allows the reader to assess in a systematic way how strong or weak a paper is in answering a clinically relevant question or whether the paper can be relied on as the basis for making clinical decisions for patients.

Deficits in knowledge and understanding of the critical appraisal process restrict the implementation of the best available evidence into clinical practice. Recog nising the need to understand evidence regard ing treatment or diagnostic options for conten tious issues is the first step in the journey. This requires an acknowledgment of equipoise (one is not sure which is the better treatment or test). The clinician may then wish to explore the quality of evidence underpinning a proven treatment or test, even if they are aware of these.

Carefully formulating a precise, answerable question that is clinically relevant (i.e. that will provide improved care or a better diagnostic test) is the starting point (why). Without knowing the why, there is no point in starting the appraisal journey. The next step is to decide where to look for the evidence. Literature searches need to be efficient, comprehensive, unrestricted and unbiased, encompassing explicit search strategies of published, citable literature databases and sources of unpublished research. How to identify good-quality studies, critically appraise those selected and apply their findings to individual patient care completes the appraisal journey.

Most readers will not limit their literature search in terms of date of publication (when) unless they are confident that a treatment or test was developed only recently, or an unlimited search yields numerous citations, which become unmanageable. As there is often a time lag between study citations being added to searchable databases, it is worthwhile considering conducting a more recent targeted search in relevant journals if the topic area is rapidly evolving. Wang and Bakhai (2006) and Pocock (1983) provide excellent further reading in the area of clinical trials, as do Greenhalgh’s ‘Education and debate’ series from the BMJ (Greenhalgh, 1997a) and Gordon Guyatt’s 2000 focus series from the JAMA (Guyatt, 2000).

Before looking for individual studies, we recom mend a concerted search for metaanalyses, which offer a useful background perspective and, if one is lucky, may even answer the research question, using the summated ‘quality overall evidence’ available so far (Mark, 2008). Meta-analyses are formally designed and properly conducted critical appraisals of intervention trials that attempt to ‘aggregate’ outcome findings from individual studies if they show a consistent effect. The presence of compelling outcome effects, consistent in direction and size across individual non-clinically heterogeneous studies of acceptable methodological quality, will likely be enough to tell you whether a proposed treatment or diagnostic test will be suited to your patient.

Critics claim that such aggregated findings cannot be applied to an individual patient; on average, the treatment effect will be qualitatively similar (if it benefits meta-analysis patients, treatment would probably be effective in your patient) but quantitatively more variable (the magnitude of benefit will vary between patients). The conclusion reached by a meta-analysis will be applicable to your patient and specific clinical setting if these characteristics are comparable to those of the study patients or study settings included in the meta-analysis.

Critically appraising a meta-analysis will still save you time and effort if many intervention trials or studies of diagnostic performance of a certain test relevant to your objective have been carried out. You need to ascertain whether the methodological rigour and quality of the meta-analysis is sufficient for conclusions and recommendations contained in the meta-analysis to reliably fulfil your objectives. If a well-conducted and reported meta-analysis is available, re-examining individual studies in detail is less worthwhile, other than for personal interest. However, a meta-analysis will not include influential studies that become available after the date of publication of the metaanalysis. Carrying out a date-of-publication limited search for newly emerging studies is thus recommended, to see whether the conclusions reached in the meta-analysis remain consistent with the newer studies. For in-depth information on systematic reviews in health care, see the standard text by Matthias Egger et al. (2001).

There may be no relevant meta-analysis to inform your objectives. If there are too few studies, or the studies are relatively small, excessively clinically heterogeneous or dissimilar, or they show inconsistent and widely varying outcomes, an aggregated outcome in a meta-analysis may not be possible. In these situations, decisions regarding patient management or investigation are based on a critical appraisal of individual studies you believe to be relevant to your practice setting, with the clinical risk–benefit analysis tailored to suit the individual needs of your patient, as well as taking into account treatment feasibility, practicability and availability.

Critical appraisal and clinical practice

Standards of clinical care now demand identification and timely delivery of the most effective treatment available in order to achieve optimal outcomes. There are high expectations among colleagues, patients and health administrators that treatments are clinically effective, cost-effective and timely with minimal adverse effects. In emergency medicine this may also involve the further consideration of time-critical situations. Treatment selection by anecdote, eminence or prior personal experience is no longer acceptable. Emergency physicians must make active, informed decisions regarding treatment selection and not simply default to in-patient teams.

In emergency medicine, critical appraisal of the evidence is most pertinent to time-critical conditions that require non-established or contentious urgent treatments that may be highly beneficial but also lead to significant harm. For example, this situation arises in thrombolytic treatment for acute ischaemic stroke, where treatment administered within three hours of symptom onset gives better neurofunctional outcome, but remains little used for fear of causing intracranial bleeding. ECASS III, a recently published RCT comparing IV alteplase with a placebo in ischaemic stroke, found alteplase to remain beneficial at three to four and a half hours after symptom onset (Hacke et al., 2008). The most recent Cochrane meta-analysis of thrombolysis trials in stroke, published in 2003, did not include ECASS III (Wardlaw et al., 2003). Evidence is in a constant state of evolution, so critical appraisal is a continuing process that aligns itself with continuing medical education and professional development. Nowadays, studies informing on therapeutic (in) effectiveness are easily and rapidly accessible through user-friendly information technology media such as the 24-hour medical cybrary. With the exception of acute resuscitation, there is never an excuse not to evaluate effectiveness prior to patient treatment.

High-standard clinical care requires the clinician to correctly select and safely deliver the best available treatment or diagnostic test for each patient in a timely fashion. A poor outcome for a patient receiving the most effective available treatment or a consequential diagnosis missed despite use of the most reliable test is ethically and medico-legally more defensible than the same adverse events in a patient after suboptimal treatment or not receiving an appropriate diagnostic test. The clinician who knows that a poor patient outcome has not resulted in some way from a knowledge gap will sleep the better for it.

Within the realm of clinical research, an unbiased comprehensive literature search is able to identify whether a research question has been answered in previous studies, and therefore whether another study is necessary or even ethical in the presence of compelling evidence. The potential impact on improving patient care and clinical relevance of a proposed study is also assessable by critical examination of the available literature. Furthermore, peer review of medical research manuscripts requires critical appraisal of a study’s internal validity as it relates to methodological rigour and its capability to be generalised to various clinical settings.

Barriers and challenges remain in keeping up to date with the latest evidence. The time pressures of increasing demand for hands-on patient care discourage evidence appraisal. This is exacerbated by perceptions that clinical care is distant and divorced from medical research, engendering the negative connotation that critical appraisal by the ‘thinker’ clinician is a diversionary activity of little relevance to direct patient care rendered by the ‘doers’. Negative perceptions exist that medical research has become an industry with little relevance to clinical practice.

Exponential growth in the medical literature and the increasing ease of access to biomedical journals has produced a ‘noise to signal ratio’ that can easily overwhelm the time-pressured clinician. Successfully identifying, or conversely not missing, crucial studies can be a challenge. The following sections provide a framework to help with this important task.

A tool kit of EBM techniques