The medical literature

Published on 14/03/2015 by admin

Filed under Emergency Medicine

Last modified 22/04/2025

Print this page

This article have been viewed 1261 times

Chapter 9 The medical literature

As we acquire more knowledge, things do not become more comprehensible, but more mysterious.

Albert Schweitzer

This chapter is presented in three parts:

• Part A introduces the basic statistics involved in the everyday practice of ordering and interpreting investigations, presented from the perspective of the clinician, not the statistician.

• Part B outlines the process of evidence-based medicine (EBM). This is particularly pertinent for the publication/presentation requirement of advanced training and ACEM Fellowship (see Chapter 8), as well as providing the structure for critical appraisal of the medical literature relevant to all sections of the exam.

• Part C provides key articles, a synopsis of current topical areas where further reading is required and a list of some useful resources on consensus treatment.

This chapter may be utilised in several ways. Those trainees commencing their approach to the fellowship exam, who have the time and motivation to use EBM techniques as a cornerstone of their preparation, may find this a useful summary of what is most relevant to have in their ‘tool kit’. Alternatively, those closer to the event may choose to focus their efforts on using the material as a review of information that could be asked in the exam — EBM would lend itself well to being an SAQ or SCE topic. The important papers section has relevance to all parts of the exam for trainees at all stages of their preparation.

Part A: the emergency physician’s guide to basic statistics

Consider the following:

A test has a specificity of 99% and sensitivity of 99% in a population where the prevalence of disease is 1%. What is the positive predictive value (PPV) of this test?

If, like most clinicians, reading this question makes your eyes glaze over and your head ache, this section is for you. We will work our way through the answer step-bystep and by the end of this section, your headache will be gone!

First, consider the possible results of a test (positive or negative) when a disease (or disorder) is present or absent. The possible combinations are shown below.

For ease of calculations, we will assume a population size of 10,000. Prevalence is the number of people in the population with the condition at a given time. So a prevalence of 1% in our population of 10,000 will therefore equal 100 people with the disease. Prevalence should not be confused with incidence, which is the number of presentations per unit of time. For example, the annual incidence of diabetes is the number being diagnosed each year, whereas the prevalence (the number of people who have diabetes) is much higher.

Sensitivity is the capacity to detect something when it is present — just like a sensitive person does. In statistical terms it is best thought of as ‘positivity in the presence of disease’ = TP/(TP + FN). Tests with high sensitivity are preferable if the desire is to ensure that a condition is detected or ‘ruled in’. For our case, a sensitivity of 99% will result in a total of 99 out of the 100 patients with the disease returning a (true) positive test and one returning a (false) negative test.

Specificity is the ability of a test to pick only the disease — just as being specific means not getting off the point. In statistical terms, specificity can be thought of as ‘negativity in the absence of disease’ = TN/(TN + FP). Tests with high specificity are preferable when it is important to ensure that a condition is not present, i.e. ‘ruled out’. For our example, a specificity of 99% will result in 99 (1%) of the 9,900 without the disease still returning a (false) positive test.

The table can now be completed with simple arithmetic.

So far, we have been working backwards from a given population with a known disease prevalence to calculate true and false negatives and positives. However, this is not the world of a clinician with a test result. The result returned will be either positive or negative (at least for the sake of this discussion). The question that the clinician must ask is: ‘What does a positive (or negative) test mean?’

Positive predictive value (PPV) and negative pre dictive value (NPV) are the likelihood that a positive (or negative) test is a true result, i.e. what pro portion of positive results are true positives and what proportion of negative results are true negatives.

• PPV = TP/(TP + FP)

• NPV = TN/(TN + FN)

This is of direct importance to you as the clinician, as it tells you whether or not you can rely on the result you have. Depending on whether the test was intended to ‘rule in’ or ‘rule out’ a particular condi tion, the focus will be more on the PPV or the NPV, respectively.

The final basic statistical value we can also now cal culate is accuracy. Accuracy is the proportion of time the test is correct (TP or TN) for the given population.

Accuracy = (TP + TN)/(TP + TN + FP + FN)

This now enables us to answer the original question. A test with a 99% sensitivity and specificity when applied to a population with a disease prevalence of 1% will have a positive predictive value of only 50%.

From this you should now be able to appreciate that prevalence of disease has a significant impact on the clinical interpretation of a test in addition to simply evaluating the sensitivity and specificity. A practical example of this is assigning pre-test prob abilities to ventilation/perfusion scanning for suspected pulmonary embolism. Assigning a pre-test probability defines the prevalence of disease and hence alters the interpretation of the test result, even though the same test has been performed!

Below is the original question with a prevalence of 10%.

Note, the PPV has now increased dramatically. If your confidence in basic statistics has now grown (and your headache has gone), try different combinations and permutations of parameters to confirm the effect on PPV and NPV. To start with, review a paper or even work through the detail for an investigation you perform on a regular basis. You may never view basic statistics with fear again!

Part B: an overview of EBM

Education’s purpose is to replace an empty mind with an open one.

Malcolm Forbes

The context

Doctors need to know about the studies that show whether new ideas work, but their volume has grown enormously. What’s more, many are published in inaccessible places, are not published at all, or are seriously flawed. Most busy doctors lack the time or skill to track down and evaluate this evidence. Although the skills of searching for evidence and critically appraising it are being mastered by growing numbers of doctors, many cannot keep up. Consequently there is a widening chasm between what we ought to do and what we actually do.

This excerpt from an editorial by Davidoff et al. in the BMJ in 1995 still holds true today. EBM comprises the latest information on the most effective or least harmful management for patients (Davidoff et al., 1995). The key processes in EBM are:

1. formulating the management or clinical research question to be answered

2. searching the literature and online databases for applicable research data

3. appraising the evidence gathered with regard to its validity, relevance and generalisability

4. integrating this appraisal with knowledge about the unique aspects of the patient (including patient preferences) (Mark, 2008).

Critical appraisal is ‘the process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision’ (Mark, 2008). It allows the reader to assess in a systematic way how strong or weak a paper is in answering a clinically relevant question or whether the paper can be relied on as the basis for making clinical decisions for patients.

Deficits in knowledge and understanding of the critical appraisal process restrict the implementation of the best available evidence into clinical practice. Recog nising the need to understand evidence regard ing treatment or diagnostic options for conten tious issues is the first step in the journey. This requires an acknowledgment of equipoise (one is not sure which is the better treatment or test). The clinician may then wish to explore the quality of evidence underpinning a proven treatment or test, even if they are aware of these.

Carefully formulating a precise, answerable question that is clinically relevant (i.e. that will provide improved care or a better diagnostic test) is the starting point (why). Without knowing the why, there is no point in starting the appraisal journey. The next step is to decide where to look for the evidence. Literature searches need to be efficient, comprehensive, unrestricted and unbiased, encompassing explicit search strategies of published, citable literature databases and sources of unpublished research. How to identify good-quality studies, critically appraise those selected and apply their findings to individual patient care completes the appraisal journey.

Most readers will not limit their literature search in terms of date of publication (when) unless they are confident that a treatment or test was developed only recently, or an unlimited search yields numerous citations, which become unmanageable. As there is often a time lag between study citations being added to searchable databases, it is worthwhile considering conducting a more recent targeted search in relevant journals if the topic area is rapidly evolving. Wang and Bakhai (2006) and Pocock (1983) provide excellent further reading in the area of clinical trials, as do Greenhalgh’s ‘Education and debate’ series from the BMJ (Greenhalgh, 1997a) and Gordon Guyatt’s 2000 focus series from the JAMA (Guyatt, 2000).

Before looking for individual studies, we recom mend a concerted search for metaanalyses, which offer a useful background perspective and, if one is lucky, may even answer the research question, using the summated ‘quality overall evidence’ available so far (Mark, 2008). Meta-analyses are formally designed and properly conducted critical appraisals of intervention trials that attempt to ‘aggregate’ outcome findings from individual studies if they show a consistent effect. The presence of compelling outcome effects, consistent in direction and size across individual non-clinically heterogeneous studies of acceptable methodological quality, will likely be enough to tell you whether a proposed treatment or diagnostic test will be suited to your patient.

Critics claim that such aggregated findings cannot be applied to an individual patient; on average, the treatment effect will be qualitatively similar (if it benefits meta-analysis patients, treatment would probably be effective in your patient) but quantitatively more variable (the magnitude of benefit will vary between patients). The conclusion reached by a meta-analysis will be applicable to your patient and specific clinical setting if these characteristics are comparable to those of the study patients or study settings included in the meta-analysis.

Critically appraising a meta-analysis will still save you time and effort if many intervention trials or studies of diagnostic performance of a certain test relevant to your objective have been carried out. You need to ascertain whether the methodological rigour and quality of the meta-analysis is sufficient for conclusions and recommendations contained in the meta-analysis to reliably fulfil your objectives. If a well-conducted and reported meta-analysis is available, re-examining individual studies in detail is less worthwhile, other than for personal interest. However, a meta-analysis will not include influential studies that become available after the date of publication of the metaanalysis. Carrying out a date-of-publication limited search for newly emerging studies is thus recommended, to see whether the conclusions reached in the meta-analysis remain consistent with the newer studies. For in-depth information on systematic reviews in health care, see the standard text by Matthias Egger et al. (2001).

There may be no relevant meta-analysis to inform your objectives. If there are too few studies, or the studies are relatively small, excessively clinically heterogeneous or dissimilar, or they show inconsistent and widely varying outcomes, an aggregated outcome in a meta-analysis may not be possible. In these situations, decisions regarding patient management or investigation are based on a critical appraisal of individual studies you believe to be relevant to your practice setting, with the clinical risk–benefit analysis tailored to suit the individual needs of your patient, as well as taking into account treatment feasibility, practicability and availability.

Critical appraisal and clinical practice

Standards of clinical care now demand identification and timely delivery of the most effective treatment available in order to achieve optimal outcomes. There are high expectations among colleagues, patients and health administrators that treatments are clinically effective, cost-effective and timely with minimal adverse effects. In emergency medicine this may also involve the further consideration of time-critical situations. Treatment selection by anecdote, eminence or prior personal experience is no longer acceptable. Emergency physicians must make active, informed decisions regarding treatment selection and not simply default to in-patient teams.

In emergency medicine, critical appraisal of the evidence is most pertinent to time-critical conditions that require non-established or contentious urgent treatments that may be highly beneficial but also lead to significant harm. For example, this situation arises in thrombolytic treatment for acute ischaemic stroke, where treatment administered within three hours of symptom onset gives better neurofunctional outcome, but remains little used for fear of causing intracranial bleeding. ECASS III, a recently published RCT comparing IV alteplase with a placebo in ischaemic stroke, found alteplase to remain beneficial at three to four and a half hours after symptom onset (Hacke et al., 2008). The most recent Cochrane meta-analysis of thrombolysis trials in stroke, published in 2003, did not include ECASS III (Wardlaw et al., 2003). Evidence is in a constant state of evolution, so critical appraisal is a continuing process that aligns itself with continuing medical education and professional development. Nowadays, studies informing on therapeutic (in) effectiveness are easily and rapidly accessible through user-friendly information technology media such as the 24-hour medical cybrary. With the exception of acute resuscitation, there is never an excuse not to evaluate effectiveness prior to patient treatment.

High-standard clinical care requires the clinician to correctly select and safely deliver the best available treatment or diagnostic test for each patient in a timely fashion. A poor outcome for a patient receiving the most effective available treatment or a consequential diagnosis missed despite use of the most reliable test is ethically and medico-legally more defensible than the same adverse events in a patient after suboptimal treatment or not receiving an appropriate diagnostic test. The clinician who knows that a poor patient outcome has not resulted in some way from a knowledge gap will sleep the better for it.

Within the realm of clinical research, an unbiased comprehensive literature search is able to identify whether a research question has been answered in previous studies, and therefore whether another study is necessary or even ethical in the presence of compelling evidence. The potential impact on improving patient care and clinical relevance of a proposed study is also assessable by critical examination of the available literature. Furthermore, peer review of medical research manuscripts requires critical appraisal of a study’s internal validity as it relates to methodological rigour and its capability to be generalised to various clinical settings.

Barriers and challenges remain in keeping up to date with the latest evidence. The time pressures of increasing demand for hands-on patient care discourage evidence appraisal. This is exacerbated by perceptions that clinical care is distant and divorced from medical research, engendering the negative connotation that critical appraisal by the ‘thinker’ clinician is a diversionary activity of little relevance to direct patient care rendered by the ‘doers’. Negative perceptions exist that medical research has become an industry with little relevance to clinical practice.

Exponential growth in the medical literature and the increasing ease of access to biomedical journals has produced a ‘noise to signal ratio’ that can easily overwhelm the time-pressured clinician. Successfully identifying, or conversely not missing, crucial studies can be a challenge. The following sections provide a framework to help with this important task.

Levels of evidence

Intervention and non-intervention studies can be stratified into several ‘levels of evidence’, according to their internal validity and dependability in informing treatment effects. A well-designed and conducted meta-analysis or randomised controlled blinded treatment trial is widely recognised as being able to offer the most reliable and least biased estimate of treatment benefit or harm (Wang et al., 2006), followed in descending order of quality of evidence by observational non-intervention studies such as case control studies and finally case series and case reports. This is variously graded (e.g. levels I–IV or grade A–C recommendations) depending on the body utilising this. Several issues should be apparent at this stage:

• For trainees, the task of tackling the mountain of literature and targeting the most relevant items to influence practice and prepare for the fellowship exam can be daunting.

• Acquiring the skills necessary to do this will be a lifelong investment in ensuring ongoing professional development.

• For some candidates, the process of seeking out and analysing existing and new offerings will become a pleasurable as well as a necessary pastime. Many will hopefully make important contributions themselves.

ACEM recognises the importance of EBM and research as being crucial for the future of the specialty. It is expected that fellowship exam candidates will be aware of major practice informing papers. The purpose of regulation 4.10 is to ensure that all trainees have exposure to research during their training. This is important so that individuals with a predilection for research can self-select and be supported in their future development.

For the working trainee who is approaching the exam, acquiring and applying EBM skills to each topic on the fellowship curriculum can be daunting. Some would say that this is unrealistic, when so many other priorities exist and time is short. Textbooks and exam-focused resources that have incorporated relatively recent clinical evidence provide an attractively efficient way to capture, in digestible portions, the latest controversy or ‘hot topic’ in emergency medicine. A great advantage to this approach is that someone else has already critically appraised the key papers for you, saving you from having to do this yourself. However, books are revised only periodically (usually every few years), so what was topical or controversial when a book was written may now be passé or resolved and no longer an attractive topic in the fellow ship exam.

In practice, the greatest proportion of a consultant emergency physician’s work time is spent ‘on the floor’, caring for patients directly or providing clinical supervision for registrars and residents. With multiple non-clinical tasks required for a functional Emergency Department, time for critical appraisal of EBM topics may be difficult to access. In terms of emergency medicine advanced training, the bread and butter of clinical emergency medicine remains the focus of the fellowship examination. The regulation 4.10 requirement and an occasional question during the fellowship examination on critical appraisal of the evidence are used to assess the trainee’s capacity for skilled self-directed learning and evidence analysis. In the process of achieving the latter aim, some trainees will aspire to becoming research leaders in the future, boosting the research credibility of our specialty.

A tool kit of EBM techniques

Was the analysis of the trial findings valid?

Intention to treat analysis

All patients should be analysed in the group to which they were randomised. Loss to follow-up greater than 20%, especially if differentially distributed between groups, will lead to post-randomisation bias if intention to treat (ITT) analysis is not used. ITT analysis means that patients are analysed according to the treatment group to which they were random ised, irrespective of whether they underwent the intended intervention or whether they adhered to protocol stipulations. ITT analysis results in an unbiased estimate of effect and more closely reflects what happens in reallife clinical practice, where patients have a range of compliance with treatment recommendations. In contrast, per protocol analysis is biased, since it includes only comparisons between patients who adhere to the treatment allocated to them. If the tested treatment works, the measured effect of the same treatment in the same study will be greater in magnitude for per protocol analysis (where only compliant patients are included in the analy sis) compared with ITT analysis (where all patient outcomes are included in the analysis whether patients comply with the treatment or not).

Statistical method

The statistical method used should be pre-specified and appropriate to the study objectives. The approp riate method depends on the outcome type (e.g. continuous variables such as BP results versus binary variables such as alive/dead) and the anticipated distribution of results (parametric tests for normally distributed data and non-parametric tests for non-normally distributed data; a transformation of raw data to approximate normality may be required). Pre-specified statistical methodology assures the reader that in the face of unimpressive or unexpected results, alternative analyses have not been used to achieve more impressive or desirable findings.

The use of post-hoc subgroup analyses, unjustified multiple outcome or interim comparisons will likely lead to a false positive finding in a small study subset (the more analyses are done, the more the risk of a false positive finding). However, it is reasonable to conduct post-hoc analysis adjustments if results indicate the method initially chosen is no longer valid. For example, a study may have been designed expecting normally distributed data, but non-normal distribution of data is unexpectedly encountered. In this situation, non-parametric methods will be required. Interested readers are referred to standard texts (Kirkwood et al., 2003) and user-friendly articles (Greenhalgh 1997b; 1997c).

What are the results?

Measures of treatment effect

Treatment effects of binary outcomes can be pre sented as an absolute difference (such as a risk difference), a relative difference (odds ratio or a risk ratio) or a relative risk (risk experimental group/risk standard group). Treatment effects of con tin uous measurements are usually analysed as absolute differences: for example, (mean blood pres sure experimental group) – (mean blood pressure standard group).

In a parallel group treatment trial an experimental treatment is being compared with standard treatment and patients are followed up for an outcome of interest (such as a specific benefit or harmful event such as death). Other metrics are used that are assisted by an outcomes matrix:

• Risk of outcome occurring in experimental treatment group

= proportion of patients in experimental arm who experience outcome of interest

= a/(a + c)

• Risk of outcome occurring in standard treatment group

= proportion of patients in standard treatment group who experience outcome of interest

= b/(b + d)

• Absolute risk difference (experimental versus standard)

= (risk of outcome occurring in experimental treatment group) – (risk of outcome occurring in standard treatment group)

= (a/[a + c]) – (b/[b + d])

The absolute risk difference is the percentage of additional (or fewer) patients allocated experimental treatment who develop the outcome of interest compared with the standard treatment.

• Relative risk (experimental versus standard)

= (risk of outcome occurring in experimental treatment group)/(risk of outcome occurring in standard treatment group)

= (a/[a + c])/(b/[b + d])

The relative risk measures the potential impact of the experimental treatment in reducing (or elevating) the risk of an outcome of interest, when compared with standard treatment.

• Odds of favourable outcome occurring

= number of patients experiencing outcome/number of patients not experiencing outcome

For experimental treatment group, odds of favourable outcome = a/c.

For standard treatment group, odds of favourable outcome = b/d.

• Odds ratio of favourable outcome (experimental versus standard)

= (odds of favourable outcome for experimental treatment group)/(odds of favourable outcome for standard treatment group)

= (a/c)/(b/d)

Absolute odds difference is not meaningful in a statistical sense. Odds ratios have similar character istics and application to the risk ratio, and although frequently used in intervention studies, are more valid to be applied to case control studies.

It is important that the results section presents both absolute and relative treatment effect measures. The latter usually gives an exaggerated impression of treatment effect if presented in isolation. In a hypothetical example comparing treatments X and Y, if beneficial outcome occurs in 4% on treatment X and 2% on treatment Y, the absolute risk difference is (4 – 2%) = 2% which is not particularly impressive. The relative risk of benefit of treatment X compared with Y is calculated as:

Risk treatment X/risk treatment Y = (4/2) = 2

If the author selectively presents only the relative risk, the reader will be led into thinking, and not incorrectly, that treatment X is twice as effective as treatment Y, but without any contextual awareness that only two out of every 100 patients attain greater benefit from treatment X. Although treatment X is better, its absolute impact is nowhere near as impres sive as the relative risk suggests.

Number needed to treat

Patients and health resource allocators need a more practical and intuitive way to understand the benefit or harm of a treatment offered to them or that they have been asked to fund. The number needed to treat (NNT) for benefit (or harm) translates previously discussed treatment effects into a more meaningful ‘How many patients do I need to treat before one of them experiences a benefit or harm?’

When we look at treatment benefit, the NNT to benefit is equivalent to (1/absolute risk difference). It is clear from this formula that the greater the absolute risk difference conferred by a treatment (the denominator), the smaller the number of patients required to be treated before one experiences a benefit. Using the previous example, where absolute risk difference is 2% benefit conferred by treatment X, the NNT for benefit is (1/0.02) = 50 for one patient to benefit. Although the relative risk of a benefit is 200% when X is compared with Y, 50 patients must be treated with X to obtain benefit for one, reflecting the small absolute risk benefit conferred by X.

Effect size

The higher the risk ratio, the more likely that the outcome will be different between the experimental and standard treatment in the real patient population. For example, for a given disease, a risk ratio of cure of 10 suggests a treatment is five times more likely to be beneficial compared to another treatment associated with a risk ratio of 2.

Precision of treatment estimates

The 95% confidence interval (95% CI) reflects the uncertainty of the study point estimate, informing the reader that the true population-level effect is likely to lie within this interval with 95% probability. The main results from a trial are best presented in terms of some treatment effect together with a confidence interval and (usually) a p-value.

How might the results be applied?

The final step is to consider the applicability of the results to your patients. This includes the similarity of their characteristics to trial patients and the practi calities of reproducing the study environment in your centre. The associated risks, costs, size and clinical significance of the treatment effect from both the patient’s and the institution’s viewpoints are practically important in deciding whether to intro duce a new therapy in your workplace.

An example of the application of these statistical factors and their clinical relevance is provided in Table 9.1 pertaining to the ECASS III study. A reasonable interpretation of this study could be that:

… for adult stroke patients similar to those enrolled in ECASS III (18–80 years admitted to a stroke centre with a diagnosis of acute ischaemic stroke able to receive the study drug within three to four hours after symptom onset), the number needed to treat to achieve a neurofunctional benefit (1 in 14) is far fewer than that required to harm (1 in 46 will suffer symptomatic intracranial haemorrhage). Despite increased risk of intracranial haemorrhage with alteplase, there was no mortality difference.

TABLE 9.1 Brief statistical analysis of ECASS III study

52.4% of ischemic stroke patients had a favourable neurological outcome with IV alteplase compared with 45.2% with placebo

Absolute risk difference for favourable outcome (alteplase vs placebo) = 52.4 – 45.2 % =7.2%.

Relative risk for favourable outcome (alteplase vs placebo) = 52.4/45.2 = 1.16 (i.e. alteplase was associated with 16% greater risk of patients having a favourable outcome (this risk ratio was not given in the study)).

Number needed to treat for benefit = 1/absolute risk difference for favourable outcome = 1/7.2% = 13.9 (i.e. we need to administer alteplase to approximately 14 patients for one of them to benefit).

Odds ratio for favourable outcome (alteplase vs placebo) is 1.28 with 95% CI, 1.00 to 1.65, p < 0.05. This means that there is a 28% greater chance for a favourable outcome with alteplase compared with the placebo. However, the lower end of the 95% CI touches the OR of 1. The true population effect lies with 95% probability within the interval from (nearly 0%) to 65% greater relative odds of favourable outcome with alteplase, with the best estimate from the study being 28%. As such, the p-value is likely to be just slightly under 0.05.

2.4% of patients given alteplase had symptomatic intracranial haemorrhage compared to 0.2% on the placebo (p = 0.008), although there was no mortality difference (7.7% v 8.4%, p = 0.68).

Alteplase incurs an additional absolute risk incidence of (2.4 – 0.2%) = 2.2% for symptomatic intracranial haemorrhage compared with placebo, with the number needed to harm = 1/(absolute risk increase for harm) = 1/(2.2%) = 45.45.

Source:Hacke W, Kaste M, Bluhmki E et al. Thrombolysis with alteplase 3 to 4.5 hours after acute ischemic stroke. N Engl J Med 2008; 359:1317−1329.

The patient could be told: ‘You have a much greater chance of recovering well from this stroke with alteplase than having a brain bleed from it.’

Critical appraisal of meta-analyses

Meta-analysis combines and summarises available research evidence quantitatively. If this is not pos sible, a more narrative systematic review is produced. Meta-analyses are most frequently and usefully employed to combine the effect estimates from multi ple randomised controlled intervention trials. An explicit, rational and comprehensive search strategy is applied to all sources of literature pertinent to a treatment topic. Unpublished or non-database cited trials should be identified and included to avoid publication bias (‘positive’ and English language trials are more likely to be published); this requires a search of ‘grey literature’ such as conference abstracts and theses and communication with researchers in the area. For single studies to be eligible for inclusion in a meta-analysis, they have to be independently evalu ated by two or more assessors as being of sufficient quality. These criteria are similar to those used to appraise single intervention studies.

The most valid meta-analyses obtain and analyse the disaggregated individual-level patient data from single trials rather than working only with aggregate data from single studies. The quality of design and conduct of meta-analyses must be appraised to ensure that their findings and conclusions are valid. The results of a well-designed and performed meta-analysis are likely to be most persuasive if it includes at least several good large-scale RCTs, the effect estimates from single studies are consistent, and the number of studies are sufficient and not clinically heterogeneous (i.e. are of similar clinical design). In cases where the available trials are small or poorly done, meta-analysis cannot compensate for the deficient primary trial data.

Forest plots are a visually excellent way to present results from meta-analyses. The general principles in assessing forest plots are:

• Look for consistent treatment effect in individual studies; that is, do the 95% CIs for included studies overlap in the main?

• Assess whether the 95% CIs encompass the vertical line, which indicates neutral or no effect; that is, there is no conclusive effect difference between the treatment and the control.

• Assess whether the 95% CIs are narrow (indicating a well-powered study) or very wide (inconclusive as sample size relatively small for study objectives).

• Assess whether individual studies are proportionately weighed according to their size in their relative contribution to the overall aggregated effect.

• Assess whether meta-analysis was valid to be used to derive an overall aggregated effect: non-overlapping 95% CIs that are distributed on both sides of the vertical line are either clinically or statistically too heterogeneous to combine.

We present two hypothetical meta-analyses for interventions to reduce the risk of failing the fellow ship examination in Figures 9.1 and 9.2.

Figure 9.1 Hypothetical forest plot: 18-month versus 12-month (control) preparation time and success in the fellowship examination

Figure 9.2 Hypothetical forest plot: overseas holiday versus weekly teaching sessions (control) and success in the fellowship examination

In Figure 9.1 there are only two studies available, A and B. Both studies have the majority of their 95% CI consistently to one side (left) of the vertical line of no effect. Both A and B are suggestive that an 18-month study phase reduces risk of failure (the risk ratio of failure is 0.6 and 0.5, respectively, with most of the 95% CI to the left of the vertical line). However, the upper limit of 95% CIs for A and B also extends to the right of the vertical line, so that an 18-month study phase may well increase the risk of failure by a factor of 3.5 and 1.05, respectively.

• Study A is relatively small with a wide CI, so A contributes less than B to the overall risk ratio.

• Study B has a much narrower CI, so it contributes most of the weighting to the overall risk ratio; this is reflected in similar 95% CIs for B and the overall risk ratio.

Avoiding failure using an 18-month study phase is not definitely proven, as the upper limit of the overall risk ratio is 1.0 (an 18-month study phase may not reduce your risk of failing the exam, but it is unlikely to exacerbate that risk); this is reflected in a p-value close to 0.05 (it achieves marginal statistical significance).

In Figure 9.2 there are four studies comparing the effect of attendance at weekly teaching sessions for six months and taking a three-month overseas holiday on failure in the fellowship examination. The individual study effects are inconsistent:

• A and B show compellingly high odds of failure with the treatment (taking the overseas holiday), with both 95% CIs located to the right of the vertical line.

• C is inconclusive with a wide 95% CI spanning both sides of the vertical line (there is little to choose between attendance at weekly teaching sessions for six months compared with taking a three-month overseas holiday, but the study is likely to be underpowered).

• D is more convincing than C as it has a much narrower 95% CI; this study suggests no difference between attending weekly teaching sessions for six months and taking a three-month overseas holiday.

The overall effect is convincing though, with a three-month overseas holiday associated with a high odds of failure compared with attendance at weekly teaching sessions for six months.

However, since the effect estimations and 95% CI for individual studies are so variable, is it appropriate to aggregate their results in producing an overall estimate?