Component	Example 1	Example 2
P: Population, patient or problem	In women suffering with heavy menstrual bleeding…	In postoperative women with swollen legs…
I: Intervention (test, medical or surgical treatment, or process of care)	… would treatment with norethisterone…	… would a Doppler ultrasound be accurate…
C: Comparison (placebo, another alternative treatment or the gold standard in a diagnostic accuracy study)	… compared with no treatment at all or alternative treatments (e.g. levonorgestrel IUS)…	… compared with venography as the gold standard…
O: Outcome(s)	… lead to an improvement in their symptoms?	… in diagnosing deep venous thrombosis?
D: Ideal design for the study	A randomized controlled trial	A test accuracy study

IUS, intrauterine system. Produced by Bob Phillips, Chris Ball, Dave Sackett, Doug Badenoch, Sharon Straus, Brian Haynes, Martin Dawes since November 1998. Updated by Jeremy Howick March 2009.

The purpose of this step is to get clarity regarding the clinical question. Writing down the question in free form often makes it possible to break it down into a PICOD format. In practice, this may not always be feasible for all clinical questions, and missing out some elements of PICOD is not necessarily a cause for concern.

Searching the literature

Approximately 17,000 journals collectively publish over 1 million biomedical articles each year. Identifying relevant articles will require the use of apposite keywords (and their combinations) to search appropriate databases. A hierarchical approach to literature searching is recommended (Figure 68.3).

Figure 68.3 Hierarchical approach to literature searching. DARE, Database of Abstracts of Reviews of Effectiveness; RHL, Reproductive Health Library; NeLH, National electronic Library of Health.

Source: Khan and Coomarasamy (2003). NeLH, National electronic Library for Health; RHL, Reproductive Health Library.

The first step is to look for an up-to-date professional guideline that has been developed on the basis of systematic appraisal of available evidence. Table 68.2 lists some sources of guidelines and evidence summaries. The Royal College of Obstetricians and Gynaecologists (RCOG) has long acknowledged the need to update clinical practice on the basis of research findings. Since 1973, the RCOG has regularly convened study groups to address important growth areas within the specialty. These groups have met, evaluated the results of research and conducted in-depth discussions on a variety of topics. These discussions have shaped the development of clinical recommendations which were initially based on consensus. Over the years, this approach has been modified in order to produce genuine evidence-based guidelines. To be effective and relevant, guidelines must fulfil fthe following three essential criteria.

• They must be multidisciplinary: in other words, they must be developed by a multiprofessional working group including clinicians, nurses and other health professionals as well as users of health services (i.e. consumer representatives).

• They must be evidence based: guideline development must include a systematic search for and synthesis of all available research evidence.

• They must be evidence linked: individual recommendations within a guideline must be explicitly linked to the strength and quality of the evidence on which they are based, using a recognized grading scheme (Centre for Evidence-based Medicine, Oxford University, www.cebm.net) (see Appendix).

Table 68.2 Sources of guidelines and evidence summaries (relevant for gynaecology)

Sources of guidelines and evidence summaries	Website
Royal College of Obstetricians and Gynaecologists (RCOG)	www.rcog.org.uk
American College of Obstetricians and Gynecologists (ACOG)	www.acog.org
The Society of Obstetricians and Gynaecologists of Canada (SOGC)	www.sogc.org
The Royal Australian and New Zealand College of Obstetricians and Gynaecologists (RANZCOG)	www.ranzcog.edu.au
International Federation of Gynecology and Obstetrics (FIGO)	www.figo.org
Faculty of Sexual and Reproductive Healthcare (FFPRHC)	www.ffprhc.org.uk
British Fertility Society (BFS)	www.britishfertilitysociety.org.uk
British Gynaecological Cancer Society (BGCS)	www.bgcs.org.uk
British Society for Colposcopy and Cervical Pathology (BSCCP)	www.bsccp.org.uk
British Society of Urogynaecology (BSUG)	www.rcog.org.uk/bsug
British Society for Gynaecological Endoscopy (BSGE)	www.bsge.org.uk
The Association of Early Pregnancy Units (AEPU)	www.earlypregnancy.org.uk
The NHS National Library for Health (NLH)	www.library.nhs.uk
The National Institute for Health Clinical Excellence (NICE)	www.nice.org.uk
Scottish Intercollegiate Guideline Network (SIGN)	www.sign.ac.uk
NHS Clinical Knowledge Summaries (NHS CKS; formerly PRODIGY)	www.cks.library.nhs.uk
The National Guideline Clearing House (NGC, US)	www.guidelines.gov

It is generally anticipated that national guidelines will, in turn, be used as a basis for the development of local protocols and guidelines in conjunction with local commissioners and providers of health care as well as service users. These should take into account the specific needs of local service provision and the preferences of the local population.

In the absence of credible evidence-based guidelines, the next step would be to search for an up-to-date, good-quality, systematic review. Some sources of systematic reviews are given in Table 68.3.

Table 68.3 Sources of evidence

Systematic reviews
Sources of systematic reviews	Website
The Cochrane Library	www.cochrane.org/reviews/
The CRD databases (including DARE)	www.crd.york.ac.uk/crdweb
The Pubmed Systematic Reviews Search Filter	www.ncbi.nlm.nih.gov/entrez/query/static/clinical.shtml
Bandolier	www.jr2.ox.ac.uk/bandolier
The WHO Reproductive Health Library (RHL)	www.rhlibrary.com
Health Technology Assessment (HTA) database	www.ncchta.org/project/htapubs.asp

Sources of primary literature
Sources of primary literature	Subject matter	Website
Pubmed (Medline)	Medicine, bioscience	www.pubmed.gov
EMBASE	Medicine, pharmacology, nursing	www.embase.com
CINAHL	Nursing and allied health care	www.cinahl.com
AMED	Allied and alternative health care	www.bl.uk (search for ‘AMED’)
BNI (British Nursing Index)	Nursing
HMIC (Health Management Information Consortium) database	Health management

WHO, World Health Organization; CRD, Centre for Reviews and Dissemination; DARE, Database of Abstracts of Reviews of Effectiveness.

Apart from the traditional sources of systematic reviews, such as the Cochrane Library and DARE, MEDLINE and Pubmed have now become a rich source of systematic reviews. Pubmed contains a systematic review filter within Pubmed Clinical Queries (http://www.ncbi.nlm.nih.gov/entrez/query/static/clinical.shtml), and MEDLINE has the indexing term ‘reviews, systematic’ as a ‘publication type’.

If no systematic reviews are identified, or if reviews are out-of-date, non-systematic or of marginal relevance to the clinical question, it is necessary to continue the literature search for primary studies. Table 68.3 provides a list of important sources of primary literature.

Efficient searching for primary literature requires training and practice, and input from an experienced librarian or information scientist. Careful selection of keywords for searching, matching these keywords to Medical Subject Heading terms, and combining the terms with ‘AND’ or ‘OR’ Boolean operators are necessary skills that are best learnt from librarians. Pubmed carries three tools that clinicians can employ to optimize their searches, and these are discussed below.

Pubmed ‘clinical queries’ (www.ncbi.nlm.nih.gov/entrez/query/static/clinical.shtml)

This has now established itself as the primary gateway for efficient literature searching to support evidence-based practice. ‘Clinical queries’ uses filters that allow retrieval of articles for the five main categories of clinical questions: therapy, diagnosis, aetiology, prognosis and clinical prediction. The option of ‘sensitive search’ in ‘clinical queries’ allows for a broad search that may identify many relevant articles, but also carries the risk of getting many irrelevant hits; the ‘specific search’, on the other hand, performs a narrow search and would retrieve fewer articles, but at the risk of missing some relevant articles. It is advisable to start with a sensitive search, and to move to a specific search if the number of hits is unmanageably large on the sensitive search.

Pubmed ‘SLIM’ (SLider Interface for Medline/Pubmed)

This exciting new development uses six different slider bars to control search variables, improving user control and the capability to instantly refine and refocus search strategies (https://pmi.nlm.nih.gov/slim/).

Pubmed ‘related articles’

Once a relevant citation is found, clicking on the ‘related articles’ function to the right of the identified citation retrieves other similar articles.

Evaluation of the literature

Once relevant articles have been identified and retrieved, the next step is to select those which are appropriate and methodologically sound. Many papers published in medical journals have serious design flaws, and most are irrelevant for everyday clinical practice. Most clinicians are accustomed to functioning as ‘users’ when dealing with evidence to support practice, but the absence of up-to-date and valid preappraised evidence makes it necessary to switch to an ‘appraiser’ mode and evaluate articles in terms of validity (methodological soundness), importance {e.g. effect sizes [relative risks, odds ratio, numbers needed to treat (NNT) for therapy studies, and accuracy estimates (e.g. likelihood ratios)] for test accuracy studies}, and applicability of the results in practice.

In addition, an overall judgement on the quality of the evidence will need to take two other issues into account: consistency, the extent to which different studies found similar results; and robustness, the extent to which minor alterations in results do not change the conclusions drawn from those data.

Various checklists can be used to appraise different types of clinical questions. Checklists for therapeutic and diagnostic questions as well as systematic reviews are given in Tables 68.5, 68.7 and 68.10. In appraising a study, it is important to assess the suitability of the research design and methods used in the context of the specific clinical question. Randomized trials provide the best evidence for treatment, but valid evidence for diagnosis, prognosis and causation may be derived from publications based on other study designs (Table 68.4).

Table 68.4 Research designs for specific clinical questions

Clinical question	Ideal research design
Effectiveness of therapy	RCT
Prevention	RCT
Screening	Cluster RCT
Diagnosis (accuracy)	Cross-sectional study (comparison of index test with reference standard test)
Prognosis	Cohort study
Aetiology	Case–control or cohort study
Adverse events	Case report, case series, case–control studies, cohort studies and RCTs
Economic assessment of medical interventions	Depending on the exact question: Cost-minimization study Cost-effectiveness study Cost–utility study Cost–benefit study

RCT, randomized controlled trial.

Appraising a paper on effectiveness of therapy (randomized controlled trial)

A randomized controlled trial reduces the risk of bias (systematic deviations or errors in the results) by minimizing the likelihood of important differences between the treatment and control arms of the study. If no randomized controlled trials are available, other research designs (e.g. a non-randomized controlled study or even a case–control study) may provide valuable information about therapy, although they may exaggerate the potential benefits of treatment and thus represent a lower grade of evidence (see evidence grades, Centre for Evidence-based Medicine, Oxford University, www.cebm.net) (see Appendix). A checklist can be used to appraise a therapy article (Table 68.5).

Table 68.5 Critical appraisal checklist for randomized controlled trials

I. Are the results valid?

1 Is the assignment of patients randomized? Appropriate methods are: random number tables and computer-generated random numbers

2 Is allocation of treatment concealed? Appropriate methods are: opaque envelopes, third party randomization, distant (telephone or Internet) allocation

3 Are patients, health workers and study personnel ‘blinded’ to the intervention?

4 Are patients analysed in the groups to which they were randomized? (‘intention to treat’ analysis)

5 Are all patients who entered the trial properly accounted for?

6 Is follow-up complete?

7 Are the randomized groups similar at the start of the trial? (normally presented in ‘Table 1’ of the article)

8 Apart from the experimental and control interventions, are the groups treated equally?

9 Was the study adequately powered?*

10 Was ethical approval obtained for the study?*

II. What are the results (i.e. importance of results)?

1 How large is the treatment effect? (reported as relative risks, odds ratios, absolute risks, numbers needed to treat or differences in means or medians)

2 How precise is the treatment effect? (95% confidence intervals around the above)

III. Will the results help me in caring for my patients (applicability)?

1 Can the results be applied to my patients?

2 Are all clinically important outcomes considered?

3 What are the likely benefits? Are they worth the potential harms and costs?

* These items do not strictly relate to ‘validity’; nevertheless, they are important and should be part of the appraisal process.

If the results for a randomized controlled trial are dichotomous (i.e. a ‘yes/no’ outcome), and there are two arms in the study, the findings can be presented and analysed in a 2 × 2 table, as shown in Table 68.6.

Table 68.6 2 × 2 table to present the results of a two-arm trial with a dichotomous outcome

	Outcome (live birth)
	Event	Non-event
Experimental group (surgery for endometriosis)	a	b
Control group (no surgery)	c	d

Risk in the experimental group (or experimental event rate) = a/(a+b)

Risk in the control group (or control event rate) = c/(c+d)

Relative risk (or risk ratio) = [a/(a+b)]/[c/(c+d)]

Absolute risk reduction (ARR) = [c/(c+d)] − [a/(a+b)]

Numbers needed to treat = 1/ARR

Odds in the experimental group = a/b

Odds in the control group = c/d

Odds ratio = (a/b)/(c/d)

NNT is a particularly useful measure to express the magnitude of treatment benefit. NNT is defined as the number of patients that will need to be treated to achieve one additional good outcome. Thus, the lower the NNT, the greater the magnitude of the outcome, and an NNT below 10 is often (arbitrarily) regarded as a ‘large effect size’. However, such arbitrary thresholds for defining benefit sizes should be used with caution, as a small effect size (with a correspondingly large NNT) can still be important if the outcome in question is serious (e.g. deaths prevented).

Appraising a diagnostic article

The checklist shown in Table 68.7 can be used to appraise a diagnostic accuracy article. It should be noted that this checklist only applies to accuracy studies; there are other aspects of testing that may need to be judged on other criteria. For example, evaluating the inter- or intraobserver reliability or the clinical impact of testing will require designs other than the one employed for accuracy evaluation, and will often need to be judged by other criteria.

Table 68.7 Critical appraisal checklist for diagnostic accuracy studies

I. Are the results valid?

1 Does the patient sample include an appropriate spectrum of patients to whom the test will be applied in clinical practice?

2 Are the test and reference (gold) standard results blinded?

3 Did everyone get both the test and the reference standard?

4 Is the test described fully?

5 Is the reference standard described fully?

II. What are the results?

1 What are the results? (sensitivity, specificity, predictive values or likelihood ratios)

2 What is their precision? (95% confidence intervals)

III. Will the results help me in caring for my patients?

1 Are the results applicable to my patients?

2 Will the results change my management?

3 Will patients be better off as a result of the test?

The sensitivity, specificity, predictive values and prevalence are defined in the marginal cells of Table 68.8. The sensitivities and specificities are often misinterpreted, and the mnemonics ‘SnNout’ and ‘SpPin’ are very useful to ensure correct interpretations.

• SnNout: when a test has high Sensitivity, a Negative test result rules out the diagnosis.

• SpPin: when a test has high Specificity, a Positive test result rules in diagnosis.

Table 68.8 2 × 2 table for test accuracy results

Thus, sensitivities relate to negative test results, whilst specificities relate to positive test results.

Likelihood ratios, which can be calculated from the 2 × 2 table, or derived from sensitivities and specificities as shown in the footnote of Table 68.8, are generally considered to be the most useful of all test accuracy summaries. The likelihood ratio indicates how much a given test result raises or lowers the probability of having the disease. The higher the likelihood ratio of an abnormal test, the greater the value of the test. Conversely, the lower the likelihood ratio of a normal test, the greater the value of the test. Although a guide to the interpretation of likelihood ratios is provided in Table 68.9, it should be noted that the value of a test may vary depending on the pretest probability and the consequences of having the condition.

Table 68.9 Interpretation of likelihood ratios (LR)

LR for positive test	LR for negative test	Value of test
>10	<0.1	Very useful
5–10	0.1–0.2	Moderately useful
2–5	0.5–0.2	Somewhat useful
>1–2	0.5–<1	Little useful
1	1	Useless

Appraising a systematic reviews or meta-analysis

A systematic review aims to provide a complete exploration of evidence on a subject. It is an overview of studies using explicit, systematic and therefore reproducible methods to locate, select, appraise and synthesize relevant and reliable evidence. Although the terms ‘systematic review’ and ‘meta-analysis’ are often used interchangeably, the term ‘meta-analysis’ has a specific meaning; it is the mathematical pooling of numerical results from individual studies. All good systematic reviews have the following five features:

• a clearly framed question;

• comprehensive literature search across multiple databases for published and often unpublished data;

• explicit inclusion and exclusion criteria for study selection;

• careful assessment of methodological quality of the included studies; and

• synthesis of the evidence (with quantitative synthesis, or meta-analysis when appropriate).

The checklist in Table 68.10 can be used to appraise a systematic review.

Table 68.10 Critical appraisal of systematic reviews

I. Are the results valid?

1 Does the article address a clearly focused question? (i.e. are the PICOD elements defined clearly?)

2 Are the criteria used to select articles for inclusion appropriate?

3 Is the literature search comprehensive? (MEDLINE, EMBASE, Cochrane Library, CINAHL, searchers for abstracts, hand-searching etc.)

4 Is the validity of the included studies appraised? (using checklists or tables like those given in Tables 68.7 and 68.9)

5 Did two or more reviewers extract the data independently?

6 Are the results consistent from study to study?

7 Is there an assessment for publication bias? (e.g. funnel plot analysis)

II. What are the results?

1 What are the results of the study?

2 How precise are the results? (95% confidence intervals)

III. Will the results help me in caring for my patients?

1 Can the results be applied to my patient care?

2 Are all clinically relevant outcomes considered?

3 Are the benefits worth the harms and costs?

One needs to be mindful of the following three issues when interpreting the findings of systematic reviews and meta-analyses.

• ‘Garbage in, garbage out’: if the studies that make up a systematic review are flawed, the systematic review itself is likely to be unreliable.

• Publication bias: unfortunately, studies that show negative and/or statistically non-significant results are less likely to be published. This could be due to apathy or conflict of interest of the researchers or sponsors, or the disinterest of journal reviewers and editors. Although systematic reviewers are expected to make all efforts to locate published and unpublished data, it is possible that negative studies remain buried, thus biasing the findings of a review. Statistical tests such as funnel plot analysis can indicate the presence of publication and related biases.

• Heterogeneity: the degree to which the effect sizes from the various individual studies in a systematic review agree (homogeneity) or disagree (heterogeneity) has important implications for inferences. If the findings are homogeneous (i.e. the results of various studies agree with each other), one is more likely to believe the meta-analysed summary result. Heterogeneity can be assessed visually by examining the forest plot, or statistically using tests such as Chi-square, Cochran Q or I² statistics.

Findings of a meta-analysis are often presented as a forest plot (Figure 68.4). Forest plots can be used to plot relative risks, risk differences, odds ratios, mean differences or other summary estimates such as sensitivities, specificities and likelihood ratios. Each individual study in the systematic review is represented in a row, with its own point estimate (the ‘box’) and the 95% confidence interval (the ‘whiskers’). The pooled summary estimate is shown as a diamond at the bottom of the chart. The centre of the diamond represents the pooled estimate and the ends of the diamond represent the 95% confidence interval for the pooled estimate.

Figure 68.4 An example of a forest plot: urinary human menopausal gonadotrophin (hMG) versus recombinant follicle-stimulating hormone (rFSH) for controlled ovarian hyperstimulation in in-vitro fertilization (IVF) treatment for the outcome of live births per IVF cycle started. RR, relative risk. See the source reference for details of the individual study references.

Source: Coomarasamy A, Afnan M, Cheema D, van der Veen F, Bossuyt PM, van Wely M. Urinary hMG versus recombinant FSH for controlled ovarian hyperstimulation following an agonist long down-regulation protocol in IVF or ICSI treatment: a systematic review and meta-analysis. Hum Reprod. 2008 Feb;23(2):310–5.

Implementation of useful findings

The final step in EBM is implementation or incorporating the identified evidence in clinical practice. This may seem a deceptively simple step (e.g. a case of writing up a prescription), but in practice can be the most complex of all EBM steps. As a minimum, this requires careful consideration of the following issues.

How generalizable are the results of the study to your specific patient or patient group?

This issue is variously described as ‘generalizability’, ‘applicability’ or ‘external validity’, and requires the examination of study population (e.g. the inclusion and exclusion criteria) and study conduct to answer the question: ‘Are there important reasons to believe that the results of this study may not apply to my patient(s)?’ It is important not to be unreasonably stringent in this evaluation. For instance, if a US study recruiting women under the age of 40 years with severe endometriosis found that treatment X was effective in improving quality of life, would the evidence apply to a 42-year-old English woman with moderate endometriosis? Although this woman may not have been eligible for the trial, there is probably no reason to believe that the results of the study should not apply to her.

What is the balance between benefit and harm?

Many effective treatments come with their own set of problems, including adverse effects, toxicity and operative complications (for surgical interventions). Whilst a full and formal clinical decision analysis or other similar techniques to comprehensively analyse the net benefits and harm with their associated values and costs would be outside the scope (and perhaps need) of most clinicians and their patients, it is essential that an overall assessment of the net balance sheet is made. To make a reliable assessment on this issue, the identified studies need to have reported all clinically relevant outcomes over a reasonable period of time. Often, it is necessary to undertake separate literature searches to identify rarer or long-term adverse outcomes.

What does the patient think?

Eliciting ideas, concerns and expectations from the patient about the condition and the potential therapeutic interventions that are available is crucial to a successful consultation. Patients may place different values on various outcomes and adverse events; unless these are elicited and incorporated into the decision-making, satisfaction with the management plan and compliance with therapy are unlikely to ensue. Building a partnership with the patient, providing a balanced discussion about the risks and benefits, and seeking agreement and providing support are all essential to the ‘Apply’ step of EBM.

Despite being accessed and appraised systematically, most research findings are currently applied intuitively. If EBM is to be seen through to its logical conclusion involving a synthesis of empirical evidence and human values, the current conflict between the explicit collection of data and its implicit use must be addressed. This area is being investigated and possible options are being evaluated. These include decision analysis, which provides an intellectual framework for the development of an explicit decision-making algorithm (Lilford et al 1998), and computerized decision support systems. When there are several treatment options which may have different effects on a patient’s life, there is a strong case for offering patients a number of choices. It is possible that their active involvement in the decision-making process may actually increase the effectiveness of the treatment (Coulter et al 1999).

Conclusion

The purpose of clinical research is to generate new knowledge on how to treat individual patients and how best to deliver healthcare services. The objective of EBM and its component activities is to maximize the extent to which clinical practice is based on research evidence, and consequently improve the net health gain for patients. Within gynaecology, the recent years have seen considerable progress in strategies aimed at seeking, appraising and applying evidence. A comprehensive approach towards getting research evidence into practice is promoted by National Health Service clinical effectiveness and clinical governance initiatives. What is being challenged is the way in which doctors make clinical decisions and the way in which patients respond to them.

KEY POINTS

1 Evidence-based medicine (EBM) involves the integration of best available research evidence with clinical expertise and patients’ views and preferences.

2 EBM has four steps: (i) Ask a structured clinical question; (ii) Acquire the relevant literature; (iii) Appraise the literature for validity and usefulness; and (iv) Apply the findings.

3 Asking a structured clinical question involves breaking it down to PICOD: Population, Intervention, Comparison, Outcomes, and Desired study design.

4 Acquiring evidence involves taking a hierarchic approach to literature searching: look for evidence-based guidelines first, then for systematic reviews, and then for primary studies if no relevant guidelines or systematic reviews exist.

5 Good guidelines are evidence-based, evidence-linked and produced by a multidisciplinary team. They can often be found in various professional body websites.

6 Appraisal of literature involves assessment for validity, usefulness and applicability.

7 Strategies need to be developed to overcome existing barriers to the wider implementation of evidence-based care in gynaecology.

Oxford Centre for Evidence-based Medicine Levels of Evidence (March 2009) (for definitions of terms used, see glossary at http://www.cebm.net/?o=1116)

Grades of recommendation

A	Consistent level 1 studies
B	Consistent level 2 or 3 studies or extrapolations from level 1 studies
C	Level 4 studies or extrapolations from level 2 or 3 studies
D	Level 5 evidence or troublingly inconsistent or inconclusive studies of any level