Finally if a number of eligible studies have been identified, obtained and have passed the selection process, relevant information has to be extracted. If possible the extraction should be standardized, for example, by using a pretested form. The format should allow one to enter the data into a database and to perform basic statistical analyses. Another efficient method is to enter the data directly into a prestructured table. Regardless of what method is used, reviewers have to have a clear idea of what information they need for their analysis and what readers will need to get their own picture. Extraction and all assessments should be done by at least two independent reviewers. Coding errors are inevitable and personal biases can influence decisions in the extraction process. The coded information of the reviewers must be compared and disagreements discussed.

A major criticism of sceptics towards meta-analysis (and, to a lesser extent, also to non-meta-analytic systematic reviews) is the garbage-in, garbage-out problem. The results of unreliable studies do not become more reliable by lumping them together. It has been shown in conventional and complementary medicine that less rigorous trials tend to yield more positive results (Schulz et al., 1995, Lijner et al., 1999 and Linde et al., 1999). Sometimes it may be better to base the conclusions on a few, rigorous trials and discard the findings of the bulk of unreliable studies. This approach has been called ‘best-evidence synthesis’ (Slavin 1986). While it is sometimes considered as an alternative, it is in principle a subtype of systematic reviews in which defined quality aspects are used as additional inclusion criteria (see White et al. 1997, as an example from complementary medicine).

However, the assessment of quality is difficult. The first problem is that quality is a complex concept. Methodologists tend to define quality as the likelihood that the results of a study are unbiased. This dimension of quality is sometimes referred to as internal validity or methodological quality. But a perfectly internally valid study may have fundamental flaws from a clinician’s point of view if, for example, the outcomes measured are irrelevant for the patients or patients are not representative of those commonly receiving the treatment.

A second problem is that quality is difficult to operationalize in a valid manner. An experienced reviewer will find a lot of subtle information giving an indication on the quality of the study ‘between the lines’ from omissions and small details. However, subjective global ways to assess quality are not transparent and prone to subjective biases.

Many systematic reviews on treatment interventions include some standardized assessments of internal validity. There is agreement that key criteria for the internal validity of treatment studies are random allocation, blinding and adequate handling of dropouts and withdrawals. In the past these and other criteria have typically been combined in scores (see Moher et al., 1995 and Moher et al., 1996 for overviews). But the validity of such scores is doubtful (Jüni et al. 1999). Today it is clearly preferred to assess single validity items without summarizing them in a score, and to investigate whether quality has an impact on findings. Currently, the most important tool is the Cochrane Collaboration’s ‘risk of bias’ assessment (Higgins & Altman 2008). Whether the criteria are combined in scores or applied separately, the problems remain that the formalized assessment is often crude and the reviewers have to rely on the information reported.

In conclusion, assessments of methodological quality are necessary but need to be interpreted with caution. The assessment of other dimensions of quality is desirable, but the problems in the development of methods for this purpose (which have to regard the specific characteristics of the interventions and conditions investigated) are even greater than for internal validity.

The clinical reader of a systematic review is mainly interested in its results. While the majority of readers will only look on the abstract and meta-analytic summary, the review also has to provide sufficient information for those who want to get their own idea of the available studies and their results. For example, in a review of acupuncture in headache it will be relevant for the specialist to know what type of headache was studied in each primary study, to have information on the sex and age of the patients, and where they were recruited. Regarding the methods readers should know what the design was, whether there was some blinding, how long the patients were followed and whether follow-up was complete. They need details on the nature of the experimental (which acupuncture points, how many treatments) and the control interventions (type of sham acupuncture). And, of course, readers want to know which outcomes have been measured and what the results were. This detailed information is typically summarized in a table.

If the primary studies provide sufficient data the results are summarized in effect size estimates. Table 6.1 lists some of the most common measures.

Tables including a graphical display of the results are extremely helpful. Table 6.2 shows the standard display from a meta-analysis from the Cochrane Library, in this case on randomized trials comparing hypericum extracts and standard antidepressants (separated in the subgroups older antidepressants and selective serotonin reuptake inhibitors (SSRIs)) in patients with major depression (Linde et al. 2008