Did the authors use randomization?

The purpose of randomization is to ensure, as far as possible, that all factors (known and unknown) that may influence the treatment outcome are balanced between the treatment groups. This is done to reduce the risk of a chance bias. Randomization requires the use of a random device; this is normally a table of random numbers. Systematic allocation; i.e., alternating between treatments, odd or even birth date or hospital number is not an acceptable method, though it is sometimes referred to as pseudo-randomization. Since the researchers know which treatment each person is allocated to before they consent, selective allocation can occur, which will skew the results. Hence the process used for randomization must ensure that neither the trial subjects nor the investigator can influence the treatment arm each person ends up in (‘allocation concealment’).

Despite their pre-eminence, RCTs do, however, have their limitations. They are usually carried out in a narrowly defined research setting. Protocols require careful selection of patients with specific characteristics. There may be a proportion of patient eligible who chooses not to participate. Because of these factors, patients in an RCT may not be typical of patients in routine health practice.

Was there a clearly defined research question?

A good study has limited objectives that are based on a biological hypothesis. Studies with no focus and without a clearly defined question are in danger of becoming a ‘fishing expedition’ with multiple comparisons.

The chances of multiple testing increases when there is numerous subgroups or endpoints or when data are reported on multiple side effects. This is particularly problematic when such comparisons are unplanned.

Did the study have appropriate end points?

Most RCTs in cancer therapy concentrate on reporting response rates (a surrogate end point of clinical benefit), disease free survival and overall survival. However, accepting that cure is out of the reach of many patients with cancer, information on symptom control and quality of life are often of major importance; they are frequently ignored in the primary research.

Optimal cut points and the problem with multiple comparisons

Where there is a continuous outcome measure, it is common to simplify analysis by defining one or more points that are used to characterize risk. The way that these points are selected may have a major effect on the analysis.

Subgroup analysis

Subgroup analyses are fairly common in trials; they use the data from a study to compare one endpoint across different subgroups. There are three major problems with this approach:

Where a subgroup is found to behave differently, you should consider if there is a plausible biological mechanism for this and whether other trials have found a similar finding. Where there is a statistical analysis, this should be done as a formal test of interaction. Strong empirical evidence suggests that post hoc subgroup analyses often lead to false positive results (Example Box 26.2).

Fraudulent conduct of clinical research

Although we all hope that such fraud is very rare, there are too many instances where there has been clear fraud (Example Box 26.3). Detecting such fraud is very difficult. Peer review may help, but in many instances journals do not have the original protocol and access to the raw data is extremely unusual. In the above example, it was an independent outside review that revealed the fraud. The review was carried out as the trial was at variance with other literature and the subject was of great importance.

Blinding

While it is often not possible to blind patients and clinicians about which arm a trial patient is in, anyone who evaluates patient outcomes should be blinded to the treatment allocation. This is especially important when there is a ‘surrogate’ endpoint subject to individual interpretation, e.g. imaging to assess response or measurement of toxicity.

Allocation concealment

For randomization to be effective, neither the trial subjects nor the investigators can influence the treatment each person is allocated. If either the patients or the investigators know the treatment each patient is allocated before they consent, selective allocation may occur which results in skewing of results. Similarly, if the patients know the treatment they are receiving, it can introduce bias in reporting the outcomes. Hence, it is important to conceal the randomization list from those recruiting patients. Only when an eligible patient has agreed to participate should the treatment allocation be carried out. Unblinded allocation methods show an average bias of 30–40%.

The only safe method of randomization is to carry out allocation, for a single patient at a time, by a central trials office or pharmacy.

Who was lost to follow-up or dropped out during the study?

Loss of some patients in a study is inevitable. These patients may have a different prognosis than those remaining in the trial. Excluding those lost to follow-up or dropped out will often result in overestimate of treatment benefit.

All patients should be analysed in the groups they were randomized to, regardless of whether they crossed over or changed treatment, were lost to follow up or dropped out. This is known as ‘intention to treat’ analysis and should always be reported, even if the authors also choose to present analyses excluding selected patients (Box 26.3). Intention to treat analysis is the only way we can ensure that the original randomization has retained and thereby, the groups are comparable.

Were measurements appropriate and well done?

Although it is probably less of a problem in cancer studies than some other areas of medicine, be wary of end points that measure short-term outcomes. For instance, in studies concerned with management of complications of cancer or the side effects of its treatment, trials may report effectiveness of an intervention after, say, six-week therapy. Since it is easier to get a short-term change, it is important to consider if the investigators should have been measuring long-term effectiveness. For instance, several treatments in the review ‘Non-surgical interventions for late radiation proctitis in patients who have received radical radiotherapy to the pelvis,’ (Denton et al., 2008), were often only given for one month and then response was assessed at one time point.

Although less of a problem in cancer studies, when compared with the general medical literature, it is sometimes difficult to obtain accurate measurements in certain types of study. If the measurement error concerns an important factor predictive of an outcome, this is of major concern. An example in the field of cancer would be the difficulty in accurately measuring dietary fat intake when studying the effect of diet on development of breast cancer.

Which is the better treatment and how much difference?

It is not sufficient to know that one treatment is better than another, it is important to quantify how much better the treatment is than the other. While a p-value may tell you which treatment is better, the confidence interval is a better measure. It not only tells you which is the better treatment, but quantifies the difference (Box 26.4). This is particularly important where any gain has to be balanced against the side effects of that treatment.

Just because a statistical test shows an intervention to be superior this does not tell you whether the difference is clinically important. This requires, in addition, an estimate of the numbers of patients seen in routine practice who might benefit from the ‘better’ treatment, the toxicity of the treatment, its ease of administration and cost. Assessing clinical significance requires sound judgment. The decision whether to use that treatment will also ultimately depend on the personal preferences of patients.

Were there sufficient subjects?

There has been a tendency for cancer RCTs to have too few subjects, or more accurately patients reaching the end point of the study. Investigators should provide an a priori justification of the sample size in the paper itself. If there are no power calculations, it is useful to look at the width of the confidence intervals. If there is an inadequate sample size, the confidence intervals will be abnormally wide.

Duplicate publication

Studies which are positive are more likely to appear more than once in publication. This is especially problematic for multi-centre trials where an individual centres may publish results specific to their site.

The limitations of a Medline search

While a Medline search is the most convenient way to identify published research, it should not be the only source of publications for a meta-analysis. Medline searches cover only about a quarter of the medical literature.

Foreign language publications

Some meta-analyses restrict their attention to English language publications only. This makes life easier for the reviewers, but there is evidence that researchers tend to publish reports of positive trials in English language journals. Conversely, they publish negative reports in a native language journal where the citation index is likely to be lower.

Managing heterogeneity

A degree of heterogeneity is to be expected and is acceptable. Showing that a treatment gives consistent results, with some variation, across a variety of trials, and in a large number of subjects, increases confidence in that treatment. Where there is excessive heterogeneity pooling of data should be avoided. The results can be explored in various ways.