Understanding the strengths and weaknesses of clinical research in cancer

Published on 09/04/2015 by admin

Filed under Hematology, Oncology and Palliative Medicine

Last modified 09/04/2015

Print this page

rate 1 star rate 2 star rate 3 star rate 4 star rate 5 star
Your rating: none, Average: 4 (1 votes)

This article have been viewed 5442 times

26 Understanding the strengths and weaknesses of clinical research in cancer

C. Williams

‘I hope that the cares of medical practice have not obliterated your interest on solving deductive problems.’

Sherlock Holmes to Watson in the Stockbroker’s Clerk*

* Quotes in this chapter are as those cleverly used by Jorgen Nordenstrom in his book ‘Evidence based medicine in Sherlock Holmes’ Footsteps’ (Wiley Black-well, 2006).

Introduction

This chapter briefly discusses cancer clinical trials and systematic reviews of the cancer literature, together with the concept of critical appraisal and evidence-based medicine (EBM). Much of literature appraisal requires a common sense, but systematic, approach to ascertaining answers to key questions about potential biases in the design, conduct and reporting of a study or review.

Appraising reports of cancer clinical trials

The difficulty in appraising clinical trials lies in assessing the strength of the evidence. On cursory reading the author of the report may seem to present a strong case for the trial, but you need to see beyond the rhetoric, the eminence of the author or institution/trials group and the status of the journal publishing the paper, to assess the evidence and to see if it is strong enough to stand on its own. Very strong evidence may be enough to influence practice by itself, but more commonly the evidence is weaker and needs support from further studies.

What are the important elements when assessing reports of trials?

By far the most important elements are the design and conduct of the trial. The methods of analysis, while still important, are generally less likely to lead to inappropriate conclusions than inappropriate design and management of a trial. This chapter is based on the methods recommended by various EBM groups (Box 26.1).

Box 26.1
Evidence-based medicine web resources

Centre for Evidence-Based Medicine, Oxford, UK – http://www.cebm.net/?o=1013
Centre for Evidence-Based Evidence, University of Toronto, Canada – http://www.cebm.utoronto.ca/
Centre for Health Evidence – http://www.cche.net/about.asp
The Cochrane Collaboration – http://www.cochrane.org/
Users’ Guides Interactive Learner Web site – http://www.usersguides.org/

Because there is such a huge volume of medical literature published each year, it is important to take a systematic approach to what you appraise thoroughly (Box 26.2). The first step in appraisal is to screen the paper to see if it is worthy of careful reading. It may be possible to answer these screening questions on reading the title and abstract.

Box 26.2
Systematic method of research appraisal

Step 1 – Screening questions

• Was there any clearly focused question?
• Did the investigators use the right type of study design?

Step 2 – Appraising the study report

1 Was there an appropriate comparison group?

• Concurrent or historic controls?
• Was the study randomized?

2 Was the study based on a pre-specified protocol?

• Did the study have a narrow focus?
• Did the study have appropriate end points?
• Did the investigators stick to the protocol?

3 Who knew, what and when?

• Blinding?
• Allocation concealment?

4 Which patients were included or excluded and what happened to those lost to follow-up or who went off protocol or were left out?

• Representative sample for the intervention studied?
• Whether all patients randomized are included in the final analysis?

5 How much did the outcomes change and were they measured and reported appropriately?

• Whether appropriate endpoints and validated instruments were used?
• Were measurements appropriate and done properly?
• Which treatment is better and how much?
• Appropriate sample size?

Step 1 – Screening questions

• Was there a clearly focused question? Important aspects include the population studied, the interventions and the outcomes used.
• Did the investigators use the right type of study design for the question being asked? If there is a clearly focused question that is addressed by a trial of appropriate design it is worth continuing on to a full appraisal of the results. However, do not save time by simply reading the abstract (Example Box 26.1).

Example Box 26.1
Abstracts

There is a tendency in abstracts to report only the statistically significant outcomes. Authors may ignore clinically important outcomes and may fail to report or discuss issues important to the validity of the study.

Piver MS, Barlow JJ, Vongtama V, et al. Hydroxyurea: A radiation potentiator in carcinoma of the uterine cervix: A randomized double-blind study. Am J Obstet Gynecol 1983;147:803–808.

In this trial the authors reported in the abstract and text, a highly significant benefit from the addition of hydroxyurea to radiation therapy for cervix cancer. Only when reading the table on side effects does it become evident that data on patients dying from causes other than cancer were censored. These included a significant excess of deaths from infection, bowel perforation and leukaemia in the hydroxyurea group. If these patients, whose deaths were potentially due to treatment toxicity, are included in an uncensored analysis the highly significant advantage for hydroxyurea disappears. These facts are not mentioned in the abstract or discussion.

Step 2 – Appraising a paper reporting a trial

There are a number of crucial questions that need to be answered (Box 26.2):

1 Was there an appropriate comparison group?

‘Though most of the facts were familiar to me, I had not sufficiently appreciated their relative importance, nor their connection to each other.’

Watson in Silver Blaze

The main aim is that the control group be identical to the intervention group in everything apart for the intervention itself.

Concurrent or historical controls?

When a new therapy is being tested some investigators will give the new treatment to all the patients and will then compare the outcomes with those obtained from the records of similar patients treated in the past – so-called historical controls. Any such comparison is fraught with danger as factors other than the new treatment may have change over time.

Did the authors use randomization?

The purpose of randomization is to ensure, as far as possible, that all factors (known and unknown) that may influence the treatment outcome are balanced between the treatment groups. This is done to reduce the risk of a chance bias. Randomization requires the use of a random device; this is normally a table of random numbers. Systematic allocation; i.e., alternating between treatments, odd or even birth date or hospital number is not an acceptable method, though it is sometimes referred to as pseudo-randomization. Since the researchers know which treatment each person is allocated to before they consent, selective allocation can occur, which will skew the results. Hence the process used for randomization must ensure that neither the trial subjects nor the investigator can influence the treatment arm each person ends up in (‘allocation concealment’).

Despite their pre-eminence, RCTs do, however, have their limitations. They are usually carried out in a narrowly defined research setting. Protocols require careful selection of patients with specific characteristics. There may be a proportion of patient eligible who chooses not to participate. Because of these factors, patients in an RCT may not be typical of patients in routine health practice.

2 Was the study based on a pre-specified protocol?

A protocol written before the trial starts is a prerequisite to good research and some journals are now recommending that the original protocol be submitted with the final paper to ensure that there was such a protocol and to identify any deviations from the original design. Any deviation from the original study design can result in skewed observation of clinical benefit.

When reading a paper bear in mind the following elements of trial design:

Was there a clearly defined research question?

A good study has limited objectives that are based on a biological hypothesis. Studies with no focus and without a clearly defined question are in danger of becoming a ‘fishing expedition’ with multiple comparisons.

The chances of multiple testing increases when there is numerous subgroups or endpoints or when data are reported on multiple side effects. This is particularly problematic when such comparisons are unplanned.

Did the study have appropriate end points?

Most RCTs in cancer therapy concentrate on reporting response rates (a surrogate end point of clinical benefit), disease free survival and overall survival. However, accepting that cure is out of the reach of many patients with cancer, information on symptom control and quality of life are often of major importance; they are frequently ignored in the primary research.

Optimal cut points and the problem with multiple comparisons

‘While the individual man is an insoluble problem, in aggregate he becomes a mathematical certainty. For example you can never foretell what any one man will do, but you can say with some precision what an average number will be up to. Individuals vary, but percentages remain constant.’

Sherlock Holmes, in The Sign of The Four

Where there is a continuous outcome measure, it is common to simplify analysis by defining one or more points that are used to characterize risk. The way that these points are selected may have a major effect on the analysis.

Subgroup analysis

Subgroup analyses are fairly common in trials; they use the data from a study to compare one endpoint across different subgroups. There are three major problems with this approach:

• The comparison is not randomized.
• The comparison is less precise as there are numbers of patients and those reaching end points is smaller.
• The power calculations for the size in a study may be invalidated by a large number of subgroups.

Where a subgroup is found to behave differently, you should consider if there is a plausible biological mechanism for this and whether other trials have found a similar finding. Where there is a statistical analysis, this should be done as a formal test of interaction. Strong empirical evidence suggests that post hoc subgroup analyses often lead to false positive results (Example Box 26.2).

Example Box 26.2
International study of infarct survival

Sleight P. Subgroup analyses in clinical trials: fun to look at – but don’t believe them! Curr Control Trials Cardiovasc Med 2000;1(1):25–27.

Analysis of subgroup results in a clinical trial is surprisingly unreliable, even in a large trial. This is due to a combination of reduced statistical power, increased variance and the play of chance. Reliance on such analyses is likely to be erroneous. Plausible explanations can usually be found for effects that are, in reality, simply due to the play of chance. When clinicians believe such subgroup analyses, there is a real danger of harm to the individual patient.

In order to study the effect of examining subgroups the investigators of the ISIS trial, testing the value of aspirin and streptokinase after MI, analysed the results by astrological star sign. All of the patients had their date of birth entered as an important ‘identifier’. They divided population into 12 subgroups by astrological star sign. Even in a highly positive trial such as ISIS-2, in which the overall statistical benefit for aspirin over placebo was extreme (p < 0.00001), division into only 12 subgroups threw up two (Gemini and Libra) for which aspirin had a non-significantly adverse effect (9% ± 13%).

ISIS-2 was carried out in 16 countries. For the streptokinase randomization, two countries had non-significantly negative results, and a single (different) country was non-significantly negative for aspirin.

There is no plausible explanation for such findings except for the entirely expected operation of the statistical play of chance. It is very important to realize that lack of a statistically significant effect is not evidence of lack of a real effect.

3 Did the investigators stick to the protocol?

Clinical research is rarely predictable, so changes or alterations to a protocol are often needed. However, they should be clearly reported, as major deviations may reduce the reliability of the data.

Buy Membership for Hematology, Oncology and Palliative Medicine Category to continue reading. Learn more here

Related posts:

Cancer in pregnancy Cancers of unknown primary Head and neck cancer Cancers of the genitourinary system Cancers of the musculoskeletal system Cancers of the haematopoietic system