Was the assignment of patients random?

Randomization ensures that both known and unknown determinants of outcome are evenly distributed between treatment and control groups, if the sample size is sufficiently large.

Were all patients accounted for and were patients analyzed in the groups to which they were randomized?

Every patient who entered the trial should be accounted for at its conclusion. If a substantial number of patients are reported as “lost to follow-up,” the validity of the study is open to question. Patients who are “lost” often have different prognoses from those who are retained; consequently, many disappear because they have suffered adverse outcomes or because they improved before the end of the study. By preserving randomization during data analysis, all outcomes will be equally distributed in the two groups and the ultimate effect can be attributed to the treatment.

Was the study “blinded”?

The best way to avoid bias is to make the study “double-blind,” so that neither investigator nor patient knows the treatment being used. For studies in which patients and treating clinicians cannot be kept blind (e.g., surgery vs. no surgery), you should note whether investigators have minimized bias by blinding those who were responsible to assess the final clinical outcomes.

Were the treatment and control groups similar?

The only important difference between groups should be whether they received the experimental therapy. Group characteristics are usually displayed in a table early in the article.

Were the groups treated equally?

Differences in care other than that under study can weaken or distort the results.

How large was the treatment effect?

Consider a study in which death is the outcome for 20% (0.20) of patients in a control group but only 15% (0.15) of patients in a treatment group (Table 7-1). Results might be expressed in terms of absolute risk reduction or relative risk reduction.

Table 7-1 Evaluating the Results of a Therapeutic Trial

A positive RRR means that the new therapy is better than the old therapy or the control; the greater the RRR, the more effective the therapy. RRR of zero indicates that the new therapy is no more efficacious than the old therapy or control. A negative RRR means that the new treatment is actually more harmful. If the authors do not specifically report RRR or ARR, use the data to calculate them for the outcomes that interest you. Be careful with relative reductions: You must know the absolute difference to interpret RRR. For example, ARR from 7% to 5% may not be impressive, yet it has the same RRR (25%) as ARR of 20% to 15%.

How precise was the estimate of treatment effect?

The best estimate of the true treatment effect is expressed by the confidence interval (CI), which is usually reported along with P values. The 95% CI includes the true RRR 95% of the time. The chance that the true RRR lies toward the extremes of this interval is low, and the chance that the true RRR lies beyond these extremes is only 5% (a property of the CI that closely relates to the “statistical significance” of P < 0.05). Thus if a study reports that RRR was 15% with a CI of 9% to 18%, then the treatment effect is positive. If the CI were reported as –5% to 17%, you cannot be sure of the efficacy of treatment because RRR ranges from positive (efficacious), to zero (no effect), to negative (adverse effect). In addition, the larger the sample size, the narrower the CI and the more likely the effect is real. If the CI is not reported, examine the P value. If the P value is exactly 0.05, then the lower bound of the 95% CI for the RRR lies exactly at 0 (no effect). As the P value decreases, the lower bound of the 95% CI for the RRR rises above 0 and an effect is more likely.

Can the study results be applied to your patient?

In general, if your patient meets all of the inclusion criteria for a therapeutic trial and does not violate any of the exclusion criteria, the results of the study are applicable. If this is not the case, ask yourself whether there is some compelling reason why the results should not be applied to your patient. Take the patient’s values into consideration.

Were all clinically important outcomes considered?

Often, researchers must substitute study endpoints that may not truly reflect the actual desired clinical endpoint. You will need to decide if these substitute outcomes are appropriate or useful.

Do benefits outweigh potential harms and costs?

Benefits are best assessed using the “number needed to treat” (NNT) (Table 7-1). The NNT is determined by the question “How many patients would need to be treated in order to prevent one adverse outcome?” In the earlier example, you would have to treat 20 patients to prevent 1 death. In addition, you need to ask whether the benefits of a given therapy outweigh its adverse effects for your patient(s). For example, treatment of mild asthmatic patients with ipratropium might result in 40 patients developing significant tachycardia for every life saved, whereas treatment of only severe asthmatics might result in 4 patients with significant tachycardia for every life saved. You need to consider your patient’s asthma and clinical status before deciding to use ipratropium for the next asthma exacerbation.