Study Design and Sample Size

Prognostic studies are inherently longitudinal in nature, most often performed in cohorts of patients who are monitored over time for an outcome to occur. The cohort is defined by the presence of one or more particular characteristics (e.g., having a certain disease, living in a certain place, being of a certain age, or simply being born alive). For example, we may track a cohort of patients admitted to the hospital with severe TBI for mortality.

A sufficient sample size is important to address any scientific question with empirical data. The effective sample size is determined mainly by the number of events in the study and not by the total number of subjects. For example, when we study a disease with a 1% chance of mortality, a study with 1000 patients will contain only 10 events, and this number determines the effective sample size. It is important to match ambitions in research questions with the effective sample size that is available. When the sample size is very small, we should ask only relatively simple questions, whereas more complex questions can be addressed with larger sample sizes.

Predictors

The choice of a predictor is based on knowledge of the subject matter: is a certain factor expected to have an effect on outcome? How the predictor should preferably be analyzed depends on the type: predictors can be continuous (age), ordinal (GCS score), categorical (pupil reactivity), or binary (present/absent). Ideally, predictors are well defined, not too costly to obtain, and reliably measurable by any observer. In practice, observer variability is a problem for many measurements. In addition, some measurements are prone to biologic variability, and a single measurement may be misleading, as for instance in the case of blood pressure.

In many studies, continuous or categorical predictors are collapsed into a binary variable by using threshold values. For example, the association between age and outcome has frequently been analyzed at a threshold value of 50 years. This approach has disadvantages.⁷ First, it is unnatural: would risks really be very different for patients who had their 50th birthday yesterday versus having their birthday tomorrow? In addition, a patient 30 years of age may have a different risk than a patient 49 years old. Yet both are below a threshold of 50. Second, from a methodologic perspective, collapsing an ordinal or continuous scale into a binary variable (dichotomization) leads to loss of information and is therefore statistically inefficient. In general, it is better to exploit the full information available. The same principle applies to the approach to analysis of outcome.

Missing Data

Missing data are a common, but as yet underappreciated problem in medical scientific research. Missing values lead to a more limited set of patients with complete data as opposed to the ideal situation of complete original data.

The best solution for missing values is of course to ensure that no data are missing. If we nonetheless have missing data, a common statistical approach is to delete patients with missing values from the analysis. This is often referred to as a complete case analysis.^8,9 Complete case analysis discards information from patients who have information on some (but not all) predictors. It is hence statistically inefficient, especially when we consider multiple predictors. Moreover, complete case analysis may lead to bias because of systematic differences between patients with complete and patients with missing data. Bias occurs when absence of a predictor is associated in some way with the outcome.¹⁰

Most statisticians now agree that we may opt for a more sophisticated statistical approach to deal with missing values in predictive regression models: single or multiple imputation.¹¹ Imputation methods substitute the missing values with plausible values so that the completed data can then be analyzed with standard statistical techniques. As in any statistical analysis, sensible judgement of the analyst based on knowledge of the subject and the research question is important. In practice, many clinicians are unaware of the problems inherent in complete case analyses and are ignorant of modern developments for dealing with missing data, in particular, the use of multiple imputation methods.

Outcome Measures for Prognostic Studies in Patients with Traumatic Brain Injury

In prognostic analysis, the outcome measure chosen should be clinically relevant, and “hard” end points are generally preferred. Mortality is often used as an end point in prognostic research, but global outcome measures (e.g., GOS), nonfatal events (e.g., disease recurrence), patient-centered outcomes (e.g., scores on quality-of-life questionnaires), or wider indicators of burden of disease (e.g., absence from work) may also be used. Whatever the end points chosen, assessment at a fixed time point is essential. Statistical power may also direct the choice of outcome. When an outcome is very infrequent, it is not suited as an end point for statistical analysis.

Most, if not all prognostic studies of TBI have used the GOS or mortality as end points. In most cases the GOS, which may be considered an ordinal scale with five categories, was collapsed into a dichotomous variable differentiating unfavorable from favorable outcome (Table 340-1). In the use of a dichotomous outcome measure, statistical power is greatest when there is a 50 : 50 distribution between outcome categories.

In TBI it is better to quantify prognostic effects across the full range of the GOS rather than after dichotomization into a binary variable. For this purpose, proportional odds methodology is appropriate.¹² The 8-point extended GOS (GOSE) has been introduced to increase the sensitivity of outcome assessment. The use of a structured interview is further advocated to obtain more consistency in outcome assignment.¹³ In severe TBI, the outcome distribution according to the 5-point GOS is U shaped, with most patients either in the lowest (dead) or highest (good recovery) categories. This U-shaped distribution of outcome has promoted the common practice of dichotomizing the GOS for analysis. There is still insufficient knowledge on how introduction of the GOSE may have changed the outcome distribution. It should be noted that the potentially increased sensitivity of the GOSE is totally lost when this is again dichotomized to a binary scale.

Despite the increased sensitivity of the GOSE, it remains a fairly global scale with broad categories aiming to capture functional reintegration without discriminating between physical and mental disabilities. More specific tools for outcome assessment include the Functional Independence Measure (FIM),¹⁴ neuropsychological tests, and quality-of-life assessments. The FIM is considered more appropriate for monitoring the course of a patient through rehabilitation, and neuropsychological tests are considered mandatory for assessing cognitive function, which is so often disturbed after TBI. Generic health-related quality-of-life measures, such as the 36-item short-form health survey (SF-36), are routinely used in many fields of medicine but have not yet become common in assessment of outcome after TBI. These generic health scales may not actually capture relevant domains after TBI, and for this reason the disease specific QOLIBRI scale (Quality of Life after Brain Injury) has been introduced.¹⁵ By definition, quality-of-life scales report the subjective experience of the patient or caretaker and consequently provide a different perspective from the possibly more objective assessment by health care professionals. Currently, there is much interest in the development of composite outcome measures for TBI in which the advantages of more detailed information from the components can be summarized. Further research is required to determine the ultimate benefits from this approach, both for prognostic analysis and in the context of clinical trial design.

Approaches to Prognostic Analysis

The first step in prognostic analysis is identification of the association between a single prognostic factor and outcome (univariate analysis). We might ask, for example, what is the effect of motor score at admission on the 6-month GOS? It should be stressed that a univariate association does not take into account the role of other predictors that may be more important or even may account for the observed association. This association does not represent causality, and the association may be secondary to other more relevant predictors.

The second step, therefore (multivariate analysis), focuses on the unique predictive value of that predictor over and above that of other covariates. Questions that require multivariable analysis are, for example, what are the most important predictors in a certain disease? Are some predictors correlated with each other such that their apparent predictive effects are explained by other predictor variables? To perform multivariate analysis, more predictors are added to the regression model as independent variables.

The third step (prognostic modeling) depends on combining information from the different individual prognostic features into a prognostic model with the aim of giving the best predictions for individual patients. The relevance of a predictor is a function of association of the predictor with the outcome and the distribution of the predictor. For example, a dichotomous predictor with an odds ratio (OR) of 2.0 and a 50% prevalence is more relevant for a prediction model than a dichotomous predictor with an OR of 2.5 and a 1% prevalence.

Predictors for inclusion in the model are usually selected from a stepwise selection procedure: we define a P value to include or exclude predictors, and the statistical package defines the final regression model based on this P value.

The relationship between predictors and outcome can be quantified in several ways (Tables 340-2 and 340-3).

TABLE 340-2 Performance Measures of Predictors

MEASURE	DEFINITION	INTERPRETATION
Relative risk (RR)	Risk for outcome in a group with a predictor/Risk for outcome without a predictor	For example, an RR of 2 means that the group with the predictor has twice the risk of the group without the predictor. When the predictor is continuous, RR represents the increase per unit
Odds ratio (OR)	Ratio of the odds for a better versus a poorer outcome in the presence of the parameter (a/b) as compared with the odds in the absence of the parameter (c/d)	If the prognostic factor is not associated with outcome, the OR will be 1. In reporting the OR, the 95% confidence interval (CI) is frequently included. Statistical significance of the relationship is present if the CI does not include the value 1
R2	Model sum of squares/Total sum of squares (Sum of squares is parameters of the regression model)	Percentage of variability in the outcome that is explained by the predictor or predictors
Sensitivity	Number of true positives/Total number with the outcome	Proportion of patients with the outcome who have the predictor (true positive)
Specificity	Number of true negatives/Total number without the outcome	Proportion of patients without the outcome who do not have the predictor (true negative)
Positive predictive value (PPV)	Number of true positives/Number of positives	Proportion of patients with the predictor who do have the outcome
Negative predictive value (NPV)	Number of true negatives/Number of negatives	Proportion of patients without the predictor who do not have the outcome.

Data from Vittinghoff E. Regression Methods in Biostatistics: Linear, Logistic, Survival, and Repeated Measures Models. New York: Springer; 2005; and Altman DG. Practical Statistics for Medical Research. New York: Chapman & Hall; 1991.

TABLE 340-3 2 × 2 Table for Explanation of Performance Measures

In the first edition of the Guidelines on Management and Prognosis of Severe Head Injury published by the Brain Trauma Foundation in 2000, the positive predictive value (PPV) was used as main measure for expressing prognostic performance. The PPV, however, has limited value because it does not take the frequency at which a predictor occurs within the population into account. Currently, the most widely used measure for expressing the strength of association in prognostic analysis is the OR, which can be obtained directly from the output of a regression model. In multivariable analysis, the ORs provided by the regression model are adjusted for the other predictors in the model. The R² is also provided by the output of the regression model. The difference in R² between a model without and with a certain predictor is the percentage of the variance that is explained by that predictor above the predictors in the former regression model.

Building Blocks for Prognostic Analysis

A wealth of literature has focused on the associations between predictors and outcome in univariate analysis. Most studies have concentrated on patients with severe and moderate TBI. Fewer studies have included multivariable analysis, and two systematic reviews on prognostic modeling have shown the shortcomings of many of the studies that reported on prognostic models previously. Much information on the univariate association between predictors and outcome is contained in the section “Early Indicators of Prognosis in Severe Traumatic Brain Injury” of the Brain Trauma Foundation’s Guidelines on Management and Prognosis of Severe Head Injury, first published in July 2000. More recently, the IMPACT study group reported the results of extensive prognostic analyses performed in a meta-analysis of individual patient data from eight randomized controlled trials and three observational series that included more than 9000 patients.¹⁶ A series of papers reported details correlating the GOS and demographic characteristics,¹⁷ cause of injury,¹⁸ GCS and pupil response,¹⁹ secondary insults,²⁰ blood pressure,²¹ computed tomography (CT) scan features,²² and laboratory parameters.²³ The results of multivariable analyses reporting also on the added predictive value were presented in the same series by Murray and colleagues.²⁴

Conceptually, the main predictors of outcome after TBI can be grouped together into “building blocks,” some of which are modifiable and some not (Table 340-4).

TABLE 340-4 Building Blocks for Prognostic Analysis

CT, computed tomography.