Concrete Examples of the Three Classes of Statistics in a Research Article

To provide a concrete example of these sometimes abstract concepts, consider a fictional study based on the simplest research design in psychiatric research: a randomized double-blind trial of a new drug versus a placebo pill for obsessive-compulsive disorder (OCD).

Figures 62-1 through 62-3 contain the annotated Method and Results sections for this fictional study, showing how the various psychometric statistics are presented in the Method section, while descriptive statistics are presented in the Method and Results sections, and inferential statistics are presented in the Results section (for definitions of terms used in these figures, refer to the section on statistical terms and their definitions).

Figure 62-1 Fictitious Method section annotated to illustrate psychometric statistics.

Figure 62-2 Fictitious Results section annotated to illustrate descriptive statistics.

Figure 62-3 Fictitious Results section annotated to illustrate inferential statistics.

Experiment-wise Error Rate

Researchers should test only a few carefully selected hypotheses (specified before collecting their data!) if their obtained p-values are to have any meaning. The more statistical tests you perform, the greater the chance of finding at least one significant by chance alone. Table 62-2 illustrates this phenomenon.

Table 62-2 Experiment-wise Error: Did the Researcher Find a Single Result Significant Solely by Chance?

* Experiment-wise error rate.

One should not be impressed by a researcher who conducts eight t-tests, finds one significant at p < .05, and proceeds to interpret the findings as confirming his theory. Table 62-2 shows us that with eight statistical tests at p < .05, the researcher had a 33% chance of finding at least one result significant by chance alone.

Selecting an Appropriate Statistical Method

The two key determinants in choosing a statistical method are (1) your research goal, and (2) the level of measurement of your outcome (or dependent) variable(s). Table 62-3 illustrates the key characteristics of the various levels of measurement and provides examples of each.

Table 62-3 Levels of Measurement of Variables

Once the level of measurement of your outcome variable has been determined, you will decide whether your research question will require you to compare two or more different groups of subjects, or to compare variables within a single group of subjects. Tables 62-4 and 62-5 will help you choose the appropriate statistical method once you have made these decisions. (Note that these tables consider only univariate statistical tests; multivariate tests are beyond the scope of this chapter.)

For example, if you want to conduct a study comparing a new drug to two control conditions, and your outcome measure is a continuous rating scale, Table 62-4 indicates that you would typically use the analysis of variance (ANOVA) to analyze your data. If you wanted to assess the associa-tion of two continuous measures of dissociation and anxiety in a single depressed sample, Table 62-5 indicates that you would usually select the Pearson correlation coefficient. (Note that the procedures listed for Ranked outcome measures are those typically referred to as “nonparametric tests.”)

The Importance of Assessing Statistical Power

A nonsignificant p-value is meaningless if the researcher studied too few subjects, resulting in low statistical power. Tables 62-6 through 62-8 will help you estimate the number of subjects required to have a reasonable chance (usually set at 80%, or power = .80) of detecting a true effect.

For example, a researcher reports that she has compared two groups of 12 depressed patients, and found that a new drug was not significantly better than placebo at p < .05, by t-test. However, this negative result is not informative, because Table 62-6 indicates that with only 12 subjects per group, this researcher had statistical power of less than .80 to detect even a “large” effect; that is, even if the drug were truly effective, this study had less than a 50/50 chance of finding a significant difference.

Power analysis is now required as part of virtually all grant and institutional review board (IRB) applications submitted today.

Analysis of Covariance (ANCOVA)

This is a form of ANOVA that tests the significance of differences between group means by adjusting for initial differences among the groups on one or more covariates. As an example, a psychologist interested in studying the effectiveness of a behavioral weight loss program versus self-dieting includes pretreatment weights as a covariate.

Analysis of Variance (ANOVA)

This is an optimal test of significance of difference among means from three or more independent groups. As an example, if a medical researcher wants to compare the effects of three or more different drugs on a single dependent measure, he or she would compute a one-way ANOVA. The more complex, factorial ANOVA also tests for interaction effects between multiple factors. For example, if the two factors being tested were “drug/placebo” and “male/female,” the ANOVA interaction test may find that the drug is more effective than the placebo in the female subjects only. The significance of the analysis of variance is tested by the F-statistic.

Analysis of Variance (ANOVA) with Repeated Measure(s)

This is the optimal test of significance for comparing continuous variables that are obtained through repeated measurements of the same subjects (because each subject’s scores are usually correlated, the regular ANOVA would give results that are “too significant”). An experimenter may select a repeated-measures design because these are generally more sensitive to treatment effects (i.e., they have high power), since score differences between subjects are ignored.

ANCOVA

See Analysis of Covariance.

ANOVA

See Analysis of Variance.

Canonical Correlation

This is a generalization of multiple regression to the case of multiple independent variables and multiple dependent variables. It is rarely used today, except in neuroimaging studies with hundreds or thousands of correlated measurements. It is considered a multivariate statistical procedure because many intercorrelated variables are analyzed simultaneously.

Chi-Square (χ²) Test

See Contingency Table Analysis by Chi-Square.

Cluster Analysis

This is a data reduction technique used to group subjects together into subgroups (or “clusters”) based on their similarities or differences on a set of variables. This technique answers questions such as, “Do my subjects fall into subgroups?” and “What variables give a profile that distinguishes subgroups of my subjects?” A simple rule of thumb is “cluster analysis groups people, while factor analysis groups variables.” It is considered a multivariate statistical procedure because many intercorrelated variables are analyzed simultaneously.

Confidence Interval

A confidence interval does more than simply report that our observed mean difference between two groups is 2.5 points, significant at p < .05; it is far more informative to report that the 95% confidence interval around our observed mean is 0.6 to 4.4. Since 0 (the null hypothesis value of the mean difference) is not included in the 95% confidence interval, we know at a glance that the difference is significant at p < .05, but we also learn of the possible values of the actual mean difference between the groups, ranging from as low as 0.6 points to a high of 4.4 points (with 95% confidence). In the case of odds ratios, a confidence interval of 0.60 to 4.40 would not be significant at p < .05, because the null hypothesis would state that the odds ratio is 1.00, and this is included in the computed 95% confidence interval. Many journals require the reporting of confidence intervals, instead of using only p-values.

Contingency Table Analysis by Chi-Square (χ²)

This is a test to determine whether the frequencies in each cell of a contingency table are different from the proportions expected by chance. It is most commonly used on a 2 × 2 contingency table, represented as four cells forming a square.

A common use is to answer the following question: “Is there a difference between the occurrence of a given side effect in the drug versus a placebo group?” In this case the table is arranged with drug versus placebo as the two rows, and side effect versus no side effect as the two columns. As the (squared) difference between the observed and expected frequencies in each cell increases, the chi-square statistic also increases, and the more significant the result becomes. If all cells contain exactly the frequencies that would be expected by chance, the chi-square statistic is zero. If the frequencies differ greatly from chance, the chi-square statistic gets larger and larger. The size of the chi-square statistic is based on the number of cells in the contingency table (since df = [# rows − 1] × [# columns − 1], a 2 × 2 table always has a single degree of freedom).

Correlation Matrix

This is a “Table” or “Matrix” of correlation coefficients for all variables of interest.

Correlation

See Pearson Correlation Coefficient.

Covariate

This is a variable that the investigator believes may influence the outcome or dependent variable and that is to be statistically adjusted for. For example, in a study of a new antidepressant, the baseline level of depression for subjects in each of the two groups may be used as a covariate.

Dependent t-Test

See t-Test for Dependent Means.

Dependent Variable

This is usually the outcome variable of interest in a study; it is also called an “endpoint.” In a study of a new antidepressant drug, a depression rating scale may be used as the dependent variable.

Descriptive Statistics

These are statistics used to describe a single population. Descriptive statistics used commonly to summarize the central tendency of a group are the mean (or arithmetic average) for continuous measures and the median (or “middle score”) for ordinal or ranked measures. Descriptive statis-tics are used commonly to describe the variability within a group. These include the variance and its square root, the standard deviation for continuous measures, and the interquartile range for ranked measures. Researchers look at descriptive statistics first to get a “big picture” of their data (and also to look for data entry errors or obvious outliers).

Discriminant Function Analysis

This is the optimal procedure to statistically distinguish between two or more groups based on a group of discriminating variables. It is an important and underused procedure. It is considered a multivariate statistical procedure because many intercorrelated variables are analyzed simultaneously.

Effect Size

This is a measure of the practice significance of a treatment effect, as opposed to the statistical significance. For comparisons of two treatments, the most common effect size measure used is Cohen’s “d” (the mean difference between the groups in standard deviation units), with values of .20, .50, and .80 defined by Cohen as small, medium, and large effects in the behavioral sciences. Effect sizes are essential to computing statistical power (see Tables 62-6 to 62-8) and also form the basis of meta-analytic research.

Endpoint

See Dependent Variable.

Experiment-wise Error Rate

As more statistical tests are performed, you are more likely to find at least one of these tests significant by chance alone. Although you may perform each test at p < .05, your chance of finding one of many tests significant by chance alone is much higher than this nominal 5% level (see Table 62-2).

Factor Analysis

This is used to statistically reduce the number of variables needed to explain or to describe a larger set of original variables, based on the correlation matrix. For example, the 10 subscale scores of the Minnesota Multiphasic Personality Inventory (MMPI) personality test were derived from the 567 items that make up the questionnaire. As a rule of thumb, “factor analysis groups variables, while cluster analysis groups people.” It is considered a multivariate statistical procedure because many intercorrelated variables are analyzed simultaneously.

Frequencies

See Descriptive Statistics.

Level of Measurement

See Table 62-3.

Logistic Regression

This is an optimal procedure to predict a binary outcome variable from a set of continuous or binary predictor variables. Logistic regression yields both a probability value and an odds ratio. Often used in epidemiological studies where the outcome is occurrence/nonoccurrence of a disease or survival/death. When more than one predictor variable is included, it is considered a multivariate statistical procedure because many intercorrelated variables are analyzed simultaneously.

MANCOVA

See Multivariate Analysis of Covariance.

Mann-Whitney U Test

This is the optimal nonparametric test of significance for difference between the sums of ranks from two independent groups or treatment levels.

MANOVA

See Multivariate Analysis of Variance.

Mean

See Descriptive Statistics.

Median

See Descriptive Statistics.

Multiple Linear Regression

This is used to predict a single continuous outcome variable from a set of continuous or dichotomous predictor variables. It is very flexible because it allows many forms of nonlinear curve-fitting as well, and automatically provides a measure of effect size as R² (this can help assess practical significance, in addition to statistical significance). When more than one predictor variable is included, it is considered a multivariate statistical procedure because many intercorrelated variables are analyzed simultaneously.

Multivariate Analysis of Variance—MANOVA

This is a generalization of ANOVA when multiple dependent variables are to be assessed simultaneously. It is not used often in psychiatric research because of the difficulty that lies with its interpretation. Instead, separate ANOVAs are often computed for each dependent variable. When more than one predictor variable is included, it is considered a multivariate statistical procedure because many intercorrelated variables are analyzed simultaneously.

Multivariate Statistical Analysis

This is a statistical procedure in which multiple correlated variables are analyzed simultaneously to account for their intercorrelation. It is contrasted with univariate analyses. Examples are discriminant function analysis, factor analysis, and MANOVA.

Null Hypothesis

Researchers usually begin with the hypothesis that there is a zero (or “null”) difference between two means, or that there is a correlation coefficient of zero (or “null”) between two measures. Since statistics cannot prove a hypothesis, we usually state the null hypothesis and present data showing that it is unlikely to be true; the p-value indicates just how unlikely. Caveat: In any very large sample, the null hypothesis is likely to be rejected. That is, two groups of individuals rarely have exactly the same mean on any two characteristics; even if they only differ by, say, 0.01 mm, the null hypothesis is not true, and a sample size in the thousands could detect even a tiny difference.

Outcome Variable

See Dependent Variable.

p-Value

This reflects the chance of a result of a statistical test being a false positive (i.e., the probability of a spurious finding). If a finding is almost certainly a spurious finding, which would not be reproducible in another sample, the p-value will be near 1.00. On the other band, if the finding almost certainly represents a “true” finding, the p-value will be near zero (small p-values are represented by several zeros after the decimal point, e.g., p < .0001—but never list a p-value as 0.00, because this is impossible, and it annoys reviewers!).

Most journals require p < 0.05 for significance. If many statistical tests are performed in a study, a more conservative p-value can be used to minimize experiment-wise error (see Table 62-2). Caveat: Do not be overly impressed by very low p-values. Remember that all this tells you is the chance that the difference is probably not zero. Also, remember that, given enough subjects, this is easy to prove. Thus, a very low p-value does not necessarily indicate a large clinical effect, but instead represents that it is a very reliable effect. Check the effect size (the correlation coefficient squared, or the size of the t-statistic) to get an idea of the magnitude of the difference or relation.

Pearson Correlation Coefficient

This is the optimal measure of the association between two continuous measures. When there is no association, r = 0; when there is a perfect positive correlation, r = 1.00; and when there is a perfect negative correlation, r = −1.00. You can easily obtain an effect size measure by squaring r (which gives the percent of variance shared by the two variables); thus, even a correlation of r = .05 can be highly significant with thousands of Ss, however, r² tells us that two variables share only a trivial percent of their variance (here, < 3/10 of 1%!). When researchers discuss correlation, they are almost always referring to the Pearson correlation coefficient.

Phi Correlation Coefficient

This is a special case of the Pearson correlation coefficient when two dichotomous variables are being compared. Note, however, that because the variables are not normally distributed, the maximum possible value of phi is often less than 1.00 or −1.00.

Planned Comparisons

Planned comparisons or contrasts are special kinds of t-tests for a specific subset of hypotheses involving mean comparisons that are formulated before collecting data for a study. Planned contrasts are typically performed instead of a full ANOVA (which is essentially a simultaneous test of all possible contrasts between means in an experiment, while planned contrasts address only a small set of hypotheses, thus providing greater statistical power).

Point-Biserial Correlation

This is a special case of the Pearson correlation coefficient when one variable is continuous and the other is dichotomous. As with the phi coefficient, the maximum value of r_pb often cannot achieve its limits of 1.00 or −1.00.

Power

This reflects the probability of finding a true difference or relation between two or more measures with a given sample size. It is analogous to the sensitivity of a medical test. An analogy: The power of a telescope (i.e., the magnification) is analogous to the power of a study; both indicate the ability to detect even tiny objects or changes. If a study had little chance of finding a true difference between two drugs (say, with only two subjects per group), the power of the study would be nearly zero. If the study had almost no chance of missing a true difference (say, comparing the mean height of 10,000 2-year-olds with 10,000 18-year-olds), the power of the study would be nearly 1.00. Power of 0.80 is usually the minimum required by statisticians when designing a study. Caveat: Just as any tiny difference can be found to be significant just by having enough subjects, it is possible to find all but huge differences to be nonsignificant, simply by having few enough subjects! Be especially wary of false negatives in studies with few subjects (e.g., < 25 per group). Several literature reviews have found that the average behavioral science study has a power of only about 40% to detect a medium-sized effect! Consult Tables 62-6 to 62-8 to estimate the statistical power of a study.

r

See Pearson Correlation Coefficient.

Random Assignment

The most effective method for assigning subjects to treatment groups or conditions is randomization. Randomization ensures that there will be no systematic bias in the makeup of the groups, so that each subject has an equal opportunity to be assigned to any one of the groups. Caveat: Strictly speaking, p-values obtained when comparing two groups are only valid if subjects have been randomly assigned to the two groups!

Rank-Order Correlation

This is a special case of the Pearson correlation coefficient when assessing the association between two variables with ranked data. With only a few tied ranks, use Kendall’s “tau;” with many tied ranks, use Spearman’s “rho.”

Regression

See Multiple Linear Regression.

Reliability

This is the dependability of a score, or the degree to which we can be certain that a measurement can be depended on (i.e., How reproducible is the score?). For self-rated scales, such as paper-and-pencil questionnaires, since there is no rater error to take into account, the main source of unreliability to assess is difference in the person’s self-rating over time. For example, if a patient completes a depression questionnaire at 3 p.m., how close would his or her score be on the same questionnaire if he or she were to take the same scale at 4 p.m., assuming no change in his or her depression? If the scores were identical, and this was the case for all patients, the correlation coefficient would be a perfect 1.00 (in this case the correlation coefficient is referred to as the “reliability coefficient”).

For scales or measures administered by a rater, the major question is, “Would this patient get the same score on this depression scale if Doctor A rated him, as if Doctor B rated him?” If the agreement was perfect for all patients, the reliability coefficient would be 1.00. If, on the other hand, there were a random relationship between the scores of the two raters, the interrater reliability would be 0.00. Reliability is necessary but not sufficient for a useful scale. Thus, a scale can be perfectly reliable, but have no validity for a particular purpose. For example, every time you ask me my phone number I will give you the same answer (perfect reliability); however, if you attempt to use my phone number to predict my anxiety level you will find a zero correlation (no validity). If a measure has no reliability (i.e., it is not reproducible), it has zero reliability. If it has perfect reliability, or repeatability, it has a reliability of 1.00.

For a continuous measure, this is assessed by the correlation coefficient, and as a rule of thumb it is generally r = 0.80 for adequate reliability. For a binary measure (e.g., the presence or absence of a disease), this is often assessed by the kappa coefficient, and a rule of thumb is generally κ = 0.70.

Rho

See Rank-Order Correlation.

Standard Deviation

See Descriptive Statistics.

Statistical Power

See Power.

Tau

See Rank-Order Correlation.

t-Test

This is the optimal test to determine whether the difference between the means of two samples of subjects is likely to be due to chance alone. If the difference between two means is zero, the t-statistic will also be zero. If all scores in both groups are the same, they both have standard deviations of zero. In this case, the t-statistic will be infinitely large.

t-Test for Dependent Means

This is the optimal test of significance for difference between means from two matched, or dependent, groups or treatment levels. As an example, a group of 30 anxious patients participated in a study of a new drug for alleviating anxiety. Anxiety levels were measured by the Hamilton Anxiety Scale at baseline and again after 3 weeks of treatment. In this test each subject’s pair of scores is directly compared, and the variation between the pairs of subjects is ignored. The same test is used to compare the means of two matched pairs of subjects (whose scores are thought to be correlated).

Univariate Statistical Analysis

There is only a single dependent variable to be considered in each analysis. Examples are t-test and ANOVA.

Validity

This is the degree of usefulness of a rating scale for a particular purpose. It is also the degree to which the test measures what it is supposed to be measuring. The determination of validity usually requires independent, external criteria of whatever the test is designed to measure. For example, if an investigator develops a single question that he or she purports to be a good screening instrument for clinical depression, patients’ responses to this question should relate well to “gold standard” measures of clinical depression, such as structured interviews and well-established rating scales for depression. There is no real rule of thumb for validity, since there is no one measure. As a bare minimum, however, the scale should at least be significantly correlated with gold-standard measures for that characteristic.

Variance

See Descriptive Statistics.