Examples of Analysis of Variance and Covariance
C. P. Doncaster and A. J. H. Davey
This page presents example datasets and outputs for analysis of variance (ANOVA) and covariance (ANCOVA), and computer programs for planning data collection designs and estimating power. All of the statistical models are detailed in Doncaster and Davey (2007), with pictorial representation of the designs and options for troubleshooting common issues with analysis.
Click here for the suite of commands in R (freeware statistical package, R Development Core Team 2010) that will analyze each of the example datasets below, and calculate the power of the design.
Contents
- What is a statistical model?
- Examples of ANOVA and ANCOVA models
3 Fully replicated factorial designs
- Analyses for figures and worked examples in Doncaster and Davey (2007)
- Computer programs for planning designs and estimating design power
- Key to types of statistical models
- Glossary
What is a statistical model?
Any statistical test of pattern requires a model against which to test the null hypothesis of no pattern. Models for ANOVA and ANCOVA take the form: Response = Factor(s) + ε, where the response refers to the data that require explaining, the factor or factors are the putative explanatory variables contributing to the observed pattern of variation in the response, and ε is the residual variation in the response left unexplained by the factor(s). For each of the examples illustrated here, we use a standard notation to describe the full model and its testable terms. For example, the two-factor nested model in Section 2 below is described by:
(i) The full model, packed up into a single expression: Y = B(A) + ε;
(ii) Its testable terms to declare in a statistics package, unpacked from the full model: A + B(A).
A statistics package will require you to specify the model desired for a given dataset. You will need to declare which column contains the response variable Y, which column(s) contain the explanatory variable(s) to be tested, any nesting or cross factoring of the explanatory variables, whether any of the variables are random rather than fixed factors, and whether any are covariates of the response.
Examples of ANOVA and ANCOVA models
Each of the links in Sections 1 to 7 below shows a full suite of analyses of a hypothetical dataset. Where appropriate, these include alternative restricted and unrestricted models (Searle 1971), and Model-1 and Model-2 designs (Newman et al. 1997). Refer to the protocols in Doncaster and Davey (2007) to see which mean squares are used for the F-ratio denominators, and consequently how many error degrees of freedom are available for testing significance. The examples have not used post-hoc pooling though this may be an option or an alternative to some quasi F-ratios, and the underlying assumptions have not been evaluated though this would need doing for real datasets.
A statistics package may give different F ratios and P values to those shown here, for a variety of reasons:
- In unbalanced and non-orthogonal designs and ANCOVA models, default use of Type-III adjusted SS for models that require Type II.
- In balanced mixed models and ANCOVA models, default use of an unrestricted model when the design may suite a restricted model.
- In designs with randomized blocks and split plots, default use of Model-1 analysis when the test hypothesis may require Model 2.
- In ANCOVA mixed models, default use of residual error for all denominator mean squares when the test hypothesis may have a different error variance.
Click here for the suite of commands that will analyze each of these example datasets in the freeware statistical package R (R Development Core Team 2010).
Computer programs for planning designs and estimating design power
1. List all terms and degrees of freedom in any model for analysis of variance or covariance. Click here to download a computer program (Terms.exe) that will list all of the main effects and interactions and their degrees of freedom for a model of your own specification with any number of levels for each of any number of cross-factored or nested categorical or continuous variables (yielding up to a maximum of 2047 terms).
Specify the model as a hierarchical nesting of sampling units in factors, representing each variable by a single letter. Thus for example, requesting: 'P(B|S(A))' will yield testable terms for any of models 3.3, 5.6 or 6.3 above, depending on the nature of the variables and the replication of sampling unit P.
A text file 'Terms.txt' will be created to store the list of terms comprising all main effects and interactions and their degrees of freedom; it will also contain the N lines of factor-level combinations against which to tag your N observations of the response.
2. List testable terms, degrees of freedom, and critical F-values for any of the numbered designs above. Click here to download a computer program (CritiF.exe) that allows you to specify your own sample sizes in the numbered ANOVA and ANCOVA designs above, assuming fixed treatment factors. For each estimable effect, it shows the test and error degrees of freedom, and the critical F at α = 0.05. For fixed effects, it shows the standardized effect size with 80% detection probability (the value of θ /σ that gives the test a power of 0.8).
Use the program to evaluate alternative experimental designs for a given workload of data points, targeting a low standardized effect size for treatments. This value will vary according to the distribution of data points between levels of sampling units and treatments. For a given total data points, it will be increased by the inclusion of nesting, covariates, blocking, split plots, or repeated measures. These may be desirable or intrinsic features of the experimental design, and they will increase power to detect treatment effects if they reduce error variances sufficiently to compensate for the reduction in error degrees of freedom.
A text file 'Factor_levels.txt' will be created to store the N lines of factor-level combinations against which to tag your N observations of the response.
3. Calculate statistical power for any balanced model. Click here to download a computer program (Power.exe) that calculates statistical power prospectively for fixed or random factors in any balanced model with a proposed size of samples, given a threshold ratio of treatment effect size, θ (the standard deviation of the treatment variability) to error effect size, σ (the standard deviation of the random unmeasured variation). It can also calculate the value of θ /σ required to achieve a target power.
A pilot study may be needed to obtain an initial observed F from samples of size n. Then [(F - 1)/n]1/2 will provide an unbiased estimate of the population θ /σ, with which to evaluate the potential to gain power from more replication (e.g., Kirk 1968). Freeware is available elsewhere on the web to further explore the relationships between n, θ, σ and power for specified designs (e.g., Piface by Russell V. Lenth).
4. Calculate relative performance for any balanced model. Click here to download a computer program (Performance.exe) that calculates the performance of a balanced analysis of variance design relative to a reference design for the same treatment(s). The relative performance of the design is given by the fractional size of its error variance that will just match the power of the reference. The value of relative performance is robustly approximated by the ratio of reference to alternative α quantiles of the F distribution, multiplied by the ratio of alternative to reference effective sample sizes (Doncaster, Davey & Dixon 2014). By comparing the precision of two designs at equal sensitivity, relative performance provides a useful way to enumerate trade-offs between error variance and error degrees of freedom when considering whether to block random variation or to sample from a more or less restricted domain.
5. Find critical F-values for any number of test and error degrees of freedom, and value of α. Click here to download a computer program (Ftable.exe) that provides critical F-values for a chosen α, and also gives the Type-I error probability associated with an observed value of F, given test and error degrees of freedom.
Program citations
Doncaster, C. P. (2007) Computer software for design of analysis of variance and covariance. Retrieved [date] from http://www.southampton.ac.uk/~cpd/anovas/.
Key to types of statistical models
Use this key to identify the appropriate section of model structures above, then look at example datasets and analyses.
1. Can you take observations with independently varying residuals that randomly sample from the populations of interest (i.e., from the levels of each factor or factor combination)?
Yes → 2.
No → identify the dependency structure and explore options to control it in your model, for example by factoring in nuisance variables or subpopulations or by sub-sampling; otherwise the data may not suit statistical analysis of any sort.
Yes → 3.
No → the data may not suit ANOVA or ANCOVA.
Yes → consider treating the continuous factor as a covariate and using ANCOVA designs in Sections 1 to 3 above; this will be the only option if each sampling unit takes a unique value of the factor. The response and/or covariate may require transformation to meet the assumption of linearity. Analyze with a General Linear Model (GLM) and for non-orthogonal designs consider using Type II adjusted SS if cross factors are fixed, or Type III adjusted SS if one or more cross factors are random (and an unrestricted model, checking correct identification of the denominator MS to the covariate). Be aware that adjusted SS can increase or decrease the power to detect main effects.
No → 4.
Yes → 5.
No → 9.
5. Are all combinations of factor levels fully replicated?
Yes → 6.
No → use an unreplicated design (Section 7 above).
Fully randomized and fully replicated designs
6. Do your samples represent the levels of more than one explanatory factor?
Yes → 7.
No → use a one-factor design (Section 1 above), considering options for orthogonal contrasts (e.g., model 1.1 above).
7. Is each level of one factor present in each level of another?
Yes → 8.
No → use a nested design with each level of one factor present in only one level of another (Section 2 above).
8. Use a fully replicated factorial design (Section 3 above), taking account of any nesting within the cross factors (model 3.3 or model 3.4 above). For balanced and orthogonal designs with one or more random cross factors, consider using a restricted model, and consider post hoc pooling if an effect has no exact F-test. If cross factors are not orthogonal (e.g., sample sizes are not balanced), use GLM and consider using Type II adjusted SS if cross factors are fixed, or Type III adjusted SS if one or more cross factors are random (and an unrestricted model).
Stratified random designs
Yes → use a design with randomized blocks (Section 4 above). If all factor combinations are fully replicated, analyze with Section-3 ANOVA tables; otherwise consider analysis by Model 1 (assumes treatment-by-block interactions) or Model 2 (assumes no treatment-by-block interactions). For a single treatment factor, consider options to use a balanced incomplete block or, with cross factored blocks, a Latin square or Youden square.
No → 10.
Yes → use a design with split plots (Section 5 above), taking account of nesting among sampling units.
No → use a repeated-measures design (Section 6 above) for repeated measurement of each sampling unit at treatment levels applied in a temporal or spatial sequence. If all factor combinations are fully replicated, analyze with Section-3 ANOVA tables.
References
Doncaster, C. P. and Davey, A. J. H. (2007) Analysis of Variance and Covariance: How to Choose and Construct Models for the Life Sciences. Cambridge University Press, Cambridge 302 pp. ISBN-13: 9780521684477. https://doi.org/10.1017/CBO978051161137
Doncaster, C. P., Davey, A. J. H. & Dixon, P. M. (2014) Prospective evaluation of designs for analysis of variance without knowledge of effect sizes. Environmental and Ecological Statistics, 21: 239-261. https://doi.org/10.1007/s10651-013-0253-4.
Kirk, R. E. (1968, 1982, 1994) Experimental Design: Procedures for the Behavioral Sciences. Brooks/Cole, Belmont, CA.
Newman, J. A., Bergelson, J. and Grafen, A. (1997) Blocking factors and hypothesis tests in ecology: is your statistics text wrong? Ecology, 78, 1312-20.
R Development Core Team (2010). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org.
Searle, S. R. (1971, 1997) Linear Models. New York: John Wiley.
Glossary
analysis of covariance: see ANCOVA
analysis of variance: see ANOVA
a priori contrasts: see orthogonal contrasts
balanced non-orthogonal designs: see balanced incomplete block and Youden square
balanced complete block: see randomized complete block
balanced incomplete block design
categorical factor: see factor
continuous factor: see covariate
contrasts: see orthogonal contrasts
crossover trials: see Latin squares
effect size: see power
effective sample size: see replication
error variance: see residual error
fixed factor: see factor
general linear model: see GLM
generalized linear model: see GLM
Helmert contrasts: see orthogonal contrasts
independent replicates: see replication
interaction: see cross factor
linear model: see GLM
main effect: see model
mean square: see F ratio
missing cell: see examples of orthogonal contrasts
multiple regression: see ANCOVA
null hypothesis: see hypothesis
observation: see replication
one-way designs: see one-factor designs
orthogonal unbalanced designs: see orthogonal contrasts
planned comparisons: see orthogonal contrasts
random factor: see factor
random replicates: see replication
regression: see ANCOVA
repeated-measures sampling units
sample: see replication
sampling unit: see replication
sum of squares: see adjusted SS and sequential SS
test hypothesis: see hypothesis
treatment: see factor
two-way designs: see factorial designs and randomized-block designs
type-II error: see type-I error and power
variance: see residual error
variance component: see power
Page maintained by C. P. Doncaster
Last updated 2 July 2024