[Coral-List] Interactions between Stressors on Coral Reefs - factorial ANOVA or Multiple Comparison Procedures

Richard Dunne RichardPDunne at aol.com
Thu Jun 3 07:21:01 EDT 2010


Following my last post below, an Excel Spreadsheet is available which 
can be used to compute several Multiple Comparison Procedures (MCP). It 
is modified and adds functions based on a template published by Brown 
(2010). The spreadsheet will simultaneously calculate Fisher's LSD, 
Tukey (both Tukey-Kramer and  Tukey HSD), Dunn-Sidak, Bonferroni, 
Scheffe and Games-Howell. It also computes a one way ANOVA and Hartley's 
Fmax test for homogeneity of variances. These are the main tools for 
conducting MCPs on both homogenous and heterogeneous data sets.

The spreadsheet works with up to 10 treatment groups and an N of 199 per 
group for all MCPs apart from the Games-Howell where the maximum number 
of groups is limited to 8 at present. It also contains 'Instructions' 
and a "Guide to the choice of MCP". All the procedures have been cross 
validated with SPSS. All that is required is for the user to paste or 
enter his/her data in a single worksheet. It is an excellent and quick 
way for the researcher to conduct MCPs on data. It requires no 
specialist statistical knowledge nor does it require access to expensive 
commercial statistics packages (several of which do not contain all of 
these tests in any case).

The output for each MCP consist of the difference between the comparison 
means, and the upper and lower 95% confidence intervals, and whether the 
comparison is significant at the 0.05 level. The output is also 
displayed graphically which makes it very easy to pick out the 
significant comparisons and also the direction and magnitude of the mean 

I can e-mail anyone who would like a copy of the spreadsheet (1.32Mb size).

Brown AM (2010) A spreadsheet template compatible with Microsoft Excel 
and iWork Numbers that returns the simultaneous confidence intervals for 
all pairwise differences between sample means. Computer Methods and 
Programs in Biomedicine 98:76-82

Also John Ware (in his post - Statistics 1 June 2010) kindly pointed out 

While I am sure we all appreciate Richard Dunne's recent and learned
comments on statistical testing, let us not forget that what most
coral-reef scientists are trying to find out is something of real

Too often, we seem to replace "statistical significance" with
"significance".  Clearly there can be statistically significant results
that have no real significance and one can obtain non-statistically
significant results when there is a very real significance of the test
variable (but the effect is not detected due to poor design or a limited

This is an important point in its own right. Statistics is merely a tool 
to help us. Just like other tools and equipment, it needs 'calibration' 
or 'validation' before it is used. Incorrect use of statistical tools 
will lead to incorrect deductions about the 'real' or 'biological' 

Richard Dunne

On 28/05/2010 09:56, Richard Dunne wrote:
> In December last year I drew the attention of readers of Coral List to 
> a Perspective that I wrote for the journal 'Coral Reefs'.
> Coral Reefs DOI: 10.1007/s00338-009-0569-6
> In that paper I examined earlier publications in the coral reef 
> literature that had used factorial ANOVA and illustrated some of the 
> errors that had been made in interpreting interaction effects. One of 
> the authors has defended his work in the face of my criticism. His 
> Comment appears online at:
> CONCLUSIONS - Comment on Dunne (2010)
> Coral Reefs DOI: 10.1007/s00338-010-0625-2
> The author sent me a copy during the peer review process and I 
> provided him with my detailed observations. As a consequence he has 
> included my name in his Acknowledgements. I would like to be clear to 
> readers of the Comment, however, that I do not agree with most of the 
> substantive points that appear and that there are still significant 
> errors. Coral Reefs does not wish to continue further discussion, 
> which is regrettable for there are important issues here. For those of 
> you who are interested, I outline some of these below.
> The author makes the perfectly valid observation that it is possible 
> to use certain multiple comparison procedures (MCP) on their own, 
> without prior protection from ANOVA to examine datasets and that they 
> can be a useful tool in their own right, sometimes with more power to 
> detect differences than ANOVA. I fully agree with this observation 
> which has been discussed in statistical publications over the last 20 
> years but rarely appears in the biological literature. However, he 
> then goes on to propose a methodology to examine interactions, 
> illustrating the technique by using a Tukey HSD test on some of his 
> datasets from 1990 whereby he concludes that interactions are present. 
> It is with this methodology and conclusion that I strongly disagree. A 
> careful reading of the Comment will reveal that the results of certain 
> of the MCP analyses do not support the logic of the interpretations, 
> notably for the MAA and Zooxanthellae numbers data, where Tukey HSD 
> finds homogenous overlapping groups. Furthermore, a MCP conducted in 
> the manner in which it has been used, across all treatment groups, 
> will not reveal interactive effects. This perhaps explains why other 
> more appropriate tools, including factorial ANOVA, exist for this 
> purpose.
> In addition, there is a serious factual error in the Comment which 
> invalidates the analysis of the Zooxanthellae catalase data. It 
> concerns the statement that: "Lesser et al. (1990) conducted a 
> balanced, multi-factorial design experiment, with a non-significant 
> Fmax test of homogeneity of variances on the zoanthid  Palythoa 
> caribaeorum,....". In 1993 I first discussed this with the author and 
> pointed out that some of the data from the 1990 experiment were not 
> homoscedastic. In fact the majority of the datasets (8 of the 13) in 
> that paper fail the Fmax test, and other tests such as Levene and 
> Bartlett. Five of the datasets are extremely heteroscedastic, with 
> Fmax (the ratio of the smallest to the largest variance) being in 
> excess of 100. This includes the Zooxanthella catalase data, where it 
> is 169. I again drew attention to this specific issue in my 
> observations, but this point seems to have been missed.
> Why is this heterogeneity important? It is because the accuracy of 
> most statistical tools is based upon underlying assumptions about the 
> data being analysed. In the case of the Tukey HSD one of these 
> assumptions is that the data are homoscedastic. In this case where 
> there is severe heteroscedasticity, some of the multiple comparisons 
> will have inflated alpha. In other words, some of the comparisons 
> which indicate significant differences at the P=0.05 will be 
> incorrect. This was clearly demonstrated by Kromery and La Rocca 
> (1995) in a study which examined several MCPs and their performance 
> when data were heteroscedastic. For the Tukey HSD they found that for 
> 5 treatment groups and a Fmax of 13, the familywise error rate was 
> inflated to 0.098. In the case of the Zooxanthella catalase data with 
> Fmax = 169 and 8 treatment groups the errors are going to be 
> considerably greater. The words of a leading statistician in the field 
> of multiple comparisons are particularly apt "Editors of many 
> scientific journals have not ....... realize[d] that the multiple 
> comparison analyses performed by some authors may have error rates 
> well beyond any acceptable level." Hsu (1996). Regrettably that 
> situation seems to have occurred once again.
> It is of course possible to avoid these problems by employing other 
> suitable MCPs which do not require homoscedasticity. For example, the 
> Games-Howell or Tamhane methods could have been used. The results from 
> these tests contradict those from a Tukey HSD, revealing no 
> differences between the treatments, which is hardly surprising given 
> the small number of samples and some of the very large variances in 
> the data.
> Rather than dwell on MCPs being used in this inappropriate manner, I 
> would wish to point out that there are valid techniques which can be 
> used notwithstanding that the data fails the assumption of 
> homosecedasticity (which is frequently the case in real life). These 
> are not difficult to perform (although they do not appear in the 
> standard computer packages, e.g. SPSS or SAS or Minitab, and cannot be 
> found in the common 'user friendly' statistical texts), indeed they 
> can be computed using an Excel spreadsheet. I have applied one of 
> these, the approximate degrees of freedom (ADF) solution (analogous to 
> factorial ANOVA, and similar to Welch's correction for t-tests), to 
> the heteroscedastic Zooxanthellae catalase data. The conclusive 
> results are that there are no significant two or three way 
> interactions for this dataset. The technique is explained in Jaccard 
> (1998) for those of you who wish to read further.
> An excellent and simple text explaining the range of MCPs and their 
> use can be found in Chapter 4 of Kirk (1995). For those with more 
> interest in the statistical technicalities, and potential pitfalls of 
> MCPs the books of Hochberg & Tamhane (1987) and Hsu (1996) make 
> excellent reading.
> Factorial ANOVA and the tools for examining any interactions that are 
> revealed was a subject to which I could give little space in the 
> original Perspective. It is clear that further discussion of this 
> topic is still warranted.
> References:
> Hochberg Y, Tamhane AC (1987) Multiple comparisons procedures. John 
> Wiley, New York. pp 450.
> Hsu J (1996) Multiple comparisons: theory and methods. Chapman & Hall, 
> London. pp 277.
> Jaccard J (1998) Interaction effects in factorial analysis of 
> variance. Sage. Thousand Oaks. pp 103.
> Kirk RE (1995) Experimental design: procedures for the behavioural 
> sciences. Brooks/Cole, Pacific Grove. pp 921.
> Kromery JD, La Rocca MAA (1995) Power and Type I error rates of new 
> pairwise multiple comparison procedures under heterogeneous variances. 
> The Journal of Experimental Education 63: 343-362.
> Richard P Dunne

More information about the Coral-List mailing list