[Coral-List] Interactions between Stressors on Coral Reefs - factorial ANOVA or Multiple Comparison Procedures

[Richard Dunne]

In December last year I drew the attention of readers of Coral List to a 
Perspective that I wrote for the journal 'Coral Reefs'.

Dunne (2010) SYNERGY OR ANTAGONISM - INTERACTIONS BETWEEN STRESSORS ON 
CORAL REEFS.
Coral Reefs DOI: 10.1007/s00338-009-0569-6

In that paper I examined earlier publications in the coral reef 
literature that had used factorial ANOVA and illustrated some of the 
errors that had been made in interpreting interaction effects. One of 
the authors has defended his work in the face of my criticism. His 
Comment appears online at:

Lesser (2010) INTERACTIONS BETWEEN STRESSORS ON CORAL REEFS: ANALYTICAL 
APPROACHES, RE-ANALYSIS OF OLD DATA, AND DIFFERENT CONCLUSIONS - Comment 
on Dunne (2010)
Coral Reefs DOI: 10.1007/s00338-010-0625-2

The author sent me a copy during the peer review process and I provided 
him with my detailed observations. As a consequence he has included my 
name in his Acknowledgements. I would like to be clear to readers of the 
Comment, however, that I do not agree with most of the substantive 
points that appear and that there are still significant errors. Coral 
Reefs does not wish to continue further discussion, which is regrettable 
for there are important issues here. For those of you who are 
interested, I outline some of these below.

The author makes the perfectly valid observation that it is possible to 
use certain multiple comparison procedures (MCP) on their own, without 
prior protection from ANOVA to examine datasets and that they can be a 
useful tool in their own right, sometimes with more power to detect 
differences than ANOVA. I fully agree with this observation which has 
been discussed in statistical publications over the last 20 years but 
rarely appears in the biological literature. However, he then goes on to 
propose a methodology to examine interactions, illustrating the 
technique by using a Tukey HSD test on some of his datasets from 1990 
whereby he concludes that interactions are present. It is with this 
methodology and conclusion that I strongly disagree. A careful reading 
of the Comment will reveal that the results of certain of the MCP 
analyses do not support the logic of the interpretations, notably for 
the MAA and Zooxanthellae numbers data, where Tukey HSD finds homogenous 
overlapping groups. Furthermore, a MCP conducted in the manner in which 
it has been used, across all treatment groups, will not reveal 
interactive effects. This perhaps explains why other more appropriate 
tools, including factorial ANOVA, exist for this purpose.

In addition, there is a serious factual error in the Comment which 
invalidates the analysis of the Zooxanthellae catalase data. It concerns 
the statement that: "Lesser et al. (1990) conducted a balanced, 
multi-factorial design experiment, with a non-significant Fmax test of 
homogeneity of variances on the zoanthid  Palythoa caribaeorum,....". In 
1993 I first discussed this with the author and pointed out that some of 
the data from the 1990 experiment were not homoscedastic. In fact the 
majority of the datasets (8 of the 13) in that paper fail the Fmax test, 
and other tests such as Levene and Bartlett. Five of the datasets are 
extremely heteroscedastic, with Fmax (the ratio of the smallest to the 
largest variance) being in excess of 100. This includes the Zooxanthella 
catalase data, where it is 169. I again drew attention to this specific 
issue in my observations, but this point seems to have been missed.

Why is this heterogeneity important? It is because the accuracy of most 
statistical tools is based upon underlying assumptions about the data 
being analysed. In the case of the Tukey HSD one of these assumptions is 
that the data are homoscedastic. In this case where there is severe 
heteroscedasticity, some of the multiple comparisons will have inflated 
alpha. In other words, some of the comparisons which indicate 
significant differences at the P=0.05 will be incorrect. This was 
clearly demonstrated by Kromery and La Rocca (1995) in a study which 
examined several MCPs and their performance when data were 
heteroscedastic. For the Tukey HSD they found that for 5 treatment 
groups and a Fmax of 13, the familywise error rate was inflated to 
0.098. In the case of the Zooxanthella catalase data with Fmax = 169 and 
8 treatment groups the errors are going to be considerably greater. The 
words of a leading statistician in the field of multiple comparisons are 
particularly apt "Editors of many scientific journals have not ....... 
realize[d] that the multiple comparison analyses performed by some 
authors may have error rates well beyond any acceptable level." Hsu 
(1996). Regrettably that situation seems to have occurred once again.

It is of course possible to avoid these problems by employing other 
suitable MCPs which do not require homoscedasticity. For example, the 
Games-Howell or Tamhane methods could have been used. The results from 
these tests contradict those from a Tukey HSD, revealing no differences 
between the treatments, which is hardly surprising given the small 
number of samples and some of the very large variances in the data.

Rather than dwell on MCPs being used in this inappropriate manner, I 
would wish to point out that there are valid techniques which can be 
used notwithstanding that the data fails the assumption of 
homosecedasticity (which is frequently the case in real life). These are 
not difficult to perform (although they do not appear in the standard 
computer packages, e.g. SPSS or SAS or Minitab, and cannot be found in 
the common 'user friendly' statistical texts), indeed they can be 
computed using an Excel spreadsheet. I have applied one of these, the 
approximate degrees of freedom (ADF) solution (analogous to factorial 
ANOVA, and similar to Welch's correction for t-tests), to the 
heteroscedastic Zooxanthellae catalase data. The conclusive results are 
that there are no significant two or three way interactions for this 
dataset. The technique is explained in Jaccard (1998) for those of you 
who wish to read further.

An excellent and simple text explaining the range of MCPs and their use 
can be found in Chapter 4 of Kirk (1995). For those with more interest 
in the statistical technicalities, and potential pitfalls of MCPs the 
books of Hochberg & Tamhane (1987) and Hsu (1996) make excellent reading.

Factorial ANOVA and the tools for examining any interactions that are 
revealed was a subject to which I could give little space in the 
original Perspective. It is clear that further discussion of this topic 
is still warranted.

References:
Hochberg Y, Tamhane AC (1987) Multiple comparisons procedures. John 
Wiley, New York. pp 450.
Hsu J (1996) Multiple comparisons: theory and methods. Chapman & Hall, 
London. pp 277.
Jaccard J (1998) Interaction effects in factorial analysis of variance. 
Sage. Thousand Oaks. pp 103.
Kirk RE (1995) Experimental design: procedures for the behavioural 
sciences. Brooks/Cole, Pacific Grove. pp 921.
Kromery JD, La Rocca MAA (1995) Power and Type I error rates of new 
pairwise multiple comparison procedures under heterogeneous variances. 
The Journal of Experimental Education 63: 343-362.


Richard P Dunne