[Coral-List] Interactions between Stressors on Coral Reefs - factorial ANOVA or Multiple Comparison Procedures
RichardPDunne at aol.com
Fri May 28 04:56:50 EDT 2010
In December last year I drew the attention of readers of Coral List to a
Perspective that I wrote for the journal 'Coral Reefs'.
Dunne (2010) SYNERGY OR ANTAGONISM - INTERACTIONS BETWEEN STRESSORS ON
Coral Reefs DOI: 10.1007/s00338-009-0569-6
In that paper I examined earlier publications in the coral reef
literature that had used factorial ANOVA and illustrated some of the
errors that had been made in interpreting interaction effects. One of
the authors has defended his work in the face of my criticism. His
Comment appears online at:
Lesser (2010) INTERACTIONS BETWEEN STRESSORS ON CORAL REEFS: ANALYTICAL
APPROACHES, RE-ANALYSIS OF OLD DATA, AND DIFFERENT CONCLUSIONS - Comment
on Dunne (2010)
Coral Reefs DOI: 10.1007/s00338-010-0625-2
The author sent me a copy during the peer review process and I provided
him with my detailed observations. As a consequence he has included my
name in his Acknowledgements. I would like to be clear to readers of the
Comment, however, that I do not agree with most of the substantive
points that appear and that there are still significant errors. Coral
Reefs does not wish to continue further discussion, which is regrettable
for there are important issues here. For those of you who are
interested, I outline some of these below.
The author makes the perfectly valid observation that it is possible to
use certain multiple comparison procedures (MCP) on their own, without
prior protection from ANOVA to examine datasets and that they can be a
useful tool in their own right, sometimes with more power to detect
differences than ANOVA. I fully agree with this observation which has
been discussed in statistical publications over the last 20 years but
rarely appears in the biological literature. However, he then goes on to
propose a methodology to examine interactions, illustrating the
technique by using a Tukey HSD test on some of his datasets from 1990
whereby he concludes that interactions are present. It is with this
methodology and conclusion that I strongly disagree. A careful reading
of the Comment will reveal that the results of certain of the MCP
analyses do not support the logic of the interpretations, notably for
the MAA and Zooxanthellae numbers data, where Tukey HSD finds homogenous
overlapping groups. Furthermore, a MCP conducted in the manner in which
it has been used, across all treatment groups, will not reveal
interactive effects. This perhaps explains why other more appropriate
tools, including factorial ANOVA, exist for this purpose.
In addition, there is a serious factual error in the Comment which
invalidates the analysis of the Zooxanthellae catalase data. It concerns
the statement that: "Lesser et al. (1990) conducted a balanced,
multi-factorial design experiment, with a non-significant Fmax test of
homogeneity of variances on the zoanthid Palythoa caribaeorum,....". In
1993 I first discussed this with the author and pointed out that some of
the data from the 1990 experiment were not homoscedastic. In fact the
majority of the datasets (8 of the 13) in that paper fail the Fmax test,
and other tests such as Levene and Bartlett. Five of the datasets are
extremely heteroscedastic, with Fmax (the ratio of the smallest to the
largest variance) being in excess of 100. This includes the Zooxanthella
catalase data, where it is 169. I again drew attention to this specific
issue in my observations, but this point seems to have been missed.
Why is this heterogeneity important? It is because the accuracy of most
statistical tools is based upon underlying assumptions about the data
being analysed. In the case of the Tukey HSD one of these assumptions is
that the data are homoscedastic. In this case where there is severe
heteroscedasticity, some of the multiple comparisons will have inflated
alpha. In other words, some of the comparisons which indicate
significant differences at the P=0.05 will be incorrect. This was
clearly demonstrated by Kromery and La Rocca (1995) in a study which
examined several MCPs and their performance when data were
heteroscedastic. For the Tukey HSD they found that for 5 treatment
groups and a Fmax of 13, the familywise error rate was inflated to
0.098. In the case of the Zooxanthella catalase data with Fmax = 169 and
8 treatment groups the errors are going to be considerably greater. The
words of a leading statistician in the field of multiple comparisons are
particularly apt "Editors of many scientific journals have not .......
realize[d] that the multiple comparison analyses performed by some
authors may have error rates well beyond any acceptable level." Hsu
(1996). Regrettably that situation seems to have occurred once again.
It is of course possible to avoid these problems by employing other
suitable MCPs which do not require homoscedasticity. For example, the
Games-Howell or Tamhane methods could have been used. The results from
these tests contradict those from a Tukey HSD, revealing no differences
between the treatments, which is hardly surprising given the small
number of samples and some of the very large variances in the data.
Rather than dwell on MCPs being used in this inappropriate manner, I
would wish to point out that there are valid techniques which can be
used notwithstanding that the data fails the assumption of
homosecedasticity (which is frequently the case in real life). These are
not difficult to perform (although they do not appear in the standard
computer packages, e.g. SPSS or SAS or Minitab, and cannot be found in
the common 'user friendly' statistical texts), indeed they can be
computed using an Excel spreadsheet. I have applied one of these, the
approximate degrees of freedom (ADF) solution (analogous to factorial
ANOVA, and similar to Welch's correction for t-tests), to the
heteroscedastic Zooxanthellae catalase data. The conclusive results are
that there are no significant two or three way interactions for this
dataset. The technique is explained in Jaccard (1998) for those of you
who wish to read further.
An excellent and simple text explaining the range of MCPs and their use
can be found in Chapter 4 of Kirk (1995). For those with more interest
in the statistical technicalities, and potential pitfalls of MCPs the
books of Hochberg & Tamhane (1987) and Hsu (1996) make excellent reading.
Factorial ANOVA and the tools for examining any interactions that are
revealed was a subject to which I could give little space in the
original Perspective. It is clear that further discussion of this topic
is still warranted.
Hochberg Y, Tamhane AC (1987) Multiple comparisons procedures. John
Wiley, New York. pp 450.
Hsu J (1996) Multiple comparisons: theory and methods. Chapman & Hall,
London. pp 277.
Jaccard J (1998) Interaction effects in factorial analysis of variance.
Sage. Thousand Oaks. pp 103.
Kirk RE (1995) Experimental design: procedures for the behavioural
sciences. Brooks/Cole, Pacific Grove. pp 921.
Kromery JD, La Rocca MAA (1995) Power and Type I error rates of new
pairwise multiple comparison procedures under heterogeneous variances.
The Journal of Experimental Education 63: 343-362.
Richard P Dunne
More information about the Coral-List