[Coral-List] Multiple Comparisons – Improved Excel Spreadsheet for MCPs, and the use of Paired Comparisons Information Criteria (PCIC)
Richard Dunne
RichardPDunne at aol.com
Thu Jul 1 04:10:29 EDT 2010
Multiple Comparisons – Improved Excel Spreadsheet for MCPs, and the use
of Paired Comparisons Information Criteria (PCIC)
On 6 June I offered Listers an Excel Spreadsheet for computing Multiple
Comparisons (MCPs). I have since sent out over 60 copies on request.
I have made additions to the original package, and it now incorporates a
total of 7 MCPs for homogenous variance datasets (Fisher LSD,
Bonferroni, Scheffe, Tukey HSD, Tukey-Kramer, Ryan F/REGW F), the
Games-Howell test for heterogeneous data, and the non parametric
Kruskal-Wallis followed by Dunn’s MCP for non-normal data.
In particular, the addition of the REGW F test provides users with the
most powerful of all the homogenous MCPs when there are 4 or more
treatments/groups (more powerful than Tukey). Because it is
computationally involved it cannot be done by hand and also only appears
in the better (and more expensive!) commercial statistics packages (eg
SAS, SPSS), although I am sure someone will tell me that it can be found
somewhere in R (free but tough to master).
The spreadsheet computes all these MCPs simultaneously on up to 10
treatments/ groups for a maximum of 199 data per treatment. Output for
most MCPs is in the form of the difference between the means, test
statistic, the critical value, 95% confidence intervals (which are also
displayed graphically) and a statement of “significance” at the 0.05
level. The use of 95% CI is increasingly recommended by scientific
publications since it involves not only a test of significance but also
an estimate of magnitude and direction of the difference between the
means. The one drawback of the REGW F test is that it cannot generate
CIs (a feature of the test not of my spreadsheet).
The only user input required is the data entry on a single sheet (type
in or paste).
Also included are tests of the assumptions of homogeneity and normality
using the Hartley's Fmax, and tests for skewness and kurtosis, together
with full descriptive statistics and a box and whisker plot for the data.
It is ‘user friendly’ with detailed 'Instructions' and a comprehensive
"Guide to the choice of MCP" and full non technical descriptions of each
MCP together with references and further reading for those who wish. All
the procedures have been cross validated with SPSS. It is an excellent
and quick way for the researcher to conduct MCPs on data. It requires no
specialist statistical knowledge nor does it require access to expensive
commercial statistics packages (many of which do not contain all of
these tests in any case). It is also an excellent teaching tool to help
users understand this area of statistics which is poorly covered in many
statistical teaching texts.
I am sending all original requesters the updated sheet (Version 5) which
was written under Excel 2003 and I can also e-mail anyone else who would
like a copy of the spreadsheet (5Mb size).
For those of you who would like to venture into new techniques, away
from these MCPs which use classical statistical hypothesis testing, a
method pioneered by Dayton in 1998 of ‘Paired Comparisons Information
Criteria’ (PCIC) is available from his website at
www.education.umd.edu/EDMS/Latent/. This uses a completely different
approach whereby an information criterion statistic (normally Akaike’s
Information Criterion - AIC) is used to select a configuration of
population means that best fits the observed data. This technique avoids
one of the problems of logic that frequently occurs with classical MCPs,
that of ‘intransitive decisions’, whereby homogenous subsets of means
overlap. For example, the MCP finds that means 1,2,3,4 are not
significantly different from each other, likewise for means 4,5. Thus
mean 1 is significantly different from mean 5 but not from mean 4 which
itself is not different from mean 5. Interpreting results such as these
can be ambiguous, particularly for mean 4.
The PCIC technique looks at all possible models of these means which
would be {1234}, {1,234}, {12,34}, {123,4}, {1,2,34}, {{12,3,4},
{1,23,4}, {1,2,3,4} where means not separated from each other by a comma
are similar. The model with the smallest AIC is considered to be the
most probable population model. The potential for overlapping subsets is
thus avoided. The technique has been shown to handle both homogenous and
heterogeneous variances and is not constrained by the distribution
shape, i.e. does not require normality.
On his website Dayton presents an Excel spreadsheet which will compute
the PCIC for groups of means of 4 or 5. For larger numbers he also
presents an SAS program file. I am not aware if anyone has written a R
program file yet. Cribbie & Keselman (2003) and Cribbie (2003) compared
the two techniques of classical MCPs and PCIC and have shown that the
latter was better able to predict the true relationships between the means.
For those who wish to read further I have listed a number of
‘digestible’ references below:
Dayton CM (1998) "Information Criteria for the Paired-Comparisons
Problem." American Statistician 52:144-151
Dayton CM (2001) SUBSET: Best subsets using information criteria.
Journal of Statistical Software 6, April
Dayton, CM (2003) Information criteria for pairwise comparisons.
Psychological Methods 8:61-71
Dayton CM (2003) Model comparisons using information measures. J Modern
Applied Statistical Methods 2:281-292
Cribbie RA, Keselman HJ (2003) Pairwise multiple comparisons: a model
comparison approach versus stepwise procedures. Brit J Mathematical
Statistical Psychology 56:167-182
Cribbie (2003) Pairwise multiple comparisons: new yardsticks, new
results. J Experimental Education 71:251-265
Richard P Dunne
More information about the Coral-List
mailing list