[Coral-List] Multiple Comparisons – Improved Excel Spreadsheet for MCPs, and the use of Paired Comparisons Information Criteria (PCIC)

[Richard Dunne]

Multiple Comparisons – Improved Excel Spreadsheet for MCPs, and the use 
of Paired Comparisons Information Criteria (PCIC)

On 6 June I offered Listers an Excel Spreadsheet for computing Multiple 
Comparisons (MCPs). I have since sent out over 60 copies on request.

I have made additions to the original package, and it now incorporates a 
total of 7 MCPs for homogenous variance datasets (Fisher LSD, 
Bonferroni, Scheffe, Tukey HSD, Tukey-Kramer, Ryan F/REGW F), the 
Games-Howell test for heterogeneous data, and the non parametric 
Kruskal-Wallis followed by Dunn’s MCP for non-normal data.

In particular, the addition of the REGW F test provides users with the 
most powerful of all the homogenous MCPs when there are 4 or more 
treatments/groups (more powerful than Tukey). Because it is 
computationally involved it cannot be done by hand and also only appears 
in the better (and more expensive!) commercial statistics packages (eg 
SAS, SPSS), although I am sure someone will tell me that it can be found 
somewhere in R (free but tough to master).

The spreadsheet computes all these MCPs simultaneously on up to 10 
treatments/ groups for a maximum of 199 data per treatment. Output for 
most MCPs is in the form of the difference between the means, test 
statistic, the critical value, 95% confidence intervals (which are also 
displayed graphically) and a statement of “significance” at the 0.05 
level. The use of 95% CI is increasingly recommended by scientific 
publications since it involves not only a test of significance but also 
an estimate of magnitude and direction of the difference between the 
means. The one drawback of the REGW F test is that it cannot generate 
CIs (a feature of the test not of my spreadsheet).

The only user input required is the data entry on a single sheet (type 
in or paste).

Also included are tests of the assumptions of homogeneity and normality 
using the Hartley's Fmax, and tests for skewness and kurtosis, together 
with full descriptive statistics and a box and whisker plot for the data.

It is ‘user friendly’ with detailed 'Instructions' and a comprehensive 
"Guide to the choice of MCP" and full non technical descriptions of each 
MCP together with references and further reading for those who wish. All 
the procedures have been cross validated with SPSS. It is an excellent 
and quick way for the researcher to conduct MCPs on data. It requires no 
specialist statistical knowledge nor does it require access to expensive 
commercial statistics packages (many of which do not contain all of 
these tests in any case). It is also an excellent teaching tool to help 
users understand this area of statistics which is poorly covered in many 
statistical teaching texts.

I am sending all original requesters the updated sheet (Version 5) which 
was written under Excel 2003 and I can also e-mail anyone else who would 
like a copy of the spreadsheet (5Mb size).

For those of you who would like to venture into new techniques, away 
from these MCPs which use classical statistical hypothesis testing, a 
method pioneered by Dayton in 1998 of ‘Paired Comparisons Information 
Criteria’ (PCIC) is available from his website at 
www.education.umd.edu/EDMS/Latent/. This uses a completely different 
approach whereby an information criterion statistic (normally Akaike’s 
Information Criterion - AIC) is used to select a configuration of 
population means that best fits the observed data. This technique avoids 
one of the problems of logic that frequently occurs with classical MCPs, 
that of ‘intransitive decisions’, whereby homogenous subsets of means 
overlap. For example, the MCP finds that means 1,2,3,4  are not 
significantly different from each other, likewise for means 4,5. Thus 
mean 1 is significantly different from mean 5 but not from mean 4 which 
itself is not different from mean 5. Interpreting results such as these 
can be ambiguous, particularly for mean 4.

The PCIC technique looks at all possible models of these means which 
would be {1234}, {1,234}, {12,34}, {123,4}, {1,2,34}, {{12,3,4}, 
{1,23,4}, {1,2,3,4} where means not separated from each other by a comma 
are similar. The model with the smallest AIC is considered to be the 
most probable population model. The potential for overlapping subsets is 
thus avoided. The technique has been shown to handle both homogenous and 
heterogeneous variances and is not constrained by the distribution 
shape, i.e. does not require normality.

On his website Dayton presents an Excel spreadsheet which will compute 
the PCIC for groups of means of 4 or 5. For larger numbers he also 
presents an SAS program file. I am not aware if anyone has written a R 
program file yet. Cribbie & Keselman (2003) and Cribbie (2003) compared 
the two techniques of classical MCPs and PCIC and have shown that the 
latter was better able to predict the true relationships between the means.

For those who wish to read further I have listed a number of 
‘digestible’ references below:
Dayton CM (1998) "Information Criteria for the Paired-Comparisons 
Problem." American Statistician 52:144-151
Dayton CM (2001) SUBSET: Best subsets using information criteria. 
Journal of Statistical Software 6, April
Dayton, CM (2003) Information criteria for pairwise comparisons. 
Psychological Methods 8:61-71
Dayton CM (2003) Model comparisons using information measures. J Modern 
Applied Statistical Methods 2:281-292
Cribbie RA, Keselman HJ (2003) Pairwise multiple comparisons: a model 
comparison approach versus stepwise procedures. Brit J Mathematical 
Statistical Psychology 56:167-182
Cribbie (2003) Pairwise multiple comparisons: new yardsticks, new 
results. J Experimental Education 71:251-265

Richard P Dunne