[Coral-List] Multiple Comparisons – an illustration of the use of Paired Comparisons Information Criteria (PCIC)

[Richard Dunne]

PAIRED COMPARISONS INFORMATION CRITERIA (PCIC) and CLASSICAL MCPs

In my last post I directed readers' attention to the Dayton PCIC 
technique. Here is an interesting illustration of the use of the 
technique in conjunction with classical MCPs.

Jason Hsu (1996 Multiple comparisons: theory and methods. Chapman & 
Hall) draws our attention to the situation where computer statistical 
programs can produce the wrong result and this also provides a useful 
illustration of MCPs and their drawbacks.

Suppose we have a data set consisting of 4 treatments/groups, each with 
2 values. The means of Groups 1 & 2 are both = -1.575 and the means of 
Groups 3 & 4 are both = 1.575. Variances for each group are = 1.0.

A one way ANOVA for this data will produce the result F (3,4) = 6.615, p 
= 0.04971. In other words there is a significant difference between at 
least two of the Group means.

The Fisher LSD identifies the following significant differences: Group 1 
vs  3, Group 1 vs 4, Group 2 vs 3, Group 2 vs 4. This is a logical 
result given that the first two Groups (1&2) have identical means, and 
Groups 3&4 have identical means. However we would normally conclude that 
we could not rely on this result because we know that the Fisher LSD 
does not control the Type I error rate.

Other MCPs which control the Type I error (Tukey, Dunn-Sidak, 
Bonferroni, Scheffe, REGW Q) all fail to find any significant differences.

The REGW F test picks out the initial difference (because the first test 
is an ANOVA F test of all the means) but then fails to detect where the 
differences lie. In 1996 Hsu showed that SAS Version 6.09 produced 
illogical results and as a result the REGW F option was removed and no 
longer appears in the SAS package. In SPSS 17.0, REGW F is still 
available and produces an absurd result for this data, namely that 
Groups 1 & 4 are different but not any others.

If we use Dayton's PCIC what can it tell us about the Groups?  This is 
the output:

Pattern    SS(m)    LN(L)    AIC    Rank
1234     23.845    -28.424    60.848    8
1,234    17.230    -21.809    49.618    6
12,34    4.000      -8.579      23.158    1      Min(AIC)
123,4    17.230    -21.809    49.618    6
1,2,34    4.000     -8.579      25.158    2
1,23,4    13.923   -18.501    45.003    5
12,3,4    4.000     -8.579      25.158    2
1,2,3,4   4.000    -8.579       27.158    4

As can be seen, it tells us that the best model for our data (the result 
with the minimum AIC) is {12,34}, in other words exactly what a logical 
inspection of the data suggests. This is the same result that was 
produced by the Fisher LSD.

Granted, this may be a highly contrived data set which produces an 
unusual  outcome but it is an interesting illustration of some of the 
potential problems in data analysis. It also challenges our reliance on 
the results of computer programs. If you had this data set how would you 
go about presenting your results in a journal manuscript?  The ANOVA has 
rejected the null hypothesis that the means are the same. So where is 
the difference? The only MCP to detect any differences in the means is 
Fisher LSD, but rely on it and you would be criticised by reviewers and 
editors. Use the results from the SPSS REGW F and your conclusions would 
defy all logic. Perhaps the answer is to use some combination of the 
significant ANOVA, the PCIC, and then report that REGW F (or Tukey) lack 
sufficient power to detect the differences which however may be inferred 
from the Fisher LSD.


Richard Dunne

On 01/07/2010 09:10, Richard Dunne wrote:
> Multiple Comparisons – Improved Excel Spreadsheet for MCPs, and the use
> of Paired Comparisons Information Criteria (PCIC)
>
> On 6 June I offered Listers an Excel Spreadsheet for computing Multiple
> Comparisons (MCPs). I have since sent out over 60 copies on request.
>
> I have made additions to the original package, and it now incorporates a
> total of 7 MCPs for homogenous variance datasets (Fisher LSD,
> Bonferroni, Scheffe, Tukey HSD, Tukey-Kramer, Ryan F/REGW F), the
> Games-Howell test for heterogeneous data, and the non parametric
> Kruskal-Wallis followed by Dunn’s MCP for non-normal data.
>
> In particular, the addition of the REGW F test provides users with the
> most powerful of all the homogenous MCPs when there are 4 or more
> treatments/groups (more powerful than Tukey). Because it is
> computationally involved it cannot be done by hand and also only appears
> in the better (and more expensive!) commercial statistics packages (eg
> SAS, SPSS), although I am sure someone will tell me that it can be found
> somewhere in R (free but tough to master).
>
> The spreadsheet computes all these MCPs simultaneously on up to 10
> treatments/ groups for a maximum of 199 data per treatment. Output for
> most MCPs is in the form of the difference between the means, test
> statistic, the critical value, 95% confidence intervals (which are also
> displayed graphically) and a statement of “significance” at the 0.05
> level. The use of 95% CI is increasingly recommended by scientific
> publications since it involves not only a test of significance but also
> an estimate of magnitude and direction of the difference between the
> means. The one drawback of the REGW F test is that it cannot generate
> CIs (a feature of the test not of my spreadsheet).
>
> The only user input required is the data entry on a single sheet (type
> in or paste).
>
> Also included are tests of the assumptions of homogeneity and normality
> using the Hartley's Fmax, and tests for skewness and kurtosis, together
> with full descriptive statistics and a box and whisker plot for the data.
>
> It is ‘user friendly’ with detailed 'Instructions' and a comprehensive
> "Guide to the choice of MCP" and full non technical descriptions of each
> MCP together with references and further reading for those who wish. All
> the procedures have been cross validated with SPSS. It is an excellent
> and quick way for the researcher to conduct MCPs on data. It requires no
> specialist statistical knowledge nor does it require access to expensive
> commercial statistics packages (many of which do not contain all of
> these tests in any case). It is also an excellent teaching tool to help
> users understand this area of statistics which is poorly covered in many
> statistical teaching texts.
>
> I am sending all original requesters the updated sheet (Version 5) which
> was written under Excel 2003 and I can also e-mail anyone else who would
> like a copy of the spreadsheet (5Mb size).
>
> For those of you who would like to venture into new techniques, away
> from these MCPs which use classical statistical hypothesis testing, a
> method pioneered by Dayton in 1998 of ‘Paired Comparisons Information
> Criteria’ (PCIC) is available from his website at
> www.education.umd.edu/EDMS/Latent/. This uses a completely different
> approach whereby an information criterion statistic (normally Akaike’s
> Information Criterion - AIC) is used to select a configuration of
> population means that best fits the observed data. This technique avoids
> one of the problems of logic that frequently occurs with classical MCPs,
> that of ‘intransitive decisions’, whereby homogenous subsets of means
> overlap. For example, the MCP finds that means 1,2,3,4  are not
> significantly different from each other, likewise for means 4,5. Thus
> mean 1 is significantly different from mean 5 but not from mean 4 which
> itself is not different from mean 5. Interpreting results such as these
> can be ambiguous, particularly for mean 4.
>
> The PCIC technique looks at all possible models of these means which
> would be {1234}, {1,234}, {12,34}, {123,4}, {1,2,34}, {{12,3,4},
> {1,23,4}, {1,2,3,4} where means not separated from each other by a comma
> are similar. The model with the smallest AIC is considered to be the
> most probable population model. The potential for overlapping subsets is
> thus avoided. The technique has been shown to handle both homogenous and
> heterogeneous variances and is not constrained by the distribution
> shape, i.e. does not require normality.
>
> On his website Dayton presents an Excel spreadsheet which will compute
> the PCIC for groups of means of 4 or 5. For larger numbers he also
> presents an SAS program file. I am not aware if anyone has written a R
> program file yet. Cribbie&  Keselman (2003) and Cribbie (2003) compared
> the two techniques of classical MCPs and PCIC and have shown that the
> latter was better able to predict the true relationships between the means.
>
> For those who wish to read further I have listed a number of
> ‘digestible’ references below:
> Dayton CM (1998) "Information Criteria for the Paired-Comparisons
> Problem." American Statistician 52:144-151
> Dayton CM (2001) SUBSET: Best subsets using information criteria.
> Journal of Statistical Software 6, April
> Dayton, CM (2003) Information criteria for pairwise comparisons.
> Psychological Methods 8:61-71
> Dayton CM (2003) Model comparisons using information measures. J Modern
> Applied Statistical Methods 2:281-292
> Cribbie RA, Keselman HJ (2003) Pairwise multiple comparisons: a model
> comparison approach versus stepwise procedures. Brit J Mathematical
> Statistical Psychology 56:167-182
> Cribbie (2003) Pairwise multiple comparisons: new yardsticks, new
> results. J Experimental Education 71:251-265
>
> Richard P Dunne
>
> _______________________________________________
> Coral-List mailing list
> Coral-List at coral.aoml.noaa.gov
> http://coral.aoml.noaa.gov/mailman/listinfo/coral-list
>
>