Duncan's new multiple range test
In statistics, Duncan's new multiple range test (MRT) is a multiple comparison procedure developed by David B. Duncan in 1955. Duncan's MRT belongs to the general class of multiple comparison procedures that use the studentized range statistic "q" to compare sets of means. David B. Duncan developed this test as a modification of the Student–Newman–Keuls method that would have greater power. Duncan's MRT is especially protective against false negative (Type II) error at the expense of having a greater risk of making false positive (Type I) errors. Duncan's  test is commonly used in agronomy and other agricultural research. The result of the test is a set of subsets of means, where in each subset means have been found not to be significantly different from one another. Definition. Assumptions: 1.A sample of observed means formula_1 , which have been drawn independently from n normal populations with "true" means, formula_2 respectively. 2.A common standard error formula_3. This standard error is unknown, but there is available the usual estimate formula_4 , which is independent of the observed means and is based on a number of degrees of freedom, denoted by formula_5 . (More precisely , formula_6, has the property that formula_7 is distributed as formula_8 with formula_9 degrees of freedom, independently of sample means). The exact definition of the test is: The difference between any two means in a set of n means is significant provided the range of each and every subset which contains the given means is significant according to an formula_10 level range test where formula_11 , formula_12 and formula_13 is the number of means in the subset concerned. Exception: The sole exception to this rule is that no difference between two means can be declared significant if the two means concerned are  both contained in a subset of the means which has a non-significant range. Procedure. The procedure consists of a series of pairwise comparisons between means. Each comparison is performed at a significance level formula_14 , defined by the number of means separating the two means compared (formula_15 for formula_16 separating means). The test are performed sequentially, where the result of a test determines which test is performed next. The tests are performed in the following order: the largest minus the smallest, the largest minus the second smallest, up to the largest minus the second largest; then the second largest minus  the smallest, the second largest minus the second smallest, and so on, finishing with the second smallest minus the smallest. With only one exception, given below, each difference is significant if it exceeds the corresponding shortest significant range; otherwise it is not significant. Where the shortest significant range is the significant studentized range, multiplied by the standard error. The shortest significant range will be designated as formula_17, where formula_13 is the number means in the subset. The sole exception to this rule is that no difference between two means can be declared significant if the two means concerned are both  contained in a subset of the means which has a non-significant range. An algorithm for performing the test is as follows: Critical values. Duncan's multiple range test makes use of the studentized range distribution in order to determine critical values for comparisons between means. Note that different comparisons between means may differ by their significance levels- since the significance level is subject to the size of the subset of means in question. Let us denote formula_27 as the formula_28 quantile of the studentized range distribution, with p observations, and formula_29 degrees of freedom for the second sample (see  studentized range for more information). Let us denote formula_30 as the standardized critical value, given by the rule: If p=2 formula_31 Else formula_32 The shortest critical range, (the actual critical value of the test) is computed as : formula_33. For formula_29->∞, a tabulation exists for an exact value of Q (see link). A word of caution is needed here: notations for Q and R are not the same throughout literature, where Q is sometimes denoted as the shortest significant interval, and R as the significant quantile for studentized range distribution (Duncan's 1955 paper uses both notations in different parts). Numeric  example. Let us look at the example of 5 treatment means: With a standard error of formula_35, and formula_36 (degrees of freedom for estimating the standard error). Using a known tabulation for Q, one reaches the values of formula_30: formula_38 formula_39 formula_40 formula_41 Now we may obtain the values of the shortest significant range, by the formula: formula_42 Reaching: formula_43 formula_44 formula_45 formula_46 Then, the observed differences between means are tested, beginning with the largest versus smallest, which would be compared with the least significant range formula_47 Next, the difference of the largest and the second smallest is computed and  compared with the least significant difference formula_45. If an observed difference is greater than the corresponding shortest significant range, then we conclude that the pair of means in question is significantly different. If an observed difference is smaller than the corresponding shortest significant range, all differences sharing the same upper mean are considered insignificant, in order to prevent contradictions (differences sharing the same upper mean are shorter by construction). For our case, the comparison will yield: formula_49 formula_50 formula_51 formula_52 formula_53 formula_54 formula_55 formula_56 formula_57 formula_58 We see that there are significant differences between all pairs of treatments except (T3,T2 ) and (T5,T1). A graph underlining those means that are not significantly different is shown below: T1 T5 T2 T3 T4 Protection and significance levels based on degrees of freedom. The new multiple range test proposed by Duncan makes use of special protection levels based upon degrees of freedom. Let formula_59 be the protection level for testing the significance of a difference between two means; that is, the probability that a significant difference between two means will not be found if the population means are equal. Duncan reasons that one has p-1 degrees of freedom for testing p ranked mean, and  hence one may conduct p-1 independent tests, each with protection level formula_59 . Hence, the joint protection level is: formula_61 where formula_62 that is, the probability that one finds no significant differences in making p-1 independent tests, each at protection level formula_59, is formula_64, under the hypothesis that all p population means are equal. In general: the difference between any two means in a set of n means is significant provided the range of each and every subset, which contains the given means, is significant according to an formula_15 –level range test, where p is the number of means in the  subset concerned. For formula_66 , the protection level can be tabulated for various value of r as follows: Note that although this procedure makes use of the Studentized range, his error rate is neither on an experiment-wise basis (as with Tukey's) nor on a per- comparisons basis. Duncan's multiple range test does not control the familywise error rate. See Criticism Section for further details. Duncan Bayesian multiple comparison procedure. Duncan (1965) also gave the first Bayesian multiple comparison procedure, for the pairwise comparisons among the means in a one-way layout. This multiple comparison procedure is different for  the one discussed above. Duncan's Bayesian MCP discusses the differences between ordered group means, where the statistics in question are pairwise comparison (no equivalent is defined for the property of a subset having 'significantly different' property). Duncan modeled the consequences of two or more means being equal using additive loss functions within and across the pairwise comparisons. If one assumes the same loss function across the pairwise comparisons, one needs to specify only one constant K, and this indicates the relative seriousness of type I to type II errors in each pairwise comparison. A study, which performed by Juliet  Popper Shaffer (1998), has shown that the method proposed by Duncan, modified to provide weak control of FWE and using an empirical estimate of the variance of the population means, has good properties both from the Bayesian point of view, as a minimum- risk method, and from the frequentist point of view, with good average power. In addition, results indicate considerable similarity in both risk and average power between Duncan's modified procedure and the Benjamini and Hochberg (1995) False discovery rate -controlling procedure, with the same weak familywise error control. Criticism. Duncan's test has been criticised as being  too liberal by many statisticians including Henry Scheffé, and John W. Tukey. Duncan argued that a more liberal procedure was appropriate because in real world practice the global null hypothesis H = "All means are equal" is often false and thus traditional statisticians overprotect a probably false null hypothesis against type I errors. According to Duncan, one should adjust the protection levels for different p-mean comparisons according to the problem discussed. The example discussed by Duncan in his 1955 paper is of a comparison of many means (i.e. 100),when one is interested only in two-mean and three-mean  comparisons, and general p-mean comparisons (deciding whether there is some difference between p-means) are of no special interest (if p is 15 or more for example). Duncan's multiple range test is very “liberal” in terms of Type I errors. The following example will illustrate why: Let us assume one is truly interested, as Duncan suggested, only with the correct ranking of subsets of size 4 or below. Let us also assume that one performs the simple pairwise comparison with a protection level formula_67. Given an overall set of 100 means, let us look at the null hypotheses  of the test: There are formula_68 null hypotheses for the correct ranking of each 2 means. The significance level of each hypothesis is formula_69 There are formula_70 null hypotheses for the correct ranking of each 3 means. The significance level of each hypothesis is formula_71 There are formula_72 null hypotheses for the correct ranking of each 4 means. The significance level of each hypothesis is formula_73 As we can see, the test has two main problems, regarding the type I errors: Therefore, it is advised not to use the procedure discussed. Duncan later developed the Duncan–Waller test which is  based on Bayesian principles. It uses the obtained value of F to estimate the prior probability of the null hypothesis being true. Different approaches to the problem. If one still wishes to address the problem of finding similar subsets of group means, other solutions are found in literature. Tukey's range test is commonly used to compare pairs of means, this procedure controls the familywise error rate in the strong sense. Another solution is to perform Student's t-test of all pairs of means, and then to use FDR Controlling procedure (to control the expected proportion of incorrectly rejected  null hypotheses). Other possible solutions, which do not include hypothesis testing, but result in a partition of subsets include Clustering & Hierarchical Clustering. These solutions differ from the approach presented in this method: