Reliability in Psychiatric Diagnosis with the DSM: Old Wine in New Barrels
Authors: Stijn Vanheule, Mattias Desmet, Reitske Meganck, Ruth Inslegers, Jochem Willemsen, Maarten De Schryver, Ignaas Devisch
Journal: Psychotherapy and Psychosomatics (2014)

Publications on the Diagnostic and Statistical Manual of Mental Disorders (DSM) often remark that with the institution of DSM-III, reliability in psychiatric decision-making strongly improved [1]. This belief is based on the fact that DSM-III implemented explicit diagnostic criteria focusing on descriptive clinical features for each disorder and formulated thresholds for the minimal number of symptoms that should be present for diagnosing a condition. The same approach can be found in the DSM-5 [[1], p. 5], which states that the 'DSM has been the cornerstone of substantial progress in reliability'. We argue that this claim is unjustified. The proof can be found in the DSM-5 field trials.In the DSM-5 field trials [2,3], the interrater reliability of 27 of the 347 DSM-5 diagnostic categories was tested. On a pairwise basis, 286 trained clinicians evaluated 1,466 adult and 616 pediatric patients. This is the largest study that has yet been done on psychiatric interrater reliability. Reliability was evaluated by means of intraclass κ-coefficients, which reflect 'the difference between the probabilities of getting a second positive diagnosis between those with a first positive and those with a first negative diagnosis' [[2], p. 47]. Accurate κ-coefficients could be calculated for 15 disorders in adults and for 8 disorders in children. In order to evaluate the κ-statistic, the DSM-5 field trials used norms formulated by Clarke et al. [2]: values ≥0.80 are excellent; values between 0.60 and 0.79 are very good; values between 0.40 and 0.59 are good; values between 0.20 and 0.39 are questionable, and values ≤0.20 are unacceptable. However, the standards for evaluating κ-statistics have relaxed substantially over time. In the early days of systematic reliability research, Spitzer and Fleiss [4] suggested that in psychiatric research κ-values ≥0.90 are excellent; values between 0.70 and 0.90 are good, while values ≤0.70 are unacceptable. In 1977, Landis and Koch [5] proposed the frequently used thresholds: values ≥0.75 are excellent; values between 0.40 and 0.75 indicate fair to good reliability, and values ≤0.40 indicate poor reliability. More recently, Baer and Blais [6] suggested that κ-values >0.70 are excellent; values between 0.60 and 0.70 are good; values between 0.41 and 0.59 are questionable, and values ≤0.40 are poor. Considering these standards, the norms used in the DSM-5 field trial are unacceptably generous.The norms proposed by Spitzer and Fleiss [4] were formulated in a major review of 6 major pre-DSM-III reliability studies published between the late 1950s and early 1970s. The accumulated data included 1,726 patients (all diagnosed by 2 psychiatrists) and evaluated 18 diagnostic categories. Starting from the 2013 κ-thresholds [2], the DSM-5 field trial in adults observed: very good reliability for 3 disorders (posttraumatic stress disorder: κ = 0.67, complex somatic disorder: κ = 0.61, major neurocognitive disorder: κ = 0.78); good reliability for 7 disorders (including schizophrenia: κ = 0.46, alcohol use disorder: κ = 0.40, borderline personality disorder: κ = 0.54); questionable reliability for 4 disorders (including major depressive disorder: κ = 0.28, generalized anxiety disorder: κ = 0.20, antisocial personality disorder: κ = 0.21), and unacceptable reliability for 1 disorder (mixed anxiety-depressive disorder: κ = 0.00), which was eventually excluded from the manual [3]. In the pediatric sample, very good reliability was observed for 2 diagnostic categories (autism spectrum disorders: κ = 0.69, attention deficit hyperactivity disorder: κ = 0.61); good reliability for 2 conditions (avoidant/restrictive food intake: κ = 0.48, oppositional defiant disorder: κ = 0.40); questionable reliability for 2 categories (disruptive mood dysregulation: κ = 0.25, major depressive disorder: κ = 0.28), and unacceptable reliability for 1 disorder (mixed anxiety-depressive disorder: κ = 0.05) [3].Starting from the κ-evaluation norms they formulated, Spitzer and Fleiss [4] observed that of the 18 conditions reviewed in 1974 only 3 had good reliability (mental deficiency: κ = 0.72, organic brain syndrome: κ = 0.77, alcoholism: κ = 0.71). The other categories did not stand to the test (including schizophrenia: κ = 0.57, sociopathic personality disorder: κ = 0.53, psychophysiological reaction: κ = 0.38). Remarkably, while the κ-value for affective disorder was 0.41, subtypes scored lower (neurotic depression: κ = 0.26, psychotic depression: κ = 0.24, manic-depressive: κ = 0.33). Overall, the authors concluded that the reliability of psychiatric diagnosis was not good, which led them to promote criteria-based diagnosis.A comparison of the conclusions drawn from the DSM-5 field trials [3] versus the pre-DSM-III studies [4] might indicate that the reliability of psychiatric diagnosis increased substantially. However, given the fact that throughout time the norms for evaluating the intraclass κ-coefficient have relaxed considerably, and that in the DSM-5 field trial unacceptably generous norms have been used, such a conclusion is unjustified. As table 1 indicates, reliabilities in the year 2013 are not better than those observed in 1974, and actually remain in the same range. While some disorders are now diagnosed more reliably (e.g. psychophysiological reaction/complex somatic disorder), the reverse is true of other conditions (e.g. alcoholism/alcohol use disorder). Moreover, the diagnosis of mood/affective disorders remains a big concern.Against our point it could be argued that since the DSM-III test-retest reliability studies with somewhat better results have been published. However, these studies calculated the κ-statistic on a limited number of patients, which undermines the accuracy of their results. Exceptions to this are two interrater reliability studies, one using SCID-I [7] and one using SCID-II [8]. In the first study, 25 trained judges evaluated 18 diagnostic categories in 390 patients [7]. As table 1 indicates, the results were only slightly better than in the DSM-5 field trials or in the review study of Spitzer and Fleiss [4], which is surprising given the involvement of highly trained judges and the use of a structured interview [9]. In the second study, 25 trained judges evaluated 10 disorders in 103 psychiatric patients [8]. The results (table 1) were no better than in the DSM-5 field trials or in the review study by Spitzer and Fleiss [4].We conclude that the reliability of the DSM is frequently overestimated. This should be a big concern in contemporary clinical research. We propose that (a) individual symptoms and clusters of symptoms should be defined more exhaustively, they should be documented with clinical examples and counterexamples, and their predictive value concerning outcome and treatment response should be studied more explicitly; (b) person-related contextual factors should be described more explicitly in diagnosing mental symptoms, and (c) clinimetric principles should be integrated more explicitly in diagnostic practice [10].There are no conflicts of interest for any of the authors.