Opinion ID: 2745691
Heading Depth: 2
Heading Rank: 1

Heading: District Court’s Validity Findings

Text: Here, in deeming the 2002 process’s testing methods valid, the district court detailed Dr. Jeanneret’s “comprehensive job analysis,” on behalf of the City, to identify the most important knowledge, skills, abilities, and personal characteristics (KSAPs) for the sergeant position. Jeanneret & Associates sought to assess all 44 of the important KSAPs identified in the job analysis and designed the test questions to meet the content validity requirements for the assessment. The investigative forms and other materials used in the investigative logic test and oral component were very similar to the actual materials used on the job and clearly simulated critical job duties. Additionally, all of the items on the job knowledge test were developed using the same reference materials used by MPD sergeants on the job. The investigative logic test involved realistic scenarios that were designed to simulate situations encountered and investigative activities performed by sergeants on the job. Likewise, the application of knowledge test was designed to evaluate how a candidate would respond to common situations encountered on the job. The [video-based] oral component also involved realistic scenarios designed to simulate situations in which a sergeant would be expected to use oral communication skills in responding to a superior officer, responding to the mother of a victim, and responding to a new partner. (R. 388, Bench Trial Op. at 17, 19–20.) Other than baldly saying that the tests did not measure traits relevant to the sergeant position (see Third Br. at 9)—arguments that appear to circle back to the claim that the 2002 process needed a work simulation instead of the video test—plaintiffs cite no evidence that contests the job-relatedness or representativeness of the KSAPs measured in each test component. We discern no clear error with these validity findings. B. District Court’s Findings Regarding Reliability & Rank Ordering Plaintiffs devote most of their alternative argument to the district court’s findings regarding reliability and rank ordering. On reliability, the court found: [The City’s expert and the designer of the 2002 process] Dr. Jeanneret testified that he did not include a reliability estimate in the validation report because the 2002 process was heterogeneous, i.e., it measured numerous broad KSAP dimensions that were correlated with one another, and he felt that there was no appropriate estimate of reliability. According to Dr. Jeanneret, the most appropriate approach to reliability for such a heterogeneous test was test-retest reliability, which was not feasible under the circumstances. A reasonable alternative, Dr. Jeanneret asserted, would have been to develop an alternate form, requiring two identical tests which, he believed, was not possible in light of the Nos. 13-5452/5454 Johnson, et al. v. City of Memphis Page 19 particular testing environment. Since neither multiple administrations of the test nor parallel administration of identical tests were practicable, Dr. Jeanneret believed the only potentially applicable method of assessing reliability was to measure internal consistency using “coefficient alpha.” Dr. Jeanneret did not initially compute coefficient alpha because he intentionally designed a very heterogeneous test and making coefficient alpha, in his opinion, an inappropriate index of reliability. Both Dr. Jeanneret and [plaintiffs’ expert] Dr. DeShon subsequently measured coefficient alpha, using somewhat different methodologies. Dr. DeShon reported an overall reliability coefficient of .76 using a method known as stratified alpha. Dr. DeShon included seniority in his analysis, which Dr. Jeanneret testified was inappropriate because seniority was not part of the measurement process. (Jeanneret, Tr. Vol. 11, 1287–88; DeShon, Tr. Vol. 5, 575; Tr. Vol. 16, 1898, 1912.) The Court agrees that inclusion of seniority was inappropriate in assessing the reliability of the test. Since seniority was an administrative add-on component, there is no reason to expect that there would be a significant correlation or internal consistency between seniority and test items. Dr. Jeanneret eventually performed a reliability analysis using a “linear composite,” which resulted in a coefficient of .82. He also computed reliability using the formula for stratified alpha, which resulted in a coefficient of .83. The Court finds credible Dr. Jeanneret’s testimony as to the limited applicability of coefficient alpha in measuring reliability of a heterogeneous test which draws material for test items from multiple sources. The Court further finds that Dr. Jeanneret’s computations of stratified alpha without inclusion of seniority scores to be more appropriate than Dr. DeShon’s computation, which included seniority. Finally, the Court finds that Dr. Jeanneret’s conclusion that the 2002 process was sufficiently reliable is consistent with professional standards and is supported by relevant law. See Hearn v. City of Jackson, 340 F. Supp. 2d 728, 740–41 (S.D. Miss. 2003) (finding that a reliability coefficient of .79 is a common and acceptable value in the context of a heterogeneous test environment). (R. 388, Bench Trial Op. at 21–22 (transcript citations omitted).) On the subject of rank ordering, the court found: Under both Sixth Circuit precedent and the Guidelines, ranking of candidates is appropriate where it can be shown that a higher score correlates with higher job performance. See Williams v. Vukovich, 720 F.2d 909, 924 (6th Cir. 1983); 29 C.F.R. § 1607.14(C)(9) (2006). The requirements for rank ordering can be met through a substantial demonstration of job-relatedness, variance in test scores, and an adequate degree of test reliability. Guardians Ass’n of New York City Police Dep’t, Inc. v. Civil Serv., 630 F.2d 79, 104 (2d Cir. 1980). Nos. 13-5452/5454 Johnson, et al. v. City of Memphis Page 20 As discussed above, the test content of the 2002 process was substantially job-related and there was an acceptable level of test reliability. Many sections of the test consisted of items in which there were several right answers, with differing point values for various elements, and/or opportunities for additional credit, all of which serve to distinguish better performing candidates from lesser performing candidates. (Def’s Ex. 22, pp. 43–46.) The written test was closely modeled after the like section in the 2000 process, which Dr. DeShon acknowledged was able to differentiate between those candidates with more job knowledge from those with less knowledge. (DeShon, Tr. Vol. 5, 546–47.) Additionally, the raw scores on the 2002 assessment show a substantial variance, with the highest raw score of 358.750 and the lowest of 174.750, among 517 candidates. (Def’s Ex. 17.) See City of Columbus, 916 F.2d at 1102–03 (upholding rank ordering where score range was 40 points among 71 candidates). Based on the foregoing, the Court finds that rank ordering of the results of the 2002 process was proper, given that the test had an acceptable level of test reliability, was substantially job-related, and had substantial variance among the scores. (Id. at 22–23.) Plaintiffs lodge several objections to the reliability and rank-ordering findings, laced with a variety of counter-evidence in the opening of their response brief. (See Third Br. at 3–15, 44– 62.) We distill three primary arguments: (1) that the district court incorrectly determined that Dr. DeShon incorporated seniority into his composite reliability score, and thus clearly erred in crediting Dr. Jeanneret’s reliability testimony; (2) that the district court applied the wrong legal standard for rank ordering, and the City failed to justify rank ordering by showing that higher test scores resulted in better job performance; and (3) that the district court erred by accepting the City’s use of seniority in the 2002 process. None demonstrates a reversible legal error or clearly erroneous factual finding. 1. Dr. DeShon’s Non-Use of Seniority & the Court’s Credibility Finding First, plaintiffs deny the district court’s factual assertion that Dr. DeShon included seniority in his reliability calculations. The City appears to concede the inconclusive nature of the evidence cited by the district court (see Fourth Br. at 27–28), but notes that any error in this regard is harmless because both experts’ reliability scores (.76 from DeShon, .82–.83 from Jeanneret) fall within the range of reliability scores accepted by courts. See, e.g., Hearn, 340 F. Supp. 2d at 740 (approving of exam with .79 reliability coefficient). Yet any mistake regarding Nos. 13-5452/5454 Johnson, et al. v. City of Memphis Page 21 the constituent parts of Dr. DeShon’s composite reliability score (.76) leaves undisturbed the court’s remaining credibility determinations pertaining to Dr. Jeanneret’s reliability methodology and testimony—namely, its approval of (1) “Dr. Jeanneret’s testimony as to the limited applicability of coefficient alpha in measuring reliability of a heterogeneous test which draws material for test items from multiple sources,” and (2) his “conclusion that the 2002 process was sufficiently reliable.” (R. 388, Bench Trial Op. at 21–22.) The court’s remaining conclusion—choosing Dr. Jeanneret’s reliability estimates (.82– .83) over that of Dr. DeShon (.76)—suffers only from the court’s mistaken belief that Dr. DeShon’s figure included seniority. So far as we can tell, plaintiffs accept the court’s related finding that these specific reliability calculations should not include seniority. Surprisingly, for all their complaints about Dr. Jeanneret’s methods, plaintiffs voice no concern for the higher result he achieved (.82 or .837) using their preferred calculation method, stratified alpha. Arguably, the district court selected Dr. Jeanneret’s number because it found his testimony more credible (consistent with its other credibility findings on this issue), not because it believed that Dr. DeShon made a calculation error. And even if the district court chose Dr. DeShon’s reliability number (.76), the district court cited authority approving a similar reliability coefficient. Hearn, 340 F. Supp. 2d at 740–41 (.79); cf. Nash, 895 F. Supp. at 1548 (stating that a reliability coefficient “above 0.70 is considered to be reliable”). Plaintiffs provide no authority compelling the conclusion that either a .76 or .82–.83 reliability score for this type of test fails as a matter of law.8 Instead, plaintiffs charge that Dr. Jeanneret conceded the inappropriateness of his own reliability estimate. To the extent plaintiffs suggest that Dr. Jeanneret rejected his own 7 Plaintiffs suggest in passing that Dr. Jeanneret did not know of “stratified alpha” and did not calculate it. (Third Br. at 52.) But Dr. Jeanneret explained that, though he initially lacked familiarity with the term “stratified alpha,” the “mathematics of the coefficient . . . [are] basically the same” as the “linear composite” figure he calculated. (R. 648-10, Trial Tr. (Jeanneret) at 1285–86.) We note that the cited evidence appears to invert the coefficient and stratified alpha scores (.83 and .82) noted by the district court and the City’s brief, but plaintiffs make no objection on this ground, and we have no reason to believe that the marginal difference between those two scores matters here. 8 Of course, we do not suggest that a reliability score of .70 suffices for all tests as a matter of law. Reliability determinations depend on the unique circumstances of the testing protocol. We simply acknowledge that this aspect of plaintiffs’ reliability argument asks us to determine credibility—something we cannot do. Harrison v. Monumental Life Ins. Co., 333 F.3d 717, 723 (6th Cir. 2003) (“Since we are not free to disregard the district court’s credibility assessment, the verdict must stand if [plausible evidence] supports [it.]”). Nos. 13-5452/5454 Johnson, et al. v. City of Memphis Page 22 calculations, they misread his testimony. (See R. 648-12, Trial Tr. (Jeanneret) at 1507 (acknowledging that his original report excluded a reliability coefficient, because it would not be an appropriate measure for the test, and stating his belief “that the coefficient alpha or internal consistency index of reliability [would not be] the most appropriate or even really an appropriate index for the reliability of the [2002 process]”).) As the district court noted, Dr. Jeanneret’s testimony explains the difficulty of calculating a reliability coefficient for a heterogenous test— i.e., one consisting of multiple, unrelated components that evaluate multiple tasks and characteristics. (See R. 648-10, Trial Tr. (Jeanneret) at 1273–81.) In choosing between the parties’ similar reliability estimates, the district court reasonably credited Dr. Jeanneret’s testimony that the best reliability measures—retesting candidates or administering duplicate tests—were impracticable for a process administered to more than 500 candidates. See, e.g., Anderson v. City of Bessemer City, 470 U.S. 564, 573–74 (1985) (“If the district court’s account of the evidence is plausible in light of the record viewed in its entirety, the court of appeals may not reverse it even though convinced that had it been sitting as the trier of fact, it would have weighed the evidence differently.”). 2. Rank Ordering Next, plaintiffs challenge the district court’s approval of the City’s use of rank ordering to distinguish between the candidates’ scores, arguing that the court misapplied three legal requirements for this scoring method set by this court in Police Officers for Equal Rights: (1) sufficient raw score spread (2) composite and component reliability, and (3) reasonable job analysis. Yet, as the City points out, our decision in Police Officers for Equal Rights included no such rule; it merely observed that the employer’s expert used those requirements. See 916 F.2d at 1102. Our standard states that “[r]anking is a valid, job-related selection technique only where the test scores vary directly with job performance.” Id. (quoting Williams v. Vukovich, 720 F.2d 909, 924 (6th Cir. 1983)). The EEOC guidelines for content-validity studies support this approach: If a user can show, by a job analysis or otherwise, that a higher score on a content valid selection procedure is likely to result in better job performance, the results may be used to rank persons who score above minimum levels. Nos. 13-5452/5454 Johnson, et al. v. City of Memphis Page 23 29 C.F.R. § 1607.14(C)(9) (emphasis added). The City satisfies this likelihood threshold with “a substantial demonstration of job relatedness and representativeness,” score variance, and an “adequate degree” of test reliability. See Guardians, 630 F.2d at 104; see also Police Officers for Equal Rights, 916 F.2d at 1100 (explaining that, while a test should “measure important aspects of the job . . . for which appropriate measurement is feasible,” the job-relatedness requirement does not demand that the test “measure all [job] aspects, regardless of significance, in their exact proportions”). The City’s evidence clears this hurdle.
First, the district court found that the City’s consultants conducted a “comprehensive job analysis” to identify the relevant KSAPs for the sergeant position, and that the test components measured relevant job tasks using similar materials to those used on the job and realistic law enforcement scenarios. (R. 388, Bench Trial Op. at 17, 19–20.) As noted above, the plaintiffs present no specific objection to these job-relatedness findings.
Second, the district court found “substantial variance” among the promotion scores: of the 517 tested candidates, the 2002 process yielded a raw-score point spread of 184 points between the highest and lowest candidates (358.75–174.75), out of a possible 384.5 points. (Id. at 23.) Our review of the exam results reveals no clear error in this finding. (R. 656-23, 2002 Process Exam Results at 1–14.) Nor do we detect clear error in the court’s finding of significant variance. Cf. Police Officers for Equal Rights, 916 F.2d at 1102–03 (permitting rank ordering where “[t]here was a spread of more than forty points among 71 test takers,” the highest score was 89.66, and the passing score was 70). Though plaintiffs stress that only one point separated approximately 30 of the more than 500 candidate scores, that circumstance pales in comparison to the sort of score-bunching found problematic elsewhere. See Guardians, 630 F.2d at 103 & nn.19–20 (finding insufficient reliability for rank ordering where nearly 9,000 applicants, or 2/3 of the passing scores, had scores between 94 and 97, out of 110 possible points). Moreover, the focus on promotional Nos. 13-5452/5454 Johnson, et al. v. City of Memphis Page 24 scores here exaggerates the 2002 process’s bunching effect, because the same candidates’ raw scores ranged between 303 and 341, or 79.0 and 88.7 on a 100-point scale. (See R. 656-23, 2002 Process Exam Results at 3–4.) Varying seniority points (1–10) contributed significantly to this purported bunching problem.
Third, the district court found sufficient test reliability, crediting Dr. Jeanneret’s composite reliability scores of .82–.83. Again, we find no clear error with the court’s factual findings and no error with its legal conclusion. Plaintiffs briefly mention that the individual components of the 2002 process received poor reliability scores ranging from .32–.79. Indeed, the relatively low component reliability scores give pause. See Police Officers for Equal Rights, 916 F.2d at 1102 (allowing rank ordering where the exam’s component tests achieved reliability scores ranging from .85–.97). Though the district court did not make specific findings regarding component reliability scores, plaintiffs point to no authority requiring such findings to sustain a rank-ordering test. Cf. id. at 1103 (holding that “the trial court was not clearly erroneous in accepting . . . [expert] testimony . . . on the issue of reliability and rank order scoring” that happened to include a component reliability estimate) (footnote omitted). “The district judge is entitled in questions of this kind which require expert [statistical] opinion to rely on that opinion.” Id. So too here, where the district court relied on Dr. Jeanneret’s opinion that the heterogeneous nature of the 2002 process’s component tests made reliability coefficients less appropriate measures of reliability than other, impracticable methods, like test/re-test consistency or dual-test administration. (R. 388, Bench Trial Op. at 21–22.) And, as we said, both the plaintiffs’ expert and the City’s expert attained composite reliability figures greater than .75 regardless of any reliability problems with the component tests. Still, the plaintiffs argue that the City produced no evidence that the test scores vary with performance so as to justify rank ordering. See Williams, 720 F.2d at 924. And, they add, high standard error measurements (SEM +3.64, +10.09 SED) belie the City’s claim of reliable test scores, rendering 428 of the 517 candidate scores statistically indistinguishable. Though the Nos. 13-5452/5454 Johnson, et al. v. City of Memphis Page 25 district court’s opinion did not specifically address SEM or SED, neither of these claims undermines its finding that the City demonstrated sufficient reliability for rank ordering. With regard to likely test-score/job-performance correlation, Dr. Jeanneret’s supplemental report cited published industry principles asserting that “cognitively based selection techniques developed by content-oriented procedures . . . can usually be assumed to have a linear relationship to job behavior.” (R. 656-7, Jeanneret Resp. Suppl. Rpt. at 35 (acknowledging that the 2002 process, while not a cognitive-ability test, had cognitive components).) We also note as significant the district court’s finding—unchallenged on appeal—that the 2002 process’s “written test was closely modeled after the like section in the 2000 process, which Dr. DeShon acknowledged was able to differentiate between those candidates with more job knowledge from those with less knowledge.” (R. 388, Bench Trial Op. at 23 (citing R. 648-4, Trial Tr. (DeShon) at 546–47).) On the topic of SEM, plaintiffs offer no authority explaining why an SEM range of 2.8 (Dr. Jeanneret’s corrected estimate calculated during trial) to 3.7, by itself, renders the 2002 process inherently unreliable or trumps other measurements of reliability. They do not show, for instance, the sort of score-bunching and passage-rates deemed problematic by the Second Circuit in Guardians. See 630 F.2d at 103 & n.19 (finding unreliable a rank-ordered promotional test with an SEM of 2.4, explaining that the test “was too easy” and resulted in “8,928 applicants, two-thirds of all who passed, [with] bunched [scores] between 94 and 97” out of a possible 110 points). As for SED, Dr. Jeanneret’s supplemental report provides detailed reasons, supported by industry publications, for not relying on this measurement. (See R. 656-7, Jeanneret Resp. Suppl. Rpt. at 34–35.) Specifically, he opposes using large SED bands to equate broad ranges of test scores, explaining that SED bands “are calculated based on the normal probability distribution,” meaning that “the further apart two scores are, the more likely those scores are to be truly different.” (Id. at 34.) He elaborates, citing an industry publication finding that “even when a test is quite reliable, a typical SED band covers so large a part of the test score range that the preferred interpretation of banding advocates . . . is false.” Dr. Jeanneret goes on to note that “test score bands . . . try[ing] to account for measurement error . . . [are] not required, or even Nos. 13-5452/5454 Johnson, et al. v. City of Memphis Page 26 endorsed by the professional standards in the field of industrial and organizational psychology (i.e., Principles, 2003; Standards, 1999).” (Id.) Ultimately, the district court heard the parties’ competing evidence regarding reliability, SEM, and SED, and the court found that the City justified the use of rank ordering with a substantial demonstration of job-relatedness, score variance, and an adequate degree of reliability supporting the likelihood that test scores would correlate to job performance. We find no clear error with the court’s findings of fact in this regard and no error with its ultimate legal conclusion regarding rank ordering. 3. Seniority Scoring Last, plaintiffs denounce the City’s use and weighting of candidates’ seniority—an item included in their Memorandum of Understanding (MOU) with the officers’ union—as a promotional factor. The Supreme Court has held that a “bona fide seniority system [is not] unlawful under Title VII,” even though “a seniority system inevitably tends to perpetuate the effects of pre-Act discrimination.” Int’l Bhd. of Teamsters v. United States, 431 U.S. 324, 352– 53 (1977) (construing 42 U.S.C. § 2000e-2(h)). Thus, this court will sustain the seniority component of a promotional procedure “so long as an intent to discriminate did not enter into its adoption and it has been maintained free from any illegal purpose.” City of Akron, 824 F.2d at 481. Though not quarreling with this standard, plaintiffs challenge the binding effect of the MOU on the City. But, contractual enforceability aside, without showing discriminatory intent or illegal purpose, plaintiffs have no grounds to impugn the City’s use of seniority. As for weighting, the plaintiffs suggest that the City’s scoring errors inflated seniority’s impact from an intended 10% to 25%. The cited testimony, however, appears to refer to something other than a tabulation error; Dr. DeShon differentiates between a “nominal weight” of 10% and an “effective” or “actual weight” of 25%, referring to the degree to which seniority affected promotion score variance. (R. 648-14, Trial Tr. (DeShon) at 1753–55.) Review of the test results (raw scores, scaled scores, and promotion scores) confirms this, revealing that seniority accounted for up to 10 points of the promotion score, out of a possible 110 points. (See generally R. 656-23.) Regardless of the nature of the alleged scoring error, in the absence of Nos. 13-5452/5454 Johnson, et al. v. City of Memphis Page 27 evidence that the City’s weighting of seniority reflects a discriminatory intent or other illegal purpose, plaintiffs gain no ground. See City of Akron, 824 F.2d at 481. Because the seniority component required no additional validation, the district court properly rejected this aspect of the plaintiffs’ challenge. V. CONCLUSION For these reasons, we affirm in part and reverse in part the district court’s judgment. We AFFIRM the district court’s immunity-based dismissal of plaintiffs’ negligence claim related to the 2000 process, but we REVERSE the district court’s Title VII judgment invalidating the 2002 process, thereby MOOTING plaintiffs’ challenge to the district court’s choice of remedies for the 2002 process. We VACATE the district court’s fees award and REMAND for further consideration in light of these developments.