Opinion ID: 3204484
Heading Depth: 2
Heading Rank: 2

Heading: Rank-Order Selection

Text: When officials at the BPD received the results of the 2005 and 2008 sergeant promotional examinations from HRD, they selected as many police officers for promotion as there were vacancies currently available, beginning with the highest-scoring name at the top of the list and moving down the list, one at a time, in order of the score each candidate received. The Officers argue that this method of selection--quite independently from the written examination itself--led to a disparate impact and the district court was obligated to conduct a separate analysis of its validity under Title VII. We review the legal sufficiency of the district court's holding on this point de novo and its subsidiary fact-finding for clear error. E.E.O.C. v. Steamship Clerks Union, Local 1066, 48 F.3d 594, 603 (1st Cir. 1995). The Officers first argue that the district court failed altogether to wrestle with the consequences of rank-order selection. This is clearly not the case. Citing section 12 The Officers did not move to strike any portion of Outtz's testimony under Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993). Hence, even if we had thought that any part of Outtz's opinion was unreliable or unsupported, we would have had to employ plain error review. See United States v. Diaz, 300 F.3d 66, 74 (1st Cir. 2002). - 30 - 1607.14(C)(9) of the Guidelines, the district court noted in its exegesis of the law that [t]he use of a ranking device requires a separate demonstration that there is a relationship between higher scores and better job performance. Lopez, 2014 U.S. Dist. LEXIS 124139, at –17. The court went on to find that Boston's selection method reliably predicts a candidate’s suitability for the job, such that persons who perform better under the test method are likely to perform better on the job. Id. at . This finding by the district court, to the Officers, is not enough. Based on their reading of the Guidelines, something more is required. The Officers argue that the use of the results of an examination that is minimally valid insofar as it tests job-related skills may not necessarily be valid if used to select candidates solely according to their scores on that exam. Two provisions of the Guidelines discuss an employer's appropriate use of a rank-ordering selection method. In the section of the Guidelines establishing General Principles, the EEOC has advised the following: The evidence of both the validity and utility of a selection procedure should support the method the user chooses for operational use of the procedure, if that method of use has a greater adverse impact than another method of use. Evidence which may be sufficient to support the use of a selection procedure on a pass/fail (screening) basis may be insufficient to support the use of the same procedure on a ranking basis under these guidelines. Thus, if a user decides to use a - 31 - selection procedure on a ranking basis, and that method of use has a greater adverse impact than use on an appropriate pass/fail basis (see section 5H below), the user should have sufficient evidence of validity and utility to support the use on a ranking basis. 29 C.F.R. § 1607.5(G) (emphasis supplied). The Guidelines also contain a refinement of this principle specific to the use of content validity studies in the Technical Standards section: If a user can show, by a job analysis or otherwise, that a higher score on a content valid selection procedure is likely to result in better job performance, the results may be used to rank persons who score above minimum levels. Where a selection procedure supported solely or primarily by content validity is used to rank job candidates, the selection procedure should measure those aspects of performance which differentiate among levels of job performance. Id. § 1607.14(C)(9). These two statements evidence some inconsistency. Section 1607.5(G) clearly indicates that an employer need have sufficient evidence of validity to support use of the exam on a ranking basis if . . . that method of use has a greater adverse impact than use on an appropriate pass/fail basis (emphasis supplied). Under this guidance, if an exam is valid, one may use it on a rank-order basis unless the use of rank ordering creates or adds to a disparate impact. One can read section 1607.14(C)(9), however, as requiring that, to defend rank ordering, the employer must first show that a higher score on a content valid selection - 32 - procedure is likely to result in better job performance; i.e., one must validate the use of ranking itself if the exam as a whole produces a disparate impact. Other provisions of the Guidelines support this latter reading, albeit without acknowledging the inconsistency. Compare, e.g., id. § 1607.5(G) ([I]f a user decides to use a selection procedure on a ranking basis, and that method of use has a greater adverse impact than use on an appropriate pass/fail basis . . . , the user should have sufficient evidence of validity and utility to support the use on a ranking basis. (emphasis supplied)), with Adoption of Questions and Answers to Clarify and Provide a Common Interpretation of the Uniform Guidelines on Employee Selection Procedures, 44 Fed. Reg. 11,996, 12,005, Question and Answer n. 62 (1979) (Use of a selection procedure on a ranking basis may be supported by content validity if there is evidence from job analysis or other empirical data that what is measured by the selection procedure is associated with differences in levels of job performance.). Several courts have seemed to approach this issue by requiring more scrutiny of the validation evidence as a whole when rank ordering is used, particularly when the exams in question have led to closely bunched scores. See Johnson v. City of Memphis, 770 F.3d 464, 479–81 (6th Cir. 2014), cert. denied, 136 S. Ct. 81 (2015); Police Officers for Equal Rights v. City of Columbus, 916 F.2d 1092, 1102–03 (6th Cir. 1990); Guardians Ass'n - 33 - of N.Y.C. Police Dep't, Inc. v. Civil Serv. Comm'n of City of N.Y., 630 F.2d 79, 100–05 (2d Cir. 1980). The district court in this case expressly adopted the approach most favorable to the Officers, citing 29 C.F.R. § 1607.14(C)(9), for the proposition that [t]he use of a ranking device requires a separate demonstration that there is a relationship between higher scores and better job performance. Lopez, 2014 U.S. Dist. LEXIS 124139, at –17. As we have noted, supra, and as the Officers seem to ignore, the court then specifically found that it was satisfied on the evidence that Boston carried its burden of showing that persons who perform better under the test method are likely to perform better on the job. Id. at –62. As a predicate to this finding, the district court observed that a group of incumbent sergeants who took an exam in 2005 that contained 53 of the questions asked of applicants on the sergeant's exam had a materially higher passing rate on those common questions than did the job applicants. Id. at – 57. The district court viewed this evidence as showing that those questions were related to the sergeants' actual performance of their jobs. Id. at . The Officers' only reply is to say that this evidence only shows that people who previously did well on the exam (and thus became sergeants) still did well on it. But the Officers point to no evidence that these incumbent sergeants in 2005 somehow achieved their positions by previously taking the - 34 - same, or more or less the same, exam that was first offered in 2005. Even accepting the district court's opinion that added scrutiny was called for because rank ordering was used, whatever added scrutiny one need apply here certainly falls short of the added scrutiny one would apply if rank ordering had been a material contributor to the disparate impact. Although they belatedly offer on appeal, without citation to the record, counsel's own calculations that banding in lieu of rank order selection would have caused more Black and Hispanic applicants to be reachable for selection by subjective performance criteria, the Officers made no effort to demonstrate that an increased number of Black and Hispanic applicants likely would have been selected under such an alternative approach. Rank ordering furthers the City's interest in eliminating patronage and intentional racism under the guise of subjective selection criteria. Such a goal is itself a reasonable enough business need so as to provide some weight against a challenge that is unaccompanied by any showing that rank order selection itself caused any disparate impact in this case.13 13 Given the absence of any showing that an equally or more valid alternative to rank-order selection would have reduced disparate impact, we need not address the Officers' arguments that any state law favoring rank order selection is unlawful or preempted. - 35 - None of this is to suggest that Boston could not have come up with an exam that did a better job of furthering its goal of selecting the best candidates for promotion to the position of sergeant. The Officers argue persuasively that Boston could have made the exam more valid. Indeed, Outtz agreed and so, too, it would appear, does the City, which, counsel tells us, has since 2008 developed a new exam that it now uses. The point, instead, is that the record contains detailed, professionally buttressed and elaborately explained support for the district court's finding that persons who perform better under the test method are likely to perform better on the job. Id. at . Given that plainly supported finding, it makes little sense to debate in the abstract how much better the exam might have been. Instead, it makes more sense to move to the next step of the inquiry to see if there is any alternative selection test that would have had less adverse impact. If so, then the court will have a meaningful gauge of validity by comparing the two tests. And if the alternative test with less adverse impact has equal or greater validity, it makes no difference how valid the employer's actual test is; the employee wins. Ricci, 557 U.S. at 578 (citing 42 U.S.C. §§ 2000e–2(k)(1)(A)(ii) and (C)). Conversely, absent proof of an equally or more valid test that has less adverse impact, there is no reason to force the employer to promote randomly if the employer has a tool that will do - 36 - meaningfully better than that. For this reason, once a court concludes that a selection device is materially more job-related than random selection would be, it makes sense to turn the focus sooner rather than later to the question of whether there is any alterative option that is as good or better, yet has less adverse impact. Otherwise, courts and litigants are left to engage in unpredictable qualitative assessments without any meaningful gauge as to what is enough. We therefore turn next to that question.