Opinion ID: 805464
Heading Depth: 5
Heading Rank: 1

Heading: Suitable Job Analysis

Text: M.O.C.H.A. posits that the district court erred in finding that the 1998 examination was based on a suitable job analysis, i.e., an “assessment ‘of the important work behavior(s) required for successful performance and their relative importance.’” Guardians I, 630 F.2d at 95 (quoting 29 C.F.R. § 1607.14(C)(2)). Specifically, M.O.C.H.A. contends that because the challenged test was premised on the Civil Service Department’s statewide job analysis, in which Buffalo barely participated, the analysis amounted to only an “other cities guess” that the survey results were consistent with the job actually performed by lieutenants in the Buffalo Fire Department. Appellants’ Br. 35. M.O.C.H.A. charges that, as a matter of law, Buffalo was not entitled to rely on such guesswork and, by extension, that the district court clearly erred in similarly finding that the Civil Service Department’s job analysis was suitable to a promotional examination for the lieutenant position in the Buffalo Fire Department. At the outset, we acknowledge that the Civil Service Department received minimal feedback from the Buffalo Fire Department in conducting its job analysis, and did not perform any on-site observations of that department or interview any of its members. 20 Further, at trial, Buffalo did not call any expert witness to opine that the results of the statewide job analysis correlated to the Buffalo fire lieutenant position. Nor did it present any direct evidence from a fact witness within the Buffalo Fire Department as to the responsibilities of a lieutenant in that department. Buffalo’s minimal participation in the Civil Service Department’s three-year statewide job analysis of firefighter positions is perplexing. So too is Buffalo’s strategic decision to defend against a disparate impact claim without calling either an expert or fact witness to link the lieutenant position within the Buffalo Fire Department to the Civil Service Department’s job analysis of that position statewide. On such a record, it would have been within the fact finder’s discretion to draw adverse inferences against Buffalo and to conclude that it had not carried its burden on the question of suitable job analysis. See Old Chief v. United States, 519 U.S. 172, 188–89 (1997) (recognizing that “triers of fact may penalize the party who disappoints them” in failing to introduce evidence to sustain his burden “by drawing a negative inference against that party” (internal quotation marks omitted)). The issue before this court, however, is not whether a fact finder could have found against Buffalo on issues on which it carried the burden, but whether the fact finder here was required by law to do so. We conclude that he was not. In reaching that conclusion, we recognize that neither the Civil Service Department nor the district court may have been able to conclude as a matter of deduction that the statewide job analysis was suitable to the position of lieutenant in the Buffalo Fire Department without knowing more about that particular department. But that is not to say that the conclusion could not be reached as a 21 matter of induction. Application of the statewide job analysis to Buffalo was not a stab in the dark. Rather, it was based on a sound inference that, because reliable statistics showed that fire lieutenants across the state (and even the nation) shared the same critical tasks requiring the same critical skills, it was more likely than not that the same tasks and skills were critical to the fire lieutenant job in Buffalo. See Clark v. Astrue, 602 F.3d 140, 147 (2d Cir. 2010) (“In the civil context, a finding that X is more likely than not true is the equivalent to a finding that X is true.”). The law permits inferential fact finding even though it may be less certain than findings from direct evidence, see, e.g., Serricchio v. Wachovia Sec. LLC, 658 F.3d 169, 186–87 (2d Cir. 2011); Cifra v. Gen. Elec. Co., 252 F.3d 205, 217 (2d Cir. 2001), and even when the burden of proof is beyond a reasonable doubt, see, e.g., United States v. Abu-Jihaad, 630 F.3d 102, 135 (2d Cir. 2010), cert. denied, 131 S. Ct. 3062 (2011). Thus, despite Buffalo’s failure meaningfully to participate in the statewide analysis of the fire lieutenant position, we are satisfied that the district court still could find from the totality of the evidence that the Civil Service Department’s statewide job analysis was—more likely than not—suitable to identifying the tasks and skills relevant to the performance of that job in Buffalo. Survey data warranting 95% statistical confidence showed that persons across New York with the title of “fire lieutenant” identified the same tasks as critical to their jobs regardless of the size or location of the fire department where they served. Indeed, 90% of surveyed New York fire lieutenants, when asked to rank specified tasks according to their criticality in the performance of the respondents’ jobs, provided virtually identical responses, 22 and those who departed did so only “slightly.” Trial Tr. 319. Such data made it highly likely that the job of fire lieutenant, wherever performed in New York, had the same critical tasks and required the same critical skills. Indeed, state survey data showed greater consistency across New York with respect to the position of fire lieutenant than with other high-ranking firefighter positions, where responses were more variable. Thus, M.O.C.H.A. mischaracterizes the Civil Service Department’s job analysis when it contends on appeal that Steinberg simply relied on shared job titles to “guess” that the content of a job in one location was the same as the content of a job with the same title in another location. Rather, the trial record demonstrates that it was the high degree of common responses to task and SKAP surveys provided by persons holding the title of fire lieutenant that permitted Steinberg to conclude with a high degree of confidence that the analysis reliably applied to the position of fire lieutenant across New York.12 Further, the trial record shows that Steinberg tested the conclusion derived from the survey data by various means consistent with the joint standards of employment test design. Not only did she arrange for the test categories created from the survey data to be reviewed and approved by experts serving on the Civil Service Department’s Fire Advisory 12 Insofar as Steinberg testified at trial that “I guess the evidence is that it would be highly unlikely that [Buffalo] would be different,” Trial Tr. 354 (emphasis added), one can discern even from a cold record that her “guess” was not a product of baseless speculation but, rather, a reasonable inference drawn from a job analysis showing that fire lieutenant tasks are essentially identical across the studied jurisdictions in New York. 23 Committee,13 she also compared the resulting six-part test plan with test plans from fourteen other large urban fire departments and found general consistency. Indeed, M.O.C.H.A.’s own expert testified that the joint standards permit both of these validation methods, although he disagreed that they were sufficient in this case to establish job relatedness. Further, as the district court noted, see M.O.C.H.A. I, 2009 WL 604898, at , Buffalo’s own job specifications for the fire lieutenant position mirrored those of other New York departments that had participated in the Civil Service Department survey, for example, White Plains and Newburgh, jurisdictions with 170 and 61 firefighting positions, respectively. From the totality of this evidence, we are satisfied that the district court could make a preponderance finding that the Civil Service Department’s statewide job analysis for the position of fire lieutenant was suitable even to a jurisdiction such as Buffalo, which had participated only minimally in that analysis, and further, that the test developed from that analysis was job related to the lieutenant position as performed in the Buffalo Fire Department. In short, this is not a case where the record shows that one municipality simply relied on another’s employment test without any evidence of a correlation between the two jurisdictions’ circumstances. See generally EEOC v. Atlas Paper Box Co., 868 F.2d 1487, 1499 (6th Cir. 1989) (holding that employer failed to sustain burden by assuming jobs were 13 Buffalo’s failure to adduce evidence that any committee member was familiar with the Buffalo Fire Department was properly considered by the district court in deciding what weight to give the Fire Advisory Committee’s approval of Steinberg’s job analysis for the position of fire lieutenant, a matter we have no reason to question on appeal. See Joseph v. N.Y.C. Bd. of Educ., 171 F.3d 87, 93 (2d Cir. 1999). 24 similar and that familiar “intelligence tests are always valid”); 29 C.F.R. § 1607.9(A) (prohibiting employer from using promotional examination based on “nonempirical or anecdotal accounts of selection practices or selection outcomes”). Here, substantial empirical evidence, reinforced by expert review and jurisdictional comparisons, showed that fire lieutenants across New York performed the same critical tasks and required the same critical skills, regardless of the location and size of their departments. While this may not absolutely foreclose the possibility that other tasks and skills might be relevant to the job of fire lieutenant in Buffalo, the law does not demand such a showing even when the standard of proof is beyond a reasonable doubt, see United States v. Reich, 479 F.3d 179, 190 (2d Cir. 2007), much less when it is a preponderance, i.e., when a party need only prove that its version of the facts is more probable than its adversary’s, see Clark v. Astrue, 602 F.3d at 147. Here, the record evidence was sufficient to permit the district court to find that it was highly unlikely that the fire lieutenant job in Buffalo required different tasks and skills from those identified in the statewide survey. This, in turn, permitted it to find that the statewide job analysis for fire lieutenants was suitable to Buffalo, notwithstanding its fire department’s minimal participation in the statewide survey. M.O.C.H.A.’s arguments to the contrary are not persuasive. Insofar as M.O.C.H.A. faults the statewide job analysis because the Civil Service Department failed to secure sufficient survey responses from the Buffalo Fire Department, we reiterate our earlier conclusion that it was not clear error for the district court to find the job analysis suitable to Buffalo in light of (1) survey results convincingly showing that the job of fire lieutenant is 25 effectively the same across New York fire departments, (2) expert approval of the statewide fire lieutenant test plan, (3) the similarity between the statewide fire lieutenant test plan and test plans of large jurisdictions nationwide, and (4) the similarity between Buffalo’s fire lieutenant job specifications and those of other New York fire departments whose members participated in greater number in the surveys. Next, to the extent that M.O.C.H.A. suggests that in-person observations or interviews within the Buffalo Fire Department were necessary for a suitable job analysis, we think it misreads our precedent. In describing “a proper job analysis” as including “a thorough survey of the relative importance of the various skills involved in the job in question and the degree of competency required in regard to each skill,” we have stated that such a survey “is conducted by interviewing workers, supervisors and administrators; consulting training manuals; and closely observing the actual performance of the job.” Guardians II, 633 F.2d at 242 (internal quotation marks omitted). Although interviews and observations may be best practices, our endorsement did not impose an absolute requirement for every job analysis. In some circumstances, it may be possible to gather reliable job-specific information by other means, such as survey instruments sent directly to employees. Here, the Civil Service Department gathered data about the job of fire lieutenant by sending surveys to incumbent fire lieutenants statewide and receiving a sufficient number of responses to warrant 95% statistical confidence in the results. Further, these responses revealed statewide uniformity among responding fire lieutenants in identifying the tasks and SKAPs critical to their job. On these facts, and in the absence of any evidence to the contrary, we conclude that the survey methodology employed by Steinberg was adequate for a suitable job analysis. 26 Nor are we persuaded to fault the district court’s finding because it was not informed by an expert opinion that the statewide job analysis was suitable to the Buffalo Fire Department. Such an expert opinion, like any direct evidence linking the Buffalo fire lieutenant’s job to the statewide job analysis, would have strengthened Buffalo’s defense. But expert opinion was not necessary as a matter of law. While the absence of expert testimony might carry more weight if Buffalo had been unable to rebut the testimony of plaintiffs’ own expert, Dr. Kevin Murphy, that is not the case. Murphy asserted that Steinberg “simply assum[ed]” that the job of fire lieutenant is “pretty similar from one place to another without, to my knowledge, any detailed analysis, any analytic demonstration that that’s true.” Trial Tr. 263. But Steinberg herself refuted that charge by denying that she made any such assumption and, more important, by explaining that she relied on survey data showing a 90% correlation between fire lieutenants’ tasks across jurisdictions. Because Steinberg provided a factual refutation to Murphy’s criticism that a fact finder would be capable of comprehending without “special or peculiar training,” we identify no clear error in the district court’s decision to credit her testimony and to find her job analysis suitable to Buffalo even without an expert witness. Wills v. Amerada Hess Corp., 379 F.3d 32, 46 (2d Cir. 2004) (internal quotation marks omitted). Similarly, we reject M.O.C.H.A.’s suggestion that Steinberg could not testify as to the challenged job analysis or 1998 examination without herself being qualified as an expert. As the district court observed, Steinberg testified to her personal knowledge regarding the statewide task and SKAP surveys, the results obtained therefrom, and the creation of the 27 1998 examination. Thus, M.O.C.H.A.’s argument is defeated by our decision in United States v. Rigas, 490 F.3d 208 (2d Cir. 2007), wherein we explained that “[a] witness’s specialized knowledge, or the fact that [s]he was chosen to carry out an investigation because of this knowledge, does not render [her] testimony expert as long as it was based on [her] investigation and reflected [her] investigatory findings and conclusions, and was not rooted exclusively in [her] expertise.” Id. at 224 (internal quotation marks and alteration omitted). In these circumstances, Steinberg’s testimony was admissible without regard to the limitations on expert opinions imposed by Fed. R. Evid. 702. Nor do we identify clear error in the district court’s notation of the absence of evidence that Buffalo fire lieutenants perform different tasks or require different skills than do firefighters elsewhere in New York. See M.O.C.H.A. I, 2009 WL 604898, at  (describing Steinberg’s testimony). We do not understand the district court to have made this observation to shift the burden of proof from Buffalo or to allow Buffalo to carry that burden simply by reference to the lack of contrary evidence, either of which would have been legal error. See 42 U.S.C. § 2000e-2(k)(1)(A)(i); Albemarle Paper Co. v. Moody, 422 U.S. at 425. Rather, we understand this observation to reference the lack of any evidence impeaching Steinberg’s conclusion that, based on the survey data, the job of fire lieutenant involved the same tasks and required the same skills across New York, which would likely be true in Buffalo as well. Finally, we do not identify clear error in the district court’s finding that the Civil Service Department’s job analysis was suitable despite Buffalo’s outlier size. The Buffalo 28 Fire Department is approximately twice the size of the Syracuse Fire Department, the next largest to participate in the statewide survey. Nevertheless, the district court could reasonably find that the job analysis was adequate even for a fire department as large as Buffalo’s in light of the Civil Service Department’s cross-validation of the test plan it created from its survey data with test plans from fourteen large fire departments nationwide. As Steinberg testified, the fourteen large departments’ test plans “were all—at least 90 percent overlap—identical to the one that we gave to Buffalo and identical to each other.” Trial Tr. 72. In sum, we conclude that, despite Buffalo’s failure meaningfully to participate in the Civil Service Department’s statewide survey of firefighters, the district court was entitled to conclude that the ensuing analysis of the job of fire lieutenant was suitable even for Buffalo because survey results showed, with 95% statistical confidence, that fire lieutenants perform the same critical tasks and require the same skills, knowledge, abilities, and personal characteristics across New York. Indeed, the conclusion was reinforced by expert review and nationwide comparisons of the statewide test plan developed from the survey data. While the conclusion would have been stronger still if Buffalo had offered any expert or direct evidence linking its fire lieutenants’ responsibilities to those identified in the statewide survey, we decline to hold that such evidence was required as a matter of law to permit the district court to make the requisite preponderance finding of a suitable job analysis. 29 b. Reasonable Competence and Content Relatedness M.O.C.H.A. further contends that the district court could not find the challenged 1998 test job related to Buffalo’s fire lieutenant position. In this respect, it maintains that the evidence was insufficient to show that the generic sub-tests in the 1998 examination—intended to assess (1) understanding and interpreting written material, (2) training practices, and (3) supervision—were the products of reasonably competent test design and were related to, and representative of, the fire lieutenant position. The record defeats these arguments.
The trial evidence was sufficient to establish that the generic sub-tests were the product of reasonably competent test design. In Guardians I, we stated that the reasonable competence of an employment examination’s design can be called into doubt if (1) the examination was not created by professional test preparers, or (2) no sample study was performed to ensure that the questions were comprehensible and unambiguous. See 630 F.2d at 96. Here, the trial record shows that the generic sub-tests were written by professional test preparers. Steinberg testified that she relied on the Civil Service Department’s units responsible for drafting cross-occupational test questions to write the generic sub-tests. Paul Kaiser, the Department’s director of testing services who supervised the creation of the Lower Level Fire Promotion test series, detailed the process by which the Department drafts cross-occupational test questions and calibrates them according to specific job responsibilities and the promotion level at issue. Kaiser explained that, in doing so, the Department routinely consults with experts and people in the field at issue. 30 Buffalo offered no evidence that the Civil Service Department conducted a sample study to determine that the generic sub-test questions were comprehensible and unambiguous. But it did submit evidence that, in creating these sub-tests, the Civil Service Department employed cross-occupational questions from previous employment examinations, which had been screened for objections from past test administrators and takers. Whatever the advantages of a prospective sample study, we conclude that the Civil Service Department’s consideration of such feedback to check the quality of crossoccupational questions, in combination with evidence that professional test preparers drafted them, was sufficient to support the district court’s finding that the generic sub-tests were the product of reasonably competent test design.
M.O.C.H.A. also contends that there was no proof that the generic sub-tests “measure[d] anything.” Appellants’ Br. 42. By this, we understand M.O.C.H.A. to be making three arguments, none of which is persuasive. First, M.O.C.H.A. submits that Buffalo failed to introduce statistical evidence demonstrating that an applicant’s success on the generic sub-tests is predictive of success as a fire lieutenant. This argument fails because the district court expressly found that the 1998 examination was content, not construct, validated, and that this content validation was an appropriate method for determining the examination’s job relatedness. See M.O.C.H.A. I, 2009 WL 604898, at –13. M.O.C.H.A. does not challenge these findings on appeal. 31 In Guardians I, we explained that construct validation is “frequently impossible” because it requires “a demonstration from empirical data that the test successfully predicts job performance.” 630 F.2d at 92; see Gulino v. N.Y. State Educ. Dep’t, 460 F.3d at 384 (discussing greater difficulty of construct validation relative to content validation). By contrast, content validation does not require this predictive validation study, and only obligates a test-maker to show that the Guardians I factors were satisfied. See Guardians I, 630 F.2d at 95. Thus, even if a predictive validation study would have been preferable, see 29 C.F.R. § 1607.14(C)(5) (stating that, “[w]henever it is feasible, appropriate statistical estimates should be made of the reliability of the selection procedure” that has been content validated), we cannot conclude that its absence was fatal to Buffalo’s defense in light of the district court’s uncontested finding that content validation was appropriate for determining the 1998 examination’s job relatedness. Second, M.O.C.H.A. asserts that the 1998 examination was not content related. See Guardians I, 630 F.2d at 97–98 (discussing content-relatedness requirement). In this sense, M.O.C.H.A. contends that the generic sub-tests were not reflective of a fire lieutenant’s tasks and, therefore, did not measure an applicant’s ability to serve in that position. The record shows, however, that in performing the job analysis, Steinberg conducted statewide surveys to determine the tasks a fire lieutenant performs and the SKAPs he would be expected to possess on the first day at that position. Based on the results of those surveys, Steinberg statistically ranked and linked the most critical tasks and SKAPs, which revealed that supervision, training, and understanding written material were related to the content of 32 the job. Cf. id. at 98 (describing process of identifying tasks and abilities in order to determine content relatedness). Further, based on those statistical results, consultation with the Fire Advisory Committee, and a comparison of nationwide test plans, Steinberg found that each sub-test area was important to the fire lieutenant position and should carry equal weight to ensure that the examination was content representative. Cf. id. at 99 (defining content-representativeness as proof that “the test measure[s] important aspects of the job, . . . but not that it measure all aspects, regardless of significance, in their exact proportions”). Having already determined that Steinberg’s underlying job analysis was suitable to the Buffalo fire lieutenant position, and in the absence of any evidence impugning Steinberg’s process of ranking and linking the task and SKAP surveys, we conclude that the evidence of Steinberg’s methodology was sufficient to support the district court’s findings that the 1998 examination was content related and representative. Finally, to the extent M.O.C.H.A. posits that questions on the generic sub-tests were unrelated to the sub-test areas, i.e., that questions on the supervision sub-test had nothing to do with supervision, no record evidence demonstrates such disjunction. Once Buffalo showed that the Civil Service Department used reasonably competent means to construct the 1998 examination, the district court was permitted to infer that the generic sub-test questions were, in fact, related to the generic sub-test areas, which had independently been shown to be content related and representative. Further, although M.O.C.H.A. received the 1998 examination questions in discovery, it never contested the relationship between test questions and the subject matters they were supposed to cover. In the absence of any evidence 33 undermining the inference that, based on the Civil Service Department’s reasonably competent test design, the 1998 examination’s questions were related to the areas they were supposed to test, the district court did not err in finding the examination job related and consistent with business necessity. Having identified no merit in M.O.C.H.A.’s various challenges to the finding of job relatedness, we conclude that the district court did not clearly err in determining that Buffalo had carried its burden at the second step of Title VII analysis. Accordingly, the district court properly entered judgment in favor of Buffalo on M.O.C.H.A.’s disparate impact challenge to the 1998 examination.14 14 Judge Kearse suggests that our affirmance of the district court’s ultimate job- relatedness finding will make it “virtually impossible” for municipalities to refuse to certify employment test results that have a proven disparate impact. See Dissenting Opinion, post at 5. Judge Kearse reasons that no municipality ever will have a “strong basis in evidence to believe it will be subject to disparate-impact liability,” the threshold necessary to discard test results and defend against a subsequent disparate treatment claim. Ricci v. DeStefano, 557 U.S. at 585. Our court has only begun to define this “strong basis in evidence” standard. See United States v. Brennan, 650 F.3d 65, 110–14 (2d Cir. 2011) (setting forth factors for determining whether municipality established “strong basis in evidence” defense, including whether there is “objectively strong evidence of non-job-relatedness,” which can be demonstrated by less than preponderance of evidence); id. at 144–45 (Raggi, J., concurring in judgment) (expressing doubt that “strong basis in evidence” can be satisfied by less-thanpreponderance showing). We have no occasion here to explore this issue further. We hold only that the district court, acting as fact finder after a bench trial, did not commit clear error in finding that a preponderance of the evidence showed that the 1998 examination was job related and consistent with business necessity. Whether such a relatively narrow and factdependent determination compels the broader legal conclusion that, at the time it certified the test results, the municipal employer lacked a “strong basis in evidence to believe it [was] subject to disparate-impact liability” is a question we leave for future courts to address. 34