Opinion ID: 805464
Heading Depth: 4
Heading Rank: 1

Heading: Test Development

Text: In December 1997, Buffalo asked the New York State Civil Service Department (“Civil Service Department”), to create a promotional examination for the position of fire lieutenant in its fire department. See N.Y. Civ. Serv. Law § 23(2) (establishing that Civil Service Department shall prepare employment examinations for municipalities upon request). It was then standard practice for Buffalo to rely on the Civil Service Department for 2 In reaching our conclusion that the district court did not clearly err in its jobrelatedness finding, we note that Judge Curtin’s experience with discrimination within the Buffalo Fire Department spans almost thirty-five years. See, e.g., United States v. City of Buffalo, 457 F. Supp. 612 (W.D.N.Y. 1978) (holding that Buffalo Fire Department engaged in pattern or practice of hiring and promotion discrimination against African Americans, Latinos, and women), aff’d, 633 F.2d 643 (2d Cir. 1980); United States v. City of Buffalo, 721 F. Supp. 463 (W.D.N.Y. 1989) (modifying injunction imposed to remedy Buffalo Fire Department’s discriminatory hiring and promotion practices). 5 examinations for municipal civil service positions rather than to devise its own tests.3 In making the request, Buffalo provided the Civil Service Department with its fire department’s most recent job specifications for the fire lieutenant position, the position’s anticipated salary level, and promotion eligibility criteria.4 3 Buffalo no longer relies on the Civil Service Department for its promotional examinations, and instead uses private contractors through a bidding process. 4 The job specifications Buffalo provided for the job of fire lieutenant stated as follows: DISTINGUISHING FEATURES OF THE CLASS This is a first line supervisory position where incumbents are responsible for the activities of a fire company during an assigned shift. Responsibilities include directing the work of Firefighters at fires and in fire stations, evaluating their work performance and instructing them in new approved firefighting methods. Work is performed in accordance with established procedures and policies as outlined by the Fire Department. The class of Fire Lieutenant is distinguished from that of Fire Captain in that the latter is in charge when both he and the Fire Lieutenant are on duty. The Fire Lieutenant has complete charge of the activities of the fire company on all shifts and is in charge of operations at the scene of a fire in the absence of or pending arrival of a superior officer. All work is performed under general departmental regulations and incumbents directly supervise Firefighters under their command. TYPICAL WORK ACTIVITIES • Responds to all alarms assigned to his company while on duty; • Directs the work of firefighters at scenes of fire and in station house; • Assigns firefighters to lay out and connect hose lines and nozzles, direct hose streams, raise ladders and ventilate buildings; • Inspects property at scene of fire to prevent re-ignition; • Supervises the cleaning, checking and replacement of tools and equipment after a fire; • Inspects personnel, station house, buildings, grounds and facilities to 6 Dr. Wendy Steinberg, an associate personnel examiner with the Civil Service Department, created the “Lower Level Fire Promotion” test series that was provided in response to Buffalo’s request. Steinberg testified that this test series was devised for use in promoting candidates to various firefighting positions, including fire lieutenant, in fire departments across New York. Indeed, municipalities across New York used the test series for that purpose. To create the Lower Level Fire Promotion test series, Steinberg spent three years, from 1994 to 1997, conducting a job analysis of firefighters of all ranks from fire departments across New York. Based on this analysis, she designed examinations for each titled position. Steinberg’s job analysis had a dual focus: (1) the tasks firefighters perform and (2) the skills, knowledge, abilities, and personal characteristics (“SKAPs”) a person would be expected to possess on the very first day in a particular position. Steinberg testified as to how her job analysis was consistent with the joint standards of employment test design ensure conformity with departmental rules and regulations; • Examines fire trucks and equipment such as ladders and hose to ensure proper order and condition; • Inspects buildings and premises for fire hazards; • Personally supervises a wide variety of cleaning and maintenance tasks performed at the station; • Maintains discipline; • Makes periodic reports of personnel and activities; • Performs related duties as required. J.A. 954, Dkt. No. 11-2184-cv. The salary was to be $49,769, and applicants needed at least three years’ firefighting experience or one year’s experience as an assistant fire alarm dispatcher. 7 published by the American Psychological Association, the American Education Research Association, and the National Council on Measurement in Education, as well as with guidelines promulgated by the Equal Employment Opportunity Commission (“EEOC”). In beginning her job analysis in 1994, Steinberg first collected job specifications from various New York fire departments for each submitted firefighting job title. From these specifications, she assembled a list of 190 identified tasks performed by firefighters of various ranks, which she reviewed with members of the Civil Service Department’s Fire Advisory Committee (“Fire Advisory Committee”), a panel of experts on the administration of fire departments. Steinberg then used the task list to create a statewide survey that, in April 1995, asked firefighters to rank each identified task according to how critical it was to the performance of firefighters’ specific jobs within their departments. With the Fire Advisory Committee’s assistance, Steinberg also created a second survey that, in October 1996, asked firefighters to rank listed SKAPs based on how critical they were to the respondents’ particular positions. Besides identifying the skills, knowledge, abilities, and personal characteristics necessary to perform the responsibilities of a specific job title, this survey was intended to provide a cross-reference for data obtained from the task survey. The task and SKAP surveys were sent to every incumbent firefighter in New York, with the exception of those serving in New York City and Rochester.5 Steinberg followed 5 New York City’s and Rochester’s fire departments were not surveyed because they do not use examinations created by the Civil Service Department, but instead create their own tests internally. In the end, however, Steinberg did compare her Lower Level Fire Promotion test series plan against New York City’s and Rochester’s, as part of an effort to 8 up with non-responsive survey recipients in order to maximize the data obtained. In the end, of 5,934 task surveys sent out, 2,502 responses were received. Of those surveys, 2,994 were sent to firefighters serving in the state’s twelve largest jurisdictions (other than New York City and Rochester),6 from whom 1,218 responses were received. Seven hundred ninety-five of the surveys were sent to firefighters holding fire lieutenant positions, with 316 responding. The responses in all three of these categories exceeded the numbers required by accepted statistical methodologies to establish 95% confidence in the survey results.7 Similarly, of 5,934 SKAP surveys sent, 1,604 responses were received, a number also sufficient to obtain 95% statistical confidence.8 By contrast, of 833 task surveys sent to Buffalo firefighters, cross-validate the plan with fire lieutenant test plans from large jurisdictions across the country. See infra at 11 & n.9. 6 These twelve largest fire departments, in descending order of size, were Buffalo, Syracuse, Yonkers, Albany, Utica, White Plains, Troy, Binghamton, New Rochelle, Niagara Falls, Schenectady, and Mount Vernon. 7 Although Steinberg did not define 95% statistical confidence, we understand her to mean that there was a 95% probability that the survey results were not random, which makes it highly unlikely that they were the result of chance. See, e.g., Smith v. Xerox Corp., 196 F.3d 358, 366 (2d Cir. 1999), abrogation on other grounds recognized by Meacham v. Knolls Atomic Power Lab., 461 F.3d 134 (2d Cir. 2006); see generally Matrixx Initiatives, Inc. v. Siracusano, 131 S. Ct. 1309, 1319 n.6 (2011) (defining statistically significant results as those that are unlikely to be result of random error); Hans Zeisel & David Kaye, Prove It with Figures: Empirical Methods in Law and Litigation 85–88 (1997) (same). 8 Although Steinberg did not provide a source for her statistical confidence rates, M.O.C.H.A. has not challenged the accuracy or validity of her statistical calculations and conclusions, thereby waiving any such argument. See Norton v. Sam’s Club, 145 F.3d 114, 117 (2d Cir. 1998) (“Issues not sufficiently argued in the briefs are considered waived and normally will not be addressed on appeal.”). Rather, M.O.C.H.A. challenges only whether, as a matter of law, the district court could find that the statewide job analysis was suitable to the Buffalo Fire Department in the absence of other direct evidence. 9 only 68 responses were received, a number too low for the results to be reliable by themselves. Further, no Buffalo firefighter responded to the SKAP survey. Upon receipt of survey responses from across New York, Steinberg grouped together the tasks identified as most critical to each firefighter title, including fire lieutenant. She performed a similar analysis of the SKAP survey responses and, with the assistance of other Civil Service Department staff, linked the most highly ranked SKAPs to corresponding highly ranked tasks. Steinberg then asked the Fire Advisory Committee to review and confirm the links drawn. This process ultimately yielded six sets of task and SKAP categories that became the sub-test areas for the challenged fire lieutenant promotional examinations: (1) fire attack and suppression, (2) fire prevention, (3) rescue and first response, (4) understanding and interpreting written material, (5) training practices, and (6) supervision. These sub-test areas were approved by the Fire Advisory Committee. Despite Buffalo’s low response rate to the task and SKAP surveys, Steinberg determined that her statewide analysis was properly relied on in responding to Buffalo’s request for a fire lieutenant promotional examination because of the overall consistency in the task and SKAP rankings of fire lieutenant respondents across New York. Despite the fact that responding fire lieutenants worked in different jurisdictions and in fire departments of varying sizes, Steinberg found a 90% correlation in the tasks they identified as critical to their job, in contrast to responses received from firefighters in other high-ranking positions, which showed more variance. Steinberg’s conclusion that common tasks and SKAPs were 10 critical to the job of fire lieutenant across New York was buttressed by (1) the Fire Advisory Committee’s review and approval of the tasks and SKAPs that she had identified for testing in a fire lieutenant examination, and (2) fire lieutenant promotional test plans from fourteen large fire departments across the United States, which were entirely consistent with the fire lieutenant test plan that Steinberg developed for general use by New York fire departments.9 Steinberg testified that obtaining such expert advice and cross-validating test plans against those of other jurisdictions are acceptable procedures under the joint standards of employment test design. In addition, Steinberg invited subject matter experts from each of New York’s fire departments to meet with her to discuss the questions to be included in the examination’s sub-tests relating to fire lieutenants’ firefighting tasks, i.e., the sub-tests assessing an applicants’ fire attack and suppression, fire prevention, and rescue and first responder knowledge. The Buffalo Fire Department did not accept this invitation. Nevertheless, the multiple-choice questions that emerged from these discussions were then reviewed and approved by the Fire Advisory Committee. Multiple-choice questions for the remaining general sub-tests, i.e., understanding and interpreting written material, training practices, and supervision, were written by another Civil Service Department unit responsible for drafting 9 Those fourteen jurisdictions were: Los Angeles, California; San Jose, California; Denver City and County, Colorado; Dade County, Florida; Fort Lauderdale, Florida; St. Petersburg, Florida; Chicago, Illinois; Baltimore, Maryland; Prince George’s County, Maryland; Reno, Nevada; New York, New York; Rochester, New York; and the District of Columbia. 11 general questions appearing on employment examinations across government agencies. Each sub-test carried the same weight in a candidate’s final score, and Steinberg set the passing score at 66 correct answers out of 105 questions, which was lower than the maximum passing score of 73 under state law.