Opinion ID: 766623
Heading Depth: 3
Heading Rank: 3

Heading: CBEST Test Validation Studies

Text: 104
105 Plaintiffs contend that the two pre-litigation studies of the CBEST, the 1982 Wheeler & Elias study and the 1985 Practitioner's Review study, failed to undertake any job analyses to identify work behaviors or job duties and thus could not have determined by professionally acceptable methods  whether the CBEST is predictive of or significantly correlated to any job related duties. Plaintiffs also challenge the skills identified in the 1995 Lundquist study, arguing that Dr. Lundquist erred in her methods for distinguishing skills which are important or critical from those which are neither. In a detailed and carefully reasoned opinion, the District Court found that all three studies adequately validated the CBEST. As we uphold the District Court's findings for the 1985 Practitioner's Review and the 1995 Lundquist study, we need focus only upon those two studies and we do not need to address the 1982 Wheeler & Elias study. 28 The 1985 Practitioners' Review 106 The Practitioners' Review was conducted by Dr. Richard Watkins in 1985. The District Court found that the Practitioners' Review in 1985 involved the `pooled judgments of informed persons such as . . . job incumbents' about the relevance of the skills tested on the CBEST to the jobs for which it is required, an appropriate formof job analysis under the professional standards of the time. AMAE II , 937 F. Supp. at 1419. 107 The Practitioners' Review polled 234 teachers, administrators, teacher educators, and other school employees, thirty-six percent of whom were members of minority groups. AMAE II, 937 F. Supp. at 1413. The participants took part in nine review panels, in which they judged the relevance of both the skills assessed by the CBEST and the test items themselves. Id. The educators were asked to rate the importance of skills according to fairly broad categories. For example, the three CBEST reading skills categories are: literal comprehension, logical comprehension, and critical comprehension. In mathematics, the three CBEST skills are problem solving skills, applied problem solution, and concepts and relationships. Id. Educators were asked to rate how relevant each of the skills tested would be to four jobs: a) elementary school teachers, b) secondary school teachers, c) pupil personnel and librarians, and d) school administrators. Id. These ratings ranged from not relevant to very relevant. Id. 108 The Practitioner's Review thus took steps to learn from teachers and administrators which categories of skills they would identify as relevant to their jobs. Moreover, although such steps might have run a risk of identifying skills at too abstract a level, the categories were defined in detail on the forms which panel members used to rate the categories. For example, the category Mathematical concepts and relationships was defined as follows: 109 Questions in this category test the understanding of basic concepts, such as the meaning of certain terms (area, for example), order among numbers, relation ships shown by graphs, elementary probability, and the like. Questions in this category may be from arithmetic, algebra, or elementary geometry. 110 It is true that the Practitioner's Review measured skills at a slightly higher level of abstraction than the Lundquist Study addressed below. Nevertheless, the educators who rated skill categories for job relevance were guided by the specific definitions provided for each category, and were instructed to rate the importance of the skills tested for teaching and nonteaching jobs. These steps are not clearly inadequate under the Craig test, that the employer must first specify the particular trait or characteristic and then determine that that particular trait or characteristic is an important element of work behavior. Craig, 626 F.2d at 662. Thus, the District Court did not clearly err when it found that the Practitioners' Review included an appropriate form of a job analysis under the professional standards of the time. 1995 Lundquist Study 111 As we noted above, Plaintiffs do not challenge Dr. Lundquist's methodology for identifying job-related skills. Plaintiffs do, however, challenge Dr. Lundquist's methods for distinguishing skills that are important or critical from those which are neither. 112 Dr. Lundquist's validation study polled experts and interviewed and observed teachers to develop a list of 59 activities, 39 reading skills, 27 writing skills, and 37 math skills used by teachers. AMAE II, 937 F. Supp. at 1414. Dr. Lundquist then surveyed 1,100 teachers and 230 administrators, asking each to rate the importance of those skills and activities on a four-point scale from 0 (not applicable) to 3 (critical). Id. 29 Skills and activities were retained only if at least 80 percent of the survey respondents rated the activity or skill as applicable to the job and the mean importance rating was 1.5 or higher on the importance scale. AMAE II, 937 F. Supp. at 1414-15. Applying that standard led tothe elimination of several job activities and skills, including 21 of 37 math skills. 113 Plaintiffs focus on the fact that skills were retained with only a mean rating of 1.5 on a scale that designated 1 as minor and 2 as important. Plaintiffs contend that using the rating of 1.5 thereby retained teacher skills and activities that are less important. The plaintiffs argue that, as a result, the Lundquist study violated the command reflected by Craig and by the Uniform Guidelines that only important or critical skills be tested. 114 The District Court disagreed with Plaintiffs, finding that Dr. Lundquist's decisions reflect manifestly reasonable professional judgments . . . . With respect to the 1.5 mean, as Dr. Lundquist testified at trial, a 1.5 rating rounds up to 2.0. It must be remembered that the mean rating of 1.5 was coupled with an 80 percent endorsement criterion, which is quite stringent. AMAE II, 937 F. Supp. at 1418 n.35. 115 We admit that it is possible to conjure up a scenario that illustrates the plaintiffs' concern. Assume, for example, that 80% of the teachers surveyed agreed that the skill of solving quadratic equations (skill QE) is relevant to the job of teaching, but that 75% of those surveyed rated the importance of skill QE as only minor. If the remaining 25% rated the skill as critical, the mean importance rating of skill QE would be calculated as 1.5 and therefore skill QE would be retained despite the fact that three-quarters of the educators considered it of minor importance. However, although this scenario is possible, it is clearly an extreme case and demonstrates that only under anomalous circumstances would an outlying question find its way into the CBEST. Plaintiffs have provided no evidence that any such outlying questions did, in fact, make its way into the CBEST. 30 116 In sum, although Dr. Lunquist's methodology is not free of any possible error, the mere possibility of error does not demonstrate that the District Court's findings were clearly erroneous. As we have previously observed, validation studies are by their nature difficult, expensive, time consuming and rarely, if ever, free of error. Cleghorn v. Herrington, 813 F.2d 992, 996 (9th Cir. 1987). Plaintiffs' challenge rests on the theoretical possibility that error could have existed in the writing and reading sections, and no such error has been shown. We conclude that the District Court did not commit clear error in upholding Dr. Lundquist's content validation study despite its possible retention of slightly less important skills on the reading or writing portions of the exam. 117
118 Title VII requires that a defendant using a test with a racially disparate adverse impact demonstrate that the test is job related for the position in question. 42 U.S.C. S 2000e2(k)(1)(A)(i). The Uniform Guidelines provide that[a]ny validity study should be based upon a review of information about the job for which the selection procedure is to be used. 29 C.F.R. S 1607.14(A). The 1985 Practitioners' Study 119 The 1985 study defined the jobs it sought to analyze as 1) elementary schoolteachers, 2) secondary school teachers, 3) librarians and pupil personnel (counselors and attendance officers), and 4) school administrators. All panel members were asked to judge the relevance of the CBEST skills by category for those jobs, ranking them from not relevant to very relevant. Although one might argue that the job classifications were too broad, we note that teachers are not infrequently called upon to teach outside of their area of primary expertise. As such, we cannot conclude that the District Court clearly erred in finding the job analysis in the Practitioners' Review to be sufficiently particularized. 1995 Lundquist Study 120 Dr. Lundquist identified teachers' job activities through several means, including literature reviews, observation of teachers on the job, and interviews with teachers regarding their job activities, their use of reading, writing, and mathematics skills, and other knowledge, skills and abilities used on the job. AMAE II, 937 F. Supp. at 1414. The Lundquist study began with an initial list of 103 specific reading, writing, and mathematics skills. From that list Lundquist created a survey in which 1,100 educators ranked the relevance and importance of those skills to the job of teaching. According to the District Court, [a] version of those survey for administrators was also created to determine whether they shared a common set of skill requirements with classroom teachers . . . . Based on her comparisons of teachers' and administrators' responses, Dr. Lundquist concluded that `all of the skills identified as important for teachers were also important for administrators.'  Id. at 1414-15. Dr. Lundquist's study reports: 121 Basic skill ratings were examined for administrators to determine if the same skill sets applied to both teacher and administrator jobs. Results showed all but one skill item (a math item) retained for teachers also applied to the administrator group. Thus the basic skill requirements identified for teachers were found to be job-related for administrators as well, and the same test specifications may be used to test basic skills for teachers and administrators. 122 Dr. Lundquist classified jobs as either teacher or administrator and determined that the CBEST, validated for teachers, was also valid for administrative positions. The District Court accepted this validation, concluding that the CBEST has been adequately validated with respect to nonteaching jobs. Id. at 1418. Reviewing the record, we find no clear error in that conclusion and we therefore uphold it. 123
124 Plaintiffs contend that no study has shown that the CBEST actually measures the job skills it purports to measure. [T]he employer must demonstrate by `professionally acceptable methods' that the selection device is `predictive of or significantly correlated' with the element of work behavior identified [by the job analysis]. Craig, 626 F.2d at 662 (quoting Albemarle Paper Co., 472 U.S. at 431). Plaintiffs claim that the District Court merely accepted the facial validity of the CBEST without any evidence in support when it concluded that the CBEST has been shown to be a valid measure of the basic skills tested. 125 The District Court did not explain the basis for its findings but unambiguously concluded that the CBEST actually measures . . . basic skills [in reading, writing, and mathematics] and repeated later that the skills tested on the CBEST are job-related, and that the CBEST has been shown to be a valid measure of those basic skills. AMAE II, 937 F. Supp. at 1411, 1419. 126 In Contreras, this Court held that a key requirement of [the] third step [of Craig], a requirement essential to the proof of job relatedness generally, is that the validation method be professionally acceptable.656 F.2d at 1282. There is evidence in the record from experts in the field to support the District Court's conclusions. Dr. William A. Mehrens, whose writings on test validity issues include a section titled Writing and validating the item reported that: 127 ETS personnel wrote some of the original items and assisted the test development committees in writing other items. ETS is well known and respected as a developer of standardized tests. They have well trained item writers and an impressive internal set of guidelines they follow with respect to item writing. 128 Asked whether the CBEST development [was] appropriate with respect to writing and evaluating the items,  Dr. Mehrens testified: 129 It has been. Many of the items came from an existing ETS pool. Others were written specifically for CBEST by members of the test development committee in concert with ETS test development specialists. The individuals on the committees 130 worked with specialists from ETS to further develop and define the content specifications, to review an existing ETS test item pool, to write new test items, and to review the items submitted by fellow committee members. In addition, the committees studied all of the data from the field testing, made recommendations for revisions as they felt necessary, and reviewed all of the final test results. (citations omitted). 131 Even if the test items are expertly developed, they still need to be matched to skills shown to be important to the job via a job analysis. The District Court described the Curriculum Matching project, in which two ETS employees . . . matched CBEST test specifications to material found in textbooks purportedly used in California public schools. AMAE II, 937 F. Supp. at 1412 n.21. The District Court criticized the study as unscientific and not particularly helpful but noted that the study did support the overall conclusion that the kinds of skills tested on the CBEST can be found in elementary and secondary school textbooks. Id. 132 Because there was some evidence that the CBEST development was appropriate with respect to writing and evaluating the items, the District Court did not clearly err in finding that the test items had been shown by professionally acceptable methods to be predictive of or significantly correlated with elements of work behavior. 133
The Uniform Guidelines provide: 134 Where cutoff scores are used, they should normally be set so as to be reasonable and consistent with normal expectations of acceptable proficiency within the work force. 29 C.F.R.S 1607.5(H). We have previously applied that standard in our own cases. See, e.g., Craig, 626 F.2d at 6655 (upholding a passing score cutoff as reasonable and consistent with normal expectations of performance in the . . . program). 135 Courts reviewing cutoff scores on exams with a disparate impact frequently quote from a thorough discussion that appears in the Second Circuit's decision in Guardians Ass'n of New York City Police Dept., Inc. v. Civil Service Commission of City of New York, 630 F.2d 79, 105 (2d Cir. 1980): 136 When a cutoff score unrelated to job performance produces disparate racial results, Title VII is violated. Consequently, there should generally be some independent basis for choosing the cutoff . . . .[A] criterion-related study is not necessarily required; the employer might establish a valid cutoff score by using a professional estimate of the requisite ability levels, or, at the very least, by analyzing the test results to locate a logical break-point in the distribution of scores. 137 (internal citations omitted). 138 In Guardians, the Second Circuit rejected a cutoff score when the employermerely chose as many candidates as it needed, and then set the cutoff score so that the remaining candidates would fail. Id. The cutoff was derived solely from the employer's staffing needs, without any reference to the job requirements. 139 Plaintiffs allege that the District Court erred in upholding a twelve-of-sixteen cutoff score on the reading component of the CBEST when an empirical study demonstrates that the proper, job-related passing score on the CBEST writing subtest is nine or ten. 140 The District Court concluded that the passing scores on the CBEST reflect reasonable judgments about the minimum level of basic skills competence that should be required of teachers. AMAE II, 937 F. Supp. at 1420. The District Court described in detail the process used to develop passing score cutoffs for the original CBEST. The District Court found that then-California Superintendent of Education Bill Honig, who had the ultimate responsibility for setting the score cutoffs, had before him the results of the Wheeler and Elias study, the results of the field test, and the recommendations of the [Commission] and the [CBEST] Advisory Board. Id. at 1421. 141 Superintendent Honig testified that he did consider both the results of the Wheeler and Elias study and the [Commission]'s advice when he made his decision, though he ultimately disagreed with both recommendations. Id. at 1422. The data show that almost twenty percent of educators polled either thought a score of eleven-out-of-sixteen was not passing or were not certain. The Superintendent located a logical breaking point based upon professional estimates when he established the bottom passing score at twelve out of sixteen instead, which all readers agreed was a passing score. In the terminology of Guardians, the Superintendent concluded that twelve of sixteen was a logical break point.  As such, District Court did not clearly err in finding Superintendent Honig's score cutoffs were reasonable. 142 E. The District Court's Appointment of a Technical Advisor 143 Plaintiffs claim that the trial court's rulings were procedurally tainted by the unreported influence of Dr. Stephen Klein, a RAND Corporation scientist who advised the District Court through exparte communications with the judge but was not appointed by the court as an expert witness, was not subject to cross-examination, and did not supply an expert's report. 144 According to the Order Appointing Technical Advisor, the District Court appointed Dr. Klein as a technical advisor, not as an expert witness. Courts have inherent authority to appoint technical advisors, although such appointments should be the exception and not the rule, and should be reserved for truly extraordinary cases where the introduction of outside skills and expertise, not possess by the judge, will hasten . . . just adjudication. Reilly v. United States, 863 F.2d 149, 156 (1st Cir. 1988). Recently, in General Electric Co. v. Joiner, Justice Breyer wrote separately to endorse such a practice: 145 I therefore want specially to note that, as cases presenting significant science-related issues have increased in number . . . judges have increasingly found in the Rules of Evidence and Civil Procedure ways to help them overcome the inherent difficulty of making determinations about complicated scientific or otherwise technical evidence. Among these techniques are . . . the appointment of special masters and specially trained law clerks. 146 118 S.Ct. 512, 520 (1997) (Breyer, J., concurring). 147 This case involves the highly technical field of psychometrics, and presents problems of unusual complexity beyond the normal questions of fact and law with which judges routinely grapple. Under the approach identified by Justice Breyerand applied by the First Circuit in Reilley, this case is a fitting case for the use of a technical advisor. The District Court invited the parties to object to Dr. Klein's appointment and plaintiffs did not do so. 148 The District Court advised the parties that, consistent with its usual practice, it expected to put Dr. Klein on the stand towards the end of the trial, ask him his views, and permit the parties to cross-examine him. Had the District Court done so, Dr. Klein would have become an expert witness and his testimony would therefore have become a part of the evidence considered by the Court in its role as the finder of fact. In addition, under Fed. R. Evid. 706(a), the parties could have cross-examined Dr. Klein as a matter of right to test this evidence. 149 However, the District Court did not call Dr. Klein, and there is no evidence that the District Court relied on Dr. Klein's opinions as a source of evidence. To the extent that the District Court relied on Dr. Klein to assist the Court in understanding the evidence submitted by the parties, Dr. Klein remained within the role of a technical advisor as defined in Reilly, similar to the role of a specialized law clerk. 31 Because Rule 706 does not apply to technical advisors, Reilly, 863 F.2d at 155, the District Court no more erred in refusing cross examination of Dr. Klein than it would have erred in refusing cross examination of its law clerks. Likewise, the District Court did not err in not requiring that Dr. Klein furnish an expert's report. 150 F. The District Court's Denial of Costs to the Defendants 151 1. This Court Reviews the Denial of Costs For An Abuse of Discretion 152 Pursuant to Rule 54(d)(1) of the Federal Rules of Civil Procedure, costs shall be allowed as a matter of course to the prevailing party unless the court directs otherwise.  F.R.C.P. 54(d)(1). Rule 54 thus creates a presumption in favor of awarding costs to the prevailing party. National Info. Servs., Inc. v. TRW, 51 F.3d 1470, 1471 (9th Cir. 1996). The District Court's denial of costs is reviewed by this Court for an abuse of discretion. United States ex rel. Newsham v. Lockheed Missiles & Space Co., Inc., 171 F.3d 1208, 1213 (9th Cir. 1999). 153 2. The District Court Abused Its Discretion When It Entirely Denied Costs to the Defendants 154 Although the decision to award costs resides within the discretion of the district court, a court generally must award costs unless the prevailing party is guilty of some fault, misconduct, or default worthy of punishment. National Info. Servs., 51 F.3d at 1472. 155 In this case, the District Court explicitly found that the Defendants had not committed any misconduct that would require a denial of costs. Order filed Feb. 12, 1997 at P2 (Order). Nevertheless, the District Court observed that our holding that a court generally must award costs implied that there were exceptional cases in which the court might deny costs. Id. (Emphasis added in original Order). In this case, the District Court found that the suit involved an issue of substantial public importance, that the issues were reasonably close and difficult, and that there was some merit to the Plaintiffs' case. Order, PP 3-4, 6-7. The District Court also noted the substantial disparity of resources between the resources of the parties. Order, P 5 (citing National Org. for Woman v. Bank of California, 680 F.2d 1291, 1294 (9th Cir. 1982) and Moore v. Hughes Helicopters, Inc., 708 F.2d 475, 478 (9th Cir. 1983)). On this basis, the District Court concluded that it was withinits discretion to completely deny costs to Defendants, and thereupon did so. Order, P 8. 156 We agree with the District Court that our precedent allows a court to consider the limited resources of a Title VII plaintiff when assessing costs. Moore, 708 F.2d at 478; Wrighten v. Metropolitan Hosp., Inc., 726 F.2d 1346, 1358 (9th Cir. 1984). However, these precedents must be read in light of our more recent decision in National Info. Servs. We held in National Info. Servs. that the denial of costs is by nature a penalty, 51 F.3d at 1472, and that such punitive measures cannot be imposed absent wrong-doing because a district court cannot punish a party by denying costs unless the party has done something to deserve it. Id. Congress has included nothing in the language of Title VII that would justify the imposition of such a penalty on an exonerated defendant. As a consequence, we conclude that a court may consider the limited resources of a Title VII plaintiff, but may not entirely deny costs to a prevailing defendant as a result. 157 We also conclude that there is no exception to Rule 54(d) that entitles a court to punish a defendant simply because the case involves an issue of national importance. The District Court relied heavily upon a sliver of language contained in two Seventh Circuit cases, which both mentioned landmark cases of national importance. Delta Airlines, Inc. v. Colbert, 692 F.2d 489, 490 (7th Cir. 1982) and Popeil Bros., Inc. v. Schick Elec., Inc., 516 F.2d 772, 776 (7th Cir. 1975). However, neither case suggested that an exception to Rule 54(d) in fact existed for landmark cases; rather, both cases expressly declined to consider the possibility of such an exception because neither case could have been considered a landmark case of national importance. Delta Airlines, 692 F.2d at 490; Popeil Bros., 516 at. 776. One could not even characterize as dicta the Seventh Circuit's position on this point, because the Seventh Circuit simply has not expressed any opinion. Turning to our own cases and to the text of Rule 54(d), we find nothing that would support such an exception, and we explicitly hold today that there is no such general exception to Rule 54(d) beyond the exception for civil rights plaintiffs. 32 158 Turning to the District Court's remaining grounds for the denial of costs, we observe that National Info. Servs. specifically rejected the denial of costs on the basis that the plaintiff's position had  `merit.'  National Info. Servs., 51 F.3d at 1471. It also specifically held that  `difficulty' alone does not justify penalizing the prevailing parties. Id. at 1473. These factors should not be considered in the assessment of costs and it was an abuse of discretion for the District Court to consider those factors. 159 Finally, none of the precedents relied upon by the District Court involved thecomplete denial of costs to a prevailing party, 33 and to the extent that they might suggest a complete denial of costs, such a suggestion would not survive our holding in National Info. Servs. As with the defendants in National Info. Servs., there is no impropriety on the part of the defendants in this case that would justify denying them their due costs under Rule 54(d)(1). Id. We therefore remand the case to the District Court for a determination of the proper costs.