Opinion ID: 1450062
Heading Depth: 3
Heading Rank: 1

Heading: Grading Procedures.

Text: Bettine argues that the ABA's grading procedure is fraught with arithmetic errors which will in many instances yield incorrect test scores. The crux of his argument is that it is error for the ABA to assign him scores with two significant figures (1.0, 1.5, 2.0, etc.), when the graders are only allowed to assign scores with one significant figure (1, 2, 3, etc.). He reasons: If respondent desires to average the applicant's essay raw score to within two significant digits of accuracy then the scores assigned by the examiners must reflect at least two significant digits of accuracy. Rounding off essay raw scores to a single significant digit of accuracy and then using these scores to calculate an applicant's average score to 0.5 points of accuracy is an incorrect and unacceptable mathematical procedure. By rounding off the raw scores during intermediate stages of computations to single digit values, respondent is in fact robbing Mr. Bettine out of a possible 0.5 points per essay question. If respondent desires to employ a grading policy which requires examiners to grade with a single significant digit of accuracy, then respondent at a minimum must implement a grading policy which assigns the higher of the two scores as the score for any question where the examiners disagree by one point. This policy would help to minimize the affect [sic] of round off error on an applicant's overall essay raw score, because an applicant would receive the benefit of the higher score for those questions where examiners are now forced to round off during intermediate stages of computations. It is also important to recognize that although the examiners assigns [sic] scores at the essay raw score level accurate to but a single significant digit, the weighted essay scores for Part A, B and C of the essay exam in addition to the applicant's combined score, are calculated to five significant digits of accuracy! It is not possible to start a series of calculations containing a number accurate to but a single significant digit and arrive at an answer with five significant digits of accuracy. Bettine's Exhibit 1 illustrates that a variation of 0.5 from the raw essay scores can have a great effect on the converted essay score. To eliminate the potentially harmful effects of the ABA's faulty grading procedure on him, he argues, his score should either be rounded off to two significant digits, or 0.5 should be added to Parts A, B and C of his essay raw scores. In either case, he argues, he would achieve a passing score of 140 points. Bettine also argues that, in addition to the round off errors described above, observation errors result when an applicant's answer does not precisely coincide with a benchmark answer, since examiners must adjust the true score of an essay to one of the integer set values, adding or discarding fractional points in the process. He writes that [t]his adjustment will be made either unintentionally by the examiner because of observation error or purposely because the examiner will consciously round off applicant's true score to one of the allowable integer values. Bettine apparently believes that the harmful effects of this round off error would be lessened if examiners were not restricted to assigning only one of five test score values. Bettine posits that the effect of the round off and observation errors is cumulative. He therefore claims that, while the grading error associated with a single question may not be significant, one must assume that all ten of his essay scores were reduced by 0.5 point of grading error in order to determine the maximum effect on his combined score of the cumulative grading errors. His Exhibit 2 shows that 5 points (0.5 points per question times 10 questions) at the raw score level are worth approximately 5.5 points at the combined essay score level. Exhibit 2 also shows that Bettine needs to accumulate only one additional point at the essay raw score level to achieve a combined passing score of 140 points. In response to Bettine's arguments, the ABA relies primarily on the affidavit of Dr. Stephen Klein, a nationally recognized expert on bar examinations. Dr. Klein does not challenge the mathematical foundations of Bettine's arguments concerning measurement error and significant figures. He instead argues that the principles of numerical analysis, upon which Bettine relies in making his criticisms of the grading procedures, are inapplicable to psychological measurements such as the grading of exams. He testified: 6. Bettine's arguments for changing the score intervals stems from his failure to recognize critical differences between physical and psychological measurement. Physical measurement is used to assess the height, weight, temperature, or other properties of physical things. These measurements are made with rulers, scales, thermometers, Geiger counters, and other mechanical devices. Such measurements can be made with more precision by using an instrument that is marked off in finer gradations, such as by replacing a ruler with a micrometer. 7. In contrast, human judgment is needed to assess the relative quality of a figure skater's performance or a candidate's answer to a bar exam question. Using more score levels (such as by having readers assign grades in half-point intervals) does not increase precision unless readers can reliably distinguish between the quality of answers in adjacent levels. Readers should use only as many score levels as are truly needed to distinguish between answers of different quality. Klein notes that most states use about five score levels to grade bar exam essay answers, since [t]hey have found this is about the right number to reflect real differences in answer quality, but not so many as to force readers to make distinctions where there are no reliably identifiable differences. After reviewing the arguments of Bettine and the ABA, we have concluded that Bettine has failed to make factual allegations sufficient to establish an abuse of discretion or improper conduct on the part of the ABA. Although Bettine's criticisms of the mathematical foundations of the grading procedure are convincing from a purely mathematical standpoint, they are undercut by Dr. Klein's statements that it is inappropriate to strictly apply numerical analysis to psychometric measurements. In view of Dr. Klein's affidavit, it cannot be said that Bettine has satisfied the heavy burden of showing an abuse of discretion or improper conduct on the part of the law examiners. Similarly, Bettine's presentation is insufficient to demonstrate that the ABA's grading procedures violated his equal protection or due process rights.