Opinion ID: 521944
Heading Depth: 1
Heading Rank: 4

Heading: the validity study issue

Text: 28 Appellants contend that the validity study conducted by McCann Associates, Inc. was flawed in two respects: 1. The inconsistency between supervisory ratings and the fire company officer test and 2. The inconsistency of ratings given by shift commanders and those given by battalion chiefs. (App.Br. pp. 14 and 16.) Furthermore, appellants criticize the fifteen-month delay between the administration of the exam and the supervisory ratings. The district court found that the evidence submitted by appellants did not support their contentions. 29 Dr. Stephen Cole, appearing on behalf of the appellants, testified at the district court hearing that he had observed inconsistencies between the test scores of the participating fire captains and the ratings given those officers by their respective shift commanders and battalion chiefs. Dr. Cole stated: 30 The point here is that there is very little correspondence with those captains between the test and the supervisory ratings. For those of correlation .33, there is very little association between how well you did on the test and how well the supervisors evaluated your job performance. Of the top 25 candidates identified by supervisors, only seven of them would be identified in the top 25 if the test alone was used. (R. 22-180). 31 Dr. Cole also pointed out that the person who ranked first on the written exam was ranked 31st by the supervisors; the person who ranked second on the written exam was ranked 68th by the supervisors; and the person who ranked 20th on the written exam was ranked 79th by the supervisors. Dr. Cole's conclusion was that ... clearly when used as the sole criterion for promotion, the test does not act as a good predictor. (R. 22-183). 32 Dr. Cole is not an Industrial Psychologist and this case apparently offered him his first opportunity to analyze a Criterion Measure Performance Rating (CMPR) system of testing for job skills and knowledge. His testimony that the ratings of the subject for job skills did not correlate with their test scores is seriously flawed. Dr. Cole relied on his (plaintiff's) Exhibit 13 which listed (not by name) the 89 subjects, the ratings given by their supervisors, and their test scores. His testimony to the district court concerned only 25 of the subjects which he picked out because their test scores differed greatly from their ratings. For example, the subject who was ranked at the bottom, 89th, was 5th from the top on test scoring. Cole used him and others to contend that the study results were skewed. 33 It is elementary statistical knowledge that the smaller the sample the more unreliable the result. McCann Associates admitted concern that having only 89 subjects was not as large a group as would be preferred. Dr. Cole shrank the group from 89 to 25 in order to reach a conclusion that the study did not accurately predict that the test would result in demonstrating which firemen would make the best fire lieutenants. The district court correctly rejected Dr. Cole's unscientific approach to the issue. 34 One of the City's experts was Dr. Chester Palmer, a professor of mathematics at Auburn University who had experience in the field of statistical analyses of employee testing for Title VII purposes and had clients such as the United States Navy, United States Air Force, and the personnel Departments of the States of Alabama and Georgia. Dr. Palmer concluded that the McCann study and tests met professional standards. 35 Dr. Palmer conducted five separate analyses of the validity of the CMPR system used by McCann. Dr. Palmer ran these analyses in an attempt to equalize--that is, to take out the extraordinary influences caused by certain factors, such as a very lenient supervisor/rater who scored everyone rather high or such as the captain (ratee) who was rated 89th by the raters but scored fifth on the test. (This person was rated by only one supervisor rather than the usual two or three, which may account in part for the deviation.) Dr. Palmer's term for such persons was an outlier. Others might use the term sport or deviant. Dr. Palmer discussed the importance of such factors in making an analysis: 36 One of the problems with this kind of study with small m is that one person can swing that correlation a large amount. So I tried just to see what would happen if I went back and I ran those charts of the correlations again but I left him out. Okay. Now, I must emphasize I don't have any justifiable reason for leaving him out other than the fact that it looks strange. Okay. That is, I'm not saying that the right thing to do is to leave him out. I can't say that but when you see one person who is that wild, it only makes sense to see what would happen otherwise. 37 The answer turns out to be quite different. Now, we are back to the back of the chart. In the back of the chart is three more of the correlation tables. You can see in the title that these are labeled in the third line of the title, 88 subjects, one outlier removed. The first one is what the correlations would be for the combined group if you did not include that individual. And we see that now if you read down under the total column we now have two of them that are almost .4. We've got one that's 393 and one that's 398. 38 So that essentially this one individual will change the correlation from .4 to .3. That's what happens in small groups when there's one person who is sort of off the wall. 39 Q. I haven't found the .4 and the .3? 40 A. I'm sorry. The .3 is not in this table. It's from the previous one. But if you read under the column that says total, if you read down for example to the one that says, criterion standard, 393, that's what McCann would have gotten if they hadn't had that one individual, if they just had the other 88 people, and the others are what the other ones would have been. They are all higher than they were before. They are all significant at the one percent level. 41 Now, the next two charts are the same thing as they were before. One set for the blacks, one set for the whites. It happens that this one individual is black so that the chart for the whites is going to be the same as it was before. They are the same people. But the chart for the blacks is different, and if you now look under total you see that all of a sudden they have gone way up. 42 For example, for the McCann standardization, the one I have labeled crit std it has now gone all way up to .45. It is now significant past the one percent level. What's happening is that the only reason that that isn't significant for blacks separately is this one person. Now, again, I have to emphasize I don't think you can just throw him away because I don't have a legitimate reason to say that's an error of some kind. But the fact is that actually the correlation would be higher for blacks than for whites if it hadn't been for one person. And that's something that I think kind of important in interpreting results. 43 R. 23 at 337-39. 44 The second issue raised by the appellants deals with the supervisory ratings given by the shift commanders and the battalion chiefs. Dr. Cole criticized these ratings on the ground that inconsistencies existed between the ratings given particular subjects by their respective shift commanders and those given by their battalion chiefs. Some fire captains were ranked at a particular level by their shift commander and ranked considerably lower by their battalion chief. (R. 22-164). Dr. Cole concluded that the supervisors were using the rating system differently, despite McCann's efforts to achieve uniformity. 2 45 The City of Atlanta responded to this criticism by pointing out that McCann had employed a standardization procedure to correct the inconsistencies between the ratings. Dr. Palmer testified that McCann had made adjustments to the ratings so that the rankings given by each supervisor were uniform. McCann examined each supervisor's rankings to determine what that shift commander or battalion chief considered to be average job performance. Those rankings were then statistically adjusted to conform with a uniform standard of average job performance based on the rankings of all supervisors. The rankings of those individuals who the supervisor felt rated above or below average were similarly adjusted to reflect the collective average. Dr. Palmer agreed that standardization was the proper procedure in this instance, but stated: 46 The next question is how should you standardize and the problem I have is that there's no professionally recognized formula for how to do this. There are different ways you could set up the way the numbers work. I don't have any particular objection to what McCann did but I would be very unhappy if the study yielded a significant correlation their way and it didn't come out any other way because I'm not sure their way is the best. So I went back and did it a bunch of different ways, ... (R. 22-297). 47 To be precise, Dr. Palmer performed five different standardization procedures. 3 Dr. Palmer testified that each of the five standardization procedures yielded virtually identical results. He stated: 48 Now, I'm pleased to say after all this--and we can look at the numbers, if you like, but it didn't make any difference. That's what it comes right down to. Actually, it made very little difference which of these you used. Including the raw score, which I thought was amazing but I ran all my analyses, you know, when I did the correlations to see if things were significant, I ran all my analyses five times. Once for each of those five ways of giving the number to see if it made any difference how you gave the number, and it really didn't. (R. 22-303). 49 Dr. Palmer expressed confidence in the validity of the supervisory ratings and their correlation with the test scores, stating: 50 That's one reason that I have faith that the numbers--the correlations reported by McCann Associates are reasonable because when I did some of these things that were totally different I cam out with almost the same thing. That's a good reason to believe that the numbers are reasonable, that they haven't just concocted a method that gives big numbers. (R. 22-303). 51 In sum, the City of Atlanta maintains that the Guideline requirements regarding the establishment of validity have been met; therefore the reliability of the criteria measurements are irrelevant. Mr. William F. Howeth testified that the exam known as Form A was found to have a correlation coefficient of + .33. 4 (R. 20-38). The statistical significance was demonstrated to be p 52 The final criticism raised by the appellants is the possibility that the fifteen-month delay between the administration of the exam and the supervisory ratings may have led to contamination. McCann allowed any fire captain participating in the validity study to learn his score on the written examination by submitting a self-addressed envelope along with his answer sheet at the completion of the test. The exam was given in March 1981. The test scores were not revealed to anyone other than the individual who had taken the exam. Appellants suggest that the shift commanders and battalion chiefs may have learned the test scores of the fire captains under their supervision. When the supervisors ranked these fire captains in June 1982, appellants maintain that knowledge of these scores may have led the shift commanders and battalion chiefs to rate individuals in accordance with their performance on the exam. Appellants point out that the Guidelines specifically warn against this form of contamination. Proper safeguards should be taken to insure the scores on selection procedures do not enter into any judgments of employee adequacy that are to be used as criterion measures. 29 C.F.R. Sec. 1607.14(B)(3). Beyond raising this possibility, the appellants offer no evidence of any actual contamination. 53 In the course of cross-examination, Dr. Palmer testified that while there were potential problems with the fifteen-month delay, McCann had taken steps to avoid the possibility of contamination. He stated: There is a possibility that the supervisors who were doing the rating might have known the scores of those people that are being rated. That is something you would normally try to prevent. (R. 23-407). Documentary evidence submitted by the City of Atlanta indicated that McCann considered the possibility of contamination and rejected it based on their analysis of the circumstances leading to the second set of performance ratings in June 1982. In addition, McCann engaged an independent psychologist to make a complete statistical study of the ratings data to determine if any evidence of contamination existed. The psychologist concluded: I find no evidence of contamination in the second set of ratings and I believe it is unlikely that such contamination occurred. 5 54 As mentioned in the beginning of this opinion, the Guidelines require consideration of fairness when a test is validated through a criterion-related study. The applicable section states: 55 (b) Investigation of fairness. Where a selection procedure results in an adverse impact on a race, sex, or ethnic group identified in accordance with the classifications set forth in section 4 above and that group is a significant factor in the relevant labor market, the user generally should investigate the possible existence of unfairness for that group if it is technically feasible to do so. The greater the severity of the adverse impact on a group, the greater the need to investigate the possible existence of unfairness. 56 29 C.F.R. Sec. 1607.14(B)(8)(b). Dr. Palmer discussed his analysis with respect to the fairness doctrine: 57 Q. Dr. Palmer, did you perform any analyses on whether the examination was fair to black applicants in this case? 58 A. Yes, I did. I should perhaps begin by explaining what fairness means in this context. The accepted definition of fairness in a criterion context like this is not whether groups differ in their test scores or not but whether, for example,, if you have a black with a score of 75 and a white with a score of 75 you would expect them to be equally good officers; that is, are the predictions roughly the same for blacks and whites. 59 Obviously, if you expected a black who made a 75 to be as good as a white you made an 80, there would be some unfairness there because you are preferring the person with an 80 on the basis of his test score. And you can really only do that if it's justified by your expectation of performance. And so there's a standard way of testing this. Actually, McCann Associates did this for their criterion measure and it appears in the report. I arranged the computations slightly differently but I did it for all five of these and they formed--the results of this formed my final exhibit which I'm sure pleases everyone, which is exhibit 18, and I should perhaps say something else before we go into the numbers, too. Both the Guidelines and the standards require investigation of test fairness when it is technically feasible. Now, I believe that it is arguable whether or not it was technically feasible under these circumstances, the groups are so small--we are talking about groups of 40 and 49--that I would probably be sympathetic to an argument that it wasn't really technically feasible. 60 On the other hand, since it was easy to do I did it anyway and these are the results. And I sort of think this is arguable whether one needs to do this but as I said McCann Associates did it and since it was easy to do I did it rather than argue that it might not be feasible because I think that's a judgment call. 61 What we have here is five sets, each of which contains three pages. 62