Opinion ID: 184778
Heading Depth: 2
Heading Rank: 2

Heading: Failure to Require a Statistically Significant Sample

Text: 20 The ATA's second claim is that the FHWA arbitrarily failed to require random selection of a statistically significant sample of records for review. Instead, the FHWA chose to use a focused sampling technique, set forth in its publicly available Field Operations Training Manual. The Manual instructs investigators to [i]dentify and list drivers and vehicles that have been involved in accidents and drivers and vehicles found in violation during roadside inspections. These drivers and vehicles will be used to focus the review.... Federal Highway Administration, Field Operations Training Manual, ch. 3, at 4. Investigators are also to focus on drivers cited for hours-of-service violations when determining the level of compliance with those regulations. Id. at 10. It is undisputed that the records and vehicles examined first under the agency's focused sampling procedure are more suspect--that is, more likely to exhibit violations than randomly selected records and vehicles. It follows that the agency will find a higher violation rate using focused sampling than it would if it used the random method petitioners favor. 21 According to the ATA, compliance reviews under the rule do not produce a representative picture of a carrier's safety fitness. Because random sampling is not required, the ATA argues, a skewed sample may produce a skewed understanding of a carrier's safety management controls. In the ATA view the FHWA therefore fails to achieve its avowed purpose, the creation of a reasonable approach for assigning a safety rating which best describes the current safety fitness posture of a motor carrier as required by the safety fitness regulations. 62 Fed.Reg. 60,035, 60,045 (1997). 22 ATA appears to assume that any rational system must estimate the proportion of violations to be found in the total population of a carrier's documents. We agree, of course, that if everything else were equal, information about this proportion would be useful. But other measures are also useful, and the agency may--if it has some reason--rationally prefer them. 23 The data yielded by the FHWA method have value, certainly for ranking carriers. It is true that a 15% violation rate in a sample composed partly or wholly of suspect documents does not support the inference that the violation rate for the entire document population is 15%. But the fact that the suspect-document population rate is not equal to the overall violation rate and does not mean the two rates are not correlated. GDP and personal consumption are correlated, though hardly equal. It seems reasonable to believe that carriers with higher observed violation rates under FHWA's system--drawing a sample of suspect documents first, with (for many factors) minimum sample numbers based on size of carrier--will generally have higher overall violation rates. 24 It is true that some carriers will have a higher proportion of suspect documents than others. But this does not destroy the value of FHWA's method. Consider two carriers of equal size, X and Y, where the sample from X has the higher observed violation rate. It is reasonable to infer that X's overall violation rate is higher regardless of which carrier has more suspect documents. To illustrate, we present two limiting cases: In Case 1, X has so many suspect documents that the X sample is entirely made up such records, while Y has no suspect documents. In this case, the carrier with more suspect documents (i.e., more roadside violations, accidents, etc.) unsurprisingly has a higher violation rate. In Case 2 we assume the reverse--that X's sampled documents are all nonsuspect and Y's are all suspect. X's non-suspect documents show a higher violation rate than Y's suspect documents. The result is a little surprising, but all it means is that is that in this particular instance suspectness turned out not to have been a good proxy for violation rate for those two carriers. The inference that X was the worse violator is not impaired. 25 Part of ATA's problem arises from a misreading of the rules. The SFRM says that [w]hen a number of documents are reviewed, the number of violations required to meet a pattern must be equal to 10 percent of those examined. 62 Fed.Reg. 60,035, 60,044 (1997). ATA acts as if this meant that an overall 10 percent rate of noncompliance with a critical regulation is satisfactory. If that were true, it would follow that only a sampling procedure aimed at estimating the total rate of noncompliance would be rational. But the total rate is not the standard. Rather than setting the acceptable noncompliance rate at 10 percent of what all documents would show, the SFRM sets it at 10 percent among examined documents. 26 As we said, the agency must of course have some reason for preferring focused over random sampling. It did. In the statement accompanying promulgation of the final rule, the FHWA defended its decision on the grounds that it is in the best interest of public safety to continue to focus its limited resources on drivers and vehicles most likely to be in violation of the regulations. 62 Fed.Reg. 60,035, 60,039 (1997). 27 To understand the FHWA's rationale, it is helpful to understand the distinction that the agency draws between acute and critical regulatory violations, a distinction unchallenged here. The FHWA defines acute regulations as those with respect to which noncompliance is so severe as to require immediate corrective actions by a motor carrier regardless of the overall safety posture of the motor carrier. 49 CFR 385 App. B, II(b), 62 Fed.Reg. at 60,044. An example is 49 CFR § 382.201, which (motorists may be cheered to read) prohibits knowing use of a driver with a blood alcohol concentration of 0.04% or greater. 49 CFR 385 App. B, VII, 62 Fed.Reg. at 60,045. Each instance of an acute violation affects the relevant factor rating. 49 CFR 385 App. B, II(g), 62 Fed.Reg. at 60,044. Critical regulations are defined as those with respect to which noncompliance relates to management and/or operational controls. These are indicative of breakdowns in a carrier's management controls. 49 CFR 385 App. B, II, 62 Fed.Reg. at 60,044. An example is 49 CFR § 391.45(b), which prohibits carriers from using a driver who has not been medically examined and certified during the past 24 months. 49 CFR 385 App. B, VII, 62 Fed.Reg. at 60,046. Violations of critical regulations do not affect the safety rating in the relevant factor unless a pattern of noncompliance is observed. There is no pattern of noncompliance unless 10% of reviewed documents, and at least two documents, show violations. 49 CFR 385 App. B., II(g), 62 Fed.Reg. at 60,044 (1997). For acute violations the reasonableness of choosing focused over random sampling is clear. Even a single acute violation is serious enough to require immediate corrective actions and to affect the carrier's safety rating for the relevant factor. 2 Thus it is eminently reasonable for the FHWA to adopt a method designed to miss as few such violations as possible. Examining the documents and vehicles most likely to exhibit violations does so. 28 The FHWA's rationale is less obvious for critical violations, because the agency has decided that a violation rate below 10% will not affect the safety rating. Why not require random sampling but impose a lower tolerance threshold? The agency's answer is that by using a technique likely to detect as many violations as possible, it can most effectively discover areas requiring carriers' attention so that carriers can improve compliance and thus, presumably, safety. Random sampling is less effective in accomplishing this goal. Although this reasoning does not emerge with limpid clarity from the relevant pages of the Federal Register, the agency's concern with husbanding resources for maximum safety effect and fostering full compliance is evident. See 62 Fed.Reg. at 60,039. The ATA says that trying to locate problems is justifiable only for enforcement activities, not for assigning safety ratings. But even in choosing among safety rating methods it makes sense for the agency to look to the overall goal of the statute, namely safety. 29 ATA can make no claim that the agency's methodology makes the resulting ratings unsuitable for their ultimate use. They are made available to other federal agencies and to the public, MST Express, 108 F.3d at 403, and, as we have said, there is no showing that the system produces skewed rankings. The direct legal effects have been limited. In the past, the only apparent legal consequence has been that the recipient of an unsatisfactory rating has been prohibited from operating a commercial motor vehicle to transport ... [h]azardous materials ... or [m]ore than 15 passengers. 49 CFR § 385.13(a). The 1998 Act expands the effect, prohibiting any unfit owner or operator from operating motor vehicles in interstate commerce starting 60 days after the determination. See Pub.L. No. 105-478, § 4009(a), 112 Stat. 107, 405-06 (1998) (to be codified at 49 U.S.C. § 31144(c)(1)). ATA has not shown that the system will produce an unfitness rating that is arbitrary. 30 Part of ATA's objection on the sampling issue is that the SFRM fails to specify how far inspectors are to go in plowing through a carrier's documents. This flexibility produces the possibility that an inspector could manipulate the process. Consider two carriers, each with 1000 documents, which the inspector attacks worst first, with violations (in each case) showing in the first ten but not beyond. If the inspector looks at 100 for carrier A and 106 for carrier B, that choice alone (assuming ordinary rounding practices) puts A but not B on the wrong side of the 10 percent divide. 31 To some extent the Manual addresses this problem by setting minimum levels of document review for specific types of rules. See Federal Highway Administration, Field Operations Training Manual, ch. 3 (1997), at 5-6 (driver factor regulations), 7-9 (operational factor regulations), 9-10 (operational factor regulations). For these categories of records, the minimum levels get at the most troubling aspect of the problem--the chance that an inspector who had it in for a particular carrier might condemn it to an unsatisfactory rating by stopping at a very low number of documents. 32 The Manual does not prescribe an upper limit on the number of documents to be reviewed. But it does guide the inspector's decision to expand the review, stating that additional driver files are to be reviewed if the focused review indicates substantial noncompliance, id. at 6, and that increased attention may be required in certain areas of a carrier's operation that have revealed noncompliance. Id. at 10. Since inspections that include extra documents focus on problem areas, they are unlikely to yield systematically better ratings for more extensively scrutinized operators. In the terms of the Carrier A/Carrier B hypothetical, the six additional Carrier B documents reviewed will be in problem areas, so there is little reason to believe they are less likely to show violations than the first 100. It was reasonable for FHWA to suppose that a system that imposes a rigid constraint on the extent of the review would yield less accurate ratings than one that allows inspectors to probe areas that they judge suspicious. 33 Furthermore, forcing the agency to specify an upper limit on the extent of each review runs counter to the general principle that courts are ill-positioned to scrutinize an agency's allocation of its scarce resources. See, e.g., Heckler v. Chaney, 470 U.S. 821, 827, 105 S.Ct. 1649, 84 L.Ed.2d 714 (1985). 34 The discretion that FHWA's scheme confers on inspectors can be abused, of course; intentionally and arbitrarily discriminatory enforcement of a statute can be unconstitutional. See Brandon v. District of Columbia Board of Parole, 823 F.2d 644, 650 (D.C.Cir.1987). But we can see no basis for restricting agents' discretion on the mere assumption--completely unsupported by factual allegations--that otherwise inspectors will act in bad faith. 35 The ATA also makes a procedural claim here--that the notice-and-comment rule is defective because it specifies no sampling procedure at all; only the Manual does so. Insofar as this is just a repeat of its early claim, our prior discussion is the answer. Beyond that claim, ATA offers no supporting reason. Here we review whether the current system for assigning ratings is arbitrary. If the FHWA changes its policy, actions under the new policy will be subject to the same standard of review.