Opinion ID: 456159
Heading Depth: 2
Heading Rank: 2

Heading: The ORNL Model

Text: 106 As we have recently reaffirmed, [a]n agency may utilize a predictive model so long as it explains the assumptions and methodology it used in preparing the model. If the model is challenged, the agency must provide a full analytical defense. Eagle-Picher Indus., Inc. v. United States Envtl. Protection Agency, 759 F.2d 905, 921 (D.C.Cir.1985) (footnotes omitted). However, we will defer to an agency's judgment to use a particular model if the agency examines the relevant data and articulates a reasoned basis for its decision. See id. at 921-22; Small Refiner Lead Phase-Down Task Force v. Environmental Protection Agency, 705 F.2d 506, 535 (D.C.Cir.1983); Sierra Club v. Costle, 657 F.2d 298, 332-33 (D.C.Cir.1981). 107 The ORNL model was originally designed to produce an analytical tool which would assist decision-makers in their evaluation of the impact of conservation strategies and policies. 1982 Economic Analysis Document at 396 (app. E), J.A. at 1333. The model was not developed specifically for this rulemaking: government-supported scientists began work on the model in 1975, before passage of NECPA, and an initial version of the model was completed in 1976. See id. Numerous changes in the model have been made since then, including changes made as a result of comments received during this rulemaking. See 48 Fed.Reg. 39,376, 39,380 (1983); see also McMahon, Residential End Use Demand Modeling: Improvements to the ORNL Model in Beyond the Energy Crisis 297 (1981). 108 DOE used the ORNL model to compare the energy used without a standard during [the periods selected by DOE] to the energy used with a particular standard during the same period, 48 Fed.Reg. at 39,380, and thus to compute the amount of energy saved during that period which is attributable to the standard, id. Petitioners attack the model's market penetration algorithm. This algorithm quantified DOE's assumption that as fuel prices go up, market distortion--that is, the tendency of consumers to make economically irrational decisions to buy inefficient appliances--will go down. See generally E. Hirst & J. Carney, The ORNL Engineering-Economic Model of Residential Energy Use 37-44 (1978). DOE expressed the overall costs of an appliance to a consumer as the life cycle cost of the appliance, which included total purchase price, energy use over the life of the product, and the maintenance costs over the service life of the product, discounted to their present values. 45 Fed.Reg. 43,976, 43,984 (1980). As DOE explained, 109 The Model recognizes that the existing market-place is not perfect; that is, all manufacturers do not make and all consumers do not purchase the type of equipment that would always be a consumer's most energy efficient choice from a life cycle cost standpoint alone. The Model calculates the difference between the optimal and the projected average life cycle costs of equipment in a base year from actual prices and sales information of equipment in that year and projected energy prices. The difference between the optimal and projected life cycle costs is defined as the market distortion. In years after the base year, the Model projects the level of market distortion based on the initial level as modified by the market penetration algorithm. 110 48 Fed.Reg. 39,376, 39,380 (1983) (citation and footnote omitted). Thus, market distortion during the base year is the difference between the life cycle cost the average consumer will in fact ultimately pay, and the life cycle cost an economically rational consumer--that is, one seeking to minimize life cycle cost--would have paid. These figures are calculated from historical data. The market penetration algorithm in the ORNL model then uses the base-year market distortion to project market distortion in subsequent years. 111 In general terms, the algorithm has the effect of diminishing the distortion between the optimal and the average life costs of equipment being purchased as a function of rising energy prices. In other words, as the price of fuel for an appliance rises, the market in that appliance will become more rational, i.e., the efficiency of equipment being purchased will be closer to the optimal efficiency (from a life cycle cost perspective) than it was when the fuel price was lower. Under the Model's algorithm, however, distortion always remains. While higher prices reduce distortion over time, they can never eliminate it altogether. 112 Id. 113 The controversial assumptions underlying the market penetration algorithm go well beyond the proposition that consumers will respond to higher energy prices by buying more efficient appliances. DOE explained this as follows: 114 The ORNL Model assumes that as energy prices increase, more efficient equipment will be produced and purchased. This is consistent with almost all analyses of the subject. However, the model goes beyond many other studies in making the assumption that the consumer criteria for selecting energy efficiency will change as energy prices increase. In effect, the model assumes that longer paybacks on investments in improved efficiency of equipment will be acceptable as energy prices rise. 115 1982 Economic Analysis Document Sec. 5.3.2.6 at 144 (emphasis in original), J.A. at 1081. 116 Until the August 1983 rulemaking, DOE also described this second assumption through the concept of an implicit discount rate. See infra at 1390 (noting technical difficulties with DOE's use of implicit discount rates). An implicit discount rate describes all of the various factors influencing the market response to higher energy prices. 1982 Economic Analysis Document Sec. 5.3.2.6 at 144, J.A. at 1081. 117 A high implicit discount rate implies that future benefits are given relatively little value by the consumer [in making purchase decisions] (or, alternatively, that the availability of efficient products in the market is limited). A low implicit discount rate indicates a relatively high value placed on future benefits and a high availability of efficient products. The reason for the relationship between the implicit discount rate and the energy efficiency of new products is that an initial investment in greater efficiency is returned in the future. Thus, the greater the value placed on future benefits, the greater the investment in efficiency improvements, and the lower the implicit discount rate. 118 Id. An implicit discount rate of 100 percent means that [the average consumer demands] a full return on the investment ... within the first year [after] purchase. 1980 Economic Analysis Document Sec. 4.2.4 at 4-9, J.A. at 431. 119 Even assuming that the implicit discount rate remained the same as energy prices increased, consumers would nonetheless purchase more efficient appliances. Suppose, for example, that the implicit discount rate for an appliance is 100 percent, and consumers thus demand a full return on efficiency-improving technology in one year. As fuel prices increase, the cost of operating an appliance during the first year of ownership also increases. The average consumer will therefore be willing to pay more--up to the increased cost of operation during the first year--for a more efficient appliance, even at the same implicit discount rate. The algorithm assumes, however, that the average consumer will spend not only amounts equal to the increased cost of operation over one year on more efficient appliances, but that the consumer will also be willing to accept a longer payback period. The implicit discount rate will consequently decline. 120 The market penetration algorithm, then, can be seen as reflecting three increasingly specific propositions about the consequences of higher fuel prices on the market. First, the algorithm assumes that consumers will purchase more efficient appliances; second, it assumes that consumers will accept longer payback periods, and market distortion will therefore diminish; and third, it quantifies the expected change in market distortion in a specific mathematical formula. 121 Some of the petitioners challenged the first of these propositions during the rulemaking. See Comment No. 2121 at 24 (June 18, 1982) (NRDC), J.A. at 2758. In reply, DOE pointed to technical literature suggesting a general correlation between the price of energy and appliance efficiency, the results of consumer surveys, and anecdotal evidence from manufacturers and trade associations. See 48 Fed.Reg. 39,376, 39,380-81 (1983). 122 DOE also noted that between 1972 and 1981, in a time of rapidly increasing energy prices, appliance efficiency improved significantly. Petitioners suggested during the rulemaking that a combination of state-mandated efficiency standards, the threat of federal standards, and increases in efficiency as a by-product of other design objectives, id. at 39,381, accounted for efficiency improvements during that period. DOE replied at some length in its August 1983 notice. In sum, DOE concluded (1) that commenters had not identified any efficiency-improving design changes that appliance manufacturers introduced as a by-product of other design objectives; (2) that while federal and state regulatory efforts may have contributed to more efficient appliances, market pressures might well have caused similar efficiency improvements without actual or threatened regulation; and (3) that the pattern and timing of improvements in appliance efficiency in the past were generally consistent with DOE's belief that market pressures, particularly increases in energy prices, were principally responsible for those improvements. See id. at 39,381-82 (1983). Rather than reciting all the evidence underlying this dispute, we merely note our conviction that the evidence was fragmentary, Comment No. 2121 at 10 (June 18, 1982) (NRDC), J.A. at 2744, conflicting, and ultimately susceptible of different interpretations. DOE's notices acknowledged those problems, reviewed the record in light of major criticisms voiced by commenters, and offered a reasoned defense of the agency's conclusions. We therefore uphold DOE's conclusion that rising energy prices tend to encourage production of more efficient appliances. 123 We pass, then, to objections focused on DOE's theory that market distortion decreases as energy prices rise. DOE commented that it found only one empirical study directly relevant to the effect of energy prices on market distortion. DOE initially summarized the findings of the study as follows: 124 The only available information regarding changes in implicit discount rates as a function of energy price is an analysis of investment in thermal integrity for new houses from 1973 to 1979.... This analysis, based on survey information from 300,000 new houses and computer analysis of the energy use of the houses, shows almost no change in implicit discount rates for new houses with electric resistance heating and a relatively small (15 to 20 percent) reduction in implicit discount rates for gas-heated houses during the time period. As the 1973 data preceded the oil embargo and rapidly rising real energy prices in the residential sector during the middle 1970s, the apparent small reduction in implicit discount rates during this period suggests that the decisionmaking process on thermal integrity in new houses changes only slowly. Higher energy prices induce higher investment in energy efficiency improvements in the shells of new houses, but they appear not to change the decision criteria. 125 1982 Economic Analysis Document Sec. 5.3.2.6 at 145, J.A. at 1082 (citation omitted). The last sentence of this passage directly challenges the assumptions of the market penetration algorithm: DOE interpreted the study to suggest that higher energy prices appear not to change the decision criteria of consumers, while the algorithm assumes that the consumer criteria for selecting energy efficiency will change as energy prices rise. Id. at 144 (emphasis in original), J.A. at 1081. 126 DOE's interpretation of the thermal integrity study changed as the rulemaking progressed. In the April 1982 notice, DOE decided that the study involved circumstances sufficiently dissimilar to render its results inconclusive in the appliance context. 47 Fed.Reg. 14,424, 14,428 n. 13 (1982) (citation omitted). By the August 1983 notice, DOE had yet a different view: 127 The study indicated that, while investment in thermal integrity in electrically heated homes showed little change in implicit discount rates over the period 1973-1979, the implicit discount rates for those investments in gas-heated homes declined 15 to 20 percent. Inasmuch as the real price of gas increased almost five times as rapidly during the period as that of electricity, and the initial implicit discount rates for electricity were only a fraction of the initial implicit discount rates for gas, such a differential in changes in implicit discount rates between gas and electricity is explainable. While DOE tentatively characterized this reduction in implicit discount rates as relatively small in the April 1982 proposal, upon further analysis DOE believes such a characterization is inapt. First, [t]hermal integrity improvements are almost entirely made by third party builders of new housing, and therefore market imperfections in this type of investment would tend to be greater than for products directly purchased by the ultimate consumer. Second, there is no labeling program for such improvements. Third, such reductions in discount rates are not inconsistent with those projected for some appliances by the ORNL model over the same period of time. More importantly, upon reconsideration of the tentative conclusion to be derived from this study [contained in the passage quoted above from the 1982 Economic Analysis Document], DOE has determined that the results of the study support the concept of the reduction of market distortion as a result of rising energy prices. 128 48 Fed.Reg. 39,376, 39,382 n. 25 (1983). DOE thus finally settled on the view that the study, while certainly not conclusive, does suggest that higher fuel prices may lead to less market distortion. Id. at 39,382 (citation and footnote omitted). 129 DOE's analysis of a technical engineering study is entitled to great deference from judges, who are hardly equipped to match the expertise of DOE's scientists. Although we are concerned by DOE's shifting position as the rulemaking proceeded and the market penetration algorithm came under increasing attack, we are unable to say that DOE's use of the thermal integrity study was arbitrary or irrational. Whether the 15 to 20 percent decline in the implicit discount rate for gas-heated homes was significant and what weight should be given to reasons why that discount rate might be particularly resistant to changes were questions addressed to the informed judgment of DOE scientists and policymakers. We cannot hold that DOE acted unreasonably in concluding that the study furnished some support, although concededly not unequivocal or decisive support, for the factual assumptions embedded in the market penetration algorithm. 130 DOE also believed that high energy prices would tend to counteract some major causes of market failure in the appliance market. For example, appliances are often purchased by landlords, builders of new homes, and other third-party purchasers who will not pay the fuel bills. Such purchasers have an incentive to minimize the initial purchase price by buying initially inexpensive but inefficient appliances with high life cycle costs. DOE believed that as energy prices went up, consumers would be more likely to demand energy-efficient housing units. These market pressures might consequently work to correct an important source of market distortion. 31 Similarly, DOE acknowledged evidence that at least for some appliances, its labeling program had been only modestly effective. 32 In DOE's view, higher energy prices would spur more consumers to consult the labels and thereby diminish market distortion. 33 131 DOE sought to confirm these general views and to verify the algorithm's specific formula for declining market distortion by testing the algorithm against past experience. As the designers of the ORNL model conceded, the algorithm's formula was initially ad hoc; it lacks a theoretical basis and empirical validation. E. Hirst & J. Carney, The ORNL Engineering-Economic Model of Residential Energy Use 71 (1978). But cf. id. at 44 (although further efforts were required to validate algorithm, it yields results that are intuitively plausible). However, DOE's tests provided inconclusive for at least two reasons. First, DOE discovered that it could not obtain the market information required by the algorithm for any year before 1978. DOE attempted to estimate the necessary figures, but it was concerned that incorrect estimates might bias its tests. Second, DOE's efforts to verify the algorithm uncovered basic difficulties in DOE's procedures: 132 [T]he scientists who developed the test designed it to measure changes in implicit discount rates, rather than changes in market distortion. In the course of making initial test runs, however, DOE discovered that under certain circumstances relatively minor changes in market distortion could result in drastic changes to the implicit discount rate. The clear implication of this was that implicit discount rates were not a good shorthand for market distortion, and that the test of the algorithm, if run in this manner, would be virtually useless because major deviations in the projected implicit discount rates from actual rates would not establish that the algorithm was inaccurate. Finally, in the course of the initial test runs and a general review of the ORNL Model, it was determined that a problem existed in the ORNL Model's life cycle cost-efficiency curve, which has the effect of tending to exaggerate the market distortion in future years. That is, separate from the disputed formula that reduces market distortion as a function of rising energy prices, the algorithm has a bias toward a larger than appropriate market distortion. 133 48 Fed.Reg. 39,376, 39,383 (1983) (footnotes omitted). DOE's scientists believed that lengthy study would be required to solve these problems. See id. at 39,383 & n. 30. Because of the time that would be required for the analyses, the doubtful usefulness of the conclusions that would be reached, and the limited effect of the market penetration algorithm on the results of the rulemaking, id. at 39,383 (footnote omitted), 34 DOE decided not to delay the rulemaking while its scientists studied the algorithm further. However, DOE noted that it continuously reviews the ORNL model and market penetration algorithm for purposes unconnected with this rulemaking. DOE declared that if it discovered flaws in the model that might have affected the final rules under review, it would immediately reconsider the final rules. See id. at 39,384. No such reconsideration has been announced. 134 DOE's efforts to verify the algorithm empirically thus proved disappointing. Moreover, the remaining support for any general correlation between market distortion and fuel prices, let alone the specific relationship described in the algorithm, was largely conjectural and anecdotal. Commenters pointed out these shortcomings at length, but they did not show that any other method of predicting future market distortion--including the assumption that it would remain stable--would produce demonstrably more accurate predictions. In short, DOE attempted to solve the difficult predictive problem before it through use of an independently created econometric model which had received favorable notice in technical literature. See id. DOE responsibly addressed alleged defects in the model by changing the model or explaining why the defects were both extremely difficult to fix and of relatively minor moment to the rulemaking. Our comments about review of a computer model in a previous case apply here as well:The safety valves in the use of such sophisticated methodology are the requirement of public exposure of the assumptions and data incorporated into the analysis and the acceptance and consideration of public comment, the admission of uncertainties where they exist, and the insistence that ultimate responsibility for the policy decision remains with the agency rather than the computer. With these precautions the tools of econometric computer analysis can intelligently broaden rather than constrain the policymaker's options and avoid the artificial narrowing of options that [can be] arbitrary and capricious. 135 Sierra Club v. Costle, 657 F.2d 298, 334-35 (D.C.Cir.1981) (footnote omitted). 136 In this case, the uncertainties underlying modeling assumptions were perhaps greater, and the factual basis for the assumptions correspondingly scanter, than in many others. But DOE's interpretation of the statute required it to hazard some predictions of market distortion, and at the time the final rules were issued the empirical foundation for any such predictions would evidently have been insecure. Under the principles of Sierra Club, we therefore decline to reject DOE's use of the ORNL model to predict energy savings, although DOE must change its unlawful definition of significant savings. Our decision does not, of course, mean that in the future DOE may continue to rely on the market penetration algorithm and the ORNL model if further study in light of more complete information shows the model's predictions to be unreliable. 35 Cf. National Lime Ass'n v. Environmental Protection Agency, 627 F.2d 416, 454 (D.C.Cir.1980). Recognizing that policymaking in a complex society must account for uncertainty, ... does not imply that it is sufficient for an agency to merely recite the terms 'substantial uncertainty' as a justification for its actions. Motor Vehicle Mfrs. Ass'n v. State Farm Mutual Life Ins. Co., 463 U.S. 29, 103 S.Ct. 2856, 2871, 77 L.Ed.2d 443 (1983); cf. 1 Davis, Administrative Law Treatise Secs. 6:13, :16 (2d ed. 1978). DOE may resolve even substantial factual uncertainties in the exercise of its informed expert judgment; but it may not tolerate needless uncertainties in its central assumptions when the evidence fairly allows investigation and solution of those uncertainties.