Since 1998, the results from state surveys of nursing homes have been the principal source of public information on nursing home quality, which is posted and routinely updated on CMS’s Nursing Home Compare Web site. Under contract with CMS, states are required to conduct periodic surveys that focus on determining whether care and services meet the assessed needs of the residents and whether homes are in compliance with federal quality requirements, such as preventing avoidable pressure sores, weight loss, or accidents. During a survey, a team that includes registered nurses spends several days at a home reviewing the quality of care provided to a sample of residents. States are also required to investigate complaints lodged against nursing homes by residents, families, and others. In contrast to surveys, complaint investigations generally target a single area in response to a complaint filed against a home. Any deficiencies identified during routine surveys or complaint investigations are classified according to the number of residents potentially or actually affected (isolated, pattern, or widespread) and their severity (potential for minimal harm, potential for more than minimal harm, actual harm, and immediate jeopardy). To improve the rigor of the survey process, HCFA contracted for the development of quality indicators and required their use by state surveyors beginning in 1999. Quality indicators are derived from data collected during nursing homes’ assessments of residents, known as the minimum data set (MDS). The MDS contains individual assessment items covering 17 areas, such as mood and behavior, physical functioning, and skin conditions. MDS assessments of each resident are conducted in the first 14 days after admission and periodically thereafter and are used to develop a resident’s plan of care. Facility-reported MDS data are used by state surveyors to help identify quality problems at nursing homes and by CMS to determine the level of nursing home payments for Medicare; some states also use MDS data to calculate Medicaid nursing home payments. Because it also envisioned using indicators to communicate nursing home quality to consumers, HCFA recognized that any publicly reported indicators must pass a very rigorous standard for validity and reliability. Valid quality indicators that distinguish between good and poor care provided by nursing homes would be a useful adjunct to existing quality data. Such indicators must also be reliable—that is, they must consistently distinguish between good and bad care. HCFA contracted with Abt to review existing quality indicators and determine if they were suitable for public reporting. Abt catalogued and evaluated 143 existing quality indicators, including those used by state surveyors. It also identified the need for additional indicators both for individuals with chronic conditions who are long-term residents of a facility and for individuals who enter a nursing home for a short period, such as after a hospitalization (a postacute stay). According to Abt, a main concern about publicly reporting quality indicators was that the quality indicator scores might be influenced by other factors, such as residents’ health status. Abt concluded that the specification of appropriate risk adjustment models was a key requirement for the validity of any quality indicators. Risk adjustment is important because it provides consumers with an “apples-to-apples” comparison of nursing homes by taking into consideration the characteristics of individual residents and adjusting quality indicator scores accordingly. For example, a home with a disproportionate number of residents who are bedfast or who present a challenge for maintaining an adequate level of nutrition—factors that contribute to the development of pressures sores—may have a higher pressure sore score. Adjusting a home’s quality indicator score to fairly represent to what extent a home does—or does not—admit such residents is important for consumers who may wish to compare one home to another. After several years of work, Abt recommended 39 risk-adjusted quality indicators to CMS in October 2001. Twenty-two were based on existing indicators and the remaining 17 were newly developed by Abt, including 9 indicators for nursing home residents with chronic conditions and 8 indicators for individuals who enter a nursing home for a short period. In September 2001, CMS contracted with the NQF to review Abt’s work with the objective of (1) recommending a set of quality indicators for use in its planned six-state pilot and (2) developing a core set of indicators for national implementation of the initiative scheduled for late 2002. NQF established a steering committee to accomplish these two tasks. The steering committee met in November 2001 and identified 11 indicators for use in the pilot, 9 of which were selected by CMS. The committee made its selection from among Abt’s list of 39 indicators but it did not recommend use of Abt’s risk-adjustment approach. Moreover, the steering committee indicated that it would not be limited to the same Abt list in developing its recommended core set of indicators for national implementation. In April 2002, NQF released a draft consensus report identifying the indicators it had distributed to its members and the public for comment on their potential inclusion in the national implementation. Under its contract, NQF was scheduled to make a final recommendation to CMS prior to the national reporting of quality indicators. CMS’s initiative to augment existing public data on nursing home quality has considerable merit but more time is needed to assure that the indicators proposed by CMS for public reporting are appropriate in terms of their validity and reliability. Based on work by Abt to validate the indicators it developed for CMS, the agency selected quality indicators for national reporting. The full Abt validation report—which is important for a thorough analysis of the appropriateness of the quality indicators--was still not available to us as of October 28, 2002. Our review of available portions of the Abt report, however, raised serious questions about whether testing and validation of the selected indicators has been sufficient to move forward with national reporting at this time. Moreover, CMS plans to initiate national reporting before it receives recommendations from NQF, its contractor, on appropriate quality indicators. On August 9, 2002, CMS announced the 10 indicators selected for its nationwide reporting of quality indicators, which it plans to launch in mid- November 2002. CMS selected these indicators from those that Abt had validated in its August 2, 2002, validation report. Abt classified the indicators it studied as to the degree of validity—top, middle, or not valid. The indicators that CMS selected were in the top category with one exception—residents in physical restraints—which was in the middle category. The objective of Abt’s validation study was to confirm that the indicators reflect the actual quality of care that individual nursing facilities provide, after taking into account resident and facility-level characteristics. For example, a validation analysis could confirm that a low percentage of pressure sores among residents was linked to a facility’s use of procedures to prevent their development. Successful validation reduces the chance that publicly reported data could misrepresent a high-quality facility as a low-quality facility—or vice versa. CMS’s decision to implement national reporting in November 2002 is troubling, given the issues raised by our review of the available portions of Abt’s validation report. Although we asked CMS for a copy of Abt’s 11 technical appendixes, as of October 28, 2002, they were still undergoing review and were not available to us. The technical appendixes are essential to adequately understand and evaluate Abt’s validation approach. Our review of the available portions of the Abt report raised serious questions about whether the effort to date has been sufficient to validate the indicators. The validation study is based on a sample that is drawn from six states; it is not representative of nursing homes nationwide and may not be representative of facilities in these six states. Selected facilities were allowed to decline participation and about 50 percent did so. For those facilities in the validation study, Abt deemed most of the indicators as valid—that is, better care processes were associated with higher quality indicator scores, taking into account resident and facility-level characteristics. However, we could not evaluate these findings because Abt provided little information on the specific care processes against which the indicators were validated. Unresolved questions also exist about the risk adjustment of the quality indicators. Risk adjustment is a particularly important element in determining certain quality indicators because it may change the ranking of individual facilities—a facility that is among the highest on a particular quality indicator without risk adjustment may fall to the middle or below after risk adjustment—and vice versa. Data released by CMS in March 2002 demonstrated that Abt’s risk adjustment approaches could either lower or raise facility scores by 40 percent or more. Although such changes in ranking may be appropriate, Abt did not provide detailed information on how its risk adjustment approaches changed facility rankings or a basis for assessing the appropriateness of the changes. In addition to the questions raised by our review of the Abt validation report, CMS is not planning to wait for the expert advice it sought on quality indicators through its contract with the NQF. Under this contract, the NQF steering committee issued a consensus draft in April 2002 with a set of potential indicators for public reporting. The steering committee had planned to complete its review of these indicators using its consensus process by August 2002. In late June, however, CMS asked NQF to delay finalizing its recommendations until early 2003 to allow (1) consideration of Abt’s August 2002 report on the validity of its indicators and risk- adjustment methods—including the technical appendices, when they become available and (2) a review of the pilot evaluation results expected in October 2002. An NQF official told us that the organization agreed to the delay because the proposed rapid implementation timeline had been a concern since the initiative’s inception. CMS’s list of quality indicators for the November 2002 national rollout did not include six indicators under consideration by NQF—depression, incontinence, catheterization, bedfast residents, weight loss, and rehospitalization (see app. I). Instead, CMS intends to consider NQF’s recommendations and revise the indicators used in the mid-November national rollout sometime next year. CMS is also moving forward without a consensus on risk adjustment of quality indicators.