Oversight of nursing homes is a shared federal-state responsibility. Based on statutory requirements, CMS (1) defines quality standards that nursing homes must meet to participate in the Medicare and Medicaid programs and (2) contracts with state survey agencies to assess whether homes meet those standards through annual surveys and complaint investigations. Although CMS has issued extensive guidance to states on determining compliance with federal quality requirements, we have found that some state surveys understate quality problems at nursing homes. Federal nursing home quality standards focus on the delivery of care, resident outcomes, and facility conditions. These standards, totaling approximately 200, are grouped into 15 categories, such as Resident Rights, Quality of Life, Resident Assessment, Quality of Care, Pharmacy Services, and Administration. For example, there are 23 standards within the Quality of Care category ranging from “promote the prevention of pressure development” to “the resident environment remains as free of accident hazards as is possible.” CMS has also developed detailed investigative protocols to assist state survey agencies in determining whether nursing homes are in compliance with federal quality standards. This guidance is intended to ensure the thoroughness and consistency of state surveys and complaint investigations. Every nursing home receiving Medicare or Medicaid payment must undergo a standard state survey not less than once every 15 months, and the statewide average interval for these surveys must not exceed 12 months. During a standard survey, teams of state surveyors—generally consisting of registered nurses, social workers, dieticians, or other specialists—evaluate compliance with federal quality standards. Based on the care provided to a sample of residents, the survey team (1) determines whether the care and services provided meet the assessed needs of the residents and (2) measures resident outcomes, such as the incidence of preventable pressure sores, weight loss, and accidents. In contrast to a standard survey, a complaint investigation generally focuses on a specific allegation regarding a resident’s care or safety and provides an opportunity for state surveyors to intervene promptly if problems arise between standard surveys. Surveyors generally follow state procedures when investigating complaints, but must comply with certain federal guidelines and time frames. When deficiencies are identified, federal sanctions can be imposed to help encourage homes to correct them. Sanctions are generally reserved for serious deficiencies—those at the G through L levels—which constitute actual harm and immediate jeopardy. Sanctions for such serious quality problems can affect a home’s revenues and provide financial incentives to return to and maintain compliance. Such sanctions include fines known as civil money penalties, denial of payment for new Medicare or Medicaid admissions, or termination from the Medicare and Medicaid programs. State surveys that miss serious deficiencies or cite deficiencies at too low a scope and severity level have enforcement implications because a nursing home may escape sanctions intended to discourage repeated noncompliance. For example, facilities that receive at least one G through L level deficiency on successive standard surveys or complaint investigations must be referred for immediate sanctions. In addition, CMS guidance calls for higher fines when a home has a poor compliance history and requires that state survey teams revisit a home to verify that serious deficiencies have actually been corrected (such revisits are not required for most deficiencies cited below the actual harm level—A through F). Statutorily required federal monitoring surveys, which are conducted annually in at least 5 percent of state-surveyed Medicare and Medicaid nursing homes in each state, are a key CMS oversight tool in ensuring the adequacy of state surveys. CMS headquarters—specifically, CMS’s Survey and Certification Group—is responsible for the management of the federal monitoring survey database and for oversight of the 10 CMS regional offices’ implementation of the federal monitoring survey program. Federal surveyors located in regional offices conduct federal monitoring surveys. The surveys can be either comparative or observational, with each offering unique advantages and disadvantages as an oversight tool. For example, an advantage of comparative surveys is that they are an independent evaluation of a nursing home recently surveyed by a state survey agency team. A disadvantage is that the time lag between the two surveys can make analysis of differences difficult. Comparative survey. A federal survey team conducts an independent survey of a home recently surveyed by a state survey agency in order to compare and contrast the findings. This comparison takes place after completion of the federal survey. When federal surveyors identify a deficiency not cited by state surveyors, they assess whether the deficiency existed at the time of the state survey and should have been cited by entering either yes or no to the question, “Based on the evidence available to the , should the team have cited this ?” This assessment is critical in determining whether understatement occurred because some deficiencies cited by federal surveyors may not have existed at the time of the state survey. For example, a deficiency identified during a federal survey could involve a resident who was not in the nursing home at the time of the earlier state survey. By statute, comparative surveys must be conducted within 2 months of the completion of the state survey. However, differences in timing, resident sample selection, and staffing can make analysis of differences between the state and federal comparative surveys difficult. On the basis of our prior recommendations, CMS has taken several steps to ensure that comparative surveys more accurately capture conditions at the time of the state survey. For example, CMS now calls for the length of time between the state and federal surveys to be between 10 and 30 working days and requires federal surveyors conducting a comparative survey in a nursing home to include at least half of the state survey’s sample of residents from that nursing home in the comparative survey sample, making it easier to determine whether state surveyors missed a deficiency. Furthermore, federal comparative survey teams are expected to mimic the number of staff assigned to the state survey. CMS also issued guidance in October 2002 defining the criteria for federal surveyors to consider when selecting facilities for comparative surveys. These selection criteria can generally be categorized as state survey team performance and facility characteristics. Regional offices were given latitude in their use of these criteria and may supplement them with other selection factors unique to their regions. For example, some regions use statistics on the prevalence of pressure sores in a nursing home’s resident population as a comparative survey selection factor. Observational survey. Federal surveyors accompany a state survey team to a nursing home to evaluate the team’s on-site survey performance and ability to document survey deficiencies. State teams are evaluated in six areas—Concern Identification, Sample Selection, General Investigation, Food-Borne Illness Investigation, Medication Investigations, and Deficiency Determination—and are rated in one of five categories for each of the six measures. The rating categories—from highest to lowest—are extremely effective, very effective, satisfactory, less than satisfactory, and much less than satisfactory. CMS annual state performance reviews require that state survey teams achieve an average rating of satisfactory. Observational surveys allow federal surveyors to provide more immediate feedback to state surveyors and to identify state surveyor training needs. However, observational surveys are not independent evaluations of the state survey. Because state surveyors may perform their survey tasks more attentively than they would if federal surveyors were not present, observational surveys may not provide an accurate picture of state surveyors’ typical performance. Since 2001, CMS has also taken steps to strengthen observational surveys. For example, the agency issued written guidance defining a standard process for resolving disagreements and a new manual to increase consistency across observational surveys. The 976 federal comparative surveys conducted from fiscal year 2002 through 2007 ranged from as few as 10 in Vermont, which has about 40 facilities, to as many as 49 in California, which has about 1,300 facilities. Of the 4,023 federal observational surveys conducted during the same period, the number ranged from 16 in New Hampshire to 346 in California. The results of federal monitoring surveys, including information on the corresponding state surveys, are entered in the federal monitoring survey database. In fiscal year 2002, CMS began including information on comparative surveys in the database, and the agency began requiring federal surveyors to determine whether a deficiency cited by federal but not state surveyors had been missed by determining whether state surveyors should have cited the deficiency. Although comparative surveys and the wide variability across states in the proportion of homes with deficiencies at the actual harm and immediate jeopardy levels indicate that state surveyors miss some serious deficiencies, our prior work has also indicated that state surveyors sometimes understate the scope and severity of a deficiency. In 2003, we found widespread understatement of actual harm deficiencies in a sample of surveys from homes with a history of harming residents. Overall, 39 percent of the 76 state surveys we reviewed had documented problems that should have been classified as actual harm instead of as lower-level deficiencies. A substantial proportion of federal comparative surveys identify missed deficiencies at the potential for more than minimal harm level or above. From fiscal year 2002 through 2007, about 15 percent of federal comparative surveys nationwide identified state surveys that failed to cite at least one deficiency at the most serious levels of noncompliance—the actual harm and immediate jeopardy levels (G through L). There was wide variation across states in the proportion of comparative surveys that found at least one missed serious deficiency, from more than 25 percent in nine states to none in seven others. In contrast to missed serious deficiencies, missed deficiencies at the potential for more than minimal harm level (D through F) were considerably more widespread, with such missed deficiencies greater than 40 percent in all but five states. Every state had at least one comparative survey with missed D through F level deficiencies. At both levels of noncompliance, the most frequently missed deficiencies involved Quality of Care standards. Federal observational survey results and prior GAO reports have highlighted several factors that may contribute to the understatement of deficiencies. About 15 percent (142) of the 976 comparative surveys conducted from fiscal year 2002 through 2007 identified state surveys that missed at least one deficiency at the actual harm or immediate jeopardy level (G through L), the most serious levels of noncompliance. This proportion fluctuated from a high of 17.5 percent in fiscal year 2003 to a low of 11.1 percent in fiscal year 2004, but it has remained relatively constant at about 15 percent for the last several fiscal years (see fig. 1). This proportion is small, but CMS maintains that any missed serious deficiencies are unacceptable. From fiscal year 2002 through 2007, federal surveyors identified missed serious deficiencies in 25 percent or more of their comparative surveys in nine states. The proportion of missed serious deficiencies in these nine states ranged from 26.3 percent in Tennessee to 33.3 percent in New Mexico, South Carolina, South Dakota, and Wyoming (see table 2). The total number of missed deficiencies at the G through L levels also varied across these nine states, from a low of 4 in South Dakota to a high of 19 in South Carolina. Federal surveyors identified no missed serious deficiencies in seven states (see app. II for complete state results). In contrast to missed serious deficiencies, missed deficiencies at the potential for more than minimal harm level (D through F) were considerably more widespread on comparative surveys conducted during fiscal years 2002 through 2007. Approximately 70 percent of comparative surveys conducted nationwide identified state surveys that missed at least one deficiency at the potential for more than minimal harm level (D through F), with such missed deficiencies identified on greater than 40 percent of comparative surveys in all but five states—Alaska, Ohio, Vermont, West Virginia, and Wisconsin. On average, state surveys selected for comparative surveys failed to identify 2.5 D through F level deficiencies per survey. Undetected care problems at the D through F level are of concern because they could become more serious over time if nursing homes are not required to take corrective actions. Missed deficiencies at the potential for more than minimal harm level were not isolated to a single year during the 6 fiscal years we examined and continued to be a problem for states in fiscal year 2007. Nationally, the proportion of comparative surveys identifying at least one missed D through F level deficiency in fiscal year 2007 was about 74 percent (see fig. 2). For results by state, see appendix III. Our analysis found that the most frequently missed deficiencies at both the potential for more than minimal harm (D through F) and the actual harm or immediate jeopardy (G through L) levels occurred in quality standards under CMS’s Quality of Care category. Missed deficiencies in this category involved residents’ receipt of the necessary care and services to attain and maintain the highest practicable physical, mental, and psychosocial well- being—such as prevention of pressure sores, nutrition and hydration, accident prevention, and assistance with bathing and grooming. From fiscal year 2002 through 2007, 11.9 percent of federal comparative surveys (116) cited at least one Quality of Care deficiency at the actual harm or immediate jeopardy level that state survey teams failed to cite. These 116 surveys contained a total of 143 missed serious Quality of Care deficiencies. The category with the next highest frequency of missed serious deficiencies was Resident Behavior and Facility Practices, with only 2.2 percent of total federal comparative surveys. At the potential for more than minimal harm level, Quality of Care was one of two categories with the highest frequency of missed deficiencies—31.7 percent. For the percentage of missed deficiencies in each of the CMS quality standard categories, see appendix IV. Both federal observational surveys and our prior reports have identified factors that may contribute to the understatement of deficiencies by state survey teams. From fiscal year 2002 through 2007, 80 percent of the 4,999 federal monitoring surveys were observational. Our review of observational survey data—which are collected during direct observation of state survey teams—found that some of the lowest state survey team ratings nationwide were in the General Investigation and Deficiency Determination areas. Together, these two areas directly affect the appropriate identification and citation of deficiencies. The General Investigation segment of an observational survey evaluates the effectiveness with which the state survey team collected information to determine how the facility’s environment and care of residents affect residents’ quality of life, health and safety, and ability to reach their highest practicable physical, mental, and psychosocial well-being. This segment includes observations of state survey team actions such as collection of information, discussion of survey observations, interviews with facility residents, and implementation of CMS investigative protocols. The Deficiency Determination segment of an observational survey evaluates the skill with which the state survey teams (1) integrate and analyze all information collected and (2) use the guidance to surveyors and regulatory requirements to make accurate compliance determinations. This segment includes observations of state survey team actions such as reviews of regulatory requirements, team participation in deficiency discussions, presentation of complete information, accurate decision making, and accurate citation of deficiencies. Nationwide, 7.7 percent of the state survey teams observed by federal surveyors received below satisfactory ratings on the General Investigation measure from fiscal year 2002 through 2007. During the same 6 fiscal years, 9.2 percent, or about 1 in 11, of the state survey teams observed by federal surveyors received below satisfactory ratings on the Deficiency Determination measure. Our analysis found variation across states in survey team performance in General Investigation and Deficiency Determination. Sixteen states had more teams than the national average receive below satisfactory ratings for both measures, while 28 states had fewer teams than the national average receive below satisfactory ratings (see app. V). Poor performance on these observational survey measures may be a contributing factor to the understatement of deficiencies by state survey teams. For example, of the nine states in table 2 with the highest percentage of missed serious deficiencies on comparative surveys, six had more teams than the national average receive below satisfactory ratings for both General Investigation and Deficiency Determination (see table 3). Our prior reports have described some other factors that may contribute to survey inconsistency and the understatement of deficiencies by state survey teams: (1) weaknesses in CMS’s survey methodology, such as poor documentation of deficiencies; (2) confusion about the definition of actual harm; (3) predictability of surveys, which allows homes to conceal problems if they so desire; (4) inadequate quality assurance processes at the state level to help detect understatement in the scope and severity of deficiencies; and (5) inexperienced state surveyors as a result of retention problems. In ongoing work, we are investigating the factors that contribute to understatement. CMS has taken steps to improve the federal monitoring survey program, but weaknesses remain in program management and oversight. For example, CMS has improved processes to ensure that comparative surveys more accurately reflect conditions at the time of the state survey, has switched control of the federal monitoring survey database to the office responsible for ensuring the effectiveness of state surveys, and has begun examining how to use monitoring survey data to improve oversight. Despite this progress, the management and oversight potential of the program has not been fully realized. In particular, CMS (1) has only begun exploring options for identifying understatement that occurs in cases where state surveys cite deficiencies at too low a level, for possible implementation in fiscal year 2009, and (2) is not effectively managing the federal monitoring survey database or using the database to oversee consistent implementation of the federal monitoring survey program by its regional offices. CMS has taken steps in three areas—time between surveys, resident sample, and survey resources—to ensure that comparative surveys more accurately capture the conditions at the time of the state survey. Time between surveys. In fiscal year 2002, CMS initiated a policy that shortened the length of time between state and comparative surveys from 2 months to 1 month. CMS relaxed the 1 month standard by changing the requirement to 30 working days in fiscal year 2003. As a result of shortening the time between the two surveys, the conditions at the time of the comparative survey are more likely to reflect those at the time of the state survey; for example, the same residents are still likely to be in the nursing home. Comparative surveys during fiscal year 2007 took place on average 21.4 working days (30.9 calendar days) after state surveys. Resident sample. Beginning in fiscal year 2003, CMS policy required that comparative surveys include at least half of the residents from state survey investigative samples. Officials from several regional offices said that examining the same resident allows for more clear-cut determinations of whether the state should have cited a deficiency. Since the policy change, about 78 percent of comparative surveys from fiscal year 2003 through 2007 included at least half the residents from state surveys’ investigative samples. By comparison, only 13 percent of comparative surveys met that 50 percent threshold in fiscal year 2002, the year before the policy went into effect. Survey resources. Beginning in fiscal year 2003, CMS initiated a policy that each comparative survey should have the same number of federal surveyors as its corresponding state survey, again to more closely mirror the conditions under which the state survey was conducted. We found that in fiscal year 2007, the average state survey team (3.4 surveyors) was larger than the average federal survey team (3.0 surveyors). However, on average, federal surveyors remained on-site longer than state surveyors— 4.3 days for federal surveyors compared with 3.7 days for state surveyors. When the number of surveyors and time on-site are taken together, state surveys averaged 12.6 surveyor-days and federal comparative surveys averaged 12.9 surveyor-days. Given these improvements, we asked the regional offices how receptive state survey teams were to feedback that they had missed deficiencies. Most regional office officials told us that in general the feedback session with state surveyors on missed deficiencies was not contentious and that state surveyors generally accepted the feedback provided. However, CMS established a formal dispute resolution process for comparative surveys in October 2007. The process is similar to the process already in place for resolving disagreements about observational survey results. While CMS requires federal surveyors to determine whether a deficiency cited on a comparative but not a state survey was missed by state surveyors, there is no comparable requirement for deficiencies that are cited at different scope and severity levels. As a result, comparative surveys do not effectively capture the extent of the understatement of serious deficiencies by state surveyors. As with missed deficiencies, a discrepancy between federal and state survey results does not automatically indicate understatement. For example, the deficiency could have worsened by the time of the federal survey. Although CMS does not require federal surveyors to evaluate scope and severity differences between the two sets of surveys, we found that some regional offices used the validation question for missed deficiencies— ”based on the evidence available to the , should the team have cited this ?”—to make such a determination. Using the validation question to make these determinations is contrary to CMS guidance issued in October 2003, which instructed comparative survey teams to only answer this question when the state failed to cite the deficiency altogether. To assess whether differences in scope and severity levels were actually understated—rather than deficiencies that worsened between the state and federal surveys—we first identified all 71 deficiencies on comparative surveys conducted from fiscal year 2002 through 2007 where federal survey teams cited actual harm or immediate jeopardy deficiencies that state survey teams cited at a lower scope and severity level. We then examined the comment fields in the federal monitoring survey database associated with those deficiencies. Our analysis identified 27 deficiencies (38 percent) in which federal survey teams determined that a state’s scope and severity citation was too low. For another 22 deficiencies (31 percent), federal survey teams found that the state’s lower scope and severity determination was appropriate, given the circumstances at the time of the state survey. The remaining 22 deficiencies (31 percent) did not have comments or contained remarks that were inconclusive about whether the state deficiency citation was too low. When the confirmed scope and severity understatement was included with understatement caused by missed deficiencies, the total percentage of comparative surveys with understatement of serious deficiencies increased by an average of about 1 percentage point over the 6 fiscal years we analyzed (see fig. 3). While CMS headquarters does not require federal surveyors to determine whether a deficiency cited by state survey teams was cited at too low a scope and severity level, some regional offices have developed their own procedures to track this information and use it to provide feedback to state survey agencies. For example, in one regional office an individual reviews all comment fields for a year’s worth of comparative surveys, makes a hand count of scope and severity differences that states should have cited, and then shares this with the state survey agencies during their annual performance reviews. Because the federal monitoring survey database does not automatically collect data on scope and severity determinations, CMS headquarters does not have access to the data analyses the regions have independently conducted. Some of the regional offices told us that they would like to have a specific way that the federal monitoring survey database could track scope and severity understatement that is similar to how deficiencies missed by state surveyors are tracked. In January 2008, CMS officials told us that they had initiated a pilot program in October 2007 to test the collection of data on understatement of scope and severity differences. According to CMS, the pilot, which will run through 2008 for possible fiscal year 2009 implementation, is necessary because the agency needs to determine which scope and severity understatement differences should be captured. For example, CMS is uncertain whether regions should only focus on differences that would raise the scope and severity level to actual harm or immediate jeopardy and not assess differences for understatement that occurs at lower scope and severity levels. Our analysis found that CMS headquarters was not effectively managing the federal monitoring survey database or using the database to oversee consistent implementation of the federal monitoring survey program by regional offices. While CMS uses data from comparative and observational surveys to provide feedback to state survey agencies during state performance reviews, CMS officials told us that they recognized the need to improve their management and use of the database for better oversight of the agency’s 10 regional offices. We identified two problems in CMS’s management of the federal monitoring survey database. CMS was not aware that (1) the results of a considerable number of comparative surveys were missing from the database and (2) the validation question for missed deficiencies was being used by some regional offices to identify scope and severity differences, contrary to CMS guidance. Missing data. In October 2007, we identified missing comparative surveys for two regional offices dating back to 2005 and asked CMS to follow up with officials in those regions. At least one of the regions had completed the surveys but had failed to upload them into the national database. We also found that CMS had not included data in the federal monitoring survey database from 162 contractor-led comparative surveys conducted between fiscal years 2004 and 2007. Use of validation question contrary to CMS guidance. Some regional offices were using the missed deficiency validation question to make determinations about whether scope and severity differences constituted understatement, making it difficult to distinguish between missed deficiencies and scope and severity understatement. In addition, we found that the regional office answer to the validation question was not always consistent with the information recorded in the comment box. Similarly, we identified weaknesses in CMS’s use of the database for regional office oversight. For example, CMS was not (1) examining comparative survey data to ensure that regional offices comply with CMS guidance intended to ensure that comparative surveys more accurately capture the conditions at the time of the state survey and (2) using the database to identify inconsistencies between comparative and observational survey results. Ensuring regional office compliance. While CMS has provided guidance to its regional offices to help ensure that comparative surveys more accurately capture the conditions at the time of the state survey, the agency is not fully using available data to ensure that the regional offices implement the agency’s guidance. For example, we found that the length of time between state and comparative surveys varied broadly by CMS region. In 2007, the average time gap ranged from a low of 15.4 working days (22.5 calendar days) in the Boston region to a high of 38.5 working days (54.4 calendar days) in the New York region. Furthermore, while 78 percent of comparative surveys from fiscal year 2003 through 2007 followed CMS’s guidance to include at least half of the residents from state surveys’ investigative samples, 22 percent of comparative surveys did not meet this threshold. Finally, when we contacted officials in CMS headquarters to ask clarifying questions about the data variables needed to conduct these analyses, the headquarters officials were not familiar with a number of the variables and referred us to a CMS staff person in one of the regional offices. Together, these three examples suggest that CMS is not effectively using the data to hold regional offices accountable for implementing guidance. Identify inconsistencies between comparative and observational results. CMS officials told us that they have begun to explore regional office differences in less than satisfactory ratings for state survey teams on observational surveys. However, CMS officials told us that they do not plan to use the database to identify inconsistencies between comparative and observational surveys that may warrant follow-up to ensure that regional offices are adhering to CMS guidance and consistently assessing state surveyor performance. For example, some states that performed below the national average in identifying serious deficiencies on comparative surveys received above-average marks on observational survey measures for Deficiency Determination and General Investigation. Wyoming’s 33.3 percent rate for surveys with missed serious deficiencies was more than double the national average of about 15 percent for surveys conducted during fiscal years 2002 and 2007. Yet Wyoming never received a below satisfactory rating on its General Investigation or Deficiency Determination measures during 18 observational surveys over that same 6-year period. We found similar inconsistencies in the results of federal monitoring surveys for South Dakota and a few other states. Although inconsistencies between comparative and observational surveys may not necessarily indicate a problem, they may warrant investigation. For example, in a small state like Wyoming it is likely that comparative and observational surveys have evaluated the same group of state surveyors. Further, Wyoming and South Dakota are two of six states whose federal monitoring surveys are conducted by CMS’s Denver regional office. Of the 140 observational surveys conducted from fiscal year 2002 through 2007, federal surveyors from the Denver regional office gave one below satisfactory rating on the Deficiency Determination measure. That 0.7 percent rate of below satisfactory performance was more than four times lower than the regional office with the next-lowest percentage—the Chicago regional office—which awarded below satisfactory ratings to 3.3 percent of state survey teams it observed. With about 1 in 6 comparative surveys concluding that state survey teams had missed a serious deficiency or understated its scope and severity level, it is evident that state survey agency performance limits the federal government’s ability to obtain an accurate picture of how often nursing home residents face actual harm or are at risk of serious injury or death. These missed serious deficiencies most frequently involved Quality of Care, reflecting shortcomings in fundamental provider responsibilities such as ensuring proper nutrition and hydration, accident prevention, and preventing pressure sores. Observational survey results also underscore problems state surveyors may face in identifying facility deficiencies; about 1 in 11 state survey teams nationwide were rated as below satisfactory by CMS surveyors on the Deficiency Determination measure. We found that comparative survey data may mask the true extent of understatement because CMS’s current protocol does not require regional offices to track in the federal monitoring survey database when state surveyors cite lower-than-appropriate scope and severity levels. As we conducted our work, CMS officials recognized this problem and in October 2007 began to experiment with a pilot program to measure understated scope and severity. However, at the conclusion of the pilot, scheduled for fiscal year 2008, CMS may decide not to implement a validation question for all scope and severity differences. We believe it is important to assess differences for understatement that occurs at the D through L levels—potential for more than minimal harm, actual harm, and immediate jeopardy. We also found that CMS was not effectively managing the federal monitoring survey database to ensure that regional offices were entering data in a timely and consistent fashion. Lack of accurate and reliable data hinders effective oversight. For example, we found that the database was missing a considerable number of comparative surveys. Further, CMS has not used the federal monitoring survey database to its full potential as an oversight tool. For example, CMS is not fully using data on comparative surveys to ensure that regional offices are implementing guidance intended to improve federal monitoring surveys. Although CMS’s Survey and Certification Group assumed control of the database in January 2007, headquarters staff often referred us to CMS regional office staff to answer specific database questions, suggesting a lack of familiarity with the organization and content of the database. In addition, agency officials told us that they do not plan to follow up on inconsistencies between comparative and observational survey results that could indicate weaknesses in how regional offices evaluate state surveyors’ performance. Identifying and following up on such inconsistencies could help ensure database reliability and hold regional office officials accountable for their implementation of the federal monitoring survey program, a program required by statute. To address weaknesses in CMS’s management of the federal monitoring survey database that also affect the agency’s ability to effectively track understatement, we recommend that the Administrator of CMS take the following two actions: Require regional offices to determine if there was understatement when state surveyors cite a deficiency at a lower scope and severity level than federal surveyors do and to track this information in the federal monitoring survey database. Establish quality controls to improve the accuracy and reliability of information entered into the federal monitoring survey database. To address weaknesses that affect CMS’s ability to oversee regional office implementation of the federal monitoring survey program, we recommend that the Administrator of CMS take the following two actions: Routinely examine comparative survey data and hold regional offices accountable for implementing CMS guidance that is intended to ensure that comparative surveys more accurately capture the conditions at the time of the state survey. Regularly analyze and compare federal comparative and observational survey results. In written comments on our draft report, HHS indicated that it fully endorsed and would implement our four recommendations intended to strengthen management and oversight of the federal monitoring survey program. The comments generally outlined CMS’s implementation plan through 2009 and indicated that some steps, such as improved management of the federal monitoring survey database, are already under way. HHS’s comments are reproduced in appendix VI. The majority of HHS’s comments focused on its strategic approach to improving oversight: (1) ensuring that all nursing homes are surveyed at least once every 15 months, (2) improving surveyor understanding of federal quality requirements through improved guidance and training, (3) increasing the consistency of state surveys through the introduction of a new nursing home survey methodology, and (4) improving the use of data generated by federal monitoring surveys. Many of these strategies aim to address the underlying causes of understatement, the topic of a forthcoming GAO report. HHS also noted that limitations in the Medicare survey and certification budget underscore the agency’s need to target resources effectively to maximize results. For example, HHS indicated that the implementation of the new survey methodology will be dependent on the level of funding in the overall survey and certification budget through fiscal year 2014. Survey and Certification funding is the subject of another forthcoming GAO report. Two of HHS’s observations merit further discussion. First, HHS noted that understatement that arises from a lack of understanding or confusion about federal requirements would generally not be detected through federal monitoring surveys because both federal and state surveyors would be affected by the same limitation. We believe that the consistency with which federal surveys have identified serious deficiencies missed by state surveyors from fiscal year 2002 through 2007—about 15 percent, on average—suggests that federal surveyors have a better understanding of CMS quality requirements than do state surveyors. We have previously reported that the limited experience level of state surveyors because of the high turnover rate was a contributing factor to deficiency understatement. Second, HHS questioned our use of “one missed deficiency per survey” as a measure of understatement. We believe that this standard is appropriate for serious deficiencies that result in harm or immediate jeopardy (G through L level) because the goal of state surveys should be to identify and require nursing homes to address all such deficiencies. CMS itself uses this standard during annual state performance reviews. We also used this standard to describe the proportion of comparative surveys that identified missed deficiencies at the potential for more than minimal harm level (D through F). Identifying and requiring nursing homes to correct such deficiencies is important because if uncorrected they have the potential to become more serious. Compared to missed serious deficiencies, we found that understatement of potential for more than minimal harm deficiencies was more widespread—about 70 percent of comparative surveys identified at least one state survey with such missed deficiencies. The number of state surveys with missed deficiencies at the D through F level was greater than 40 percent in all but five states, and state surveys selected for comparative surveys failed to identify an average of 2.5 deficiencies in this range per survey. In short, the magnitude of understatement at the potential for more than minimal harm level should be a cause for concern. HHS also provided technical comments, which we incorporated as appropriate. As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies to the Administrator of the Centers for Medicare & Medicaid Services and appropriate congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov. If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VII. In order to identify trends in the percentage of nursing homes cited with actual harm or immediate jeopardy deficiencies, we analyzed data from the Centers for Medicare & Medicaid Service’s (CMS) On-Line Survey, Certification, and Reporting system (OSCAR) database for fiscal years 2002 through 2007 (see table 4). Because homes must be surveyed at least every 15 months, with a required 12-month statewide average, it is possible that a home was surveyed more than once in any fiscal year. To avoid double counting homes, we included only a home’s most recent survey from each fiscal year. Because CMS conducts a relatively small number of comparative surveys, it is not possible to compare the results of comparative surveys to the results of all state surveys. In addition to the contact named above, Walter Ochinko, Assistant Director; Katherine Nicole Laubacher; Dan Lee; Elizabeth T. Morrison; Steve Robblee; Karin Wallestad; and Rachael Wojnowicz made key contributions to this report. Nursing Home Reform: Continued Attention Is Needed to Improve Quality of Care in Small but Significant Share of Homes. GAO-07-794T. Washington, D.C.: May 2, 2007. Nursing Homes: Efforts to Strengthen Federal Enforcement Have Not Deterred Some Homes from Repeatedly Harming Residents. GAO-07-241. Washington, D.C.: March 26, 2007. Nursing Homes: Despite Increased Oversight, Challenges Remain in Ensuring High-Quality Care and Resident Safety. GAO-06-117. Washington, D.C.: December 28, 2005. Nursing Home Deaths: Arkansas Coroner Referrals Confirm Weaknesses in State and Federal Oversight of Quality of Care. GAO-05-78. Washington, D.C.: November 12, 2004. Nursing Home Fire Safety: Recent Fires Highlight Weaknesses in Federal Standards and Oversight. GAO-04-660. Washington D.C.: July 16, 2004. Nursing Home Quality: Prevalence of Serious Problems, While Declining, Reinforces Importance of Enhanced Oversight. GAO-03-561. Washington, D.C.: July 15, 2003. Nursing Homes: Public Reporting of Quality Indicators Has Merit, but National Implementation Is Premature. GAO-03-187. Washington, D.C.: October 31, 2002. Nursing Homes: Quality of Care More Related to Staffing than Spending. GAO-02-431R. Washington, D.C.: June 13, 2002. Nursing Homes: More Can Be Done to Protect Residents from Abuse. GAO-02-312. Washington, D.C.: March 1, 2002. Nursing Homes: Federal Efforts to Monitor Resident Assessment Data Should Complement State Activities. GAO-02-279. Washington, D.C.: February 15, 2002. Nursing Homes: Sustained Efforts Are Essential to Realize Potential of the Quality Initiatives. GAO/HEHS-00-197. Washington, D.C.: September 28, 2000. Nursing Home Care: Enhanced HCFA Oversight of State Programs Would Better Ensure Quality. GAO/HEHS-00-6. Washington, D.C.: November 4, 1999. Nursing Home Oversight: Industry Examples Do Not Demonstrate That Regulatory Actions Were Unreasonable. GAO/HEHS-99-154R. Washington, D.C.: August 13, 1999. Nursing Homes: Proposal to Enhance Oversight of Poorly Performing Homes Has Merit. GAO/HEHS-99-157. Washington, D.C.: June 30, 1999. Nursing Homes: Complaint Investigation Processes Often Inadequate to Protect Residents. GAO/HEHS-99-80. Washington, D.C.: March 22, 1999. Nursing Homes: Additional Steps Needed to Strengthen Enforcement of Federal Quality Standards. GAO/HEHS-99-46. Washington, D.C.: March 18, 1999. California Nursing Homes: Care Problems Persist Despite Federal and State Oversight. GAO/HEHS-98-202. Washington, D.C.: July 27, 1998.

Summary: GAO reports since 1998 have demonstrated that state surveyors, who evaluate the quality of nursing home care on behalf of CMS, sometimes understate the extent of serious care problems in homes because they miss deficiencies. CMS oversees the effectiveness of state surveys through the federal monitoring survey program. In this program, federal surveyors in CMS's regional offices either independently evaluate state surveys by resurveying a home (comparative surveys) or directly observe state surveyors during a routine nursing home survey (observational surveys). GAO was asked to evaluate the information federal monitoring surveys provide on understatement and the effectiveness of CMS management and oversight of the survey program. To do this, GAO analyzed the results of federal monitoring surveys for fiscal years 2002 through 2007, reviewed CMS guidance for the survey program, and interviewed headquarters and regional office officials. A substantial proportion of federal comparative surveys identify missed deficiencies at the potential for more than minimal harm level or above. During fiscal years 2002 through 2007, about 15 percent of federal comparative surveys nationwide identified state surveys that failed to cite at least one deficiency at the most serious levels of noncompliance--actual harm and immediate jeopardy. Overall, nine states missed serious deficiencies on 25 percent or more of comparative surveys; in seven states federal surveyors identified no such missed deficiencies. During the same period, missed deficiencies at the lowest level of noncompliance--the potential for more than minimal harm--were more widespread: nationwide, approximately 70 percent of federal comparative surveys identified state surveys missing at least one deficiency at the lowest level of noncompliance, and in all but five states the number of state surveys with such missed deficiencies was greater than 40 percent. Undetected care problems at this level are a concern because they could become more serious if nursing homes are not required to take corrective action. The most frequently missed type of deficiency on comparative surveys, at the potential for more than minimal harm level and above, was poor quality of care, such as ensuring proper nutrition and hydration and preventing pressure sores. Federal observational surveys highlighted two factors that may contribute to understatement of deficiencies: weaknesses in state surveyors' (1) investigative skills and (2) ability to integrate and analyze information collected to make an appropriate deficiency determination. These factors may contribute to understatement because they directly affect the appropriate identification and citation of deficiencies. CMS has taken steps to improve the federal monitoring survey program, but weaknesses remain in program management and oversight. For example, CMS has improved processes to ensure that comparative surveys more accurately reflect conditions at the time of the state survey, such as requiring that comparative surveys occur within 30 working days of the state survey rather than within the 2 months set in statute. Despite these improvements, the management and oversight potential of the program has not been fully realized. For example, CMS has only begun to explore options for identifying understatement that occurs in cases where state surveys cite deficiencies at too low a level, for possible implementation in fiscal year 2009. In addition, CMS is not effectively managing the federal monitoring survey database to ensure that the regional offices are entering data accurately and reliably--CMS was unaware, for example, that a considerable number of comparative surveys had not been entered. Furthermore, CMS is not using the database to oversee consistent implementation of the program by the regional offices--for example, the agency is not using the database to identify inconsistencies between comparative and observational survey results.