When investigating complaints about nursing homes, state survey agencies follow state policies and procedures based on CMS instructions. To oversee state survey agencies’ complaint investigation processes, CMS uses data from its complaints database and State Performance Standards System. CMS’s State Operations Manual outlines procedures for state survey agencies’ investigation of nursing home complaints. This manual is based on requirements in statutes and regulations and includes a detailed protocol for handling complaints and incidents, such as directions for key parts of the complaints process—intake, prioritization, investigation, and reporting of results. Intake. State survey agencies receive complaints via phone calls, e-mails, or letters. At intake, staff review the information provided by the complainant and, because each complaint can have more than one allegation, determine the type(s) of allegations involved, such as resident abuse or poor quality of care. Prioritization. Based on the nature of the allegations, staff assign a priority level to the complaint, which determines if an onsite investigation is required. Four of the eight priority levels require an onsite investigation. (See table 1.) For example, investigations for complaints that allege “immediate jeopardy” to a resident’s health, safety, or life must be started within 2 working days of receipt, while investigations for complaints that allege a high level of actual harm (“actual harm-high”) to a resident must be started within 10 working days of prioritization. Investigation. During the unannounced investigation, state agency surveyors may conduct a document review and observe nursing home conditions. Additionally, surveyors interview witnesses, including the resident about whose care the complaint was filed and other residents with similar care needs, being careful to protect the anonymity of those involved in the complaint. Surveyors determine whether the allegations are substantiated and whether the nursing home should be cited for any deficiencies (failure to meet federal or state quality standards), which may be related or unrelated to the complaint allegations. Deficiencies are categorized according to scope and severity. Scope refers to the number of residents potentially or actually affected and has three levels—isolated, pattern, or widespread. Severity refers to the degree of relative harm and has four levels—immediate jeopardy (actual or potential for death or serious injury), actual harm, potential for more than minimal harm, or potential for minimal harm. Reporting of Results. After the complaint investigation is completed, the state survey agency notifies the complainant and the nursing home of the outcome of the investigation, following guidelines specified in the State Operations Manual. CMS oversees state survey agencies’ complaint investigation processes using its complaints data and State Performance Standards System. CMS’s Complaints Data. As of January 1, 2004, state survey agencies were required to enter data about all complaints and incidents into the ACTS—Automated Survey Processing Environment (ASPEN) Complaints/Incidents Tracking System—database according to guidance provided by CMS. Officials in CMS’s headquarters and regional offices can access all information in ACTS, though the information is stored on individual state servers. CMS provides guidance to state survey agencies regarding ACTS database procedures, including what complaint information states are required to enter. The information is then uploaded into CMS’s national complaints database, which contains a variety of information about complaints, such as the date of the alleged event, the name of the nursing home involved, and the source of the complaint. (See table 2.) State Performance Standards System. CMS’s 10 regional offices are responsible for annually evaluating state survey agencies’ nursing home complaint investigations using four performance standards. (See table 3.) CMS developed the State Performance Standards System in fiscal year 2001 to assess whether state survey agencies were meeting the requirements for the survey and certification program and to identify areas for improvement. In fiscal year 2006, CMS reorganized the performance standards system, and in the following years made several revisions to the four nursing home complaint performance standards. None of the standards focus exclusively on nursing home complaints. For some standards, the scope of review includes incidents as well as complaints, facilities other than nursing homes, or standard surveys as well as complaint investigations. For all except the timeliness standard, the review is based on samples rather than the universe of complaints and incidents. Upon completion of the performance evaluation, CMS regional offices share the results with each respective state survey agency and CMS headquarters, which in turns shares each state’s scores with all of the other states. State survey agencies that fail performance standards must submit corrective action plans to their CMS regional offices, which the regional offices can accept or reject, depending on whether they believe the state has outlined appropriate steps to address poor performance. The regional offices use these plans to follow up with state survey agencies as part of their monitoring activities. CMS’s national complaints data show that state survey agencies received over 50,000 complaints about nursing homes in calendar year 2009. The number and types of complaints varied among states. State survey agencies investigated all but 102 of the complaints that required an investigation. Among complaints that were investigated and uploaded to CMS’s national database for 2009, 19 percent were substantiated with at least one federal deficiency cited. State survey agencies reported receiving 53,313 complaints about nursing homes in 2009. In 2009, 9 states received fewer than 100 complaints while 17 states received more than 1,000. Six states—Illinois, Missouri, New York, Ohio, Texas, and Washington—accounted for roughly half of all 2009 complaints in CMS’s database. Although the number of nursing home residents has remained relatively stable, the number of complaints received generally increased by about 1,000 complaints a year from 2004 to 2008. In 2009, the number of complaints dropped by about 5,000. Complaint Rate. Nationally, in 2009, CMS’s database showed a complaint rate of roughly 38 complaints per 1,000 nursing home residents. The complaint rate ranged from less than 1 (0.77) in South Dakota to about 137 in Washington. Additionally, 11 states received 15 or fewer complaints per 1,000 nursing home residents, while 14 states received more than 45. (See fig. 1.) Submission of Complaints and Sources. CMS data show that state survey agencies received three-quarters of complaints in 2009 by phone. Complaints also were submitted through other means, such as in writing, through e-mail, or in person. In 2009, complaints were typically submitted by family members (47 percent), anonymously (19 percent), or by residents (10 percent). Complaints were also submitted by current nursing home staff or other sources. Prioritization of Complaints. In 2009, among the complaints in CMS’s national data, state survey agencies prioritized most as either actual harm- high (45 percent) or actual harm-medium (33 percent). Roughly 10 percent of complaints were prioritized as immediate jeopardy and about 4 percent were prioritized as actual harm-low. Approximately 8 percent of complaints were prioritized at the four lowest levels and did not require an onsite investigation. State survey agencies varied in the percentage of complaints they prioritized at different levels. For example, 23 state survey agencies prioritized more than 50 percent of complaints as immediate jeopardy or actual harm-high, while 7 state survey agencies prioritized fewer than 10 percent of complaints they received at these two levels. Allegations. Allegations are specific charges within complaints; each complaint can have multiple allegations. In 2009, according to CMS’s national data, the average number of allegations per complaint was 2.3. Allegations that focused on quality of care or treatment accounted for about 40 percent of all allegations in 2009. (See table 4.) CMS data show that in 2009 about 48,900 of the approximately 53,300 complaints received required an investigation and that state survey agencies investigated all but 102 of those complaints. Among those 102 complaints, 25 percent were prioritized as either immediate jeopardy or actual harm-high (6 and 19 percent respectively). The remaining 75 percent were complaints prioritized as actual harm-medium or actual harm-low. The percentage of complaints investigated from 2004 through 2009 remained relatively stable even as the number of complaints increased in all years except 2009. In 2009, an investigation was initiated within CMS’s required time frames for most complaints prioritized as either immediate jeopardy or actual harm-high. Among immediate jeopardy complaints, an investigation was initiated within 2 working days of receiving the complaint for 88 percent of complaints. Among complaints prioritized as actual harm-high, an investigation was initiated within 10 working days of prioritization for 72 percent of complaints. Roughly 19 percent of the complaints that were investigated and uploaded into CMS’s complaints database for 2009 were substantiated with at least one deficiency cited. However, there was considerable variation across states. In 19 states, more than 30 percent of the complaints investigated were substantiated with at least one deficiency cited, while in 5 states, the proportion was less than 10 percent. Of the approximately 16,000 nursing homes nationwide, about 2,800 had one substantiated complaint where at least one deficiency was cited. In addition, about 1,100 nursing homes had two such complaints. The percentage of immediate jeopardy complaints that were substantiated with at least one deficiency cited was higher than for complaints prioritized at lower levels in 2009. According to CMS’s complaints database, roughly 26 percent of the immediate jeopardy complaints that were investigated were substantiated with at least one deficiency cited. Among complaints prioritized at lower levels, the percentage was around 21 percent for actual harm-high complaints, 17 percent for actual harm- medium complaints, and 12 percent for actual harm-low complaints. In 2009, among the complaints prioritized as immediate jeopardy or actual harm-high, the percentage substantiated with at least one deficiency was higher if the investigation was initiated within required time frames than if it was not. For example, among actual harm-high complaints that were investigated within 10 working days of prioritization, 22 percent were substantiated with at least one federal deficiency cited. (See table 5.) In contrast, among actual harm-high complaints that were investigated late, the proportion was 17 percent. (App. I contains state-level data on complaints received, investigated, and substantiated by state survey agencies, according to CMS data.) Many state survey agencies did not meet some of CMS’s performance standards for nursing home complaints in fiscal year 2009. In particular, 19 state survey agencies had difficulty investigating complaints and incidents prioritized as actual harm-high within the required time frame. State survey agencies reported that they have taken or plan to take steps in four key areas—staffing, agency restructuring, training and guidance, and monitoring—to meet CMS’s nursing home complaint standards. Although the standards do not assess state survey agencies’ communication with complainants, CMS does expect agencies to convey investigation findings to complainants in accordance with CMS’s State Operations Manual. We found that agencies varied in their interpretations of the manual’s instructions, and some provided limited information to complainants. More than half of state survey agencies had difficulty meeting certain CMS performance standards pertaining to nursing home complaints. According to CMS’s assessment for fiscal year 2009, 28 state survey agencies failed the timeliness of investigations standard for either immediate jeopardy or actual harm-high complaints, the prioritization of complaints standard, or both. Timeliness of Investigations Standard. CMS’s assessment of state survey agencies’ performance found that some had difficulty meeting the timeliness of investigations standard, which evaluates: (1) whether an investigation was initiated within 10 working days of prioritization for actual harm-high complaints and incidents for nursing homes, and (2) whether an investigation was initiated within 2 working days of receipt for immediate jeopardy complaints and incidents for nursing homes and other facilities. State survey agencies must begin investigating at least 95 percent of complaints and incidents within required time frames. For actual harm-high complaints and incidents, CMS evaluates performance for nursing homes separately from that of other facilities. For immediate jeopardy complaints and incidents, CMS evaluates performance for both nursing homes and other types of facilities. CMS found that in fiscal year 2009, 19 state survey agencies failed to meet the timeliness of investigations standard for complaints and incidents prioritized as actual harm-high. This marked an improvement from fiscal year 2008, when 25 states failed. States’ fiscal year 2009 scores varied widely. For example, among states failing this standard, Louisiana nearly passed with 94.4 percent of actual harm-high complaints and incidents investigated within the required time frame, while Michigan’s score was 17.3 percent. (For information on all state survey agencies’ performance on this standard, see app. II.) According to CMS’s national data for calendar year 2009, the 19 states that failed this standard in fiscal year 2009 accounted for more than half (52 percent) of all actual harm-high complaints received nationally. In these 19 states, at least 43 percent of actual harm-high complaint investigations were initiated late, and at least 33 percent were initiated more than 11 working days late. Officials from the three state survey agencies in our sample that failed to meet the timeliness standard for actual harm-high complaints cited long- standing workload and staffing issues as reasons. More specifically, officials with the Michigan and Texas survey agencies said they had difficulty because of staffing shortages and because the volume of complaints and incidents increased. Tennessee officials noted that the state has tried to hire the additional staff needed to investigate the state’s backlog of complaints, but has been hampered by low salaries for surveyor positions as well as a cumbersome state hiring process. Nationwide, state survey agencies generally performed better on CMS’s timeliness standard for immediate jeopardy complaints and incidents than they did for actual harm-high complaints and incidents. In CMS’s assessment for fiscal year 2009, all but nine state survey agencies passed this standard by initiating investigations within 2 working days of receipt for at least 95 percent of the immediate jeopardy complaints and incidents they received about nursing homes and other facilities. Among the nine state survey agencies that failed this standard, four had scores at or below 50 percent. As with actual harm-high complaints and incidents, the two state survey agencies in our sample that failed the timeliness standard for immediate jeopardy complaints and incidents—Michigan and Tennessee— cited staffing shortages or increases in the number of complaints and incidents as key reasons. Fourteen state survey agencies that met CMS’s timeliness standard for immediate jeopardy complaints and incidents did not meet the timeliness standard for actual harm-high complaints and incidents. An official in one CMS regional office noted that immediate jeopardy complaints are the highest priority and therefore rightly received the most attention. Prioritization of Complaints Standard. CMS’s assessment of state survey agencies’ performance found that most agencies (32) consistently passed this standard for the past four years. State survey agencies must appropriately prioritize at least 90 percent of complaints and incidents. CMS evaluates performance for nursing homes separately from that of other facilities. In CMS’s assessment for fiscal year 2009, all but nine state survey agencies passed this performance standard. Among the nine state survey agencies that failed this standard in fiscal year 2009, most had scores between 70 percent and 88 percent. (See app. II for information on all state survey agencies’ performance on this standard.) All but one of the six state survey agencies in our sample passed the prioritization standard in fiscal year 2009. Officials from Tennessee said that the agency had difficulty meeting this standard because of personnel changes and because it took time for new management to fully understand how the agency operates. Officials from the five state survey agencies in our sample that passed this standard generally attributed their agencies’ performance on the prioritization standard to staff skills and experience, training, and processes for quality control. For example, officials from two state survey agencies—Arkansas and Texas—attributed their states’ success, in part, to a supervisor’s or quality assurance specialist’s review of the priority levels assigned by the staff members who received the complaint. State survey agencies reported that they have taken or plan to take steps in four key areas—staffing, agency restructuring, training and guidance, and monitoring—to either improve or maintain performance on CMS’s nursing home complaint standards. Staffing. Officials from three of the state survey agencies in our sample indicated that because staff shortages affected their ability to meet CMS standards, they had taken steps to increase staffing. For example, officials of the Michigan survey agency, which repeatedly failed the timeliness of investigations standard between 2006 and 2009, reported that beginning in fiscal year 2009, the agency was able to hire additional surveyors and as of June 1, 2010, had eliminated its backlog of complaints. Tennessee officials indicated that the agency received state legislature approval in February 2009 to hire additional surveyors to fill vacant positions. Texas officials also hired additional surveyors to conduct complaint investigations. Officials of state survey agencies in our sample that met all or most of CMS’s nursing home complaint standards credited, among other factors, experienced agency staff. For example, Wisconsin officials indicated that the agency’s ability to meet CMS’s standards was partly due to the quality of the staff hired by the agency—specifically, some staff members’ experience in the regulatory process, as both health care providers and regulators. Agency Restructuring. Some state survey agencies restructured complaint investigation operations to address performance issues, either consolidating regional offices or creating separate units to investigate complaints. For example, to provide better statewide coverage with available staff, the Tennessee survey agency downsized from three regional offices to two. Arkansas and Texas both established separate complaint investigation units—in Arkansas’s case, more than 10 years ago—in an effort to better manage large volumes of complaints. Officials of state survey agencies that have separate complaint investigation units cited several advantages to dividing complaint investigation functions from standard survey functions, including greater efficiency and flexibility. For example, some officials said that staff assigned to the complaints unit are able to build experience and familiarity with the process and thus conduct more efficient investigations and prepare more accurate reports; likewise, staff that focus on standard surveys are able to conduct these inspections more efficiently because they do not have to investigate complaints at the same time. One official also said that a separate complaint investigation unit affords managers more flexibility—for example, by allowing them to more easily change staff members’ assignments from day to day to respond to high priority complaints. Training and Guidance. Officials of some state survey agencies attributed their agencies’ successful performance on the prioritization of complaints standard partly to staff training. State survey agencies also issued guidance, including policy manuals and standardized forms or templates, to guide staff through the complaint investigation process. For example, Florida provides staff with a 44-page manual, with chapters on intake, prioritization, and investigation of complaints, and created an automated complaint investigation form that captures information about each allegation in a complaint, as well as the evidence collected and findings reached with respect to each. Monitoring. Among the state survey agencies in our sample that failed to meet some of CMS’s standards, officials indicated that their agencies had implemented or planned to implement additional monitoring efforts. For example, Texas officials indicated that the agency conducts reviews throughout the complaint process. For example, after a complaint has been prioritized, a quality assurance specialist reviews the information to ensure that the prioritization was appropriate. Similarly, officials from Tennessee’s survey agency indicated that the agency planned to increase monitoring. In particular, the officials indicated that each of the state’s regional offices would track and report quarterly on the timeliness of investigations for all immediate jeopardy and actual harm-high complaints. Tennessee officials indicated that surveyors in the state’s regional offices would be immediately alerted when they are assigned an immediate jeopardy complaint to investigate, something not always done in the past. State survey agencies in our sample that generally passed CMS’s performance standards indicated that monitoring programs contributed to the agencies’ success. For example, a Florida official indicated that a supervisor reviews a sample of complaints received on the previous day to determine whether they were prioritized appropriately. Although the CMS performance standards do not assess whether state survey agencies are providing sufficient information to complainants about investigation results, CMS’s State Operations Manual indicates that state survey agencies should provide a written report to complainants in accordance with certain guidelines specified in the manual. The manual specifies that the state agency should acknowledge the complainant’s concerns, identify the agency’s regulatory authority to investigate, provide a summary of investigation methods and the date of the investigation, summarize the investigation findings, and identify any follow-up action to be taken. The six state survey agencies in our sample varied in their interpretations of the manual, particularly the instruction to provide a summary of the investigation findings. Two of the six agencies consistently provided detailed information that specifically addressed complainants’ allegations. For example, one sample letter we received from the Wisconsin survey agency lists four specific allegations made by the complainant and then describes the agency’s finding with respect to each, including whether a deficiency was cited. (See fig. 2 for an excerpt from this letter.) The other state survey agency that provided detailed information (Michigan) did so by enclosing the investigation report with the letter, along with the statement of deficiencies, if any were cited. A Michigan survey agency official said that staff also make at least one attempt to contact a complainant by telephone to explain the findings. In contrast, four of the state survey agencies sent complainants only boilerplate descriptions of the complaint investigation, typically sending one type of form letter if surveyors cited deficiencies and another if they did not. For example, in the sample letter we received from Florida, the survey agency varied the middle paragraph of its three-paragraph letter depending on whether deficiencies were cited (see fig. 3). An official of this agency said the letter was intended to let complainants know that the point of an investigation is to determine a nursing home’s compliance with regulations. Of the four state survey agencies that provided boilerplate descriptions of their investigation findings, two told complainants how to obtain a more detailed report. For example, a sample letter from the Arkansas state survey agency noted that the agency’s report on the deficiencies cited and the nursing home’s plan of correction should be posted in the nursing home. An Arkansas survey agency official said that complainants could also request a copy of the investigation report, but that it might be heavily redacted to protect medical and identifying information. CMS’s oversight of state survey agencies’ complaint investigation processes, through its performance standards system and complaints database, is hampered by data reliability issues. While the four performance standards CMS uses to assess state survey agencies’ processes for investigating nursing home complaints are consistent with certain key criteria for performance measures identified by GAO and other audit agencies, the standards have weaknesses in areas related to other key criteria, particularly data reliability, due in part to inadequate sample sizes and inconsistent interpretation of some standards by CMS reviewers. In addition, CMS has not made full use of the information it collects about state survey agencies’ complaint investigation processes. For example, in part because of data reliability concerns, CMS does not routinely use data from the complaints database to calculate certain measures that could enhance its understanding of state survey agencies’ performance. Although CMS requires state survey agencies that fail performance standards to develop corrective action plans, these plans do not necessarily address the underlying causes of performance issues, such as staffing shortages. CMS’s four nursing home complaint performance standards— (1) prioritization of complaints, (2) timeliness of investigations, (3) quality of investigations, and (4) documentation of deficiencies—are consistent with some, but not all, of the key criteria for performance measures identified by GAO and other audit agencies. Specific weaknesses we identified include a lack of comparability over time in the performance scores and thus an inability to assess trends; a lack of balance among some standards; and, most critically, a lack of data reliability, due in part to inadequate sample sizes and varying interpretations of the standards. Consistent with key criteria for performance measures, CMS’s performance standards are comprehensive and limited in number and overlap. Officials of all of the state survey agencies and CMS regional offices in our sample indicated that they considered the four nursing home complaint standards comprehensive. Although the performance standards system does not include standards for certain steps in the complaint investigation process, such as intake, officials indicated that the standards cover key steps, which include prioritizing complaints, scheduling and conducting investigations, and documenting any deficiencies identified. The standards are also limited in number and overlap, with each focused on different aspects of the nursing home complaint process than the others. Performance trends cannot be easily assessed because scores are not comparable over time. Because CMS changed the scoring methodologies for three of the four nursing home complaint standards during the past 4 years, it is not readily apparent from scores on these standards whether state survey agencies’ performance improved or worsened over that time period. CMS officials generally felt that the changes had enhanced the standards—in the case of the documentation of deficiencies and quality of investigations standards, by holding state survey agencies accountable for meeting all of the underlying requirements or by highlighting specific areas in need of improvement. Further, they did not identify the lack of trend data as a major concern. Officials noted that CMS judges state survey agencies’ performance for a given year, not in relation to prior years, and does not count scores on a standard in the first year after a significant change in methodology. However, a lack of consistent trend data makes it more difficult for CMS to assess whether the steps that it and the states are taking to improve performance on the nursing home complaint standards are having the desired effect. The balance among standards may be undermined by how the prioritization standard is scored. In general, the standards are balanced, so that the incentives created by one standard are counterbalanced by the incentives created by other standards. However, because the prioritization standard requires only that complaints be assigned a priority level at or above the level assigned by CMS reviewers, this standard may create an incentive for state survey agencies to assign higher priority levels than are warranted—which may jeopardize the timeliness of investigations. As one state survey agency official pointed out, the staff members who prioritize complaints may not be responsible for conducting investigations; consequently, these staff may be more focused on the agency’s meeting the prioritization standard than the timeliness standard and thus err on the side of caution in prioritizing complaints. According to CMS headquarters officials, the prioritization standard is scored this way because the agency was most concerned about complaints being prioritized at too low a level and did not want to fault state survey agencies for investigating complaints sooner than necessary. However, officials of two CMS regional offices noted that assigning complaints too high a priority level can cause misallocation of resources, as state survey agencies that prioritize complaints at higher levels than are warranted must investigate these complaints within shorter time frames than they otherwise would. Some performance scores are unreliable because of inadequate sample sizes and varying interpretations of standards among CMS reviewers. For three of the four CMS performance standards, the samples specified by CMS are in some cases too small to yield reliable data. Scores on the prioritization of complaints, quality of investigations, and documentation of deficiencies standards were generally based on a sample of 10 to 40 cases (10 percent, up to a maximum of 40). With samples this small, the margin of error around states’ scores on the prioritization of complaints standard, for example, was as much as 19 percentage points in fiscal year 2009. Accordingly, at least some of the states that received passing marks on this standard may actually have failed, and at least five of the nine states that received failing marks may actually have passed. Although the small sample sizes CMS requires make the reviews involved in certain standards more practical, by reducing the documentation CMS reviewers must examine, the trade-off is a lack of precision in the scores for these standards. Moreover, interpretation of some standards has varied among CMS reviewers—in terms of both the materials reviewed to assess performance and how certain requirements were construed by reviewers. Materials reviewed. To assess the quality of investigations, some CMS regional offices reviewed only information surveyors entered into the complaints database, while other CMS regional offices reviewed more extensive hard-copy notes from complaint investigations. CMS headquarters officials indicated that relying solely on the information in the complaints database to assess the quality of investigations was not consistent with federal guidance, stating that regional office officials should follow the guidance for the standard, which calls for reviewers to examine a variety of documents, including surveyor worksheets and investigation notes. They also noted that the investigation notes are not required data elements in the complaints database. Some state survey agency officials said that their scores on this standard have suffered because the investigation notes in the database do not always provide a complete picture of the agency’s complaint investigations. How requirements were construed. State survey agency officials we interviewed also noted differences in how CMS reviewers understood certain requirements in the standards, particularly in the documentation of deficiencies standard. For example, officials described differences in reviewers’ interpretations of what it means to quantify the extent of a deficient practice, one of the requirements in that standard. One state survey agency official said that his agency’s scores on the standards improved from one half of the year to the next simply because the CMS staff conducting the review changed. Officials in one of the CMS regions where all state survey agencies failed the documentation of deficiencies standard acknowledged the 100 percent failure rate was at least partially due to a change in the regional office’s review—specifically, regional managers having issued more explicit instructions to staff about how to assess states’ performance on particular requirements. The clustering of failing scores on this standard within certain CMS regions also suggests regional variation in interpretation; in three regions, all of the state survey agencies failed the documentation of deficiencies standard in fiscal year 2009, while in the other seven regions, half or fewer of the state survey agencies failed. Although some CMS regional offices have tried to ensure consistent interpretation of the standards within their own regions—for example, by requiring that multiple reviewers concur on any failing marks given to state survey agencies and encouraging ongoing dialogue about the standards—some officials we interviewed believe CMS should do more to ensure consistency across regions. CMS headquarters officials told us that the agency has issued additional guidance when officials became aware of a need for clarification, but some CMS regional office officials said that parts of the guidance need enhancement and that CMS headquarters should have more staff dedicated to developing guidance and answering questions from regional office staff. In addition, some state survey agency officials suggested that CMS regional offices should have less autonomy in the performance review process. One official suggested that CMS headquarters should exert more control over the regional offices with respect to the review process, and others indicated a need for more “review of the reviewers”—for example, by having the performance reviews conducted by each regional office validated by another. Officials of one state survey agency, noting that state survey agencies can appeal their performance scores only to the same regional office that conducted their performance review, suggested that a second regional office should at least be involved in the appeals process. CMS has not made full use of the information it collects about state survey agencies’ complaint investigation processes through its complaints database and performance standards system. For example, CMS does not routinely use data from its complaints database to calculate certain measures that could enhance its understanding of state survey agencies’ performance investigating complaints and has not publicly reported state survey agencies’ scores on the performance standards. CMS has not made full use of data in the complaints database to monitor performance. In part because of data reliability concerns, CMS does not routinely calculate certain measures that could shed additional light on state survey agencies’ performance—such as substantiation rates or additional measures of the timeliness of investigations. Substantiation rates, if interpreted by state survey agencies in a consistent manner, could provide insight into the quality of complaint investigations. Given the many factors that influence these rates, including whether the complaints have a basis in fact, it would not be appropriate to require state survey agencies to achieve a particular rate. However, substantial variation in rates, either among states or over time, could signal issues with complaint investigations and prompt further inquiry by CMS. A CMS headquarters official told us that because some state survey agencies may consider a complaint to be substantiated even if no federal deficiencies are cited, CMS headquarters does not systematically monitor substantiation rates and most CMS regional offices probably do not do so either. The Patient Protection and Affordable Care Act (PPACA), enacted March 23, 2010, requires HHS to post on the Nursing Home Compare Web site summary information on substantiated complaints, including their number, type, severity and outcome, by March 23, 2011. Accordingly, a CMS official told us that CMS headquarters will issue guidance to ensure that state survey agencies interpret substantiation in a consistent manner. Additional measures of timeliness—such as the number of days by which state survey agencies miss the deadlines for some complaint investigations—could provide CMS with a more comprehensive picture of performance in this area. We found that some state survey agencies with similar scores on CMS’s timeliness standard for actual harm-high complaints in fiscal year 2009 had very different backlogs of complaint investigations. For example, looking at two state survey agencies with performance scores of 82 and 85 percent—which indicates, respectively, that 18 and 15 percent of their investigations were late—we found that 51 percent of one agency’s late investigations were initiated more than 30 days late in calendar year 2009, compared with 4 percent for the other agency. Currently, the reliability of timeliness measures such as this is uncertain because state survey agencies do not necessarily enter all complaints into CMS’s database or prioritize complaints in the same way. Responsibility for training to address performance issues has generally been left to CMS regional offices. The CMS regional offices in our sample have used information from the performance standards system to identify performance issues, but training designed to address these issues has generally been undertaken by individual CMS regional offices and, as a result, has varied in content and scope. Complaint investigation training at the national level has been limited and was not designed to address specific performance issues identified during reviews. Officials of most of the state survey agencies in our sample indicated that CMS’s training and guidance was sufficient, but officials of two state survey agencies noted that their agencies provide any training above the basic level. One state survey agency official said that CMS should offer more comprehensive training, including more material on complaint investigations, so that states are not “sinking or swimming” on their own and are able to conduct investigations in a more consistent manner. PPACA directed HHS to enter into a contract to establish a National Training Institute to help surveyors develop complaint investigation skills. However, as of March 2011, funds had not yet been appropriated to implement this provision of the act, and CMS estimates that it would cost about $12 million to establish the institute. As a start, CMS has redirected about $1 million from other projects to initiate a project which will provide instruction on all aspects of complaint surveys for all facility types, including nursing homes. Corrective action plans are not timely and may not address the underlying causes of performance issues. CMS requires state survey agencies that fail performance standards to submit plans to improve their performance, but CMS does not require these plans to be submitted until halfway through the next performance cycle, which allows little time for corrective actions to take effect before the next performance review. (See fig. 4.) Moreover, despite CMS regional office input, the plans do not necessarily address the underlying causes of state survey agencies’ failure to meet performance standards. For example, all three of the state survey agencies in our sample that failed the timeliness of investigations standard for immediate jeopardy complaints, actual harm-high complaints, or both in all 4 fiscal years from 2006 through 2009 cited staff shortages as a reason, but two of the three submitted at least one corrective action plan during that period that did not propose hiring the additional staff needed. CMS regional office officials indicated that they had accepted such corrective action plans because the steps the state survey agencies did propose—such as developing a graphic analysis tool to track performance or implementing additional central oversight of regional offices—were likely to improve performance to some extent, and because CMS does not have the authority to require state survey agencies to hire or reallocate staff. Only one of the CMS regional offices in our sample reported ever having rejected a corrective action plan, and officials of one CMS regional office told us they preferred that a corrective action plan provide a realistic account of what a state survey agency was going to try to achieve rather than propose actions that the agency could not carry out. Some CMS officials view the penalties the agency might impose for failure to meet nursing home complaint standards as counterproductive or unrealistic. CMS’s regulations provide for penalties to be imposed on a state survey agency for failure to follow procedures specified by CMS for complaint investigations, such as reducing funding or terminating the contract under which the state survey agency conducts standard surveys and complaint investigations. CMS headquarters officials noted that while CMS has reduced funding to state survey agencies for failure to meet requirements for standard surveys, such as statutory time frames, the agency has not done the same for complaint investigations. One official said that CMS has not done so partly because of concerns about the fairness of penalizing states for failure to meet standards that may vary from year to year, as well as concerns that reducing states’ funding might make it even more difficult for them to meet the standards. Some CMS regional office officials said that reducing state survey agencies’ funding for failure to complete complaint investigations on time made sense, but others said that taking resources away from the agencies could be counterproductive, further hampering their ability to carry out investigations. Although CMS could terminate its contract with a state survey agency, CMS officials we interviewed indicated that this was not a realistic option. CMS has not publicly reported state survey agencies’ performance scores. Public reporting of performance information has been advocated by GAO and other auditors as a critical step in performance management because it provides policymakers and the public with information needed to assess progress and may also serve to motivate agency managers and staff. While CMS has shared state survey agencies’ scores on the performance standards with all of the other state survey agencies, it has not made the scores available to other stakeholders, such as residents, family members, or advocates. According to a CMS headquarters official, some state survey agencies have made their own scores publicly available, but CMS has not yet issued any guidance to the states on public disclosure of scores. This official told us that CMS plans to issue a policy memo affirming state survey agencies’ right to disclose their own scores and is also considering making all of the scores publicly available, possibly on CMS’s Web site. Although some CMS regional office officials questioned whether performance reports might too easily be misconstrued by the public and necessarily gloss over details that would provide a more nuanced picture of performance, GAO’s prior work on performance management suggests reports can be structured to avoid these potential pitfalls—for example, by explaining the limitations of the data and using clearly defined terms and readily understood tables and graphs to convey information. In the past decade, CMS has made several efforts to improve the intake and investigation of nursing home complaints by state survey agencies, including (1) implementation of a database that not only helps state survey agencies track complaints but also helps CMS monitor the state survey agencies’ performance and (2) establishment of and refinements to its performance standards related to nursing home complaints. However, our review indicates that challenges remain. CMS’s complaint data have limitations. We found that the lack of consistency in state surveys agencies’ use of the database—particularly in terms of which complaints are entered and how certain fields are interpreted—undermines the reliability of some of the data and limits the usefulness of the database as a monitoring tool. CMS does not routinely use the data to calculate measures such as substantiation rates that could enhance its understanding of complaint investigations partly because of concerns about the reliability of the data. CMS’s performance reviews highlight state workload issues. Although state survey agencies generally prioritized nursing home complaints in accordance with CMS’s performance standard, we found that many agencies had difficulty managing a heavy workload of actual harm-high complaints. In 2009, state survey agencies prioritized 45 percent of the more than 53,000 nursing home complaints they received as actual- harm high, which requires initiation of an investigation within 10 working days of prioritization. In fiscal year 2009, 19 state survey agencies failed to meet the CMS timeliness standard for these complaints. Staffing shortages and heavy workloads were cited as key reasons by survey agency officials we interviewed whose states had failed this standard. CMS’s policy for scoring the prioritization standard may contribute to these workload issues by creating an incentive for the agency staff who prioritize complaints to assign higher priority levels than are warranted. While CMS is correct in asserting that prioritizing complaints at too high a level is preferable to the reverse, this practice can have a significant impact on state survey agencies’ workload and thus on their ability to meet requirements for timely investigations. Additionally, CMS data for 2009 showed that, among investigated complaints prioritized as either immediate jeopardy or actual harm-high, the percentage substantiated with at least one federal deficiency cited was higher if the investigation was initiated within required time frames than if it was not. Though many factors can affect whether complaints are substantiated, including whether there is evidence to support them, considerable variation in substantiation rates, among the states or over time, could indicate potential concerns with state survey agencies’ complaint investigations. Some performance standards scores are unreliable due to small samples and varying interpretations of requirements. CMS has also made efforts to refine its performance standards for nursing home complaints. However, as with the complaints data, scores on some standards are unreliable, because of inadequate sample sizes and varying interpretations of the standards by the CMS regional office officials who conduct the performance reviews. While we recognize that CMS may have opted for small samples for some standards in order to limit the amount of documentation reviewers must examine each year, sample sizes could be increased without increasing reviewers’ workloads if performance on certain standards—those that require document review—were assessed less frequently than once a year. Less frequent reviews could also help address the issue of state survey agencies receiving their final scores and submitting their corrective action plans so far into the next performance cycle that little time remains for them to improve their performance. The credibility of the scores could be further enhanced by ensuring that the standards are consistently interpreted by the CMS regional offices. Clarifying CMS guidance could help in this regard as well as in ensuring that state survey agencies understand their responsibilities with respect to each aspect of the complaint investigation process, including the manner in which investigation results are communicated to complainants. CMS is considering making state survey agencies’ scores on the performance standards publicly available. While we support such a step, we believe that it is important to consider the reliability of data, as well as its comparability over time, when deciding which scores to publish. For such performance reports to be useful to the public, they should also include meaningful trend data that reflect agencies’ actual progress over time, as well as a clear explanation of the limitations of the data. To ensure that information entered into CMS’s complaints database is reliable and consistent, we recommend that the Administrator of CMS: Identify issues with data quality and clarify guidance to states about how particular fields in the database should be interpreted, such as what it means to substantiate a complaint. To strengthen CMS’s assessment of state survey agencies’ performance in the management of nursing home complaints, we recommend that the Administrator of CMS take the following three actions: Conduct additional monitoring of state performance using information from CMS’s complaints database, such as additional timeliness measures. Assess state survey agencies’ performance in certain areas—specifically, documentation of deficiencies, prioritization of complaints, and quality of investigations—less frequently than once a year. Assure greater consistency in assessments by identifying differences in interpretation of the performance standards and clarifying guidance to state survey agencies and CMS regional offices. To strengthen and increase accountability of state survey agencies’ management of the nursing home complaints process, we recommend that the Administrator of CMS take the following three actions: Clarify guidance to the state survey agencies about the minimum information that should be conveyed to complainants at the close of an investigation. Provide guidance encouraging state survey agencies to prioritize complaints at the level that is warranted, not above that level. Implement CMS’s proposed plans to publish state survey agencies’ scores but limit publication to those performance standards that CMS considers the most reliable and clear. We received written comments on a draft of this report from HHS and from the Association of Health Facility Survey Agencies (AHFSA), the organization that represents state survey agencies. HHS provided written comments, which are reproduced in app. III. HHS generally concurred with all of our recommendations. With respect to our first recommendation, HHS agreed that CMS should take steps to ensure that information entered into the agency’s complaints database is reliable and consistent. HHS said that CMS will convene a workgroup—including staff from CMS headquarters, CMS regional offices, and state survey agencies—to address data quality issues. HHS also agreed that CMS needs to strengthen its assessment of state survey agencies’ performance in the management of nursing home complaints. HHS said that CMS’s planned workgroup will review the three specific actions we recommended and identify ways to strengthen the agency’s oversight process. Finally, HHS agreed that CMS needs to strengthen and increase accountability of state survey agencies’ management of the nursing home complaints process. Regarding the specific actions we recommended, HHS said that CMS will provide increased guidance to states regarding the minimum information that must be conveyed to complainants at the close of an investigation and provide clarification and guidance to ensure that complaints are prioritized at the appropriate level. With respect to our recommendation that CMS publish state survey agencies’ scores on certain nursing home complaint performance standards, HHS said that CMS will work with state officials and others to identify key information about state survey agencies’ performance that would be of public value. HHS also provided technical comments, which we incorporated as appropriate. AHFSA emphasized the critical importance of enforcing federal and state survey and certification standards and noted that in many states, complaint systems have significant connections to state and local licensing and enforcement activities, which are outside CMS’s jurisdiction. AHFSA noted that several of the policy and operational issues raised in our report create challenges for states. These include lack of clarity about what it means to substantiate a complaint and lack of timely notification to the states of any changes in CMS’s performance standards for nursing home complaints. AHFSA also commented that CMS’s guidance on prioritizing complaints could be improved but questioned whether many states were prioritizing complaints at a higher level than is warranted in order to meet CMS’s prioritization standard. In addition, AHFSA said that the complaint system is the primary safety net for vulnerable nursing home residents and therefore suggested that states should err on the side of caution when prioritizing complaints in order to better protect residents. AHFSA also provided some state-specific comments, which we incorporated as appropriate. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the Secretary of Health and Human Services, the Administrator of the Centers for Medicare & Medicaid Services, and other interested parties. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. This appendix provides additional information on the number of complaints received, investigated, and substantiated by all 50 state survey agencies and the survey agency for the District of Columbia for 2009 based on complaints in Centers for Medicare & Medicaid Services’ (CMS) national complaints database. We included only complaints and excluded facility-reported incidents, which nursing homes are required to self-report to state survey agencies. Additionally, we included only complaints that alleged a violation of federal requirements. In the course of our work, we found some limitations to the data we obtained, including that state survey agencies interpret certain variables, such as substantiation, differently from one another and that data are missing for certain variables, such as the date on which the state survey agency acknowledged the complaint. Additionally, we learned that CMS’s national database may not include all complaints because the state survey agencies may not have entered all of the complaints they received. Because of the data limitations we found, we included in our analysis only those variables that we found to be reliable, and we consider the number of complaints from CMS’s national complaints database to be a conservative estimate of the total number of complaints received by state survey agencies. Score (Percent) Pass (√) or fail (X) Score (Percent) Pass (√) or fail (X) Score (Percent) Score (Percent) Pass (√) or fail (X) Score (Percent) Pass (√) or fail (X) Score (Percent) √= passed performance standard X= failed performance standard. A blank in the score column indicates that the state received a passing score (at least 90 percent for the prioritization of complaints standard and at least 95 percent for the timeliness of investigation standards for immediate jeopardy and actual harm-high complaints). Pennsylvania officials reported that the state did not pass the prioritization of complaints standard because it required all complaint investigations to be initiated within 48 hours and survey agency staff therefore assigned a priority level of immediate jeopardy to nearly all complaints. Because CMS guidance on this standard was not clear in fiscal year 2009, the CMS regional office that assessed Pennsylvania’s performance considered complaints assigned a priority level higher than warranted to be inappropriately prioritized and therefore gave the state a failing score on this standard. In addition to the contact name above, Walter Ochinko, Assistant Director; Jennie Apter; Shaunessye Curry; Christie Enders; Nancy Fasciano; Dan Lee; Lisa Motley; Matthew Rae; and Jessica Smith made key contributions to this report. Nursing Homes: Complexity of Private Investment Purchases Demonstrates Need for CMS to Improve the Usability and Completeness of Ownership Data. GAO-10-710. Washington, D.C.: September 30, 2010. Poorly Performing Nursing Homes: Special Focus Facilities Are Often Improving, but CMS’s Program Could Be Strengthened. GAO-10-197. Washington, D.C.: March 19, 2010. Nursing Homes: Addressing the Factors Underlying Understatement of Serious Care Problems Requires Sustained CMS and State Commitment. GAO-10-70. Washington, D.C.: November 24, 2009. Nursing Homes: Opportunities Exist to Facilitate the Use of the Temporary Management Sanction. GAO-10-37R. Washington, D.C.: November 20, 2009. Nursing Homes: CMS’s Special Focus Facility Methodology Should Better Target the Most Poorly Performing Homes, Which Tended to Be Chain Affiliated and For-Profit. GAO-09-689. Washington, D.C.: August 28, 2009. Medicare and Medicaid Participating Facilities: CMS Needs to Reexamine Its Approach for Funding State Oversight of Health Care Facilities. GAO-09-64. Washington, D.C.: February 13, 2009. Nursing Homes: Federal Monitoring Surveys Demonstrate Continued Understatement of Serious Care Problems and CMS Oversight Weaknesses. GAO-08-517. Washington, D.C.: May 9, 2008. Nursing Home Reform: Continued Attention Is Needed to Improve Quality of Care in Small but Significant Share of Homes. GAO-07-794T. Washington, D.C.: May 2, 2007. Nursing Homes: Efforts to Strengthen Federal Enforcement Have Not Deterred Some Homes from Repeatedly Harming Residents. GAO-07-241. Washington, D.C.: March 26, 2007. Nursing Homes: Despite Increased Oversight, Challenges Remain in Ensuring High-Quality Care and Resident Safety. GAO-06-117. Washington, D.C.: December 28, 2005. Nursing Home Quality: Prevalence of Serious Problems, While Declining, Reinforces Importance of Enhanced Oversight. GAO-03-561. Washington, D.C.: July 15, 2003. Nursing Homes: Public Reporting of Quality Indicators Has Merit, but National Implementation Is Premature. GAO-03-187. Washington, D.C.: October 31, 2002. Nursing Homes: Federal Efforts to Monitor Resident Assessment Data Should Complement State Activities. GAO-02-279. Washington, D.C.: February 15, 2002. Nursing Homes: Sustained Efforts Are Essential to Realize Potential of the Quality Initiatives. GAO/HEHS-00-197. Washington, D.C.: September 28, 2000. Nursing Home Care: Enhanced HCFA Oversight of State Programs Would Better Ensure Quality. GAO/HEHS-00-6. Washington, D.C.: November 4, 1999. Nursing Home Oversight: Industry Examples Do Not Demonstrate That Regulatory Actions Were Unreasonable. GAO/HEHS-99-154R. Washington, D.C.: August 13, 1999. Nursing Homes: Proposal to Enhance Oversight of Poorly Performing Homes Has Merit. GAO/HEHS-99-157. Washington, D.C.: June 30, 1999. Nursing Homes: Complaint Investigation Processes Often Inadequate to Protect Residents. GAO/HEHS-99-80. Washington, D.C.: March 22, 1999. Nursing Homes: Additional Steps Needed to Strengthen Enforcement of Federal Quality Standards. GAO/HEHS-99-46. Washington, D.C.: March 18, 1999. California Nursing Homes: Care Problems Persist Despite Federal and State Oversight. GAO/HEHS-98-202. Washington, D.C.: July 27, 1998.

Summary: CMS, the agency within HHS that manages Medicare and Medicaid, contracts with state survey agencies to investigate complaints about nursing homes from residents, family members, and others. CMS helps assure the adequacy of state complaint processes by issuing guidance, monitoring data that state survey agencies enter into CMS's database, and annually assessing performance against specific standards. Concerns have been raised about the timeliness and adequacy of complaint investigations and CMS's oversight. GAO examined (1) complaints received, investigated, and substantiated by state survey agencies; (2) whether those agencies were meeting CMS performance standards and other requirements; and (3) the effectiveness of CMS's oversight. In addition to analyzing CMS data on complaints and performance reviews, GAO examined CMS guidance and conducted interviews with officials from three high- and three low-performing state survey agencies and their CMS regional offices. GAO addressed data reliability concerns by reporting only data we determined to be reliable. CMS's complaints data showed that state survey agencies received 53,313 complaints about nursing homes in 2009. The number and types of complaints varied among states. For example, 11 states received 15 or fewer complaints per 1,000 nursing home residents while 14 states received more than 45. State survey agencies assess the severity of a complaint and assign a priority level, which dictates if and when an investigation must be initiated. About 10 percent of complaints were prioritized as immediate jeopardy, requiring investigation within 2 working days of receipt, while 45 percent were prioritized as actual harm-high, requiring investigation within 10 working days of prioritization. State survey agencies investigated all but 102 complaints that required an investigation. Among investigated complaints, 19 percent were substantiated and resulted in the citation of at least one federal deficiency. The percentage of immediate jeopardy and actual harm-high complaints that were substantiated with at least one federal deficiency cited was higher if the investigation was initiated on time. In CMS's performance assessment for fiscal year 2009, many state survey agencies had difficulty meeting some of CMS's nursing home complaint standards, most of which also assess performance with regard to incidents--specific care issues that nursing homes are required to report. In particular, 19 state survey agencies had difficulty investigating actual harm-high complaints and incidents within the required time frame. However, most states were able to meet other CMS standards--timely investigation of immediate jeopardy complaints and incidents and appropriate prioritization of complaints and incidents. Although CMS's performance assessment does not review state survey agencies' communication with complainants, CMS does expect the agencies to convey investigation findings according to CMS guidelines. GAO found state survey agencies had varied interpretations of those guidelines, and some provided limited information to complainants. CMS's oversight of state survey agencies' complaint investigation processes, through its performance standards system and complaints database, is hampered by data reliability issues. While CMS's performance standards are consistent with certain key criteria for performance measures identified by GAO and other audit agencies, performance scores are not always reliable, due in part to inadequate sample sizes and inconsistent interpretation of some standards by CMS reviewers. In addition, CMS has not made full use of the information it collects. For example, in part because of data reliability concerns, CMS does not routinely use data from the complaints database to calculate certain measures that could enhance its understanding of agencies' performance. Although CMS requires state survey agencies that fail performance standards to develop corrective action plans, states' plans do not necessarily address the underlying causes of performance issues, such as staffing shortages. GAO recommends that the CMS Administrator take several steps to strengthen oversight of complaint investigations, such as improving the reliability of its complaints database and clarifying guidance for its state performance standards to assure more consistent interpretation. HHS generally agreed with our recommendations.