Health care quality measures are standard, evidence-based metrics designed to assess the performance of health care providers, such as hospitals, in providing care. These measures are intended to (1) inform providers about opportunities for potential improvements in their delivery of care, (2) incentivize providers to consistently provide high quality care, and (3) inform consumers about which providers are most likely to deliver high quality care. There are broad categories of clinical quality measures that address various aspects of quality of care. See table 1 for a description of these broad categories of quality measures. These broad measure categories can be further broken down into more specific groups of related measures. For example, outcome measures can include measures of patient safety, such as the incidence of healthcare-associated infections (HAI) or complications, as well as measures of hospital readmissions and mortality, and results obtained from ambulatory care, such as the proportion of patients with hypertension whose blood pressure is reduced to the normal range. The data used to calculate the results of health care quality measures can come from a number of different sources. Some measures often require detailed clinical information obtained from patient medical records, such as process measures that indicate whether timely and effective care was provided in a specific situation, for example, or whether stroke patients received clot-dissolving medication appropriately. Other measures are designed to use information on patient demographics and diagnoses that can be obtained from more readily accessible sources, such as claims data or other administrative data that have already been collected for other purposes such as billing. In addition, patients can be asked directly, usually through surveys, to report on their experiences receiving care. One key method for disseminating information on health care quality measures to consumers—including which providers are delivering high or low quality care and the costs of care—is through websites that can convey this information to anyone with internet access. HHS is one of the organizations that provide such information to consumers and others. Specifically, HHS’s Centers for Medicare & Medicaid Services (CMS) maintains a series of websites that provide information on health care quality, including a series of websites for hospitals, nursing homes, and certain other providers that participate in the Medicare program. Since 2005, CMS has increased the number of health care quality measures it posts on one of its websites, known as Hospital Compare, that covers more than 4,000 hospitals that participate in the Medicare program. These hospitals supply data to CMS for quality measures of inpatient and outpatient care in return for higher payments on their Medicare claims. Each year HHS goes through a formal process, including receiving input from experts and stakeholders, to review and revise the mix of quality measures that these hospitals report. The Hospital Compare website allows anyone with internet access to select up to three hospitals to compare their performance on each of these measures side-by-side. CMS uses contractors to collect and process the data submitted by individual hospitals, and it posts the results on each of the quality measures on the Hospital Compare website. In implementing a requirement under the Choice Act, VA reports some of the 79 possible health care quality measures reported by non-VA hospitals on HHS’s Hospital Compare website. As of June 2017, VA was reporting 35 measures that it had determined were applicable to its individual medical centers. Of the remaining 44 measures that non-VA hospitals report on Hospital Compare, VA plans on reporting on 12 measures in future years and does not plan to report 32 measures. See table 2 for a summary of VA’s reporting of quality measures on Hospital Compare. The 35 measures VA reported on Hospital Compare as of June 2017 include all of the available patient experience measures (e.g., patient perspectives on how well physicians communicated with them), most of the process measures (e.g., whether stroke patients received appropriate clot-dissolving drugs), and some of the outcome measures (e.g., whether pneumonia patients were readmitted to the hospital within 30 days). (See app. I for more detailed information on the specific measures VA reports on Hospital Compare.) VA officials told us that when they began to report quality measures to Hospital Compare in 2010, the measures they first decided to report included measures that VA was already reporting for other purposes. For example, these included process measures of timely and effective inpatient care that VA reported for Joint Commission hospital accreditation. According to VA officials, choosing quality measures it already reported for other purposes minimized the additional resources needed to report the measures on Hospital Compare. From 2017 through 2019, VA plans on reporting an additional 11 outcome measures on Hospital Compare. These include 6 measures of various HAIs at VAMCs, 4 measures of mortality and readmission rates associated with additional medical conditions and procedures, such as stroke and hip/knee replacement surgery, and 1 measure related to patient safety. Specifically: Regarding the 6 additional HAI measures, VA officials told us that they will report these measures when they can develop a new data collection process that will allow them to meet the requirements for reporting HAI measures in a way that minimizes demands on VA resources. According to VA officials, complying with the existing process for collecting data for and reporting on HAIs would require infection control staff at each VAMC to fill out forms with information on individual patients, whereas VA currently collects its own information on HAIs based on aggregated data assembled by VA Central Office staff. VA officials told us that their new data collection process for reporting HAI measures is intended to meet those Hospital Compare requirements through automating much of the required data entry at the VAMCs. However, VA officials noted that this new reporting process is still in an early stage of development, and they expressed uncertainty about how long it would take to implement the process. Regarding the 4 additional mortality and readmission measures, CMS faces challenges in integrating VA’s clinical information into the Hospital Compare database. According to both VA and CMS officials, VA has had to implement new readmissions and mortality rate measures incrementally due to the limited capacity of one of CMS’s Hospital Compare contractors to develop the programming needed to calculate these measures using VA patient data. Because the readmissions and mortality rate measures require complex programming to implement risk adjustments, based on a number of different diagnoses recorded in patient medical records over time, VA and CMS officials agreed to add only two new measures each year until VA can report all 4 measures. There are another 32 measures non-VA hospitals report on the Hospital Compare website that VA has no plans to report. VA officials determined that these measures are not relevant for VA’s health care system, given its distinctive funding sources, patient population, and health care delivery structure. For example, VA officials stated that they did not plan to report any of the Hospital Compare measures related to costs of care, such as Medicare spending per beneficiary, because those measures are based on Medicare payments and VA does not receive any payments from Medicare. In other cases, VA officials do not plan on reporting measures that relate to health care services that VAMCs rarely, if ever, provide, such as obstetrics and hip and knee replacements. In implementing a requirement under the Choice Act, VA has posted on the Hospital Compare website a link to the notification on the quality measures VA is not reporting. The most recent notice, dated February 2015, broadly explains that VA faces three main challenges that affect the availability of certain measures. According to the notice, these challenges relate to data quality, standard data collection processes, and funding to support the collecting and reporting of quality measures. VA officials told us that they plan on adding to the notification information about the specific measures VA expects to report in the future and when it expects to report them. As of June 2017, VA reported on its own website 110 health care quality measures for its VAMCs. These include measures in many of the same categories that VA reports on Hospital Compare, but they also include several additional categories of quality measures that are not available on Hospital Compare. These additional categories address quality issues VA officials deem relevant for veterans, such as various measures of access to care. For example, the additional measures include measures of how long veterans must wait to obtain care at VAMCs and measures of the quality of care related to ambulatory care, such as colorectal cancer screening rates. See Table 3 for the categories of measures VA reports on its website. Some of the specific measures reported on VA’s website are the same as those reported on Hospital Compare and some are entirely different. For example, most of the patient experience measures reported on VA’s website are the same as those reported on Hospital Compare, while the access to care measures are only reported on VA’s website. In addition, some measures reported on VA’s website are similar to, but not exactly the same as, Hospital Compare measures. For example, among similar measures, the target population may be defined somewhat differently or the result may be calculated differently. (See app. I for more information on the specific measures VA reports on its website.) In implementing a requirement under the Choice Act, VA has also posted a notification of unavailable measures on its website. This is the same February 2015 notice VA provided on the Hospital Compare website. VA publicly reports 110 health care quality measures on two separate webpages on its website—the “Access and Quality” webpage, launched in April 2017, and the “Quality of Care” webpage, which VA has used since 2008. The Access and Quality webpage is intended to be the primary source of information on quality of care at VA for veterans, according to VA officials. VA officials also told us that this webpage was developed to present quality of care information in ways that are easy for veterans and other stakeholders to understand. A link to the primary Access and Quality webpage can be found on the homepage of VA’s website, making it relatively easy to find. VA officials also told us that they retained the older Quality of Care webpage in an effort to be transparent as well as provide historical data on the many quality measures it has tracked. The older Quality of Care webpage is not linked to the homepage of VA’s website. Additionally, neither the primary Access and Quality webpage nor the older Quality of Care webpage provides any link or makes any mention of the other. See fig. 1 and fig. 2, which show the primary Access and Quality and older Quality of Care webpages, respectively. On the two webpages, VA reports its 110 health care quality measures across various subpages, with some measures reported on multiple subpages. Specifically, as of June, 2017: VA reported 15 of its 110 quality measures on the primary Access and Quality webpage, which comprises three subpages. Two subpages focus on measures related to access (which we refer to as the “Wait Times” and “Experience With Access” subpages) and the third subpage compares how VAMCs perform relative to non-VA hospitals in their geographic area on a few selected Hospital Compare measures (which we refer to as the “Non-VA Hospital Comparison” subpage). According to VA officials, the 15 quality of care measures on the primary Access and Quality webpage were selected because they provide information that is useful to veterans in making health care choices. On the older Quality of Care webpage, which comprises four subpages, VA reported 100 quality measures. According to VA officials, each of the four subpages of this webpage was created for a specific purpose and as a result, some measures are reported on multiple subpages. We also found that VA reported five of the same quality measures on both the primary and older webpages, which include measures reflecting patient ratings of their experience in the hospital and healthcare-associated infection rates. VA officials told us that they do not plan on consolidating the information currently reported on the two webpages since they serve different purposes, as explained earlier. See Table 4 for a summary of the health care quality measures reported on VA’s Access and Quality and Quality of Care webpages. We found that VA’s primary Access and Quality webpage generally reports health care quality of care information in ways that are more accessible and understandable than VA’s older Quality of Care webpage. We assessed both webpages using criteria we identified in prior work for evaluating how well websites present information on health care quality to the public. Specifically, we found that VA’s primary webpage meets four of six presentation criteria compared with VA’s older webpage, which met none of the criteria. (See table 5.) We found that VA’s primary webpage does especially well at presenting information on quality of care on the two subpages that focus on access to care. In particular, these subpages: provide information written in plain language with accompanying enable consumers to customize the information presented, so that they can, for example, select the types of medical appointments of interest to them; and allow users to rank order VAMCs by level of performance on a given quality measure. In contrast with VA’s primary webpage, we found that VA’s older Quality of Care webpage displays quality measures in ways that generally do not meet any of the presentation criteria. In particular, we noted that the information presented across the four subpages on the Quality of Care webpage shares the following key limitations: none are written in plain language or use graphics to convey key none summarize related quality information or organize the data to show patterns, such as rank ordering VAMCs on a given performance measure; none enables comparison of multiple VAMCs in one view, but instead requires users to look up information on VAMCs one at a time; none enables customization of how the information is displayed so that users can focus on the quality measures most relevant to them; and none are advertised to potential users, with no indication provided on the VA website homepage that the subpages exist, what information they provide, and where to find them. Representatives from two of the three VSOs we spoke with said that the information on the Quality of Care webpage is not utilized by veterans because links to the webpage are not prominently displayed on the VA website. According to one VSO official we spoke with, when VA sought feedback from the veterans service organizations about its overall plans for its primary Access and Quality webpage, the official found that the main advantage of this webpage is that it presents information in a more visually compelling way that allows veterans to directly compare quality information for multiple VAMCs. According to officials from the three VSOs with whom we spoke, this capability is important because veterans can be overwhelmed by information that is not conveyed in an easily digestible format. We also assessed VA’s webpages using criteria we identified in prior work for evaluating the extent to which websites provide consumers with information relevant for making health care decisions. Using these criteria, we found that VA’s primary Access and Quality webpage does not provide veterans as much relevant information as VA’s older Quality of Care webpage. Our analysis shows that VA’s primary webpage meets only two of the seven relevance criteria. In contrast, we found that VA’s older webpage performs relatively well on the relevance criteria—meeting six of the seven criteria. (See table 6.) As Table 6 shows, VA’s primary webpage does not provide information on a broad range of health care services, nor does it highlight key differences in clinical quality of care. For example, the primary webpage reports only on the incidence of two types of HAIs and not on any other types of outcomes, such as readmissions and mortality rates associated with different medical conditions. Additionally, VA’s primary webpage only provides limited information about key differences in patient experiences and the key strengths and limitations of the data reported. Although VA officials told us that they expect to expand the number of reported measures in future and intend to make the webpage more comprehensive, they did not identify specific measures to be implemented. In contrast, VA’s Quality of Care webpage performs better on the relevance criteria primarily because its four subpages together provide information on 100 different measures spread across a broader range of measure categories (see tables 4 and 6). As a result, it provides information on a broad range of services and on key differences in clinical quality of care. Furthermore, representatives of two veterans service organizations told us that much of the information reported on VA’s older webpage would be relevant to veterans, including rates of hospital readmissions, mortality, and complications—none of which are included on VA’s primary webpage. While VA has improved how it presents information on quality to veterans through its primary Access and Quality webpage, there are still gaps in the relevance of information reported on the webpage. Specifically, while it may not be necessary for VA to report on all 110 measures on its primary webpage to meet the relevance criteria, the 15 measures reported on the Access and Quality webpage (of which 10 focus on access) clearly do not provide the same breadth of relevant information relevant for veterans. VA officials told us that they began with these 15 quality measures that they deemed most useful to veterans for making health care choices, and in the short term, they are focused on improving the user experience of the webpage. However, until VA’s website provides information on a broader range of health care services, highlights key differences in clinical quality of care, and reports this information in a manner that is easily accessible and understandable, VA is missing an opportunity to provide veterans with relevant quality information that it has already collected to help veterans make informed decisions about their care. Studies and other evidence we reviewed indicate potential problems with the completeness and accuracy of the clinical information in patient records that is used to calculate VA’s publicly reported health care quality measures on Hospital Compare and its own website. Moreover, VA Central Office—which has responsibility for calculating and reporting health care quality measures for each VAMC—has not systematically assessed the completeness and accuracy of this clinical information across its VAMCs and the effects, if any, on the accuracy of its health care quality measures. Studies and other evidence we reviewed indicate potential problems with the completeness and accuracy of the clinical information recorded in patient medical records at some VAMCs (e.g., diagnoses given and treatments received). This is significant because the accuracy of VAMCs’ performance on quality measures that are calculated and reported by VA on Hospital Compare and its own website depends on the completeness and accuracy of this clinical information. For example, VA reports readmission measures on Hospital Compare and on its website that compare VAMCs in terms of the proportion of their patients with a medical condition, such as heart failure, who are readmitted to a hospital within 30 days of an initial inpatient stay. Because VAMCs as well as non-VA hospitals that treat healthier patients are likely to have lower readmission rates independent of the quality of care that they provide, these readmission measures incorporate information about whether a hospital’s patients have certain other diagnoses that indicate their overall level of health. However, the calculation of hospital readmission rates will not work as intended if VAMCs do not record in their medical records complete and accurate information on patient diagnoses. In particular, if some VAMCs record that information more completely and accurately than others, that may distort any comparison of the readmission rates among the VAMCs. However, according to an independent assessment of VA’s clinical documentation procedures conducted by McKinsey & Company in 2015, VA as a whole is below industry standards. The independent assessment’s report looked specifically at VA’s implementation of clinical documentation improvement (CDI) programs, which typically provide a combination of provider education and assessments of provider performance to promote more complete and accurate clinical documentation. The independent assessment noted that such CDI programs have been widely adopted by hospitals across the U.S. health care system and play a critical role in producing more complete and accurate clinical information. However, the assessment found that only 62 of 134 VAMCs examined had a CDI program as of 2014. The number of VAMCs with CDI programs has increased since the study was completed; VA officials reported to us that as of July 2017, 99 VAMCs had a CDI program in place, and an additional 11 VAMCs are planning to implement a CDI program. Moreover, the independent assessment found evidence suggesting that deficiencies in the completeness and accuracy of the clinical information recorded in VA patient medical records may have affected the assessed quality performance of at least some VAMCs. Specifically, the independent assessment cited multiple instances where VAMC officials observed that their facility’s assessed performance on VA’s quality measures markedly improved after the facility took steps to improve the completeness and accuracy of the clinical information recorded in the VAMC’s patient records. We could not quantify the potential effect of incomplete and inaccurate clinical information on VAMC assessed quality performance without conducting an audit of the clinical information recorded at each VAMC. Other components of the independent assessment also highlighted longstanding issues with VA’s electronic health records (EHR) system, which stores the clinical information used to calculate VAMC performance on the quality measures. The summary report for all 12 independent assessments noted that VA’s EHR system is outdated. The EHRs used at each VAMC do not consistently use standard data elements and algorithms for recording clinical information, and this variability in underlying clinical information from one VAMC to the next has led to an inability to convey consistent and complete information across the VA health care system. In 2016, the Commission on Care determined that the deficiencies in VA’s clinical documentation were so pervasive that it recommended that VA procure an entirely new EHR system that would replace its increasingly obsolete EHR system. These findings are similar to other studies conducted by the VA Office of Inspector General (OIG) that found deficiencies in VAMCs’ operations that can lead to the recording of incomplete and inaccurate patient information. For example, in July 2017, a VA OIG investigation of colonoscopy practices concluded that more accurate and stringent clinical data collection would enable VA to improve its monitoring of the quality of providers’ colonoscopies. Additionally, in May 2012, the OIG found that some VAMCs do not consistently conduct reviews intended to ensure that their clinicians properly enter information into EHRs, including ensuring the appropriate use of cut and paste functions. In April 2010, the OIG also found that VAMC emergency departments do not consistently document all of the information that they are required to report when transferring patients to other facilities. Furthermore, as one VA official observed to us, because of the way in which they are funded, VAMCs do not have the same financial incentives that non-VA hospitals do to ensure that the clinical information stored in patient medical records is complete and accurate. In addition, because most of VA health care is funded through direct appropriations, rather than by payments of claims filed with an insurance company or with government programs such as Medicare, VA does not routinely produce the claims-based data on health care services provided to patients that non-VA hospitals and clinics typically generate in the course of doing business. Thus, to calculate VAMC performance on quality measures, VA officials told us that VA has to extract data from its patient medical records specifically for this purpose. In contrast, non-VA hospitals routinely collect this information—such as data on readmissions and mortality—as part of their claims processing and reimbursement from Medicare and other health care payers. Within VA, VA Central Office is responsible for calculating and reporting the health care quality measures that VA publicly reports for each of its VAMCs and ensuring that these measures provide accurate information on the quality of care at these facilities. However, while studies and other evidence we reviewed indicate potential problems with the clinical information recorded at some VAMCs, VA Central Office has not determined the extent to which these problems exist across VAMCs and adversely affect the accuracy of the quality measures VA publicly reports on Hospital Compare and its own website. Specifically, VA Central Office has not conducted a systematic assessment of the completeness and accuracy of the clinical data recorded in VA patient medical records across VAMCs. (There are models for this type of assessment; see app. III for examples of methodologies that have been applied to different types of medical record systems.) When asked why they have not conducted such an assessment, VA Central Office officials told us that they have focused instead on improving the accuracy of medical coding and on specific clinical quality issues identified at individual VAMCs. However, accurate medical coding depends on the completeness and accuracy of the underlying clinical information in patient records. Moreover, VA Central Office policy assigns responsibility for monitoring the completeness and accuracy of patients’ clinical information to each individual VAMC. Each VAMC has a Health Record Review Committee that is charged with monitoring the clinical documentation practices at that facility. This committee determines the frequency of medical record reviews at its facility, decides what focused reviews should occur, and identifies clinicians that record clinical information poorly until they improve to an acceptable rate of completeness and accuracy. According to officials in VA’s Central Office who are responsible for health information management, the results of individual VAMC Health Record Review Committee reviews are not reported to VA Central Office. As a result, VA Central Office lacks information that could help it systematically determine whether or to what extent VAMCs’ clinical information is incomplete and inaccurate, and if so, the extent to which these deficiencies affect VAMCs’ reported performance on the quality measures VA publicly reports. The results of such a systematic analysis could also help identify the deficiencies, if any, in the recording of patient clinical information and what steps, if any, VA Central Office may need to take to address them. VA Central Office’s lack of a systematic assessment of the completeness and accuracy of clinical information recorded in patient medical records and the extent to which this affects the accuracy of its quality measures is inconsistent with federal standards for internal controls related to information and monitoring. These standards call for agencies to use accurate information to achieve objectives, and to monitor agency activities and evaluate results. Without a systematic assessment of the completeness and accuracy of the clinical information recorded in VAMC patient medical records and their potential effects on the health care quality measures VA reports, VA Central Office does not have reasonable assurance that differences in VAMCs’ performance on VA’s quality measures reflect true differences in the quality of care and not differences in the accuracy and completeness of the underlying clinical information. This may hinder VA Central Office’s ability to appropriately assess VAMC performance and offer accurate publicly available information on its website to veterans so that they can make informed choices about their care. VA uses Hospital Compare and its website to provide veterans with information on how VAMCs perform on a range of health care quality measures. By providing information on the quality of care at VA facilities, the quality measures are intended to help veterans make informed decisions about their care. However, two key limitations in VA’s efforts may hinder veterans’ ability to use the information VA provides to make informed decisions about their health care. First, while VA’s primary Access and Quality webpage provides generally accessible and understandable information on the quality of VA health care, the breadth of information it provides is too limited for veterans to make informed choice about their care. VA intends for this webpage to be veterans’ primary source of information on the quality of care at VAMCs, but the 15 measures reported on the webpage represent only a small subset of the 110 measures and only a few of the measure categories VA makes available elsewhere on its website. Until VA can provide information on a broader range of health care measures and services, highlight key differences in the quality of clinical care at VAMCs, and present this information in a way that is easily accessible and understandable, VA cannot ensure that its website is functioning as intended in helping veterans make informed choices about their care. Second, VA does not have reasonable assurance that the health care quality measures it reports on Hospital Compare and its own website accurately assess the relative performance of VAMCs. This is because the quality measures are calculated using clinical information recorded in patient medical records, but VA Central Office does not know to what extent this information has been accurately and completely recorded across VAMCs. This is because VA has not conducted a systematic assessment and does not have reasonable assurance about the accuracy of the information that its quality measures are based on. Therefore, VA lacks reasonable assurance that the quality measures reported on Hospital Compare and its own website provide accurate information to veterans so they can make informed choices about their care. We are making two recommendations to the Department of Veterans Affairs. We recommend that the Undersecretary for Health take additional steps to ensure that VA’s website reports health care quality measures that cover a broad range of health care services, highlights key differences in the clinical quality of care, and presents this information in an easily accessible and understandable way (Recommendation 1). We recommend that the Undersecretary for Health direct VA Central Office to conduct a systematic assessment of the completeness and accuracy of patient clinical information across VAMCs that is used to calculate the health care quality measures VA reports and address any deficiencies that affect the accuracy of these measures (Recommendation 2). We provided a draft of this report to VA and HHS for review and comment. HHS had no comments on this report. VA provided written comments, which we have reprinted in appendix IV, and provided technical comments, which we have incorporated as appropriate. In its comments, VA concurred with our first recommendation to take additional steps to ensure that VA’s website reports health care quality measures that cover a broad range of health care services, highlights key differences in the clinical quality of care, and presents this information in an easily accessible and understandable way. VA provided additional information about its plans to expand the quality measures reported on its Access and Quality webpage, including information on which additional measures it was planning to add to the webpage and the steps it planned to take to continue to enhance the presentation of this information for veterans. VA indicated that its target completion date for these activities was December 2017. In addition, VA concurred in principle with our second recommendation for VA Central Office to conduct a systematic assessment of the completeness and accuracy of patient clinical information across VAMCs that is used to calculate the health care quality measures VA reports and address any deficiencies that affect the accuracy of these measures. In its comments, VA acknowledged the importance of improving the reliability of data used to calculate its health care quality measures and described its plans to establish a workgroup under the Deputy Under Secretary for Health for Organizational Excellence to determine the best approach for conducting a systematic assessment of the completeness and accuracy of the patient clinical information across VAMCs. In its response, VA described the workgroup’s focus on determining whether current validation processes ensure completeness and accuracy and stated that this determination would be completed by December 2017. In conjunction with reviewing these validation processes, VA should also assess the completeness and accuracy of clinical information recorded in patient records that is used to calculate the quality measures. This will allow VA to determine the extent to which the quality measures it reports accurately reflect VAMCs’ performance in delivering care to veterans. We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or williamsonr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V. This appendix presents the specific health care quality measures that the Department of Veterans Affairs (VA) publicly reports, plans to report, or does not report on the Department of Human and Health Services’ (HHS) Hospital Compare website. HHS’s Hospital Compare website provides information on the quality of care at non-VA hospitals and VA Medical Centers (VAMCs). This information is captured across various quality measures covering topics such as patient experience. Additionally, this appendix presents the specific health care quality measures that VA reports on its own website about the quality of care at VAMCs. Some of the specific measures reported on the Hospital Compare website are the same as those reported on VA’s website and some are entirely different. In the table below, measures listed on the same row (e.g., patient experience measures) are measures that we determined were the same, based on information we obtained from VA and HHS. Measures that are grouped together but listed on different rows may address similar quality issues (e.g., hospital readmissions) but are calculated differently. Additionally, a dash in a table cell indicates that the measure listed on that row is not reported on the webpage in question. See Table 7. The VA website locations refer to subpages of VA’s two webpages with quality measures. They are coded as: Access and Quality webpage: (1) Wait Times; (2) Experience with Access; and (3) Non-VA Hospital Comparison. Quality of Care webpage: (1) VA Quality Scores (Strategic Analytics for Improvement and Learning (SAIL)); (2) MCP--Medical Center Performance; (3) WNTB--Why Not the Best; and (4) PT EXP--Patient Experience on the Quality of Care. The 13 criteria we used to assess the presentation and relevance of information on quality of care reported on VA’s primary Access and Quality and older Quality of Care webpages were identified as part of GAO’s prior work on health care transparency. These criteria identify the key characteristics that make websites effective in communicating information on health care quality of care to consumers. We used a four- scale rating system—yes, no, limited or very limited—to determine if each criterion had been met. A “limited” rating indicates that the webpage has discrete areas where the webpage has implemented the characteristic to some degree, but those areas are not representative of the webpage as a whole. A rating of “very limited” indicates that the webpage has largely not implemented the characteristic (with a few exceptions). Two analysts applied the criteria separately and reconciled any differences. Two external experts we also consulted generally agreed with our assessments of the presentation and relevance of VA’s publicly reported quality measures on its website. Six of the 13 characteristics of effective consumer websites focus on the extent to which a website presents its information in a way that enables the consumer to grasp and interpret it. Specifically, the research we reviewed shows that more effective websites: 1. Use plain language with clear graphics. Effective consumer websites use labels and descriptions that make sense to consumers who typically are unfamiliar with clinical terminology and who often have difficulty interpreting numerical information. Graphics, including symbols, can help to readily convey information on relative provider performance, especially when they are designed to display a summary assessment of that performance as part of the symbol itself, for example one that incorporates the words “superior” or “poor.” 2. Explain purpose and value of quality performance ratings to consumers. Effective consumer websites address prevalent misleading preconceptions by providing consumers coherent explanations of how different quality measures relate to the aspects of quality that consumers find relevant. These explanations work best when they link individual measures to overarching categories indicating what is being achieved, such as effectiveness of care, safety, or patient-focused care. 3. Summarize related information and organize data to highlight patterns and facilitate consumer interpretation. Two techniques that consumer websites can use to help consumers make sense of large amounts of information are (a) combining information from multiple related measures into summary or composite scores, and (b) structuring presentation of the data in ways that make patterns evident. For example, listing providers in rank order on selected cost and quality dimensions greatly simplifies identification of high and low performers. 4. Enable consumers to customize information selected for presentation to focus on what is most relevant to them. Consumers differ in the priority they assign to different aspects of quality. Websites that enable consumers to customize which quality information is presented help consumers filter out information of lesser consequence to them, and hone in on the information that they find most compelling. For example, one consumer may choose to focus on providers’ capacity to communicate well with patients, while another may focus on providers’ rates of complications and infections. 5. Enable consumers to compare quality performance of multiple providers in one view. Websites are most effective when they present side-by-side assessments of providers’ performance on a given aspect of cost or quality, so consumers can most easily compare providers. 6. Enable easy use and navigation of the tool. Unless consumers can quickly find information of interest to them, they are likely to quickly dismiss the potential utility of a consumer website and move on. Extensive testing with consumers can help public and private entities providing websites to develop intuitive, user-friendly approaches to website navigation and for manipulating how the data are presented. Seven of the 13 characteristics of effective websites we identified address the extent to which a website provides substantive quality and cost information of relevance to consumers. Specifically, the research we reviewed shows that more effective websites: 1. Review a broad range of services so that more consumers’ particular needs are included. The more services that are covered by the website, the more likely it is that the website will have information relevant to the particular services of interest to any given consumer. It is especially important to include services that are predictable and non-urgent, because these services are most likely to afford consumers the opportunity to evaluate cost and quality information before receiving the service. 2. Cover a broad range of providers. Websites that provide information for all or most of the available providers in a given geographic area, regardless of network status or practice setting, give consumers more information about their full range of options. For example, for procedures that can be conducted in either a hospital outpatient department or ambulatory surgical center, it helps consumers to provide comparable information for both settings, so that consumers can choose from a larger number of providers that offer those procedures. 3. Describe key differences in clinical quality of care, particularly patient-reported outcomes. Assessments of the clinical quality of care that have been shown to have particular relevance to consumers are those that relate to long-term outcomes of the care experienced by other patients. Often this is best addressed by patient-reported outcomes, which tell consumers the eventual outcome of treatments, as reported by previous patients of a particular provider. For example, patients receiving hip replacements can be asked, through such patient-reported outcomes, to rate their ability to climb stairs both before and after their procedures, which enables assessments of the procedures’ effects on patients’ mobility. 4. Describe key differences in patient experiences with providers. Another outcome that matters is patients’ assessment of their interactions with providers. Effective websites include information on how past patients have evaluated providers in terms of dimensions such as how well nurses communicate with patients, or the responsiveness of clinicians to patients’ needs. 5. Describe other information related to quality, where appropriate. There may be other quality indicators that could have major significance to consumers for certain types of services. For example, facility inspection results and staffing levels are of particular relevance to nursing home care. 6. Provide timely Information. More recent data are intrinsically more relevant than data that are several years old. Because consumer websites necessarily rely on past data to assess likely cost and quality performance in the future, some lag in collecting, analyzing, and providing data is inevitable. Data that are no more than two years old are generally considered timely. 7. Describe key strengths and limitations of the data. Although the research we reviewed shows that few consumers are inclined to delve into the many methodological issues that concern appropriate techniques for collecting, checking, and analyzing cost and quality data, effective websites can provide both summary assessments of strengths and limitations for most consumers, and links to more complete explanations for those wanting to pursue these issues in greater detail. Such information, along with identification of the organization responsible for the website, provides consumers a basis to judge the credibility of the cost and quality information provided. In recent years, a number of researchers have examined methods for assessing the quality of clinical information recorded in electronic medical records, including data completeness and accuracy. A wide variety of approaches have been proposed and used, depending on the clinical focus of the research and the institutional context in which the clinical information is recorded. The following provides some examples drawn from this body of research. 1. Nicole Gray Weiskopf and Chunhua Weng, “Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research,” Journal of the American Medical Informatics Association, vol. 20, no. 1 (2013), 144–151. This article identifies a range of different tests that had been applied by various researchers to determine both data completeness and data accuracy in electronic patient records. Most of these tests involved comparison with another data source, such as paper records, patient interviews, or other alternative data sources, that served as a “gold standard” for purposes of assessing the information recorded in the electronic medical record. Other tests involved comparison between different pieces of information recorded in the electronic medical record. 2. Philip J.B. Brown and Victoria Warmington, “Data quality probes— exploiting and improving the quality of electronic patient record data and patient care,” International Journal of Medical Informatics, vol. 68 (2002), 91-98. The article describes the use of “data quality probes” to assess data quality by matching the result of database queries relative to established clinical knowledge for a given condition. 3. Edwin R Faulconer and Simon de Lusignan, “An eight-step method for assessing diagnostic data quality in practice: chronic obstructive pulmonary disease as an exemplar,” Informatics in Primary Care, vol. 12, no. 4, (2004), 243–53. This article describes an eight-step methodology for assessing the completeness and accuracy of diagnoses for a specific condition. 4. Michael G. Kahn, Marsha A. Raebel, Jason M. Glanz, Karen Riedlinger, and John F. Steiner, “A Pragmatic Framework for Single- site and Multisite Data Quality Assessment in Electronic Health Record-based Clinical Research,” Medical Care, vol. 50, no. 7 (2012), S21-S29. This article proposes an approach that prioritizes assessment of selected variables and data quality dimensions; iterative cycles of assessment within and between sites; targeting assessment toward data domains known to be vulnerable to quality problems; and detailed documentation of the rationale and outcomes of data quality assessments to inform data users. It presents a comprehensive set of data quality rules that look for anomalies in data values and distributions as well as inconsistencies across related variables. 5. William R. Hogan and Michael M. Wagner, “Accuracy of Data in Computer-based Patient Records,” Journal of the American Medical Informatics Association, vol. 4, no. 5 (1997), 342–355. This article emphasizes the importance of assessing data completeness and accuracy in conjunction with each other. It recommends adopting a continuous improvement approach to improving data quality based on regular cycles of monitoring, analysis of errors, and interventions to address the factors found to cause errors. In addition to the contact named above, Rashmi Agarwal, Assistant Director; Eric Peterson, Analyst in Charge; Dee Abasute; Krister Friday; Jacquelyn Hamilton; Wati Kadzai, and Vikki Porter made key contributions to this report. VA Health Care: Improvements Needed in Data and Monitoring of Clinical Productivity and Efficiency. GAO-17-480 (Washington, D.C.: May 24, 2017). Federal Health Care Center: VA and DOD Need to Develop Better Information to Monitor Operations and Improve Efficiency. GAO-17-197 (Washington, D.C.: Jan. 23, 2017). Veterans Health Care: Improvements Needed in Operationalizing Strategic Goals and Objectives.GAO-17-50 (Washington, D.C.: Oct. 21, 2016). VA Primary Care: Improved Oversight Needed to Better Ensure Timely Access and Efficient Delivery of Care. GAO-16-83 (Washington, D.C.: Oct. 8, 2015). Health Care Transparency: Actions Needed to Improve Cost and Quality Information for Consumers. GAO-15-11 (Washington, D.C.: Oct. 20, 2014). VA Health Care Management and Oversight of Consult Process Need Improvement to Help Ensure Veterans Receive Timely Outpatient Specialty Care. GAO-14-808 (Washington, D.C.: Sep. 30, 2014).

Summary: To help veterans make informed choices about their care, the Veterans Access, Choice, and Accountability Act of 2014 (Choice Act) directs VA to publicly report applicable health care quality measures for its medical facilities on HHS's Hospital Compare website and on VA's own website. The Choice Act also contains provisions for GAO to review the health care quality measures VA publicly reports. In this report, GAO 1) describes the quality measures VA reports on Hospital Compare and its own website; 2) evaluates VA's reporting of quality measures on its website; and 3) examines the extent to which VA has assessed the accuracy of the quality measures it publicly reports. GAO reviewed the quality measures VA publicly reports, reviewed studies and interviewed VA officials about the accuracy and completeness of the clinical information used to calculate the measures, and assessed the presentation and relevance of VA's information on quality of care using criteria identified in previous GAO work to evaluate health care websites. As of June 2017, the Department of Veterans Affairs (VA) publicly reported 35 health care quality measures on the Hospital Compare website, which is maintained by the Department of Health and Human Services. Veterans can use information on this website to compare the performance of VA medical centers (VAMC) and non-VA hospitals on a common set of quality measures. Those measures include patient reports of their experience of care, such as how well doctors and nurses communicated with them, and actual outcomes of care, such as readmissions to the hospital. On its own website, VA reported 110 quality measures, including some of the same measures reported on Hospital Compare . VA also reports quality measures not found on Hospital Compare , such as measures of how long veterans must wait to access care at VAMCs. VA reports health care quality measures on two separate webpages of its website. VA launched the Access and Quality webpage in April 2017, which according to VA officials is the primary source of information for veterans on the quality of care at VAMCs. GAO found that this information is generally presented in a way that is accessible and easy to understand. However, GAO also found that the primary webpage provides information from a small subset—15—of the 110 measures VA reports on its website as of June 2017. Most of the other measures are available on a second, older webpage that resides elsewhere on VA's website and is generally not easily accessible and understandable. Until VA can provide information on a broader range of health care measures and services and present this information in a way that is easily accessible and understandable, VA cannot ensure that its website is functioning as intended in helping veterans make informed choices about their care. Within VA, VA Central Office is responsible for calculating the health care quality measures that VA publicly reports for each of its VAMCs and ensuring that these measures provide accurate information on the VAMCs' quality of care. However, GAO found that VA Central Office has not systematically assessed the completeness and accuracy of the underlying clinical information that is used to calculate these measures. This clinical information is recorded in veterans' medical records and includes diagnoses given and treatments provided. Several studies have found potential problems with the accuracy and completeness of this clinical information at some VAMCs. For example, a 2015 independent assessment conducted by McKinsey & Company found that VA's clinical documentation procedures are below industry standards and that many VAMCs do not have programs in place to improve clinical documentation practices. VA Central Office officials told GAO that they have not systematically assessed the completeness and accuracy of the clinical information across VAMCs and the extent to which this affects the accuracy of its quality measures because they have focused on other priorities. However, the lack of such an assessment is inconsistent with federal standards for internal controls related to information and monitoring. As a result, VA does not have assurance that the quality measures it publicly reports on Hospital Compare and its own website accurately reflect the performance of its VAMCs and provide veterans with the information they need to make informed choices about their care. GAO recommends that VA 1) report a broader range of health care quality measures in an accessible and understandable way on its website and 2) conduct a systematic assessment of the patient clinical information across VAMCs to ensure its accuracy and completeness. VA concurred with GAO's first recommendation and concurred in principle with the second recommendation, and described steps to implement the recommendations.