VA pays monthly disability compensation to veterans with service- connected disabilities (i.e., injuries or diseases incurred or aggravated while on active military duty) according to the severity of the disability. VBA staff in 57 regional offices process disability compensation claims. These claims processors include Veterans Service Representatives who gather evidence needed to determine entitlement, including VA and military medical records, and Rating Veterans Service Representatives who decide entitlement and the rating percentage. Each claim requires a determination on one or more medical conditions. In fiscal year 2013, VBA decided more than 1 million compensation claims. Since fiscal year 1999, VBA has used the STAR program to review the decisional accuracy of disability compensation claims. Under the STAR program, VBA reviews a stratified random sample of completed claims, and certified reviewers use a checklist to assess specific aspects of each claim. Specifically, for each of the 57 regional offices, completed claims are randomly sampled each month and the data are used to produce estimates of the accuracy of all completed claims. VA reports national estimates of accuracy from its STAR program to Congress and the public through its annual performance and accountability report and annual budget submission. VBA also produces regional office accuracy estimates, which it uses to manage the program. These and its national accuracy rates are reported in a publicly-available performance database, the ASPIRE dashboard. Prior to October 2012, VBA’s estimates of accuracy were claim-based; that is, claims with zero errors were considered accurate and, conversely, claims with one or more errors were considered inaccurate. Beginning in October 2012, VBA also began using STAR data to produce issue-based estimates of accuracy that measure the accuracy of decisions on the individual medical conditions within each claim. For example, a veteran could submit one claim seeking disability compensation for five disabling medical conditions. If VA made an incorrect decision on one of those conditions, the claim would be counted as 80 percent accurate under the issue-based measure. By comparison, under the claim-based measure, the claim would be counted as 0 percent accurate. In March 2014, VBA reported a national estimate of issue-based accuracy in its fiscal year 2015 annual budget submission and plans to update this estimate in VA’s next performance and accountability report. VBA also produces issue- based estimates by regional office, and reports them in the ASPIRE dashboard. For fiscal year 2013, the regional office claim-based accuracy rates ranged from 78.4 percent to 96.8 percent, and the issue-based accuracy rates ranged from 87.0 percent to 98.7 percent. Beyond STAR, VBA has programs for conducting regional office quality reviews and for measuring the consistency of decisions. In March 2012, VBA implemented a quality review team (QRT) at each regional office, consisting of staff dedicated to conducting quality reviews. A QRT conducts individual quality reviews of claims processing staff members’ work, for performance assessment purposes. The QRT also conducts in- process reviews before claims are finalized to help prevent inaccurate decisions by identifying specific types of common errors. Such reviews also serve as learning experiences for staff members. Since fiscal year 2008, VBA has also conducted studies to assess the consistency of disability claims decisions across regional offices. Initially, these studies have included inter-rater reliability studies that assess the extent to which a cross section of claims processors from all regional offices agree on an eligibility determination when reviewing the entire body of evidence from the same claim. In 2013, VBA moved beyond the inter-rater reliability studies and introduced consistency questionnaires as its primary means for assessing consistency. A questionnaire includes a brief scenario on a specific medical condition for which claims processors must correctly answer several multiple-choice questions. When calculating accuracy rates, VBA does not follow generally accepted statistical practices. For example, VBA does not calculate the margin of error associated with each estimate that it generates, which prevents a complete understanding of trends over time and comparisons among offices. In addition, VBA does not weight the results of its STAR reviews to reflect its approach to selecting claims by regional office, which can affect the accuracy of estimates. According to our analysis of VBA data, weighting would have resulted in a small change to VBA’s nationwide claim-based accuracy rate for fiscal year 2013: from 89.5 percent to 89.1 percent. However, 29 of the 57 regional offices’ would have experienced a somewhat greater increase or decrease in their accuracy rates. According to VBA officials we interviewed, although STAR management used a statistician initially to help develop the way in which they measure accuracy, it currently does not use a statistician to, for example, weight STAR results and calculate margins of error for accuracy estimates. Further, VBA officials said they did not consult a statistician when developing the new issue-based accuracy measure, but rather relied on the same sampling methodology and approach for estimating accuracy as for the claim-based measure. We have previously reported that to be useful, performance information must meet users’ needs for completeness, accuracy, consistency, and validity among other factors. In addition, VBA’s accuracy reporting lacks methodological details that would help users understand the distinction between its two accuracy measures and their associated limitations. While VBA’s new issue-based measure provides additional perspective on quality of claims decisions, to date VBA has not fully explained in its public reports how the issue-based and claim-based measures differ. VBA began reporting the issue-based measure in its ASPIRE dashboard in 2013. The issue-based measure tends to be higher than the claim-based measure because the former allows for claims to be considered partially correct, whereas the claim- based measure does not. According to VBA officials, the issue-based estimate provides a better measure of quality because veterans’ claims have increasingly included multiple medical issues. Our analysis of STAR data confirms that, as the number of issues per claim increases, the chance of at least one issue being decided incorrectly within a single claim increases because there are more opportunities for error (see fig. 1). However, VA did not report in its fiscal year 2015 budget request how these measures are calculated and why the issue-based measure might be higher than the claim-based measure. Further, VA has not explained in public reports that its accuracy measures are estimates which have distinct margins of error and limitations. These margins of error are necessary for users to make meaningful comparisons, for example, between the two measures or over time for the same measure. Further, each accuracy measure has distinct limitations, but VA does not report this information. For example, the claim-based measure does not provide a sense of the proportion of issues that the agency decides correctly because the measure counts an entire claim as incorrect if any error is found. On the other hand, the issue-based measure does not provide a sense of the proportion of claims that the agency decides with no errors. In prior work, we identified clarity as a key attribute to a successful performance measure, meaning that the measure is clearly stated and the associated methodology for the measure is identified. Measures that lack clarity may confuse or mislead users. We have also reported on best practices in implementing related federal performance reporting requirements, such as those in the GPRA Modernization Act of 2010. Specifically, agencies must disclose information about the accuracy and validity of their performance information in their performance plans, including the sources for their data and actions to address any limitations. Finally, VBA’s approach to measuring accuracy is inefficient because it reviews more claims than needed to estimate accuracy. VBA randomly selects about 21 claims per month from each of its regional offices for STAR review, regardless of the offices’ varying workloads and historical accuracy rates. According to VBA, this uniform approach allows the agency to achieve a desired level of precision of its accuracy estimates for each regional office. However, this approach leads VBA to select more claims for review than are needed at regional offices where the number of claims processed has been relatively small or accuracy has been high. According to our analysis of fiscal year 2013 regional office workload and accuracy results, VBA could reduce the overall number of claims it reviewed annually by about 39 percent (over 5,000 claims) and still achieve its desired precision for its regional office accuracy estimates. Only for one regional office did we find that VBA would need to increase the number of claims currently reviewed to achieve its confidence level goal for that office. More efficient sampling could allow VBA to select fewer cases for review and free up limited resources for other important quality assurance activities, such as additional targeted accuracy reviews on specific types of error-prone or complex claims. Specifically, reviewing about 5,000 fewer claims could free up about 1,000 staff days because, according to VBA officials, STAR staff review at least 5 claims per day. Our review of STAR is ongoing. Specifically, we plan to analyze STAR data and processes to assess whether any types of claims are systematically underrepresented in accuracy reviews. In addition, we are further considering the implications of a more efficient sampling methodology on developing issue-based accuracy estimates. In addition to its STAR reviews, VBA’s quality assurance framework includes other complementary activities, some of which have been enhanced to help meet its 98-percent goal for fiscal year 2015. For example, VBA established QRTs in regional offices as a means of strengthening its focus on quality at regional offices, where claims are processed. QRT personnel—like STAR reviewers—are required to pass an annual skills certification test. In addition to conducting individual quality reviews to determine whether claims processors are achieving individual accuracy targets, QRT personnel are charged with conducting in-process reviews of claims not yet finalized, looking for specific types of errors commonly made. Quality reviewers are also responsible for providing feedback to claims processors on the results of their quality reviews, including formal feedback from the results of individual quality reviews and more informal feedback from the results of in-process reviews. Typically, feedback on quality reviews is provided as reviews are completed.