Within VA, its OSDBU has overall responsibility for the SDVOSB and VOSB verification program. OSDBU’s Center for Verification and Evaluation (CVE) maintains the mandated database of verified SDVOSBs and VOSBs and is responsible for verification operations, such as application processing. CVE is led by a director and deputy director, and staff are organized into seven teams that either assist with a verification phase or supporting function. A federal employee leads each team and CVE contracts with several SDVOSBs to provide contractors that conduct verifications and supporting activities. As of January 2016, CVE had 16 federal employees and 156 contract staff (employed by five different SDVOSB contractors) verifying applications or filling supporting roles. CVE and its information technology are funded by VA’s Supply Fund, a fund that recovers its operating expenses through fees and markups on different products or services. CVE’s final obligations for fiscal year 2014 were $17.9 million and its approved budget for fiscal year 2015 was $16.1 million, representing a decrease of about 10 percent ($1.8 million). VA developed eligibility requirements and a process to verify the ownership and control of firms seeking contracting preferences as SDVOSBs and VOSBs and to confirm the status of any owner who indicates a service-connected disability. To be eligible for verification under VA’s rules the small business concern (hereafter, firm) must be unconditionally owned and controlled by one or more eligible parties (veterans, service-disabled veterans, or surviving spouses); the owners of the firm must have good character (any small business owner or concern that has been debarred or suspended is ineligible); the applicant cannot knowingly make false statements in the application process; the firm and its eligible owners must not have significant financial obligations owed to the federal government; and the firm must not have been found ineligible due to a Small Business Administration protest decision. VA’s verification process consists of reviewing and analyzing a standardized set of documents submitted with each verification application. VA’s current verification process has four phases—initiation, examination, evaluation, and determination (see fig. 1). Denied applicant firms can request a reconsideration of the denial decision, but if the denial is upheld, must wait 6 months before submitting another application. VA maintains a database on its Vendor Information Pages (VIP) website of verified SDVOSBs and VOSBs. Once VA verifies a firm, the firm name appears with a verified logo in VIP. Verification is valid for 2 years, with the option of renewing for 2 additional years. To apply for renewal, verified firms must submit an application before expiration of their verification, answer questions about any key changes to their business structure, submit supporting business documents such as updated tax returns and any amended operating documents, and receive a full evaluation by CVE. Firms that receive a renewal have to undergo a full verification (reverification) after the 2-year renewal period expires (that is, firms must go through the full process every 4 years). In 2014, VA launched the MyVA Reorganization Plan in an effort to improve the efficiency and effectiveness of VA’s services to veterans. The plan’s strategy emphasizes improved service delivery, a veteran-centric culture, and an environment in which veteran perceptions are the indicator of VA’s success. MyVA extends to all aspects of the agency’s operations, including the verification program. In response to this organizational change, OSDBU is required to align its own strategy with MyVA and take steps to make its operations more customer service- oriented and veteran-centric. OSDBU has established in its strategic plan that its longer-term goals for the verification program are to transform the verification process to make it more veteran-friendly and to expand the program’s capacity to serve more veterans. In our May 2010 and January 2013 reports, we found that VA faced challenges in establishing a program to verify firms on a timely and consistent basis. We made a number of recommendations to address these issues. Since that time, CVE has made significant improvements to the verification program—consistent with our recommendations—such as improving application processing times, quality controls, and communication with veterans. Based on CVE’s administrative data, application processing times have decreased by more than 50 percent since October 2012—from an average of approximately 85 days—to 42 days in fiscal year 2015 (see fig. 2). VA officials attributed the decreased processing time to a number of process improvements, such as moving from paper to electronic applications, developing detailed written work instructions for staff and contractors conducting verification activities, and enhancing resources available to applicants to make them more aware of program requirements. Officials told us that they have been generally meeting their regulatory processing goal of 60 days (from receipt of a complete application) and had 5 applications (out of 3,129) in fiscal year 2014 and 11 applications (out of 4,651) in fiscal year 2015 that did not meet this goal. Our review of randomly selected application files corroborated that CVE generally met its processing goals, but the verification process can take longer from a veteran’s perspective. In calculating processing times, CVE excluded any time spent waiting for additional information it asked firms to supply, so the number of days it took an applicant to become verified typically was longer than CVE reported. To illustrate how long the process could take from the veteran’s perspective, we used the results of our case file review to estimate the average number of days it took veterans to receive an initial determination—including any time a firm took to prepare and submit additional requested documentation (time that VA excluded from its estimated processing times). Based on our analysis, it took firms that applied for verification from June through September 2014 an average of 68 days to receive an initial determination on verification eligibility. Additionally, firms can submit and withdraw their application multiple times if they need to correct issues or wish to apply at a later date, an option that can lengthen the verification process for some firms. Each time a firm resubmits an application, CVE resets the processing clock. Based on our case file review, we estimated that for 15 percent of applications submitted from June 2014 through September 2014, it took more than 4 months from the initial application date for firms to receive a determination from CVE. In 5 of the 96 applications we reviewed, the verification process took more than 6 months to complete. For 2 of the 5 applications, the process took 1 year or more to complete. In each of the 5 cases, the firm had submitted and withdrawn an application at least one time before submitting an application that received a final determination. VA officials said that weeks or months could pass between a firm’s withdrawal of its application and resubmission of a new application. Additionally, VA officials said that allowing applicants to withdraw and resubmit multiple applications was advantageous to the veteran (without the option, more veterans would receive denials and have to wait 6 months before submitting another application). CVE has continued to refine its quality management system since our May 2010 and January 2013 reports, including developing comprehensive work instructions, conducting site visits, and revising policies for investigating allegations of noncompliance. In particular, VA has made progress since our 2013 report. For example, at that time, VA was introducing significant changes to its procedures and operations, and we determined that our original focus on evaluating VA’s compliance with policies and procedures would be of limited value. Since our 2013 report CVE has put into place detailed written work instructions—which are used by CVE staff and contractors conducting verification activities—for each part of the verification process, and a quality manual that documents the requirements of its quality management system. According to CVE officials, the work instructions are updated on a regular basis. CVE officials said the agency received certification in 2015 that its revised quality management system was compliant with the International Organization for Standardization 9001:2008 quality management standards. CVE also has implemented an internal audit process and a continuous improvement process. As of September 2015, CVE had taken action on and closed 332 of 350 (95 percent) internal audit recommendations made since February 2013. Based on our case file review, we estimate that VA followed its policies and procedures for verifying SDVOSBs and VOSBs in 100 percent of applications that were approved or denied from June through September 2014. This included checking the veteran and disability status of the applicant, conducting research on the firm from publicly available information, and reviewing business documents to determine compliance with eligibility requirements (such as direct majority ownership by the veteran and experience of the veteran manager). In our May 2010 report, we found that VA had a large backlog of firms awaiting site visits, including some high-risk firms. As a result, we recommended that VA develop and implement a plan to, among other things, conduct timely site visits to high-risk firms. VA reported in October 2013 that it had conducted more than 1,000 site visits in fiscal year 2013 and there was no longer a backlog of firms awaiting site visits, which was consistent with our recommendation. In fiscal years 2014 and 2015, CVE conducted 1,750 site visits to gather additional information about firms during its application review, or check the accuracy of verification teams’ decisions and help ensure that verified firms continue to comply with program regulations. Specifically, CVE conducted 1,144 site visits in fiscal year 2014 and 606 site visits in fiscal year 2015 on verified firms and firms applying for verification. Of the fiscal year 2015 site visits, CVE used risk-based determinations to select the vast majority of firms for visits (93.7 percent or 568 firms). The remainder were randomly selected or chosen because they were in the application process. CVE officials said that firms identified through risk-based selection were chosen based on their risk to VA (for instance, if the firm had a VA contract). Officials reported low error and noncompliance rates as identified through site visits, described specifically below. CVE officials said the site visits identified two instances in fiscal year 2015 and nine instances in fiscal year 2014 in which CVE evaluators mistakenly verified a firm (a less than 1 percent error rate). CVE issued 25 cancellations to firms found out of compliance with program regulations (a 4 percent noncompliance rate) in fiscal year 2015 and 57 cancellations in fiscal year 2014 (a 5 percent noncompliance rate), an outcome that can result from changing characteristics of the firm after verification. These statistics, particularly the identification of a small number of instances in which CVE evaluators mistakenly verified a firm in the past 2 fiscal years, are consistent with the findings from our case file review that VA has been following its policies and procedures for verifying firms. CVE officials said they have been working with VA’s Office of Enterprise Risk Management to determine how many site visits should be conducted annually and how firms should be selected. VA spent about $3 million in fiscal year 2014 to conduct 1,144 site visits (about 16 percent of all verified firms). Officials stated that because of the cost of conducting the site visits (about $2,600 per visit)—and the low rate of noncompliance identified by site visit examiners (about 4.7 percent in fiscal year 2014)— the agency has been looking to reduce the number of site visits while maintaining the effectiveness of the site visit program. CVE also randomly selected a sample of 104 of 2,312 firms that received VA contracts from March 2014 through April 2015 for site visits, in order to obtain a statistical estimate of the noncompliance rate for the program. CVE officials said they have compiled data from visits performed in fiscal years 2014 through 2016, which they will use to identify risk factors that could affect compliance (such as a firm’s business type, industry type, and size of VA contract). VA officials said they plan to use these data to monitor emerging risk factors and adjust their selection of firms to receive site visits accordingly. CVE also monitors program compliance through investigations of suspicious firms identified through tips from external sources. CVE officials told us they received about 400 tips in 2014 about noncompliance with program regulations. CVE is responsible for investigating noncompliance with program requirements, and the VA OIG is responsible for investigating fraud. Officials said that they investigate credible allegations they receive by conducting public research to substantiate or disprove the tip, reviewing eligibility requirements related to the tip, and making a recommendation for corrective action, if necessary. We reviewed case files associated with 10 firms for which CVE received such allegations (between June 2014 and May 2015). These allegations were made by a veteran-owned small business advocate. We found that CVE investigated 6 of the 10 allegations. For the 4 cases that it did not review, CVE officials said the allegations were not specific enough to conduct an investigation. Additionally, officials said the allegations were not sent to the e-mail address that VA had established for this purpose, and thus, may not have been routed to the correct individuals within VA charged with investigating allegations. According to a policy memorandum issued in October 2015, CVE previously only accepted referrals on its non-compliance referral form. However, the policy memorandum changed CVE’s policy regarding reviews of non- compliance referrals, and CVE began reviewing and responding to all allegations of noncompliance, whether received on the referral form or not. Additionally, CVE will notify the sender if the agency has enough information to investigate the allegation and request additional information, if necessary. Based on our review of the files for these 10 allegations, we found that CVE had not always documented that a noncompliance allegation had been received or that it was conducting a review of the firm’s eligibility based on the allegation. CVE officials said they adopted a policy in July 2015 to upload all findings from investigations that result from noncompliance allegations to the case-management system so that CVE staff and contractors working on a firm’s verification have access to that information. VA has taken steps to improve communication with veterans since our January 2013 report, in which we discussed concerns of some veterans’ organizations about the verification program. VA implemented additional procedures to improve communication with verified firms about their verification status. Specifically, according to agency officials, VA sends e-mail reminders 120, 90, and 30 days before the expiration of a firm’s verification status; contacts verified firms by telephone 90 days before expiration of verification status; and notifies firms in writing 30 days before cancelling a firm’s verification. Additionally, VA communicates with applicants at several points in the verification process, such as to indicate that an application is complete, additional documents are needed, and a determination has been made. VA also e-mails applicants about issues that would result in a denial to offer them the ability to make changes to the application or to withdraw it prior to receiving the denial. Our case file review found VA generally followed its procedures to send reminder notices to applicants who needed to submit additional documentation, and to send notices that an application was complete. In our January 2013 report, we noted that VA had recognized that some applicants needed additional support and launched a Verification Counseling program in June 2012 to assist firms interested in becoming verified. Since that time, VA has continued to work with PTACs across the United States to provide verification assistance to veterans free of charge. VA officials said they have trained more than 300 procurement experts on the verification process so they can assist veterans applying for verification. In addition, we determined that VA provides other resources such as fact sheets, verification assistance briefs, and standard operating procedures for the verification program on its website. VA also provides a tool on its website that allows firms to obtain a list of documents required for an application depending on the type of company they own (such as a limited liability corporation or sole proprietorship). Moreover, CVE officials said that they have increased interaction with veterans seeking verification through web-based activities and outreach at conferences and other events. For instance, since our 2013 report, CVE began conducting monthly pre-application, reverification, and town hall webinars to provide information and assistance to verified firms and others interested in the verification process. VA officials also told us that they attend veteran small business conferences and other meetings in which they can conduct outreach for the verification program. CVE has taken steps to collect veteran feedback about the program. The agency held two focus groups in July 2015 and began surveying firms in August 2015. Several veterans who participated in the focus groups commented on the lack of clarity of VA’s communications. For example, one veteran said that there was a lack of clarity about the documents that should be submitted with an application. Two veterans noted issues with the timing and redundancy of document requests during the application process. And two veterans said there was lack of clarity around certain program rules. Although survey results cannot be generalized to all veterans going through the verification process, the contractor administering the surveys reported in September 2015 that feedback from firms that had been through the verification process appeared positive, particularly with respect to improvements in the verification process. CVE officials stated they hope the surveys will allow them to more systematically collect feedback from veterans on different aspects of the program, including the pre-application experience, the verification process from submission to determination, and site visit examinations. All the counselors and representatives of veterans’ service organizations with whom we spoke noted that VA has improved its verification process, but most had suggestions for continued improvement. One PTAC representative noted that VA could better leverage organizations such as PTACs, veteran small business outreach centers, and state-level Departments of Veterans Affairs to disseminate information. Another PTAC representative noted that PTACs and other veterans’ groups could host events for CVE to interact with veterans to increase awareness of the verification program and the services PTACs provide. Additionally, this PTAC representative said CVE could do a better job referring veterans to PTACs that could assist them with the verification process. He added that having PTACs assist veterans before they started the application process would reduce applicant error and frustration and processing times. VA officials stated that they make PTAC referrals through the help desk and monthly preverification webinars and participate in outreach events when possible. Officials also noted that travel dollars to attend and conduct outreach at external events are limited and they try to use the monthly webinars to interact with and inform veterans. In addition, VA has taken steps to address external stakeholders’ concerns about the information available on VA’s website and clarity of communications to applicants. Three of the four counselors noted that resources on VA’s website for the verification program can be difficult to locate. Representatives from one of two service organizations said VA does not provide adequate documentation of program standards for applicants. Officials said that they have been working with OSDBU to redesign the website to make documents, such as verification assistance briefs and the tool to identify required documents, easier to locate. Additionally, the counselors we interviewed noted that VA’s communications to applicants were at times unclear. VA’s procedures require that determination letters to applicants include specific reasons for denial or potential denial. All four of the counselors we interviewed also stated that VA’s determination letters to applicants could be clearer and that they include regulatory compliance language that could be difficult for some applicants to understand. We used an automated readability test on five determination letters from our case file review (written from August through December 2014). According to the test, the reading level required to understand the letters ranged from a college sophomore to a college senior level, so readability might present a challenge for some applicants. VA officials maintained that the inclusion of regulatory language in the determination letters was necessary, but acknowledged the language could present readability challenges. Moreover, the officials noted that they encourage veterans to obtain free assistance with their applications from VA certified verification counselors. We also observed several instances in our review in which a letter initially stated that documents were due on one date, and then later stated the applicant should disregard the initial statement and that documents were due on a different, earlier date. VA officials said this was due to a glitch in the system that generated the letters and a software update issued in May 2015 resolved the issue. Officials also said document requests are now generated using a new template, instead of the older template that previously caused issues. VA has efforts under way (discussed below) to replace the program’s case management system, which generates the templates for document requests. Although VA improved application processing times and communication with veterans since our 2013 report, VA has recognized that the verification process can be more cost-effective and veteran-friendly. OSDBU developed a strategic plan for 2014–2018 that included longer- term goals for the program, such as making the verification process more veteran-friendly. Additionally, OSDBU officials also told us that in 2015 the Supply Fund Board asked OSDBU to design a veteran-centered process that highlights customer service and maximizes cost efficiency. OSDBU and CVE have various efforts under way—such as restructuring the verification and reverification processes and revising program regulations—intended to improve veterans’ experience with the program and provide cost efficiencies. These changes are also intended to better align the verification program with MyVA—VA’s organization-wide transformation initiative. In August 2015, VA began to test (VA officials refer to it as a pilot) a restructured verification process that gives veterans a case manager who serves as a point of contact throughout the process and allows veterans to communicate directly with the individual processing their application. According to OSDBU officials, the new process is expected to provide cost savings to the agency by reducing the amount of time spent reviewing applications and addressing veterans’ questions. For example, VA officials told us that under the current four-phase process, eligibility issues are not identified and communicated to the veteran until the later stages of the process, which could be 35–40 days after application submission. Under the new process, veterans would be interviewed shortly after submitting their application, which would allow VA to identify issues up front and avoid multiple reviews of applications for firms not meeting program requirements. CVE officials also stated that changes to the process are intended to help improve communication with applicants. For example, under the current process, applicants may correspond with several different customer service representatives, each of whom would have to read through case notes before addressing the veteran’s question. Under the new process, a case manager, who serves as the point of contact for the veteran and coordinates staff evaluating the application, would be familiar with the application status and any issues that arose throughout the process. Additionally, by interviewing the applicant at the beginning of the application process, CVE may be able to reduce some of the problems caused by applicants not understanding the written communications sent by CVE. Key differences between the pilot and current processes as described by CVE officials are shown in table 1. CVE officials stated that as of November 2015, 369 applications had been reviewed using the new process and that CVE had processed about 15 percent to 20 percent of applications under the new process. CVE officials said they plan to process about half of all applications using the new process by April 2016. CVE officials stated that they plan to fully transition to a new process by September 2016. The new process will be based on the approach used during the pilot with some adjustments, as determined by CVE’s evaluation of the pilot procedures. The officials stated that they developed a number of metrics to inform adjustments to the pilot, which include length of application processing times, number of approvals, denials, and withdrawals, and number of applications processed by case-management teams each month. Additionally, to help evaluate the verification procedures used during the pilot, CVE officials stated that VA held one focus group in October 2015 and since September 2015 has been surveying firms that participated in the new process to obtain feedback. CVE has made adjustments to the process in response to metrics and feedback. For example, CVE decided to have case analysts process the majority of applications, which allowed assessors more time to process complex applications and provide guidance to the case analysts and other assessors, when needed. CVE’s Acting Director and Deputy Director were responsible for evaluating data obtained from these metrics according to CVE officials. In addition, CVE officials stated that information obtained from the metrics was intended to provide them with a better sense of how many teams will be needed to conduct verification and what types of skills verification staff and contractors needed for the work. CVE officials stated that VA has used current staffing resources for the pilot and transitioned more staff from the current verification process to the pilot, as the pilot expanded. Officials said that VA has developed a range of preliminary cost estimates based on different staffing levels used during the pilot. The agency plans to develop a cost estimate for average and total processing costs after they have transitioned staff to the new process. As of March 2016, CVE officials stated they completed their evaluation of the pilot and have selected a process they determined was cost effective and efficient and did not compromise quality. VA also recently revised its reverification process to improve efficiency and customer service. According to CVE officials, reverification used to require nearly the same effort of CVE staff, contractors, and veterans as the full initial verification process. Under a new process CVE implemented in October 2015, CVE contractors review documentation from the veteran’s previous application, determine what additional documentation is needed to reverify the firm, and conduct an initial interview with the veteran to identify and provide information on what documents need to be updated for reverification. CVE officials said these changes are intended to improve veterans’ understanding of the requirements for reverification, enhance the veteran- centric nature of the program, and further reduce application processing times—by reducing the number of documents the veteran uploads at the beginning of the process and therefore the time CVE contractors and staff spend reverifying applications. According to CVE, it is too soon to determine if these changes have achieved the desired effect, but they intend to evaluate the new procedures by developing survey questions on customer satisfaction and reviewing processing times for reverification applications. In addition to changes to the verification and reverification processes, VA has continued to make revisions to its program regulations to streamline the process and provide clarity for veterans. In our 2013 report, we found that VA had begun modifying program regulations to extend the verification period from 1 to 2 years and published an interim final rule to this effect in late June 2012. More recently, in November 2015 VA published a proposed rule for clarifying and simplifying eligibility criteria in the Federal Register, on which it had been working since 2013. VA obtained input and recommendations from veteran stakeholder groups to inform the new regulations. According to officials, the proposed revisions are intended to simplify the criteria used to determine veteran ownership and control, and account for common business practices that might otherwise lead to a denial decision under the current regulation. For example, in addressing the challenges associated with one current regulatory provision, CVE officials said that the proposed rule simplifies the ownership criteria by modifying the term “unconditional” to allow businesses to include rights of first refusal and “tag-along” rights in their operating documents and still participate in the program. Similarly, officials said that VA plans to allow minority owners to vote on extraordinary business decisions such as closing or selling the business. Officials stated that the revisions to the regulation were not expected to provide cost and resource efficiencies, or affect the new verification process being developed through the pilot. Comments on the proposed rule were due January 5, 2016, and officials expect to finalize the proposed rule in mid-2016. Officials said they received about 100 comments and that they still were evaluating and categorizing the comments as of January 21, 2016. When the rule changes are finalized, VA plans to train staff on the revised regulation and update its review sheet used during application review. We previously found that leadership and staff vacancies contributed to the slow pace of implementation of the verification program. In our May 2010 report, we found that leadership in OSDBU was lacking because the position of Executive Director remained vacant from January 2009 until January 2010. Furthermore, one of two leadership positions directly below the Executive Director had been vacant since October 2008, and the other positions had been filled by an Acting Director. In 2010, we recommended that VA develop and implement a plan that ensures a more thorough and effective verification program and addresses actions and milestone dates for filling vacant positions within OSDBU, including the leadership positions. By July 2011, VA had filled the vacant leadership positions but has since experienced turnover in the leadership positions for the verification program. Specifically, CVE has had four different directors since 2011, including two acting directors in 2015. In addition, the position of Deputy Director was vacant from March 2014 to September 2015. VA posted a job announcement for the CVE director position in October 2015 that closed in November 2015. OSDBU’s Executive Director told us in December 2015 that VA planned to complete the recruiting process and hire a permanent director by January 2016. VA hired a permanent director in February 2016. In addition to taking steps to acquire permanent leadership for the program, VA has made changes to CVE’s organizational structure to align staffing resources with agency needs and reflect the new pilot verification process. CVE officials said that under the new organizational structure, a federal employee heads each of three teams—critical path, risk and compliance, and verification support—and oversees the contractors who conduct the majority of the work for the program. The critical path team is responsible for verifying applications. All federal staff responsible for overseeing the verification process and the contractors reviewing applications have been moved to this team, according to CVE officials. The risk and compliance team oversees CVE’s site visit program, investigates allegations of noncompliance, and makes referrals to VA’s Debarment and Suspension Committee and OIG. The verification support team staffs the help desk and provides support processing communications to applicants. CVE officials also said that the quality assurance team has been moved from within CVE to the larger OSDBU organization and will continue to support CVE internal audits as well as provide support to OSDBU. OSDBU officials indicated that VA made this change to share more responsibilities between CVE and OSDBU and reduce program operating costs. According to CVE officials, VA has developed position descriptions for the pilot verification process and used data from the pilot to determine optimal staffing levels needed to process applications under the new procedures. As discussed earlier, CVE relies heavily on contractor support to conduct its verification activities and currently has 16 federal employees and 156 contractors working on the verification program. CVE officials stated that VA plans to continue using contractor staff to conduct verification activities because the use of such staff gives VA the flexibility to adjust staffing levels as needed to respond to changes in the number of verification applications received. According to VA officials, and generally consistent with findings from our case file review, federal employees make the final determination for verification decisions. Officials said that based on information obtained through the pilot, VA has determined that it needs a total of 10 federal reviewers on the critical path team; however, it still was determining its needs for contractor staff as of November 2015. According to OSDBU officials, VA has contracts in place for the verification program staff through April 2016 and plans to start the process for securing new contracts in January 2016. We reported in January 2013 that VA moved from using a paper-based verification application to an electronic application when it implemented a new case-management system in 2011, consistent with our prior recommendation. However, we identified significant shortcomings in VA’s data system, including that the system did not collect important data and had limited reporting and workflow management capabilities. We recommended that VA integrate its efforts to modify or replace the program’s data system with a broader strategic planning effort to ensure that the system addressed the program’s short- and long-term needs. VA concurred with the recommendation and included a strategic performance goal to improve information technology capabilities in its strategic plan, which was consistent with our recommendation. VA awarded a contract for an enhanced data system; however, VA has since faced delays in developing the system. VA’s Office of Information Technology hired a contractor in September 2013 to develop the new system, but VA cancelled the contract in October 2014 due to poor contractor performance. VA paid the contractor about $871,000 for work that had been performed before the contract’s termination, and received several planning documents from the contractor that helped inform its current acquisition effort, according to CVE officials. VA officials told us that CVE and VA’s Office of Information and Technology then began working with a contractor in September 2014 to identify data systems in existing federal programs that VA could use to build its own system. In February 2015, officials told us that they planned to award a contract for development of the new system in the first quarter of fiscal year 2016 based on the contractor’s research to identify existing federal systems. VA was unable to award a contract based on this effort. Specifically, the contractor identified two programs that had the potential to be used in developing a new case-management system for VA. However, VA officials conducted additional research on each program’s system and found technical or administrative reasons that prevented VA from partnering with any of the identified programs to use their existing systems to develop a new case-management system for CVE. VA’s more recent efforts to develop a new case-management system also have faced setbacks. In May 2015, VA established an internal working group consisting of staff from OSDBU and VA’s Office of Information and Technology and Office of Acquisition and Logistics to plan and manage the development of the new system, based on the planning documents provided by the contractor and in accordance with internal guidelines for managing new information technology projects. In July 2015, VA officials told us they decided to develop a pilot system through another existing contract. Officials said they intended to use the pilot system to provide VA with the opportunity to evaluate the capabilities of a new system without the time and expense of putting an entire new system in place. VA developed specifications and other planning documents for the pilot system, and planned to develop and evaluate the system from November 2015 through January 2016. If the pilot was successful, VA had planned to issue a solicitation and award a contract for development of a full system by April 2016 and fully transition to the new system by September 2016. However, in November 2015, OSDBU officials told us that the Supply Fund board had requested that OSDBU develop and provide a business case for the new system. VA officials stated that as a result, they have revised the timeline for developing the new system, and expect to begin the pilot in January 2016. If the pilot is successful, VA plans to fully transition to a new system in early 2017. But as a result, VA continues to use a data system that does not collect important data and has limited reporting and workflow management capabilities. In both our May 2010 and January 2013 reports, we found that VA faced challenges in developing and implementing plans for establishing an effective verification program. Specifically, in our May 2010 report, we found that VA did not have a plan or specific time frames for implementing a thorough and effective verification program, including filling vacant staff positions; improving verification procedures to ensure greater completeness, accuracy, and consistency of verification reviews; and conducting timely site visits at high-risk firms. We recommended that VA develop and implement such a plan. VA took actions that addressed the specific actions referenced in our recommendation, obviating the need for a plan to accomplish these actions. In our January 2013 report, we found that VA faced challenges in its strategic planning efforts and recommended that VA refine and implement a strategic plan with outcome-oriented longer-term goals and performance measures. Subsequently, VA developed a strategic plan for fiscal years 2014–2018 that, consistent with our recommendation, described OSDBU’s vision, mission, and performance goals for its programs, including the verification program. Additionally, VA has developed a high-level operating plan for fiscal year 2016 that identifies key actions needed to meet OSDBU’s objectives, such as transitioning to a new verification process, completing revisions to verification regulations, and developing a new case- management system. But, VA’s operating plan is not comprehensive and does not include an integrated schedule with specific actions and milestone dates for achieving program changes or discuss how the efforts described in the previous sections might be coordinated. For example, the operational plan states that VA needs to have in place an information technology system that allows both case-management and client-relationship management. However, it does not describe the specific actions that VA must take to acquire such a system or how system development will be integrated with other ongoing efforts, such as adoption of a new verification process. Instead, the operating plan states that a main element to achieving the goal is to develop a case-management system. In another example, the operating plan states that VA must have a verification process that provides a positive veteran experience and that a main element to achieving that goal is to transition to the new process. But the plan does not explain the specific actions necessary to fully transition to the new process or the timetable for the transition. In addition, VA does not have a process in place to update the verification program operating plan on a timely basis to ensure that it reflects current initiatives, conditions, and long-term goals. We previously reported that useful practices and lessons learned from organizational transformation show that a transformation, such as CVE’s efforts to make the verification process more efficient and veteran-friendly, is a substantial commitment that could take years before it is completed, and therefore must be carefully and closely managed. As a result, setting implementation goals and a timeline to build momentum and show progress from day one is essential for organizations. Tracking implementation goals and establishing a timeline can help pinpoint performance shortfalls and gaps and identify the need for midcourse corrections. According to OSDBU officials, each OSDBU program team (such as CVE) is to develop an action plan for its specific program that includes resource needs and expected timelines. The Executive Director stated that OSDBU intends to review and incorporate each action plan into the operating plan for all of OSDBU, so that OSDBU has a detailed plan for the verification and other OSDBU programs. CVE officials told us that they had initially delayed finalizing CVE’s plan because they were waiting to evaluate the results of the verification pilot, and intended to finalize the plan with actions and milestone dates in December 2015. As of January 29, 2016, CVE had yet to complete an operating plan for the verification program. Without a plan that contains actions and milestone dates for the multiple efforts CVE has been undertaking, VA may face difficulties in managing these efforts to completion. Furthermore, without engaging in effective operational planning moving forward, VA lacks assurance that it can achieve its longer-term objectives for the verification program. VA has made significant improvements in its verification program since our 2013 report, including application processing time, quality control, and communication with veteran applicants. Nonetheless, the agency continues to face challenges in making the program less resource- intensive and more efficient and veteran-friendly. VA has acknowledged these issues and has begun to transform the program to address them. VA’s efforts to restructure the verification process, realign organizational structure, and acquire a new case-management system represent significant efforts for CVE’s team of 16 federal employees. But these efforts began and have continued in the absence of a detailed operational plan to guide and integrate them with VA’s strategic objectives. And the agency has faced challenges with planning—both strategic and operational—as we found in previous reviews of the program dating to 2010. By putting such a plan in place to guide the program’s transformation, VA could obtain reasonable assurance that these efforts will be properly sequenced, managed to completion, and help VA accomplish its longer-term goals. Moreover, having a detailed plan to accomplish multiple ongoing efforts is critical given the repeated delays in VA’s efforts to acquire a new case-management system and the lack of continuity in CVE leadership. Such a plan is also critical in the context of VA’s efforts to carry out an organizational transformation and its long-term goals to expand the program’s capacity to serve more veterans. Without a policy to review and update the operating plan to reflect current conditions and priorities, VA would continue to be at risk for delays in implementing its initiatives and achieving its long-term goals. To improve the management and oversight of VA’s SDVOSB and VOSB verification program, we recommend that the Secretary of Veterans Affairs direct OSDBU to complete its fiscal year 2016 operating plan and include an integrated schedule that addresses key implementation goals and the actions and milestone dates for achieving them, such as the coordination of the redesign of the verification process and the design, acquisition, and deployment of a new case-management system; and establish a process to review and update the operating plan for the verification program on a timely basis to address new VA initiatives, other changing conditions, and long-term goals. We provided a draft of this report to the Department of Veterans Affairs for comment. In its written comments, VA agreed with our recommendations. Specifically, VA said that it completed a draft of the fiscal year 2016 operating plan, which received input from the major OSDBU program areas (including the verification program). A final version of the plan, which will incorporate key implementation goals and milestones, will be released by March 31, 2016. VA also stated that OSDBU has implemented a process to review and update the operating plan for the verification program and all other program areas on a timely basis. According to VA, the process will allow OSDBU to address VA initiatives and programmatic contributions linked to realizing those initiatives and articulating how other changing conditions and long-term goals will be managed. VA also provided timeframes for completing its planned actions. VA provided technical comments and updates on the status of some of its ongoing initiatives, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-8678 or shearw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report assesses progress by the Department of Veterans Affairs (VA) in (1) establishing a timely and consistent verification program and improving communication with veterans, and (2) the steps VA has taken to identify and address program challenges and longer-term goals. We and others previously identified verification program challenges, including application processing timelines and quality controls, communication with veterans, case-management system, and strategic and operational planning. To assess VA’s progress in establishing a timely and consistent verification program, we reviewed relevant statutes, regulations, and procedures for the verification program. We also reviewed the verification program quality manual to understand what the Center for Verification and Evaluation’s (CVE) quality management standards were and how CVE ensured quality of the program, and reviewed internal audit reports from December 2014 to March 2015. We interviewed officials in VA’s Office of Small and Disadvantaged Business Utilization (OSDBU) and CVE about their policies and procedures for processing applications, for quality control, for investigating allegations of noncompliance with program regulations, and changes to the verification program since our last report (2013). We conducted a case file review to determine the extent to which VA had followed its policies, procedures, and quality controls for processing applications, as well as the extent to which VA had processed applications within regulatory time limits. We selected a stratified probability sample of all verification applications (initial and renewal applications) submitted to VA between June and September 2014. We chose this time period so we could obtain a sample of applications for which CVE had completed its application review and so that the applications in our sample would have been processed under a recent and similar set of verification procedures. The applications were stratified by two groups of decision outcomes: (1) withdrawals and (2) approvals and denials. The sample was designed to make generalizeable estimates for approvals and denials only, with the sample of withdrawals providing nongeneralizeable examples. We used simple random sampling methods to select 96 of the 1,306 applications submitted to VA during this time frame that resulted in an approval or denial. We developed and pre-tested an instrument to collect data from the case files on application processing time frames, completeness of VA’s review, and documentation of key decisions and rationales. We assessed the reliability of these data by interviewing VA officials knowledgeable about the data, reviewing documentation related to the data systems, and checking the data for illogical values or obvious errors and found them to be sufficiently reliable for estimating population values. The sample allowed us to estimate the proportion of cases for which VA consistently followed its policies and procedures and met regulatory time frames for reviewing applications for the verification program for applications submitted from June through September 2014. Because we used simple random sampling methods to select approvals and denials, our estimated proportions did not require weighting. We used hypergeometric methods to estimate 95 percent confidence intervals, which account for the small size of the sample and population, estimated proportions near 0 or 100, and a nonignorable sampling fraction. We reviewed administrative program data obtained from VA on application processing times for fiscal years 2012 through 2015 and compared those numbers to our findings from the case file review. We also reviewed administrative data from VA on the number and type of site visits conducted in fiscal years 2014 and 2015. We assessed the reliability of CVE’s administrative program data by interviewing VA officials and reviewing documentation related to VA’s data system, and we found the data to be sufficiently reliable for describing VA’s reported processing statistics. We also reviewed the files of 10 verified businesses for which CVE received allegations of noncompliance with program regulations to identify the steps CVE took to investigate the allegations. To select these files, we reviewed more than 100 allegations of noncompliance with program regulations sent to VA between June 2014 and May 2015 by a veteran-owned small business advocate. We catalogued tips that were relevant to the verification program (tips that dealt with potential problems in the verification process versus tips that dealt with VA contracting issues) and selected 10 firms for which to conduct a more in-depth review of how VA reviewed or addressed the alleged fraud. We purposefully selected these firms to obtain variation in the type of allegation (e.g., that the firm was not owned by a service-disabled veteran but instead was a “pass-through” or that the firm did not meet the criteria for a small business), whether an official protest was filed, and results (whether the firm remained in the verification database). We reviewed the files associated with each of these firms and collected information about the steps VA took to respond to allegations of noncompliance and status protests. To assess VA’s progress in addressing communication challenges with veterans, we reviewed VA’s work instructions for processes that involve communicating with veterans. We also reviewed VA’s website to identify guidance available for applicants, such as an applicant guide, frequently asked questions, verification assistance briefs, and an online self- assessment tool for prospective applicants. We interviewed VA officials to determine what procedures they have in place to communicate with applicants and verified businesses and obtain feedback from these entities on VA’s verification process and communication efforts. We interviewed representatives of two veteran service organizations and four Procurement Technical Assistance Centers (PTACs)—which provide verification assistance to veterans—to obtain information about their opinions of VA’s procedures to verify applications and communicate with veterans. We selected the veterans groups based on our prior work in the area and the PTACs based on recommendations from the Defense Logistics Agency and the Association for Procurement Technical Assistance Centers and to obtain geographic diversity. There are 98 PTACs in the United States with more than 300 local offices. We interviewed counselors at the Florida, Missouri, Nevada, and Washington PTACs. We also analyzed data collected through the case file review to determine the extent to which VA complied with its procedures for communicating with applicants and verified businesses. We reviewed methods VA used to collect feedback from program participants, such as documents relating to VA’s help desk customer satisfaction surveys, survey instruments approved by the Office of Management and Budget, results from surveys that had been deployed as of October 2015, and results from focus groups conducted to identify areas for improvement in the verification process. We also assessed the readability of five determination letters that VA sent to veteran applicants from August through December 2014 to corroborate testimonial evidence from veterans groups indicating that these letters can be difficult to understand. We selected these letters by taking the first five cases from our case file review sample that had been issued either a predetermination or denial decision. To determine the reading level at which determination notices were written, we used an automated readability tool, the Flesch-Kincaid Grade-Level test, which rates text on a U.S. school grade level. To assess the steps VA has taken to identify and address verification program challenges and longer-term goals, we reviewed prior work on the verification program that we and VA’s Office of Inspector General (OIG) conducted. We also reviewed VA’s planning, organizational, and budget documents, such as OSDBU’s 2014-2018 Strategic Plan, OSDBU’s 2016 Operating Plan, CVE organizational charts, and CVE’s budget for fiscal years 2014 and 2015. We compared these planning documents with useful practices and lessons learned on organizational transformations, as identified in previous GAO work. We interviewed VA officials to determine what steps, if any, they have taken to address issues we or the OIG identified, and identify and address other challenges associated with the verification program. We also discussed VA’s plans to restructure the verification process with officials from VA’s OSDBU and CVE. We also reviewed documents pertaining to the new verification process, such as process maps and work instructions for the pilot verification process, the reverification policy issued in October 2015, and the revised program regulations posted for public comment in November 2015. We reviewed VA’s human capital and staff management practices, including CVE’s organizational structure, leadership, and reliance on contractors to conduct verification activities. We analyzed VA data on the number of contractors and federal staff working on the verification program and compared these numbers to those found in our 2010 and 2013 reports. We used testimonial evidence obtained during interviews with contractors and agency officials to describe the responsibilities of contractor and federal staff. We also reviewed VA’s policy and planning documents and position descriptions to describe the changes it plans to make to its organizational structure. To assess the progress VA has made in modifying or replacing its information technology system for case management, we reviewed project planning documents that included a list of technical requirements, as well other contract documents. We interviewed VA officials about their plans for developing the new case-management system, including plans for issuing a solicitation, fully transitioning to the new system, and ensuring the system supports the pilot verification process. We conducted this performance audit from December 2014 to March 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Harry Medina (Assistant Director); Katie Boggs (Analyst-in-Charge), Meghana Acharya, Mark Bird, Charlene Calhoon, Pamela Davidson, Kathleen Donovan, Beth Faraguna, John McGrail, Barbara Roesmann, and Jeff Tessin made key contributions to this report.

Summary: In fiscal year 2014, the VA made contract awards totaling $4.0 billion to veteran-owned small businesses, including $3.6 billion to service-disabled veteran-owned small businesses. VA must verify the ownership, control, and status of firms seeking such preferences. GAO found in January 2013 ( GAO-13-95 ) that VA faced challenges verifying firms on a timely and consistent basis, communicating with veterans, enhancing information technology systems, and developing and implementing long-term strategic plans. GAO assessed (1) VA's progress in establishing a timely and consistent verification program and improving communication with veterans, and (2) the steps VA has taken to identify and address program challenges and longer-term goals. GAO reviewed VA's verification procedures and strategic plan, reviewed a generalizable random sample of 96 verification applications, and interviewed VA officials and representatives from two veterans' organizations selected from prior work and four verification assistance counselors selected to obtain geographic representation. Since GAO's 2013 report, the Department of Veterans Affairs (VA) took significant steps to improve how it verifies and communicates with veteran-owned small businesses, consistent with several of GAO's previous recommendations. VA reported that due to process improvements, it reduced average application processing times by more than 50 percent—from 85 days in 2012 to 41 in 2015. VA reported that it generally met its regulatory goals for application processing, and GAO's review of randomly selected application files generally corroborated this statement. VA refined the program's quality controls and implemented an internal audit process. Veterans' organizations and verification counselors with whom GAO spoke noted improvements in VA's communications and interactions with veterans, although three of the four verification counselors with whom GAO spoke suggested the program's website could be clearer and all four said the agency's letters to veterans could be clearer. In response, VA officials said they have been redesigning the website to make documents easier to locate. Officials also said the regulatory language in the letters was necessary and they encourage veterans to obtain free assistance with their applications from VA-certified counselors. VA has been undertaking multiple efforts to address continuing verification program challenges (such as an outdated case-management system) and long-term goals (making processes more veteran-friendly). However, the agency has not had a comprehensive operational plan for managing these efforts to completion. GAO previously recommended that VA's Office of Small and Disadvantaged Business Utilization (OSDBU), which oversees the verification program, establish a strategic plan for the program and integrate efforts to replace an outdated case-management system with agency strategic planning. VA developed a strategic plan for 2014–2018 that included longer-term goals for the program, such as making the verification process more veteran-friendly. In August 2015, VA began piloting a restructured process that allows veterans to communicate directly with the individual processing their application. VA plans to fully transition to the new process by September 2016. VA recently hired a new director for the program, which has had four directors since 2011, including two acting directors in 2015. VA also has continued its efforts to replace the outdated case-management system for the program, but has faced delays due to contractor performance and funding issues. As a result, VA officials do not anticipate the replacement system will be in place until early 2017. While VA has developed a high-level operating plan for OSDBU, the plan does not integrate schedules or specify actions and milestone dates for achieving the multiple changes under way or discuss how to integrate the efforts. VA officials told GAO they were working on developing a detailed operating plan but were waiting to evaluate preliminary results of the verification pilot. GAO's work on organizational transformations states that organizations should set implementation goals and develop timelines to show progress. A detailed plan to guide multiple ongoing efforts is critical given repeated delays in VA's efforts to acquire a new case-management system and the lack of continuity in the program's leadership. Once such an operating plan is developed, it also will be important to update it on a timely basis. Otherwise, VA would continue to be at risk for delays in implementing its initiatives and achieving its long-term goals. GAO recommends that VA: (1) complete its fiscal year 2016 operating plan and include an integrated schedule addressing key actions for the verification program and milestone dates for achieving them, and (2) establish a process to review and update the operating plan to address changing conditions. VA agreed with these recommendations.