In 2007, VA established the VCL, a 24-hour crisis line staffed by responders trained to assist veterans in emotional crisis. Through an interagency agreement, VA collaborated with SAMHSA to use a single, national toll-free number for crisis calls that serves both Lifeline and the VCL. Through this interagency agreement, VA and SAMHSA set up a cooperative relationship between the agencies that would provide consistent suicide-prevention techniques to callers. The national toll-free number presents callers with choices. Callers are greeted by a recorded message that explains the function of the crisis line and prompts individuals to press “1” to reach the VCL. Callers who do not press “1” by the end of the message are routed to one of Lifeline’s 164 local crisis centers. All callers who press “1” are routed first to the VCL primary center. Calls that are not answered at the VCL primary center within 30 seconds of the time that the caller presses “1” during the Lifeline greeting are automatically routed to one of five VCL backup call centers. If a call is not answered by the VCL backup call center that initially receives it, the call may be sent to another VCL backup call center. VA entered into a contract with a firm to oversee the operations of the VCL backup call centers. At the time of our 2016 report, there were a total of 164 Lifeline local crisis centers, 5 of which also serve the VCL. VA added online chat and text message capabilities to the VCL in fiscal years 2009 and 2012, respectively. The number of online chats and text messages handled by the VCL generally increased every year, though the number of online chats decreased in fiscal year 2015. In our covert testing of the VCL’s call response time in July and August 2015, we found that it was uncommon for VCL callers to wait an extended period before reaching a responder since all of our calls that reached the VCL were answered in less than 4 minutes. However, we also found VA did not meet its goal of answering 90 percent of calls to the VCL within 30 seconds for test calls that we made. Our test calls included a generalizable sample of 119 test calls that could be used to describe all callers’ wait times when calling the VCL during this period. On the basis of our test calls, we estimated that during July and August 2015 about 73 percent of all VCL calls were answered at the VCL primary center within 30 seconds. VA officials told us that, during fiscal year 2015, about 65 to 75 percent of VCL calls were answered at the VCL primary center and about 25 to 35 percent of VCL calls were answered at the backup call centers. These VA-reported results indicate that about 65 to 75 percent of VCL calls were answered within either 30 or 60 seconds. These results are consistent with our test results for July and August 2015. During our 2016 review, VA officials told us that VA attempts to maximize the percentage of calls answered at the VCL primary center because these responders have additional resources—including access to veterans’ VA electronic medical records—that are unavailable to VCL backup call center responders. All responders—whether at primary or backup centers—receive specialized training to assist callers in crisis. To improve its performance toward meeting the goal of answering 90 percent of calls at the VCL primary center within 30 seconds, VA implemented two changes in fiscal year 2015—namely, staggered work shifts for responders and new call-handling procedures. Staggered work shifts. VA implemented staggered shifts for responders at the VCL primary center on September 6, 2015. Staggered shifts are work schedules that allow employees to start and stop their shifts at different times as a way to ensure better coverage during peak calling periods. Specifically, it helps schedule more employees to work when call volume is highest and fewer employees to work when call volume is lowest. Additionally, staggered shifts help limit disruptions in service as responders begin and end their shifts. By comparing VCL telephone call data from September through December of 2014 to that of September through December of 2015, we found that VA’s implementation of staggered shifts at the VCL primary center had mixed results. For example, the average percentage of calls answered per hour at the VCL primary center from September through December 2015—after staggered shifts were implemented—was 75 percent, slightly less than the average of 79 percent answered during the corresponding period in 2014 before staggered shifts were implemented. However, the VCL received an average of about 1.3 more calls per hour during this period in 2015 than it received during the corresponding period in 2014 and, according to VA officials, the VCL primary center employed fewer responder staff in 2015 than 2014. New call-handling procedures. VA implemented new call handling procedures at the VCL primary center beginning in June 2015 that provided responders with specific guidance to more efficiently handle “noncore” callers—those callers who were not seeking crisis assistance but rather seeking help with other issues, such as help with veterans’ benefits questions. For example, if a caller reached the VCL with a question about VA disability benefits, the VCL primary center responder would verify that the caller was not in crisis and transfer the caller to the Veterans Benefits Administration to address the question. VCL telephone call data provided by VA suggest that the average time VCL primary center responders spent handling noncore calls decreased by approximately 30 percent over a 5-month period after responder training began on these new call-handling procedures. We would expect that as the average time VCL primary center responders spent handling noncore calls decreased, these responders would have more time available to answer additional incoming calls. To determine the timeliness of the VCL’s responses to online chats and text messages, we conducted a covert test in July and August 2015 using nongeneralizable samples of 15 online chats and 14 text messages. All 15 of our test online chats received responses within 60 seconds, 13 of which were within 30 seconds. This result was consistent with VA data that indicated VCL responders sent responses to over 99 percent of online chat requests within 1 minute in fiscal years 2014 and 2015. During our 2016 review, VA officials told us that all online chats are expected to be answered immediately. Although this was an expectation, we found in 2016 that VA did not have formal performance standards for how quickly responders should answer online chat requests and expected to develop them before the end of fiscal year 2016. However, our tests of text messages revealed a potential area of concern. Four of our 14 test text messages did not receive a response from the VCL. Of the remaining 10 test text messages, 8 received responses within 2 minutes, and 2 received responses within 5 minutes. As we reported in May 2016, VA officials stated that text messages are expected to be answered immediately, but, as with online chats, VA had not developed formal performance standards for how quickly responders should answer text messages. VA data indicated that VCL responders sent responses to 87 percent of text messages within 2 minutes of initiation of the conversation in both fiscal years 2014 and 2015. During our 2016 review, VA officials said that VA planned to establish performance standards for answering text messages before the end of fiscal year 2016. VA officials noted and we observed during a site visit that some incoming texts were abusive in nature or were not related to a crisis situation. According to VA officials, in these situations, if this is the only text message waiting for a response, a VCL responder will send a response immediately. However, if other text messages are awaiting responses, VA will triage these text messages and reply to those with indications of crisis first. This triage process may have contributed to the number of our test texts that did not receive responses within 2 minutes. The VCL’s text messaging service provider offered several reasons for the possible nonresponses that we encountered in our test results. These included: (1) incompatibilities between some devices used to send text messages to the VCL and the software VA used to process the text messages, (2) occasional software malfunctions that freeze the text messaging interface at the VCL primary center, (3) inaudible audio prompts used to alert VCL primary center responders of incoming text messages, (4) attempts by people with bad intentions to disrupt the VCL’s text messaging service by overloading the system with a large number of texts, and (5) incompatibilities between the web browsers used by the VCL primary center and the text messaging software. At the time of our 2016 review, VA officials told us that they did not monitor and test the timeliness and performance of the VCL text messaging system, but rather relied solely on the VCL’s text messaging service provider for such monitoring and testing. They said that the provider had not reported any issues with this system. According to the provider, no routine testing of the VCL’s text messaging system was conducted. Standards for internal control in the federal government state that ongoing monitoring should occur in the course of normal operations, be performed continually, and be ingrained in the agency’s operations. We concluded that without routinely testing its text messaging system, or ensuring that its provider tests the system, VA cannot ensure that it is identifying limitations with its text messaging service and resolving them to provide consistent, reliable service to veterans. We recommended that VA regularly test the VCL’s text messaging system to identify issues and correct them promptly. In response, VA developed and implemented procedures to regularly test the VCL’s text messaging system, as well as its telephone and online chat systems. We believe this change will allow VA to more reliably and quickly identify and correct errors in the text messaging system and therefore help veterans reach VCL responders in a timelier manner. As we reported in May 2016, VA had sought to enhance its capabilities for overseeing VCL primary center operations through a number of activities—including establishing a call center evaluation team, implementing revised performance standards for VCL primary center responders, implementing silent monitoring of VCL primary center responders, and analyzing VCL caller complaints. Establishment of a call center evaluation team. In October 2013, VA established a permanent VCL call center evaluation team that is responsible for monitoring the performance of the VCL primary center. As we reported in May 2016, the call center evaluation team analyzes VCL data, including information on the number of calls received and the number of calls routed to backup call centers from the primary center. VA officials told us that they use these data to inform management decisions about VCL operations. Implementation of revised performance standards for VCL primary center responders. In October 2015, VA implemented new performance standards for all VCL primary center responders that will be used to assess their performance in fiscal year 2016. According to VA officials, these performance standards include several measures of responder performance—such as demonstrating crisis-intervention skills, identifying callers’ needs, and addressing those needs in an appropriate manner using VA approved resources. Silent monitoring of VCL primary center responders. In February 2016, VA officials reported that they were beginning silent monitoring of all VCL responders using recently developed standard operating procedures, standard data collection forms, and standard feedback protocols. Analysis of VCL caller complaints. In October 2014, VA created a mechanism for tracking complaints it receives from VCL callers and external parties, such as members of Congress and veterans, about the performance of the VCL primary and backup call centers. According to VA officials, each complaint is investigated to validate its legitimacy and determine the cause of any confirmed performance concerns. The results and disposition of each complaint are documented in VA’s complaint tracking database. In 2011, VA established key performance indicators to evaluate VCL primary center operations; however, in our May 2016 review, we found these indicators did not have established measureable targets or time frames for their completion. VCL key performance indicators lacked measurable targets. We found that VA’s list of VCL key performance indicators did not include information on the targets the department had established to indicate their successful achievement. For example, VA included a key performance indicator for the percentage of calls answered by the VCL in this list but did not include information on what results would indicate success for (1) the VCL as a whole, (2) the VCL primary center, or (3) the VCL backup call centers. As another example, VA had not established targets for the percentage of calls abandoned by callers prior to speaking with VCL responders. Measureable targets should include a clearly stated minimum performance target and a clearly stated ideal performance target. These targets should be quantifiable or otherwise measurable and indicate how well or at what level an agency or one of its components aspires to perform. Such measurable targets are important for ensuring that the VCL call center evaluation team can effectively measure VCL performance. VCL key performance indicators lack time frames for their completion. We found that VA’s list of VCL key performance indicators did not include information on when the department expected the VCL to complete or meet the action covered by each key performance indicator. For example, for VA’s key performance indicator for the percentage of calls answered by the VCL, the department had not included a date by which it would expect the VCL to complete this action. As another example, VA had not established dates by which it would meet targets yet to be established for the percentage of calls abandoned by callers prior to speaking with VCL responders. Time frames for action are a required element of performance indicators and are important to ensure that agencies can track their progress and prioritize goals. Guidance provided by the Office of Management and Budget states that performance goals—similar to VA’s key performance indicators for the VCL—should include three elements: (1) a performance indicator, which is how the agency will track progress; (2) a target; and (3) a period. Without establishing targets and time frames for the successful completion of its key performance indicators for the VCL, we concluded that VA could not effectively track and publicly report progress or results for its key performance indicators for accountability purposes. We recommended that VA document clearly stated and measurable targets and time frames for key performance indicators needed to assess VCL performance. While VA officials have informed us that they have created scorecards that track information related to calls answered, staffing, and average handle times, as of March 2017, clearly stated and measurable targets and time frames have not yet been developed. As we reported in May 2016, VA’s backup call coverage contract, awarded in October 2012 and in place at the time of our review, did not include detailed performance requirements in several key areas for the VCL backup call centers. Clear performance requirements for VCL backup call centers are important for defining VA’s expectations of these service partners. However, VA had taken steps to strengthen the performance requirements of this contract by modifying it in March 2015 and was beginning the process of replacing it with a new contract. According to VA officials, the new contract was awarded in April 2016. October 2012 backup call coverage contract. This contract provided a network of Lifeline local crisis centers that could serve as VCL backup call centers managed by a contractor. This contractor was responsible for overseeing and coordinating the services of VCL backup call centers that answer overflow calls from the VCL primary center. This contract as initially awarded included few details on the performance requirements for VCL backup call centers. For example, the contract did not include any information on the percentage of VCL calls routed to each VCL backup call center that should be answered. Detailed performance requirements on these key aspects of VCL backup call center performance are necessary for VA to effectively oversee the performance of the contractor and the VCL backup call centers. By not specifying performance requirements for the contractor on these key performance issues, we believe that VA missed the opportunity to validate contractor and VCL backup call center performance and mitigate weaknesses in VCL call response. As we reported in May 2016, VA officials told us about several concerns with the performance of the backup call centers operating under the October 2012 contract based on their own observations and complaints reported to the VCL. These concerns included the inconsistency and incompleteness of VCL backup call centers’ responses to VCL callers, limited or missing documentation from records of VCL calls answered by VCL backup call center responders, limited information provided to VA that could be used to track VCL backup call center performance, and the use of voice answering systems or extended queues for VCL callers reaching some VCL backup call centers. For example, VA officials reported that some veterans did not receive complete suicide assessments when their calls were answered at VCL backup call centers. In addition, VA officials noted that they had observed some VCL backup call centers failing to follow VCL procedures, such as not calling a veteran who may be in crisis when a third-party caller requested that the responder contact the veteran. According to VA officials, these issues led to additional work for the VCL primary center, including staffing one to two responders per shift to review the call records submitted to the VCL primary center by backup call centers and to determine whether these calls required additional follow-up from the VCL primary center. These officials estimated that 25 to 30 percent of backup call center call records warranted additional follow-up to the caller from a VCL primary center responder, including approximately 5 percent of backup call center call records that needed to be completely reworked by a VCL primary center responder. March 2015 backup call coverage contract modification. Given these concerns, in March 2015, VA modified the October 2012 backup call coverage contract to add more explicit performance requirements for its backup call coverage contractor, which likely took effect more quickly than if the department had waited for a new contract to be awarded. These modified requirements included (1) the establishment of a 24- hours-a-day, 7-days-a-week contractor-staffed emergency support line that VCL backup call centers could use to report problems, (2) a prohibition on VCL backup call centers’ use of voice answering systems, (3) a prohibition on VCL backup call centers placing VCL callers on hold before a responder conducted a risk assessment, (4) documentation of each VCL caller’s suicide risk assessment results, and (5) transmission of records for all VCL calls to the VCL primary center within 30 minutes of the call’s conclusion. Development of new backup call coverage contract. In July 2015, VA began the process of replacing its backup call coverage contract by publishing a notice to solicit information from prospective contractors on their capability to satisfy the draft contract terms for the new contract; this new backup call coverage contract was awarded in April 2016. We found that these new proposed contract terms included the same performance requirement modifications that were made in March 2015, as well as additional performance requirements and better data reporting from the contractor that could be used to improve VA’s oversight of the VCL backup call centers. Specifically, the proposed contract terms added performance requirements to address VCL backup call center performance—including a requirement for 90 percent of VCL calls received by a VCL backup call center to be answered by a backup call center responder within 30 seconds and 100 percent to be answered by a backup call center responder within 2 minutes. In addition, the proposed contract terms included numerous data reporting requirements that could allow VA to routinely assess the performance of its VCL backup call centers and identify patterns of noncompliance with the contract’s performance requirements more efficiently and effectively than under the prior contract. The proposed terms for the new contract also state that VA will initially provide and approve all changes to training documentation and supporting materials provided to VCL backup call centers in order to promote the contractor’s ability to provide the same level of service that is being provided by the VCL primary center. In May 2016, we found that when callers did not press “1” during the initial Lifeline greeting, their calls may take longer to answer than if the caller had pressed “1” and been routed to either the VCL primary center or a VCL backup call center. As previously discussed, VA and SAMHSA collaborated to link the toll-free numbers for both Lifeline and the VCL through an interagency agreement. The greeting instructs callers to press “1” to be connected to the VCL; if callers do not press “1,” they will be routed to one of SAMHSA’s 164 Lifeline local crisis centers. To mimic the experience of callers who did not press “1” to reach the VCL when prompted, we made 34 covert nongeneralizable test calls to the national toll-free number that connects callers to both Lifeline and the VCL during August 2015 and we did not press “1” to be directed to the VCL. For 23 of these 34 calls, our call was answered in 30 seconds or less. For 11 of these calls, we waited more than 30 seconds for a responder to answer— including 3 calls with wait times of 8, 9, and 18 minutes. Additionally, one of our test calls did not go through, and during another test call we were asked if we were safe and able to hold. VA’s policy prohibits VCL responders from placing callers on hold prior to completing a suicide assessment; Lifeline has its own policies and procedures. According to officials and representatives from VA, SAMHSA, and the VCL backup call centers, as well as our experience making test calls where we did not press “1,” there are several reasons why a veteran may not press “1” to be routed to the VCL, including an intentional desire to not connect with VA, failure to recognize the prompt to press “1” to be directed to the VCL, waiting too long to respond to the prompt to press “1” to be directed to calling from a rotary telephone that does not allow the caller to press “1” when prompted. As we found in May 2016, VA officials had not estimated the extent to which veterans intending to reach the VCL did not press “1” during the Lifeline greeting. These officials explained that their focus had been on ensuring that veterans who did reach the VCL received appropriate service from the VCL primary center and backup call centers. In addition, SAMHSA officials said that they also did not collect this information. These officials reported that SAMHSA did not require the collection of demographic information, including veteran status, for a local crisis center to participate in the Lifeline network. However, they noted that SAMHSA could request through its grantee that administers the Lifeline network that local crisis centers conduct a one-time collection of information to help determine how often and why veterans reach Lifeline local crisis centers. SAMHSA officials explained that they could work with the Lifeline grantee to explore optimal ways of collecting this information that would be (1) clinically appropriate, (2) a minimal burden to callers and Lifeline’s local crisis centers, and (3) in compliance with the Office of Management and Budget’s paperwork reduction and information collection policies. The interagency agreement between VA and SAMHSA assigns SAMHSA responsibilities for monitoring the use of the national toll-free number that is used to direct callers to both the VCL and Lifeline. These responsibilities include monitoring the use of the line, analyzing trends, and providing recommendations about projected needs and technical modifications needed to meet these projected needs. Using the information collected from the Lifeline local crisis centers on how often and why veterans reach Lifeline, as opposed to the VCL, VA and SAMHSA officials could then assess whether the extent to which this occurs merits further review and action. Although the results of our test were not generalizable, substantial wait times for a few of our covert calls suggested that some callers may experience longer wait times to speak with a responder in the Lifeline network than they would in the VCL’s network. We concluded that without collecting information to examine how often and why veterans do not press “1” when prompted to reach the VCL, VA and SAMHSA could not determine the extent veterans reach the Lifeline network when intending to reach the VCL and may experience longer wait times as a result. In addition, limitations in information on how often and why this occurs did not allow VA and SAMHSA to determine whether or not they should collaborate on plans to address the underlying causes of veterans not reaching the VCL. Standards for internal control in the federal government state that information should be communicated both internally and externally to enable the agency to carry out its responsibilities. For external communications, management should ensure there are adequate means of communicating with, and obtaining information from, external stakeholders that may have a significant impact on the agency achieving its goals. We recommended VA and SAMHSA collaborate in taking the following two actions: (1) collect information on how often and why callers intending to reach the VCL instead reach Lifeline local crisis centers and (2) review the information collected and, if necessary, develop plans to address the underlying causes. We understand that VA and SAMHSA have been coordinating on these issues. However, as of March 2017, both of these recommendations remain open. Chairman Roe, Ranking Member Walz, and Members of the Committee, this concludes our statement for the record. For questions about this statement, please contact Seto J. Bagdoyan at (202) 512-6722 or bagdoyans@gao.gov or Randall B. Williamson at (202) 512-7114 or williamsonr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. In addition to the contacts listed, GAO staff who made key contributions to this statement are Jennie F. Apter, Christie Enders, Cathleen J. Hamann, Marcia A. Mann, Vikki Porter, Lisa Rogers, and Julie T. Stewart. This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.

Summary: VA established the VCL in July 2007 to provide support to veterans in emotional crisis. Between fiscal years 2008, its first full year of operation, and 2015, the number of calls received by the VCL increased almost 700 percent, exceeding VA's expectations. As VA began to address increasing numbers of requests for assistance, reports of dissatisfaction with VCL's service periodically appeared in the media. This statement summarizes GAO's May 2016 report on VA's oversight of the VCL. Specifically, the statement describes (1) the extent to which VA met response-time goals for VCL calls and text messages, (2) how VA monitored VCL primary center call center operations, and (3) how VA worked with VCL service partners to help ensure veterans receive high-quality service. For the May 2016 report, GAO visited the VCL's primary center and two backup call centers; tested VCL response time through a generalizable sample of covert telephone calls and a nongeneralizable sample of text messages in July and August 2015; reviewed internal reports and policies and plans; and interviewed VA officials. GAO contacted VA and SAMHSA for an update on actions taken in response to the report's recommendations. In May 2016, GAO found that the Department of Veterans Affairs (VA) did not meet its call response-time goals for the Veterans Crisis Line (VCL), although extended call wait times were not common; issues with text messages were also found. VA's goal was to answer 90 percent of VCL calls at the VCL primary center within 30 seconds in the period GAO studied. Calls not answered within 30 seconds were to route to VCL backup call centers; however, for 5 months of fiscal year 2015, calls were routed to VCL backup call centers after 60 seconds. According to VA officials about 65 to 75 percent of VCL calls were answered at the VCL primary center in fiscal year 2015 within either 30 or 60 seconds. GAO's covert testing in July and August 2015 confirmed this. Specifically, 119 covert test calls showed that an estimated 73 percent of calls made during this period were answered within 30 seconds. GAO also covertly tested the VCL's text messaging services and found that 4 of 14 GAO test text messages did not receive responses. GAO recommended that VA regularly test the VCL's text messaging system to identify issues and correct them promptly. In response, VA developed and implemented procedures to regularly test the VCL's text messaging system, as well as its telephone and online chat systems. In May 2016, GAO found that VA had taken a number of steps to improve its monitoring of the VCL primary center operations. For example, VA had established a permanent VCL call center evaluation team and created a mechanism for tracking complaints about the performance of the VCL primary center from VCL callers or external parties. However, GAO found that VA had not specified quantifiable or otherwise measurable targets and had not included dates for when it would expect the VCL to complete actions covered by each key performance indicator. This was inconsistent with guidance provided by the Office of Management and Budget. GAO recommended that VA document clearly stated and measurable targets and time frames for key performance indicators needed to assess VCL performance. Although VA agreed, as of March 2017, this recommendation remains open. GAO also found that VA had established an interagency agreement with its service partner, the Department of Health and Human Services' (HHS) Substance Abuse and Mental Health Services Administration (SAMHSA), to manage the shared operations of the VCL and Lifeline (a public-private network that provides free and confidential support to people in suicidal crisis or emotional distress). Despite these efforts to coordinate, VA and SAMHSA did not collect information needed to assess how often and why callers intending to reach the VCL do not follow voice prompts and instead reach Lifeline local crisis centers. As a result, VA and SAMHSA did not know the extent to which this occurred and could not determine the underlying causes that may need to be addressed. GAO recommended that VA and SAMHSA (1) collect information on how often and why callers intending to reach the VCL instead reach Lifeline local crisis centers and (2) review the information collected and, if necessary, develop plans to address the underlying causes. Although VA and SAMHSA agreed, as of March 2017, these recommendations remain open. In its May 2016 report, GAO made four recommendations to VA and HHS to improve the VCL. VA and HHS concurred with the recommendations and while action has been taken on one, three remain open as of March 2017.