In 2007, VA established the VCL, a 24-hour crisis line staffed by responders trained to assist veterans in emotional crisis. Through an interagency agreement, VA collaborated with SAMHSA to use a single, national toll-free number for crisis calls that serves both Lifeline and the VCL. Through this interagency agreement, VA and SAMHSA set out to establish a seamless crisis-management system through a collaborative and cooperative relationship between the agencies that would provide consistent suicide-prevention techniques to callers. The national toll-free number presents callers with choices. Callers are greeted by a recorded message that explains the function of the crisis line and prompts individuals to press “1” to reach the VCL. Callers who do not press “1” by the end of the message are routed to one of Lifeline’s 164 local crisis centers. All callers who press “1” are routed first to the VCL primary center. Calls that are not answered at the VCL primary center within 30 seconds of the time that the caller presses “1” during the Lifeline greeting are automatically routed to one of five VCL backup call centers. If a call is not answered by the VCL backup call center that initially receives it, the call may be sent to another VCL backup call center. VA entered into a contract with a firm to oversee the operations of the VCL backup call centers. There are a total of 164 Lifeline local crisis centers, 5 of which also serve the VCL. (See fig. 1.) The number of calls reaching the VCL has increased substantially since the VCL’s first full year of operation. Increases in the number of VCL calls received have corresponded with increased annual funding obligations for the VCL. (See fig. 2.) VA added online chat and text message capabilities to the VCL in fiscal years 2009 and 2012, respectively. The number of online chats and text messages handled by the VCL generally increased every year, though the number of online chats decreased in fiscal year 2015. (See fig. 3.) To determine how well VA performed against its goal for responding to VCL callers, we covertly tested the VCL’s call response time in July and August 2015. During this testing we found that it was uncommon for VCL callers to wait an extended period before reaching a responder since all of our calls that reached the VCL were answered in less than 4 minutes. According to VA officials, VA established a goal that the VCL primary center would answer 90 percent of calls to the VCL within 30 seconds. Our test included a generalizable sample of 119 test calls that can be used to describe all callers’ wait times when calling the VCL during this period. On the basis of our test calls, we estimate that during July and August 2015 about 73 percent of all VCL calls were answered at the VCL primary center within 30 seconds. (See fig. 4.) VA officials told us that, during fiscal year 2015, about 65 to 75 percent of VCL calls were answered at the VCL primary center and about 25 to 35 percent of VCL calls were answered at the backup call centers. These VA-reported results indicate that about 65 to 75 percent of VCL calls were answered within either 30 or 60 seconds. These results are consistent with our test results for July and August 2015. According to VA officials, VA attempts to maximize the percentage of calls answered at the VCL primary center because these responders have additional resources— including access to veterans’ VA electronic medical records—that are unavailable to VCL backup call center responders. All responders receive specialized training to assist callers in crisis. To improve its performance toward meeting the goal of answering 90 percent of calls at the VCL primary center within 30 seconds, VA implemented two changes in fiscal year 2015—namely, staggered work shifts for responders and new call-handling procedures. Staggered work shifts. VA implemented staggered shifts for responders at the VCL primary center on September 6, 2015. Staggered shifts are work schedules that allow employees to start and stop their shifts at different times as a way to ensure better coverage during peak calling periods. Specifically, it helps schedule more employees to work when call volume is highest and fewer employees to work when call volume is lowest. Additionally, staggered shifts help limit disruptions in service as responders begin and end their shifts. By comparing VCL telephone call data from September through December of 2014 to that of September through December of 2015, we found that VA’s implementation of staggered shifts at the VCL primary center had mixed results. Overall: The average percentage of calls answered per hour at the VCL primary center from September through December 2015—after staggered shifts were implemented—was 75 percent, slightly less than the average of 79 percent answered during the corresponding period in 2014 before staggered shifts were implemented. However, the VCL received an average of about 1.3 more calls per hour during this period in 2015 than it received during the corresponding period in 2014 and, according to VA officials, the VCL primary center employed fewer responder staff in 2015 than 2014. By day of the week: The average percentage of calls answered per hour at the VCL primary center increased on Mondays to 89 percent and Tuesdays to 83 percent after VA implemented staggered shifts, up from 78 percent and 79 percent, respectively, during the corresponding period in 2014. These increases suggest that staggered shifts may have helped VA answer more calls at the VCL primary center on these days because VCL call data from our analysis indicate that these days of the week typically experienced the highest number of calls prior to implementing staggered shifts, and VA officials told us that they used the implementation of staggered shifts to schedule more responders on these days. However, the average percentage of calls answered per hour at the VCL primary center decreased on Saturdays to 61 percent and Sundays to 70 percent after VA implemented staggered shifts, down from 78 percent and 80 percent, respectively, during the corresponding period in 2014. By hours of the day: VA answered a higher percentage of calls at the VCL primary center during the mid-day and evening hours after the implementation of staggered shifts. Specifically, from 11:00 a.m. to 4:00 p.m. and from 9:00 p.m. to 11:00 p.m. the VCL primary center answered a higher percentage of calls compared with the corresponding periods in 2014. However, VA answered a lower percentage of calls at the VCL primary center during overnight hours—midnight to 9:00 a.m.—and in the early evening—5:00 p.m. to 8:00 p.m.—compared to corresponding periods in 2014. To address staffing limitations and align the number of responders available for each staggered shift according to demand, VA officials said that VA planned to hire 63 additional responders for the VCL primary center in fiscal year 2016 and assign these new responders to weekend and evening shifts. This change would likely help improve the mixed results we identified in our analysis of VA’s implementation of staggered shifts for responders. As of February 2016, VA officials said that 22 applicants had accepted employment offers and that VA planned to extend employment offers to an additional 15 applicants. These officials also noted that recent attrition at the VCL primary center was largely due to VCL primary center responders being promoted into new positions at the VCL primary center or to VCL primary center responders leaving because their work with the VCL did not qualify as clinical hours required for licensure in their specialties. New call-handling procedures. VA implemented new call handling procedures at the VCL primary center beginning in June 2015 that provided responders with specific guidance to more efficiently handle “noncore” callers—those callers who were not seeking crisis assistance but rather seeking help with other issues, such as help with veterans’ benefits questions. For example, if a caller reached the VCL with a question about VA disability benefits, the VCL primary center responder would verify that the caller was not in crisis and transfer the caller to the Veterans Benefits Administration to address the question. VCL telephone call data provided by VA suggest that the average time VCL primary center responders spend handling noncore calls decreased by approximately 30 percent over a 5-month period after responder training began on these new call-handling procedures. We would expect that as the average time VCL primary center responders spend handling noncore calls decreases, these responders should have more time available to answer additional incoming calls. To determine the timeliness of the VCL’s responses to online chats and text messages, we conducted a covert test in July and August 2015 using nongeneralizable samples of 15 online chats and 14 text messages. All 15 of our test online chats received responses within 60 seconds, 13 of which were within 30 seconds. This result is consistent with VA data that indicate VCL responders sent responses to over 99 percent of online chat requests within 1 minute in fiscal years 2014 and 2015. VA officials said that all online chats are expected to be answered immediately. Although this is an expectation, VA does not yet have formal performance standards for how quickly responders should answer online chat requests and expects to develop them before the end of fiscal year 2016. However, our tests of text messages revealed a potential area of concern. Four of our 14 test text messages did not receive a response from the VCL. Of the remaining 10 test text messages, 8 received responses within 2 minutes, and 2 received responses within 5 minutes. VA officials stated that text messages are expected to be answered immediately, but, as with online chats, VA has not yet developed formal performance standards for how quickly responders should answer text messages. VA data indicate that VCL responders sent responses to 87 percent of text messages within 2 minutes of initiation of the conversation in both fiscal years 2014 and 2015. VA officials said that VA plans to establish performance standards for answering text messages before the end of fiscal year 2016. VA officials noted and we observed during a site visit that some incoming texts were abusive in nature or were not related to a crisis situation. According to VA officials, in these situations, if this is the only text message waiting for a response, a VCL responder will send a response immediately. However, if other text messages are awaiting responses, VA will triage these text messages and reply to those with indications of crisis first. This triage process may have contributed to the number of our test texts that did not receive responses within 2 minutes. The VCL’s text messaging service provider offered several reasons for the possible nonresponses that we encountered in our test results. These included: (1) incompatibilities between some devices used to send text messages to the VCL and the software VA used to process the text messages, (2) occasional software malfunctions that freeze the text messaging interface at the VCL primary center, (3) inaudible audio prompts used to alert VCL primary center responders of incoming text messages, (4) attempts by people with bad intentions to disrupt the VCL’s text messaging service by overloading the system with a large number of texts, and (5) incompatibilities between the web-browsers used by the VCL primary center and the text messaging software. VA officials told us that they do not monitor and test the timeliness and performance of the VCL text messaging system, but rather rely solely on the VCL’s text messaging service provider for such monitoring and testing. They said that the provider had not reported any issues with this system. According to the provider, no routine testing of the VCL’s text messaging system is conducted. Standards for internal control in the federal government state that ongoing monitoring should occur in the course of normal operations, be performed continually, and be ingrained in the agency’s operations. Without routinely testing its text messaging system, or ensuring that its provider tests the system, VA cannot ensure that it is identifying limitations with its text messaging service and resolving them to provide consistent, reliable service to veterans. VA has sought to enhance its capabilities for overseeing VCL primary center operations through a number of activities—including establishing a call center evaluation team, implementing revised performance standards for VCL primary center responders, implementing silent monitoring of VCL primary center responders, and analyzing VCL caller complaints. Establishment of a call center evaluation team. In October 2013, VA established a permanent VCL call center evaluation team that is responsible for monitoring the performance of the VCL primary center. The call center evaluation team analyzes VCL data, including information on the number of calls received and the number of calls routed to backup call centers from the primary center. VA officials told us that they use these data to inform management decisions about VCL operations. For example, these data were used as part of its decision to implement staggered shifts for VCL primary center responders in an attempt to increase the number of calls answered at the VCL primary center. Implementation of revised performance standards for VCL primary center responders. In October 2015, VA implemented new performance standards for all VCL primary center responders that will be used to assess their performance in fiscal year 2016. According to VA officials, these performance standards include several measures of responder performance—such as demonstrating crisis-intervention skills, identifying callers’ needs, and addressing those needs in an appropriate manner using VA approved resources. VA officials told us that by the summer of 2016 VCL primary center supervisors will have access to real-time information on VCL primary center responders’ performance against these standards and can track their workload and performance periodically. These officials explained that they anticipate these performance standards will be reviewed and revised as needed for the fiscal year 2017 performance year. Silent monitoring of VCL primary center responders. In February 2016, VA officials reported that they were beginning silent monitoring of all VCL responders using recently developed standard operating procedures, standard data collection forms, and standard feedback protocols. These officials explained that the VCL primary center silent monitoring would begin in mid-February 2016 with four VA medical center–based suicide-prevention coordinators completing silent monitoring of 15 to 20 calls a week to the VCL primary center through March 2016. These officials explained that six full-time silent monitors had been hired as part of the VCL quality assurance staff and would begin conducting silent monitoring of VCL primary center calls in April 2016 once their training had been completed. During the initial rollout, the four VA medical center–based suicide-prevention coordinators will remotely access VCL primary center calls, complete the standard data collection form, and send the information to the observed VCL primary center responders’ supervisors for feedback delivery. Once the six full- time silent monitors begin completing these activities, they will complete all call monitoring and deliver feedback to VCL primary center responders and will coordinate with VCL primary center supervisors on an as-needed basis. VA officials explained in February 2016 that they were unsure how many VCL primary center calls these six full-time silent monitors would be able to observe and will clarify this expectation once these silent monitors begin their duties in April 2016. Analysis of VCL caller complaints. In October 2014, VA created a mechanism for tracking complaints it receives from VCL callers and external parties, such as members of Congress and veterans, about the performance of the VCL primary and backup call centers. Complaints can be about services provided by either the VCL primary center or one of the VCL backup call centers. In fiscal year 2015, the VCL received over 200 complaints from veterans and others regarding call center operations. These complaints included issues with VCL primary center and backup call center customer service and wait times to reach a responder. According to VA officials, each complaint is investigated to validate its legitimacy and determine the cause of any confirmed performance concerns. This validation process includes speaking with the complainant and VA staff, as applicable. The results and disposition of each complaint are documented in VA’s complaint tracking database. For complaints that include details on specific responders, VA officials told us that they investigate complaints and use legitimate complaints as part of the performance evaluation process for VCL primary center responders. Specifically, these officials explained that when a complaint about a VCL primary center responder’s customer service is verified as accurate by a VA psychologist or supervisor after it is investigated, it can affect a VCL primary center responder’s annual performance appraisal. The investigation process also includes verifying any associated documentation of the activities at the source of the complaint. In 2011, VA established key performance indicators to evaluate VCL primary center operations; however, we found these indicators did not have established measureable targets or time frames for their completion. VCL key performance indicators lack measurable targets. We found that VA’s list of VCL key performance indicators did not include information on the targets the department had established to indicate their successful achievement. For example, VA included a key performance indicator for the percentage of calls answered by the VCL in this list but did not include information on what results would indicate success for (1) the VCL as a whole, (2) the VCL primary center, or (3) the VCL backup call centers. As another example, VA did not establish targets for the percentage of calls abandoned by callers prior to speaking with VCL responders. Measureable targets should include a clearly stated minimum performance target and a clearly stated ideal performance target. These targets should be quantifiable or otherwise measurable and indicate how well or at what level an agency or one of its components aspires to perform. Such measurable targets are important for ensuring that the VCL call center evaluation team can effectively measure VCL performance. VCL key performance indicators lack time frames for their completion. We found that VA’s list of VCL key performance indicators did not include information on when the department expected the VCL to complete or meet the action covered by each key performance indicator. For example, for VA’s key performance indicator for the percentage of calls answered by the VCL, the department did not include a date by which it would expect the VCL to complete this action. As another example, VA did not establish dates by which it would meet targets yet to be established for the percentage of calls abandoned by callers prior to speaking with VCL responders. Time frames for action are a required element of performance indicators and are important to ensure that agencies can track their progress and prioritize goals. Guidance provided by the Office of Management and Budget states that performance goals—similar to VA’s key performance indicators for the VCL—should include three elements: (1) a performance indicator, which is how the agency will track progress; (2) a target; and (3) a period. VA officials reported that they are currently implementing a comprehensive process improvement plan, discussed later in this report, that will help ensure the right structures and processes are in place, which they believe are logical precursors to examining VCL outcomes and establishing targets and time frames for performance indicators. Without establishing targets and time frames for the successful completion of its key performance indicators for the VCL, VA cannot effectively track and publicly report progress or results for its key performance indicators for accountability purposes. VA’s backup call coverage contract, awarded in October 2012 and in place at the time of our review, did not include detailed performance requirements in several key areas for the VCL backup call centers. Clear performance requirements for VCL backup call centers are important for defining VA’s expectations of these service partners. However, VA has taken steps to strengthen the performance requirements of this contract by modifying it in March 2015 and beginning the process of replacing it with a new contract. October 2012 backup call coverage contract. This contract provided a network of Lifeline local crisis centers that could serve as VCL backup call centers managed by a contractor. This contractor was responsible for overseeing and coordinating the services of VCL backup call centers that answer overflow calls from the VCL primary center. This contract as initially awarded included few details on the performance requirements for VCL backup call centers. For example, the contract did not include any information on several key aspects of VCL backup call center performance, including: (1) the percentage of VCL calls routed to each VCL backup call center that should be answered, (2) VA’s expectations on whether or not VCL backup call centers could use voice answering systems or caller queues for VCL calls, and (3) VA’s documentation requirements for VCL calls answered at the VCL backup call centers. Detailed performance requirements on these key aspects of VCL backup call center performance are necessary for VA to effectively oversee the performance of the contractor and the VCL backup call centers. By not specifying performance requirements for the contractor on these key performance issues, VA missed the opportunity to validate contractor and VCL backup call center performance and mitigate weaknesses in VCL call response. For example, representatives from one VCL backup call center provided data that showed that the backup call center answered about 50 percent of the VCL calls it received. However, without a performance requirement establishing a standard for the percentage of calls each VCL backup call center should answer, VA could not determine whether this was acceptable performance for a VCL backup call center. As of December 2015, this VCL backup call center reported that it had improved its performance and answered about 66 percent of calls it received from July 2015 to December 2015. VA officials told us about several concerns with the performance of the backup call centers operating under the October 2012 contract based on their own observations and complaints reported to the VCL. These concerns included the inconsistency and incompleteness of VCL backup call centers’ responses to VCL callers, limited or missing documentation from records of VCL calls answered by VCL backup call center responders, limited information provided to VA that could be used to track VCL backup call center performance, and the use of voice answering systems or extended queues for VCL callers reaching some VCL backup call centers. For example, VA officials reported that some veterans did not receive complete suicide assessments when their calls were answered at VCL backup call centers. In addition, VA officials noted that they had observed some VCL backup call centers failing to follow VCL procedures, such as not calling a veteran who may be in crisis when a third-party caller requested that the responder contact the veteran. According to VA officials, these issues led to additional work for the VCL primary center, including staffing one to two responders per shift to review the call records submitted to the VCL primary center by backup call centers and to determine whether these calls required additional follow-up from the VCL primary center. These officials estimated that 25 to 30 percent of backup call center call records warranted additional follow-up to the caller from a VCL primary center responder, including approximately 5 percent of backup call center call records that needed to be completely reworked by a VCL primary center responder. March 2015 backup call coverage contract modification. Given these concerns, in March 2015 VA modified the October 2012 backup call coverage contract to add more explicit performance requirements for its backup call coverage contractor, which likely took effect more quickly than if the department had waited for a new contract to be awarded. These modified requirements included (1) the establishment of a 24- hours-a-day, 7-days-a-week contractor-staffed emergency support line that VCL backup call centers could use to report problems, (2) a prohibition on VCL backup call centers’ use of voice answering systems, (3) a prohibition on VCL backup call centers placing VCL callers on hold before a responder conducted a risk assessment, (4) documentation of each VCL caller’s suicide risk assessment results, and (5) transmission of records for all VCL calls to the VCL primary center within 30 minutes of the call’s conclusion. Development of new backup call coverage contract. In July 2015, VA began the process of replacing its backup call coverage contract by publishing a notice to solicit information from prospective contractors on their capability to satisfy the draft contract terms for the new contract; this new backup call coverage contract was awarded in April 2016. We found that these new proposed contract terms included the same performance requirement modifications that were made in March 2015, as well as additional performance requirements and better data reporting from the contractor that could be used to improve VA’s oversight of the VCL backup call centers. Specifically, the proposed contract terms added performance requirements to address VCL backup call center performance—including a requirement for 90 percent of VCL calls received by a VCL backup call center to be answered by a backup call center responder within 30 seconds and 100 percent to be answered by a backup call center responder within 2 minutes. In addition, the proposed contract terms include numerous data reporting requirements that could allow VA to routinely assess the performance of its VCL backup call centers and identify patterns of noncompliance with the contract’s performance requirements more efficiently and effectively than under the prior contract. The proposed terms for the new contract also state that VA will initially provide and approve all changes to training documentation and supporting materials provided to VCL backup call centers in order to promote the contractor’s ability to provide the same level of service that is being provided by the VCL primary center. We found that when callers do not press “1” during the initial Lifeline greeting, their calls may take longer to answer than if the caller had pressed “1” and been routed to either the VCL primary center or a VCL backup call center. As previously discussed, VA and SAMHSA collaborated to link the toll-free numbers for both Lifeline and the VCL through an interagency agreement. The greeting instructs callers to press “1” to be connected to the VCL; if callers do not press “1,” they will be routed to one of SAMHSA’s 164 Lifeline local crisis centers. To mimic the experience of callers who do not press “1” to reach the VCL when prompted, we made 34 covert nongeneralizable test calls to the national toll-free number that connects callers to both Lifeline and the VCL during August 2015 and we did not press “1” to be directed to the VCL. For 23 of these 34 calls, our call was answered in 30 seconds or less. For 11 of these calls, we waited more than 30 seconds for a responder to answer— including 3 calls with wait times of 8, 9, and 18 minutes. Additionally, one of our test calls did not go through, and during another test call we were asked if we were safe and able to hold. VA’s policy prohibits VCL responders from placing callers on hold prior to completing a suicide assessment; Lifeline has its own policies and procedures. According to officials and representatives from VA, SAMHSA, and the VCL backup call centers, as well as our experience making test calls where we did not press “1,” there are several reasons why a veteran may not press “1” to be routed to the VCL, including an intentional desire to not connect with VA, failure to recognize the prompt to press “1” to be directed to the VCL, waiting too long to respond to the prompt to press “1” to be directed to the VCL, or calling from a rotary telephone that does not allow the caller to press “1” when prompted. VA officials said they had not estimated the extent to which veterans intending to reach the VCL did not press “1” during the Lifeline greeting. These officials explained that their focus has been on ensuring that veterans who did reach the VCL received appropriate service from the VCL primary center and backup call centers. In addition, SAMHSA officials said that they also do not collect this information. These officials reported that SAMHSA does not require the collection of demographic information, including veteran status, for a local crisis center to participate in the Lifeline network. However, they noted that SAMHSA could request through its grantee that administers the Lifeline network that local crisis centers conduct a onetime collection of information to help determine how often and why veterans reach Lifeline local crisis centers. SAMHSA officials explained that they could work with the Lifeline grantee to explore optimal ways of collecting this information that would be (1) clinically appropriate, (2) a minimal burden to callers and Lifeline’s local crisis centers, and (3) in compliance with the Office of Management and Budget’s paperwork reduction and information collection policies. The interagency agreement between VA and SAMHSA assigns SAMHSA responsibilities for monitoring the use of the national toll-free number— 1-800-273-TALK (8255)—that is used to direct callers to both the VCL and Lifeline. These responsibilities include monitoring the use of the line, analyzing trends, and providing recommendations about projected needs and technical modifications needed to meet these projected needs. Using the information collected from the Lifeline local crisis centers on how often and why veterans reach Lifeline, as opposed to the VCL, VA and SAMHSA officials could then assess whether the extent to which this occurs merits further review and action. Although the results of our test are not generalizable, substantial wait times for a few of our covert calls suggest that some callers may experience longer wait times to speak with a responder in the Lifeline network than they would in the VCL’s network. Without collecting information to examine how often and why veterans do not press “1” when prompted to reach the VCL, VA and SAMHSA cannot determine the extent veterans reach the Lifeline network when intending to reach the VCL and may experience longer wait times as a result. In addition, limitations in information on how often and why this occurs do not allow VA and SAMHSA to determine whether or not they should collaborate on plans to address the underlying causes of veterans not reaching the VCL. Standards for internal control in the federal government state that information should be communicated both internally and externally to enable the agency to carry out its responsibilities. For external communications, management should ensure there are adequate means of communicating with, and obtaining information from, external stakeholders that may have a significant impact on the agency achieving its goals. In June 2014, VA assessed the operational state of the VCL and, based on its findings, designed a performance-improvement plan that outlined actions to address problems VA identified regarding the VCL’s workforce, processes, technology, and infrastructure. To implement this plan, in March 2015 VA began a series of rapid process-improvement events, such as improvements to VCL primary center responder training, designed to solve problems identified by VCL staff and stakeholders with actions that could be implemented within 60 to 90 days. According to VA officials and documentation provided by these officials, these rapid process-improvement events led to several changes at the VCL primary center in 2015 and 2016. As we previously noted, these changes include implementation of staggered shifts; development of silent monitoring procedures, and the hiring of dedicated staff to complete this monitoring; and new call-handling procedures previously discussed. They also include some remaining follow-up activities, such as completing the implementation of remaining planned quality-assurance activities in fiscal year 2016. These measures—if fully implemented—represent positive steps to improve VCL operations. VA has developed additional plans to address other concerns with VCL operations. These plans address issues at the VCL primary center related to renovation of new space, upgrades to telecommunications, and the introduction of a caller queue. Renovation of new space for VCL primary center operations. We found that the VCL primary center responders are housed in two different buildings originally designed for patient care delivery. According to VA officials, these buildings do not reflect call center leading practices that recommend large, open rooms that provide supervisors greater access to the responders they oversee. However, in February 2016, VA officials reported that the department committed funding to relocate the VCL primary center operations to a renovated space on the VA medical center campus. The relocation is to be implemented in two phases. VA officials expect that the first phase, which includes moving administrative and monitoring staff, will be completed in June 2016; the second phase will relocate the rest of the VCL staff, including all responders. VA officials said they anticipate that the second phase will be completed in fiscal year 2017. VA officials told us that they plan on using the National Call Center- Health Resource Center’s large open-space layout as a model in designing the VCL primary center’s new space. According to VA officials, the National Call Center-Health Resource Center follows leading practices for call center operations as set by the International Call Management Institute. Upgrade of VCL primary center telecommunication infrastructure. VA officials told us that the VCL primary center uses the telephone infrastructure of the VA medical center rather than a separate telephone system that would be more conducive to operating a call center. According to a telephone infrastructure change justification that VA information-technology officials prepared, the VCL primary center’s existing telephone system does not meet the requirements for operating a call center of its size. This documentation indicates that improvements are needed in several features of the VCL’s existing telephone system— including call routing, call recording, data capture, and automatic callback. In February 2016, VA officials reported that planned improvements to the VCL primary center’s telephone system would be implemented by June 2016; however, the VCL primary center will continue to operate using part of the VA medical center’s telephone system. Introduction of VCL primary center caller queue. VA’s evaluation of the VCL conducted in 2014 noted that a possible option for improving VCL call response would be to implement a queue at the VCL primary center that would allow callers to wait a longer period for a VCL primary center responder before being sent to a VCL backup call center. Currently, VA allows VCL primary center responders 30 seconds to answer calls before routing them to VCL backup call centers for a response. In February 2016, VA officials told us that they are considering implementing this type of queue. According to these officials, they are considering allowing VCL calls to remain at the VCL primary center for up to 5 minutes and they explained that this 5-minute period was determined based on feedback they received from veterans on how long they would be willing to wait for a responder. These officials further explained that voice prompts would offer callers options as they waited in the queue to reach the next available VCL primary center responder or to be transferred to other VA call centers for concerns unrelated to crisis situations. The VCL plays an important role in providing a means by which veterans and those concerned about them can discuss unique challenges and crises they face, and provides a way to access VA’s mental health care services. However, the rapid growth of the VCL in recent years has coincided with operational and planning challenges that constrain its ability to serve veterans in crisis in a timely and effective manner. To its credit, VA has taken some interim but noteworthy steps to address these challenges. Building on these steps, VA and SAMHSA need to take additional actions to provide reasonable assurance that the VCL’s mission to serve veterans and others in crisis situations is met. As our testing demonstrates, VA has not yet achieved its goal of answering 90 percent of all VCL calls within 30 seconds at the VCL primary center, but its planned and recently implemented changes, such as staggered shifts and enhanced call-handling procedures, are intended to gain VA system efficiencies that will help the department meet its goal once additional responders are hired. However, VA has not applied the same level of attention to its text messaging service and does not regularly test the VCL’s text messaging system. Without doing so, VA cannot ensure that veterans are receiving timely responses from VCL responders to their text messages. In addition, while VA has taken a number of steps to improve its monitoring of the VCL, VA continues to experience challenges related to weaknesses in VCL key performance indicators—including a lack of measurable targets and time frames. If left unresolved, these weaknesses will likely have negative effects on VA’s ability to ensure the VCL is providing the best service possible to veterans. Despite efforts to coordinate the operations of the VCL and Lifeline through an interagency agreement, VA and SAMHSA have not collected information necessary to determine how often and why veterans intending to reach the VCL reach Lifeline instead. As a result, neither VA nor SAMHSA can assess the extent this occurs and the underlying causes that may need to be addressed. To improve the timeliness and quality of VCL responses to veterans and others, we recommend that the Secretary of Veterans Affairs direct the Under Secretary for Health to take the following two actions: regularly test the VCL’s text messaging system to identify issues and correct them promptly; and document clearly stated and measurable targets and time frames for key performance indicators needed to assess VCL performance. We further recommend that under the applicable terms of their interagency agreement, the Secretary of Veterans Affairs and the Secretary of Health and Human Services direct the Under Secretary for Health and the Administrator of the Substance Abuse and Mental Health Services Administration (SAMHSA), respectively, to collaborate and take the following two actions: collect information on how often and why callers intending to reach the VCL instead reach Lifeline local crisis centers; and review the information collected and, if necessary, develop plans to address the underlying causes. We provided a draft of this report to VA and HHS for review and comment. In their written comments, summarized below and reprinted in appendixes II and III, both agencies concurred with our recommendations. VA and HHS described ongoing or planned actions and provided a timeline for addressing our recommendations. HHS also provided technical comments, which we incorporated as appropriate. In response to our first recommendation, to regularly test the VCL’s text messaging system to identify issues and correct them promptly, VA said that the VCL’s analytics department will develop and implement a more robust and proactive system to test daily the VCL’s text messaging service by July 2016. In the interim, VA stated that it has a process for identifying, addressing, and troubleshooting problems that utilizes e-mail templates to notify its contract text service provider of issues or errors that require a response to troubleshoot the error. In response to our second recommendation, to document clearly stated and measurable targets and time frames for key performance indicators needed to assess VCL performance, VA said that it is in the process of developing a monthly scorecard with elements assessing call center, staffing, quality-assurance, and crisis-response metrics with specific performance targets. VA estimates that by October 2016 it would establish targets and time frames for its performance indicators. In response to our third recommendation, to collaborate with SAMHSA to collect information on how often and why callers intending to reach the VCL instead reach Lifeline local crisis centers, VA said that the VCL’s newly formed Clinical Advisory Board would foster collaboration amongst capable experts and leverage their collective expertise in facilitating an improved experience for callers, greater operational efficiencies, and increased access to the VCL for veterans in crisis. VA noted that the Clinical Advisory Board included members of SAMHSA, the VA Suicide Prevention Office, and other VA clinical offices. VA estimates that it would collect sufficient data, conduct a collaborative analysis with SAMHSA, and complete reporting to both agencies on this issue by October 2016. HHS said that in response to this recommendation it would review ways to collect data on callers intending to reach the VCL but instead reaching Lifeline local crisis centers. In response to our fourth recommendation, to collaborate with SAMHSA to review the information collected and, if necessary, develop plans to address the underlying causes for callers intending to reach the VCL instead reaching Lifeline local crisis centers, VA said that the Clinical Advisory Board referenced above would evaluate this issue as a standing agenda item in its monthly meetings. VA said that the Clinical Advisory Board would establish a baseline regarding the frequency of this issue’s occurrence, monitor reported complaints about the press “1” functionality, and provide us with data from Clinical Advisory Board meetings to demonstrate action taken toward implementing our recommendation. VA expects to complete these actions by January 2017. HHS said that in response to this recommendation it would review the data collected as described above and, if necessary, address the underlying causes as appropriate. These VA and HHS actions, if implemented effectively, would address the intent of our recommendations. In its technical comments, HHS emphasized the distinction between the Lifeline network and the VCL, noting that the two programs operate with different policies, procedures, and resources. We revised the draft to more clearly reflect this distinction. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time we will send copies of this report to the appropriate congressional committees, the Secretary of Veterans Affairs, the Secretary of Health and Human Services, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions about this report, please contact Seto J. Bagdoyan at (202) 512-6722 or bagdoyans@gao.gov, or Randall B. Williamson at (202) 512-7114 or williamsonr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV. To determine the extent to which the Department of Veterans Affairs (VA) meets response-time goals for calls, online chats, and text messages received through the Veterans Crisis Line (VCL), we conducted several tests of VCL services during July and August 2015. These tests were designed to measure the timeliness of the VCL’s response to calls, online chats, and text messages. We conducted a covert test of the VCL’s call response time using a generalizable sample of 119 test calls placed in July and August 2015. To develop this generalizable sample, we interviewed VA officials with knowledge about VCL primary and backup call center operations; obtained the VCL primary center’s historical call volume data in hourly increments for fiscal year 2013 through the end of the second quarter of fiscal year 2015; and generated a schedule of days and times during which our test calls would be made. This test call schedule was created by dividing the 62-day sample period into 496 primary sampling units, which we defined as 3-hour blocks of time. We then defined secondary sample units as 10-minute increments within each 3-hour block of time and selected a stratified two-stage random cluster sample of 144 10- minute increments during which our test calls would be made. We selected the 144 10-minute increments by: (1) stratifying the primary sampling units into four strata—overnight, morning, afternoon, and evening—based on time of day; (2) identifying a stratified sample of 36 primary sampling units that were allocated across the four strata based on call volume and our available resources; and (3) randomly selecting four 10-minute increments from each selected primary sampling unit. The results of this test can be used to estimate all VCL callers’ wait times for July and August 2015. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we express our confidence in the precision of our particular sample’s results as a 95 percent confidence interval (i.e., a margin of error of within plus or minus a certain number of percentage points). This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. Percentage estimates from our analysis included in this report have a margin of error of within plus or minus 9 percentage points at the 95 percent confidence level. Estimates of the median wait time have a margin of error of within plus or minus 10 percent at the 95 percent confidence level. When placing test calls, we used 20 telephone numbers with randomly selected area codes to mask the origin of the calls. Two analysts then independently measured and documented wait times by reviewing audio recordings of each test call. Wait times were measured from the time that the caller pressed “1” to reach the VCL and the time that a responder answered the test call. The final wait time for each test call was the lowest of the two wait times recorded by these analysts. We successfully completed and measured the wait times for 119 test calls in 30 of the 36 selected primary sampling units. We did not complete calls in 25 of our selected 10-minute increments due to technical or scheduling issues. The resulting completion rate for our test calls was 83 percent (119 out of 144). The omitted test calls were distributed across all four strata and were unrelated to the time of day. To test VA’s online chat and text message response timeliness, we reviewed VA’s procedures and training materials for operating both services. We then interviewed and observed VCL responders at the VCL primary center who responded to online chat and text messages. We also spoke with the VCL text messaging service provider to learn about the text messaging operations. To test the VCL’s online chat and text message response, we scheduled one covert test online chat or text message during each of the 30 primary sampling units used for the generalizable sample described above and recorded our wait times for a response. We measured the wait time for online chats and text messages as the elapsed time between when we sent the online chat or text message to the VCL and when we received a response from a responder. We initiated our test online chats through a link provided on the VCL’s website. We sent test text messages to the VCL through an Internet text messaging service provider in order to record our test data electronically. We removed one test text message attempt from the sample because of technical issues we experienced that may have prevented the message from reaching the VCL. As a result, our final samples consisted of 15 test online chats and 14 test text messages. We verified the reliability of VA’s reported VCL call data by interviewing officials responsible for managing them and reviewing reports that VA’s backup call coverage contractor provided to VA that documented the time, duration, and routing of every VCL call. The routing information included details on the call centers where each call was routed and identified the call center that ultimately answered each call. We were able to identify our test calls in these reports and confirmed that the data matched records we maintained for our test calls. This exercise also allowed us to confirm whether our test calls were answered at the VCL primary center or a VCL backup call center. On the basis of these actions, we found these data to be sufficiently reliable for the purposes of describing the quantity of requests for services reaching the VCL. We used these data to evaluate the timeliness of the VCL’s call response and compared the data to the department’s own goals. To assess the effectiveness of the implementation of staggered shifts for responders at the VCL primary center, we compared VCL call data from September 6, 2015, through December 31, 2015, to that of September 1, 2014, through December 31, 2014. We selected September 6, 2015, as the start date for our 2015 period of analysis because it was the first day that VA fully implemented staggered shifts at the VCL primary center. We chose the cutoff of December 31, 2015, because it corresponded to the most recent complete month of data available at the time of our analysis. We used call data from September 1, 2014, through December 31, 2014, because they reflected a comparable period from the year prior. We used these 2014 data as a comparison group to account for any seasonality patterns, variations, or fluctuations that might affect the demand for VCL services within a particular season, day of the week, or other periods. Our evaluation compared the average hourly call response percentages of the periods we examined and included analysis for the time of day using hourly intervals, day of the week, and holidays within each period. The average hourly response percentages are likely affected by several factors—such as call volume, staffing levels, and complexity of calls, for which we did not control. Our analysis examined differences by day of the week, time of day, and holidays, but did not control for the above- mentioned or other factors that may affect the percentage of calls answered at the VCL primary center. To determine whether callers attempting to reach the VCL who did not press “1” experienced longer wait times than those that did, we conducted a nongeneralizable test. The VCL is accessed by calling a single national toll-free number—1-800-273-TALK (8255)—shared by both the VCL and the National Suicide Prevention Lifeline (Lifeline). This toll-free number is managed by the Substance Abuse and Mental Health Services Administration (SAMHSA). To conduct our nongeneralizable test, we used a random sample of 34 covert test calls to conduct these tests where we mimicked the experience of VCL callers who do not follow the instructions of the voice prompt on this single national toll-free number to press “1” in order to reach the VCL. To do this, we placed two test calls where we did not press “1” as prompted to reach the VCL during each of the scheduled primary sampling units in August 2015. We recorded the wait times for each of the 34 test calls by calculating the amount of time that elapsed between the moment that an automated message informed us that the call was being transferred to a Lifeline local crisis center and when a responder answered our call. We masked the origin of these calls in a manner similar to that described for our generalizable sample of 119 test calls placed to the VCL. Although the 34 test calls were randomly made, the results of these test calls are not generalizable due to the small number of calls included in our sample. In addition to the contacts named above, Gabrielle M. Fagan (Assistant Director), Marcia A. Mann (Assistant Director), James D. Ashley, Dean Campbell, Shaunessye D. Curry, Amber D. Gray, Katherine Nicole Laubacher, Olivia Lopez, Maria McMullen, Brynn P. Rovito, Amber H. Sinclair, and Shana B. Wallace made key contributions to this report. Members of our investigative staff also contributed to the report.

Summary: VA established the VCL in July 2007 to provide support to veterans in emotional crisis. Between fiscal years 2008, its first full year of operation, and 2015, the number of calls received by the VCL increased almost 700 percent, exceeding VA's expectations. As VA began to address increasing numbers of requests for assistance, reports of dissatisfaction with VCL's service periodically appeared in the media. GAO was asked to review VA's administration of the VCL. This report, among other issues, examines (1) the extent to which VA meets response-time goals for VCL calls and text messages, (2) how VA monitors VCL primary center call center operations, and (3) how VA works with VCL service partners to help ensure veterans receive high-quality service. GAO visited the VCL's primary center and two backup call centers; tested VCL response time through a generalizable sample of covert telephone calls and a nongeneralizable sample of text messages in July and August 2015; reviewed internal reports and policies and plans; and interviewed VA and SAMHSA officials. GAO found that the Department of Veterans Affairs (VA) did not meet its call response time goals for the Veterans Crisis Line (VCL), although extended call wait times were not common. VA's goal is to answer 90 percent of VCL calls at the VCL primary center within 30 seconds. Currently, calls not answered within 30 seconds route to VCL backup call centers; however, for 5 months of fiscal year 2015, calls were routed to VCL backup call centers after 60 seconds. VA officials told GAO that VA data show about 65 to 75 percent of VCL calls were answered at the VCL primary center in fiscal year 2015 within either 30 or 60 seconds. GAO's covert testing in July and August 2015 confirms VA's data. Specifically, 119 covert test calls show that an estimated 73 percent of calls made during this period were answered within 30 seconds. GAO also estimates that 99 percent of all VCL calls during this period were answered within 120 seconds. GAO also covertly tested the VCL's text messaging services and found that 4 of 14 GAO test text messages did not receive responses. VA officials said they do not monitor or test the timeliness and performance of the VCL text message system and instead rely solely on the VCL's text messaging provider for these functions. VA officials told GAO that the provider had not reported any issues with the system, but the provider told GAO that routine testing of the VCL system is not conducted. Without routinely testing its text messaging system or ensuring that its provider does so, VA cannot identify limitations to this service. While VA has taken a number of steps to improve its monitoring of the VCL primary center operations, VA has not developed measurable targets and time frames for its key performance indicators, such as the program's percentage of abandoned calls. VA established a permanent VCL call center evaluation team and created a mechanism for tracking complaints about the performance of the VCL primary center from VCL callers or external parties. However, GAO found that VA has not specified quantifiable or otherwise measurable targets and has not included dates for when it would expect the VCL to complete actions covered by each key performance indicator. This is inconsistent with guidance provided by the Office of Management and Budget. As a result, VA cannot ensure that the VCL is providing consistent, high-quality services to callers and cannot effectively track and publicly report progress or results. VA established an interagency agreement with its service partner, the Department of Health and Human Services' (HHS) Substance Abuse and Mental Health Services Administration (SAMHSA), to manage the shared operations of the VCL and the National Suicide Prevention Lifeline (Lifeline), which include a single national toll-free number used by both. Despite these efforts to coordinate, VA and SAMHSA do not collect information needed to assess how often and why callers intending to reach the VCL do not follow voice prompts and instead reach Lifeline local crisis centers. VA officials told GAO that the type of information that would be needed to do so is not collected because VA has focused on addressing the concerns of those callers who did reach the VCL. In addition, SAMHSA officials said that they do not require Lifeline local crisis centers to collect this type of information, noting that it would be possible to collect it. As a result, VA and SAMHSA do not know the extent to which this occurs and cannot determine the underlying causes that may need to be addressed. GAO recommends that VA regularly test VCL's text messaging system and document targets and time frames for key performance indicators. GAO also recommends that VA and SAMHSA collect information on how often and why callers reach Lifeline when intending to reach the VCL, review this information, and, if necessary, develop plans to address the causes. VA and HHS concurred with GAO's recommendations and described planned actions to address them.