VA has undertaken a number of initiatives to help prevent veteran suicide, including identifying suicide prevention as VA’s highest clinical priority in its strategic plan for fiscal years 2018 through 2024 (see fig. 2). VA uses CDC’s research on risk factors and prevention techniques to inform its approach to suicide prevention in the veteran community. There is no single determining cause for suicide; instead, suicide occurs in response to biological, psychological, interpersonal, environmental, and societal influences, according to the CDC. Specifically, suicide is associated with risk factors that exist at the individual level (such as a history of mental illness or substance abuse, or stressful life events, such as divorce or the death of a loved one), community level (such as barriers to health care), or societal level (such as the way suicide is portrayed in the media and stigma associated with seeking help for mental illness). According to VA, veterans may possess risk factors related to their military service, such as a service-related injury or a difficult transition to civilian life. CDC reports that protective factors—influences that help protect against the risk for suicide—include effective coping and problem- solving skills, strong and supportive relationships with friends and family, availability of health care, and connectedness to social institutions such as school and community. VA’s 2018 National Strategy for Suicide Prevention identifies four focus areas: (1) healthy and empowered veterans, families, and communities; (2) clinical and community preventative services; (3) treatment and support services; and (4) surveillance, research, and evaluation. Collectively, these four areas encompass 14 goals for preventing veteran suicide, one of which is implementing communication designed to prevent veteran suicide by changing knowledge, attitude, and behaviors. VHA’s suicide prevention media outreach campaign is just one of its initiatives intended to reduce veteran suicide. For example, in 2007, VHA established the Veteran’s Crisis Line (VCL), a national toll-free hotline that supports veterans in emotional crisis. Veterans, as well as their family and friends, can access the VCL by calling a national toll-free number—1-800-273-8255—and pressing “1” to be connected with a VCL responder, regardless of whether these veterans receive health care through VHA. VHA added the option to communicate with VCL responders via online chat in 2009, followed by text messaging in 2011. Another VHA suicide prevention initiative is the Recovery Engagement and Coordination for Health – Veterans Enhanced Treatment initiative, or REACH VET. Established in 2016, REACH VET uses predictive modeling to analyze existing data from veterans’ health records to identify veterans at increased risk for adverse outcomes, such as suicide, hospitalization, or illness. Suicide prevention officials within VHA’s Office of Mental Health and Suicide Prevention (OMHSP) are responsible for implementing the suicide prevention media outreach campaign. Since 2010, VHA has used a contractor to develop suicide prevention media outreach content and monitor its effectiveness. In September 2016, VHA awarded a new contract to the same contractor to provide both suicide prevention and mental health media outreach. Under the 2016 contract, the suicide prevention and mental health outreach campaigns remain separate and are overseen by separate suicide prevention and mental health officials, both within OMHSP. VHA officials told us that beginning in fiscal year 2019, VHA will separate the contract for suicide prevention and mental health media outreach. Specifically, VHA will utilize an existing agreement with a different contractor for suicide prevention media outreach while the existing contractor will continue to provide mental health media outreach. According to VHA, the purpose of its suicide prevention media outreach campaign is to raise awareness among veterans, their families and friends, and the general public about VHA resources that are available to veterans who may be at risk for suicide. The primary focus of the outreach campaign since 2010 has been to raise awareness of the services available through the VCL. VHA’s suicide prevention media outreach falls into two main categories: unpaid and paid. Unpaid media outreach content is typically displayed on platforms owned by VA or VHA, or is disseminated by external organizations or individuals that share VHA suicide prevention content at no cost, as discussed below (see fig. 3). Social media. VA and VHA each maintain national social media accounts on platforms such as Facebook, Twitter, and Instagram, and post content, including suicide prevention content developed by VHA’s contractor. VHA also works with other federal agencies, non-governmental organizations, and individuals that post its suicide prevention content periodically. Public service announcements (PSA). VHA’s contractor typically develops two PSAs per year, which various local and national media networks display at no cost to VHA. Website. VHA’s contractor maintains the content displayed on the VCL website (veteranscrisisline.net), including much of the content it develops for other platforms, such as PSAs and social media content. Visitors to the website can both view the content on the website and share it on their own platforms. Paid digital media. An example of paid digital media includes online keyword searches, in which VHA pays a search engine a fee for its website to appear as a top result in response to selected keywords, such as “veterans crisis line” or “veteran suicide.” Paid digital media also includes social media posts for which VHA pays a fee to display its content to a widespread audience, such as users with a military affiliation. Paid “out-of-home” media: “Out-of-home” refers to the locations where this type of content is typically displayed. Examples include billboards, bus and transit advertisements, and local and national radio commercials. VHA recognizes September as Suicide Prevention Month each year. During this month, VHA establishes a theme and increases its outreach activities, including a combination of both paid and unpaid media outreach. According to VHA, it typically incorporates additional outreach techniques during this month, such as enlisting the support of celebrities or hosting live chat sessions on social media platforms, including Facebook and Twitter. VHA’s suicide prevention media outreach activities declined in fiscal years 2017 and 2018 compared to earlier years of the campaign. We identified declines in social media postings, PSAs, paid media, and suicide prevention month activities, as discussed below. Social media. The amount of social media content developed by VHA’s contractor decreased in 2017 and 2018, after increasing in each of the prior four years. Specifically, VHA reported that its contractor developed 339 pieces of social media content in fiscal year 2016, compared with 159 in fiscal year 2017, and 47 during the first 10 months of fiscal year 2018 (see fig. 5.). PSAs. VHA’s contractor is required to develop two suicide prevention PSAs in each fiscal year. VHA officials said that the development of the two PSAs was delayed in fiscal year 2018. Specifically, as of August 2018, VHA reported that one PSA was completed, but had not yet aired, and another PSA was in development. As a result of this delay, VHA had not aired a suicide prevention PSA on television or radio in over a year; this is the first time there has been a gap of more than a month since June 2012. Paid media. VHA had a total budget of $17.7 million for its suicide prevention and mental health media outreach for fiscal year 2018, of which $6.2 million was obligated for suicide prevention paid media. As of September 2018, VHA said it had spent $57,000 of its $6.2 million paid media budget. VHA officials estimated that they would spend a total of $1.5 million on suicide prevention paid media for fiscal year 2018 and indicated that the remaining funds would be de-obligated from the contract at the end of the fiscal year and not used for suicide prevention media outreach. VHA officials indicated that the reason they did not spend the remaining funds on suicide prevention paid media in fiscal year 2018 was that the approval of the paid media plan was delayed due to changes in leadership and organizational realignment of the suicide prevention program. As a result, VHA officials said they limited the paid media outreach in fiscal year 2018 to activities that were already in place, including 25 keyword search advertisements, and 20 billboards and 8 radio advertisements in selected cities across the United States. In prior fiscal years, VHA conducted a variety of digital and out-of- home suicide prevention paid media. For example, in fiscal year 2015, with a suicide prevention paid media budget of more than $4 million, VHA reported that it ran 58 advertisements on Google, Bing, and Facebook, and ran 30 billboards, 180 bus advertisements, more than 19,000 radio advertisements, 252 print advertisements, and 39 movie theatre placements in selected cities across the United States. VHA ran similar types of paid media in fiscal years 2013, 2014, and 2016 with variation in quantities based on the approved budget in each of these years. In fiscal year 2017, VHA had a budget of approximately $1.7 million to spend on paid media for both the suicide prevention and mental health outreach campaigns. However, VHA spent less than 10 percent of the funds (approximately $136,000) to run paid advertisements on Google and Bing for suicide prevention in fiscal year 2017; the remainder was spent on mental health outreach. Suicide Prevention Month. VHA documentation indicated that Suicide Prevention Month 2017 was a limited effort. VHA officials said that this was because they did not begin preparing early enough. In May 2018, VHA officials indicated that they were similarly behind schedule for planning Suicide Prevention Month 2018, though they told us in August 2018 that they had caught up. VHA officials told us that the decrease in suicide prevention media outreach activities was due to leadership turnover and reorganization since 2017. For example, VHA officials said the National Director for Suicide Prevention position was vacant from July 2017 through April 2018. VHA filled the role temporarily with a 6-month detail from another agency from October 2017 through March 2018 and then hired this individual as the permanent director on April 30, 2018. VHA officials that worked on the campaign told us they did not have leadership available to make decisions about the suicide prevention campaign during this time. For example, VHA officials said they did not have a kick-off meeting between VHA leadership and VHA’s contractor at the beginning of fiscal year 2018—a requirement of the contract—because there was no leadership available to participate in this meeting. The officials also reported that suicide prevention leadership was not available for weekly meetings to discuss suicide prevention outreach activities, even after the suicide prevention program obtained an acting director on detail from another agency. VHA staff said that at that time, they focused their suicide prevention media outreach efforts on areas that did not require leadership input, such as updating the VCL website. The absence of leadership available to provide direction and make decisions on the suicide prevention media outreach campaign is inconsistent with federal internal control standards for control environment, which require agencies to assign responsibilities to achieve its objectives. If a key role is vacant, management needs to determine by whom and how those responsibilities will be fulfilled in order to meet its objectives. Officials that worked on the campaign told us they shifted their focus away from the suicide prevention media outreach campaign toward the mental health outreach campaign due to reorganization of the offices responsible for suicide prevention activities in 2017. Specifically, under the new organization, and in the absence of suicide prevention program leadership, the officials began reporting directly to mental health program leadership and became more focused on the mental health outreach aspects of the contract. Following the reorganization, officials that worked on the campaign did not have a clear line of reporting to the suicide prevention program. This is also inconsistent with federal internal control standards for control environment, which require agencies to establish an organizational structure and assign responsibilities, such as establishing lines of reporting necessary information to management. VHA officials told us that one of the highest priorities for the suicide prevention program since the beginning of fiscal year 2018 was to establish a national strategy for preventing veteran suicides. The national strategy, issued in June 2018, includes suicide prevention outreach as one of the strategy’s 14 goals. The national strategy also emphasizes VHA’s plans to shift to a public health approach to suicide prevention outreach. The public health approach focuses less on raising awareness of the VCL and more on reaching veterans before the point of crisis. VHA officials told us they have been trying to shift to a public health approach since 2016. Some of the campaign themes and messages have reflected this shift; for example, the “Be There” campaign theme that was adopted in fiscal year 2016—and has remained the theme since— emphasizes the message that everyone has a role in helping veterans in crisis feel less alone and connecting them to resources. However, VHA officials told us in May 2018 that they were just beginning to conceptualize what the suicide prevention outreach campaign should look like moving forward. Leadership officials also said that while they were developing the national strategy, they delegated the responsibility for implementing the suicide prevention outreach campaign to other officials working on the campaign. The decline in VHA’s suicide prevention media outreach activities over the past 2 fiscal years is inconsistent with VA’s strategic goals, which identify suicide prevention as the agency’s top clinical priority for fiscal years 2018 through 2024. Further, VHA has continued to obligate millions of dollars to its suicide prevention media outreach campaign each year. Since fiscal year 2017, VHA has obligated $24.6 million to the contract for media outreach related to both suicide prevention and mental health. By not assigning key leadership responsibilities and clear lines of reporting, VHA’s ability to oversee the suicide prevention media outreach activities was hindered and these outreach activities decreased. As a result, VHA may not have exposed as many people in the community, such as veterans at risk for suicide, or their families and friends, to its suicide prevention outreach content. Additionally, without establishing responsibility and clear lines of reporting, VHA lacks assurance that it will have continuous oversight of its suicide prevention media outreach activities in the event of additional turnover and reorganization in the future, particularly as VHA begins implementing the suicide prevention media outreach campaign under its new agreement that begins in fiscal year 2019. VHA works with its contractor to create and monitor metrics to help gauge the effectiveness of its suicide prevention media outreach campaign in raising awareness among veterans and others about VHA services, such as the VCL. The metrics primarily focus on the number of individuals who were exposed to or interacted with VHA’s suicide prevention content across various forms of outreach, including social media, PSAs, and websites. According to VHA, the metrics are intended to help VHA ensure that its media outreach activities achieve intended results, such as increasing awareness and use of the resources identified on the VCL website. Examples of metrics monitored by VHA and its contractor include those related to (1) social media, such as the number of times a piece of outreach content is displayed on social media; (2) PSAs, such as the total number of markets and television stations airing a PSA; and (3) the VCL website, such as the total traffic to the website, as well as the average amount of time spent on a page and average number of pages viewed per visit. VHA’s contractor is required to monitor the metrics and report results on a monthly basis. Specifically, the contractor provides monthly monitoring reports to VHA that summarize how outreach is performing, such as the number of visits to the VCL website that were driven from paid media sources. Officials noted these reports are key sources of information for VHA on the results of its outreach. VHA officials also told us they informally discuss certain metrics during weekly meetings with VHA’s contractor. In addition, VHA works with its contractor to conduct a more in-depth analysis of outreach efforts during and after Suicide Prevention Month each year. VHA has not established targets for the majority of the metrics it uses to help gauge the effectiveness of its suicide prevention media outreach campaign. As a result, VHA does not have the information it needs to fully evaluate the campaign’s effectiveness in raising awareness of VHA’s suicide prevention resources among veterans, including the VCL. For example, we found that VHA’s contractor’s monitoring reports—a summary of key metrics that VHA uses to routinely monitor information regarding the campaign—generally focused on outreach “highlights” and positive results. The reports did not set expectations based on past outreach or targets for new outreach, and lacked more comprehensive information on whether outreach performed against these expectations. For example: A monitoring report from 2018 showed that during one month, there were 21,000 social media mentions of keywords specific to VA suicide prevention, such as “VCL” or “veteran suicide,” across social media platforms. These mentions earned 120 million impressions; however, there was no indication of the number of keyword mentions or impressions that VHA expected based on its media outreach activities. In addition, the report did not indicate the proportion of mentions that VHA believed were specifically driven by its outreach activities, and there also was no indication of whether these mentions were positive or negative, or what actions to take based on this information. Another monitoring report from January 2017 showed that paid advertising drove 39 percent of overall website traffic during one month, while unpaid sources drove the remaining 61 percent. However, there was no information indicating the amounts of paid advertising that VHA conducted during this monitoring period, and whether this amount of website traffic from paid advertising met expectations. VHA’s 2016 Suicide Prevention Month summary report showed that there were 194,536 visits to the VCL website, roughly an 8 percent increase from the Suicide Prevention Month in 2015. However, the report did not indicate whether this increase from the prior year met expectations, or a different result was expected. VHA officials told us that they have not established targets for most of the suicide prevention media outreach campaign because they lack meaningful targets for the metrics to help evaluate the campaign. VHA officials said that the only target they have established is for each PSA to rank in the top 10 percent of the Nielsen ratings because this is the only meaningful target available that is accepted industry-wide. VHA officials stated that using any other targets would be arbitrary. For the remaining metrics, VHA officials told us they assess the outcomes of their campaign by comparing data from year to year, and examining any changes in the outcomes over time. However, VHA could set targets that capture the number of people who viewed or interacted with its outreach content, similar to its Nielsen target set for television viewership. Doing so would help VHA evaluate whether the campaign has been effective in raising awareness of VHA’s suicide prevention resources. Further, creating targets for these additional metrics need not be arbitrary, because VHA could use information about how its metrics performed in the past to develop reasonable and meaningful targets for future performance. VHA could also adjust the targets over time to reflect changes in its metrics or approach to the campaign, such as changes to its paid media budget each year. Federal internal control standards for monitoring require agencies to assess the quality of its performance by evaluating the results of activities. Agencies can then use these evaluations to determine the effectiveness of its programs or need for any corrective actions. Further, VA’s June 2018 National Strategy for Preventing Veteran Suicide also emphasizes the importance of the agency evaluating the effectiveness of its outreach. The absence of established targets leaves VHA without a framework to effectively evaluate its campaign. Our prior work has shown that establishing targets allows agencies to track their progress toward specific goals. In particular, we have developed several key attributes of performance goals and measures including, when appropriate, the development of quantifiable, numerical targets for performance goals and measures. Such targets can facilitate future evaluations of whether overall goals and objectives were achieved by allowing for comparisons between projected performance and actual results. Further, establishing targets for its outreach metrics will enable VHA officials to determine whether outreach performed as expected and raised awareness of VHA resources such as the VCL, including identifying outreach efforts that worked particularly well and those that did not. In doing so, VHA officials will have the opportunity to make better informed decisions in their suicide prevention media outreach campaign to support VA’s overall goal of reducing veteran suicides. VA has stated that preventing veteran suicide is its top clinical priority; yet VHA’s lack of leadership attention to its suicide prevention media outreach campaign in recent years has resulted in less outreach to veterans. While VHA identifies the campaign as its primary method of raising suicide prevention awareness, it has not established an effective oversight approach to ensure outreach continuity. This became particularly evident during a recent period of turnover and reorganization in the office responsible for the suicide prevention outreach campaign. Moving forward, VHA has an opportunity to improve its oversight to ensure that its outreach content reaches veterans and others in the community to raise awareness of VHA’s suicide prevention services, particularly as VHA begins working with a new contractor beginning in fiscal year 2019. VHA is responsible for evaluating the effectiveness of its suicide prevention media outreach campaign in raising awareness about VHA services that are available to veterans who may be at risk for suicide. To do so, VHA collects and monitors data on campaign metrics to help gauge the effectiveness of its suicide prevention media outreach campaign in raising such awareness, but has not established targets for the majority of these metrics because officials reported that there are no meaningful, industry-wide targets for them. We disagree with VHA’s assertion that other targets would not be meaningful; VHA collects data on its metrics that it can use to develop reasonable and meaningful targets for future performance. In the absence of established targets, VHA cannot evaluate the effectiveness of the campaign, and make informed decisions about which activities should be continued to support VA’s overall goal of reducing veteran suicides. We are making the following two recommendations to VA: 1. The Under Secretary for Health should establish an approach for overseeing its suicide prevention media outreach efforts that includes clear delineation of roles and responsibilities for those in leadership and contract oversight roles, including during periods of staff turnover or program changes. (Recommendation 1) 2. The Under Secretary for Health should require officials within the Office of Suicide Prevention and Mental Health to establish targets for the metrics the office uses to evaluate the effectiveness of its suicide prevention media outreach campaign. (Recommendation 2) We provided a draft of this report to VA for review and comment. In its written comments, summarized below and reprinted in Appendix I, VA concurred with our recommendations. VA described ongoing and planned actions and provided a timeline for addressing our recommendations. VA also provided technical comments, which we incorporated as appropriate. In response to our first recommendation, to establish an oversight approach that includes delineation of roles and responsibilities, VA acknowledged that organizational transitions and realignments within OMHSP contributed to unclear roles and responsibilities in 2017 and 2018. VA said that OMHSP has made organizational improvements, including hiring a permanent Director for Suicide Prevention and establishing a new organizational structure. In its comments, VA requested closure of the first recommendation based on these actions. However, to fully implement this recommendation, VA will need to provide evidence that it has established an oversight approach for the suicide prevention media outreach campaign. This would include providing information about the roles and responsibilities, as well as reporting requirements, for contract and leadership officials involved in the suicide prevention media outreach campaign under the new organizational structure and the new contract. VA will also need to demonstrate that it has a plan in place to ensure continued oversight of the suicide prevention media campaign in the event of staff turnover or program changes. In response to our second recommendation, to establish targets against which to evaluate suicide prevention metrics, VA said it has plans to work with communications experts to develop metrics, targets, and an evaluation strategy to improve its evaluation of its suicide prevention program efforts, including outreach. VA expects to complete these actions by April 2019. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees and the Secretary of Veterans Affairs. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7114 or at DraperD@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix I. In addition to the contact named above, Marcia A. Mann (Assistant Director), Kaitlin McConnell (Analyst-in-Charge), Kaitlin Asaly, and Jane Eyre made key contributions to this report. Also contributing were Jennie Apter, Emily Bippus, Valerie Caracelli, Lisa Gardner, Jacquelyn Hamilton, Teague Lyons, Vikki Porter, and Eden Savino.

Summary: Veterans suffer a disproportionately higher rate of suicide than the civilian population. VA has estimated that an average of 20 veterans die by suicide per day, and in 2018, VA identified suicide prevention as its highest clinical priority. VHA's suicide prevention media outreach campaign—its collective suicide prevention outreach activities—helps raise awareness among veterans and others in the community about suicide prevention resources. VHA has contracted with an outside vendor to develop suicide prevention media outreach content. GAO was asked to examine VHA's suicide prevention media outreach campaign. This report examines the extent to which VHA (1) conducts activities for its suicide prevention media outreach campaign, and (2) evaluates the effectiveness of its campaign. GAO reviewed relevant VHA documents and data on the amount, type, and cost of suicide prevention outreach activities since fiscal year 2013. GAO also reviewed VHA's contract for developing suicide prevention outreach content and interviewed VA and VHA officials. The Department of Veterans Affairs' (VA) Veterans Health Administration (VHA) conducts national suicide prevention media outreach on various platforms to raise awareness about VHA's suicide prevention resources. The primary focus of this campaign since 2010 has been to raise awareness of the Veterans Crisis Line (VCL), VHA's national hotline established in 2007 to provide support to veterans in emotional crisis. GAO found that VHA's suicide prevention media outreach activities declined in recent years due to leadership turnover and reorganization. For example, the amount of suicide prevention content developed by VHA's contractor for social media decreased in fiscal years 2017 and the first 10 months of 2018 after increasing in each of the 4 prior years. VHA officials reported not having leadership available for a period of time to make decisions about the suicide prevention media outreach campaign. GAO found that VHA did not assign key leadership responsibilities or establish clear lines of reporting, and as a result, its ability to oversee the outreach campaign was hindered. Consequently, VHA may not be maximizing its reach with suicide prevention media content to veterans, especially those who are at-risk. VHA evaluates the effectiveness of its suicide prevention media outreach campaign by collecting data on metrics, such as the number of people that visit the VCL website. However, VHA has not established targets for the majority of these metrics. Officials said they have not established targets because, apart from one industry-wide target they use, they lack meaningful targets for evaluating the campaign. However, VHA could use information about how its metrics performed in the past to develop reasonable and meaningful targets for future performance. Without established targets for its metrics, VHA is missing an opportunity to better evaluate the effectiveness of its suicide prevention media outreach campaign. VHA should (1) establish an approach to oversee its suicide prevention media outreach campaign that includes clear delineation of roles and responsibilities, and (2) establish targets for its metrics to improve evaluation efforts. VA concurred with GAO's recommendations and described steps it will take to implement them.