Under Title 10 of the United States Code, the Secretary of each Military Department (the Army, the Navy, and the Air Force) has the responsibility to recruit personnel, subject to the authority, direction, and control of the Secretary of Defense. As such, each Secretary has the authority to organize and delegate responsibility for advertising efforts within each military service or components within each military service and, as a result, the organizational structure of advertising programs and associated recruiting organizational structure for each of the military departments differs. While advertising is carried out by the military services, some roles and responsibilities for advertising reside with the Office of the Under Secretary of Defense for Personnel and Readiness. Within that office, the Accessions Policy office within Manpower and Reserve Affairs has responsibility for the (1) policy, planning, and program review of active and reserve personnel, procurement, and processing and (2) development, review, and analysis of policies, resource levels, and budgets for recruiting enlisted personnel and for officer commissioning programs. One function for recruiting enlisted personnel and officer commissioning programs is advertising. Further, JAMRS reports to the Director of DOD’s Defense Manpower and Data Center and is responsible for joint marketing communications and market research and studies. JAMRS is to conduct research about the perceptions, beliefs, and attitudes of American youth as they relate to joining the military, which is often referred to as the propensity to serve. JAMRS officials stated that understanding these factors helps ensure recruiting efforts are directed in the most efficient and beneficial manner. JAMRS has in the past carried out joint advertising aimed at “influencers,” or the adults that can influence or impact a potential recruit’s decisions regarding military service. JAMRS also maintains a database that is used across the military services to begin their outreach to potential recruits. The military service components conduct advertising in support of their recruitment missions. Consistent with the private sector, the components’ advertising programs follow a strategy that considers the phases of an individual’s decision-making process, sometimes referred to as the consumer journey. The decision to enlist in the military is a significant commitment and can be affected by numerous factors, such as other employment or educational opportunities available to an individual who is considering a career in the military. According to military service officials, these phases are generally characterized by the components as awareness, engagement, and lead generation, as illustrated in figure 1. The goal of military advertising is to move a potential recruit through each phase and, ultimately, to a decision to enlist. Further, each military service conducts advertising throughout each phase that according to military service officials is intended to communicate and reinforce a certain brand or image among potential recruits, leading potential recruits to determine that a particular military service is the best fit for their individual interests, beliefs, or goals. Awareness. The military service components conduct general awareness advertising to inform members of an audience about the opportunity to serve in the military and the distinct characteristics of each military service. The components typically pursue awareness through traditional advertising formats such as television commercials, print advertisements, and banners at events or signs within a community. Engagement. Advertising focused on building engagement targets individuals who are aware of the military as a career option and have begun to consider the possibility of enlisting. During this phase, the components seek to provide recruits with additional information to aid in their decision-making process. Often this phase of advertising takes place in the digital environment, as components seek to provide informative social media posts and use banner advertisements to attract individuals to visit their websites for more information. Figure 2 describes a variety of digital advertising activities conducted by the components. Lead Generation. Lead generation advertising targets individuals who have considered military service and are ready to discuss the possibility of enlistment. As such, lead generation activities seek to encourage these individuals to provide their contact information in order to schedule an opportunity to meet with a recruiter. Lead generation is often conducted in person, such as through recruiters’ presence at events like career fairs or sports games. Lead generation may also be conducted through other means—such as direct mail and online or print classified advertisements—as long as the advertisement features a “call to action” intended to prompt viewers to provide their contact information. Further, the military services often employ the use of “mobile assets,” such as large trucks and trailers fitted with equipment and activities intended to draw crowds and encourage and facilitate public interaction with a recruiter at an event in order to generate leads. Figure 3 shows some examples of various types of military service advertising used for recruiting purposes, such as mobile assets used at recruiting events to advertise a specific military service, digital advertising on social media, and print brochures. DOD has taken steps to coordinate some advertising activities among the military service components. Within the military departments, there are seven military service component advertising programs (see app. II for more details) that compete to attract recruits from a relatively small pool of individuals that are eligible for military service. Private sector advertising industry experts we spoke to emphasized the importance of maintaining a unique brand and strategy for each of the service components when there is competition for a target audience. According to DOD officials, each component works to develop a unique brand that differentiates the military services in order to compete for potential recruits. While industry experts stated that competition is inherent to advertising, these experts also stated that coordination can sometimes be beneficial to increase efficiencies and effectiveness, and that DOD could pursue greater coordination in some instances to help address any inefficiencies. Despite the competition that exists among the military components’ advertising programs, DOD has coordinated certain advertising activities. For example, DOD established the JAMRS office in 2002 to create a centralized program for joint market research and communication. JAMRS provides the military components with information that informs their advertising programs from surveys of U.S. youth attitudes toward joining the military, which change and evolve considerably over time. Officials across the military components stated that they heavily relied on the market research conducted by JAMRS and reported using the information to tailor their advertising and recruiting activities to address the interests of U.S. youth. JAMRS used to provide joint advertising campaigns directed at adults who might influence an individual’s decision to join the military, such as parents or coaches, known as influencers. According to JAMRS officials, joint advertising campaigns of any type have not been carried out in recent years due to budget constraints. Military service component officials stated that JAMRS’ joint advertising campaigns had been important for building awareness among influencers and assisting in promoting a positive image of military service to the U.S. public. Further, these officials stated that joint advertising for influencers provided by JAMRS was beneficial for all the military services and that due to the resources needed to conduct service-specific advertising, it is not feasible for their components to conduct additional advertising focused on influencers. Additionally, the Office for Accessions Policy within the Under Secretary of Defense for Personnel and Readiness has taken specific steps to coordinate among the components to attempt to increase the effectiveness of DOD’s advertising or to address shared challenges in advertising that are crosscutting in nature and can affect all of the military service components. For example, the Office coordinated a response to media and congressional interest in recent activities conducted as part of sports advertising that were questioned as being inappropriate. Further, the Under Secretary responded to this crosscutting challenge in September 2015 by issuing interim guidance for sports marketing that applied to all service components to prevent inappropriate activities from occurring as part of sports advertising. In addition, DOD officials stated that, in an effort to improve the effectiveness of DOD’s advertising, in 2008 the Office for Accessions Policy convened a cross-service working group of military service component advertising officials to develop a set of consistent performance measures that each military service component could use to assess performance. However, DOD officials stated that consensus was not reached, and thus the measures were not developed. Officials from the military service components stated that they meet about quarterly with their counterparts at meetings held by JAMRS to obtain their respective results from joint market research conducted by JAMRS, and at these meetings they will on occasion discuss crosscutting issues that affect the services’ advertising programs. Military service component advertising officials stated that these discussions have at times provided an opportunity to engage in discussions on effective advertising practices and lessons learned across service programs. Further, military service officials stated that they have long-standing working relationships with their counterparts among the services and that they do on occasion share some information during these discussions. However, the officials stated that they may not share comprehensive details of best practices or lessons learned because of the competition for recruits among the components and because they are not required to share information or coordinate. The private sector executives and professional association representatives we spoke with stated that while competition is an inherent aspect of advertising, increasing coordination should be considered in order to reduce inefficiencies and leverage resources effectively. Further, we have found in prior work that mechanisms to coordinate programs that address crosscutting issues may reduce the instances of potentially duplicative, overlapping, and fragmented efforts. While DOD has taken some steps to coordinate, the department has not established a formal process to ensure that service component advertising officials have a forum to share information and address crosscutting issues systematically, and that these discussions are consistently occurring, to help ensure an efficient use of resources in the competitive environment for recruitment. In the absence of a formal process for coordination, we found examples of possible unnecessary duplication, overlap, and fragmentation within DOD’s advertising activities. Risk of Increased Cost of Advertising Media Purchases. Industry experts and various military service officials stated that as each military service component purchases advertising space in the same media market, such as the time to air an advertisement on radio or television, competition among the services could increase the prices of these purchases. Further, they stated that working together could increase the buying power for certain activities. As such, from a department-wide perspective, DOD is at risk of making duplicative purchases in the same market with no coordination regarding the cost and effectiveness of those purchases. Officials from one military service component acknowledged that the competition for media purchases most likely results in a more expensive manner of doing business for the entire military. Multiple Contracts for Similar Functions. While each service component’s advertising agency works with the component to develop a unique brand that differentiates the military services, the components each contract for some functions that are not brand specific. For example, service officials reported that each component has advertising contracts that include services such as call centers and website chat-function support that respond to requests for information generated by advertising, and that screen potential recruits against general criteria for military service. As such, there are at least seven call center functions that the department uses to screen potential recruits. Most of these call centers are staffed by contractors from the general public and do not provide a service-specific function. Thus the department is paying several different companies to provide a similar service. Advertising industry experts pointed to functions that are not brand specific as possible opportunities for obtaining efficiencies through coordination or consolidation. However, military service officials disagreed with this assessment and stated that benefits that might result from consolidation of these types of functions are unclear and that call centers may provide brand-specific information. For example, officials from the Army National Guard stated that trained individuals, sometimes Army National Guard members, can offer state- or territory guard- specific information when responding to calls from their contracted call center. As discussed below, we also found examples where better coordination within and among the military service components could increase efficiencies or effectiveness by addressing the fragmentation of these advertising programs. Three Advertising Programs within the Air Force. The Air Force has three components that each contract with a different advertising agency to develop and implement three separate advertising programs. Maintaining three separate programs can lead to inefficiencies. For example, officials from the Air Force reserve and guard components stated that they do not have the resources to do marketing mix modeling. Marketing mix modeling is a best practice employed by agencies to determine the most efficient and effective allocation of a client’s budget toward media buys including print, television, and digital advertising. However, if the Air Force components coordinated on similar advertising functions that are not part of their unique branding, such as marketing mix modeling, they could potentially afford to jointly contract for such functions. In response to our questions about why they had not coordinated these programs, officials from the components stated that both the reserve component and Air National Guard need to focus on the geographic location of potential recruits to fill vacancies, whereas this is not a concern for the active duty components. However, they further stated that there had been discussion of consolidating some components in the past, possibly the guard and reserve components, but that this was not pursued. Officials could not provide any further rationale for requiring separate programs or for why further efforts for consolidation were not pursued. Two Army Advertising Programs. There are two Army advertising programs, one for the Army active duty and reserve components and another for the Army National Guard. Better coordination between these two programs could result in more effective use of resources for a common purpose. Army National Guard officials stated that they do not have the resources to fund some needed services that the active Army could support. For example, Army National Guard officials stated that they do not procure warehouse space for the storage of advertising materials for guard units throughout the country or mobile assets to deploy for National Guard campaigns, which the active Army could potentially support. Army National Guard officials stated regular coordination does not exist between these two programs and to address these shortcomings they have begun to engage in discussions with the active Army component to further explore coordination for these activities. A January 2016 report from the National Commission on the Future of the Army stated that separation of the Army’s recruiting programs and associated advertising is inefficient and unproductive. The report recommended the establishment of pilot programs that align the recruiting efforts of the active duty Army, Army National Guard, and Army Reserve as consolidating the administration and budgeting of recruiting, advertising, and branding for all components will yield increased effectiveness and efficiency. Army National and Local Advertising. We also found fragmentation in the Army’s active and reserve advertising programs resulting in coordination and communication challenges. Within the active and reserve Army components’ advertising program, the Army has two organizations that share some degree of responsibility for carrying out advertising activities: its Army Marketing and Research Group advertising office which directs the Army’s national level advertising and U.S. Army Recruiting Command which directs local advertising for recruiting. Senior officials from both organizations stated their organization is responsible for lead generation, and recruiters generate leads at both national and local events. While we acknowledge the importance of maintaining a unique brand for the service components, a formal process for coordination could allow the military service components to more effectively share best practices. As part of this process, a review of DOD’s existing programs could potentially identify opportunities to obtain efficiencies by reducing unnecessary duplication, overlap, and fragmentation that may exist within and among military service components’ advertising programs. As numerous department officials cited a lack of needed resources to appropriately carry out advertising in order to best execute the difficult task of recruiting, identifying and reducing unnecessary duplication, overlap, and fragmentation could potentially free up additional resources. In the absence of a formal process for coordination of the department’s several advertising programs, DOD may not be positioned to best leverage its advertising funds. With some exceptions, DOD—through its military service components and together with their contracted advertising agencies—generally follows commercial best practices that we identified for assessing the effectiveness of advertising, shown in table 1. We determined that a well- defined and widely accepted list of best practices had not been established; therefore, we asked a nongeneralizable sample of advertising experts and professional associations to identify best practices. We compiled and condensed the identified best practices into a list that we then validated with those same industry experts. However, when we compared the military service components advertising programs against the best practices that we identified, we found variations among the components in establishing measurable goals. Further, differences across the components in their processes for collecting and reviewing performance data have resulted in varying abilities to measure the effectiveness of advertising on generating leads, especially at the local level. We found that the military service components have largely established roles and responsibilities to address the need for unbiased performance information, and ensured that coordinator roles were assigned when multiple vendors were used. Industry experts we spoke with stated that advertising decisionmakers should consider whether to distribute responsibilities for various aspects of advertising—including creative development, media buying, and performance analysis—across different vendors, but when doing so, should ensure that a coordinator role is assigned to maintain a cohesive advertising strategy. While the majority of the military service components rely heavily on their contracted advertising agencies to assess the effectiveness of their advertising activities, they also adhered to these best practices, as described in the examples below. Consistent with this commercial best practice that we identified, for example, we found the following: A Marine Corps official responsible for overseeing the component’s advertising program stated that four Marine Corps project officers are designated to work closely with its advertising agency in monitoring and analyzing performance across key areas. Officials from the Army, which has the largest advertising budget of the components, stated that the Army had additionally contracted with third-party research firms to further assess the effectiveness of its advertising. All of the components independently review quantitative performance data, which they may access through their own data systems or reports provided by the advertising agencies. Military service officials stated that monitoring such data can help them to assess the performance of both their advertising activities and contracted advertising agencies. Each of the components also cited research conducted by DOD’s JAMRS program as a source of unbiased performance data. For example, JAMRS conducts a quarterly advertising tracking survey, evaluating the target audience’s recall and reactions to the components’ television advertising campaigns, which service officials stated can help the components assess the effectiveness of that specific type of advertising. As each component contracts with a lead advertising agency, the components typically adhere to this best practice as the agency fulfils the coordinator role in conjunction with the component. While all of the military service components develop evaluation frameworks that identify a target audience in accordance with the commercial best practice, the goals set by the components vary in measurability. In our prior work, we have also found that high-performing organizations have goals that are aligned with performance management. We found that almost all of the service components typically develop an annual advertising plan, including information about their target market and goals to be met by advertising that year. All of the plans we reviewed demonstrated a detailed understanding of the target market. For example, the Marine Corps’ fiscal year 2015 plan recognizes the varying levels of the propensity to join the military among its target market and identifies an opportunity to focus on “the movable middle”—the portion of prospective recruits who are not currently inclined to join the military but might be willing to consider it. Likewise, reflecting its organizational structure of 54 state and territorial units, the Air National Guard’s 2015 annual advertising plan emphasizes targeting advertising to audiences located near units where career opportunities are available. However, the commercial best practice on planning that we identified also calls for advertising plans to include measurable goals. In our review of the most recent version of each component’s annual advertising plan, we found that the components’ plans varied in the extent to which their goals can be measured. For example, the Army plan we reviewed identifies a series of marketing objectives supported by measurable, numeric goals related to the public perception of the Army. In contrast, the Marine Corps and active duty Air Force plans state goals that relate to emphasizing positive aspects of the components, such as diversity and core values, but these goals are not numeric and measurable with related performance measures. For example, the most recent annual advertising plan of the Marine Corps includes a goal to weave diversity into all advertising efforts, while the Air Force annual plan contains a goal that all advertising tells the Air Force a story in a way that highlights Air Force core values. Neither plan includes information on how these goals would be measured. We also found that the annual plans of the Air Force National Guard and Air Force reserve had a mixture of the types of goals identified, some of which could be measured. In addition, Army National Guard officials stated that while it requires each state unit to create an annual plan that includes goals, the Army National Guard headquarters does not establish annual goals at a national level. Army National Guard officials stated that there is not a national level annual advertising plan with associated goals because each state unit has a unique recruiting mission, including the types of positions that need to be filled and the demographic makeup of the target audience. The absence of measurable goals at the national level may limit Army National Guard headquarters’ ability to determine the success of any national level advertising efforts or to distribute advertising funds strategically and efficiently among the state units. Military service officials stated that the annual advertising campaigns and associated advertising plans for each component can change considerably from year to year. Ensuring that future iterations of each component’s annual advertising plan contain measurable goals could enhance their ability to demonstrate the success of their advertising programs. The components also vary in their use of sophisticated modeling to determine how to distribute available advertising funds across different types of advertising (e.g., television, print, Internet, etc.), which was a key planning best practice cited by industry experts. Specifically, industry experts stated it is a best practice to use some form of modeling, such as marketing mix modeling, to determine the optimum distribution of advertising funds. Officials from the active duty Army, Marine Corps, and Air Force active duty components stated that they use such modeling, provided by their contracted advertising agencies, to determine how to spend their advertising funds. However, the remaining components do not currently leverage marketing mix modeling, and Air Force Reserve and Army National Guard officials cited its high cost as a barrier. Industry experts acknowledged that, while marketing mix modeling is a best practice, it can present a significant expense, and other methods of modeling may be more appropriate in some situations. As such, Army National Guard officials stated that marketing mix modeling might not be best suited for modeling when mission requirements fluctuate year to year or when geographic location of vacancies are of a primary concern. We found that the military service components successfully applied industry standard measures appropriate for two of the purposes of advertising activities—awareness and engagement— but varied in their ability to assess the effectiveness of advertising related to the third activity, lead generation. For example, to measure effectiveness of awareness related advertising, such as television commercials or national print advertisements, the components typically measure the number of times an advertisement is viewed by a member of the audience, numbers known as impressions. To measure engagement, the components use a range of real-time digital analytics, including click-through rates and social media “likes,” among others, when they conduct digital advertising. According to military officials and representatives of their advertising agencies, it is an industry standard to directly link the results demonstrated by these analytics to the purchase of advertising for awareness and engagement. The components rely on their advertising agencies to negotiate these purchases with the intent of achieving a level of performance that is consistent with industry standards for these forms of advertising. In contrast, we found that the components varied in their ability to use industry standard performance measures to assess the effectiveness of advertising activities focused on lead generation. Service officials stated that such activities can include recruiter booths at events, direct mail, and certain types of print or digital advertising, for which performance is measured on the basis of how many leads are generated by the activity. Whereas military service officials stated that advertising focused on generating awareness and engagement is generally executed at the national level, lead generation activities are often carried out at the local level and can depend on recruiters’ familiarity and knowledge of their local markets. For example, service officials stated that recruiters may select high school populations to receive direct mailings or interact with potential recruits at local events such as career fairs or sports games and may also be responsible for identifying the number of leads generated by such activities. As a result, the responsibility for measuring the performance of locally executed lead generation advertising activities is carried out, in part, at the local level, and some components do not obtain or measure the performance data needed to assess the effectiveness of these activities. Insufficient data to measure the performance of local lead generating activities has diminished some of the components’ ability to understand how to attribute outcomes to specific advertising activities. According to industry experts and our prior work, determining the precise impact of advertising on outcomes, such as recruitment, is inherently challenging, in part due to the concurrent effects of external factors, such as the influence of family support and the availability of other career or educational opportunities. In addition, the length and complexity of a decision to enlist in the military necessitates the use of multiple types of advertising throughout the recruiting process, and an individual may ultimately decide to enlist months or years after first being exposed to advertising by the military. Although industry experts acknowledged that understanding how outcomes are impacted by advertising is a challenge in both the private sector in general and for the military in particular, they stated it is nonetheless important for advertising managers to develop an understanding of how outcomes can be attributed to advertising. With the exception of certain new requirements specific to sports advertising contracts, DOD does not require that the service components measure the performance of their advertising activities, and the extent to which the components can currently measure the effectiveness of their advertising activities varies. As such, measurement and monitoring performance is at the discretion of the military service components, and there is variation in the components’ processes for collecting and reviewing advertising activity performance data. According to service officials, we found that some components either do not collect local advertising performance data related to lead generation in their systems or cited issues with the reliability of such data. Specifically, in terms of collecting this type of performance data, service component officials stated the following: The Marine Corps’ leads tracking system allows the Marine Corps to link multiple exposures to advertising—such as a direct mailing or interaction with a recruiter at an event—to a prospective recruit’s lead record, enhancing the ability to analyze performance in lead generation. While the Navy requires that performance data for local advertising be entered into its data systems, currently it does not have the capability to analyze performance in the same manner and cannot attribute potential leads to multiple exposures to advertising. Air Force active duty recruiting squadrons are not required to report to their headquarters on the performance of their advertising activities, including the performance of local advertising activities in lead generation. However, Air Force officials stated they conduct and assess the performance of the majority of their advertising at the headquarters level, and a comparatively small portion of advertising funds are distributed to the squadrons for local advertising activities. Both the Air Force Reserve and Air National Guard require that recruiters submit performance data, such as attendance and leads generated for any local advertising activity and that headquarters officials review these data in their lead tracking data systems. The Army National Guard does not routinely require state units to provide headquarters with performance data related to advertising, including lead generation. The Army requires in its policy that subheadquarters units submit advertising performance data for headquarters review, but Army advertising officials cited concerns with the reliability of these data for lead generation. Regarding the Army’s concerns regarding unreliable data, Army officials stated that when leads are collected by recruiters at advertising events, in many cases those leads are coded in their data system as “recruiter generated,” rather than being attributed to the appropriate advertising activity. As a result, Army officials stated that although they believe their locally executed advertising activities are a good investment, they do not have sufficient evidence to demonstrate their effectiveness. For example, while Army Recruiting Command officials cited “register to win” giveaways for promotional items at local events as an effective lead generator, Army Marketing and Research Group officials stated they do not have data to support the effectiveness of this advertising activity and questioned the quality of the leads produced. Army officials stated that they are currently working to address this issue to ensure leads are properly coded and thus improve the reliability of this data. We did not find the proper coding of lead generation to be an issue for other service components; for example, according to service officials, recruiters from the Marine Corps and active Air Force that attend an event typically record a potential lead’s information into an electronic tablet that automatically assigns the proper event code for the lead based on the recruiter’s location and the date and time of the lead’s entry. Federal internal controls standards state that program managers need appropriate data to determine whether they are meeting stated goals and achieving an effective and efficient use of resources. Without access to the necessary performance data, such as reliable leads collection and attribution data, a military service component may be limited in its ability to measure the performance of its advertising activities against stated goals. Without processes in place to facilitate the measurement and monitoring of advertising performance across all levels—especially at the local level—the military service components may be unable to ensure advertising dollars are used efficiently to help meet stated recruiting goals could result in inefficient use of advertising dollars. DOD does not have comprehensive oversight of the military service components’ advertising activities, as it does not have a policy that defines its oversight role as well as procedures to guide the components’ respective advertising activities. Federal internal control standards for an agency’s organizational structure call for planning, directing, and controlling activities to ensure goals and objectives can be achieved. Further, these standards require that an agency’s activities be directed by policies and that management should provide appropriate oversight of activities. When DOD issues policy, such as directives and instructions, the department requires that the policy establishes roles and responsibilities and defines the procedures that are to be followed by all defense offices and organizations that are involved with the activity or program the policy directs. As there is no department-wide policy that defines DOD’s role in overseeing advertising activities or the procedures that should be followed when the components carry out their advertising activities, the department’s advertising activities are overseen at the military service level and in some cases, within a service’s individual active, reserve, or guard components. Each military service component has component- level policy and guidance that defines the overall objectives of its advertising program and sets forth the roles and responsibilities for the component’s advertising program. The high-level policy is in many cases supplemented by yearly guidance that communicates the recruiting goals and the priorities of the commanding officer or senior leader responsible for a component’s advertising. However, the high-level policies issued by each service component for its advertising program vary considerably in the level of oversight and direction provided, as the procedures the components are to follow as they carry out their respective advertising activities have not been defined by DOD policy. For example, the Navy’s guidance includes financial thresholds for headquarters’ level review of certain expenditures and the types of cost- effectiveness reviews that must be performed, while other components’ guidance does not address the review of expenditures. The Marine Corps guidance specifies some advertising activities that are prohibited, whereas other components’ guidance—including the Air Force—does not include this type of detail. The variation in the oversight and direction provided to each service component’s advertising program has allowed for an inconsistent understanding of digital advertising rules and regulations and for negatively perceived activities to occur in some instances. Digital Advertising. We found differences in the understanding of rules and regulations that apply to digital advertising activities. The government is restricted in tracking the digital behavior of those that visit government websites. However, we observed differing opinions and understanding among military service officials regarding what types of digital tracking was permissible. Officials from the Air Force active duty component’s advertising program and representatives from their contracted advertising agency stated that persistent tracking would be used in their new website development and that these plans adhere to relevant DOD guidance. Further, they stated that subsequent advertising could be sent to individuals after visiting the Air Force website. Officials from the Office of the Under Secretary of Defense for Personnel and Readiness stated that they had reviewed the plans of the Air Force regarding digital tracking and subsequent advertising and agreed that these actions would be permissible and in accordance with DOD guidance. However, officials from other military services we spoke with described these same digital advertising strategies as not allowed per DOD guidance and stated that they could not pursue digital tracking similar to that of the private sector. The other services could be at a disadvantage as a result of these different interpretations of regulations. Sports-Related Advertising. While contracting with sports teams and events for advertising is widely practiced among the military service components, media reports from 2015 and congressional attention revealed that some components’ contracts with professional sports teams included provisions to conduct ceremonies that honored servicemembers and provide items such as tickets to games, which were perceived to be inappropriate and came at a cost to the federal government. In our review of contracts from fiscal years 2013 and 2014, we found several contracts with a marketing firm that connects brands to collegiate sports teams that included costs for honorary or swearing in ceremonies and items that are personal in nature. For example, we identified a contract spanning fiscal years 2013 and 2014 in which a major public university received $20,000 from a state Army National Guard unit for two swearing in ceremonies of recruits conducted at sporting events, and another state Army National Guard unit contracted in fiscal year 2014 for a “VIP experience” at university sporting events costing approximately $8,700. Subsequent to the negative media reports in 2015 and resulting DOD reviews in September 2015, the acting Under Secretary for Personnel and Readiness issued interim guidance that provided more focused direction and strengthened oversight of sports marketing and advertising contracts. The interim guidance acknowledged the inappropriateness of paying for recognition or swearing in ceremonies during sports events or the inclusion of items that are personal in nature, typically sports tickets or parking, for which the receipt of those items is not clear or controlled. Specifically, this guidance (1) requires that a senior military component reviewing official approve sports marketing contracts, (2) prohibits paying for recognition ceremonies, (3) restricts items that are personal in nature, and (4) requires reporting and analysis on the returns generated by larger sports partnerships. While the issuance of the interim DOD-level sports marketing guidance is a positive step, a senior DOD official acknowledged that the interim guidance was vague regarding how some of these new requirements are to be implemented, did not include some monetary thresholds that it could have identified, and applied only to sports marketing and advertising contracts. As such, in our discussions with service officials responsible for carrying out local level advertising, we observed discrepancies in how the guidance was being interpreted. For example, there were differences in opinions among local level advertising officials about the inclusion of honorary or swearing in ceremonies in contracts with sports teams. One local level official stated that the recent interim guidance now restricts any mention of an honorary ceremony in a contract, while other officials stated that such activities could still be included in a contract as long as they were listed with a cost of $0.00 and considered “added value.” Further, as the interim guidance only applies to sports advertising, it does not address the procedures that the components should follow when carrying out other types of advertising. For example, the components may contract to carry out advertising at music concerts or festivals. These events are similar in nature to sporting events as they may also allow for the inclusion of tickets other premium items in advertising contracts, but currently there is no DOD-wide advertising guidance that prevents the inclusion of such items in contracts. DOD officials stated that they are currently in the beginning stages of developing guidance for advertising that is more comprehensive to replace and build upon the interim guidance that was focused solely on sports-related advertising, but the details of what this proposed expanded guidance will cover are not clear. These officials stated that the department’s process for development, coordination, approval, and publication of new DOD guidance can be a lengthy one. Further, they stated that they planned to review the implementation of the interim guidance and incorporate any lessons learned from the issuance of the September 2015 guidance, as well as incorporate the findings of our report before moving forward to create a DOD-wide policy. Because DOD officials were not able to provide us a draft of this proposed expansion of guidance, and they stated that they are in the earliest stage of a lengthy process, it is unclear when DOD will issue department-wide guidance for advertising. Further, it is unclear whether the guidance will clearly define DOD’s role in oversight, clarify all remaining issues related to sports advertising, and provide direction to other types of advertising such as digital advertising and concerts or other types of event advertising. Without a policy that clearly defines DOD’s role in oversight of the advertising activities of the military service components and outlines the procedures the components should follow for all types of advertising activities, DOD may not be able to ensure that the components are carrying out other types of advertising in a manner that it considers appropriate. More broadly, while it may be appropriate for the service components to have variations in advertising policies given that they operate separate programs in some instances and we are encouraged that DOD officials stated that they will address current weaknesses such as those we identify in this report, without a policy that defines procedures for all types of advertising activities, DOD risks abuses or inappropriate activities in other advertising in the future. Advertising activities provide information and seek to influence the beliefs and understanding of potential recruits about each military service, and the services conduct advertising to help meet their recruitment goals. The unique branding of each service plays a role in the decision of an individual to become a soldier, sailor, airman, or marine and, as such, the department relies on the military services to carry out their own advertising programs. However, despite the competition for potential recruits, a formal process for coordination of advertising activities among the military service components could improve the department’s ability to leverage resources and thus improve the efficiency of DOD’s advertising activities. Further, while the components generally follow commercial best practices we identified for evaluating advertising, DOD has not addressed variations in measurable goals among the components or insufficient data that has prevented some components from being able to assess the effectiveness of their advertising activities in generating leads. Lastly, the absence of a department-wide policy that clearly defines DOD oversight of and procedures to guide the advertising activities has allowed for activities of questionable appropriateness to occur in some instances. Without such a policy, DOD cannot ensure—through comprehensive oversight—that each service or component is carrying out advertising that meets departmental standards and that appropriately invests taxpayer dollars. We recommend that the Secretary of Defense take the following three actions: Direct the Under Secretary of Defense for Personnel and Readiness, in consultation with officials from the military service components and the JAMRS office, to develop a formal process for coordination on crosscutting issues to facilitate better leveraging of resources. As part of this process, DOD could review existing advertising programs to identify opportunities to reduce unnecessary duplication, overlap, and fragmentation and obtain potential efficiencies. Direct the Secretaries of the Departments of the Army, the Navy, and the Air Force to ensure that each military service component fully measure advertising performance. This should include both the identification of measurable goals in future versions of the service components’ advertising plans and assurance that the service components have access to the necessary performance data to determine the effectiveness of their advertising activities for lead generation activities. Direct the Under Secretary of Defense for Personnel and Readiness to ensure, as the department undertakes its effort to issue a department-wide policy for advertising, that this policy (1) clearly defines DOD’s role in overseeing the advertising activities of military service components; (2) clarifies issues related to sports-related advertising; and (3) outlines procedures that should guide the components’ advertising activities for other types of advertising, such as music concerts, other event advertising, and digital advertising. We provided a draft of this report to DOD for comment. In written comments, DOD generally concurred with our three recommendations. Specifically, DOD concurred with our recommendations aimed at improving coordination and providing more direction to military service advertising. DOD partially concurred with our recommendation regarding the need for better performance measurement. DOD stated in its written comments that it will work with the military services to develop guidance that addresses our recommendations, which will be provided in the form of a DOD issuance. DOD’s comments are reprinted in their entirety in appendix IV. DOD also provided technical comments, which we incorporated into the report as appropriate. DOD concurred with our first recommendation that the department develop a formal process for coordination on crosscutting issues to facilitate better leveraging of resources. In its written comments, DOD stated that it is developing a DOD instruction for marketing and that this guidance will formalize coordination among the military services, which it states should facilitate better leveraging of resources. DOD partially concurred with our second recommendation that the military departments fully measure advertising performance. We recommended that this performance measurement should include both the identification of measurable goals in future versions of the service components’ advertising plans and assurance that the service components have access to the necessary performance data to determine the effectiveness of their advertising activities for lead generation activities. In its written comments, DOD stated that it agrees with our recommendation in broad terms. DOD highlighted actions already underway, and stated that as part of the development of its instruction for marketing, it will further clarify and codify guidance related to performance measurement. However, DOD further states that not all goals and measures relate to lead generation and that other goals and objectives can be used to measure success. We acknowledge in this report that the goal of some forms of advertising is to improve awareness or engagement, and not solely lead generation. Therefore, we do not believe that all measurement and performance data should be tied back to lead generation if that is not the goal of the advertising activity in question. However, as we state in our report, we found that not all military services collect the necessary performance data to determine if activities intended to generate leads are performing as intended, and DOD did not address this issue of data collection in its written comments. We believe that DOD and GAO largely agree on this issue and are encouraged that DOD plans to issue an instruction that will clarify the need for better performance measurement of advertising activities. While its September 2015 interim guidance was an important step that required assessment of performance of sports related advertising, we reiterate that the draft DOD instruction in development should clarify that performance measurement is important for all types of advertising, not only sports related advertising activities, and that the appropriate goals, performance data, and performance measures be used to assess the performance of all advertising activities. DOD concurred with our third recommendation that the department issue policy for advertising. In its written comments, DOD stated that it is developing a DOD instruction for marketing and that this instruction will clearly define DOD’s role in overseeing the advertising activities of military service components; clarify issues related to sports-related advertising; and outline procedures that should guide the components’ advertising activities for other types of advertising. We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense; the Under Secretary of Defense for Personnel and Readiness; the Director of JAMRS; and the Secretaries of the Army, the Navy, and the Air Force. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact Andrew Von Ah at (213) 830-1011 or vonaha@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. The objectives of our review were to examine the extent to which the Department of Defense (DOD) (1) has coordinated its advertising activities among the military service components, (2) has followed commercial best practices to assess the effectiveness of advertising activities, and (3) has oversight of the components ‘advertising activities. To determine the extent to which DOD has coordinated its advertising activities among the military service components, we reviewed department- and service-level guidance pertaining to roles and responsibilities for advertising. We interviewed officials from the Accessions Policy office within the Office of the Under Secretary of Defense for Personnel and Readiness, the Joint Advertising and Marketing Research Service (JAMRS), and each of the military service components’ advertising programs to discuss the ways in which DOD has taken steps to coordinate advertising activities as well as ways in which the military service components coordinate with each other to achieve efficiencies in advertising. Further, we discussed any methods used to share best practices or lessons learned for advertising, either formally or informally, among the military service components. We also conducted interviews in Fort Knox, Kentucky, with officials from the U.S. Army Recruiting Command, the U.S. Army Cadet Command, and the U.S. Army Accessions Support Brigade, organizations that play a role in advertising for the active and reserve Army components. We selected these Army organizations for this objective because the Army is the only service component that has an office responsible for advertising functions—the Army Marketing and Research Group—that is separate from the recruiting function, carried out by the U.S. Army Recruiting Command. We compared identified policies and practices that describe any coordination, as well as any instances of coordination described by officials, against a best practice identified by private sector advertising experts during the course of this review and our duplication, overlap, and fragmentation evaluation and management guide. Specifically, the private sector advertising experts we met with during the course of this review stated that it is a best practice to consider an effective use of coordination to increase efficiencies in advertising. As there are seven advertising programs carried out by the military services as well as JAMRS, we reviewed these programs against the evaluation and management guide as it describes how to identify and evaluate instances of fragmentation (more than one agency involved in the same broad area), overlap (multiple agencies or programs with similar goals, activities, or beneficiaries), and duplication (two or more agencies or programs engaged in the same activities or services for the same beneficiaries) among programs. Further, the guide can help identify options to reduce or better manage the negative effects of fragmentation, overlap, and duplication, and evaluate the potential trade-offs and unintended consequences of these options. We took four major steps to determine the extent to which DOD has followed commercial best practices to assess the effectiveness of advertising activities. First, we interviewed officials from the Office of the Under Secretary of Defense for Personnel and Readiness and each of the advertising programs of the military service components to determine the types of advertising they conduct and how its effectiveness is determined. Second, to identify existing best practices for assessing effectiveness of advertising, we conducted a preliminary search, including a literature review, and determined that a well-defined and widely accepted list of best practices had not been established. Therefore, we selected a nongeneralizable sample of advertising companies and professional organizations using a “snowball sampling methodology,” which consisted of interviewing advertising industry experts from an initial set of organizations and requesting those experts to refer additional contacts to participate in the review. Based on this approach, we identified and interviewed the following organizations: Companies: Ad Council, Agent, WideOpen, Widmeyer Professional organizations: Association of National Advertisers, An executive from another private sector advertising company preferred that we not include the name of the organization that currently employs the individual, stating that the input provided during the course of our review reflected experience obtained from numerous positions held at various advertising companies. During the interviews, we asked officials to describe their knowledge of industry best practices for assessing the effectiveness of advertising. Based on the interviews, we identified key practices reported by each expert or organization, including organizational structures to safeguard against bias in performance evaluation, processes for planning and goal- setting in advance of advertising, and standard performance measures used to assess advertising. We compiled these practices into a list and provided the list to the organizations for review and comment. Although the perspectives of the organizations included in our sample are not generalizable to the advertising industry as a whole, we found a sufficient degree of consensus among the sample for the purpose of developing criteria for this review. Third, following the development of the best practices, we interviewed officials from each of the military service components responsible for advertising, as well as the components’ contracted advertising agencies, to determine the extent to which the components follow the practices we identified. We compared the most recent annual advertising plan of each of the military service components (which ranged from fiscal year 2014 through 2016, depending on the component) and other department- and service-level guidance pertaining to advertising with the Standards for Internal Control in the Federal Government, which requires that an organization have relevant, reliable, and timely information and that there is communication of that information throughout the agency in order to achieve its objectives. We also considered regulatory requirements or other potential barriers to government agencies following the best practices we identified. Finally, to determine how performance data related to advertising activities is collected and reviewed below the headquarters level, we developed and administered a structured questionnaire to lower level recruiting officials with responsibility for local advertising from two lower level units per service component. Based on our discussions with the components’ headquarters-level advertising officials, we determined the appropriate officials to participate in the semistructured interview were located at the brigade level of the Army, district level of the Navy and Marine Corps, squadron level of the Air Force active and reserve components, and state level of the Army and Air National Guards. Given the total population of 177 units, we chose to use a judgmental, nonprobability sampling approach to select 2 subordinate units from each of the seven components, for a total 14 interviews. We selected the units, in consultation with service component headquarters officials, using criteria intended to obtain a variation of perspectives within each of the components. Due to the degree of differences among the components in command structure and methods for allocating advertising funds across subordinate units, we tailored the criteria to the characteristics of each component. For several components, we selected units based on the size of the budget received for local advertising, choosing a unit with a larger- than-average advertising budget and a unit with a smaller-than-average advertising budget. If a component provided roughly equal local advertising funding to all subordinate units, we used another selection criterion to obtain varying perspectives, such as selecting two units that represented regions with different propensities to serve or two units with different recruiting missions such as an enlisted unit and a health professionals unit. We relied on the headquarters officials from each of the service components to identify states or units that matched our selection criteria. Because we interviewed a nongeneralizable sample, the results cannot be used to make inferences about the population as a whole. To determine the extent to which DOD has oversight of its advertising activities, we reviewed (1) department-level guidance on advertising and recruiting and (2) service-level guidance on the review and approval processes related to the services components’ advertising programs. We interviewed officials from the Under Secretary of Defense for Personnel and Readiness about the oversight provided to service advertising programs and the role of JAMRS. In addition, to determine how the military service components provide direction to and oversight of their respective advertising programs, we interviewed officials from the headquarters of each of the military services components’ advertising programs and we developed and administered a structured questionnaire to lower level recruiting officials with responsibility for local advertising from two lower level units per service component. We met with officials from a judgmental, nonprobability sample of lower level units that represent a geographic region or area of the United States, such as a recruiting district, squadron, or brigade depending on the service. The subordinate level units were selected to obtain distinct views within each component, based on variations in the size of budgets, recruiting goals, or the propensity of youth to serve in the military within the geographical area of the unit. We compared any departmental and service guidance, as well as the information obtained during our interviews, against Standards for Internal Control in the Federal Government, which states that an agency’s organizational structure should feature planning, directing, and controlling activities to ensure that goals and objectives of the agency can be achieved. Further, internal controls require that policies be in place to direct an agency’s activities. Lastly, we reviewed contracting data from fiscal year 2014 and some of fiscal year 2015 to identify variations in the types of activities included in military service component advertising contracts. We obtained contracting data from fiscal year 2014 and some of 2015 because this time frame included information before and after the issuance of interim guidance related to sports advertising, including the most recently available data as of March 2016. Through our discussions with relevant experts and our review of past work, we determined the data to be sufficiently reliable for the purpose of corroborating the types of activities included in advertising contracts. We conducted this performance audit from June 2015 to May 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Within the military departments, there are seven military service component advertising programs and the organizational structure of each program differs. Further, there is a joint advertising program—Joint Advertising and Marketing Research Service—that is to provide some advertising functions for the department. Within all of the military service components, except for the Army Active Duty and Reserve Component Advertising Program, the remaining six programs’ advertising function is part of the recruiting function of the respective military service component. Each of the military services has its own recruiting structures and organizations, which are responsible for the military service’s recruiting mission and functions. The role of a military service’s recruiting command is to provide support to the recruiting force and guidance for the recruitment and enlistment process. In addition, a recruiting command plays a role in developing the recruiting goals. The commands are structured similarly across the military services with some variation in organizational structure. The recruiting command is the recruiting headquarters for each military service, with subordinate commands between the headquarters level and recruiting stations or substations where frontline recruiters work to reach out to prospective applicants and discuss the benefits of joining the military. Table 2 describes each of the seven advertising programs within the military departments. Each military service component receives an annual appropriation to carry out operations and maintenance activities, including advertising and marketing. Table 3 shows the DOD reported amounts allotted to the military service components’ respective advertising and marketing activities from each component’s annual operations and maintenance appropriations for fiscal years 2015 through 2017. The Army has received the highest allotments for its advertising programs. Specifically, the Army has received about at least two to three times the amount that the other active-duty components have received. In addition, each military service component sets annual goals for recruitment and retention, for both enlisted servicemembers and officers, in order to meet defined endstrength requirements. Recruits must meet numerous standards before they are accessed into the military, specifically physical, educational, and other standards (e.g., an acceptable record of behavior). For the reserve and guard components, the geographic location of vacancies is also important for understanding and addressing recruitment and retention goals because these components recruit from a geographic area to fill vacancies in that area and recruits typically continue to live and work in that area. Conversely, active duty components recruit from anywhere within the United States and its territories to meet recruitment goals and recruits are assigned subsequently to duty stations. Table 4 shows the recruitment goals by fiscal year from fiscal year 2014 through fiscal year 2016, as of March 1, 2016. The Army and the Army National Guard had the largest recruiting missions over this time period. In addition to the individual named above, key contributors to this report were Margaret Best (Assistant Director), Serena Epstein, Mae Jones, William Lamping, Felicia Lopez, Suzanne Perkins, Carol Petersen, Ophelia Robinson, Andrew Stavisky, and Amie Lesser.

Summary: As part of its efforts to meet yearly recruitment goals for the military, DOD requested almost $575 million in fiscal year 2017 to conduct advertising intended to increase awareness of military service and ultimately generate leads for potential recruits. Senate Report 114-49 included a provision for GAO to assess DOD's advertising activities. This report examines the extent to which DOD (1) has coordinated advertising activities among the military service components, (2) has followed commercial best practices to assess the effectiveness of advertising activities, and (3) has oversight of its components' advertising activities. GAO identified best practices for assessing the effectiveness of advertising in consultation with advertising industry experts and reviewed DOD and service policies and the most recent version of each military service components' advertising plans, from fiscal years 2014-16. The Department of Defense (DOD) has taken steps to coordinate some advertising activities among the military service components, but it has not developed a formal process for coordination. DOD conducts joint market research and service officials responsible for advertising at times share some information about lessons learned. However, there is no formal process for addressing inefficiencies and to ensure information sharing among the services. GAO found examples of possible unnecessary duplication, overlap, and fragmentation that may result from the absence of coordination. For example, the Air Force has three advertising programs that contract with three advertising agencies, but officials could not provide a rationale for requiring separate programs. In the absence of a formal process for coordination, the services may be missing opportunities to effectively leverage advertising resources. While DOD has generally followed commercial best practices GAO identified to assess the effectiveness of advertising, DOD's components vary in their ability to determine whether their activities are generating leads for potential recruits. For example, while the Marine Corps has developed a framework to assess the effectiveness of its advertising including leads generated from advertising activities at the local level, Army officials stated they do not have reliable data to evaluate whether locally executed advertising activities are generating leads, and the Army National Guard does not require state units to report on the performance of their advertising activities. Without fully measuring advertising performance, especially at the local levels, DOD may be unable to ensure advertising dollars are used efficiently to help meet recruiting goals. DOD does not have comprehensive oversight of the components' advertising activities; instead, DOD's components oversee their own programs. However, examples identified by GAO and others of some components paying sport teams to provide recognition ceremonies for servicemembers—a practice later deemed unacceptable by DOD—suggest that the absence of DOD oversight may have contributed to some activities of questionable appropriateness. Further, GAO observed discrepancies in how recent sports advertising guidance was being interpreted and in service officials' understanding of regulations that direct digital advertising. Without a department-wide policy that clearly defines its oversight role, DOD lacks reasonable assurance that advertising is carried out in an appropriate manner. GAO recommends that DOD develop a formal process for coordination among its components on crosscutting advertising issues; ensure that the components fully measure the performance of advertising activities; and issue department-wide policy that, among other things, defines DOD's oversight role in advertising. DOD generally agreed with GAO's recommendations.