As identified and described in figure 1, we reported in October 2016 that agencies have frequently used five open innovation strategies to collaborate with citizens and other external parties, and encourage their participation in agency initiatives. As we described in our October 2016 report, agencies can use these open innovation strategies singularly, or in combination as part of a larger open innovation initiative, to achieve a number of different purposes. Agencies can use them to efficiently engage a broad range of citizens and external stakeholders in developing new ideas, solutions to specific problems, or new products ranging from software applications to physical devices. For example, DOE’s Wave Energy Prize competition was designed to achieve a ground-breaking advancement in technology that produces electricity by capturing energy from ocean waves. DOE’s goals for the competition were to engage a diverse collection of developers to create more efficient devices that would double the energy captured from ocean waves. According to the Wave Energy Prize website, the competition, which was concluded in November 2016, produced a significant advancement in this technology. Ultimately, 4 of 9 finalist teams—selected from 92 that registered to participate—developed devices that surpassed DOE’s goal, while the winning team achieved a five-fold improvement. Agencies can also use these strategies to enhance collaboration among citizens and external stakeholders or organizations interested in an issue, and leverage the resources, knowledge, and expertise of citizens and stakeholders to supplement that which is available within the agency. These contributions enhance the agency’s capacity and its ability to achieve goals that would be more difficult to reach without this additional capacity or expertise. In another example highlighted in our October 2016 report, the Federal Highway Administration’s (FHWA) Every Day Counts ideation initiative helps states implement innovations that improve the efficiency and quality of their highway transportation and construction projects. Every two years, FHWA engages a range of stakeholders to identify innovative technologies and practices that merit more widespread deployment. Once innovations are identified, FHWA creates deployment teams comprised of experts from FHWA, state transportation agencies, and industry to help states and other stakeholders implement these innovations. According to an FHWA report, this open innovation initiative has had significant and measurable effects in participating states. In one case, FHWA reported that deploying Accelerated Bridge Construction as an Every Day Counts innovation has allowed states to reduce the time it takes to plan and construct bridges by years, significantly reducing traffic delays, road closures, and often project costs. In our October 2016 report, we also identified seven practices and 18 related key actions, described in figure 2, that federal agencies can use to help effectively design, implement, and assess their open innovation initiatives. To do so, we analyzed and synthesized suggested practices from relevant federal guidance and literature, including public and business administration journals, and publications from research organizations, as well as interviews with experts and agency officials with experience implementing open innovation initiatives. In that report, we also illustrated how the application of these practices helped agencies effectively implement open innovation initiatives in a way that achieved intended results. Three agencies with government-wide responsibilities—OMB, OSTP, and GSA—have taken steps to support and encourage the use of open innovation strategies by federal agencies. OMB is the largest component of the Executive Office of the President, and is responsible for helping agencies across the federal government implement the commitments and priorities of the President. Among other things, it develops policies and provides direction on the use of Internet-based technologies that make it easier for citizens to interact with the federal government. OSTP is also a component of the Executive Office of the President. Among other things, it has responsibility for leading interagency efforts to develop and implement sound science and technology policies. GSA is responsible for helping federal agencies obtain the facilities, products, and services that they need to serve the public. Among other things, GSA builds, provides, and shares technology, platforms, and processes that support initiatives to invite participation from stakeholders and make data available to the public. OMB, OSTP, and GSA developed government-wide policies and implementation guidance to encourage agencies to use open innovation strategies to, among other things, advance their missions, improve programs and services, and inspire new ideas to address specific challenges. According to staff from these agencies, policies and guidance do this by: clarifying the legal authorities available to use specific strategies; highlighting the benefits and results that agencies can produce using suggesting actions for agency staff to take when designing and implementing an open innovation initiative. Table 1 lists the government-wide policies and guidance that we, in consultation with OMB, OSTP, and GSA staff, identified as key for implementing each open innovation strategy. As table 1 shows, the focus of the guidance varies across the different types of open innovation strategies. For crowdsourcing and citizen science, as well as prize competitions and challenges, toolkits provide step-by-step guidance for agency staff to follow when implementing those types of initiatives. OMB, OSTP, and GSA staff identified the U.S. Public Participation Playbook as key guidance for a variety of approaches to engage the public, including two open innovation strategies: ideation and open dialogues. They also identified the Open Data Policy and Project Open Data as the key policy and source of guidance, respectively, that cover open data collaboration initiatives. They told us that these and other resources have a broader focus, generally to help agencies inventory, manage, and release their open data. These sources do suggest some actions for agencies to take when implementing open data collaboration initiatives. However, OMB, OSTP, and GSA staff have not yet developed more detailed, step-by-step guidance for implementing specific initiatives where agencies use events or websites to engage and collaborate with outside stakeholders using open data. As one example of policy and guidance for an open innovation strategy, in September 2015, the Director of OSTP issued a memorandum encouraging agencies to use crowdsourcing and citizen science to enhance scientific research and address problems by drawing on the voluntary participation of the public. The memorandum also directs agencies to improve their ability to use crowdsourcing and citizen science by identifying internal agency coordinators responsible for seeking opportunities to use this strategy to meet agency goals. Crowdsourcing and citizen science coordinators from four of the selected agencies— DOE, HHS, EPA, and NASA—told us that having this clear statement of support has helped enhance awareness of and interest in the strategy. The memorandum was supplemented by the Crowdsourcing and Citizen Science Toolkit, which provides a step-by-step list of actions agencies can take to carry out a specific initiative, as shown in figure 3. These officials from DOE, EPA, HHS, and NASA told us that that the toolkit is an important educational resource for agency staff, helping those interested in using crowdsourcing and citizen science understand what the strategy can be used to achieve and the full range of actions they can take to develop and implement an initiative. An official from GSA’s Technology Transformation Service (TTS) told us GSA is planning updates to the toolkit to reflect the authorities provided by, and requirements of, the American Innovation and Competitiveness Act (AICA) for crowdsourcing and citizen science initiatives. In addition to developing relevant policies and implementation guidance, staff from OMB, OSTP, and GSA told us they also provide ongoing support for the use of open innovation strategies across the federal government by: answering questions from agency staff, sharing lessons learned, and providing advice and assistance; hosting training sessions and other events for agency staff to highlight how agencies can use open innovation strategies and how agencies have had success using them; and matching agency staff in need of assistance with mentors, advisors, or partners in other agencies. Table 2 provides additional information on the types of support they provide. One example of staff supporting the use of open innovation strategies is GSA’s Challenge.gov program management team. According to an official from GSA’s TTS, this team is the primary point-of-contact for staff from other federal agencies, fielding their questions about the site or other aspects of managing a prize competition or challenge. To help familiarize agency staff with how to manage a prize competition or challenge and use the Challenge.gov platform, the team conducts training sessions and has developed on-demand videos and webinars that are available through the website. In addition, to provide tailored assistance, the team created the Challenge and Prize Mentorship Program, which matches staff seeking support with experienced practitioners from other agencies who can assist with all aspects of planning and implementing an initiative using this strategy. OMB, OSTP, and GSA staff also support communities of practice for various open innovation strategies. According to staff from these agencies, these communities provide venues to bring experienced and knowledgeable agency staff together so that they can learn from one another. Through regularly-scheduled meetings and e-mail lists, the communities allow members to share experiences and lessons learned, seek and provide advice and assistance, and ensure members are up to date on relevant issues, such as the development and release of new resources or upcoming initiatives. OMB, OSTP, and GSA staff also have leveraged the expertise of community members to develop toolkits and case studies, which capture leading practices and lessons learned to ensure they are readily available to inform future initiatives. Collectively, these communities, which are described in table 3, help staff across the federal government communicate and collaborate with one another. The federal Challenges and Prizes Community of Practice further illustrates the support such communities can provide. According to a GSA official, this community was created in 2010 to bring together staff from across the federal government interested in that open innovation strategy. According to information on the Challenge.gov website, the community is comprised of federal employees representing a wide range of agency perspectives and experience levels. Staff from OSTP’s Technology and Innovation Division and GSA’s TTS told us that they have worked with volunteers from the community to develop resources, including the federal Challenges and Prizes Toolkit, to try to meet the diverse needs of its members. The official from GSA’s TTS also told us that they plan to work with members of the community to revise the toolkit to reflect the updated authorities and requirements for prize competitions and challenges in AICA. This official further stated that the Challenge.gov program team provides ongoing support to this community by maintaining its e-mail list, which allows for information sharing between members, and working with members to identify topics and arrange speakers for the community’s quarterly meetings. For instance, the Challenge.gov team helped organize the March 2017 meeting, which focused on how AICA updates agency authorities and requirements for prize competitions and challenges. According to staff from OMB, OSTP, and GSA, their agencies developed several websites, described in table 4, that support the use of open innovation strategies by: making relevant guidance and information easy to access in a single, providing agency staff access to additional resources, such as applications, templates, and documents from other agencies, that they can replicate to save time and effort; and providing online platforms for agencies to reach the public with information on their open innovation initiatives. As examples of such websites, OMB, OSTP, and GSA developed Project Open Data and Data.gov to help agencies manage and release their open data and meet requirements under OMB’s Open Data Policy. According to officials from DOE, EPA, HHS, HUD, and NASA, Project Open Data has made it easier for them to find the guidance they need by making it accessible in one place. Project Open Data also encourages agencies to hold open data events to engage with data users, and to use these opportunities to expand awareness about their open data, and collect feedback and ideas. The website provides an overview of the main types of open data community events that federal agencies hold, as well as instructions and templates to help agencies organize, publicize, and carry out various types of events. To increase awareness about the open data events that agencies are holding, and to help agencies increase participation in these events, GSA has also added an “Events” page to Data.gov. As shown in figure 4, this provides agencies with an additional online platform to inform potential participants about open data events they will be hosting. We determined that key government-wide guidance developed by OMB, OSTP, and GSA to support the implementation of various open innovation strategies reflects practices for effective implementation to differing extents. Several factors led to these variances, including differing scopes and methodologies used in their development and the dates when they were issued. We identified key guidance for the implementation of initiatives using each open innovation strategy in consultation with OMB, OSTP, and GSA staff. As noted above, staff from these agencies identified the U.S. Public Participation Playbook as key guidance for both ideation initiatives and open dialogues. Therefore, we present our assessment of guidance for those two strategies together below. OMB, OSTP, and GSA staff also identified the Open Data Policy and Project Open Data as key sources of guidance for the implementation of open data collaboration initiatives. These sources suggest some actions for agencies to take when implementing such initiatives. However, as noted earlier, they have a broader focus to help agencies inventory, manage, and release their open data. As such, our assessment found that this guidance generally did not reflect practices for effective implementation of individual open data collaboration initiatives. Our October 2016 report identified two key actions agencies should take when attempting to select the most appropriate strategy or strategies for their initiatives: 1. Clearly articulate the purpose(s) they hope to achieve by engaging the public. As we found in that report, agencies have used open innovation strategies for a number of purposes, including to develop solutions to specific problems and to leverage the expertise of external stakeholders to enhance an agency’s ability to achieve a goal. 2. Consider the agency’s capability to implement a strategy. Considerations should include whether agency leadership supports the use of a strategy, and whether they have legal authorities, financial and technological resources, and staff available to support an initiative. As shown in figure 5, we determined that guidance for all but one of the strategies—open data collaboration—fully reflects these key actions. For example, the Challenges and Prize Toolkit encourages agencies to clarify their purpose in engaging the public, including by developing a detailed understanding of the specific problem they want to address through the competition. In addition, when considering a prize competition or challenge, the toolkit recommends that agency staff: secure the approval of agency leadership to move forward; work with legal counsel to identify the most appropriate authority under which to conduct the competition, and to address any other potential legal issues; estimate the budget and resource needs of a competition; and ensure the availability of staff to monitor or run the challenge throughout its life cycle. Guidance for open data collaboration initiatives does not address the first key action—defining the purpose—and partially reflects the second key action. Although it encourages agencies to consider agency capacity and applicable laws, regulations, and policies, and the availability of resources, it does not address leadership support. To guide agencies in designing and implementing initiatives, and to provide those involved with a clear understanding of what they are working to achieve, we identified three key actions that agencies should take: 1. Define specific and measurable goals for their initiatives. 2. Identify performance measures to assess progress toward those goals. 3. Align the goals of their open innovation initiatives with the agency’s broader mission. This final action helps to demonstrate the relevance and value of an initiative to others in the agency, and reinforces the connection between the agency’s goals and the day-to-day actions of those carrying out the initiative. As shown in figure 6, our assessment shows that the guidance for all but one of the strategies fully reflects these three key actions. For example, the Federal Crowdsourcing and Citizen Science Toolkit encourages agencies to define their goals for an initiative while considering how they will measure and evaluate these outcomes. In addition, the toolkit encourages agencies to identify the specific measures that they will use to track the initiative’s outputs and activities, such as the number of samples collected or training sessions held, and to determine whether the initiative is achieving its goals. Lastly, it states that those managing initiatives should ensure that the initiative is aligned with their agency’s mission, and be able to specify how it will help the agency meet its goals. As also shown in figure 6, we determined that guidance for the implementation of open data collaboration initiatives partially addresses the first key action. Specifically, the guidance identifies illustrative goals for various types of open data events. For example, it states that the goal of a “hackathon” is to build relationships with a community of developers and designers, and to see immediate tools and prototypes built using open data. However, as written, these illustrative goals are not measurable, nor does the guidance directly encourage agencies to develop such goals for their own specific initiatives. Furthermore, the guidance does not reflect the other two key actions. To leverage the experience, insights, and expertise of those interested or engaged in the area to be addressed by an initiative, agencies should: 1. Identify and engage with external stakeholders. These are individuals or organizations that share an interest in the issue being addressed and may already be active in related efforts. For a federal agency, external stakeholders can include representatives of relevant nonprofit organizations and foundations, community or citizens’ groups, universities and academic institutions, the private sector, members of Congress and their staffs, other federal agencies, and state and local governments. 2. Look for opportunities to partner with other groups and organizations. Partners are organizations and individuals that play a direct role in designing and implementing an initiative. They provide staff capacity, resources, administrative and logistical support, assistance with communications and community building, or ongoing advice and expertise. In considering potential partners, agencies should look for groups and organizations that would be interested in, or could benefit from, the results of an open innovation initiative. As shown in figure 7, we determined that guidance for all of the open innovation strategies fully reflects both key actions. For example, guidance for implementing open data collaboration efforts in OMB’s Open Data Policy encourages agencies to engage with various stakeholders, including entrepreneurs and innovators in the private and nonprofit sectors. This engagement could then lead to these stakeholders participating in specific initiatives to use agency data to build products, applications, and services. Guidance on the Project Open Data website also suggests agencies use open data collaboration events to bring together various stakeholders, such as entrepreneurs, technology leaders, and policy experts, to explore available data and discuss new ideas for tools that the private sector could create using agency data. Project Open Data also encourages agencies to invite and partner with other government entities and private companies when developing and carrying out these events. To ensure that tasks and time frames are clear for all involved in implementing and managing an open innovation initiative, agencies should document roles and responsibilities and develop implementation plans for those initiatives. The plans should clearly identify specific tasks, the parties responsible for completing them, and the time frames for doing so. They should also outline when and how the agency will reach out to various participant groups, and how data will be collected and evaluated to determine results. As shown in figure 8, we determined that guidance for crowdsourcing and citizen science initiatives, and prize competitions and challenges, fully reflects these key actions. For example, the Challenges and Prizes Toolkit encourages agencies to create an implementation plan that clearly outlines the roles and responsibilities for those involved in an initiative. As the toolkit emphasizes, managing a competition can involve a wide range of government and contract staff with varying responsibilities, from project management to providing subject matter expertise and technical, communications, and legal support. Because of this, it is important to be clear about the role of each team member, the duties each is assigned, and how his or her work fits into the timeline for the competition. As part of this, the toolkit also states that the plan should specify the procedures that will be used to collect relevant data during the course of the challenge. Lastly, the toolkit encourages agencies to develop a communications plan that defines the audiences they want to reach with information on the initiative; the websites, news outlets, social media, and other outlets it will use to reach them; and the messages they will use to reach potential participants and encourage them to participate. In contrast, as also shown in figure 8, we found that guidance for open data collaboration initiatives does not reflect either of these key actions, and guidance for open dialogue and ideation initiatives partially reflects these actions. Specifically, the guidance for open dialogue and ideation initiatives encourages agencies to document roles and responsibilities, and to develop a plan that specifies the tasks and time frames for recruiting participants. However, the guidance does not discuss planning for data collection. To reach the right potential participants, motivate them to participate, and keep partners and participants engaged throughout the implementation of an initiative, we previously identified four key actions agencies should take. They should: 1. Use multiple outlets and venues to reach potential participants. In their outreach, agencies should use the initiative’s website, social media, press releases, journals, newsletters, and professional conferences and networks to reach potential participants. 2. Craft announcements in a way that motivates people to participate. In doing so, agencies should address the interests of potential participants and explain why it is important for them to participate. 3. Engage with participants to provide answers to questions and any necessary assistance. While an initiative is ongoing, agencies can use websites, question-and-answer sessions, e-mails, and other forms of communication to engage with participants and provide ongoing support and assistance. 4. Hold regular check-ins with those involved in implementation. Agencies should hold regular meetings to help ensure that those working to implement an initiative are aware of the status of efforts and have an opportunity to raise and discuss any concerns. As shown in figure 9, our assessment found that guidance for crowdsourcing and citizen science fully reflects all of these key actions, while guidance for the other strategies reflects some, but not all, of these key actions. For example, the Crowdsourcing and Citizen Science Toolkit encourages agencies to find the best platforms for reaching communities with information about their initiatives, and to reach out and communicate with potential participants using media and messages that respond to their interests. The toolkit also encourages agencies to let those participating in an initiative know how they can engage with the agency to provide information and feedback, and states that the agency should pay attention to shifts in participant needs and interests over time. Lastly, the toolkit instructs agency staff to hold regular meetings with an initiative’s implementation team so that everyone can understand how the project is progressing, and discuss new developments and concerns. As also shown in figure 9, for the other strategies, we determined that guidance fully reflected actively engaging participants, but none of them reflected holding regular check-ins with those involved in implementation. Guidance for ideation and open dialogue initiatives, and prize competitions and challenges reflects two additional key actions—using multiple outlets and crafting announcements to participants’ interests— while the guidance for open data collaboration initiatives does not. After an initiative has concluded, or at regular intervals if it is a long- standing or continuous effort, agencies should take three steps to assess and report results and identify potential improvements: 1. Assess the data collected during implementation to determine whether the initiative met its goals. 2. Analyze feedback from partners and participants. Agencies should identify lessons learned about what went well and what would need to be adjusted or improved for similar initiatives in the future. 3. Report publicly on results achieved and lessons learned. Doing so can demonstrate the value of an initiative and sustain a dialogue within the community of interested organizations and individuals. As shown in figure 10, we determined that guidance for crowdsourcing and citizen science initiatives and prize competitions and challenges fully reflected these key actions. For example, the Challenges and Prizes Toolkit states that agencies should assess the data they have collected to determine how well the competition achieved its goals, and to identify other results and outcomes it produced, such as quantifiable improvements to existing solutions and technologies. The toolkit also states that agencies should conduct an after-action assessment to capture feedback, lessons learned, and other institutional knowledge so the agency can improve its challenges in the future. Specifically, it suggests that agencies consider the following questions when conducting this assessment: What worked well? What would you have done differently in challenge design looking back? How might agency clearance and coordination go more smoothly next time? What could have been improved in judging, communications, and operations? Did the evaluation process result in the selection of the best submissions? What were any unintended consequences, both positive and negative? Lastly, the toolkit reminds agencies to complete required public reporting, sharing results, lessons learned, and success stories, which can be critical to improving how challenges are designed and implemented. As figure 10 also illustrates, our assessment found that guidance for open data collaboration initiatives did not reflect any of these key actions, and guidance for ideation and open dialogue initiatives reflected most of them. Specifically, that guidance encourages agencies to collect and analyze data to assess goal achievement and results, conduct an after-action review, and report publicly on results. It does not, however, encourage agencies to report publicly on lessons learned through their experience. Given the time and resources that agencies may invest to build or enhance communities of partners and participants for open innovation initiatives, our October 2016 report identified two key actions agencies should take to sustain these connections over time: 1. Acknowledge and, as appropriate, reward the efforts and achievements of partners and participants. 2. Seek ways to maintain communication with members of the community. Doing so can keep them informed of future initiatives and other opportunities, and facilitate communication within the community. As figure 11 shows, we determined that none of the guidance fully reflects the first action. The guidance for crowdsourcing and citizen science, and prize competitions and challenges, both encourage agencies to acknowledge the contributions of participants, and reward participants with monetary and nonmonetary incentives (as appropriate). However, guidance for these strategies does not also encourage agencies to acknowledge the contributions of partner organizations, which can provide critical resources, expertise, and capacity for open innovation initiatives. The guidance for ideation, open dialogues, and open data collaboration initiatives did not reflect this key action. As also shown in figure 11, our assessment found that guidance for most open innovation strategies fully reflect the second action; however, guidance for open data collaboration does not. For example, the Federal Crowdsourcing and Citizen Science Toolkit encourages agencies to continue actively engaging partners and participants, and help direct participants to other initiatives that might interest them. The toolkit also encourages agencies to create opportunities for participants to socialize and communicate with each other by supporting discussion forums, and identifying group leaders that can help carry forward a discussion among members of the community. In instances where guidance does not fully reflect the practices and key actions identified in our October 2016 report, agency staff may not be aware of certain steps they should take to better ensure the success of their open innovation initiatives. In part, the various guidance resources developed by OMB, OSTP, and GSA do not fully reflect some practices and key actions because those resources almost all pre-date our report. We also used different methodologies and sources in pulling together our practices and key actions than they did in developing their guidance. As noted earlier, we developed our practices by analyzing and synthesizing suggested practices and key actions from a wide range of relevant literature, as well as interviews with nongovernmental experts and agency officials. Our scope also was to identify practices applicable to implementing individual initiatives using any open innovation strategy. The guidance developed by OMB, OSTP, and GSA, by contrast, is generally strategy specific and based on the experiences of and lessons learned from individuals involved in the relevant communities of practice. For example, as was highlighted earlier, OSTP and GSA staff worked with volunteers from the Challenges and Prizes Community of Practice to develop the Challenges and Prizes Toolkit. Other factors contributed to the guidance for open data collaboration not fully reflecting our practices and key actions for effectively implementing open innovation initiatives. As was previously described, OMB, OSTP, and GSA developed various resources, including websites and guidance, to help agencies inventory, manage, and release their open data. According to staff from these agencies, the focus of their efforts and resources generally has been assisting agencies with the management and release of open data, while providing some support for agencies interested in mobilizing and collaborating with participants to use open data (i.e., open data collaboration), which we identify as an open innovation strategy. The existing guidance does suggest some actions for agencies to take when implementing open data collaboration initiatives. In addition, staff from OMB’s Office of the Chief Information Officer and GSA’s TTS shared several ways in which they and members of the Open Data Working Group provide support for staff interested in carrying out open data collaboration initiatives. For example, they told us that when agency staff ask for advice on how to conduct such initiatives, they often match the individual making the request with experienced staff in another agency, or rely upon members of the Open Data Working Group to answer their questions. An official from GSA’s TTS stated that because members of this community meet regularly, there are frequent opportunities to share best practices and lessons learned to inform the planning of any current or future agency open data collaboration initiatives. In addition, the official told us that agency staff can also use the open data community’s e-mail list to seek and receive assistance from fellow members. Although OMB, OSTP, and GSA staff identified various resources that are available to support agency staff interested in carrying out open data collaboration initiatives, those resources do not include detailed and consistent, step-by-step guidance for the implementation of such initiatives. The agencies we selected for this review—DOE, HHS, HUD, DOT, EPA, and NASA—have put in place various resources to support the use of open innovation strategies. These resources—policies and implementation guidance, supporting staff and organizations, and websites—complement what exists at the government-wide level. The selected agencies have generally developed, or are developing, resources for the open innovation strategies they use frequently, to provide staff with tailored guidance and support. This helps ensure staff carry out initiatives in a manner that is informed by the agency’s previous experience and that is consistent with agency procedures. In some instances, agency officials told us that they have not developed certain agency-level resources, generally for one or more of the following reasons: they found government-wide resources to be sufficient; they have not used a strategy, or use it infrequently, limiting the need for agency-specific resources; or they have not yet had sufficient experience using a strategy to be able to craft policies or other resources informed by experiences and lessons learned. Officials from the selected agencies told us that agency-level policies and guidance help to raise awareness among agency staff of the value of using open innovation strategies, and how these strategies can be an effective tool for engaging the public to generate new ideas and solutions. In instances where agency leaders have approved and issued these policies, this also helps demonstrate to agency staff that leadership supports using these strategies. The policies and guidance at these agencies (examples provided in table 5) also clarify agency-specific steps that staff should take when implementing an initiative, which, according to agency officials, helps ensure that staff take actions necessary to meet requirements and successfully carry out open innovation initiatives. As illustrated in table 5, NASA’s Policy Directive on Challenges, Prize Competitions, and Crowdsourcing Activities, which was signed by the NASA Administrator and issued in February 2014, encourages agency staff to use these open innovation strategies. The directive states that it is NASA’s policy to encourage the use of challenge activities to obtain solutions and stimulate innovation. According to an official from NASA’s Center of Excellence for Collaborative Innovation (CoECI), the directive was created to supplement government-wide policies and guidance. It defines the roles that certain agency officials, such as the Associate Administrator for the Space Technology Mission Directorate, are to play in designing and implementing challenges. It also directs staff to develop and maintain agency-level best practices and implementation guidance for challenges. In keeping with this policy, in 2014, CoECI, which provides advice and support for NASA teams to design, implement, and evaluate challenges conducted through the NASA Tournament Lab, developed a Challenge Coordinator Toolkit. The toolkit provides a detailed checklist for CoECI challenge coordinators to follow when supporting teams that are conducting those challenges. According to a CoECI official, staff with varying levels of experience working on challenges serve as challenge coordinators. The toolkit is intended to ensure that coordinators consistently follow a set of specific procedures for managing all phases of a challenge being conducted using the NASA Tournament Lab, from completing necessary documentation before a challenge is launched to holding a close-out interview to capture feedback and lessons learned. See figure 12 for an overview of the steps highlighted in the toolkit. According to officials at the selected agencies, their agencies have created staff positions and internal organizations to serve as a source of information and guidance, sharing agency-specific and government-wide resources with staff who are conducting open innovation initiatives. In some instances, agencies have dedicated staff responsible for leading efforts to use a certain strategy in their agency and providing internal support to other staff interested in using the strategy. In other instances, agency officials told us they also have staff who, given their interest and experience using a strategy, have taken on responsibilities to support others in their agency, but are not devoted full time to these efforts. Some agencies also have brought together multiple staff to create internal organizations responsible for supporting the use of open innovation strategies. These staff and organizations—selected examples of which are highlighted in tables 6 and 7, respectively—can provide tailored advice and support based on their experience with these strategies. According to agency officials, this can involve helping to ensure staff meet requirements and take other actions to successfully propose, design, and implement an open innovation initiative. As noted in table 6, DOT’s Chief Data Officer (CDO) told us that he provides technical support for the department’s open data efforts. As part of his responsibilities, he works with staff to improve how the agency collects, manages, and publishes data. He also has coordinated and supported several open data events that have allowed DOT staff to collaborate with the public, including private sector companies using transportation data, state and local stakeholders, academic researchers, and those working on new technologies. He told us that these events have served to help the agency better understand how people are using transportation data and what improvements that they would like to see made. For example, in 2015, the CDO worked with staff from throughout DOT to organize a Transportation “Datapalooza.” According to the CDO, the event allowed DOT to both highlight its own data initiatives and learn about the applications and products that private companies, developers, and technologists are developing with DOT data. It also gave DOT staff an opportunity to speak directly with data users, identifying areas where they might work collaboratively to expand the use of the data and improve quality. According to the EPA memorandum that established it in July 2015, EPA’s Challenge Review Team (ChaRT), mentioned above in table 7, is intended to help ensure that agency prize competitions and challenges meet applicable legal requirements, and have an adequate scientific foundation, financial support, and a clear communications plan. As shown in figure 13, ChaRT brings together officials from several EPA offices who, according to an official from EPA’s Office of Research and Development, need to be involved in prize competitions and challenges, creating an efficient “one-stop” approach to reviewing and approving proposals. According to the official from EPA’s Office of Research and Development, the officials who serve on ChaRT provide legal, financial, and scientific expertise and support to EPA teams developing proposals for prize competition or challenge initiatives. As part of the team’s review processes, it requires EPA staff to complete a ChaRT review form to ensure that (1) new initiatives will address agency needs, (2) leadership is aware of and supports the challenge, and (3) teams have completed required steps and addressed any concerns before they can proceed. According to the memorandum that established it, ChaRT members must concur before any prize competition or challenge can be announced. In addition to dedicating staff and organizations, according to officials from selected agencies, some of the agencies also created communities of practice (COP) for those staff interested or involved in open innovation initiatives, particularly citizen science. These include NASA’s Citizen Science COP; EPA’s Citizen Science COP and recently-created Prize and Challenge COP; and within HHS, citizen science working groups at the Centers for Disease Control and Prevention and National Institutes of Health (NIH). Like government-wide communities, those COPs and working groups at the agency level can provide a venue for sharing experiences and lessons learned from designing and implementing open innovation initiatives, or can be used to develop guidance for agency staff. For example, according to NIH officials, the agency’s Citizen Science Working Group was established in 2012 to address interest among NIH staff about how the agency could use crowdsourcing and citizen science methods. NIH officials told us the working group brings together, as of March 2017, approximately 60 staff from 14 of the 27 NIH institutes and centers to discuss various topics and share their experiences with and knowledge of implementing these initiatives. The group also hosts presentations by outside speakers. Agencies selected for this review have developed websites that, like the government-wide websites described above, can provide a central, publicly available location for stakeholders and participants to access information, including agency data that can be used in open data collaboration initiatives and details of specific projects in which they can volunteer to participate. However, officials from DOE, EPA, HHS, and NASA told us that websites developed by agencies, unlike the government-wide websites, are often designed to reach specific audiences and stakeholders that may be likely to participate in the agencies’ open innovation initiatives and in some cases provide an online forum for these audiences to collaborate. Table 8 provides examples of websites that agencies have developed. For example, DOE’s Open Energy Data website, as shown in figure 14, provides energy data users and developers with access to DOE’s catalog of open data sets that they can use to conduct research and develop applications. In addition, it provides visitors with information on how to engage with DOE to provide feedback on the department’s open data. Open Energy Data also highlights and links to Open Energy Information (OpenEI), a website sponsored by DOE and other partner organizations that allows users to share datasets and collaborate on energy data initiatives. HUD’s Switchboard, an online ideation platform that was also highlighted in our October 2016 report, allows employees, stakeholders, and other members of the public to offer ideas for improving HUD’s processes, programs, and administration. According to a HUD official, the platform was originally created in 2009 to collect input for an update to the agency’s strategic plan. Over time, however, HUD expanded the purpose of the platform to make it a more general forum for the public and external stakeholders, as well as HUD employees. Now Switchboard is intended to ensure there is an avenue for anyone to reach HUD with any ideas or feedback that could improve the department. Beyond that, the website allows visitors to offer comments on other submissions, and vote in favor of those ideas that they support. It also helps designated HUD staff collect, review, and respond to the ideas submitted, after which they will work to determine whether they can be implemented. Figure 15 illustrates different aspects of Switchboard’s capabilities. OMB, OSTP, GSA, and selected agencies have developed various resources to support the use of open innovation strategies. Of these, government-wide guidance is particularly important, as it helps agency staff understand the full range of actions they should take when designing, implementing, and evaluating an open innovation initiative. However, key government-wide implementation guidance, in particular for open data collaboration, does not always fully reflect demonstrated good practices and key actions. Better incorporating those practices and key actions would help ensure agency staff are aware of, and can take, the full range of steps to effectively design, implement, and assess their open innovation initiatives. Consistently applying these practices can help agencies ensure that their initiatives successfully achieve intended results. To help ensure federal agencies effectively design, implement, and assess open innovation initiatives in line with the practices and key actions identified in our past report, we make 22 recommendations for GSA, OMB, and OSTP to enhance relevant implementation guidance. We recommend that the Director of the Office of Management and Budget, the Director of the Office of Science and Technology Policy, and the Administrator of the General Services Administration enhance key guidance for open data collaboration initiatives to fully reflect the following 15 key actions: Clearly define the purpose of engaging the public; Consider the agency’s capability to implement a strategy, including leadership support, legal authority, the availability of resources, and capacity; Define specific and measurable goals for the initiative; Identify performance measures to assess progress; Align goals of the initiative with the agency’s broader mission and Document the roles and responsibilities for all involved in the initiative; Develop a plan that identifies specific implementation tasks and time frames, including those for participant outreach and data collection; Use multiple outlets and venues to announce the initiative; Craft announcements to respond to the interests and motivations of Hold regular check-ins for those involved in the implementation of the Collect and analyze data to assess goal achievement and results of Conduct an after-action review to identify lessons learned and Report on results and lessons learned publicly; Acknowledge and, where appropriate, reward the efforts and achievements of partners and participants; and Seek to maintain communication with, and promote communication among, members of the community. We recommend that the Administrator of the General Services Administration enhance key guidance for ideation and open dialogue initiatives to fully reflect the following four key actions: Develop a plan that identifies specific implementation tasks and time frames, including those for participant outreach and data collection; Hold regular check-ins for those involved in the implementation of the Report on results and lessons learned publicly; and Acknowledge the efforts and achievements of partners that have contributed to the implementation of an initiative. We recommend that the Director of the Office of Science and Technology Policy and Administrator of the General Services Administration enhance key guidance for crowdsourcing and citizen science initiatives to fully reflect the key action to acknowledge the efforts and achievements of partners that have contributed to implementing an initiative. We recommend that the Director of the Office of Management and Budget and the Administrator of the General Services Administration enhance key guidance for prize competitions and challenges to fully reflect the following two key actions: Hold regular check-ins with those involved in the implementation of an initiative; and Acknowledge the efforts and achievements of partners who contributed to the implementation of an initiative. We provided a draft of the report to the Director of the Office of Management and Budget, the Acting Director of the Office of Science and Technology Policy, the Acting Administrator of the General Services Administration (GSA), the Secretary of Energy, the Secretary of Health and Human Services, the Secretary of Housing and Urban Development, the Secretary of Transportation, the Administrator of the Environmental Protection Agency, and the Acting Administrator of the National Aeronautics and Space Administration for comment. In its comments, reproduced in appendix II, GSA agreed with the recommendations in this report. Staff from OMB’s Office of the Federal Chief Information Officer and Office of General Counsel provided oral comments on May 15, 2017, stating that OMB generally agreed with the recommendations in this report. In comments provided by email, OSTP’s General Counsel stated that OSTP neither agreed nor disagreed with the recommendations in this report. She stated that, given their past and ongoing responsibilities related to open innovation in the federal government, OMB and GSA are best positioned to address these recommendations. She further stated that OSTP may support OMB and GSA in these efforts to the extent that OSTP has appropriate staff in the future. EPA, NASA, and OMB provided technical comments, which we incorporated as appropriate. DOE, DOT, HHS, and HUD informed us that they had no comments. We are sending copies of this report to interested congressional committees, the heads of the agencies identified above and other interested parties. This report will also be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. Key contributors to this report are listed in appendix III. As part of the federal performance management framework originally put into place by the Government Performance and Results Act of 1993 (GPRA), and updated and expanded by the GPRA Modernization Act of 2010 (GPRAMA), agencies are to identify the strategies and resources they will use to achieve their goals. GPRAMA also includes a provision for us to periodically review how implementing the act’s requirements is affecting agency performance. This report is part of our response to that mandate, and builds on our October 2016 report that described how agencies are using open innovation strategies, and identified practices to ensure they are implemented effectively. Specifically, this report (1) identifies key government-wide resources the Office of Management and Budget (OMB), the Office of Science and Technology Policy (OSTP), and the General Services Administration (GSA) put in place to support the use of open innovation strategies; (2) examines the extent to which key government-wide guidance reflects practices for effectively implementing these strategies; and (3) identifies resources that selected agencies have developed to support the use of open innovation strategies. To identify the various government-wide resources that OMB, OSTP, and GSA have developed, we reviewed relevant policy and guidance documents related to each open innovation strategy (see table 9 below). We also reviewed information available on relevant websites, including Citizenscience.gov, Challenge.gov, Digitalgov.gov, Data.gov, and Project Open Data, and corroborated information collected from these websites through interviews with relevant agency staff. Lastly, we interviewed staff from these agencies involved in efforts to support and encourage the use of various open innovation strategies across the federal government. In these interviews, we asked staff to identify key government-wide policies and guidance for each type of strategy, and to describe their roles and responsibilities in supporting and encouraging agency use of those strategies. We also asked about their collaboration with staff in other agencies, among other things. To determine the extent to which government-wide policies and guidance reflect practices and key actions for effectively implementing open innovation strategies, we first, in consultation with OMB, OSTP, and GSA staff, identified those that were considered key for each type of strategy. These key policies and guidance are listed in table 9. For each strategy type, we then compared key guidance contents to the seven practices and 18 related key actions, listed in figure 16, that we identified in October 2016. We identified those practices and key actions by analyzing and synthesizing suggested practices from relevant federal guidance and literature, including public and business administration journals, and publications from research organizations, as well as interviews with experts and agency officials with experience implementing open innovation initiatives. We then analyzed the content of the implementation guidance for each strategy to determine the extent to which it reflects the practices and key actions we previously identified. First, two analysts reviewed each source of guidance to identify all excerpts suggesting actions in line with those we identified. Next, a third analyst separately reviewed the guidance documents to verify the accuracy of the initial determinations, or identify those areas in need of additional discussion. The analysts involved in the first and second stages of this analysis then made final determinations about whether the actions suggested in the guidance reflected those we identified. We considered guidance to fully reflect a key action when it suggested steps in line with those outlined in our report. If guidance suggested some but not all steps, we considered it to partially reflect the key action. Finally, when guidance did not suggest any steps in line with a key action, we considered guidance to not reflect it. Lastly, to identify the resources that selected agencies have put in place, we reviewed agency policies and guidance, and websites related to the use and implementation of open innovation strategies from six agencies: the Departments of Energy (DOE), Health and Human Services (HHS), Housing and Urban Development (HUD), and Transportation (DOT); the Environmental Protection Agency (EPA); and the National Aeronautics and Space Administration (NASA). We selected these agencies based on various criteria, including the number and variety of open innovation strategies outlined in their individual agency Open Government Plans. These selections were also in line with suggestions from knowledgeable staff at OMB, OSTP, and GSA familiar with agencies that have actively used such strategies. In interviews with officials from these agencies, we asked them to identify agency-specific policies, procedures, or other guidance that has been developed to aid staff carrying out open innovation initiatives. Through this consultation, we were able to identify relevant policy documents and guidance developed by these agencies. For instance, we identified EPA’s memorandum establishing the agency’s Challenge Review Team, which is responsible for approving proposals for the agency’s prize competitions and challenges. Similarly, we were able to use this consultation to identify the Federal Highway Administration’s Public Involvement Techniques for Transportation Decision-Making guide, which provides staff with guidance on how to use various open dialogue approaches. We were also able to use these interviews to identify relevant websites the agencies have developed that support open innovation efforts. For instance, we identified the openNASA website, which provides the public with access to NASA’s datasets, code, and other tools, as well as information on opportunities to collaborate with others on the use of NASA’s data. Similarly, we also used this consultation to identify the National Institutes of Health’s (NIH) Biomedical Citizen Science Hub website, which is an online space for researchers and stakeholders interested in the use of citizen science in biomedicine to collaborate. In our interviews with officials supporting the use of various open innovation strategies at these six agencies, we asked them to describe, among other things: the steps staff would go through to design, implement, and evaluate a specific type of open innovation strategy; the support available to staff throughout that process; and the ways resources at the government-wide level have helped support the use of open innovation strategies in their agency. We conducted this performance audit from February 2016 to June 2017 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Benjamin T. Licht (Assistant Director) and Adam Miles supervised the development of this report. Steven Putansu, Lauren Shaman, Erik Shive, Wesley Sholtes, and Andrew J. Stephens made significant contributions to this report. John Hussey and Donna Miller also made key contributions. James Cook, Eric Gorman, Danielle Novak, and Jason Vassilicos verified the information in the report.

Summary: To address the complex and crosscutting challenges facing the federal government, agencies need to effectively engage and collaborate with those in the private, nonprofit, and academic sectors; other levels of government; and citizens. Agencies are increasingly using open innovation strategies for these purposes. The GPRA Modernization Act of 2010 requires agencies to identify strategies and resources they will use to achieve goals. The act also requires GAO to periodically review how implementation of the act's requirements is affecting agency performance. This report identifies the open innovation resources developed by GSA, OMB, OSTP, and six selected agencies, and examines the extent to which key guidance reflects practices for effective implementation. To address these objectives, GAO identified various resources by reviewing relevant policies, guidance, and websites, and interviewing staff from each agency. GAO selected the six agencies based on several factors, including the number and type of open innovation initiatives outlined in their agency Open Government Plans. GAO also compared guidance to practices and key actions for effective implementation. Open innovation involves using various tools and approaches to harness the ideas, expertise, and resources of those outside an organization to address an issue or achieve specific goals. Agencies have frequently used several open innovation strategies—crowdsourcing and citizen science, ideation, open data collaboration, open dialogues, and prize competitions and challenges—to engage the public. Staff from the General Services Administration (GSA), Office of Management and Budget (OMB), and Office of Science and Technology Policy (OSTP) developed resources to support agency use of these strategies: to encourage use, clarify legal authorities, and suggest actions for designing and implementing an open innovation initiative; staff to advise and assist agency staff implementing initiatives and open innovation-related communities of practice; and websites to improve access to relevant information and potential participants. Six selected agencies—the Departments of Energy, Health and Human Services, Housing and Urban Development, and Transportation; the Environmental Protection Agency; and the National Aeronautics and Space Administration—also developed resources for those strategies they frequently use. These resources complement those at the government-wide level, providing agency staff with tailored guidance and support to help ensure they carry out initiatives consistent with agency procedures. For the open innovation strategies identified above, GAO determined that key government-wide guidance developed by GSA, OMB, and OSTP reflect to differing extents practices GAO previously identified for effectively implementing specific initiatives (see table). Several factors led to these variances, including differing scopes and methodologies used in their development, and when they were issued. Better incorporating these practices could help ensure agency staff are aware of, and are able to take, steps to effectively design, implement, and assess their initiatives. GAO recommends GSA, OMB, and OSTP enhance key guidance for each open innovation strategy to fully reflect practices for effective implementation. GSA and OMB generally agreed with these recommendations. OSTP neither agreed nor disagreed with the recommendations.