The federal performance management framework put into place by GPRA and GPRAMA requires agencies to develop long-term strategic plans that identify their missions, along with long-term goals and objectives (often referred to as strategic goals and objectives) aimed at achieving their missions. Agencies are to develop performance plans with near-term goals annually, to show progress towards their long-term goals and objectives. These near-term goals are called performance goals. In both of these plans, agencies are directed to identify the various strategies and resources they will use to achieve their goals. In line with these requirements, the Open Government Directive instructs federal agencies to develop Open Government Plans detailing the strategies and initiatives they would use to improve public engagement and collaboration on the agency’s core mission activities. It directs agencies to describe how they would use innovative feedback mechanisms, technology platforms, and such methods as prize competitions to increase opportunities for public participation and collaboration with those outside the agency and in other levels of government. These outside parties include those in the private, nonprofit, and academic sectors. Agencies were directed to release their initial plans in 2010, and to update these plans every 2 years. In July 2016, OMB released guidance for the development of 2016 Open Government Plans, which were to be published in September 2016. The new guidance instructs agencies to describe their activities to increase the use of open innovation initiatives. In early 2010, OMB also created an Interagency Open Government Working Group to provide a forum for open government professionals to share best practices across agencies. Representatives from 41 federal agencies made up the initial working group. OMB, OSTP, and GSA have taken additional steps to support and encourage agency use of open innovation strategies. They have developed specific policy and guidance documents, built websites that facilitate their use, and supported knowledge sharing communities of practice. For example: In July 2010, GSA launched Challenge.gov. This site is designed to help agencies find participants for prize competitions and challenges by providing a centralized list of all competitions sponsored by federal agencies. After the America COMPETES Reauthorization Act authorized federal agencies to conduct prize competitions, OMB issued guidance in August 2011 to help agencies use this authority. GSA also hosts the Challenges and Prizes Community of Practice. This group meets quarterly to discuss policies and procedures, and share ideas and practices. According to information from Challenge.gov, agencies have conducted more than 700 distinct prize competitions or challenges since the site was first launched in 2010. In May 2013, the President released an executive order requiring OMB to issue an Open Data Policy. This policy, also released by OMB in May 2013, directs agencies to collect or create information using open formats that are non-proprietary and publicly available, and to build or modernize information systems in a way that maximizes the accessibility of information. The President’s executive order also called for the creation of an Open Data Cross- Agency Priority Goal, which is designed, among other things, to provide support to help agencies release high priority data sets and facilitate the use of open data by those outside the agency. In May 2014, the administration also released an Open Data Action Plan. This plan called on agencies to use online and in-person mechanisms to engage with open data users and stakeholders to prioritize open data sets for release, improve data based on feedback, and encourage its use. OMB and OSTP have created a website called Project Open Data to provide good practices and examples to assist agencies. OMB, OSTP, and GSA also manage the Open Data Working Group, which meets every 2 weeks to share best practices and tools, and allow agencies to learn from one another. In September 2015, OSTP released a memorandum that outlined principles agencies should use when designing a crowdsourcing or citizen science initiative. The memorandum also outlined actions the agencies should take to build their respective agency capacity to use that type of strategy. At the same time, OSTP released the Crowdsourcing and Citizen Science Toolkit with practices, lessons learned, and case studies to inform agency efforts to design, implement, and sustain these initiatives. GSA has also launched Citizenscience.gov, which is a centralized repository of information on agency citizen science initiatives. As of September 2016, the Crowdsourcing and Citizen Science Catalog on Citizenscience.gov lists 303 active crowdsourcing and citizen science projects across 25 agencies. Lastly, practitioners from across the federal government have come together to form the Federal Community of Practice for Crowdsourcing and Citizen Science, which meets monthly to share lessons learned and practices for implementing and evaluating crowdsourcing and citizen science initiatives. In some agencies, this government-wide infrastructure has been supplemented by agency-level policies and organizations with dedicated staff and resources. For instance, NASA has created the Center of Excellence for Collaborative Innovation (CoECI), which assists teams from NASA and other agencies with implementing open innovation strategies, particularly prize competitions and challenges. Based on our review of agency Open Government Plans and other sources, we found that agencies have frequently used the five open innovation strategies shown below to collaborate with citizens and external parties, and encourage their participation in agency efforts. Figure 1 identifies and describes these strategies, and we provide further information about them in appendix II. Agencies can use these strategies singularly, or in combination as part of a larger open innovation initiative. For example, an open innovation initiative could primarily involve a prize competition or challenge that also has an idea generation component focused on the identification of promising new ideas or issues to be addressed. It could also have a component where participants are asked to use open data to develop new products or applications based on those ideas. We identified seven practices that federal agencies can use to help effectively design, implement, and assess open innovation initiatives. These practices are detailed below. We drew from our analysis of federal resources and relevant literature with suggested practices for the implementation of open innovation strategies. We also interviewed experts and agency officials with expertise in implementing such initiatives. While we present these practices in a certain order, this is not meant to imply they should be implemented in this sequence. Relevant literature, agency officials, and an expert we consulted emphasized that, in practice, agencies often take some of these actions concurrently or will use an iterative approach. Through our analysis of relevant literature and interviews with experts, we identified several factors agency officials should consider when selecting the most appropriate open innovation strategy or strategies to use for an initiative. First, agency officials considering the use of an open innovation strategy should clearly articulate the purpose(s) they hope to achieve by engaging the public. Through our literature review and interviews we found that agencies generally used open innovation strategies to achieve one or more of five high-level purposes. These purposes, which are not mutually exclusive, are summarized below in table 2. implement a strategy, including leadership support, legal authority, the availability of resources, and capacity. Description Agencies can collect the perspectives of a broad group of citizens and external stakeholders to identify problems or challenges, gauge perceptions of a program or service, gather reactions to proposed actions, or better understand their priorities, values and preferences. Agencies can then use this information to inform decisions about policies, plans, and the allocation of resources. Agencies can efficiently engage a broad range of citizens and external stakeholders in developing new ideas, solutions to specific problems, or new products ranging from software applications to physical devices. Agencies can also have them evaluate the quality and feasibility of the ideas and solutions proposed by others, or test the products that were developed. If it uses a successive or iterative process, the agency can help build the capacity of participants in these efforts to further develop or refine their ideas or products. Agencies can also use open innovation initiatives to stimulate the creation of new markets and companies that will then commercialize products and technologies developed for an initiative. Agencies can leverage the time, resources, and expertise of citizens and external stakeholders to supplement their own internal resources, data, and expertise. These contributions enhance the agency’s capacity, and therefore, its ability to achieve goals that would be more difficult to reach without this additional capacity or expertise. Open innovation initiatives may also allow agencies to achieve goals more efficiently and effectively than more traditional federal program types, such as grants or contracts. Agencies can establish or enhance collaboration among citizens and external stakeholders or organizations interested in an issue. This can be done, in part, by developing relationships among involved individuals and organizations. These relationships can then be leveraged to achieve common or complementary goals. Agencies can also enhance previously-established communities by using open innovation initiatives to strengthen existing relationships. This also can be done to bring new individuals and organizations into the community. An agency can provide participants or the broader public with balanced and objective information and data to help them understand an issue or problem. Information can also be provided to help them understand opportunities and various alternatives for addressing an issue or problem. To determine how frequently agencies identified these as purposes for each type of open innovation strategy, we identified both the primary strategy and the purposes agencies articulated for each initiative in their most recent open government plans. The results of this analysis are summarized below in figure 2. Through this analysis, we found that agencies identified certain purposes more frequently for different types of strategies. For example, we found that, of the 26 prize competitions or challenges identified in agency plans, agencies indicated that developing new ideas, products, or solutions was a specific purpose for 25 (or 96 percent) of the initiatives. Similarly, of the 74 open dialogue initiatives we identified in agency plans, agencies indicated that collecting information and perspectives was a specific purpose for 57 (77 percent) of them. In addition to the purpose(s) agency officials hope to achieve through open innovation, through our literature review and interviews we identified additional factors agency officials should consider when selecting the strategy or strategies that will be used: Leadership support: The support and approval of agency leaders for the potential use of an open innovation strategy is particularly important. Such leadership support can lend credibility and visibility, help generate support from others throughout the agency, and increase the likelihood an initiative will receive necessary approvals and resources. Legal authorities: Agency officials should work with their respective agencies’ legal staff to ensure that they have appropriate legal authority to use a strategy, and are aware of any relevant requirements that need to be met as they work to implement a strategy. For instance, the legal requirements that an agency must meet when conducting a prize competition or challenge can be more detailed and specific than those that apply to certain other open innovation strategies. Those considering a strategy should also be aware of any government-wide and agency-specific policies or guidance that can help guide planning and implementation of these tools. Resource needs and availability: Agency officials should also work with other relevant staff to understand what financial and information technology resources are necessary and available to support the use of various open innovation strategies. For example, agency officials could work with staff to understand whether they can design or leverage an existing website or other tool to engage and manage a community of widely-dispersed participants. Assessing resource needs and availability helps determine the costs and feasibility of implementing the selected strategy. Capacity to implement the strategy: Agency officials should consider whether their staff has sufficient time and expertise to design and implement a strategy. Agency officials could work with staff with prior experience developing and implementing open innovation initiatives. Such staff can help ensure successful practices from previous initiatives are replicated and previously-identified problems avoided. Similarly, officials can also work with agency contracting and acquisition staff to contract for additional capacity and expertise to support implementation. Below we provide illustrative examples of how NASA, EPA, and DOT selected and used various open innovation strategies to achieve specific purposes. As part of NASA’s strategic goal to expand the frontiers of knowledge, capability, and opportunity in space, the agency is examining near-Earth asteroids to determine whether any of these objects threaten Earth. In June 2013, NASA also announced its Asteroid Grand Challenge, which is a large-scale effort to use partnerships and collaboration to find all asteroid threats to human populations. NASA officials also reported that the algorithm that astronomers have been using to analyze images of space to detect asteroids can produce false detections, and the process to screen out those false detections is labor intensive and inefficient. According to NASA officials, beginning in January 2014, staff working on the Asteroid Grand Challenge began working with staff from NASA’s CoECI, who have expertise in executing prize competitions and challenges and are responsible for managing competitions launched through the NASA Tournament Lab. To ensure they could access necessary technical expertise to develop an improved algorithm to identify asteroids in images captured by ground-based telescopes, officials decided to leverage an existing NASA contract with Harvard University to carry out a series of competitions. These competitions were conducted by Harvard University’s subcontractor Topcoder, a private- sector company that administers contests in computer programming and has an existing community of expert developers and data scientists. In March 2014, NASA officially announced the Asteroid Data Hunter challenge and citizen science effort to develop the more accurate algorithm. The effort was also designed to develop a software application that would allow citizen scientists to genuinely contribute to asteroid detection, supplementing the efforts of professional astronomers. According to an April 2015 report from OSTP on the implementation of federal prize competitions and challenges, through the challenge’s 10 months, more than 1,200 participants submitted 700 potential solutions. This resulted in the development of a new algorithm and software package. Figure 3 provides a screenshot from the website where interested members of the public can download the application. According to NASA, the improved algorithm has led to a faster, more accurate asteroid detection process. NASA and Planetary Resources, Inc., a private-sector company also involved in the initiative, analyzed the results and found that the new algorithm resulted in a 15 percent increase in the positive identification of new asteroids in the main belt of asteroids that orbit between Mars and Jupiter. Furthermore, NASA also stated that the software application could increase the number of new asteroids discovered by citizen astronomers. According to NASA officials, the application has been downloaded over 8,000 times as of March 2016. NASA obtained these results with a total project cost of less than $200,000, which OSTP reported and NASA officials confirmed is less than the fully loaded cost of employing an engineer for the same time period. Excessive levels of nutrients such as nitrogen and phosphorus can harm aquatic environments, according to EPA. Governments, academic research organizations, environmental organizations, utilities, and the agriculture community are collecting data on nutrient levels. However, EPA and its partners say the general public cannot easily access or understand these data. To raise public awareness and identify new and innovative ways to communicate data on nutrient pollution to the public, EPA collaborated with the United States Geological Survey (USGS) and Blue Legacy International to conduct the Visualizing Nutrients Challenge. USGS is a scientific organization within the Department of the Interior that collects and distributes scientific data and information on the health of ecosystems and the environment. Blue Legacy International is a non- profit organization focused on the protection of water resources. The goal of the challenge, which ran from April to June 2015, was to invite participants to design innovative and compelling web applications, images, and videos to help individuals and communities understand the causes and consequences of, and solutions to, nutrient pollution. According to EPA officials, as the idea came together for an effort to identify innovative ways to translate and communicate information about nutrient pollution, EPA staff reached out to colleagues at USGS to gauge their interest in partnering. EPA officials said they did this because of the role USGS plays in collecting data on the nation’s surface and ground waters, and their interest in seeing those data communicated and used more broadly. According to EPA officials, this relationship with USGS was important because of the additional expertise and capacity USGS staff provided, as well as their support in publicizing the challenge. EPA officials explained that because of Blue Legacy International’s mission and interest in using digital media to build public awareness about the importance of local watersheds and more sustainable stewardship of water resources, it approached EPA about becoming involved. EPA officials also stated that Blue Legacy International provided $10,000 to fund its own independently-selected awards to help incentivize participation. According to EPA officials, they determined that conducting a challenge with an open call for submissions would be the preferred approach to achieve the goals established for the effort. Before the challenge could move forward it had to be reviewed and approved by all members of EPA’s Challenge Review Team. EPA officials explained that the review team consists of representatives from key offices throughout EPA. This includes individuals from the Office of General Counsel, who determine whether there is sufficient statutory authority to carry out a challenge, and the Office of the Chief Financial Officer, who ensure there are sufficient financial resources available to support the challenge. To secure additional capacity to implement the challenge, EPA contracted with InnoCentive, a private-sector contractor that manages prize competitions and challenges. According to EPA officials, they also used the contract to access InnoCentive’s large existing network of potential challenge participants with expertise in relevant disciplines, including design, physical science, and data analysis. InnoCentive played a central role by recruiting potential participants, assisting with design and development, and prioritizing issues that needed to be addressed each week by EPA, USGS, and Blue Legacy International. According to EPA officials, a competition was selected because it offered a superior cost-benefit ratio to more traditional federal contracting. According to an August 2016 report from OSTP, using this approach, EPA and USGS were able to collect 20 submissions. EPA officials said these submissions provided a wide range of examples for how to present and communicate data on nutrient pollution. They also said they achieved this in approximately 3 months and at the cost of staff time—with responsibilities shared among EPA and USGS, and Blue Legacy International—and $16,500 that EPA paid to InnoCentive to administer the competition. By contrast, EPA officials estimated that using traditional procurement processes to produce a single visualization would have cost significantly more and taken longer. In addition, they said a more traditional procurement may not have resulted in a product of the quality that was received through the competition. The Moving Ahead for Progress in the 21st Century Act (MAP-21), signed into law in July 2012, required DOT to develop a National Freight Strategic Plan in consultation with stakeholders. As we have reported, involving stakeholders in strategic planning can help ensure that efforts and resources are targeted at the highest priorities, and that stakeholders appreciate how competing demands and resource limitations require careful balancing. To inform the development of the freight strategic plan, DOT officials, led by staff from the Office of the Secretary, the Office of Public Engagement, and the Federal Highway Administration (FHWA) Office of Freight Management and Operations, decided to engage a broad range of stakeholders through a series of both online and in-person open dialogues. For example, beginning in 2012, DOT used an online platform called IdeaScale to launch an online dialogue and roundtables to leverage web-based communications technology to engage with stakeholders. According to DOT officials, the online dialogue session and online roundtables allowed stakeholders to comment and provide suggestions on various topics, including developing guidance for state freight plans and potential measures of conditions and performance for a national freight system. FHWA has also continued to conduct monthly webinars to provide information on freight issues, technical assistance, and training for those in the freight and transportation planning communities. Since 2012 these webinars have been used to cover a range of topics, including freight-related provisions in MAP-21 and other legislation, state freight planning, and improving freight system performance in metropolitan areas. In addition to its web-based outreach, DOT also used in-person meetings to engage with and collect recommendations from a range of stakeholders. For example, in May 2013 the then-Secretary of Transportation chartered the National Freight Advisory Committee (NFAC), which was comprised of 47 stakeholders from different organizations and groups with an interest in freight policy. It included representatives from state and local governments, port and transportation authorities, transportation-related companies and associations, unions, and public interest groups. DOT officials emphasized that NFAC was created to advise the department on matters related to freight transportation. They added that it was critical to ensure a wide range of perspectives would be represented. NFAC met in person 7 times between June 2013 and November 2015, and ultimately provided DOT with nearly 100 recommendations. DOT leaders also conducted nearly 60 roundtables and public meetings across the country to collect the perspective of stakeholders at the regional and local levels on various freight policy issues. According to DOT officials, the insights collected through this outreach had a large influence on the development of the draft National Freight Strategic Plan, which was released in October 2015. DOT officials told us that they received substantial public input on issues such as freight transportation safety, the adoption of new technologies, workforce development, opportunities to strengthen connections between different modes of transportation, and the need for reliable funding for freight infrastructure. Each of these issues was then addressed in specific sections of the draft freight strategic plan. DOT officials stated that these insights also informed recent action by Congress. Specifically, in December 2015, Congress enacted and the President signed into law the Fixing America’s Surface Transportation (FAST) Act which created a new grant program for nationally significant freight and highway projects and authorized appropriations for this new program as well as existing grant programs through fiscal year 2020, among other things. According to relevant literature and our interviews with experts and agency officials, once the agency has identified the high-level purposes it wants to achieve through an open innovation initiative and selected the strategy or strategies it will use, it should clearly define specific and measurable goals for the initiative. Specific goals can help guide the design and implementation of an initiative. They also can help those involved maintain a sense of direction by providing a clear understanding of what they are working to achieve. Define specific and measurable goals for the initiative. Identify performance measures to assess progress. Align the goals of the initiative w ith the agency’s broader mission and goals. Relevant literature, experts, and agency officials we consulted highlighted that the agency should also identify the performance measures it will use to assess progress towards the goals and overall results. For open innovation initiatives, measures can be used to assess the achievement of specific outcomes, participation and engagement, and resources invested in the initiative. Outcome measures could include the successful achievement of a goal, improvements in the quality of a policy or process, or the improved delivery of a service. Participation and engagement measures could include the number or diversity of participants engaged in the initiative; the number of ideas submitted; the amount of time it takes to respond to participant questions, comments, or feedback; and the satisfaction of participants with their experience. Measures of resources invested (input measures) could include the money, staff resources, and time dedicated to implementing the initiative. This information can also help an agency determine whether it would be appropriate to expand—or “scale”—an approach if it is found to be successful. Lastly, the literature and experts also emphasized that the agency should seek to align the specific goals of an open innovation initiative with the agency’s broader mission and goals. Aligning initiative-specific goals with agency priorities can help ensure the relevance and value of an initiative, by showing how its successful implementation could advance progress on the agency’s mission and goals. This alignment also reinforces the connection between the agency’s mission and goals and the day-to-day activities of those carrying out an initiative. The following two examples illustrate how DOE and EPA defined goals and performance measures for selected open innovation initiatives. According to an official in DOE’s Wind and Water Power Technologies Office (WWPTO), its Wave Energy Prize (WEP) competition is designed to dramatically improve devices that produce electricity by capturing energy from ocean waves. WEP began in April 2015 and is scheduled to conclude in November 2016. WWPTO specified in its contest documentation that the effort could stimulate private sector innovation and contribute to energy security and international competitiveness in the wave energy conversion sector. This was aligned with DOE’s strategic objective to support a more economically competitive, environmentally responsible, secure, and resilient U.S. energy infrastructure. During the planning phase, WWPTO established a specific, measurable goal in its rules for the competition. The goal required that devices developed for the competition at least double the energy capture of current technology. According to a DOE National Laboratories analysis, the average rate of wave energy capture for a group of current devices is 1.5m/$M (or 1.5 meters per million dollars). To be eligible for a monetary prize, which will range from $1.5 million for the winning team to $250,000 for the third place team, participants would have to develop a device that would achieve 3m/$M. WWPTO officials told us that this target gave participants a clear, achievable goal for which to strive. They added that the goal also was aggressive enough to represent a ground-breaking advancement over current technology. Although the competition is still ongoing, according to information on the contest website, WEP has demonstrated early success as a number of the teams are proposing innovative technologies and have demonstrated a potential to achieve or exceed WWPTO’s stated goal. To help guide its outreach efforts, WWPTO also established a goal to alert potential participants about the WEP, and have them take action by registering to participate. WWPTO officials and the prize administration team developed a detailed Communications and Outreach Plan for the competition. The plan outlined the types of metrics that could be tracked to determine the effectiveness of its outreach efforts. These metrics include the number of registered teams, and traffic to the competition website and social media pages. According to WWPTO officials, 92 teams registered to participate in the competition thanks to their aggressive communications and outreach strategy. This number was three times more than they had initially expected. EPA has a strategic objective to protect and restore watersheds and aquatic ecosystems, and has reported that it is working with external partners and stakeholders to spur technological innovations to reduce costs and pollution through improved and less-expensive monitoring. In 2013, OSTP convened the Challenging Nutrients Coalition (CNC). CNC is a group of federal agencies, including EPA, nongovernmental organizations, and academia, working together to address the issue of nutrient pollution. In November 2013, OSTP hosted a meeting of agencies and experts familiar with nutrient pollution. According to EPA, experts found that more affordable and reliable sensors are needed to collect more data on nutrient levels to inform decisions about how to manage and reduce these levels. In December 2014, the Nutrient Sensor Challenge was announced, led by EPA and supported by the National Oceanic and Atmospheric Administration (NOAA) and other agencies. The goal of the Nutrient Sensor Challenge is to accelerate the commercial development of accurate, reliable, and affordable devices that will meet user needs and be available for purchase by 2017. According to EPA officials, EPA aligned the goals of the challenge with EPA’s strategic objective. The challenge offers participants non-monetary rewards and incentives like visibility in an emerging market and access to testing services and other resources. In June 2014, the Partnership on Technology Innovation and the Environment, another member of the CNC, conducted a study to clarify the specific needs of potential sensor users. Through this study they identified standards for accuracy, precision, and cost that the vast majority of potential users would look for in devices. These became the technical requirements that devices developed for the challenge must meet to be eligible for awards. For example, most of the study’s participants identified the $1,000-to-$5,000 price range as affordable for their purposes. For this reason, EPA required that the devices built for the competition have a purchase price of less than $5,000. As of August 2016, EPA and its partners are conducting final testing on the devices submitted by participants to determine if any meet the technical requirements, and plan to announce final awards in December 2016. However, EPA officials stated that preliminary results indicate that the devices developed through the competition will meet the technical requirements that have been established. They added that several companies are developing instruments of similar capabilities and price outside of the challenge. Another goal of the competition is to produce an identified, mobilized market of community organizations, state and federal agencies, and researchers. According to EPA officials, EPA and other CNC partners, including USGS, NOAA, and the National Institute for Standards and Technology, are creating pilot programs that will allow organizations to deploy and test these sensors following the completion of the competition in late 2016. According to EPA officials, as of March 2016, 14 organizations have expressed interest in participating in EPA’s pilot program. EPA officials also stated that this pilot program will help identify organizations that may want to purchase and deploy the sensors in a more widespread way in the future. EPA officials stated that having these specific goals has been critical given the focus that they have provided. For example, the goals will help ensure that the devices developed through the challenge serve as the reliable and affordable devices necessary to stimulate the market, and to expand how widely they are deployed. Identify and engage outside stakeholders interested in the issue addressed by the initiative. Look for opportunities to partner w ith organizations on the design and implementation of the initiative. Our literature review and agency officials highlighted the importance of identifying and engaging with external stakeholders who share an interest in the issue being addressed and may already be active in related efforts. For a federal agency, external stakeholders can include representatives of relevant non-profit organizations and foundations, community or citizens’ groups, universities and academic institutions, the private sector, members of Congress and their staffs, other federal agencies, and state and local governments. By engaging with outside stakeholders, agencies can gain their support for the initiative, gain insights from their prior experience working on an issue, and see how they might use the results (e.g., products) of an initiative. This can help clarify the goals and design of an initiative. This engagement can also be used to determine what motivates stakeholders to get involved in an effort, and to identify additional stakeholders, partners, or potential participants to engage in the initiative. The literature, experts, and agency officials also emphasized that agencies should look for opportunities to partner with other groups and organizations that would be interested in, or could benefit from, the results of an open innovation initiative. Partners are organizations and individuals that play a direct role in designing and implementing an initiative. They provide staff capacity, resources, administrative and logistical support, assistance with communications and community building, or ongoing advice and expertise. Partner organizations provide these resources and assistance because they have missions or goals that overlap or align with what the agency wants to achieve through an open innovation initiative. Agencies can also consider the most appropriate and effective mechanism for formalizing these partnerships, such as collaboration agreements, contracts, or interagency agreements. Agency officials can identify partner organizations through discussions with external stakeholders, professional contacts, or research into organizations with complementary goals. Finally, agency officials we interviewed emphasized the especially important role that agency leaders can play with respect to this practice. The support of agency leaders can be particularly important, as their involvement can lend credibility and visibility to an initiative to those outside the agency. It can also help mobilize a broader community of external stakeholders and partner organizations. Below we provide illustrative examples of how DOT, HUD, EPA, and HHS identified and engaged external stakeholders and partners for three open innovation initiatives. The Federal Highway Administration’s (FHWA) Every Day Counts (EDC) is an example of an ideation initiative. EDC is designed to identify effective, market-ready innovations states could implement to improve highway project delivery. According to an FHWA official, from the beginning of the initiative in 2009, the then-FHWA Administrator and Deputy Administrator (who are now Deputy U.S. Secretary of Transportation and FHWA Administrator respectively) established and supported EDC as a state-based, stakeholder-driven program. They established the Center for Accelerating Innovation (Center) to implement the program, and worked with internal and external stakeholders to promote the idea of using innovative practices to improve how highway construction projects are performed. Every 2 years, FHWA works with various stakeholders to identify innovative technologies and practices that merit more widespread deployment through EDC. The process begins when FHWA publishes a Request for Information inviting suggestions for new innovations to consider from state, local, tribal, and industry experts. According to FHWA officials, the agency typically receives more than 100 suggestions and comments. FHWA staff review these submissions to develop a list of those innovations that are market ready, could be implemented across the country, and have the greatest potential to improve efficiency and quality in highway transportation and construction. According to an FHWA official, once this list of EDC innovations is finalized, the Center works with FHWA program offices to identify leaders for Innovation Deployment Teams. The deployment team leaders identify other team members, such as communication specialists, subject matter and technical experts from state transportation agencies, and key stakeholders like industry representatives. The deployment teams work with state transportation agencies and other stakeholders to implement the innovations that best fit their needs by providing technical assistance, training, and outreach. Once the EDC innovations are selected, transportation leaders from across the country gather at regional summits to learn about and discuss the innovations. According to a March 2015 report from FHWA, the summits are used to disseminate information on innovations so states can identify those that best fit the needs of their highway programs. The summits include interactive working sessions to foster connections among regional transportation professionals, and encourage longer-term collaboration on the deployment of innovative practices. In 2014, the summits introduced online broadcasts of the presentations and discussions so that a wider audience could participate. The President’s Hurricane Sandy Rebuilding Task Force launched Rebuild by Design (RBD), a prize competition overseen by HUD, in June 2013 to generate innovative and implementable design ideas to rebuild communities affected by Hurricane Sandy. According to HUD officials, HUD searched for external organizations and foundations with complementary missions to partner with on implementing RBD. In particular, it sought established organizations with resources, capabilities to administer a design competition, and the ability to engage local residents and stakeholders in affected communities. Several philanthropic organizations, including the Rockefeller Foundation, provided financial support to fund the administration of the competition, $200,000 cash prize awards to finalist design teams, and project evaluation. According to a 2014 evaluation of RBD conducted by the Rockefeller Foundation and HUD officials, direct outreach to potential philanthropic partners by the then-Secretary of HUD played a key role in securing their financial commitments. To help administer the competition, HUD also partnered with four local research and advocacy organizations to support the work of RBD design teams at the local level. Figure 4 summarizes the network of organizations involved in RBD. According to HUD officials, each administering partner organization was chosen for its complementary resources and expertise in research, design competitions, community outreach, regional planning and design, and local ties to the region. HUD staff also established a management plan early in the process that outlined roles and responsibilities for how these partner organizations would work together through each stage of the competition. According to HUD officials, this partnership with local organizations supporting the competition’s implementation was critical to RBD’s success. HUD officials were unfamiliar with local networks of community groups and other relevant organizations in each region, so the ability to partner with those that had knowledge, networks, and skills that HUD could leverage was valuable. These networks helped facilitate community engagement by design teams, who used meetings, community design workshops, site visits, and social media to engage hundreds of local stakeholder groups from communities affected by Hurricane Sandy. According to HUD officials, this outreach was critical to meet HUD’s expectation that projects receiving support be co-designed with communities, have local support, and be financially viable. They also said that RBD demonstrated the value that external partnerships can bring in providing expertise, capacity, and connections that help an agency achieve its mission and goals. According to EPA and HHS officials, both agencies shared an interest in developing affordable, wearable sensors that would provide wearers with information on air quality and the body’s reaction to it. The agencies jointly sponsored the My Air, My Health challenge, asking participants to develop a device that would do these things in tandem. The challenge was held in two phases, and ran from June 2012 to June 2013. According to EPA officials, EPA and HHS created a cross-agency design team that included experts from EPA’s Offices of Air and Radiation and Research and Development, and the National Institutes of Health (NIH), a medical research agency within HHS. Within that design team, one cross- agency work group focused on identifying the air pollutants and health concerns the competition would target, while another work group focused on the technology and how the devices would communicate health data. According to EPA officials, creating this collaborative design team helped ensure key subject matter experts from each agency could guide the development of technical requirements for the competition in a way that would address the shared goals of each agency. According to an HHS official, for example, during the development of these technical requirements, EPA staff identified what air quality data would need to be collected, while HHS staff identified what would need to be measured to determine the health effects of exposure to air pollution. According to EPA officials, the agencies shared responsibilities for implementing the competition’s phases. EPA implemented the first phase of the competition, which was focused on developing plans and proposals for prototypes. HHS then implemented the second phase, in which finalists developed and validated proposed prototypes. EPA and HHS officials told us that the agencies used the competition to communicate their shared interest in the technology and encourage further private-sector development. The agencies used My Air, My Health to demonstrate that open innovation initiatives involving partnerships between agencies were feasible, and that collaboration between agencies and with the private sector can allow agencies to achieve goals that they may not have the capability to achieve alone. Relevant literature and agency officials highlighted how important it is for agencies to ensure that roles, responsibilities, expectations, and time frames are clear for all involved in implementing and managing an initiative. The agency and any of its partners can do this by establishing and documenting a governance structure for the initiative that clarifies the processes that will be used to ensure regular communication; raise, discuss, and resolve any pressing issues; and make decisions. According to our literature review and interviews with experts and agency officials, the agency and any partners should develop a detailed implementation plan for the initiative that clearly identifies the specific tasks and actions needed to carry out the initiative, the parties responsible for completing them, and the timeframes for doing so; potential participant groups to engage in the initiative, including when and how the agency and any partners will reach out to various participant groups and encourage them to participate, and how they will engage with participants during and after the initiative’s implementation; and what data will be collected, and how, during and after implementation, and how the data will be evaluated to determine overall results and progress towards the initiative’s stated goals. The following two examples show how HUD and HHS developed plans for implementing and recruiting participants for selected open innovation initiatives. Switchboard is an online idea generation initiative that HUD uses to collect ideas from citizens, stakeholders, and HUD staff on how the agency can improve its processes, programs, and administration. HUD officials can then consider these ideas for potential implementation. HUD drafted a charter in 2011 to guide the initiative’s implementation that describes the overall team structure, defines the roles and responsibilities of each staff member involved in reviewing and responding to ideas submitted through the website, and names liaisons for program offices throughout HUD to review and respond to ideas that fall within their programmatic jurisdiction. See table 3 for a summary of the roles and responsibilities from the Switchboard charter. Table 3. Information on Roles and Responsibilities from HUD’s Switchboard Charter Responsibility Champion of the project. Approval and sign off of project components and requirements. Overall ownership of project from an organizational perspective; management of budget. Overall management of the project timelines and scope. Oversight of internal and external communications; sets direction for messaging. Manages day-to-day activities of project. Provide input into process, manage ideas and responses. The charter also explains the process and criteria used to evaluate an idea, and determine whether it should be elevated for consideration and potential implementation. HUD supplemented this charter with a document outlining policies and procedures for investigating, responding to, and implementing an idea. Figure 5 summarizes these procedures. According to HUD staff, Switchboard has become a tool for more effective customer service by providing an easy way for anyone to contact HUD with ideas for how the agency could do things more effectively. It has also provided the agency with a platform to host specific issue forums that are sponsored by various HUD program offices and targeted toward specific segments of the public. For example, in 2011, the HUD Office of HIV/AIDS Housing used Switchboard (then called HUD Ideas in Action) to ask for public input on how HUD should update the Housing Opportunities for Persons with AIDS program funding formula to better target resources to need. In response to this request, HUD received 17 submissions with ideas—many of which generated additional comments from participants in the forum—and a total of more than 500 votes. HUD then selected four of these submissions for further review, and incorporated recommendations from one of them into the department’s fiscal year 2013 budget request. The Neuro Startup Challenge was created by NIH and the Center for Advancing Innovation (CAI), a non-profit organization with a mission to accelerate knowledge and technology transfer, and entrepreneurship. Conducted from April 2014 to August 2015, the challenge was designed to generate promising start-up companies with business plans to commercialize NIH inventions for use in treating brain and neurological disorders. According to the collaboration agreement between NIH and CAI, the challenge supported NIH’s mission to advance research, innovation, and education to protect public health. It also aligned with CAI’s goals to encourage the commercialization of new technologies. NIH and CAI used this collaboration agreement to outline a detailed governance structure that specified the roles each organization would play in implementing the competition. The agreement also identified the respective tasks each would be responsible for completing during the various phases of the competition, along with the timeframes for each phase. For example, the agreement specified that during the planning phase of the competition, which was scheduled to run from April to August 2014, CAI would be responsible for identifying and engaging stakeholders and potential participants, as well as other deliverables, including the development of an advertising and marketing plan for the competition. The agreement also specified that NIH would provide input on the rules and criteria for the competition, the selection of inventions, and the identification of potential participants. According to an NIH official, this delineation of responsibilities was particularly important to help frame and focus efforts at the beginning of the project. In the agreement, NIH and CAI also identified the potential participants they wanted to reach through the competition. Participants included graduate and post-doctoral students and experienced entrepreneurs. According to an NIH official, NIH and CAI particularly focused on engaging those affiliated with universities, given the focus on connecting university students with real-world experience in business planning. Prior to launching the initiative, CAI planned for extensive contact with university faculty and students to get feedback on the concept and to make them aware of the challenge. CAI then conducted an extensive series of phone conversations and in-person meetings to connect with stakeholders and potential participants at 37 universities in 14 states. Through this outreach they reached approximately 1,500 people with information on the challenge. According to NIH officials, many of the more than 70 teams that participated in the competition were from those universities contacted through this outreach. CAI also reached out to local economic development groups and universities to identify entrepreneurs and business developers who would be interested in supporting participating teams. Relevant literature, experts, and agency officials emphasized that when agencies are ready to move forward with implementation, they should announce the initiative in a way that generates interest among potential participants. This involves using multiple outlets and venues—including the initiative website, social media, press releases, press conferences, journals, newsletters, and professional conferences and networks—to ensure they reach the right potential participants and make them aware of the initiative. The participants that an agency and any partners seek to engage, and how they decide to solicit participation, will vary depending on the purposes of the initiative. For instance, if an agency wants to use an initiative to address a very specific technical issue it may attempt to identify and engage individuals with the requisite skills through an existing network of experts. However, if an agency intends to use an initiative to collect a wide range of perspectives on an issue, it will likely need to be much more open and inclusive in its outreach and encourage diverse groups to participate. Efforts to promote the initiative are important because reaching the right participants and motivating them to participate is critical to the overall success of an initiative. According to the literature and our interviews, the initial outreach to potential participants should be crafted and communicated in a way that responds to the interests and motivations of potential participants, and explains why it is important for them to participate. In addition, the agency should also establish clear expectations for participants, describing in detail what they will be expected to contribute; how and when their contributions will be collected, evaluated, and used; and what participants must do to receive any monetary or non-monetary incentives that may be provided. Once the initiative begins, the agency and any partners should use websites, question-and-answer sessions, emails, and other forms of communication to keep participants apprised of progress. Through the literature and our interviews we also found that agencies and their partners can actively engage participants to solicit and respond to any questions, comments, and feedback, and provide any necessary assistance. These actions can increase the likelihood that participants will have a positive experience, and can help show that their participation and contributions are valued. According to experts and agency officials with whom we consulted, however, doing this can be a very resource-intensive activity, particularly if the initiative has a large number of participants and there is a high volume of communication from participants. Therefore, during the planning phase, the agency and any partners should work together to ensure that the party responsible for this aspect of implementation has sufficient capacity to respond in a timely fashion. Agency officials highlighted that the agency and any partners should also use regular check-ins to discuss the progress of the initiative. Such check-ins can help ensure those involved in implementation know the status of specific implementation tasks against established time frames, and any decisions that may be needed. The agency and partners should also review the data and feedback that are being collected during implementation. This will allow them to identify and make any necessary adjustments to improve implementation and the experience of the participants. As illustrated below, HUD, DOE, and HHS engaged participants and partners during the implementation of three open innovation initiatives. HUD’s objective for its outreach to potential participants for Rebuild by Design (RBD), according to an April 2015 report from OSTP on the implementation of federal prize competitions and HUD officials, was to recruit world class design talent to participate in the competition. It used its network of project partners, professional associations, university programs, as well as websites focused on planning, design, and urban issues, to promote the competition. For example, the American Institute of Architects launched a communications campaign urging its membership to participate in RBD. According to the April 2015 OSTP report, this outreach was successful, as HUD ultimately received high-quality proposals from 148 teams representing top engineering, architecture, and design firms. According to HUD officials, after 10 design teams were selected to participate in RBD, HUD and its partners regularly communicated with the teams to identify challenges they faced and assistance that they needed. HUD officials explained that RBD was designed to allow more than one winner, as each finalist team worked to develop innovative approaches for rebuilding and resilience in a different community. As a result, the RBD management team facilitated collaboration between the design teams. This allowed the teams to share good practices and learn from each other’s experiences. According to a 2014 evaluation of RBD conducted by the Rockefeller Foundation, HUD’s local administering partners supporting RBD’s implementation also worked closely with the design teams and provided logistical support and connections to community-based organizations and public officials. To ensure clarity about reporting requirements and deadlines, those managing RBD also instituted other means of communication. This included biweekly memorandums for the design teams and weekly phone and e-mail communications with partner organizations providing support to teams at the local level. The Rockefeller Foundation also reported that effective management practices and regular communication allowed the design teams to meet all procedural deadlines and milestones despite the initiative’s fast pace and logistical challenges. In the Communications and Outreach Plan developed for the Wave Energy Prize (WEP), DOE’s Wind and Water Power Technologies Office (WWPTO) set a goal to expand the community of developers involved in wave energy conversion technology. It sought to do this by drawing in both experienced energy device developers and newcomers representing a diverse group of companies, universities, and individuals. According to WWPTO officials, to generate a large pool of new and experienced developers for the competition, which began in April 2015 and is scheduled to conclude in November 2016, they used multiple outlets and venues to encourage WEP participation. As outlined in the Communications and Outreach plan for the competition, this included the WEP website, social media, email marketing, presentations, and outreach to various media outlets to reach a broad range of potential participants. Communications used to recruit participants also emphasized several key messages to motivate interested individuals and teams to participate. These messages included the availability of a monetary prize, the opportunity to help solve a difficult technological problem, and the chance to work on technologies that could contribute to the nation’s energy independence. See figure 6 for examples of these communications. According to WWPTO officials, to ensure there would be participants with technical expertise in energy production technology, WWPTO officials reached out to individuals who previously had contacted WWPTO regarding other projects involving wind and water power. WWPTO also promoted the competition through specific industry publications, outreach to professional and academic organizations focused on relevant technical specialties, and presentations at energy technology-oriented conferences. According to WWPTO officials, through this outreach, they attracted both new and experienced developers to participate in WEP. Of the 92 teams that registered to participate in WEP, most were previously unknown to WWPTO. Furthermore, out of the nine finalists and two alternates that were chosen to participate in the final phases of the competition, only two had received any prior funding from WWPTO. WWPTO officials also reported that they were successful in reaching teams with sufficient technical expertise to reach aggressive technical goals. According to information on the competition website from March 2016, while the devices of finalist teams are currently undergoing final building and testing, preliminary evaluations indicate that many of them could achieve or exceed WWPTO’s goals for the competition. According to WWPTO officials, the prize administration team has also created processes to regularly engage with teams participating in the competition. For example, the prize administration team holds biweekly calls with participating teams and technical experts. These calls prepare them for the final testing program, solicit and respond to participants’ questions and comments, and provide any necessary technical assistance. According to WWPTO officials, these interactions can be time- and resource intensive, so they planned for them during the early phases of the competition. This ensured that the prize administration team allocated sufficient resources to fulfill their participant management responsibilities. Furthermore, WWPTO and the prize administration team also hold weekly conference calls to discuss progress on key tasks and any adjustments that may be needed. These check-ins help ensure that WWPTO and the prize administration team are working from a common set of expectations. It also allows WWPTO to provide the prize administration team with any necessary information it needs to successfully implement WEP. OpenFDA is an open data platform released by the Food and Drug Administration (FDA) in June 2014. FDA, an agency within HHS responsible for assuring the safety of drugs, medical devices, and food, uses OpenFDA to make several key datasets available in a format that allows researchers and developers to more easily use the data. According to an August 2014 report from Iodine, a private health data company that assisted FDA in the development of OpenFDA, as the platform was developed and became available for testing, FDA officials actively engaged potential users. The officials solicited input from a group of individuals and organizations that had expressed interest in the platform and were willing to contribute feedback. The report also stated that FDA officials observed that some of those testing the platform had difficulty using it. As a result, FDA took actions to make the platform more user friendly. These actions included adding an interactive tool that allows users to filter and visualize the data more intuitively. According to an FDA official, these changes permitted OpenFDA users without technical expertise to more easily use and benefit from the platform. In addition, FDA officials actively monitored the online forums created for users of OpenFDA, and responded to any requests for clarity or information. According to FDA officials, engagement with users has been a priority. Through direct contact with the community of users, the agency has collected information to help ensure OpenFDA will serve their needs. For instance, in December 2015, FDA made the data on OpenFDA available for direct download as a result of requests from users. In June 2016, FDA also launched an updated version of OpenFDA that was redesigned in response to user feedback. This feedback included the need to improve the website’s layout. Relevant literature and agency officials emphasized that after the initiative has concluded, or at regular intervals if it is a long-standing or continuous effort, the agency should assess whether the initiative has achieved its goals. By analyzing the data it has collected, including quantitative performance data and qualitative data provided by participants on the effects of an initiative, the agency can determine if it has met its goals. When a goal is unmet, the agency should conduct additional analyses to understand why. In addition, because some outcomes may not be observable until months or years later, agencies can consider whether a long-term monitoring or assessment plan is needed and appropriate. According to relevant literature we reviewed, the agency should also conduct an after-action review to analyze feedback from partners and participants. Such a review can help identify lessons learned and process improvements that could be applied in future initiatives. For example, participant feedback may provide insights on parts of the process that went well and others that could have been executed better. These can then be replicated or adjusted, accordingly, for reoccurring or similar initiatives in the future. The agency can also engage with partners to review planning and implementation activities to identify what worked well and any notable gaps or challenges that may need to be addressed in future initiatives. Lastly, relevant literature and experts emphasized that once the agency has assessed the initiative it should publicly report on the results achieved and lessons learned. This transparency can help build trust with partners and participants, demonstrate the value of open innovation initiatives to other stakeholders and the public, and build momentum for future initiatives. Reporting results while partners and participants are still engaged can also help sustain a dialogue and increase awareness within the community of interested organizations and individuals. For the following three open innovation initiatives, we present how DOT, NASA, and DOE collected data, and assessed and reported results. The Federal Highway Administration’s (FHWA) Every Day Counts (EDC) initiative focuses on ensuring that proven innovations to improve highway construction and safety are quickly and broadly deployed. FHWA launched EDC in 2009. FHWA tracks progress toward this goal primarily by measuring the number of states that are deploying specific innovations being supported by EDC, along with whether the innovation is being developed, tested, assessed, or adopted as a standard practice. According to FHWA officials, staff from the Center for Accelerating Innovation (Center), which is responsible for implementing EDC, and deployment teams use this data to track how the level of deployment compares with goals established at the beginning of each 2-year cycle. Figure 7 shows the January 2015 baseline data for the e-Construction innovation, the progress made through December 2015, and the overall goal the agency is working to achieve by December 2016. According to an FHWA official, staff from the Center work with deployment teams to develop implementation plans for each innovation, which include identifying interim performance goals that will be used by the team to track implementation progress. FHWA officials say setting specific performance goals for deployment helps to ensure accountability for the advancement of innovations. For instance, the Director of the Center meets with the leader of each deployment team each quarter to review progress toward established goals. According to an FHWA official, these review meetings can result in the provision of additional resources or assistance to deployment teams, or, in some circumstances, adjustments to team leadership. FHWA has also established regular reporting cycles for EDC. It releases two progress reports each year that summarize the status of each innovation. In addition, it has also produced a final report at the end of previous two-year cycles summarizing the highway community’s accomplishments and progress. The final report includes data on how widely each innovation was deployed, accomplishments in states where innovations were deployed, and explanations of benefits and lessons learned through implementation. According to FHWA officials, publicly reporting results increases transparency and shows the effects of the EDC program. It also highlights successes achieved by state and local agencies in deploying innovations faster. For instance, FHWA reported in its July to December 2015 EDC progress report that the program has accelerated the deployment of innovations across the country. Every state implemented at least 8 of the 38 innovations promoted under the initiative since 2010, while some have adopted over 20. Furthermore, in August 2014, FHWA released a report with examples demonstrating that implementing EDC innovations has had significant and measurable effects in participating states. For example, FHWA reported that deploying Accelerated Bridge Construction as an EDC innovation has allowed states to reduce the time it takes to plan and construct bridges by years. This significantly reduces traffic delays, road closures, and often project costs. In 2015, Congress enacted and the President signed into law a requirement that FHWA continue to use EDC to work with states, local transportation agencies, and industry stakeholders to identify and deploy proven innovative practices and products. NASA worked with Expert and Citizen Assessment of Science and Technology (ECAST), a network of institutions that encourages public input on science and technology policy issues, to solicit the views of citizens on options for defending the Earth against an asteroid strike and exploring asteroids. These in-person and online forums, known collectively as the Asteroid Initiative Citizen Forums, took place in November 2014 and February 2015. The forums were used to obtain information on participant preferences, priorities, and values. NASA officials used this input to inform, among other things, decisions about its future mission and technology investment goals. This includes detecting asteroids, mitigating asteroid threats, and exploring asteroids with astronauts. For example, after the forums were held, relevant results were shared with NASA managers to inform the selection of a specific technology and approach that would be used for a future mission to capture an asteroid. According to NASA officials, the results of these forums provided NASA with insights into public understanding and views on NASA’s asteroid work. Figure 8 illustrates the platforms used for both the in-person and online forums. According to NASA officials, NASA also wanted to use the forums to identify lessons that could guide its future efforts to engage citizens. To do this, ECAST had participants complete post-forum surveys and provide written comments on their experience in the forum. ECAST then analyzed this information. Observers at selected tables also helped assess the meaning of written comments from participants. Through an after-action review, ECAST identified a small number of issues to address in preparation for any future forums. These included insights into the ability of citizens to understand complex information, and the need to provide clearer information to participants about how the forum results would be used. Members of ECAST involved in designing and implementing the forums also summarized their observations on potential refinements in their final report to NASA. For example, ECAST found that connecting attendee background information to individual responses could have also provided context for interpreting the written results. ECAST members also found they needed more time to test background materials given to participants to read before the event, and needed to take additional steps to increase consistency across table facilitators. DOE’s SunShot Catalyst initiative (Catalyst) was a series of competitions first begun in May 2014. It was designed to engage entrepreneurs, solar professionals, and software and data experts to help them rapidly develop start-up companies with viable technologies to address identified challenges in the solar and energy efficiency markets. By providing intensive training and support to those with the most promising ideas, DOE officials also wanted to ensure that teams would have market-ready innovations and viable business plans at the end of the competition. According to DOE officials, DOE selected 35 teams to participate in the initiative. According to a DOE official, in order to determine the initiative’s effectiveness, DOE developed a long-term effort to monitor the status of the companies created through the competition. DOE officials said that they collected publicly-available information on the status of the 35 teams that participated in Catalyst. DOE officials reported in June 2016 that through collecting this information they found that 28 teams were still actively pursuing their startups. DOE officials also invited all 35 teams to one-on-one discussions, and were able to meet with 24 of them. Through these discussions, DOE collected information on the amount of capital the teams had raised, projected annual revenues, and the benefits they gained from participating in Catalyst. For example, the 24 teams reported that they had collectively raised a total of $6.4 million in private capital or public funding, had 95 full-time employees, and had total expected annual revenue of $5.6 million. As DOE also reported, officials also identified specific lessons learned at each stage of the Catalyst process that can be used to inform how future competitions are conducted, and improve and expand the Catalyst program. For example, DOE reported that the most effective way to reach potential Catalyst participants was through the networks of previous participants, along with recruitment efforts involving local partners and events. DOE also reported that the 60-day period provided for participants to develop their prototypes was challenging, and that the department should consider adding time to that phase of the competition. DOE also compared the cost and time for product development under the Catalyst approach to those supported through DOE’s traditional financial assistance awards, including cooperative agreements and grants. According to DOE officials, the agency had learned through earlier efforts to engage developers that the application process for traditional funding opportunities can create a barrier for those who may not be interested in or able to go through what can be seen as an extensive review and approval process. DOE wanted to use Catalyst to test a faster, more open way of engaging developers and entrepreneurs. Through its assessment, DOE found that, under a traditional funding opportunity, it typically takes 9 months to move from the announcement of the opportunity to the award being made, with minimum awards ranging from $300,000 to $500,000 for software or applications. By contrast, for Catalyst, this process was completed in 3 months, with $25,000 prizes awarded to rapidly test and validate prototypes. Given the time and resources that agencies may invest to build or enhance communities of partners and participants for open innovation initiatives, agencies can take steps to sustain these connections over time. This is particularly important if one purpose of the initiative is to build a new, or bring greater coherence to an existing, community of interested organizations and individuals to work together on an issue. However, this may be less applicable when an initiative is discrete in scope and intended to be a one-time occurrence. Seek to maintain communication w ith, and promote communication among, members of the community. According to relevant literature and agency officials, agencies should acknowledge and, where appropriate, reward the efforts and achievements of partners and participants so that they feel their contributions are valued and appreciated. This can be done in conjunction with reporting the results of and lessons learned from the initiative, or through separate venues such as announcements, award ceremonies, or recognition on the initiative website. As part of this effort, it is also important for agencies to explain how the contributions of partners and participants helped the agency achieve, or progress toward, its goals, and to communicate the next steps that will be taken following an initiative. Relevant literature and experts we consulted also highlighted that agencies can seek ways to maintain communication with members of the community to keep them informed of future initiatives and other opportunities of interest, and facilitate communication within the community. To ensure these activities receive sufficient attention over time, an agency may need to assign staff the responsibility of maintaining contact with these communities. Efforts to sustain a community over time can help enhance collaboration to continue progress on addressing an issue, and provide the agency with a network that could be more easily mobilized again for future initiatives. At some point, these communities may become self-sustaining, with members continuing to collaborate with little or no involvement from the agency. To illustrate how agencies have built and sustained communities of interested partners and participants by implementing open innovation initiatives, we provide the following three examples from EPA, NASA, and HHS. From 2012 to 2015, EPA’s Office of Research and Development held a series of air pollution sensor workshops that were, according to EPA officials, designed to better understand the needs of governments and community groups interested in using these sensors, and to build a more coherent community of users and developers. EPA held the first workshop in March 2012. It provided a forum for the exchange of ideas and collaboration among people who use and research air pollution sensors to learn from their successes and challenges. Seventy people representing federal agencies, state and local governments, academia, private industry, and community-based organizations attended the workshop. According to EPA officials, workshop attendees agreed that it was helpful to have EPA convene these groups so that they could learn from each other, and build greater trust and understanding through collaboration and communication. Subsequent workshops held from 2013 to 2015 focused on specific issues, including data quality, citizen science, and community-based monitoring. In addition to in-person attendance, the workshops were also broadcast as webinars to allow those unable to attend in person to participate. Each year, there was increased interest in the workshops. More than 800 people participated in the 2015 workshop, both in person and via the webinar. The 2015 event, which was used to provide training on how to conduct community air monitoring, was, according to EPA officials, designed to build on the three previous annual workshops, whose participants requested more hands-on training opportunities. EPA officials reported that these regular workshops helped sustain this growing community, providing opportunities to build partnerships and identify and address stakeholder needs. In addition to these workshops, EPA officials continue to share information and resources to keep individuals in the community engaged in efforts to develop and deploy improved air sensors. For example, after the 2015 workshop, EPA officials told us that they hosted regular follow- up conference calls with 30 in-person attendees chosen because of their involvement in air-monitoring projects in local communities. EPA officials also said they periodically e-mail past workshop participants to inform them about webinars, funding opportunities, and other items of interest. In addition, according to documentation from EPA, officials have regularly given presentations to stakeholder groups on community air monitoring. EPA has also made resources available to support sensor developers and citizen scientists, establishing a sensor technology testing program to provide feedback to developers and users, and an online “Air Sensor Toolbox” that offers training videos and answers to frequently asked questions about community air monitoring. According to EPA officials, these efforts to build and sustain the community interested in air sensor technology have contributed directly to EPA’s strategic goal to improve air quality. Prior to the series of workshops, EPA had little in terms of ongoing research related to the development, testing, and use of air sensor technology. The workshops identified a need for, and inspired, a concerted research effort to identify promising technologies, evaluate technology performance in field and laboratory tests, and explore the use of these new technologies and the data they produce. Each year since 2012, NASA has held an annual 2-day event called the International Space Apps Challenge. At this event, teams of scientists, developers and students use publicly-available data to design solutions to identified challenges. According to NASA’s report on the 2015 event, Space Apps is used to make the agency’s open data and assets available to the public, with the aim of giving people new ways to produce relevant open-source solutions to global challenges. According to a NASA report on the Space Apps Challenge and a NASA official, beginning with local Space Apps events in 25 cities across the world in 2012, each year the number of local events has increased. In 2016, local events were held in 161 locations spanning 61 countries. NASA relies on local volunteer hosts to secure venues, manage logistics, and promote the events. Given the importance of sustaining relationships with those at the local level experienced in hosting these events, NASA has provided tools to help ensure local hosts have a positive experience. For example, NASA created a toolkit that provides prospective hosts with practical advice, guidelines, and best practices for hosting a local event. According to the NASA Space Apps report, three months prior to the event, NASA staff begins to actively engage with those organizing local events. They provide weekly suggestions, reminders, and resources to help hosts plan and manage their local events. NASA staff also convenes periodic planning conference calls with local hosts to communicate new information and answer questions. According to a NASA official, actively engaging through planning calls makes a significant difference. New hosts can ask questions of, and learn from, experienced hosts and NASA staff. It also allows local hosts to share ideas and advice with one another. This official also stated that, by engaging with the community, NASA can learn more about the support that local hosts need and collect their suggestions. For example, rather than having one planning call each week, NASA holds three different calls to accommodate the varying schedules of local hosts in different time zones. According to the NASA Space Apps report, after the completion of each year’s event, NASA also acknowledges and honors hosts and winning participants by recognizing them on the Space Apps website, in public reports, and through other venues, like invitations to launches. Figure 9 provides an excerpt from the website used to acknowledge finalists and winners from previous challenges. According to a NASA official, all of these elements combined have helped them maintain a strong level of involvement by hosts at the local level. For example, she said 78 percent of the hosts for 2016 local events had returned after hosting events in previous years. She also stated that, by regularly engaging with a community of people using the agency’s data, Space Apps has helped NASA meet its open data goals and mandates. For example, through feedback from Space Apps participants, NASA officials learned how difficult it could be to use the agency’s open data. This led to action to improve the usefulness of the datasets and house them in one location to make them more accessible. NASA also used feedback from Space Apps participants to redesign the agency’s websites and make it easier for visitors to understand and use NASA data. According to FDA officials, one of the key goals of the OpenFDA initiative is to build an open community of users around FDA data. OpenFDA developers emphasized the importance of direct contact with, and feedback from, external users. However, due to resource limitations, the developers knew it would be difficult to actively monitor online feedback boards and regularly address individual questions or requests. They wanted to create infrastructure to both support users and make the community somewhat self-sustaining. According to FDA officials, they believed that an engaged community would help provide resources and assist new users. According to an August 2014 report from Iodine, the private health data company that assisted FDA with the development of OpenFDA, the infrastructure that FDA put in place relies upon two online forums, StackExchange and GitHub. These forums facilitate communication and information sharing among members of the community. They allow developers and researchers who use OpenFDA to ask questions of the broader community of users, and get answers to those questions. This allows lessons learned and insights to be spread among the community. According to FDA officials, these forums also allow users to recommend fixes to problems with, and make improvements to, the OpenFDA source code. According to data available on the StackExchange and GitHub websites, both forums have been actively used. For example, since OpenFDA’s June 2014 launch, members of the community of users have submitted more than 100 questions on the StackExchange forum. Nearly 90 percent of those questions have been answered by other members of the community. According to an FDA official, since OpenFDA’s launch GitHub has also been used to identify nearly 50 issues with the OpenFDA platform. As of June 2016, 39 of those issues have been addressed. We provided a draft of the report to the Office of Management and Budget, the Office of Science and Technology Policy, the General Services Administration, the Departments of Energy, Health and Human Services, Housing and Urban Development, and Transportation, the Environmental Protection Agency, and the National Aeronautics and Space Administration for comment. These nine agencies provided responses via emails transmitted between September 16 and September 27, 2016. All nine agencies concurred with the findings of the report. In its response, provided in an email from the OSTP General Counsel transmitted on September 22, 2016, OSTP raised a concern that the report does not include an example of an initiative that only involved the use of citizen science. Our primary objective for this report was to identify, and illustrate through selected agency examples, practices that promote the effective implementation of open innovation strategies. Therefore, our focus was on selecting those initiatives with the greatest potential to illustrate aspects of these practices. In making those selections, we ensured that the sample covered the five types of open innovation strategies frequently used by federal agencies. Although we did not include an initiative that only used citizen science, we included initiatives that involved the use of citizen science in combination with other strategies, such as NASA’s Asteroid Data Hunter initiative and EPA’s efforts to encourage the use of air pollution sensors. In addition, DOE, HHS, HUD, NASA, and OSTP provided technical comments, which we have incorporated as appropriate. We are sending copies of this report to interested congressional committees, the heads of the agencies identified above and other interested parties. This report will also be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or mihmj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of our report. Key contributors to this report are listed in appendix III. The GPRA Modernization Act of 2010 includes a provision for us to periodically review how implementation of its requirements is affecting agency performance. This report is part of our response to that mandate. Our specific objective for this report is to identify, and illustrate through selected agency examples, practices that facilitate the effective implementation of open innovation strategies and the effects, if any, the use of those strategies have had on agency performance and opportunities for citizen engagement. To identify the various open innovation strategies federal agencies have used to facilitate participation by, and collaboration with, citizens and other non-profit, academic, and private sector partners, we reviewed documents, reports, and resources from the Office of Management and Budget (OMB), the Office of Science and Technology Policy (OSTP), and the General Services Administration (GSA), and analyzed the Open Government Plans of federal agencies. Through our review of these reports, and the most recent open government plans from 35 agencies, we identified 5 open innovation strategies that agencies have frequently used to engage citizens and external stakeholders. To identify practices that can facilitate the effective implementation of open innovation strategies, we analyzed and synthesized information gathered from a number of different sources. First, we collected relevant federal resources, including guidance with suggested practices for implementing various open innovation strategies developed by OMB, OSTP, and GSA. Through a literature review of relevant publications from public and business administration journals, and research organizations we identified those with suggested practices for the design and implementation of open innovation initiatives in the public sector. We then analyzed and synthesized suggested practices in these sources to identify areas of commonality between them. We interviewed 14 open innovation experts with experience in implementing open innovation initiatives or with academic or consultative expertise in this area. We also interviewed officials involved in implementing open innovation initiatives at six selected agencies, as well as staff from OMB, OSTP, and GSA. We initially selected and interviewed experts based on the results of our literature review (e.g., the authors of relevant articles or books with suggested practices for the design and implementation of open innovation initiatives). Based on suggestions from those individuals, we expanded our list of experts and conducted additional reviews. Through our analysis and expert interviews, we developed a broad set of practices that facilitate the effective implementation of open innovation initiatives. We refined the list of practices through our audit work at selected agencies (see below); reviewing our body of work on performance management and collaboration; and incorporating feedback from the open innovation experts we had previously interviewed, and from knowledgeable federal officials at OSTP, GSA, and other agencies. To illustrate how actions that selected agencies have taken to carry out open innovation initiatives have reflected effective practices, and the effects the application of these practices had on agency performance and citizen engagement, we selected six agencies for more in-depth review: the Departments of Energy, Health and Human Services, Housing and Urban Development, and Transportation (DOT); the Environmental Protection Agency; and the National Aeronautics and Space Administration. We selected these agencies based on several criteria, including the number and variety of open innovation strategies outlined in their individual agency open government plans. These selections were also in line with suggestions we independently obtained from knowledgeable staff at OMB, OSTP, and GSA that were familiar with agencies that have actively used such strategies. We also identified and selected 15 specific open innovation initiatives led by these 6 agencies which would allow us to illustrate how these agencies have applied effective practices for implementing open innovation initiatives. We selected these initiatives based on our review of the open government plans for the 6 selected agencies, and of OSTP reports on the implementation of prize competitions and challenges. Suggestions from knowledgeable agency staff also contributed to our selection process. These initiatives are listed below in table 4. At these agencies, we reviewed relevant agency documents and interviewed knowledgeable agency officials responsible for designing and implementing these selected initiatives. We asked these officials how they defined goals and selected specific strategies, how they designed and implemented their initiatives, and what steps they took to collect data and assess results. These interviews allowed us to capture detailed illustrations showing how agencies took actions that reflect aspects of effective practices in the implementation of their initiatives. The scope of this review was to identify practices for the effective implementation of open innovation initiatives, and to describe actions agencies took in carrying out open innovation initiatives that reflect aspects of those practices. While we present information on the implementation of agency open innovation initiatives, we did not assess the success of the underlying agency programs and activities that these initiatives were designed to support. For example, while we examined the implementation of DOT’s open dialogues on freight transportation, we have ongoing work reviewing various DOT activities related to issues mentioned in the draft National Freight Strategic Plan and have not evaluated the plan nor determined its effectiveness in helping DOT meet its freight goals. We conducted this performance audit from July 2015 to October 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Based on our review of agency open government plans and other sources, we found that agencies have frequently used the five open innovation strategies below to collaborate with citizens and external parties, and encourage their participation in agency efforts. Crowdsourcing and Citizen Science. In crowdsourcing, agencies submit an open call, generally through the Internet, for voluntary assistance from a large group of individuals to complete defined tasks. This can help the agency complete projects, such as transcribing large numbers of historical documents, while also producing usable products that benefit the broader community, like searchable databases. Similarly, agencies can use citizen science to encourage members of the public to voluntarily assist with science- related tasks. Such tasks can include conducting experiments, making observations, collecting and analyzing data, and interpreting results. This can supplement an agency’s own data collection efforts. It also allows agencies to study complex issues by conducting research at large geographic scales and over long periods of time in ways that professional scientists working alone cannot easily duplicate. Idea Generation (Ideation). In idea generation, or ideation, an agency asks participants to submit ideas to address a specific issue or problem, and may allow them to provide comments on ideas submitted by other participants, and vote to express their support for an idea. Open Data Collaboration. In open data collaboration, an agency mobilizes participants to share, explore and analyze publicly-available data sets. Examples of open data collaboration may include using open data to conduct research, design data visualizations, or create web and mobile applications and websites that help people access and use the data. Participants can also be mobilized through in- person or online events, often referred to as “data jams” or “hackathons,” or through websites that provide access to open data and facilitate ongoing communication. Open Dialogue. In an open dialogue, an agency collects and responds to information, observations, and perspectives provided by a range of citizens and other external experts and stakeholders. They can do this using online tools, including websites or interactive webinars, and in-person meetings or forums. The agency can also use open dialogues to request input and suggestions on a set of options under consideration, and to better understand the values, perspectives, and preferences of citizens and stakeholders. Prize Competition or Challenge. When an agency identifies a problem to solve or a specific goal it wants to achieve with the assistance of members of the public, it can hold a prize competition or challenge. In a competition or challenge, the agency invites interested members of the public to submit potential solutions to this problem or challenge. The agency then evaluates these proposals and provides a monetary or non-monetary award for those that meet specific criteria and are selected as winners. In addition to the contact named above, Benjamin T. Licht (Assistant Director) and Adam Miles supervised the development of this report. Theodore Alexander, Joyce Y. Kang, Steven Putansu, Lauren Shaman, Erik Shive, Wesley Sholtes, and Andrew J. Stephens made significant contributions to this report. Sarah Gilliland, Robert Robinson and Stewart Small also made key contributions. Shea Bader, Giny Cheong, Jeffrey DeMarco, Alexandra Edwards, Anthony Patterson, and Timothy Shaw verified the information in the report. Ballentyne, Perrie. Challenge Prizes: A Practice Guide. United Kingdom: Nesta, 2014. Brabham, Daren C. Crowdsourcing in the Public Sector. Georgetown Digital Shorts. Washington, D.C.: Georgetown University Press, 2015. Brabham, Daren C. Using Crowdsourcing in Government. Washington, D.C.: IBM Center for the Business of Government Collaborating across Boundaries Series, 2013. Department of Health and Human Services IDEA Lab. The HHS COMPETES Playbook, accessed on January 12, 2016, http://www.hhs.gov/idealab/what-we-do/hhs-competes. Desouza, Kevin. Challenge.gov: Using Competitions and Awards to Spur Innovation. Washington, D.C.: IBM Center for the Business of Government Using Technology Series, 2012. Eggers, William D. and Paul MacMillan. “A Billion to One: The Crowd Gets Personal.” United Kingdom: Deloitte Review issue 16 (2015). Federal Public Participation Working Group. U.S. Public Participation Playbook, 2015, accessed on July 27, 2015, https://participation.usa.gov. Goldhammer, Jesse, Kwasi Mitchell, Anesa “Nes” Parker, Brad Anderson, and Sahil Joshi. “The Craft of Incentive Prize Design: Lessons from the Public Sector.” Deloitte University Press, June 2014. Kannan, P. K. and Ai-Mei Chang. Beyond Citizen Engagement: Involving the Public in Co-Delivering Government Services. Washington, D.C.: IBM Center for the Business of Government Collaborating across Boundaries Series, 2013. King, Andrew and Karim R. Lakhani. “Using Open Innovation to Identify the Best Ideas.” MIT Sloan Management Review, September 11, 2013. Lee, Gwanhoo. Federal Ideation Programs: Challenges and Best Practices. Washington, D.C.: IBM Center for the Business of Government Using Technology Series, 2013. Lee, Gwanhoo and Young Hoon Kwak. An Open Government Implementation Model: Moving to Increased Public Engagement. Washington, D.C.: IBM Center for the Business of Government Using Technology Series, 2011. Luciano, Kay. Managing Innovation Prizes in Government. Washington, D.C.: BM Center for the Business of Government Collaborating across Boundaries Series, 2011. Lukensmeyer, Carolyn J., Joe Goldman, and David Stern. Assessing Public Participation in an Open Government Era: A Review of Federal Agency Plans. Washington, D.C.: IBM Center for the Business of Government Fostering Transparency and Democracy Series, 2011. McKinsey & Company. “And the Winner is…” Capturing the Promise of Philanthropic Prizes. McKinsey & Company, July 2009. Mergel, Ines. “Opening Government: Designing Open Innovation Processes to Collaborate with External Problem Solvers.” Social Science Computer Review vol. 33, no. 5 (2015): 599-612. Mergel, Ines and Kevin Desouza. “Implementing Open Innovation in the Public Sector: The Case of Challenge.gov.” Public Administration Review vol. 73, no. 6 (November/December 2013): 882–890. Nabatchi, Tina and Matt Leighninger. “Participation Scenarios and Tactics,” Public Participation for 21st Century Democracy. Hoboken, NJ: 2015, 241–285. Nambisan, Satish. Transforming Government through Collaborative Innovation. Washington, D.C.: IBM Center for the Business of Government Innovation Series, 2008. Nambisan, Satish and Priya Nambisan. Engaging Citizens in Co-Creation in Public Services: Lessons Learned and Best Practices. Washington, D.C.: IBM Center for the Business of Government Collaborating across Boundaries Series, 2013. Noveck, Beth Simone. Smart Citizens, Smarter State: The Technologies of Expertise and the Future of Governing. Cambridge, MA: Harvard University Press, 2015. Office of Management and Budget, Executive Office of the President of the United States. The Common Approach to Federal Enterprise Architecture. Washington, D.C.: May 2, 2012. Office of Science and Technology Policy, General Services Administration, and Federal Crowdsourcing and Citizen Science Community of Practice. “Federal Crowdsourcing and Citizen Science Toolkit,” adaptation of Bonney et al., “Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy.” BioScience 59(11), 977-984 (2009), accessed on January 26, 2016. https://crowdsourcing-toolkit.sites.usa.gov/howto. Tong, Raymond and Karim R. Lakhani. Public-Private Partnerships for Organizing and Executing Prize-Based Competitions, Research Publication no. 2012-13. Cambridge, MA: The Berkman Center for Internet and Society at Harvard University, June 2012.

Summary: To address the complex and crosscutting challenges facing the federal government, agencies need to effectively engage and collaborate with those in the private, nonprofit, and academic sectors, other levels of government, and citizens. Agencies are increasingly using open innovation strategies for these purposes. The GPRA Modernization Act of 2010 (GPRAMA) requires federal agencies to identify strategies and resources they will use to achieve their goals. GPRAMA also requires GAO to periodically review how implementation of its requirements is affecting agency performance. This report identifies and illustrates practices that help agencies effectively implement open innovation strategies, and how the use of those strategies has affected agency performance and opportunities for citizen engagement. To identify these practices, GAO analyzed relevant federal guidance and academic literature, and interviewed open innovation experts. To refine and illustrate the practices, GAO reviewed documents and interviewed officials from the Office of Management and Budget, Office of Science and Technology Policy, General Services Administration, and six selected federal agencies. GAO selected the agencies and a sample of their initiatives based on several factors, including the number and type of initiatives outlined in their Open Government Plans. Open innovation involves using various tools and approaches to harness the ideas, expertise, and resources of those outside an organization to address an issue or achieve specific goals. GAO found that federal agencies have frequently used five open innovation strategies to collaborate with citizens and external stakeholders, and encourage their participation in agency initiatives. GAO identified seven practices that agencies can use to effectively implement initiatives that involve the use of these strategies: Select the strategy appropriate for the purpose of engaging the public and the agency’s capabilities. Clearly define specific goals and performance measures for the initiative. Identify and engage external stakeholders and potential partners. Develop plans for implementing the initiative and recruiting participants. Engage participants and partners while implementing the initiative. Collect and assess relevant data and report results. Sustain communities of interested partners and participants. Aspects of these practices are illustrated by the 15 open innovation initiatives GAO reviewed at six selected agencies: the Departments of Energy, Health and Human Services, Housing and Urban Development, and Transportation (DOT); the Environmental Protection Agency; and the National Aeronautics and Space Administration (NASA). For example: With the Asteroid Data Hunter challenge, NASA used a challenge and citizen science effort, beginning in 2014, to improve the accuracy of its asteroid detection program and develop an application for citizen scientists. Since 2009, DOT’s Federal Highway Administration has used an ideation initiative called Every Day Counts to identify innovations to improve highway project delivery. Teams of federal, state, local, and industry experts then implement the ideas chosen through this process.