The focus of patent examination is to determine whether the invention in a patent application satisfies the legal requirements for a patent, including that the invention be novel and not obvious. As shown in figure 1, USPTO’s patent examination process involves a variety of steps, at least one of which includes prior art searches that examiners use to determine whether an invention is novel and not obvious. USPTO’s manual for patent examiners establishes certain requirements and processes that examiners must follow in examining patent applications, including performing prior art searches. There are generally two types of prior art—patent literature or nonpatent literature. Patent literature consists of previously issued U.S. or foreign patents and published patent applications. Nonpatent literature consists of other publicly available documents and can include such things as product manuals, standards established by international organizations, textbooks, periodicals, or conference presentations. Both patent and nonpatent literature may be written in a language other than English, referred to as foreign-language art in this report. USPTO’s manual for patent examiners requires them to conduct a thorough prior art search and directs them to consider U.S. patents, foreign patents, and nonpatent literature unless they can justify with reasonable certainty that no more pertinent prior art references can be found. Examiners are expected to complete their examination of an application in a certain number of hours. The time allotted varies depending on factors such as the technology and the seniority of the examiner. For example, an examiner reviewing an application related to wire fabrics and structure may be allotted about 14 hours for examination, while an examiner at the same experience level would be allotted about 32 hours for an application related to data processing, such as database and file management. The allotted time includes the time needed to review the application, perform a search for prior art, and complete all office actions. Examiners have minimum production goals, based on the time allotted, for the number of office actions they must complete, and examiners may earn bonuses for exceeding these minimum production goals. USPTO uses several different information technology systems to assist examiners in conducting prior art searches. For example, to search U.S. patent literature, examiners use two systems, the Examiner’s Automated Search Tool (EAST) and the Web-Based Examiner Search Tool (WEST), to search the full texts of published patent applications since 2001, patents granted since 1970, and optically scanned U.S. patents granted from 1920 through 1970. These systems also include abstracts of some foreign patents. Examiners may access additional foreign patent documents through other web-based tools. In addition, USPTO’s Scientific and Technical Information Center (STIC) operates systems that can search U.S. patent literature and that store some nonpatent literature. STIC also provides subscriptions to various web-based sources of nonpatent literature. According to a USPTO document, the agency had subscriptions to 119 different journals or external databases in 2014. USPTO’s current search tools do not provide examiners with immediate access to computer-generated translations, known as machine translations; however, examiners can request human and machine translation services from STIC and three contracted translation vendors that currently cover 35 languages. As of May 2015, USPTO had nearly 8,300 patent examiners across the eight technology centers we reviewed (see fig. 2). USPTO uses the General Schedule (GS) classification system for patent examiners, whose levels range from GS-5 to GS-15. Examiners at the GS-14 level or above (44 percent of the examiners in the technology centers we reviewed, as shown in fig. 3) are generally primary examiners. Primary examiners may accept or reject a patent application without additional review. This level of authority is in contrast to junior examiners—most examiners below the GS-14 level—whose work must first be reviewed by a primary examiner before an office action can be sent to the applicant. At the GS-13 level, some examiners are in the process of becoming primary examiners. Supervisory patent examiners are at the GS-15 level and are responsible for the day-to-day management of examiners. In addition to applying for a patent from USPTO, an inventor may also seek patent protection in other countries for the same invention by filing in multiple patent offices. Such interrelated patent applications are described as a patent family. The World Intellectual Property Organization has estimated that approximately half of all applications worldwide are repetitive filings in a patent family, and the rest are initial filings. According to the World Intellectual Property Organization, around 2.7 million patent applications were filed worldwide in 2014, of which 2.2 million applications were filed with patent offices in the United States, China, Japan, South Korea, and Europe—known as the IP5. These five offices, including EPO and JPO, each receive hundreds of thousands of patent applications each year. EPO issues patents that cover 42 countries, most of which are member countries that are party to the European Patent Convention. Applicants may apply to national patent offices or apply through EPO for coverage in some or all of these 42 countries. EPO has three official languages, one of which must be used for processing an application. These three languages are English (about 80 percent of applications), French (about 5 percent), and German (about 15 percent), according to EPO officials. EPO and JPO have about one- half and one-fifth as many examiners as USPTO, respectively. According to JPO officials, there is a limit on the number of federal employees JPO can have; therefore, 494 of its 1,702 examiners are fixed-term rather than permanent employees, and JPO has begun outsourcing aspects of prior art search. JPO officials stated that outsourcing some aspects of prior art searches based on instructions from a government examiner frees up the examiners’ time and allows the office to review more applications. Table 1 describes the workload and workforce of USPTO, EPO, and JPO. USPTO examiners face a variety of challenges in identifying relevant prior art during patent examination. As shown in figure 4, the experts we interviewed and examiners we surveyed cited challenges related to certain attributes of prior art and patent applications and USPTO examination policies, search tools, and human capital management. Our survey results show the extent of the challenges may vary by technology center or examiners’ GS level (see app. III). Because we surveyed a generalizable stratified random sample of examiners and adjusted for nonresponse, our results provide estimates for the entire population of examiners in our study and, when reported by technology center, for each of the technology centers we reviewed. Several attributes of prior art and patent applications present challenges for examiners in identifying relevant prior art, including the quantity of prior art, amount and relevance of prior art cited by applicants, availability of prior art, and clarity of patent applications. Quantity of prior art. The large volume of prior art available from multiple sources makes searching for relevant prior art challenging, according to most experts we interviewed as well as examiners responding to our survey. For example, one expert noted that the amount of patents, publications, and other nonpatent literature has grown exponentially, making it harder to find relevant prior art in the time allotted. Another expert said that technological innovations are occurring at a tremendous rate, and that the growing volume of prior art domestically and worldwide can be overwhelming. Based on our survey, we estimate that 45 percent of all examiners in the eight technology centers we reviewed find that the large quantity of art makes it somewhat or much more difficult to complete a thorough prior art search in the time allotted, while fewer examiners—34 percent—find that the quantity makes it somewhat or much easier. However, responses varied among technology centers, as shown in appendix III. For example, 30 percent of examiners in the Computer Networks, Multiplex Communication, Video Distribution, and Security technology center find that the quantity of art makes it somewhat or much more difficult to complete a thorough prior art search in the time allotted, compared to 60 percent of examiners in the Mechanical Engineering, Manufacturing, and Products technology center and 60 percent of examiners in the Chemical and Materials Engineering technology center. Amount and relevance of prior art cited by applicants. Examiners we surveyed reported difficulties with the amount and relevance of prior art references provided by applicants. USPTO requires applicants and others assisting in filing an application to submit all information known to be material to patentability. This information may include search results from foreign patent offices or publications known to the individual. According to USPTO policy, examiners will consider this information when reviewing a patent application, which may require reviewing numerous prior art references submitted by the applicant. Based on our survey, we estimate that 82 percent of examiners sometimes, often, or always encountered applications with what they considered an excessive number of submitted art references in the past quarter. We estimate that for most examiners (64 percent), excessive references make it somewhat or much more difficult to complete a thorough prior art search in the time allotted. Considering all of the prior art references submitted by applicants can be particularly challenging for examiners because applicants are generally not required to explain the relevance of the references or to point examiners to the particular portions of references that are relevant. In commenting on this issue in our survey, one examiner recalled often receiving information disclosure statements from applicants with numerous prior art references, of which only a handful were relevant. Based on our survey, we estimate that 88 percent of examiners sometimes, often, or always encountered applications with irrelevant references in the past quarter. Moreover, most examiners (57 percent) find that irrelevant references make it somewhat or much more difficult to complete a thorough prior art search in the time allotted. In contrast, 87 percent of examiners find that an application with relevant references makes it somewhat or much easier to complete thorough prior art searches in the time allotted. Availability of prior art. The availability of prior art and difficulties obtaining certain types of prior art are also challenges, according to most experts we interviewed as well as examiners we surveyed. For example, some relevant prior art may require a fee to access, may not be in a text- searchable format, may not be in a database, or may otherwise be difficult to access. In particular, experts told us that certain general types of prior art are more difficult to find, such as nonpatent literature overall and software-related prior art (most experts) and foreign-language prior art (6 of the 18 experts). Specific types of nonpatent literature that experts identified as difficult to find include product documentation and summaries, product databases, and user manuals; offers for sale and public use; and information from standards-setting organizations. Prior art related to computer technologies has also been difficult to find, according to one expert we interviewed, in part because a lot of information is commonly known in this field but may not be found by searching a public database. In addition, one expert we interviewed suggested that examiners do not have ready access to textbooks that are a good source of prior art. Another suggested that prior art from before the mid-1970s is difficult to find because patents issued before then have not been fully digitized. This expert stated that this is a particularly challenging issue for examiners in the mechanical technology centers because those examiners tend to use older art more often than other examiners. USPTO examiners we surveyed also reported difficulties obtaining relevant prior art from searches for certain types of prior art more than other types. In particular, on the basis of our survey, we estimate that 51 percent of all examiners find it somewhat or very difficult to obtain relevant art from searches for foreign-language nonpatent literature. Difficulties obtaining certain types of prior art may influence how often patent examiners search for them. For example, 8 of the 18 experts we interviewed suggested that examiners focus on searching patent literature and may not thoroughly search nonpatent literature. Similarly, our survey results in table 2 show that nearly all examiners always or often search for U.S. patents and applications (an estimated 99 percent); we also found that nearly all examiners always or often view this as the most relevant type of art they consider (an estimated 98 percent of examiners). In contrast, we estimate that 67 percent of examiners always or often search for foreign patents, and 20 percent of examiners always or often search for foreign-language nonpatent literature. More examiners also find that foreign patents and foreign-language nonpatent literature are difficult to obtain, compared to those that find U.S. patent literature difficult to obtain. In analyzing our survey results, we found that the difficulty examiners ascribed to finding foreign patent literature and foreign-language nonpatent literature was statistically associated with how often they reported searching for these types of prior art. How often examiners search for certain types of prior art, and how difficult examiners find those searches, also varies by technology center (see app. III). Clarity of patent applications. According to most of the experts we interviewed and examiners we surveyed, the clarity of applications can pose a challenge to finding relevant prior art. For example, as one expert noted, there often are no standard terms to describe technologies, and different applications may use different terms to describe the same thing. According to four of the experts we interviewed, the absence of standard terms is particularly a challenge for software-related applications. Inconsistent terminology can make it more difficult for examiners to find relevant prior art because searching for one term using keyword searches—one common method of searching for prior art described by USPTO officials—will not identify documents that use a different term. As shown in table 3, examiners reported that issues with the clarity of the application make it more difficult to complete thorough prior art searches in the time allotted. In addition, based on comments examiners made in our survey, examiners may face difficulties associated with applications that have been translated from a foreign language. For example, one examiner stated that “translation quality is often poor, and claims routinely contain non-standard industry terms. Issued patents and publications containing these non-standard terms also make searching in foreign collections exceptionally challenging because it is not possible to anticipate which synonyms to use.” In our patent quality report (GAO-16-490), we provide additional information on how the clarity of applications affects patent quality. The effect of unclear applications may be exacerbated by USPTO’s practice that examiners should attempt to identify all applicable grounds for rejecting a claim or claims during their first review—a practice called compact prosecution. According to USPTO’s manual for patent examiners, this practice aims to avoid unnecessary delays. Such delays could be caused by examiners waiting to continue examining an application until previously identified issues are resolved. However, the practice of compact prosecution may discourage examiners from resolving any issues of clarity or ambiguity before conducting their initial searches for relevant prior art. For example, as one examiner commented in our survey, in order to follow the compact prosecution practice, an examiner must guess what unclear claims mean in order to search for prior art related to the claims. If examiners do not have a clear understanding of the scope and claims of the invention, they may not choose the most appropriate keywords for conducting their prior art searches, which may lead examiners to miss relevant prior art that could be found with more relevant keywords. Several aspects of USPTO’s patent examination policies, prior art search tools, and human capital management present challenges for examiners in identifying relevant prior art, including the time pressures examiners experience for prior art searches, USPTO search tools and capabilities, the misclassification of patent applications, and examiners’ technical competence. Time pressures for prior art searches. According to most of the experts we interviewed and examiners we surveyed, time pressures may reduce examiners’ ability to conduct thorough prior art searches. These pressures relate to USPTO’s system for allotting an expected amount of time for examiners to complete an examination. For example, one expert noted that the amount of time examiners are allotted decreases as they become more experienced, and this may lead more senior examiners to increasingly rely on art they know well instead of searching for new art. As figure 5 shows, we estimate on the basis of our survey that 67 percent of examiners find they have somewhat or much less time than needed to complete thorough prior art searches given a typical workload. Our survey also found that examiners’ perception of the sufficiency of time for completing thorough prior art searches varies by technology center (see app. III). For example, an estimated 37 percent of examiners in the Mechanical Engineering, Manufacturing, and Products technology center reported having much less time than needed to complete a thorough prior art search, compared to 20 percent of examiners in the Biotechnology and Organic Chemistry technology center. In analyzing our survey results, we found that how often examiners searched for foreign patent literature, scientific articles or presentations, or foreign-language nonpatent literature was statistically associated with their description of the sufficiency of time they had to complete a thorough prior art search. Further, we asked examiners about overtime worked to meet their minimum production goals. A majority of examiners (an estimated 72 percent) worked voluntary/uncompensated overtime in the past 6 months to meet their goals, as shown in figure 6. An estimated 30 percent of examiners worked an average of more than 10 hours of voluntary/uncompensated overtime per biweekly period, although examiners’ overtime varied by GS level (see app. III). However, on the basis of our survey, an estimated 56 percent of examiners experience no pressure to work overtime. In a 2007 report on USPTO’s efforts to hire and retain an adequate workforce, we found that an estimated 70 percent of examiners worked voluntary/uncompensated overtime in the previous year. Our patent quality report (GAO-16-490) provides additional information on how time pressures may affect patent quality. Search tools and capabilities. Experts we interviewed and examiners we surveyed had mixed opinions on USPTO’s search tools and capabilities. Of the 18 experts, 10 agreed that the search tools and capabilities available to examiners create challenges to conducting thorough prior art searches. In general, these experts characterized USPTO’s search tools as being less advanced and more heavily reliant on keyword searching than other available tools. For example, one expert said that examiners’ electronic prior art searches predominantly rely on using keywords, and this is a problem if the examiners do not know the most appropriate keywords. Another expert suggested that USPTO needs more comprehensive databases of prior art. Based on our survey, examiners generally find that the search tools available to them from USPTO and from third parties make it easier to complete prior art searches, but they also find that other tools would help. Specifically, we estimate that a majority of examiners agree that certain search tools not currently available would make prior art searches somewhat or much easier, including a search engine that (1) can automatically search for concepts and synonyms related to the search terms entered by the examiner (an estimated 76 percent of examiners), and (2) automatically generates relevant art, without keyword entry, based on all the claims in an application (69 percent) or an application’s specification (69 percent). In addition, a group of four supervisory patent examiners we interviewed said that it would be more efficient to search for prior art in one tool, with a single search method that covered multiple sources of prior art, including nonpatent literature. Currently, relevant nonpatent literature may appear in different journals or databases that cannot be searched with a single search function. Similarly, examiners commented in our survey about the difficulty of searching nonpatent literature or requested an easier way to search for it. For example, examiners requested a method for searching nonpatent literature that is integrated with their systems for searching patent literature. In some cases, USPTO examiners may not be able to use public search engines because searching for specific terms or concepts from an unpublished application may put the confidentiality of the application at risk. Examiners also requested improvements to USPTO’s current translation capabilities. On the basis of our survey, a majority of examiners (an estimated 86 percent) agree that having access to immediate machine translation of foreign-language documents would make it somewhat or much easier to complete a thorough prior art search in the time allotted, while about half of examiners (an estimated 50 percent) find that additional translators would make it somewhat or much easier. Misclassification of patent applications. According to examiners, another challenge they face is misclassification of patent applications. Misclassification occurs when an application is not classified into the group most closely associated with the invention. This can result in applications being routed to the wrong USPTO technology center or art unit for examination. Although only 2 of the 18 experts we interviewed raised concerns about the misclassification of patent applications, on the basis of our survey, an estimated 75 percent of examiners encountered misclassified applications sometimes, often, or always in the past quarter, and an estimated 76 percent of examiners find that the misclassification of patent applications makes it somewhat or much more difficult to complete a thorough prior art search in the time allotted. According to our survey, how often examiners encountered misclassified applications in the past quarter varied by technology center (see app. III). Misclassified applications pose particular challenges for examiners, according to examiners’ comments to our survey. For example, while examiners receiving misclassified applications can request that an application be transferred to a different art unit, they may encounter difficulties doing so, according to examiners’ comments in our survey. Misclassified applications that are not transferred can negatively affect examiners’ confidence in their work and the quality of the examination, based on examiners’ comments in our survey. This is because when applications are routed to the wrong art areas, they may be reviewed by examiners who do not have appropriate knowledge and experience to understand the invention or relevant prior art and determine appropriate search strategies and terms. Additionally, misclassified applications that are never corrected could be difficult for examiners to subsequently use as prior art in later examinations. This is because misclassified applications may be difficult to find when examiners search for patent literature using technology-specific patent classification categories. Ensuring examiners’ technical competence. According to most experts we interviewed, ensuring that examiners have sufficient and appropriate technical backgrounds, knowledge, or skills for conducting thorough prior art searches is also a challenge. The role of patent examiner is a difficult one that can take years to learn, and examiners with less education or work experience, or who are not abreast of advances in a particular technology area, may not have the technical knowledge necessary to identify relevant prior art. According to USPTO officials, an examiner’s technical knowledge enables him or her to understand the invention being searched, and if the examiner does not understand the invention, he or she may not know what to search for, where to look, and when to stop searching. In addition, we estimate on the basis of our survey that 82 percent of examiners find it somewhat or much easier to complete a thorough prior art search in the time allotted for applications with a subject matter in which they have knowledge of existing prior art based on their education or previous work experience. As of May 2015, approximately 39 percent of all examiners in the technology centers we reviewed had been at the agency for less than 5 years, and USPTO has historically faced challenges in retaining examiners. In addition, as of September 2015, a majority of examiners in the technology centers we reviewed (61 percent) did not have a degree beyond a bachelor’s degree when hired. USPTO officials told us that the agency has aimed to match new hires’ previous work experiences and educational backgrounds to technology centers. When we asked USPTO how often examiners in each technology center have technical work experience or education relevant to their art unit, agency officials answered “always” for five centers, “often” for three centers, and “sometimes” for one center. However, our survey found that in the past quarter, less than half of examiners—an estimated 42 percent—always or often encountered applications with a subject matter in which they have knowledge of existing prior art based on their education or previous work experience. The training examiners receive—particularly continuing training in evolving technologies—may also affect their ability to maintain the technical competence they need to effectively identify relevant prior art. Overall, an estimated 60 percent of examiners found that the continuing education they received from USPTO in their art area in the past year was at least somewhat useful. However, 22 percent of examiners had not taken such training or had not been offered such training, with responses by technology center ranging from an estimated 10 percent to 34 percent of examiners (see app. III). We found in 2005 that examiners were reluctant to attend voluntary training given the time demands involved. Similarly, according to the group of five examiners we interviewed in 2015, USPTO did not offer enough ongoing technical training and did not always give examiners sufficient time to complete training. EPO and JPO use several approaches that may help their examiners address challenges in identifying prior art similar to those that experts and respondents to our survey of patent examiners cited. These approaches include worksharing, creating internal databases of nonpatent literature, using patent classification systems, adopting advanced search tools, hiring and training of examiners to promote technical expertise, and incorporating review and audit procedures. Worksharing. EPO and JPO have systems to share search and examination results. The exchange of prior art searches or patent examination results among patent offices—referred to as worksharing— facilitates more efficient prior art searches for applications filed in multiple countries, according to foreign officials. As we discussed above, approximately half of all patent applications worldwide are part of patent families filed in multiple offices. To gain patent protection in multiple countries, inventors may file applications for the same invention in several different offices. Because the underlying invention is the same, the search and examination results at one office may be useful to examiners in another office. Therefore, worksharing can ameliorate the challenges posed by the quantity of potential prior art that examiners must search and the time pressure examiners face by allowing patent offices to leverage others’ work and knowledge. The IP5, a multilateral forum of the five largest patent offices, identified worksharing as the main tool for addressing an increasing number of patent applications, while helping the patent offices conduct timely, quality examinations. JPO officials noted that worksharing systems allow more efficient access to foreign patent literature and help JPO examiners more easily see foreign examiners’ work. JPO examiners still have to review the relevant literature themselves, but access to foreign examination results may help JPO examiners identify relevant art or sources of prior art. An EPO official also noted that worksharing could give offices access to search reports of unpublished applications at other offices, and that these search reports may include prior art relevant to similar applications filed with EPO. Creating internal databases of nonpatent literature. EPO and JPO incorporate specific nonpatent literature resources into their offices’ main search tools, which may allow examiners to more efficiently search both patent and nonpatent literature by keyword, rather than searching multiple sources individually. Officials at these offices said that incorporating these resources into their main search tools allows examiners to consider a wide array of nonpatent literature sources. According to EPO officials, EPO’s primary search tool allows examiners to use a single interface to search through all of EPO’s internal databases as well as some external databases of prior art. In addition to creating an internal database of nonpatent literature documents to facilitate examiners’ searches, EPO also added descriptive metadata to index these documents to help examiners find the most relevant documents. EPO’s system also incorporates features that make it user-friendly—for example, grouping documents to avoid examiners facing dozens of copies of the same article from different sources. EPO officials stated that examiners find that these groupings enhance their ability to search through larger quantities of prior art. Overall, EPO officials identified their search engine and the large number of indexed prior art resources available within the search engine as strengths of the office in addressing the quantity of prior art available. EPO officials said that the office prefers to have internal collections of these resources so they may be integrated into its search tools, but some sources remain external and separate from EPO’s search systems. Because the cost of bringing sources into EPO’s search systems is high, the office needs a strong business case to justify the expense. According to these officials, publishers of academic journals and other nonpatent literature often raise subscription prices annually, so the office also must decide what to exclude, because there is simply too much nonpatent literature for the office to include everything. According to JPO officials, JPO has internal databases of both patent and nonpatent literature and also subscribes to external databases for nonpatent literature. JPO’s internal databases include JPO patent literature, some foreign patent literature, and selected nonpatent literature. JPO annually selects useful nonpatent literature to add to its database based on recommendations from an internal committee of examiners and officials. In addition, JPO uses commercial databases of nonpatent literature to augment its internal database and annually evaluates whether each external database is useful for prior art search. Using patent classification systems. EPO and JPO use classification systems to help examiners narrow their searches and find relevant patent literature even if the patent application uses inconsistent terms or is written in a foreign language. EPO and JPO classify applications and patents into groups that describe specific components of the invention. Examiners may then look at other applications and patents within the same group to find similar inventions. A system that has more groups— sometimes called classes—allows for finer distinctions, and prior art searches within a class will produce fewer, more focused results. Reducing the number of results an examiner needs to consider decreases the time needed for the prior art search, thereby improving efficiency. Furthermore, by grouping similar patents into classes that describe details of the inventions, classification systems allow examiners to conduct prior art searches based on the classes assigned to an application, independent of its language and wording. This can improve search quality by producing results that would not be found from a specific keyword search but are nonetheless closely related. According to EPO officials, this can be particularly helpful for finding documents in Asian languages that are otherwise difficult for EPO examiners to search, and it facilitates searches that are less dependent on the examiner’s choice of keywords. Adopting advanced search tools. EPO and JPO officials told us that examiners in both offices primarily use classification and keyword searches instead of more complex search tools. However, EPO uses some automated tools that assist in keyword-based searches or provide automated search results. One EPO tool provides recommended search terms in its three official languages to examiners, based on a database of examiners’ past searches. This tool can improve search efficiency by providing additional search terms, such as synonyms or translations, that the examiner may not have initially considered. Another EPO tool provides examiners with the results of an automated search based on the concepts and words drawn from the claims in applications. The results are in the form of a ranked list of potentially related documents, which provides a starting point for examiners before they begin a manual search. Hiring and training examiners to promote technical expertise. EPO and JPO hire examiners with technical expertise and provide several years of training, according to officials from these offices. EPO only recruits students with at least a master’s degree and almost all JPO examiners join the office after graduate school, according to EPO and JPO officials. In addition, JPO’s outsourced aspects of prior art searches are often carried out by retired engineers or technical experts, and according to JPO officials, their experience may enhance the quality of patent application examinations. At EPO, examiners must also meet a requirement that they understand English, German, and French. EPO examiners spend their first 2 years in training, alternating periods of classroom training with periods of on-the-job training. JPO examiners spend their first 2 to 4 years (depending on their academic degree and work experience) as assistant examiners and receive practical training by a supervising examiner. Thereafter, examiners in both offices have opportunities for additional technical training. According to EPO officials, EPO examiners need 3 to 4 years before they are fully trained, and 4 to 5 years in certain fields, such as biotechnology. At both offices, examiners also tend to spend their entire career with the office. JPO officials identified the longevity of examiners as a strength, stating that examiners are well trained and have experience that helps ensure examination consistency. Incorporating review and audit procedures. EPO and JPO have procedures to review examiners’ work before issuing or denying a patent. At EPO, applications are examined by a panel of three examiners: a first examiner, second examiner, and chair. According to EPO officials, the first examiner generally performs the bulk of the search and administrative tasks, with the other two examiners actively discussing and approving office actions. Examiners from a particular technology area are assigned to each panel randomly. An individual examiner may serve as first examiner on some applications and second examiner or chair on others. EPO’s quality management system also calls for random audits, including a minimum of two prior art search audits for each examiner per year, and chairs of the examination committees are required to record data on the quality of their examinations. JPO officials also reported taking steps to ensure that the patents it issues are of high quality. These steps include a director quality check, consultations with other examiners, and quality audits of a sample of examinations. According to JPO’s quality manual, JPO directors conduct a quality check on examiners’ decisions to grant or reject applications before the office actions are sent to the applicants. The manual also encourages examiners to consult with their directors or other examiners who may be able to provide guidance in examining an application. In 2014, examiners recorded approximately 83,000 consultations and received approximately 243,000 new applications, according to JPO officials. Experienced managers or examiners who serve as quality management officers also perform quality audits on randomly selected applications. According to JPO officials, the office has around 90 such officers in particular technology areas who review entire examinations and complete independent searches, and 4 wide-area officers who focus on the appropriateness of examiners’ decisions but do not conduct additional searches. According to these officials, while the number of audits is not large enough to allow tracking of individual examiner performance, it allows for statistical monitoring of examination quality overall. USPTO has taken or begun planning various actions that may help address challenges in identifying relevant prior art, but some of these actions have limitations that may hinder their effectiveness. USPTO’s actions span the following areas: (1) leveraging the work of foreign patent offices, (2) encouraging submission of prior art from third parties, (3) improving prior art search tools, (4) monitoring examiners’ prior art searches, (5) evaluating the agency’s system for determining the amount of time examiners are allotted to examine patent applications, and (6) strengthening the technical competence of the examiner corps. These actions may help address challenges related to prior art and applications and to examination policies, search tools, and human capital management. In some cases, these actions are coordinated with, similar to, or could be informed by approaches taken by EPO or JPO. USPTO has taken actions to leverage the work of foreign patent offices just as EPO and JPO have done, but the agency has experienced challenges working with EPO on a new patent classification system. USPTO is collaborating with foreign patent offices to give examiners access to examination files from those offices. Specifically, USPTO has (1) contributed data to information technology systems that share published information among patent offices; (2) engaged in pilot programs to collaborate with other offices during search and examination of related patent applications; and (3) adopted a new, joint classification system with EPO. These efforts may help address several of the challenges we identified relating to prior art and applications, including the large quantity of art available and the availability of prior art, particularly foreign patents and art written in foreign languages. First, USPTO contributes data to three worksharing systems that make applications, prior art citations, and examination results from foreign patent offices available to examiners and the public. One system, the Common Citation Document, provides bibliographic information, such as title and source, of patent and nonpatent prior art citations. USPTO, EPO, and JPO launched this system in 2011, and have since expanded it to include information from additional patent offices. Another system, Global Dossier, provides access to the examination history of applications, such as office actions and other correspondence. USPTO examiners gained access to Global Dossier in 2015. USPTO officials described Global Dossier as a system that allows examiners to quickly and easily view office actions from participating foreign patent offices. EPO officials described these two systems as complimentary, with each system providing advantages based on the specific search task involved. A third system, PATENTSCOPE, run by the World Intellectual Property Organization, provides access to applications filed with multiple offices under the Patent Cooperation Treaty as well as patents granted by several regional and national offices. By allowing examiners to review prior art that examiners from other offices found useful in examining patent applications, these systems may help address the challenges related to managing the large quantity of prior art and accessing prior art that would otherwise be difficult to obtain. Second, to enhance collaboration on related patent applications filed in multiple countries, USPTO has entered into two pilot programs to jointly examine certain applications: one with JPO and one with the Korean Intellectual Property Office. Applicants who file related patent applications with USPTO and either JPO or the Korean Intellectual Property Office may request to enter the pilot—known as the Collaborative Search Pilot Program—and receive expedited review of their applications. In the pilot with JPO, USPTO examiners consider prior art from both JPO’s search results and their own search results before responding to the applicant with the first office action. In the pilot with the Korean Intellectual Property Office, examiners perform independent searches and examinations, then compare results prior to final office actions. According to USPTO officials, these collaborative efforts could improve searches for patent and nonpatent literature in foreign languages, as examiners from multiple countries will perform searches in their native languages and share the results. Third, USPTO, in collaboration with EPO, designed and implemented a new, joint patent classification system meant to improve examiners’ ability to find relevant patent literature from the United States, Europe, or other offices that adopt the system. However, USPTO and patent examiners have experienced some challenges with this effort. As discussed earlier, by grouping similar patents, classification systems allow examiners to conduct prior art searches independent of an application’s language and wording and can improve search quality by producing results that would not be found from a specific keyword search. According to USPTO officials, the U.S. Patent Classification system became outdated following budget constraints in the early 2000s that undermined the agency’s efforts to update the classification system in response to technological change. After identifying a need to revise its classification system, USPTO partnered with EPO to create a new system to be jointly used by both offices. USPTO officially adopted the Cooperative Patent Classification (CPC) system in January 2015. According to USPTO officials, CPC gives examiners greater access to foreign patent literature compared to the prior system and can be expanded more easily to include new technologies. USPTO has found that classifications for some U.S. patent applications under the CPC system are inconsistent with EPO’s classifications, and USPTO examiners have reported difficulty with the system. Officials from both USPTO and EPO told us that there were disagreements between the two offices on how to classify some applications, such as on which categories to use or the number of categories to which applications are assigned. Additionally, USPTO examiners in certain technology centers reported difficulty with the CPC system in our survey. On the basis of survey responses from our random probability sample of examiners, we found that some examiners find that the CPC classifications make it somewhat or much more difficult to complete a thorough prior art search in the time allotted, as shown in table 4. For example, on the basis of our survey, CPC classifications make it somewhat or much more difficult to complete a thorough prior art search in the time allotted for an estimated 53 percent of the examiners in the Mechanical Engineering, Manufacturing, and Products technology center. On the other hand, an estimated 42 percent of examiners in the Computer Networks, Multiplex Communication, Video Distribution, and Security technology center find CPC classifications make it somewhat or much easier to complete a thorough prior art search in the time allotted. In response to an open- ended survey question, one examiner described their difficulty, saying that proper classification of prior art is essential to quality and efficiency of examination, that classification of prior art in the CPC system is inconsistent, and that many CPC categories still contain large amounts of prior art despite having more detailed technology categories. USPTO officials we interviewed also told us that the number of documents in some CPC categories was greater than those in comparable categories under USPTO’s previous classification system. According to USPTO officials, adopting the CPC system helps examiners give greater consideration to foreign patent documentation. As we note in this report, foreign patent and foreign-language nonpatent literature is sometimes difficult for examiners to find, according to the experts we interviewed and our survey of examiners. However, the CPC system’s usefulness to examiners depends, in part, on the consistency with which the system is applied by its users. Inconsistently applying the classification system to applications undermines its usefulness because similar applications may not be placed in the same category by USPTO and EPO. Also, misclassified applications may require examiners to familiarize themselves with new technologies, which increases the amount of time it takes for them to conduct their prior art searches and may negatively affect their confidence in the results. At USPTO, classification contractors apply an initial classification to applications before they are routed to an examiner. The examiner may revise the classification during examination. To help transition to the new system, USPTO provided examiners and classification contractors with training on how to classify applications and how to search for references using the CPC system, assigned lead quality experts to each technology center to provide training and assistance, and established a feedback tool for examiners to report issues with the new system. According to USPTO officials, the agency has not identified specific reasons that account for differences between its classifications and EPO’s. However, USPTO is updating its contract for initial classification services to require a higher level of expertise and is drafting a plan to assess performance under the new contract. The agreement between USPTO and EPO that supports the CPC system commits both offices to exchange best practices on classifications and aim for a high level of consistency, but it does not include a specific target or method for assessing consistency. USPTO officials told us that they are actively working with EPO to identify ways to measure and improve consistency, and that both offices have developed their own methods for measuring the consistency of CPC classifications. However, as of March 2016, EPO and USPTO had not agreed upon a measure of consistency between the offices. In the context of interagency collaborations, we have previously reported that agencies can enhance and sustain their collaborative efforts by, among other things, defining and articulating a common outcome and establishing mutually reinforcing or joint strategies. Establishing agreement with EPO on a target of consistency in classifying patent applications and a plan for monitoring consistency would help to implement this practice and would establish an internal control to support the reliability and usefulness of the CPC system. Since 2012, USPTO has had authority to receive submissions from third parties that may help identify potentially relevant prior art for published patent applications. This process, sometimes described as crowdsourcing, allows experts and other interested parties to share documents that they believe will aid USPTO’s examination of an application. Prior art submitted by third parties can help identify prior art that is not immediately available to examiners (e.g., because it is not available electronically or without payment) or that may be difficult to find. By potentially reducing the need to search for this prior art, these submissions may address challenges related to the quantity and availability of prior art. According to information from USPTO, third parties submitted prior art for less than 1 percent of the approximately 600,000 patent applications received per year since third-party submissions began in 2012. In addition, based on our survey, we estimate that 83 percent of examiners have rarely or never seen an application with a third-party submission. However, USPTO evaluations of third-party submissions found that they were generally useful to examiners and that the documents were sometimes not available to examiners through USPTO sources. According to USPTO officials we interviewed, an internal review of 300 randomly selected applications for which a third party submitted prior art found that examiners used a submission to reject a patent application in 20 to 25 percent of such cases. Furthermore, USPTO reviewed approximately 2,500 submissions of nonpatent literature prior art it received since September 2012, and found 753 unique, English- language prior art documents (30 percent of the submissions received) that were otherwise not available to examiners. These documents were mostly journal or other scholarly articles, but also included other types of nonpatent literature, such as marketing materials, book chapters, conference proceedings, presentations, and manuals that could not be found in the agency’s internal or external nonpatent literature sources. Because potential third-party submitters have to know about an application before they can submit potentially related art, USPTO has taken steps to allow third parties to more easily monitor published applications. The agency created a subscription service, whereby users will be notified when applications are published that contain user-chosen keywords. Beyond these efforts, USPTO also considered expanding use of third-party submissions of prior art by allowing examiners to use crowdsourcing to solicit such art for specific applications or topics. According to USPTO officials, the agency decided not to pursue this option, at least in part, because the agency is uncertain about its legal authority to do so. Specifically, while statute allows USPTO to collect third-party submissions, the law requires the agency to establish procedures to ensure that no protest or opposition is initiated during examination of a patent application. According to USPTO officials, it is unclear if requests for third-party submissions on a specific application could be viewed as allowing opposition. USPTO is making improvements to its prior art search tools that may help address some of the challenges examiners face in identifying relevant prior art, but USPTO has not developed a strategy to assess incorporating new sources of art into these tools over time. As part of its Enhanced Patent Quality Initiative, USPTO is procuring an automated prior art search capability that could enhance examiners’ ability to identify relevant prior art. In June 2015, USPTO requested information on a search system that uses the claims and specification within a patent application to search for patent and nonpatent literature automatically, without human involvement. In its request for information, USPTO described the intent of the system as providing a useful prior art baseline for patent examiners to begin their own searches. Such a system could improve the search tools available to examiners and help address the challenge of managing the quantity of potentially relevant prior art. USPTO anticipates that a pilot system will be available to a limited number of examiners, and agency officials told us that they anticipate awarding a contract in late summer 2016. The new system could expand upon the capabilities of a currently available system called the Patent Linguistic Utility Service. According to USPTO officials, the Patent Linguistic Utility Service has been in use for 20 years and has limitations that prevent it from meeting the agency’s search needs. For example, the Patent Linguistic Utility Service only searches U.S. patent literature and cannot perform the advanced search techniques contemplated for the new system. USPTO uses the current system to perform searches for about one-tenth of the agency’s incoming applications, whereas USPTO expects to use the new system for every application. Based on our survey, an estimated 12 percent of examiners find that the current system makes it somewhat or much easier to complete a thorough prior art search in the time allotted, whereas 52 percent of examiners expect that an automated pre-examination search would make it somewhat or much easier to complete a thorough prior art search in the time allotted. Moreover, USPTO’s 2014-2018 Strategic Plan includes an objective to ensure optimal information technology services, including upgrading search systems and prior art access. Toward this objective, USPTO is in the process of a major, multiyear $405 million effort to upgrade its information technology tools to provide examiners with a new system to manage all aspects of patent examination, including certain aspects of their prior art searches. The new system, called Patents End-to-End, will, according to USPTO’s Strategic Information Technology Plan for fiscal years 2015 through 2018, replace nearly 20 systems currently used to search patent applications. The new system will initially replicate the prior art search capabilities of USPTO’s current systems, such as EAST and WEST, which focus on U.S. patent literature and include only one source of nonpatent literature. Although searching for nonpatent literature is required by the agency’s manual for patent examiners, under the current and planned systems, examiners need to individually access and search a variety of external sources to look for nonpatent literature during their examinations. Consequently, neither the current nor planned systems provide USPTO examiners with the capability to efficiently search for prior art using a single, integrated search that includes both patent literature and multiple sources of nonpatent literature. The time it takes examiners to search the large and increasing volume of nonpatent literature and the inefficiency of having to search many different sources individually may lead examiners to conduct less thorough searches of nonpatent literature, potentially missing relevant prior art. As discussed earlier, our analysis of examiners’ survey responses and experts’ statements suggests that examiners are less likely to search for certain types of prior art, particularly foreign-language patents and nonpatent literature, from which it is more difficult to find relevant prior art. Integrating additional sources of prior art into USPTO’s search tools is one way USPTO could increase the types and sources of prior art that examiners consider, and would be similar to approaches that EPO and JPO cited as helping examiners consider a wide array of nonpatent literature sources. According to USPTO officials, the capabilities of the new Patents End-to-End system can be expanded in the future to include additional nonpatent literature sources. However, USPTO officials told us that, as of March 2016, the agency does not currently have specific plans to add additional nonpatent literature sources to its new system because of its initial focus on developing parity with the existing system. In addition, as of March 2016, USPTO had not established a documented strategy to identify and assess new sources in the future or the most optimal means of providing access to them. According to federal standards for internal control, control activities are the policies, procedures, techniques, and mechanisms that help ensure that actions are taken to address risks. These activities are an integral part of an agency’s planning to achieve effective results and efficiently manage government resources, including the development and maintenance of information systems. Because information technology changes rapidly, the internal control standards note that controls must evolve to remain effective. These standards also highlight the importance of clearly documenting internal controls. Without a documented, periodically updated strategy to evaluate new sources of prior art to include in the Patents End-to-End system and a process to periodically assess this strategy, USPTO will not have the assurance that it is taking full advantage of its information technology investment to help examiners more efficiently access a variety of resources for their prior art searches. USPTO is taking steps to strengthen monitoring of examiners’ work; however, these efforts may not provide USPTO with adequate data to identify and address shortcomings with examiners’ prior art searches specific to individual technology centers and monitor search thoroughness over time. USPTO uses two methods to review examinations, which may help the agency monitor the effects of the challenges described above as well as the thoroughness of searches. First, USPTO’s Office of Patent Quality Assurance (OPQA) conducts audits of a random sample of office actions, and some of these audits will review examiners’ prior art searches. Prior to recent changes in their audits, reviewers in OPQA performed about 400 audits per year that focused on assessments of examiners’ prior art searches. Second, supervisory patent examiners review examiners’ work products as part of each examiner’s annual performance appraisal. USPTO supervisory patent examiners are required to review at least four office actions of each of their primary examiners per year—with additional reviews for junior examiners—and to evaluate the thoroughness of examiners’ prior art searches during these reviews. However, in recent years, the number and consistency of OPQA staff and supervisory patent examiners’ reviews of examiners’ prior art searches have not been sufficient to examine trends in the thoroughness of prior art searches at the technology center or art unit level. Using the 400 audits OPQA performed annually, USPTO officials said that they could perform statistically valid assessments of prior art searches only for the examiner corps as a whole. Further, because the supervisory patent examiners’ reviews have not been conducted or documented in a consistent manner, USPTO could not examine trends at the technology center or art unit level by combining supervisory reviews with OPQA’s reviews. Specifically, past OPQA audits have evaluated prior art searches by, for example, considering if the examiner used reasonable search terms and synonyms, and may have included an independent search to discover prior art missed by the examiner. Auditors in OPQA documented the results of their reviews, and the office used them to evaluate the examiner corps from year to year. In comparison, according to a USPTO training document, supervisory patent examiners should check that a thorough search was conducted, but should also adjust their reviews based on an examiner’s skills, abilities, and performance history. Supervisory examiners do not need to perform an independent search for prior art. Supervisory examiners may document errors found during their reviews to inform individual employee performance assessments but are otherwise not required to record their reviews. Because USPTO does not require supervisory examiners to document these reviews in a consistent form, the agency cannot analyze the data to examine issues with prior art search quality in specific technology centers and art units. Early in fiscal year 2016, USPTO took two steps that if finalized could enable monitoring of prior art search trends at the technology center or art unit level, according to agency officials. Beginning in November 2015, OPQA made changes to its review processes that along with an increase in staff, should allow the office to perform about 12,000 audits in 2016, according to OPQA officials. In addition, in 2015, USPTO drafted a master review form that could standardize OPQA and supervisory examiner reviews with a single, consistent approach and documentation. OPQA began using a draft version of the form in November 2015, and USPTO began to pilot the form with some supervisory patent examiners in 2016. As of March 2016, USPTO had not made a decision about the final content of the review form, when supervisory patent examiners might begin to use the new form, or how the data from OPQA and supervisory examiner reviews would be used to assess examiners’ prior art searches. Furthermore, despite potential improvements in how OPQA and supervisory patent examiners conduct and document reviews of examiners’ prior art searches, USPTO’s ability to use these data to monitor prior art searches may be limited because USPTO (1) does not have a clear definition of what constitutes a thorough prior art search, (2) may not collect sufficient information to assess examiners’ search strategies or the sources of prior art they consider, and (3) has not established goals and indicators for improving prior art searches. USPTO requires examiners to perform a thorough prior art search and record their search results or search history, but the agency has not clearly defined what constitutes a thorough prior art search. USPTO’s manual for patent examiners requires examiners to conduct a thorough prior art search by identifying the field of search, the appropriate tools, and a search strategy. It also requires examiners to consider U.S. patents, foreign patents, and nonpatent literature, unless they can justify with reasonable certainty that no more pertinent references can be found. USPTO has no single method for assessing the thoroughness of prior art searches because, as supervisory patent examiners told us and our survey results show, examiners’ search strategies and the sources they use will differ based on the technology (see app. III). Examiners have access to guidance and training on classification-based searches, but USPTO has not documented technology-specific guidance—such as by technology center or art unit—on what constitutes a thorough prior art search. Specifically, in its definitions of CPC classes, USPTO provides suggested search areas of related classes, but these suggestions do not specify what USPTO would consider a thorough prior art search for each class or describe the sources of prior art the examiner should consider. As of March 2016, USPTO’s draft of the new master review form did not require OPQA or supervisory examiners to evaluate the thoroughness of an examiner’s search. Instead, the form asks if the examiner (1) searched for prior art associated with the inventor’s name, (2) searched for prior art using classification results for the application, and (3) recorded his or her search strategy. If a reviewer finds that the examiner should have made a rejection but did not, the form asks the reviewer to identify the source of prior art needed for the missed rejection. The March 2016 draft of the form does not include, as an October 2015 draft we reviewed did, questions assessing if search queries were likely to result in identification of relevant prior art. The March 2016 draft of the form also does not include questions that address whether the examiner searched foreign patent literature and U.S. and foreign nonpatent literature, as is required by the agency’s manual for patent examiners. While USPTO has taken steps to improve the quality of prior art searches, the agency has not yet established performance goals and indicators for improving prior art searches. One of the stated objectives in USPTO’s 2014-2018 Strategic Plan is to enhance accurate and consistent results in examination quality, and USPTO’s Enhanced Patent Quality Initiative also aims to achieve excellence in measuring patent quality. However, USPTO’s goals for patent quality in the strategic plan and for the quality initiative do not currently include goals or indicators assessing the thoroughness of prior art searches. Although USPTO is not required to establish goals or indicators for improving prior art searches, we have previously reported that establishing program goals and associated indicators constitutes a leading practice for planning within federal agencies. Among other things, the Government Performance and Results Act Modernization Act of 2010 requires that agencies (1) establish objective, quantifiable, and measurable performance goals and (2) establish performance indicators to measure progress toward each performance goal. The limitations to USPTO’s ability to monitor examiners’ prior art searches at a technology center or art unit level may hinder the agency’s ability to identify and address issues in the thoroughness of examiners’ prior art searches and quality of the agency’s examinations. Federal standards for internal control state that agencies’ monitoring of their internal controls should assess the quality of performance over time, and that internal controls should generally be designed to assure that ongoing monitoring occurs in the course of normal operations. Without consistently collecting the information needed to assess the thoroughness of prior art searches and monitoring at a technology center or art unit level, the agency cannot identify and address issues that are more prevalent in certain technology centers or art areas, such as variations in the extent to which examiners in certain areas search for foreign patents or nonpatent literature. Similarly, without greater clarity on what constitutes a thorough prior art search in different technologies, it will be difficult for USPTO to assess the adequacy of examiners’ searches, and examiners may vary in their thoroughness. Moreover, without goals and indicators for assessing the thoroughness of prior art searches, USPTO cannot reliably assess the thoroughness of its searches or improvement in searches over time. USPTO plans to evaluate changes to the agency’s system for determining the number of applications a patent examiner is expected to review within a specified period of time; however, the agency has not identified plans to assess the amount of time examiners in different technologies need to perform thorough prior art searches. According to USPTO officials we interviewed and a document we reviewed, the time allotted for each individual technology was determined when the examiner performance and production system was first created in the 1970s or when subsequent art units were added. USPTO adjusted the time allotted to examiners between fiscal years 2010 and 2012 and gave all patent examiners a total of 2.5 additional hours per application. However, according to USPTO officials, the agency did not evaluate art unit or technology-specific factors prior to making this change. USPTO also adjusted the time allotted in April 2016, when approximately 1,000 examiners received an additional 2.7 hours for examinations of certain technologies to address concerns related to the transition to the CPC system. According to technical comments USPTO provided on a draft of this report, these changes were based on an initial investigation of time needed to perform a thorough prior art search in these technologies. The time pressures created by USPTO’s system for determining allotted examination time have important implications for examiners’ ability to conduct thorough prior art searches. As noted above, about two-thirds of examiners responding to our survey reported having insufficient time to complete a thorough prior art search given their typical workload. Furthermore, examiners in different technologies may need different amounts of time to conduct thorough prior art searches. For example, we estimate based on our survey that a higher percentage of examiners in the Mechanical Engineering technology center find that they have much less time than needed, compared to examiners in other technology centers (see app. III). Supervisory patent examiners we interviewed told us that prior art searching is the most time-consuming aspect of patent examination. Based on our survey, an estimated 80 percent of examiners spent, on average, from 6 to 20 hours initially examining patent applications and completing a first office action. For the prior art search portion of these efforts, we estimate that examiners spent an average of 8 hours per application during the past quarter. If examiners feel pressure to complete their examinations quickly rather than thoroughly, it could have important implications for their prior art search efforts. In fact, as shown in figure 7, less than half of the examiners—an estimated 46 percent—were moderately or very confident that they found the most relevant prior art during the time allotted. However, when they included their additional voluntary overtime, the number of examiners who were moderately or very confident increased to an estimated 69 percent of examiners. Even with the inclusion of their overtime, though, an estimated 23 percent of examiners remained not at all confident to somewhat confident. Federal standards for internal control specify that agencies should assess the risks the agency faces, including identifying relevant risks associated with achieving the agency’s objectives, assessing a risk’s significance and likelihood of its occurrence, and deciding what actions should be taken to manage the risk. In addition, these standards note that an agency’s operational success requires providing personnel with the right incentives for the job. In USPTO’s 2014-2018 Strategic Plan, the agency indicated its intent to evaluate changes to the its system for evaluating if examiners are completing office actions in the time allotted and to make additional modifications as needed. In November 2015, USPTO’s Commissioner for Patents affirmed the agency’s intent to examine this system and told us that he had committed to the examiners’ union to do so. USPTO has historically faced a backlog of applications, and changing the time allotted may make it more difficult for the agency to address this backlog. However, changing the time allotted to examiners may address the challenge of time pressures on examiners that we discuss above. As of May 2016, USPTO had not clarified the extent to which this evaluation will specifically consider the time needed for a thorough prior art search for different technologies. Without evaluating the technology-specific time pressures examiners face in their efforts to identify prior art in patent examination, USPTO does not have assurance that the time allotted to examiners reflects the evolving complexity of different technologies and their associated prior art. USPTO has taken several actions related to the challenge of ensuring that the agency has an examiner workforce with the technical competence—backgrounds, knowledge, or skills—needed for identifying relevant prior art; however, USPTO does not have a process to assess and measure each technology center’s progress toward closing any gaps in examiners’ technical knowledge and skills. Effective management of an organization’s workforce—its human capital—is essential; in particular, identifying critical occupations, skills, and competencies and analyzing workforce gaps are leading principles in workforce planning, as we and the Office of Personnel Management have previously identified. Further, as described in the federal standards for internal control, all personnel need to possess and maintain a level of competence that allows them to accomplish their assigned duties. Management needs to ensure that skill needs are continually assessed and that the organization is able to obtain a workforce with the skills required to achieve organizational goals. Management also needs to provide employees the right training to develop and retain skill levels to meet changing organizational needs. Accordingly, USPTO developed a human capital strategy, its 2015-2018 People Plan, which replaced its 2011-2015 Strategic Human Capital Plan. The agency also identified the role of patent examiner as a mission- critical occupation and identified the competencies needed for this occupation. These competencies include technical work experience/education and technical competence, which we refer to collectively as technical competence. USPTO defines technical competence as the ability to analyze and interpret written technical materials, rules, regulations, instructions, and reports. According to USPTO officials, specific technical competencies vary depending on the technology examined by each technology center and art unit. USPTO has further documented specific technical competencies for examiners in a series of “job analysis worksheets,” which describe the knowledge, skills, and abilities needed for examiner positions. Agency officials said these were based on the hiring needs derived from a hiring model created by senior management. For example, the job analysis worksheet for electrical engineering patent examiners considers their education, training, and experience in researching, analyzing, and applying scientific principles in specified technical areas, such as image analysis, power systems, and computer architecture. To develop and maintain examiners’ technical competence, USPTO offers three programs, all of which are voluntary for examiners (see table 5). In addition, USPTO officials said that technology centers provide technology-specific training through paid vendors, periodic internal meetings to discuss technical and quality issues, and technology fairs. USPTO does not require examiners to complete a minimum amount of ongoing technical training; however, based on information from USPTO officials, 74 percent of examiners participated in at least one of the 153 technical training events held through the agency’s Patent Examiner Technical Training Program in fiscal year 2015, with an average of 6.7 training hours per participant. The extent to which examiners participated in this technical training program varied by technology center, from a low of 48 percent to a high of 96 percent in fiscal year 2015, as well as by art unit, according to information from USPTO officials. While these are important steps that can help develop and maintain examiners’ technical competence, USPTO officials told us that the agency has not conducted an analysis to identify any gaps in examiners’ competence, either for the agency as a whole or for each technology center. USPTO strategic or human capital plans since 2007 have called for measuring the agency’s performance on closing competency/skill gaps for mission-critical occupations. USPTO officials told us that the technology centers use performance appraisal plans to individually assess examiners’ skills and competency gaps. However, doing so does not address the question of whether broader competency gaps exist at the technology center level or how any gaps can be addressed. Further, using USPTO’s job analysis worksheets in hiring does not eliminate the need for ongoing workforce assessment. In particular, as technologies change, the knowledge and skills required of examiners may evolve accordingly, and examiners may move from one technology center or art unit to another during their careers. According to USPTO officials, there are also times when patent examiners are ultimately assigned to a different technology center upon completion of their initial training than the assignment given to them when they were hired. In addition, USPTO has reported that examiner attrition has increased over the past 5 years. Historically, attrition has been highest amongst examiners at USPTO for less than 5 years. As we found in 2007, attrition of less experienced examiners is a significant loss for the agency, in part because examiners require 4 to 6 years of on-the-job experience before they become fully proficient in conducting patent application reviews. When these staff leave USPTO, the agency loses as much as 5 years of training investment in them, and continuing turnover of many new patent examiners makes the overall workforce less experienced. Moreover, USPTO’s 2015-2018 People Plan notes that that the agency’s ability to retain experienced and high-performing employees may prove difficult with continued improvements to the economy. Because of these factors, the technical knowledge and skills needed in a technology center may differ from the knowledge and skills of the individual examiners assigned to that center. Without periodically evaluating gaps between the technical competence of examiners and the knowledge and skills needed in each technology center, USPTO cannot ensure that it has appropriate strategies, such as training or other efforts, to close any gaps. Moreover, as specified in the federal standards for internal control, ongoing internal control monitoring should assess the quality of performance over time. The agency’s 2015-2018 People Plan, however, does not include measures for monitoring progress in closing gaps. Without monitoring and evaluating progress toward closing any identified gaps in technical competence, USPTO may not have reasonable assurance that examiners in all technology centers have the skills and knowledge to identify relevant prior art during patent examination. Examiners face a number of challenges in their efforts to search for prior art, including the large volume of prior art from multiple sources to consider, unclear patent applications, difficulties identifying or accessing relevant nonpatent literature and prior art in foreign languages, and limits on the time available to search for relevant prior art, among others. Some of these challenges may reinforce one another or affect examiners in some technology centers more than others. USPTO, recognizing the importance of issuing quality patents, is taking a number of steps to improve patent quality, as we discuss in our patent quality report (GAO-16-490). USPTO is also is taking a number of steps to help address challenges to completing a thorough prior art search. However, while USPTO’s steps to address prior art search challenges are promising, opportunities exist for USPTO to address limitations that may hinder the effectiveness of some of its efforts. For example, USPTO has experienced some challenges working with EPO to adopt and consistently implement a new patent classification system. Because USPTO and EPO have not identified a target level of consistency or a plan to monitor consistency between USPTO contractor and EPO classifications—important aspects of interagency collaboration—the potential benefits of adopting the new classification system may be reduced as inconsistently classified patents make it more difficult for examiners to identify relevant U.S. and foreign patent literature. Similarly, USPTO is undertaking a major investment in information technology tools for examiners to manage their work and search for prior art. The new search system USPTO plans has the potential to integrate additional sources of nonpatent literature, an approach that EPO and JPO have taken and cited as helping examiners consider a wide array of nonpatent literature sources. However, USPTO has not developed and documented a strategy to identify and assess the optimal means of incorporating new sources of art into these tools. Such a strategy would be consistent with federal standards for internal control. Developing and periodically updating such a strategy could help USPTO take full advantage of its investment in its new information technology tools to address some of the challenges examiners face in identifying relevant prior art. USPTO is also taking steps to augment the number and consistency with which reviews of examiners’ prior art searches are conducted and documented that could help to address a weakness of its past efforts—an insufficient number of consistently performed reviews to enable USPTO to identify issues with examiners’ prior art searches within individual technology centers. However, USPTO still faces limitations in its ability to use the data to monitor the thoroughness of examiners’ prior art searches. First, USPTO has not clearly defined what constitutes a thorough prior art search for different technology centers, which is important given differences in the appropriate search strategies and prior art resources across technology areas. Second, USPTO’s draft review form, as of March 2016, does not include questions that are important to assess the thoroughness of examiners’ prior art searches, such as the extent to which an examiner searched for foreign patent and nonpatent literature. These are aspects of prior art search that are generally required by USPTO’s manual for patent examiners but that our survey results showed many examiners may not perform. Third, the agency has not established goals or indicators covering prior art search performance. Monitoring the thoroughness of examiners’ prior art searches at a level of rigor and consistency to assess trends within individual technology centers would better enable USPTO to identify issues that vary by technology center and develop appropriate responses as called for by federal internal control standards. Such monitoring would also allow USPTO to have greater assurance that examiners are searching all relevant sources of prior art, including foreign patents and nonpatent literature. Furthermore, establishing goals and indicators for prior art search would help USPTO reliably assess the thoroughness of its searches and improvements over time. Additionally, while monitoring the thoroughness of examiners’ prior art searches is important, it is essential that examiners have sufficient time to perform high-quality work. USPTO expects examiners to examine an application within a specific amount of time. This means that examiners may need to make trade-offs among the amount and types of prior art sources they consider because of the limited time available to perform their searches. However, as we also note in our patent quality report (GAO-16-490), USPTO has not recently assessed the time needed for examinations in different technologies. Without this information, the agency does not know if it is allotting examiners appropriate amounts of time to complete their work. Based on our survey, most examiners find that they do not have sufficient time for thorough prior art searches, and many were not confident in their ability to identify the most relevant prior art without working voluntary overtime. USPTO’s planned effort to evaluate the agency’s system for determining the time allotted for examination could help the agency better understand the time pressures examiners face; however, USPTO has not yet clarified the extent to which the evaluation will look at any differences among art units and technologies in the time needed for thorough prior art searches. Given the importance of thorough prior art searches to USPTO’s patent quality objectives, specifically assessing the time needed for a thorough prior search for different technologies would provide USPTO with greater assurance that it will address the risks posed by the time pressures examiners experience, consistent with federal internal control standards. Finally, examiners’ technical competence is an important factor in understanding patent applications and locating relevant prior art in the time allotted. USPTO has taken steps to identify the technical competencies of various examiner positions and to offer voluntary technical training opportunities to examiners. However, the agency has not conducted an overall analysis of the technical competence of its examiners to identify potential competency gaps for each technology center. Periodically conducting such analyses to reflect evolving technologies and workforce changes, as called for by federal standards for internal control, would give USPTO greater assurance that examiners in all technology centers have the technical skills and knowledge to identify the most relevant prior art during patent examination. Furthermore, analyzing the results of such analyses would help the agency define and prioritize its strategies for closing any competency gaps, such as through training or other efforts. Lastly, these steps, in conjunction with developing measures to monitor progress toward closing any gaps, would help USPTO address the effects of evolving technologies and workforce changes over time. To enhance USPTO’s ability to identify relevant prior art, we recommend that the Secretary of Commerce direct the Director of USPTO to take the following seven actions as the agency continues to implement its efforts to improve prior art searches. To ensure that USPTO’s collaborative efforts on classification help examiners find relevant prior art, USPTO should work with EPO to identify a target level of consistency of Cooperative Patent Classification decisions between USPTO and EPO and develop a plan to monitor consistency to achieve the target. To ensure that USPTO is able to take full advantage of its investment in new information technology tools and capabilities, USPTO should develop and periodically update a documented strategy to identify key sources of nonpatent literature for individual technology centers and to assess the optimal means of providing access to these sources, such as including them in USPTO’s search system. To improve its monitoring of prior art searches and provide USPTO the ability to examine and address trends in prior art search quality at the technology center level, USPTO should take the following three actions: Develop written guidance on what constitutes a thorough prior art search within each technology field (i.e., mechanical, chemical, electrical), technology center, art area, or art unit, as appropriate, and establish goals and indicators for improving prior art searches. Ensure that sufficient information is collected in reviews of prior art searches to assess the quality of searches at the technology center level, including how often examiners search for U.S. patents, foreign patents, and nonpatent literature. Use the audits and supervisory reviews to monitor the thoroughness of examiners’ prior art searches and improvements over time. To ensure that examiners have sufficient time to conduct a thorough prior art search, USPTO should, in conjunction with implementing the recommendation from our patent quality report to analyze the time examiners need to perform a thorough examination, specifically assess the time examiners need to conduct a thorough prior art search for different technologies. To ensure that examiners have the technical competence needed to complete thorough prior art searches, USPTO should assess whether the technical competencies of examiners in each technology center match those necessary; develop strategies to address any gaps identified, such as a technical training strategy; and establish measures to monitor progress toward closing any gaps. We provided a draft of this report to the Department of Commerce for its review and comment. In the Department of Commerce’s written comments, reproduced in appendix IV, the Department concurred with our recommendations. USPTO provided additional technical comments that we incorporated, as appropriate. We also provided a draft of this report to EPO and JPO for their views and comment. EPO indicated it had no comments on the draft, and JPO provided technical comments that we incorporated, as appropriate. In its written comments, the Department concurred with our recommendation to work with EPO to identify a target level of consistency for CPC decisions and develop a plan to monitor consistency. The Department noted ongoing efforts to work with EPO and other patent offices to develop objective metrics to measure the level of consistency. If finalized, such metrics and a plan to monitor consistency could allow USPTO to more fully benefit from adopting the new CPC system. In regard to our recommendation to develop and periodically update a strategy related to key sources of nonpatent literature, the Department concurred and stated that USPTO’s STIC analyzes nonpatent literature sources used by examiners and assesses them for incorporation into USPTO’s search system. While we did not receive evidence of such assessments during our review, they may be a useful step toward a strategy to periodically assess the optimal means of providing examiners access to key nonpatent literature sources and to ensure the effectiveness of USPTO’s planned Patents End-to-End search system. In regard to our recommendation to develop written guidance on what constitutes a thorough prior art search and establish goals and indicators for improving prior art searches, the Department concurred and said USPTO would develop technology-based search training guidance and establish enhanced goals and indicators for improving prior art searches. Specifically in regard to developing guidance on what constitutes thorough prior art searches, USPTO’s technical comments noted that our report did not recognize that USPTO’s definitions of CPC classification areas include suggestions for related classes for search. We adjusted our report to acknowledge the information USPTO provides on these suggested search areas. However, these suggested search areas do not fully establish what constitutes a thorough prior art search for each CPC class, and they do not provide guidance on examiners’ searches for nonpatent literature. In regard to our recommendations to ensure that reviews of examiners’ prior art searches collect sufficient information to assess search quality at the technology center level and use these reviews to monitor examiners’ prior art searches over time, the Department concurred. The Department said it would ensure adequate search data are collected to assess the quality of searches at the technology center level, and would investigate using audits and reviews to monitor the thoroughness of examiners’ prior art searches over time. In its written comments, the Department noted that a great deal of information on examiners’ prior art searches is potentially available from examiners’ records and reviews by primary examiners, supervisors, OPQA, and others. As we describe in our report, these reviews have not been carried out or recorded in a consistent manner in the past. Therefore, as a whole, these data do not provide USPTO with reliable information to assess trends in prior art search quality at the technology center level. USPTO’s recent effort to improve the consistency of these reviews is an important step to address this issue. However, additional steps are needed to ensure that USPTO’s reviews of examiners’ prior art searches collect information on whether, for example, examiners are searching patent literature, foreign patents, and nonpatent literature, as is required by the manual for patent examiners. Without collecting such information at the technology center level, USPTO may be challenged to use its reviews to monitor the thoroughness of examiners’ prior art searches and improvements over time. In regard to our recommendation on assessing the time examiners need to conduct a thorough prior art search for different technologies, the Department concurred and stated that USPTO intends to further investigate the time needed. The Department also mentioned the April 2016 changes to the time allotted for approximately 1,000 examiners, noting that they were based on an initial evaluation of the time examiners need to perform a thorough prior art search. This initial evaluation stemmed from feedback from examiners during the transition to CPC. We did not receive details of this evaluation during our review, but recognize that this action may have provided the agency with important information about the time needed for prior art search in some technologies. We continue to believe that the agency should review the time allotted for all technologies, and not only those identified through feedback related to CPC. Finally, the Department concurred with our recommendation to assess whether the technical competence of examiners in each technology center matches those necessary and develop strategies to address gaps if identified. As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Commerce, the Director of the USPTO, the Commissioner for Patents, and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-3841 or neumannj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. This report (1) describes the challenges examiners face in identifying relevant prior art, (2) describes how selected foreign patent offices have addressed challenges in identifying relevant prior art, and (3) assesses the extent to which the U.S. Patent and Trademark Office (USPTO) has taken steps to address any challenges in identifying relevant prior art. Our work for this report was coordinated with our report on patent quality. To describe the challenges examiners face in identifying relevant prior art, we reviewed relevant laws and USPTO documents and interviewed USPTO officials; a group of four supervisory patent examiners; officials of the examiners’ union, the Patent Office Professional Association; and a group of five examiners serving as union representatives. We also conducted semistructured interviews with subject matter experts (experts) active in the intellectual property field and knowledgeable about the subject of prior art. We selected and interviewed 18 experts to obtain views of different stakeholder communities, including four academic experts on the basis of a literature search we conducted for articles related to USPTO and prior art; four patent holders from the following technology fields: chemical technologies, electrical technologies, and mechanical technologies, based on data from USPTO that identified the companies receiving patents in these fields from 2010 through 2014; five attorneys based on their leadership (e.g., as chairs or co-chairs) of patent-related committees or sections of various bar associations; representatives of three nongovernmental organizations we identified during background research that had activities or publications related to patents; and two patent data experts who were listed in a 2015 USPTO analysis evaluating search technologies. In addition, we conducted a web-based survey of a stratified random sample of 3,336 eligible USPTO patent examiners from across 8 of the 11 technology-based subject matter groups (referred to as technology centers) into which USPTO examiners are divided. Fielded between August and November 2015, the survey was designed to collect information on challenges USPTO faces in finding relevant prior art and how USPTO might improve its prior art search capabilities. To identify our survey population, we obtained from USPTO a list of patent examiners as of May 2015. We excluded examiners from the following 3 technology centers: We excluded the Designs technology center because these examiners work on design patents instead of utility patents; design patents are outside the scope of this engagement and have different statutory and administrative requirements than utility patents. We excluded examiners who perform “reexamination” work and not initial patent examinations. We excluded examiners in the patent training academy because these examiners are recent hires who are in a 12-month training program. We also excluded examiners employed at USPTO for less than 1 year. We then defined nine strata by technology center, with one technology center separated into two strata, as described in table 6. Specifically, the Transportation, Construction, Electronic Commerce, Agriculture, National Security and License and Review technology center includes a diverse set of technologies, including transportation, construction, agriculture, and business methods. In our review, we separated the art units—subunits of a technology center—focused on electronic commerce and business methods (collectively referred to as business methods) in light of recent legislation and court decisions related to business methods. This resulted in nine strata with a target survey population totaling 7,825 eligible examiners. From this list, we drew our stratified random sample of 3,336 eligible USPTO patent examiners. We received responses from 2,669 eligible examiners for an 80 percent response rate. Because we used a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. Since each sample could have provided different estimates, we quantified the sampling error and express our confidence in the precision of our particular sample’s results at a 95 percent confidence interval. This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. Within each stratum and overall, the confidence intervals for survey results for percentages are generally within +/- 5 percentage points. The only estimates for which the confidence intervals exceed 5 percentage points are certain results for the business methods stratum. In these instances, the confidence intervals are from 5 to 6 percentage points. In this report, our figures containing survey results show the upper and lower bounds for estimates at the 95 percent confidence interval. For other estimates in the report, we have not provided the upper and lower bounds in the text or tables; however, those details for all survey results are available in the e- supplement related to this report. The quality of survey data can also be affected by nonsampling error, which includes, for example, variations in how respondents interpret questions, respondents’ willingness to offer accurate responses, nonresponse errors, and data collection and processing errors. To minimize nonsampling error, we took several steps in developing the survey and in collecting and analyzing survey data. Specifically, in developing the survey, we worked with our survey professionals, among other things, to draft questions that were clear and unbiased. We pretested the survey in person with five USPTO staff: three examiners who are also representatives to the examiners’ union, a supervisory patent examiner, and a quality assurance specialist. We used these pretests to check that the questions were clear and unambiguous, used correct terminology, requested information that could be feasibly obtained, and were comprehensive and unbiased. We also obtained comments on the survey from USPTO management and leadership from the examiners’ union. In addition, we obtained a quality review by a separate GAO survey methodologist. Based on these activities, we made changes to the survey before administering it. Further, using a web-based survey provided several advantages, including allowing examiners to enter their responses into an electronic instrument that created an automatic record for each respondent. This eliminated the potential for errors that could have resulted if we had used a manual process to enter respondents’ data from paper surveys. In addition, to account for the complex sample design, we used survey software in our analyses to produce appropriate estimates and confidence intervals, and the programs we used to process and analyze the survey data were independently verified to ensure the accuracy of this work. To minimize nonresponse error, we made a variety of contacts with the sample of examiners during the survey, including follow-up e-mails to encourage responses. In addition, from October 20 through 23, 2015, we attempted to follow up via telephone calls to all 1,102 examiners who had neither completed the survey nor told us that they were no longer examiners. We also analyzed nonresponse bias to (1) assess whether any factors were associated with examiners’ propensity to respond and (2) allow our analysis of respondents to properly reflect the sampling universe of eligible examiners. To adjust the sampling weight for potential nonresponse bias, we used standard weighting class adjustments based on the sampling strata and the examiners’ years of experience at USPTO. The weighted response rate was also 80 percent. In this report and in the related e-supplement, we present the survey results using the nonresponse adjusted weights, which are generalizable to the eligible population of examiners. We analyzed the responses to the survey for all examiners, as well as responses by technology center and by the General Schedule (GS) level of the examiners. We selected three categories of GS levels—less than GS-13, GS-13, and greater than GS-13—because examiners at these levels have different responsibilities and authorities when examining patent applications. Specifically, examiners at the GS-14 level or above generally may grant a patent or reject a patent application without additional review, most examiners below the GS-14 level must have their actions reviewed by a more senior examiner, and some GS-13 examiners are in the process of becoming GS-14 examiners. Question 3 in the survey asked examiners approximately how many hours they spent per application on prior art searches in the past quarter, when preparing the original First Action on the Merits, the means by which examiners initially notify applicants about the patentability of their inventions. Examiners provided a variety of open-ended responses. Some respondents chose to provide a range, such as “5 to 10,” while others provided a single number for hours spent per application on prior art searches. Some examiners also provided responses that did not clearly indicate the approximate hours per application spent on prior art searches in the past quarter; we excluded those responses. Where possible, we coded responses to reflect a range of numbers by assigning a low and a high value; when a single number was provided, we coded that number as both the low and the high value. A second analyst verified the initial analyst’s coding. We checked the sensitivity of results overall, within strata, and within GS level categories; results were not statistically different when using the low value, midpoint (average between low and high values), or high value of a range. As a result, we present results for question 3 based on the midpoint. The 95 percent confidence intervals for these results are within +/- 10 percent of the estimates themselves, except for the estimate for technology center 2100 (Computer Architecture, Software, and Information Security). The estimate for technology center 2100 has a 95 percent confidence interval of within +/- 18 percent of the estimate itself. For some other survey questions, we also reviewed examiners’ open- ended responses on selected topics. We selected those topics based on our interviews with experts and USPTO officials as well as our analysis of closed-ended survey responses. We selected the questions for which examiners’ responses most frequently included keywords we identified for each topic. An analyst conducted a keyword search of all responses to the selected open-ended questions and coded responses containing the keywords. A second analyst verified the initial analyst’s coding. Our report provides some examples or summaries of examiners’ comments based on this review. Examiners’ responses to open-ended questions are not generalizable to other examiners. In addition, because we did not conduct a systematic review of all open-ended responses to our survey, we do not report the exact number of examiners who provided responses on the topics we reviewed. In addition, we conducted statistical tests of association on the results of certain survey questions; all tests were independently verified to ensure their accuracy. All tests of association were carried out at the 5 percent level of significance and were Cochran-Mantel-Haenszel (CMH) Chi- square tests of general association. The testing was carried out in SUDAAN, which is statistical software appropriate for the analysis of survey data. The null hypothesis was that there is no association between the two tested variables. When the association between two variables, conditional on a third variable, is of interest, this is referred to as the stratum-adjusted CMH test. The test statistic is Wald Chi-Square. Specifically, among other survey questions, we performed CMH tests on examiners’ responses to questions on how often they search for foreign patent literature or foreign-language nonpatent literature, and how difficult it is to obtain relevant art from these searches. We also performed CMH tests on responses to questions on how often examiners search for certain types of prior art and whether they have sufficient time to complete a thorough prior art search. The types of art included were foreign patent literature, scientific articles or presentations, foreign- language nonpatent literature, and industry-related nonpatent literature. We performed these tests on responses overall as well as conditional on the examiners’ technology center. To describe how selected foreign patent offices have addressed challenges in identifying relevant prior art, we first reviewed information identifying the world’s five largest patent offices. From the list of the other four offices similar in size to USPTO, we selected the European Patent Office (EPO) and the Japan Patent Office (JPO) based on comments observers made about the quality of their work. We conducted site visits with EPO and JPO, during which we interviewed officials from these offices and reviewed documents they provided. We also interviewed stakeholders knowledgeable about the offices’ practices, such as academics who study these offices. Because these individuals were not necessarily knowledgeable about USPTO practices, they are not included among the 18 experts discussed above. To assess the extent to which USPTO has taken steps to address any challenges in identifying relevant prior art, we reviewed documents from the agency related to prior art search procedures and capabilities, including ongoing and planned initiatives related to information technology resources or capabilities, the examiner workforce and related human capital management efforts, training practices, and international cooperation. We also interviewed or obtained written responses to our questions from officials from USPTO’s Office of the Commissioner for Patents, Office of Patent Examination Policy, Office of Patent Information Management, Office of International Patent Cooperation, Office of Human Resources, and Office of Patent Quality Assurance, among others, and conducted interviews with technology center directors, supervisory patent examiners, and representatives from the USPTO examiners’ union. In addition, we reviewed the results of the survey of examiners. In assessing USPTO’s efforts, we identified criteria in the federal standards for internal control; the Government Performance and Results Act of 1993, as amended; USPTO’s Manual of Patent Examining Procedure; and USPTO’s strategic plan. We conducted this performance audit from November 2014 to June 2016 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Appendix II: Patent Examination Process at the U.S. Patent and Trademark Office (USPTO) (Corresponds to Fig. 1) This appendix provides details on steps in the patent examination process, including rollover information, depicted in figure 1. Based on our survey results, U.S. Patent and Trademark Office (USPTO) patent examiners in some technology centers or at some General Schedule (GS) levels face certain challenges in identifying relevant prior art more than examiners in other technology centers or at other GS levels. Tables 8 through 15 provide information related to the effects of the quantity of prior art, how often examiners search for certain types of prior art and how difficult they find it to obtain relevant art from certain types, the sufficiency of examiners’ time to complete a thorough prior art search, average time examiners spent on prior art searching and each First Action on the Merits, examiners’ overtime, how often examiners encountered misclassified patent applications, and continuing education provided by USPTO in examiners’ art areas. Because we surveyed a generalizable stratified random sample of examiners, our results provide estimates for each technology center included in our study. See the related e-supplement, GAO-16-478SP, for the original survey language and more detailed results. In addition to the contact named above, Chris Murray (Assistant Director), Tind Shepper Ryen, Krista Breen Anderson, Hilary Benedict, Richard Burkard, John Delicath, Alice Feldesman, Armetha Liles, Rebecca Makar, Rob Marek, Eleni Orphanides, Kelly Rubin, Monica Savoy, Ardith Spence, and Sonya Vartivarian made key contributions to this report. Intellectual Property: Assessing Factors That Affect Patent Infringement Litigation Could Help Improve Patent Quality. GAO-13-465. Washington, D.C.: August 22, 2013. U.S. Patent and Trademark Office: Performance Management Processes. GAO-10-946R. Washington, D.C.: September 24, 2010. Intellectual Property: Enhanced Planning by U.S. Personnel Overseas Could Strengthen Efforts. GAO-09-863. Washington, D.C.: September 30, 2009. U.S. Patent and Trademark Office: Hiring Efforts Are Not Sufficient to Reduce the Patent Application Backlog. GAO-08-527T. Washington, D.C.: February 27, 2008. U.S. Patent and Trademark Office: Hiring Efforts Are Not Sufficient to Reduce the Patent Application Backlog. GAO-07-1102. Washington, D.C.: September 4, 2007. Intellectual Property: Improvements Needed to Better Manage Patent Office Automation and Address Workforce Challenges. GAO-05-1008T. Washington, D.C.: September 8, 2005. Intellectual Property: Key Processes for Managing Patent Automation Strategy Need Strengthening. GAO-05-336. Washington, D.C.: June 17, 2005. Intellectual Property: USPTO Has Made Progress in Hiring Examiners, but Challenges to Retention Remain. GAO-05-720. Washington, D.C.: June 17, 2005.

Summary: USPTO examines patent applications to ensure that inventions are, among other requirements, novel and not obvious. USPTO patent examiners accomplish this by comparing applications to “prior art”—existing patents and applications in the United States and abroad, and nonpatent literature, such as scientific articles. Thorough prior art searches help ensure the validity of granted patents. GAO was asked to identify ways to improve patent quality through use of the best available prior art. This report (1) describes the challenges examiners face in identifying relevant prior art, (2) describes how selected foreign patent offices have addressed challenges in identifying relevant prior art, and (3) assesses the extent to which USPTO has taken steps to address challenges in identifying relevant prior art. GAO surveyed a generalizable stratified random sample of USPTO examiners with an 80 percent response rate; interviewed experts active in the field, including patent holders, attorneys, and academics; interviewed officials from USPTO and similarly sized foreign patent offices, and other knowledgeable stakeholders; and reviewed USPTO documents and relevant laws. Experts and U.S. Patent and Trademark Office (USPTO) examiners described a variety of challenges in identifying information relevant to a claimed invention—or “prior art”—that can affect examiners' ability to complete a thorough prior art search in the time allotted and their confidence in their search efforts. These challenges include, among others, the quantity and availability of prior art, the clarity of patent applications, and USPTO's policies and search tools. The European Patent Office and Japan Patent Office face similar challenges to USPTO in identifying prior art and use various approaches to help address them, such as leveraging work of other patent offices on related patent applications and integrating nonpatent literature into their search tools. In some cases, these approaches are coordinated with, similar to, or could inform USPTO actions. USPTO has taken actions to address challenges in identifying prior art, but some actions have limitations. For example, USPTO is in the process of upgrading its search tools. However, examiners will still need to access a variety of external sources to meet USPTO's requirement to consider nonpatent literature. Federal internal control standards call for controls to evolve to remain effective and USPTO officials noted that the new search system can be expanded to include more nonpatent literature as the European and Japan patent offices have done. However, USPTO does not have a documented strategy for identifying additional sources. Without such a strategy, USPTO cannot be assured that its information technology investment will improve examiners' searches. USPTO is also taking steps to augment the number of, and consistency with which, reviews of examiners' work are conducted and documented, which could improve USPTO's monitoring of examiners' work. However, USPTO still faces limitations in assessing the thoroughness of examiners' prior art searches, because, for example, the agency has not established goals or indicators for search quality and may not be collecting sufficient information on examiners' searches to assess prior art search quality. Without monitoring examiners' prior art searches, the agency cannot be assured that examiners are searching all relevant sources of prior art and may not be able to develop appropriate responses as called for by federal internal control standards. GAO is making seven recommendations, among them, that USPTO develop a strategy to identify key sources of nonpatent literature, establish goals and indicators for prior art search quality, and collect sufficient information to assess prior art search quality. USPTO concurred with GAO's recommendations.