Throughout history, new technologies have transformed societies. Many technological advances, ranging from the steam engine to electricity and personal computers, have enhanced productivity and improved societal standards of living. At the same time, many technological advancements have led to increases in automation—modifying processes to become more automatic by reducing human involvement—and corresponding changes in the workforce. For example, researchers have noted that automation has replaced tasks performed by workers and also increased production, creating a greater demand for other types of workers. Although automation has historically been a labor disrupter in manufacturing and physical work, various researchers have observed that recent progress in fields such as artificial intelligence (AI) and robotics are enabling machines to perform cognitive tasks currently performed by humans. Artificial intelligence refers to machines and computers that attempt to mimic various aspects of human intelligence, as we have reported. The field of AI can be traced back to the 1950s. Early AI often consisted of expert systems programmed by humans to perform predefined tasks. This form of AI resulted in some degree of productivity gains and remains an active area of development. However, numerous factors, primarily the trends underlying big data (i.e., increased data availability, storage, and processing power), have contributed to rapid innovation and accomplishments in AI in recent years. Present-day AI innovation centers more on machine learning, including deep neural network architectures, in which systems are trained against observational or simulated outcomes—applications include language translation and machine vision (i.e., systems that use cameras, radar, or lasers to observe their surroundings or recognize content). Industrial robots and robotic machinery are often more comparable to expert systems that are programmed to perform predefined tasks, but they can also incorporate machine learning, such as having machine vision capabilities (e.g., object recognition). Below are some examples of expert system and machine learning applications of artificial intelligence. Examples of expert system applications of AI: software programs that prepare tax filings or schedule logistics; and industrial robots that perform predefined or routine tasks, such as lifting, placing, and welding pieces of metal together. Examples of machine learning applications of AI: software that uses a training dataset to “learn” how to read information from a form filled out by a person; collaborative robots that can sense when they touch a physical obstruction and shut down to safely work alongside humans; industrial robots with machine vision incorporated to identify and pick up specific parts from a collection of randomly strewn pieces; and automated guided vehicles that transport materials around a production plant and use cameras and radar to navigate independently and re-route around obstacles. Advanced technologies, including AI and other technological drivers of workforce changes, are continually progressing and new developments emerge regularly. For example, automated vehicles have varying levels of autonomy. Similarly, while robots have existed for decades, today’s generation of robots may be equipped with machine vision and learning capabilities that enable them to perform a more expansive array of tasks. How, when, or whether technologies progress from development to commercialization (i.e., readiness for adoption), and how, when, or whether firms adopt the technologies is generally dependent on context- specific considerations, which are difficult to predict. To better understand these developments and how they affect the economy, the National Academies report recommended developing three indexes (technology progress index; AI progress index; and organizational change and technology diffusion index) to measure technology progress and the extent of adoption. The study suggested that indexes could be valuable for identifying what fields are advancing rapidly and what benchmarks might indicate the imminence of significant economic impact, as well as tracking and predicting the types of human tasks that can be automated and the impacts of technology adoption by industry. Stanford University’s AI Index project is another initiative that aims to track, collate, and visualize data related to artificial intelligence. The data collected by the AI index measure, among other things, volume of AI activity (e.g., published papers, course enrollment, AI-related startups, job openings) and technical performance (e.g., object detection and speech recognition). However, the potential uses and limitations of the data being compiled are yet to be seen, as this initiative is still in its early stages. While national employment data measure jobs and workers by occupation and industry, the adoption of advanced technologies generally affects specific work tasks, and can materialize in a variety of ways. As shown in figure 1, industries are made up of various occupations, which in turn are formed by a group of jobs. Underlying all, jobs are comprised of a collection of varied work tasks. By analyzing tasks within jobs or occupations to determine their susceptibility to automation, a number of studies have developed models to estimate the future workforce effects of advanced technology adoption. The three example studies below each developed similar models, though differences in methods and data sources produced varying conclusions about the number of jobs that may be automated in the future. In a 2016 article, researchers Frey and Osborne estimate that 47 percent of total U.S. employment is in occupations that are at high risk of automation over the next decade or two (i.e., by 2030). For example, the authors observe both that industrial robots will be able to perform a wider scope of non-routine manual tasks and that a substantial share of employment in services, sales, and construction occupations exhibit high probabilities of automation. A 2017 report by the McKinsey Global Institute estimates that 23 percent of total U.S. work hours could be automated by 2030 or as high as 44 percent under other assumptions. The report predicts that while labor demand will enable some re-employment of displaced workers, up to one-third of the workforce may need to change occupational categories. In a 2016 paper, researchers Arntz, Gregory, and Zierahn estimate that 9 percent of all U.S. workers hold jobs that are at high risk of automation. The authors observe that susceptibility to automation is lower for jobs that require cooperating or influencing others. Studies by Autor and others also develop theoretical models exploring the effects of automation. For example, they noted that while automation can substitute for some tasks, it can also complement others. This can lead to increasing value for tasks that require other attributes like creativity and intuitive judgement. These models hypothesize that automation may have a net positive effect on employment, or at least on employment in certain sectors, which is consistent with historical employment trends. However, researchers have also noted that machine learning may affect different tasks than earlier forms of automation and may be less likely to automate low-wage jobs—though low-wage workers may be affected in other ways. Although the models discussed above represent ways of identifying jobs that may be affected by the adoption of advanced technologies, they do not provide a model for tracking the current or to-date workforce effects of technology adoption. As the recent National Academies report states, “making forecasts about social phenomena is perilous… doing so with respect to the fast-changing and dynamic area of technology is even more challenging.” According to a different project by some of these same experts, several factors unrelated to whether a task or job could be automated contribute to these challenges. For example, technologies may substitute for human labor in some tasks, but: may also complement human labor in other tasks—increasing the demand for, or value of, human labor (e.g., the automation of calculation tasks leading to increased demand for human programmers); prices and demand for products may counteract this human labor substitution (e.g., technology reducing the price of air travel, and thus leading to increased demand for flights, and thus increased employment in the aviation industry); and firms may redesign operations in response to the substitution in ways that lead to employment increases or decreases that are greater than the direct substitution. As discussed in the National Academies report and elsewhere, researchers have tried to disentangle workforce effects in various ways, such as analyzing productivity data to examine workforce trends in the context of other economic factors, such as globalization. As the National Academies report observes, “Predictions that new technologies will make workers largely or almost entirely redundant are as old as technological change itself…. However, predictions of widespread, technologically induced unemployment have not come to pass, at least so far.” Since recovering from the recession of 2007-2009, the economy has recently experienced low unemployment rates—4.0 percent in January 2019—despite continued strides in advanced technologies. However, other indicators have not recovered. For example, the labor force participation rate—the percentage of the population that is either employed or seeking work—declined significantly through the recession and has generally remained at this lower level. This may indicate that the post-recession decline in the unemployment rate may over-represent the health of the labor market, according to BLS. Advanced technologies and automation may also affect workers in other ways, beyond potential changes in the workplace, such as by reducing production costs and thus lowering the prices of consumer goods. There are currently no comprehensive data on firms’ adoption and use of advanced technologies. As a result, researchers have difficulty determining whether changes in the U.S. workforce observed in existing employment data are related to advanced technologies. The National Academies report states that federal household and employer surveys, such as the CPS, ACS, and OES, provide useful information about changes to the occupational mix of the U.S. workforce over time. However, these data cannot identify the causes of employment shifts. For example, these data do not identify whether an employment decline in one occupation is due to jobs being replaced as a result of automation, or to other factors unrelated to automation. Other federal data, such as the Job Openings and Labor Turnover Survey, provide useful information on employment turnover and opportunities. However, although these data are available by industry sector and firm size, the data do not capture reasons for layoffs and discharges, and thus cannot be linked to advanced technologies. In the absence of comprehensive data that definitively link employment trends to technology adoption, we analyzed occupations that researchers Frey and Osborne identified as being susceptible to automation (see sidebar) to determine whether changes due to advanced technologies are appearing in employment data. By exploring concentrations of these occupations in industries, job displacements in these occupations, and the characteristics of workers in these occupations, we found minor indications that advanced technologies are changing the workforce and could affect some worker populations. However, the conclusions that can be drawn from these analyses are limited by the unpredictability of when, if, or how automation materializes—e.g., whether worker positions are eliminated or shifted to other non-automated tasks. Industries with higher concentrations of jobs susceptible to automation were more likely than others to have experienced significant growth in their concentration of tech jobs from 2010 to 2016, according to our analysis of employment data from the American Community Survey. For example, as shown in figure 2, the plastics product manufacturing industry has a relatively high concentration of jobs susceptible to automation. Many of these jobs are in production occupations. From 2010 through 2016, this industry experienced about 11 percent annual growth in tech jobs (i.e., jobs in the fields of computing, engineering, and mathematics). More than half of this growth was the result of increases in industrial engineers, engineering technicians, and miscellaneous engineers. As we observed at some firms we visited, some of these engineers may have been hired to program or maintain newly installed robots. However, the data do not provide this level of information about job tasks. Similar dynamics could also be occurring in other industries. Across all 69 industries that had statistically significant changes in the concentration of tech jobs, we found a positive, though weak, correlation with the concentration of jobs susceptible to automation (see fig. 2). This suggests that growth in tech jobs may be an indicator of industries’ preparation for, or adoption of advanced technologies. However, given the complex causes of employment changes, there could be other reasons for tech job growth in these industries that are unrelated to firms’ adoption of advanced technologies. The growth in tech jobs in certain industries suggests firms in these industries may be using more advanced technologies, which could also signal that jobs susceptible to automation are being replaced. However, our analysis of ACS data showed no correlation between an industry having a higher concentration of jobs susceptible to automation and employment changes in that industry (i.e., total employment increases or decreases). We also found no meaningful differences in job losses, according to our analysis of employment data from the Current Population Survey’s Displaced Worker Supplement. Specifically, the relative rate at which workers in occupations susceptible to automation lost a job because their position or shift was abolished or there was insufficient work for them to do was not meaningfully different than workers in other occupations. There could be a number of reasons we did not find a relationship between susceptibility to automation and employment changes in both of these analyses, including: a relationship does not exist; such a relationship is too complex to measure in this way (e.g., automation may lead to decreases in employment in some industries, while also leading to increases in employment in other industries due to improved competitiveness, productivity, and profitability); it is too soon to observe the employment effects of automation (e.g., growth in tech jobs in an industry may be a leading indicator of employment disruption); or our analysis covered a period of overall economic growth, which could obscure or overwhelm other employment trends. Existing data cannot predict with certainty when or if automation will materialize in the workforce, as suggested by our analyses. However, the tendency of particular worker groups to hold jobs susceptible to automation suggests that some communities may be disproportionately affected by changes if they occur. For example, according to our analysis of 2016 ACS data, workers with lower levels of education are more likely than those with higher levels to hold jobs in occupations that the Frey and Osborne study identify as susceptible to automation. Specifically, 60.7 percent of workers with a high school degree or less hold these types of jobs, as compared to 46.7 percent of workers with some college, 26.9 percent of workers with a bachelor’s degree, and 11.3 percent of workers with a graduate degree. In addition, 54.1 percent of Hispanic workers hold jobs in occupations susceptible to automation, as compared to 46.4 percent of Black workers, 40.0 percent of White workers, and 35.9 percent of Asian workers. Certain geographic areas also rely more heavily than others on occupations identified as susceptible to automation, according to OES data. We identified areas where the proportion of jobs susceptible to automation is at least 5 percentage points greater than the national average (see fig. 3). These occupations are comprised of a diverse set of jobs that may experience automation in different ways and at different times, if at all. However, if employment disruptions are regionally concentrated, groups of workers with similar skills in the same labor market may need to adapt to changes simultaneously, which could strain the availability of local job opportunities and support resources. Workers in occupations that the Frey and Osborne study identify as susceptible to automation earn less on average than other workers. For example, the median hourly wage for workers in occupations susceptible to automation is $14.26, compared to $22.06 for other workers, according to our analysis of 2016 ACS data. After controlling for factors that may affect wages, such as age, education, and industry, we found that workers in jobs susceptible to automation earn about 17.2 percent less, on average, than similar workers in other occupations. These results show that, on average, workers in jobs susceptible to automation are already in more vulnerable economic circumstances than other workers. When or if changes brought on by automation materialize, these workers may face additional hardships in adapting to changing workforce demands. In the absence of comprehensive data, researchers have taken differing approaches to exploring the relationships between technology adoption and workforce trends. We identified some examples of recent and ongoing work that attempt to measure workforce effects directly attributable to technology adoption. These examples illustrate types of data that may be useful for better understanding and measuring the use of specific technologies (e.g., robot sales), the spread of technologies generally (e.g., automation patents), and how specific work tasks are changed by technology use (e.g., firm-level operations data). Some researchers have used data on industrial robot sales collected by the International Federation of Robotics (IFR) to approximate robotics adoption worldwide and in the United States and to model its direct effects on employment. Analysis by Furman and Seamans (2018) shows that annual sales of industrial robots in the United States increased substantially between 2010 and 2016. The analysis attributes this growth to a combination of factors, including lower robot prices, improved robot functionality, and greater awareness of the benefits of robots. They also observe that the automotive sector was the largest customer for industrial robot sales in the United States from 2004 through 2016, though robot sales to the consumer electronics sector grew the most over that period. Studies by Acemoglu and Restrepo (2017) and by Graetz and Michaels (2017) both use IFR data through 2007 to model the workforce effects of robot adoption in the United States, though their methods, results, and conclusions differ. Acemoglu and Restrepo estimate that each additional robot used in a geographic area reduces employment by about six workers in that area. They observe that their estimated employment effects are greatest in manufacturing and other industries most exposed to robots, in routine manual work-related occupations, and for workers with less than a college education. They do not find corresponding employment gains in any other occupation or education groups. They also estimate that one more robot used per thousand workers reduces wages by about 0.5 percent. They conclude by noting that, so far, relatively few robots have been used in the U.S. economy and thus the effect on jobs has been limited; however, they state that if robot usage continues to grow as researchers expect, these effects could be more substantial. Graetz and Michaels estimate that increased robot use did not significantly affect total hours worked across the 17 developed countries in their analysis, but that work shifted from low-skilled workers to middle-skilled and high-skilled workers. They also estimate that increased robot use increases productivity and average wages. While their analysis covers 17 developed countries, they note that robot use in the United States was marginally lower than the average across all countries. They also observe that while their results differ from Acemoglu and Restrepo, it is possible that the effects of robot usage are different in the United States than across the 17 countries they analyze. Other researchers have used U.S. patent data as an alternative way to approximate the spread of advanced technologies and to examine the resulting workforce effects. Mann and Püttman (2017) use machine learning algorithms to identify patents related to automation technology. They find that automation patents grew substantially from 1976 through 2014. After linking the patents to industries where they may be used, they estimate that automation causes manufacturing employment to fall, though it increases employment in the service sector, as well as overall employment. They observe that their results depict a more positive picture of the employment effects of new technology use than the studies that used industrial robot sales data (discussed above). Lee Branstetter, a researcher at Carnegie Mellon University, and his colleagues have a similar ongoing project that uses a machine learning algorithm to identify patents related to AI technologies. According to these researchers, their initial results suggest a rapid rise in AI patents over the past decade and also that AI patents are emerging in a variety of application areas. They are also in the early stages of work linking AI patents to industries to explore how new technology use affects the workforce. Researchers have also identified how important micro-level data could be for understanding the workforce effects of advanced technology adoption. For example, reports by the National Academies and others highlight the potential for firm-level information to augment traditional survey data to enable analyses of the conditions under which advanced technologies complement or substitute for workers, and what types of firms invest in advanced technologies. Other researchers have emphasized the importance of focusing on work tasks to analyze the effects of technological change at workplaces. Erica Fuchs, a researcher at Carnegie Mellon University, and her colleagues Christophe Combemale, Katie Whitefoot, and Laurence Ales use a combined firm-level, task- based approach by collecting and analyzing production floor data from four semiconductor firms with different levels of process automation and parts consolidation. They map out detailed versions of firms’ production processes and then use existing data and technical knowledge to simulate each step to analyze the effects of technology changes. Their preliminary results estimate that automation replaces some routine tasks, leading to estimated declines in the number of production floor jobs requiring medium skill levels. According to the authors, this firm-level, task-based approach may be applicable to other manufacturing industries and could provide insight on how the adoption of different technologies may produce different labor outcomes. However, they note that the approach requires detailed production process data, which may be difficult to collect for many firms or industries. Commerce’s Census Bureau has begun administering surveys with questions that focus specifically on firms’ adoption of advanced technologies and resulting workforce changes. According to Census, this data collection is part of a long-standing, coordinated effort to measure the impact of technology. In addition, consistent with Commerce’s strategic plan, these represent new efforts to provide a timely, in-depth, and accurate picture of the economy amidst the economic shifts and technological advances of the 21st century. However, none of the survey results will be available until late 2019 and later. The new Annual Business Survey (ABS) is a joint effort by Commerce and the National Science Foundation that has the potential to provide insight on the spread of advanced technologies in the economy and could be used to examine the workforce effects of technology adoption, but the first ABS results are not expected until late 2019. Census administered the 2017 ABS in June 2018 to collect information on firms’ use of advanced technologies, such as automated guided vehicles, machine learning, machine vision, and robotics, among other things (see example in sidebar). The survey asks whether firms are testing a given technology or using it for either less than 5 percent, 5 to 25 percent, or more than 25 percent of their production or service. Census officials said this question should provide information about the extent of technology adoption nationwide, including whether there are any industry concentrations of advanced technologies. Census plans to add questions on the workforce effects of advanced technologies when it administers the 2018 ABS during July through December 2019, pending final approval by the Office of Management and Budget. Census plans to release these survey results in December 2020. Specifically, Census plans to include new questions that ask firms about: (1) their use of advanced technologies such as AI, cloud computing, robotics, and specialized software and equipment; (2) their motivation for adopting and using artificial intelligence and advanced technologies; (3) the impact these technologies might have on the number and skill level of workers; and (4) the factors that could adversely affect the adoption or production of these technologies. The new questions also ask about changes in the number of production workers, non-production workers, supervisors, and non-supervisors. These new questions could be used to characterize the prevalence of workforce changes in the economy caused by advanced technology adoption (e.g., declines in production workers, or increases in supervisory workers) and whether this differs by industry sector. However, these planned questions are not intended to provide information to quantify the magnitude of workforce changes, in part to minimize respondent burden and potential survey error, according to Census. In addition, until the ABS data are available and evaluated, it remains unclear what limitations, if any, the data may have. Census also plans to expand other surveys to track the spread of advanced technologies in the economy, including its Annual Survey of Manufactures (ASM) and Annual Capital Expenditures Survey (ACES). Census plans to administer the 2018 ASM in May 2019, pending final approval by the Office of Management and Budget. The survey will collect capital expenditures data for industrial robotics at approximately 50,000 manufacturing plants, as well as the number of industrial robots purchased by and in use at these plants. Census officials stated these two measures might be useful in understanding the impact that industrial robots could have on productivity as well as the impact robots could have on the manufacturing labor force once the survey results are available in the spring of 2020. Census plans to administer the 2018 ACES during March through May 2019 and to have the survey results available in February 2020.The survey will include questions on robotics expenditures, similar to those in the 2018 ASM. However, the ACES collects expenditure data from 50,000 employer firms across all non-farm sectors of the economy—instead of just manufacturers—and will also ask about firms’ use of both industrial and service robots. Some Commerce offices also track issues related to the adoption and workforce effects of advanced technologies on a limited or intermittent basis. For example, National Institute of Standards and Technology officials stated that the Hollings Manufacturing Extension Partnership collects limited information about the number of jobs gained and retained by small and medium businesses adopting new technologies. National Telecommunications and Information Administration officials said they monitor developments in AI on an intermittent basis and also direct a project that examines new applications of small and large internet devices. DOL has a role in collecting data that track changes occurring in the U.S. economy and workforce, including developing new ways to track emerging economic trends, though as we previously discussed, currently available federal data do not link shifts in the workforce to technological changes. BLS is the principal federal statistical agency responsible for measuring labor market activity. According to DOL’s strategic plan, BLS is to support public and private decision-making and meet the needs of its many stakeholders, including the general public, educational institutions, and the public workforce system. This includes regularly identifying structural shifts in the economy and developing new data products that reflect economic changes. In addition, DOL’s Employment and Training Administration (ETA) is to assist workers’ entry and reentry into in- demand industries and occupations. This assistance includes providing job seekers with accurate labor market data and guidance about opportunities, aligning training services to industry needs, and helping connect businesses with properly skilled workers. Internal control standards state that agencies should use quality information to identify, analyze, and respond to significant changes, including external conditions such as economic and technological changes that may affect an agency’s ability to achieve its objectives. DOL collects workforce data through various surveys, including the Current Population Survey’s Displaced Worker Supplement, and produces other data products such as the occupational employment projections and Occupational Information Network database that include information related to advanced technologies. However, these data are limited, and according to BLS, provide some, but not all, of the information required to assess the impact of automation on the workforce. BLS’s Employment Projections program identifies and provides limited information about occupations expected to experience declines in their share of employment in an industry or group of industries as a result of the adoption of advanced technologies. On a biennial basis, this program analyzes changes in the economy to project how employment by occupation may change over 10 years, including which occupations may be affected by advanced technologies. Factors that can affect occupational employment include but are not limited to technological innovation; changes in business practices or production methods; organizational restructuring of work; changes to the size of business establishments; and offshore and domestic outsourcing, according to BLS. As part of this program, BLS develops a table of occupations that are projected to have direct employment changes due to some identified reason. This table identifies projected staffing pattern changes and BLS’s qualitative judgment of the most significant factor or factors projected to affect the occupation. The table also indicates whether an occupation’s share of employment is expected to change within a single industry or within multiple or all industries. For example, the table includes the following selected entries: Librarians: Employment share is projected to decline in the information services industry as internet-based research continues to displace library-based research. Stock clerks and order fillers: Employment share is projected to decline in two industries (the warehousing and storage industry and the grocery and merchant wholesalers industry) as firms increasingly adopt automated storage-and-retrieval systems. Aircraft structure and systems assemblers: Employment share is projected to decline in all industries as collaborative robotics increase efficiency, producing more output with the same amount of labor. We identified 100 occupations in BLS’s table that are projected to experience declines in their shares of employment in an industry or group of industries as a result of the adoption of advanced technologies. Similar to the examples above, reasons could be related to automation, the increased use of robots or artificial intelligence, advances in machine or software technologies, or other changes resulting from the adoption of advanced technologies. As shown in figure 4, most of these occupations are production occupations (40 of 100) or office and administrative support occupations (30 of 100). BLS officials told us they do not currently track groups of occupations projected to experience employment share declines due to specific reasons, such as advanced technology adoption. Officials also said they do not aggregate total projected employment effects stemming from similar causes because they are unable to identify ripple effects in all occupations—e.g., automation in one occupation affecting employment in a different occupation. Information contained in ETA’s Occupational Information Network (O*NET) database includes, among other things, information about work activities, tools and technologies used, and required skills associated with over 1,000 occupations. According to ETA officials, the primary purpose of O*NET is to assist job seekers in making employment decisions. However, the O*NET database can be used to identify occupations that use certain types of advanced technologies. For example, we identified 15 occupations in which workers monitor, install, develop, troubleshoot, debug, or perform other tasks with robots as part of their daily work activities and 63 occupations in which workers use robots as a tool or technology in their daily work activities (see table 1). In addition, states, federal officials including at BLS, and academic researchers use these data to inform, among other things, worker support programs. DOL officials told us they do not use O*NET data to analyze changes in occupations over time, such as robots being used in additional occupations, because the methodology is not currently structured to capture these kinds of changes systematically. For example, data are collected from a selection of occupations at varying frequencies, rather than at the same time, which could make it challenging to track changes in certain occupations over time. Without comprehensive data linking employment shifts and technological changes, policymakers and DOL may not be prepared to design and implement programs that both encourage economic growth and provide support for workers affected by changes. DOL-funded programs rely on accurate information to guide job seekers to employment opportunities and to help align training services with local industry needs. For example, the O*NET database identifies high-growth, high-demand occupations for job seekers based largely on BLS employment projections data. While these employment projections provide valuable information, they are not designed to identify the full extent of occupational shifts due to advanced technology adoption. Similarly, other workforce surveys, such as the Current Population Survey’s Displaced Worker Supplement and the Job Openings and Labor Turnover Survey, do not collect information about the causes of job losses and gains. This information could be a valuable tool for designing programmatic or policy supports for workers. For example, data on whether advanced technologies have resulted in worker displacements, work hour reductions, or substantial adjustments to work tasks could better position BLS to meet stakeholder needs. Congress has expressed concern that there continues to be insufficient data on the effects advanced technologies are having on the U.S. workforce. On January 2, 2019, BLS reported to Congress that it plans to work with a contractor during fiscal year 2019 to study the interaction between labor and capital in the workplace and how it is affected by new technologies; identify ways to supplement BLS data with additional information on automation; and produce a report that recommends data collection options to fill those gaps. In fiscal year 2020, BLS also plans to identify pilot projects to test the feasibility of new data collection based on the recommendations in its final report, resources permitting. However, these plans are still in their early stages, according to BLS officials. Officials at Commerce and DOL stated that collecting data on the adoption and workforce effects of advanced technologies is challenging because it is difficult to identify which new and emerging technologies to track; employment trends generally occur at the occupation and industry levels but the effects of advanced technologies typically occur at the task or job level; and employment trends have a complex and diverse set of causes. Specifically: Identifying which new and emerging technologies to track. Census officials said there is uncertainty about how an emerging technology might affect the economy and thus whether it should be tracked systematically. For example, self-service technology appeared at grocery stores in 1916, other self-service technology appeared at gas stations later, and more recently self-service technologies are being adopted by some restaurants, according to researchers. Periodically, Census has included questions in its firm surveys about the use of these technologies. Past surveys asked questions about the use of self-service at gas stations until the technology became ubiquitous and was dropped from the survey. As self-service technologies have expanded to other areas of the economy such as restaurants, Census has again added questions about self-service to recent surveys because information is lacking on the growth of this phenomenon. Trends and effects appear at different levels. BLS officials said employment changes due to technology typically occur at the individual task or job level and employment trend data are at the industry and occupation levels. Officials also said that identifying technology-related effects in occupations, such as changes related to uses of machine learning algorithms, is difficult because some workers within an occupation might be affected by the technology while others might not. For example, some computer scientists and engineers might be involved in the development or application of machine learning algorithms while others are not. Causes of trends are complex and diverse. BLS officials said that employment trends’ complex and diverse causes make it difficult to identify occupations that are changing because of advanced technologies. Changes in one occupation may have ripple effects in other occupations. Partly as a result of this complexity, BLS’s Employment Projections program identifies examples of technology- impacted occupations, but it does not attempt to identify all instances where technology impacts occupations nor does it attempt to quantify an overall projected employment effect of advanced technologies. The White House Office of Science and Technology Policy (OSTP) is responsible for coordinating AI related policy across government agencies and for overseeing the National Science and Technology Council’s subcommittees and their ongoing activities. For example, the Subcommittee on Machine Learning and Artificial Intelligence was originally chartered in 2016 to monitor machine learning and artificial intelligence and to watch for the arrival of important technology milestones in the development of AI, among other things. OSTP officials told us that the Subcommittee has been re-chartered, now receives direction from OSTP’s Select Committee on Artificial Intelligence, and is presently focused on federal resources related to AI research and development. Selected firms generally adopted advanced technologies through a phased process of innovation and technology adoption (see fig. 5). We met with officials representing 16 firms that are using advanced technologies and a systems integrator who spoke for a number of his customer firms. Many firm officials described the path to integrating technology into operations as lengthy, complex, and iterative. For example, some firms we visited have had to build and test different mechanical “grippers” attached to robot arms to pick up and handle particular objects; one firm had high school participants at a local training center develop a gripper solution for one of the firm’s robots. Some of the large firms we visited had their own internal teams that identified, tested, and integrated advanced technologies. Other firms we visited used third- party integrator companies to help with incorporating technologies into their operations. We spoke with firm officials about their motivations for adopting advanced technologies, as well as challenges they faced throughout the process, and they identified a number of similar issues. Most selected firms cited cost savings as a primary consideration for adopting advanced technologies. Firm officials discussed cost-related motivations in various forms, such as remaining competitive in a global economy, increasing productivity (i.e., lower cost per unit), decreasing labor costs, and saving on physical space. Firms said they adopted advanced technologies as a way of reducing operational costs—including labor costs—to increase competitiveness and profitability. Some officials also specifically identified the pressure of large low-cost competitors, both in the United States and globally, as a major motivation to reduce costs and product prices. Officials at a medium-sized door manufacturer told us that increased use of advanced technologies, such as robots, enabled the firm to increase efficiency, reduce labor costs, and re-focus its product line on custom doors to survive the entry of manufacturers in China that could sell mass-produced doors for lower prices. The original motivation for adopting robots at a medium-sized automotive parts manufacturer was a customer’s price demand that the firm could not meet and still remain profitable, according to officials. Integrating more robots enabled the firm to reduce production costs by using fewer workers. At a large manufacturing corporation of household and personal care goods, officials told us the company had a goal of reducing its workforce size by 1,500 full-time positions per year for 5 years (across its subsidiaries), and specifically using robotic automation to accomplish 40 percent of its reduction goal. The constant pressure to keep costs low in the health care sector motivated a university-affiliated medical center we visited to explore adopting more advanced technologies, such as autonomous mobile robots that could decrease expenses by reducing the number of positions in some departments. Firm officials also told us about other, non-labor-related cost savings considerations that led to the adoption of advanced technologies. Officials at a large automotive manufacturer told us they recently upgraded a laser welding system to use fewer, more advanced robots to save production line space—which is a valuable commodity in manufacturing. They also pursued this change to increase overall production capacity because the physical space they saved could be used to install more robots for other production steps. The integration of autonomous mobile robots to deliver prescription drugs to patient wards at a university-affiliated medical center was intended, in part, to save costs related to medicines that go missing when delivered and processed manually, according to officials. According to officials at selected firms, the desire to improve jobs led firms to adopt advanced technologies. The firms wanted to automate tasks that are dangerous, difficult, dull, or dirty in large part to improve worker safety, and to optimize the value added by workers. For example: Dangerous work: Two robots were installed to pick up doors weighing between 90 and 300 pounds, and place them on a paint line at a medium-sized door manufacturer we visited. Prior to the robots, workers who performed this dangerous task experienced work related injuries, and the firm paid large amounts of money in workers’ compensation claims, according to officials. Once the robots were installed, the firm experienced a decrease in the number of worker compensation claims. Dull work: A small automotive parts manufacturer we visited installed an industrial robot to perform a machine-to-machine transfer of a heavy part. Prior to the robot, the firm had three workers performing this task—even though the task only required two—because workers would eventually quit due to the tedium of the job and new workers would require time to be trained, according to officials. Value-added work: Some officials told us they adopted advanced technologies because they wanted to maximize human labor that provided value to the firm and reduce labor that did not. Officials at a warehouse for a regional grocery store chain and a university- affiliated medical center said they wanted to minimize time workers spent traveling between tasks (as opposed to performing tasks). Warehouse officials said their workers spend up to 60 percent of their time traveling back and forth between shelves and products, which is time that could be spent selecting and sorting items. Thus, at the time of our visit, the warehouse was in the early stages of adopting automated guided vehicles to eliminate the need for workers to travel between points. Similarly, officials at a university-affiliated medical center that adopted autonomous mobile robots to transport, among other things, prescription drugs, said nurses and pharmacy technicians used to walk back and forth between the patient ward and the pharmacy to pick up and deliver these drugs, which diverted them from performing other tasks. They said that the medical center wanted them to have more time to provide valuable work, especially for employees who are highly-paid. Officials at many firms said that adopting advanced technologies can help them deal with the challenges of recruiting and retaining skilled workers. They explained that worker shortages and high turnover can result from skill gaps in the local or national workforce, low unemployment, and certain work being viewed as unappealing, among other reasons. For example, officials at a warehouse for a regional grocery store chain we visited told us they struggle with high worker turnover and the constant need to hire new workers. In addition, low unemployment can make it difficult to retain workers with the right skills to operate machinery according to officials at a small automotive parts manufacturer. Similarly, at the university-affiliated medical center, an official said that positions for pharmacy and other types of medical technicians can be difficult to fill. By using autonomous mobile robots to automate some tasks, the medical center can streamline its operations to more efficiently use the technicians it already has. Recruitment in Manufacturing Officials at some manufacturing firms we visited said they have had trouble attracting new workers into the sector, and officials at two firms said that adopting advanced technologies is one way they have sought to make manufacturing more attractive and to appeal to more and younger workers. One younger worker at a small automotive parts manufacturer talked about how appealing his workplace was due to the firm’s use of advanced technologies, specifically robots. Officials at a large automotive manufacturer viewed their tech development facility, which includes spaces to tinker with virtual reality, augmented reality (i.e., technology that superimposes images on a user’s view of the real world; for example, by wearing augmented reality glasses), and other emerging technologies, as an asset to recruit young talent. Improving product quality, expanding product offerings, and supply chain reliability were primary motivations for adopting advanced technologies, according to officials at some firms. Product quality: Quality is paramount in the automotive industry, where mistakes are costly and can have implications for a firm’s reputation, according to officials at a medium-sized automotive parts manufacturer we visited. For this reason, they decided to use robots rather than workers for welding in order to standardize the processes, reduce errors, and improve product consistency and quality. Officials at a large automotive manufacturer similarly said that the firm has pursued machine learning technologies to ensure fewer defects and problems in vehicles. Engineers at the firm are developing a smart watch for workers who connect wires that will provide feedback to these workers if a proper connection is not made, based on the sound of the connection. The firm is already using machine vision technology that inspects vehicles as they pass through a section of the production line to ensure the correct pieces have been used for each vehicle model. Expanding product offerings: At a medium-sized fruit processing plant, an official said that integrating robots, an advanced conveyer system, and machine vision inspection technologies, among other advanced technologies, enabled the firm to begin producing applesauce in a highly automated and safe way. Had manual production been the only option, officials said they would not have considered producing applesauce due, in part, to safety issues. Supply chain reliability: One small manufacturer of rubber stamps and embossing seals (hereafter referred to as a small stamp manufacturer) used to rely on a single supplier for pre-cut materials, which was not always reliable. The firm adopted a collaborative robot, in part, so it could purchase raw materials directly and then have the robot cut the materials as part of the production process (see fig. 6). In addition to the capital cost of advanced technologies, which some firms told us can be substantial, firms face a number of risks that can affect their return on investment, such as the reliability of technology and working with new tech developers. While the firms we met with had already adopted advanced technologies, officials had to consider and overcome various risks during the adoption process. Some of these firms decided against adopting other advanced technologies upon evaluating these risks. Being an early adopter of a technology is risky because the new technology may not yet be sufficiently reliable for firms’ operations. Officials at a large appliance manufacturer we visited showed us technology that was supposed to use machine vision to autonomously inspect the wire connections for clothes dryers. They told us that the vision technology had been ineffective, so they took it off the production line for engineers to continue working with it in the lab; they planned to bring the technology back onto the line a few weeks after our visit. Officials at this firm said that the vision technology was still relatively immature, as it had a limited field of vision and yielded numerous false readings. Similarly, a warehouse we visited that invested in automated guided vehicles used them to move pallets for a short time, but then put them into storage because these vehicles did not have mature enough machine learning and vision capabilities for the firm’s purposes. Eventually, officials from this warehouse began working closely with the developer firm to improve the vehicle technology, which advanced enough that it could be used. For instance, officials from the warehouse suggested adding turn signals to the vehicles to alert nearby workers of intended movements and improving the vehicles’ ability to travel over spills without triggering the system’s sensors to shut down. Firm Size Might Affect Risk Tolerance An official at one small manufacturing firm stated that larger firms may be more willing to be early adopters of technology, as they may be able to absorb the high risks of experimenting with expensive technologies, while smaller firms tend to wait until a technology has been optimized before deciding to adopt it. Accordingly, his firm only purchases industrial robots from an established manufacturer, although it would like to experiment with newer technologies in the future, such as augmented reality. Officials at a large manufacturing firm told us they have purchased a number of advanced technologies to experiment with, even though they do not know yet how the technologies may ultimately be used in their production process. This firm also has teams of technicians and engineers who can adapt the technology for operations. During our visit, we met with engineers who demonstrated different potential applications of technologies that are still being tested, including using virtual reality to test new part design and augmented reality glasses to provide interactive training to workers. Officials at some firms explained that installing advanced technologies at times necessitated building manual redundancies into their operations due to reliability concerns. Officials at a construction consulting company and a municipal township that adopted a machine learning technology to inspect roads said the technology would miscategorize road quality at times, such as identifying tree branch shadows on the road as pavement cracks. While working with the developer to improve the technology, officials said they continued to conduct redundant manual inspections to ensure they were making road repair decisions based on accurate information. During our visit to a large appliance manufacturer, we saw multiple collaborative robots that were not working properly. As a result, workers were performing these tasks manually while the robots were down; officials told us that each of the firm’s automated processes has workers trained to perform the tasks in case a technology was not working properly. Technologies Viewed Differently by Firms Some firms find a technology to be useful while others find little practical application for that technology, as illustrated by the various opinions firm officials had about collaborative robots. Officials at one small manufacturer we visited said that a collaborative robot was well suited for the firm’s production process and environment because, among other reasons: (1) the firm produces small durable goods that require dexterity rather than speed, which the collaborative robot could provide; (2) the collaborative robot would be safe around workers and could be trained by non-technical staff, so the firm’s small workforce could adapt to its use; and (3) the collaborative robot could fit in the firm’s limited floor space, as it would not require a cage. On the other hand, officials at other manufacturing firms we visited told us that collaborative robots were less useful in their settings because they have significant weight and speed limitations in order to be safe enough to operate outside of a cage, limiting their usefulness for their firms. Some firm officials told us it could be risky to work with tech developers with limited experience. Officials at a large appliance manufacturer said that newer developers may go out of business or be bought out by a larger firm, which could render the technology acquired from them obsolete (especially in terms of future servicing of parts and software updates). The officials stated that emerging technologies, both hardware and software, tend to not be standardized, so investing in a developer likely means investing in a type of technology that may not be supported by other developers if issues arise. We heard from some firms that they purchased technology from developers who already had established reputations and longevity. For example, a small manufacturer of durable goods selected a robotics company because of the founder’s reputation and track record, among other reasons. Operational slowdowns: The time period between initial adoption and optimization of a technology varies widely and can sometimes be a lengthy and ongoing process, according to officials. One small stamp manufacturer experienced a lengthy and iterative implementation process for an off-the-shelf collaborative robot they purchased. For example, they had to construct a customized environment for the robot to function in, make parts by hand, purchase a 3-D printer to develop tools for the robot, and build additional parts to take care of increased byproducts like sawdust. Officials at a large automotive manufacturer told us that new technology, such as machine vision technology used for automated inspections, is often integrated on the weekends or during off-shifts. Then, on the first day of production after the new technology is integrated, the production line starts slowly and speeds up as worker comfort and experience increases. Outside of manufacturing, a consultant that helps facilitate the adoption of advanced technologies at firms said that firms’ existing, or legacy, computer infrastructure can be a barrier to integrating machine learning technology, increasing complexity and causing an extended implementation process as his firm integrates the new technology platform with the legacy infrastructure. Worker concerns: Officials at some manufacturing firms said they have encountered worker concerns with advanced technologies, and have employed various tactics to mitigate this, such as introducing workers to the technology in offsite demonstrations and involving them during the decision-making and planning before the technology was integrated. In one case, workers were able to ask questions about a collaborative robot as it was being installed and were provided with orientation training. The robot was then phased into operations—used initially for short periods of time so workers would become accustomed to its physical presence and proximity to their workstations. Deciding Not to Adopt Advanced Technologies Officials at the firms we visited identified instances in which they chose not to adopt certain advanced technologies, or not to use advanced technologies that were working well in other processes. Reasons we heard included: a product line had too much variation to benefit from advanced technologies (i.e., that some advanced technologies work better for standardized products and processes); a certain manufacturing process was too low-volume to invest time and resources into automation; and human dexterity is difficult to replicate. Officials from a large appliance manufacturer showed us an instance where using automation would not make sense. We observed a worker performing a simple, single task: grabbing a metal heat shield and plastic dishwasher spinner from separate bins and clipping one on to the other. Because of the shape of the pieces and because they were lying unorganized in boxes, the task requires human dexterity, making the process difficult to automate, according to officials. Officials at many of the firms we visited said they needed fewer workers in certain positions after adopting advanced technologies to perform tasks previously done by workers. Officials at these firms generally told us they adjusted by redeploying workers to other responsibilities and, in certain instances, reducing the firm’s workforce size through attrition. We also heard examples of direct layoffs due to the adoption of technologies. There may also be other types of adjustments firms can make that we did not observe or discuss with these officials. The complexity of these workforce adjustments makes it difficult to determine or measure the effects of technology adoption on workers. For example, although workers may not have lost their jobs due to an adopted technology taking over specified work tasks—either because of redeployments or attrition— fewer job opportunities might be available in the future for workers with similar skills. In addition, the iterative and sometimes lengthy process of incorporating advanced technologies can delay workforce effects. Thus, the absence of short-term effects of technology adoption does not necessarily preclude long-term implications, such as reductions or slower growth rates in workforce size over time (see text box below). As discussed in the prior section, one reason firm officials are motivated to use advanced technologies is to decrease labor costs. Slower Workforce Growth than Revenue Growth An official from a small automotive parts manufacturer told us that advanced technologies and automation resulted in revenue increasing by more than 400 percent over the last 12 years while the workforce increased about 15 percent. Production workers now make up a smaller percentage of the overall firm workforce than prior to automation, and sales and support staff now make up a greater percentage. The firm official described this change as an increase in higher-skilled jobs and a decrease in lower-skilled jobs. Similarly, according to firm officials at a different medium-sized automotive parts manufacturer, revenue has grown six times in the past 15 years while the workforce has grown four times, largely as a result of adopting robotics technology. Redeployments without job loss: When advanced technologies replaced positions, firms we visited often shifted, or redeployed, workers to different responsibilities. For example, officials at a medium-sized automotive parts manufacturer we visited told us they had nine workers who smoothed sharp edges and removed burrs on hydraulic cylinders prior to installing two robots to perform these tasks. Now, with the robots in these positions, three workers load the robots and then inspect and de- burr any parts of the cylinders the robots missed. The other six workers were redeployed to other tasks, according to a firm official. At a large appliance manufacturer we visited, officials told us that two workers used to move large parts from one line to another line to be painted. Now, as we observed, a collaborative robot performs this function alone; a worker monitors the operation to ensure it is running smoothly, and the original workers were moved to different tasks on the production line, according to officials. Although the size of these firms’ workforces did not decrease as a result of the technology adoption, the numbers of certain positions were adjusted—for example, production positions decreased while monitoring positions increased. Differences in skills required for these positions may also affect the ability of current workers to transition and could have implications for individual workers even though the number of jobs at the firm does not change. These sorts of changes may or may not appear in firms’ reported employment data, depending on whether redeployed workers change occupations or what other workforce changes may be occurring simultaneously (e.g., if other production workers are hired for reasons unrelated to the technology adoption). Redeployments with job loss through attrition: Officials at some of the selected firms that redeployed workers said they also reduced their overall workforce size through attrition, as a result of adopting advanced technologies. Autonomous mobile robots independently transported biohazardous waste, linens, meals, and prescription drugs throughout the university- affiliated medical center we visited. Officials told us they eliminated 17 positions after they deployed the robots. No workers were laid off; instead, they relied on high staff turnover rates and moved workers to vacant positions elsewhere. At a medium-sized fruit processing plant, firm officials told us they replaced 150 to 200 jobs with various advanced technologies over the past 3 to 4 years. However, they relied on attrition rather than layoffs. For example, the plant adopted a robot to pack food into boxes. Prior to using the robot, officials told us there were 26 workers per shift performing this job; as of our visit, there were 13 workers per shift. A medium-sized door manufacturer reduced its workforce from 650 employees to less than 500 over approximately the last 20 years due to, among other things, their adoption of robots, according to firm officials. For example, we observed industrial robots that load steel sheets into a cutting machine, reading a barcode on each sheet that tells them what size sheet is being lifted and how it should be placed in the cutting machine. This process only requires a single worker to monitor the robots during each of two shifts, where previously three workers per shift were on this production step (i.e., a change from six to two workers total). How quickly workforce reductions materialize for firms using attrition can vary greatly. We visited firms with low employee turnover rates and firms with high turnover rates. High worker turnover rates allowed some firms to more quickly adjust their workforces when deploying advanced technologies and may be a reason we were told about job loss through attrition rather than layoffs at these firms. Job loss through layoffs: An official from a systems integrator firm (“integrator”) provided examples of significant layoffs as a direct result of advanced technologies. This integrator provides machine learning technology and other similar products to automate office and administrative processes, among other things. One of the integrator’s customers—a U.S. automotive parts firm facing competition from online retailers—adopted machine learning technology to take over its accounts payable and distribution system. As a result, according to the integrator’s official, this firm reduced the number of employees in one of its U.S. offices from 500 to 200. Another of this same integrator’s customers—a firm that sells telecommunication circuits—adopted machine learning technology to automate product returns processing. As a result, the firm experienced a 30 percent reduction in customer care calls, and replaced about 150 jobs in a U.S. call center with 110 jobs at a call center in a different country (i.e., about 150 U.S. jobs lost; and an overall workforce reduction), according to the integrator’s official. According to officials at some selected firms, greater competitiveness and productivity due to the adoption of advanced technologies (see sidebar) has helped firms grow their workforces. For example, some hired additional production workers due to increased production (despite some production tasks being taken on by the adopted technologies), or new types of workers, such as technicians to maintain the technologies. Some officials also said that although they may not have grown their workforces, adopting advanced technologies helped them stay in business by allowing them to compete effectively, and thus to preserve jobs and retain workers. For example, officials at a medium-sized door manufacturer, where we observed numerous robots in the production facility, told us that their firm “could not survive” global competition without the use of advanced technologies. Productivity and Efficiency Gains Adopting advanced technologies has helped some firms improve their product quality and increase their production efficiency. For example, according to officials at a medium-sized fruit processing plant, after the firm began using an automated fruit grading technology, the process took significantly less time and resulted in far fewer complaints from farmers about the grading. Farmers thought the automated grading technology was fairer and more accurate than having workers manually and subjectively grade the fruit. A large appliance manufacturer that began using a collaborative robot to apply sealant to an appliance door observed improved consistency, which led to fewer service calls from retailers and customers about excessive, insufficient, or incorrect seals. One medium-sized door manufacturer said that automation technologies enabled them to produce and ship doors in 3 days, as opposed to 4 to 6 weeks. An official from a warehouse for a regional chain of grocery stores said that using automated guided vehicles allowed the firm to save time moving pallets from one end of the warehouse to the other, and also save worker hours. The warehouse saves just over $2 per pallet moved by an automated guided vehicle rather than a worker, and up to $3,500 a day based on volume, according to the official. Advanced technologies enabled some selected firms to increase production or produce a larger range of goods, and thus to hire additional production workers. This also led to workforce increases for suppliers and other firms, according to officials. One large appliance manufacturer increased its use of robots and other advanced technologies to produce more of its own component parts internally instead of relying on suppliers. As a result, the firm was also able to increase the number of production jobs, according to firm officials. Due to advanced technologies, a small automotive parts manufacturer was able to bid on a contract to produce a new and more intricate part for a major automotive manufacturer. An official described how the part was so intricate that it could not have been produced manually with the required level of consistency and speed. Although the firm adopted six robots to produce this part, winning the contract also created nine new jobs. While the robots are completing much of the production, the volume of parts demanded and the existence of some tasks that only workers can complete has led to this job growth. A developer of autonomous mobile robots said that, as a result of increased business, his firm has created jobs among its eight local suppliers where he buys parts, such as motherboards for the robots. Growth of Developer and Integrator Firms Selected developer firms we met with said they grew their technical and non-technical staff as a result of increasing demand for their technologies. A firm that develops and produces robots had tripled its workforce size, to about 130 employees, in the last year alone, according to officials. An official at another developer firm that makes inspection robots said they had grown from three workers to about 20 and envisions expanding to 100 in the near future. The official said that the firm’s first years were spent on technology development, but that once the technology was deployable to customers, the firm grew its workforce size. Integrator firms that help companies adopt advanced technologies have also grown in size, and new types have emerged, according to integrators we visited. For example, with the development of smarter robots, one integrator firm we visited entered the industry to recondition and sell old robots; the firm also adds newer technology to these robots if requested. This integrator has grown from 35 to 45 employees in the last 10 years, according to officials, with the new positions being primarily robot technician jobs. As a result of technology adoption, some firms hired more workers with technical skills, and in other instances lower-skilled workers, according to firm officials. An official from a warehouse for a regional chain of grocery stores said that adopting an advanced automation system created a need for three additional workers to provide preventive maintenance on the machines. These additional positions pay about 25 percent more than the standard warehouse positions, according to officials. At a large automotive manufacturer, officials told us the firm increased its number of lower-skilled cleaning jobs when robots began producing large amounts of byproduct. At the firms we visited, workers changed roles and tasks as a result of advanced technology adoption, such as focusing more on interactive, cognitive, higher-skilled, and monitoring tasks, and in other cases focusing more on lower-skilled tasks. Workers who can adapt and be flexible to task changes may experience positive effects, including work that is less physically taxing, safer, more ergonomic, less monotonous, or higher paying. On the other hand, workers who are unable to adjust to changing tasks may be negatively affected. Officials at some of the firms told us that their firms provided internal training or leveraged external resources to develop workers’ skills to help them move into new positions. During our visits to selected firms, we saw a variety of ways in which tasks for workers are changing. Interactive work: The use of autonomous mobile robots to deliver prescription drugs for patients enabled nurses at the university-affiliated medical center we visited to focus more of their time on patient interaction, according to officials. The small stamp manufacturer we visited would like to continue to automate its ordering process and focus more on providing customer service. Officials there said for future hires, they plan to recruit for data and people skills, rather than production skills. Cognitive work: A federal statistical agency adopted machine learning technology to automatically interpret text narratives on forms and assign codes to the data. As a result, staff who previously entered this information manually are able to spend more time on analytical tasks such as reviewing the accuracy of the auto-coding, correcting issues, obtaining clarifications about information submitted on the forms, and following up with non-respondents, according to officials. Higher-skilled work: At a large automotive manufacturer, due to increased use of advanced technologies, workers who are hired today need to have greater technical proficiencies than workers hired in the past. For example, to adapt to their changing roles working with robotic equipment, non-technical production staff need machine maintenance and technical skills, rather than only manual dexterity skills. Officials at a large appliance manufacturer that adopted an automated machine to stamp metal said that the resulting process required a single worker to monitor the machine and provide basic maintenance. This worker needed technical skills and at least 6 months of training to effectively perform these duties. In contrast, at another one of this firm’s global plants, four separate pressers are used and each requires workers to load and unload metal. Monitoring work: Officials at the large appliance manufacturer mentioned above showed us a step in their production process in which two small pieces of plastic and metal need to be attached. Three workers used to perform this task by hand, which caused ergonomic challenges, and inconsistencies in both quality and production cycle times. Now, the firm uses three robots to perform this work and a single worker loads the pieces for all three robots and monitors their performance. At a small automotive parts manufacturer, production operators who work in cells with robots monitor multiple machines and sometimes also monitor multiple work cells, so a greater aptitude level is needed. As a result, these operators earn $3 per hour more than operators in work cells without robots, according to a firm official. Less physically taxing work: Staff at some firms also told us how advanced technologies have made worker tasks less physically demanding. For example, we talked with one warehouse worker who used to lift heavy boxes, but who now operates a forklift after his old task was automated with a conveyer belt and sorting system. He described his new position as having ergonomic benefits, including experiencing less back pain. At a large automotive manufacturer, officials said the firm installed six robots to paint vehicle interiors. This production step was a major ergonomic hazard and workers who did this painting had a relatively high injury rate, according to officials. Officials told us that adopting the robots lowered the injury rate among these workers and resulted in faster vehicle painting. Simplified work: At a small stamp manufacturer that adopted a collaborative robot, officials told us that as the firm continues to redesign and optimize operations, the robot will take on more complex tasks. As a result, the remaining production work performed by the firm’s production worker will be simpler (see fig. 7). Officials said that in the future, after the firm’s current production worker retires, the firm may rely on contingent workers to perform any needed production work not completed by the robot because the tasks will be simpler and easier to train a new, temporary worker to complete. Officials said the firm may also hire a worker with a different and more varied skillset who can perform the few remaining production tasks along with other types of tasks. Lower-skilled work: Officials at a medium-sized door manufacturer installed a robot to facilitate the firm’s redesigned door sealant system and production process. The original process of manually applying door sealant was physically-intensive, ergonomically challenging, and required significant skill and experience to precisely apply the sealant. With the new design, a robot applies the sealant autonomously. As a result, workers perform lower-skilled tasks in this process, including placing a piece on a platform, visually inspecting the robot’s work, cleaning and setting up the robot’s work station, and confirming the correct program is entered in the computer. Adaptability to changing daily work demands: Officials from selected firms told us that due to advanced technology adoption, workers need to change tasks depending on the day and circumstances. For example, at a large appliance manufacturer some workers serve in different capacities depending if the robots are functioning properly and depending on the production needs of that day. On the day we visited the plant, several of the robots were malfunctioning and workers were performing the robots’ tasks. Firm officials said that some of their workers serve in swing roles and move around to different production processes and assist as needed. Training Centers for Advanced Tech Skills We met with officials at a training center that re-trains adults and teaches high school students to work with advanced technologies used in manufacturing. We visited two firms in the area that told us that this training center helps fill a local shortage in maintenance technician skills, and that they have hired workers who graduated from the center. Officials at the training center said that there is a high demand in the area for maintenance technicians. For example, they said that a large automotive manufacturer in the area is planning to hire 800 maintenance technicians over the next 3 years, and that the firm is worried about how it will fill these positions. Officials at the training center also said that some firms have such a high demand for maintenance technicians that they hire high school students who complete the training program before they graduate high school. The training center is piloting its adult training program. The program recruits adults who are underemployed and have some mechanical aptitude, then trains them in advanced technologies used in manufacturing. Most of the students who participated in an early pilot obtained higher paying jobs than those they held before the program, according to officials at the training center. Many firms we visited offered training for workers to adapt to their changing roles and tasks, particularly when the tasks or roles became more technical. Some firms used internal training resources and some leveraged local training centers (see sidebar). Some technology developers also offered training to firms that adopted their technologies. Officials at some firms told us that training current workers for more technical positions was easier than finding workers with the appropriate skills. For example, officials at one medium-sized door manufacturer said they needed highly specialized engineers, but could not find any in the region. As a result, this firm offered tuition reimbursement for workers who were willing to go back to school to become engineers. They also partnered with local community colleges to train students to become future maintenance technicians. Officials at a large automotive manufacturer said that due to increases in the firm’s use of advanced technologies, the plant has needed to hire more technicians. As a result, this firm added programs to its on-site training center to train workers for these roles. The complex job changes we observed at the selected firms we visited are not currently captured in federal data, though they may have significant implications for broader employment shifts. As the primary agencies responsible for monitoring the U.S. economy and workforce, the Departments of Commerce and Labor are aware of the importance of advanced technologies as major drivers of changes. For example, Census’ newly administered Annual Business Survey may provide valuable information in the future about the adoption and use of advanced technologies nationwide and the prevalence of resulting workforce effects. However, comprehensive data on firms’ adoption and use of advanced technologies do not currently exist, which prevents federal agencies and others from fully monitoring the spread of advanced technologies throughout the economy and linking their use to changes in employment levels or structural shifts in the tasks and skills associated with jobs. Observations from our visits to selected firms illustrate the complex and varied workforce effects that result from firms’ adoption of advanced technologies. In some circumstances, technology adoption will lead to increases in different types of jobs and in other cases technology adoption will lead to workforce reductions—either over time or immediately. Regardless of the firm-level workforce effects, worker roles and responsibilities are likely to change as advanced technologies take over tasks that workers previously performed. These changes could positively affect some workers, but could also have negative consequences for other workers, especially those who are unable to adapt to changes. For example, workers whose previous work tasks are automated and who are unable to perform new tasks required of them may need to seek new employment. If these changes occur occupation- wide, across many firms, workers may need to re-train or seek new employment in entirely different occupations or industries. To the extent that these changes are concentrated among occupations susceptible to automation, certain groups of workers (e.g., those with lower education levels) may be disproportionately affected and may lack the opportunity to develop skills needed to enter growing occupations. These workers will be in greater need of programmatic or policy supports, and federal workforce programs will need to be aligned with in-demand skills for the changing economy. Without comprehensive data that can measure the magnitude and variety of these firm-level changes, the workforce effects of the adoption of advanced technologies will remain unclear, job seekers may not be fully informed about their best future career prospects, and federally funded programs to support workers may be misaligned with labor market realities. DOL’s ability to collect information regularly on jobs and workers may enable the agency to fill these information gaps. Specifically, better data could be used by policymakers and DOL to proactively design and fund worker training programs that meet the job needs of the future. The Secretary of Labor should direct the Bureau of Labor Statistics (BLS) and the Employment and Training Administration (ETA) to develop ways to use existing or new data collection efforts to identify and systematically track the workforce effects of advanced technologies. For example, the Secretary could select any of the following possibilities, or could identify others. BLS could expand existing worker or firm surveys to ask respondents whether advanced technologies have resulted in worker displacements, work hour reductions, or substantial adjustments to work tasks. BLS could expand its employment projections work to regularly identify occupations projected to change over time due to advanced technologies. ETA could expand the O*NET data system to identify changes to skills, tasks, and tools associated with occupations, as the information is updated on its rotational basis, and consider how this could be used to track the spread of advanced technologies. (Recommendation 1) We provided a draft of this report to DOL, Commerce, NSF, and OSTP for review and comment. We received written comments from DOL that are reprinted in appendix II and summarized below. DOL and Commerce provided technical comments, which we incorporated as appropriate. NSF and OSTP told us that they had no comments on the draft report. DOL agreed with our recommendation to develop ways to identify and track the workforce effects of advanced technologies. DOL stated that it will continue coordinating with the Census Bureau on research activities in this area, and that it plans to identify and recommend data collection options to fill gaps in existing information about how the workplace is affected by new technologies, automation, and AI. DOL also stated that it plans to release employment projections annually instead of every 2 years, beginning in 2019. As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees, the Secretary of Labor, the Secretary of Commerce, the Director of the National Science Foundation, the Director of the White House Office of Science and Technology Policy, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (202) 512-7215 or brownbarnesc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. The objectives of this review were to examine (1) what is known about how the adoption of advanced technologies affects the U.S. workforce; (2) selected federal agency efforts to track and monitor the adoption and workforce effects of advanced technologies; (3) considerations that led selected firms to adopt advanced technologies and the risks they faced; and (4) ways technology adoption has affected the workforce at selected firms. Throughout the report, we use “advanced technologies” as a broad term to describe technological drivers of workforce changes, including but not limited to those identified in the National Academies study: artificial intelligence; machine learning; robotics; autonomous transport; advanced manufacturing; 3D printing; advanced materials; computing power; and internet and cloud technology. The technologies we observed at work sites could generally be categorized as applications of robotics, machine learning (e.g., machine vision or autonomous navigation), or both. However, not all technologies that may affect the U.S. workforce in the future—through automation or in other substantial ways—fall into these categories. Our use of the broad term “advanced technologies” leaves open the possibility that new technologies and other areas of focus are likely to emerge. To examine what is known about how the adoption of advanced technologies affects the U.S. workforce, we explored the extent to which available federal data could identify and measure these effects, and we identified limitations with available data. Because there was no comprehensive data that link employment trends to technology adoption, we used a study by Frey and Osborne to identify a group of occupations susceptible to automation. We then analyzed whether the concentration of these occupations in industries is correlated with growth in tech jobs or employment declines in those industries, whether job displacements are more common in these occupations than in others, the characteristics of workers who hold jobs in these occupations, and the geographic concentration of jobs in these occupations. We analyzed employment data from the Census Bureau (Census) and the Bureau of Labor Statistics (BLS); specifically, the American Community Survey (ACS), the Current Population Survey’s (CPS) Displaced Worker Supplement, and the Occupational Employment Statistics (OES) survey. For more information, see detailed discussions of our data analyses in sections 1-3 below. Identifying occupations susceptible to automation: Using a model that evaluates tasks within an occupation, Frey and Osborne estimate a probability of automation for 702 occupations. They identify occupations with a probability greater than 0.7 as being at high risk of automation. In our analyses, we thus consider this collection of occupations as those susceptible to automation. While there are different studies that attempt to predict what occupations or jobs may be automated in the future, we use the work by Frey and Osborne because it is widely cited and because its results are structured to allow us to identify a broadly inclusive collection of occupations susceptible to automation. The results of our analyses could be affected by using other studies to the extent that they identify different occupations as susceptible to automation. The accuracy of any collection of occupations is limited by the unpredictability of when or if jobs are automated, as well as the fact that occupations are comprised of a variety of jobs, which may experience automation to varying degrees or in different ways. We also reviewed examples of recent and ongoing studies that attempt to measure workforce effects directly attributable to technology adoption. We identified examples of research through interviews with knowledgeable individuals and from among those included in a recent review of the state of empirical work. Our review of studies was not meant to be comprehensive of the research in this area. To identify selected federal agencies’ current and planned efforts to collect data on, and monitor the prevalence and effects of advanced technologies in the economy, we met with the Departments of Labor (DOL) and Commerce (Commerce), as the principal federal agencies responsible for collecting data on the U.S. economy and workforce; the White House Office of Science and Technology Policy (OSTP), which leads interagency science and technology policy-coordination efforts across federal agencies; and the National Science Foundation (NSF), which was involved in the development of the Annual Business Survey. We interviewed officials and reviewed data and information collected by these agencies. We also reviewed the Annual Business Survey’s questionnaire to consider the potential uses of data being collected by the survey, and analyzed data from DOL’s Employment Projections program and Occupational Information Network (O*NET) database to identify information related to the adoption and workforce effects of advanced technologies. Annual Business Survey: The Annual Business Survey was administered for the first time in summer 2018, and collects information from firms about various topics, including innovation and technology use. The survey is a joint effort by the Census Bureau and the National Center for Science and Engineering Statistics within the National Science Foundation and Census plans to administer the survey annually for 5 years. The Annual Business Survey replaces the 5-year Survey of Business Owners, the Annual Survey of Entrepreneurs, the Business R&D and Innovation for Microbusinesses survey, and the innovation section of the Business R&D and Innovation Survey. Employment Projections program: BLS’s Employment Projections program analyzes changes in the economy, among other things, to project how employment by occupation may change over the next 10 years, including which occupations may be affected by advanced technologies. BLS’s projections are for the most part structured around the Occupational Employment Statistics, which produces employment and wage estimates for over 800 occupations. As part of this program, BLS develops a table of occupations that are projected to have direct employment changes due to some identified reason. According to BLS officials, the specific reason listed for each occupation is based on BLS’s judgment of the most significant factor or factors affecting the occupation (i.e., based on a qualitative assessment). We examined the reasons listed in this table and identified those related to the adoption of advanced technologies in an occupation, such as through automation, the increased use of robots or artificial intelligence, advances in machine or software technologies, or other similar changes. We then counted the number of unique occupations projected to experience declines in their shares of employment in an industry or group of industries due to one of these reasons. We also counted these occupations according to their major occupation group. BLS projected that some of these occupations would experience employment share declines in all industries and some would experience employment share declines in a single industry only. We counted unique occupations regardless of what industries or how many were noted (e.g., all industries or only one). We chose to do this to capture an inclusive list of occupations projected to be affected by advanced technologies, and because we are not using the list to quantify total projected employment changes. Of the 247 unique occupations BLS includes in its table as projected to have direct employment changes due to some identified reason, BLS projects that 163 will experience employment share declines— 100 of those occupations are projected to change broadly as a result of the adoption of advanced technologies. An employment share decline indicates that employment in an occupation will decline relative to others in a given industry or group of industries, not that the occupation will necessarily experience a decrease in employment in absolute terms. Occupational Information Network (O*NET) database: The O*NET database contains information about the skills, tasks, and tools (i.e., use of technology) associated with specific occupations. We downloaded two components of the database that (1) list the various work tasks associated with each occupation, and (2) list the various tools and technologies used by each occupation. In each database component, we searched for and identified tasks, tools, and technologies that involved robots in some way—e.g., tasks such as working with robots, robotic systems, or robotic applications, and tools such as welding robots, loading robots, or robot automation tools. We then counted the number of unique occupations that (1) had an associated work task related to robots, or (2) used a robot-related tool in the occupation. To understand firms’ adoption of advanced technologies and any resulting workforce effects, we met with officials representing 16 different firms that are using advanced technologies in their operations, as well as a systems integrator who provided detailed information about how several customer firms are using advanced technologies. Most of the meetings with firms were in-person site visits; three of the meetings with firms and the meeting with the systems integrator were by phone. Throughout this report, we use the term “firm” for simplicity, although the “firms” we met with included production plants of large manufacturers, single-location firms, public sector agencies, and other entities (see below). We also identify the manufacturing firms we visited as falling into one of three different size groups to describe their relative size differences from each other. The manufacturing firms we visited ranged from eight employees to thousands, according to firm officials. For the purposes of our study, we define small as fewer than 200 employees; medium as 200 employees to 1,000; and large as over 1,000 employees. Among the 16 firms we met with that are using advanced technologies, 10 are manufacturing firms: a small manufacturer of rubber stamps and embossing seals (also referred to as a small stamp manufacturer); two medium-sized door manufacturers; a small automotive parts manufacturer; a medium-sized automotive parts manufacturer; two large appliance manufacturers; a large automotive manufacturer; a large manufacturing corporation of household and personal care a medium-sized fruit processing plant. Six are non-manufacturing firms of various types: a construction consulting company; a federal statistical agency; a food retail corporation; a municipal township; a university-affiliated medical center; and a warehouse for a regional grocery store chain. The firms about which we received information from the systems integrator were business, administrative, and customer relations offices of various firm types. To identify firms to meet with, we consulted and sought referrals from a variety of knowledgeable sources, including academic researchers, technology developer firms, technology integrator firms, state economic development associations, and our own research. We selected firms that varied in size, industry sector, types of advanced technology used, and geography. We limited our focus to firms that had adopted advanced technologies and had experienced workforce effects. Our selection of firms is not a generalizable sample, but does provide illustrative examples of the adoption and workforce effects of advanced technologies. During our site visits at firms, we met with one or more management officials and, at times, with workers. We were also able to view the advanced technologies being used in operations. Our discussions with officials included topics such as motivations for adopting advanced technologies, the integration process, and any workforce effects that resulted from the technologies, including positions lost or gained and how workers’ tasks and skills may have changed. Our site visits and interviews with firm officials ranged from hour-long conversations to full-day visits, so some site visits yielded more detailed information than others. In addition to the firms that use advanced technologies, we interviewed seven technology developer firms and two robotics integrator firms (in addition to the systems integrator mentioned above). We met with these firms to learn more about some of the technologies being used and the adoption process, as well as about workforce effects at these firms. We identified these developer and integrator firms from various sources, including our conversations with academic researchers and our own research. We conducted additional interviews to obtain background and context for our work. We met with individuals knowledgeable about issues related to the adoption and workforce effects of advanced technologies, such as academic researchers and economists, officials from two unions representing manufacturing workers, officials at three industry-based organizations, officials from two state economic development associations, and officials at two worker training centers. For all objectives, we also reviewed relevant federal laws and regulations. The remainder of this appendix provides detailed information about the data and quantitative analysis methods we used to examine what is known about the workforce effects of automation and the adoption of advanced technologies (objective 1), as follows: Section 1: Analyses using data from the ACS Section 2: Analyses using data from the CPS’s Displaced Worker Section 3: Analyses using data from the OES survey For each of the datasets described below, we conducted a data reliability assessment of variables included in our analyses. We reviewed technical documentation and related publications and websites with information about the data. We spoke with BLS and Census officials who maintain the datasets to gain an understanding of and provide context for the various data that we analyzed, as well as to resolve any questions about the data and to identify any known limitations. We also tested the data, as applicable, to check for logical consistency, missing data, and consistency with data reported in technical documentation. We determined that the variables we used from the data we reviewed were sufficiently reliable for the purposes of this report. This section describes the quantitative analysis methods we used to examine employment trend correlations and the characteristics and earnings of workers in occupations susceptible to automation (as identified by Frey and Osborne; see above). We used ACS data for these analyses. The ACS is administered by the Census Bureau and is an ongoing national survey that uses a series of monthly samples to produce annually updated estimates for the same areas surveyed via the decennial census. The ACS collects a range of information about individuals from a large sample of households—over 2.2 million respondent households in 2016—including employment information such as occupation, industry, and earnings, and demographic information such as age, gender, race, ethnicity, and educational attainment. We limited our analysis to workers who were classified as current employees, and who had earned positive wage and salary income in the prior 12 months. In 2016, this resulted in observations representing 136 million workers, close to the number reported by BLS for that same period using a different survey. This report primarily used ACS data from 2010 through 2016—specifically, we relied on the Census Bureau’s Public Use Microdata Sample of the ACS for the single years 2010, 2011, 2012, 2013, 2014, 2015, and 2016. To test whether industries with higher concentrations of individuals in occupations susceptible to automation (as identified by Frey and Osborne) have experienced employment changes, we examined their correlation with changes in tech job concentration and changes in overall employment from 2010 through 2016. We limited the analysis to this period both because the ACS occupation codes changed in 2010 and because it allowed our results to post-date the economic recession of 2007-2009. We used industry definitions set by the ACS data, which groups some industries together—e.g., residential and nonresidential construction industries are combined in a single construction industry grouping. We defined tech jobs as those in computing, engineering, and mathematics occupations, consistent with previous GAO work on the tech field. We also examined an alternative definition of tech jobs in which we included those with “computer” in the occupation title. For both definitions, we estimated the number of tech jobs in each industry in each year, 2010-2016. We then calculated the growth rate in the number of tech jobs in each industry, and correlated that growth rate with the percentage of workers in that industry in occupations susceptible to automation (as identified by Frey and Osborne). We also estimated the number of workers overall in each industry in each year (2010-2016) and correlated the trend in total employment with the percentage of workers in that industry in occupations susceptible to automation (as identified by Frey and Osborne). We restricted our correlation analyses to those industries where the tech job growth rate or the overall employment trend was statistically significant. We performed two correlation tests. The Spearman test measures correlation between the rank of the two sets of values. The Pearson test measures correlation between the values themselves. As shown in table 2, we found a positive but weak correlation between industries with higher concentrations of jobs susceptible to automation and their concentration of tech jobs, based on both correlation tests and both definitions of tech jobs, and we found no meaningful correlation with change in overall employment in either test. To explore an example industry—the plastics product manufacturing industry—in further detail, we identified the number of jobs susceptible to automation within that industry, by occupation and groups of occupations. We also examined the growth in tech jobs within the industry, by tech occupation. We approximated each occupation’s contribution to the overall growth of tech jobs in the industry by multiplying their individual growth rates over the period 2010-2016 by their employment in 2010. The growth rates for the three engineering occupations, which when combined, account for more than half of the industry’s growth in tech jobs, were each significant at the 85 percent confidence level. To analyze the characteristics of workers in occupations susceptible to automation (as identified by Frey and Osborne), as well as the characteristics of workers with tech jobs, we used 2016 ACS data. We examined data on the workers’ gender, level of education, age, race and ethnicity, and hourly wage, and compared distributions of workers in occupations susceptible to automation and workers in all other occupations (see table 3). For race and ethnicity categories, we included only non-Hispanic members of White, Black, Asian, and Other categories, and the Hispanic category included Hispanics of all races. The “Other” category included American Indian or Alaskan Native, Native Hawaiian or Pacific Islander, two or more races, and other race. To analyze education level, we combined all attainment levels from a high school degree or less. To estimate the hourly wage of workers, we divided the wage and salary earnings of the worker by their usual hours worked and weeks worked. To test the reliability of this measure, we compared our results to average hourly wages reported by other BLS surveys; we found that the average values were sufficiently close to determine that this method was sufficiently reliable for our purposes. To investigate whether differences in hourly wage might be due to other factors, we estimated multiple regression models that enabled us to control for additional variables. Specifically, we estimated wage differences between workers in occupations susceptible to automation and workers in other occupations—i.e., whether a worker was in an occupation susceptible to automation (as identified by Frey and Osborne) was our primary independent variable (a binary, yes/no variable). Because we used the natural log of the hourly wage as the dependent variable, the standard interpretation of the regression coefficient of this variable is that it represents the average log point difference in hourly wages between occupations susceptible to automation and all other occupations. This coefficient can be made to more closely approximate a percentage difference in hourly wages or an earnings gap by taking the exponent and subtracting 1. As noted previously, we limited our analysis to workers who earned positive wage and salary income in the prior 12 months. We also removed observations with outlier values for wages (e.g., wage rates above $140 per hour); this represented about 1 percent of the sample in 2016. We ran five regression models with different sets of independent variable controls. Regression (1) estimates the earnings gap without any controls (the uncorrected earnings gap). Regression (2) estimates the earnings gap with a set of independent variables that control for characteristics of the individual; these variables included age, race and ethnicity, gender, marital status, state of residence, and education level. Regression (3) estimates the earnings gap with independent dummy variables for 2-digit industry codes added; this corrects for any differences between industries at the 2-digit level. Regression (4) estimates the earnings gap with independent dummy variables for 2-digit occupation codes added; this corrects for any differences between occupations at the 2-digit level. Regression (5) includes both 2-digit industry and 2-digit occupation code dummy variables. As table 4 shows, we found a significant difference in hourly wages between workers in occupations susceptible to automation compared to workers in other occupations, even after independent variables to control for worker characteristics, industry, and occupation codes were included. Including the additional independent variables caused the earnings gap to fall from just over -34 percent to just over -10 percent. Regression model 3, which estimated an earnings gap of about -17.2 percent, is our preferred model, as it controls for individual worker characteristics and for any differences between industries at the 2- digit level, but does not include occupation as an independent variable. Including occupation variables controls for any differences between occupations at the 2- digit level. However, because we identify workers in jobs susceptible to automation based on their occupations, these occupation control variables are likely highly predictive of Frey and Osborne’s estimated probability of automation, which is used to categorize workers in jobs susceptible to automation. We also ran these regression models for other years from 2010 to 2016 and we found substantively similar results. This section discusses the quantitative analysis methods we used to compare relative job displacement rates between workers in occupations susceptible to automation (as identified by Frey and Osborne; see above) and workers in other occupations. We used data from the CPS’s Displaced Worker Supplement for these analyses. The CPS is sponsored jointly by Census and BLS and is the source of official government statistics on employment and unemployment in the United States. The basic monthly survey is used to collect information on employment, such as employment status, occupation, and industry, as well as demographic information, among other things. The survey is based on a sample of the civilian, non-institutionalized population of the United States. Using a multistage stratified sample design, about 56,000 households are interviewed monthly based on area of residence to represent the country as a whole and individual states; the total sample also includes additional households, some of which are not interviewed in a given month for various reasons, such as not being reachable. The CPS Displaced Worker Supplement has been administered every other year since 1984, and provides supplemental data on persons age 20 years or older who lost a job involuntarily in the prior 3 years, including data on reasons for job displacement, as well as industry and occupation of the former job. This report used data from the January 2016 Displaced Worker Supplement. To analyze whether workers in occupations susceptible to automation (as identified by Frey and Osborne) experience job displacement at differing rates than workers in other occupations, we used data from the CPS’s January 2016 Displaced Worker Supplement. We identified workers who lost or left a job involuntarily during the 3 calendar years prior to the survey (i.e., January 2013 through December 2015) because their position or shift was abolished or because there was insufficient work for them to do. We focused on these reasons for displacement as those that most closely approximate how advanced technologies could replace workers at a given firm. We also limited our analysis to those workers who did not expect to be recalled to their jobs within the next 6 months. We categorized these displaced workers according to the occupations from which they were displaced (e.g., workers displaced from occupations susceptible to automation and workers displaced from all other occupations). We calculated relative job displacement rates as the number of displacements over the period 2013-2015 reported by a given population (e.g., workers in occupations susceptible to automation), over that population’s total current employment in January 2016. Although this measure does not represent the total number of jobs that existed annually that could have resulted in displacements, it allows us to control for population size and to approximate a relative displacement rate. We examined various populations, including occupations identified as susceptible to automation by Frey and Osborne, occupations BLS projects will experience declines in their share of employment due to advanced technologies (see above), and production occupations. To categorize occupations, Frey and Osborne and BLS use Standard Occupational Classification (SOC) codes, whereas the Displaced Worker Supplement uses Census occupation codes. We used a crosswalk provided by Census to match these occupation classifications. SOC codes have a hierarchical structure—e.g., a “broad” occupation group contains a subset of “detailed” occupations. For example, SOC code 13- 1031 is the detailed occupation “claims adjusters, examiners, and investigators” within the broad group SOC 13-1030 (“claims adjusters, appraisers, examiners, and investigators”). When a direct crosswalk between SOC and Census occupation codes was not available at the detailed level, we used the associated broad SOC group to identify a Census occupation code. There were some respondents in the Displaced Worker Supplement who did not report the occupation from which they were displaced, and these were dropped from our analysis. To estimate the sampling errors for each estimate, we used strata defined by state because the Displaced Worker Supplement data did not provide replicate weights or the sampling strata necessary to obtain standard errors. When estimating the number of job displacements over the period 2013-2015 reported by a given population (e.g., workers in occupations susceptible to automation), we used the supplement weight for respondents. When estimating the population’s total current employment in January 2016, we used the CPS 2016 weight for respondents. We used a Taylor series linearization to estimate the sampling error of the ratio of estimated number of job displacements over the period 2013- 2015 to the estimated number of current employment in 2016. While our primary analysis examined relative displacement rates for workers in occupations susceptible to automation, we also conducted sensitivity analyses by considering other groups of occupations. Specifically, we examined the relative displacement rates of the following groups: Jobs susceptible to automation had a relative displacement rate of 3.4 percent +/- 0.3, and all other jobs combined had a relative displacement rate of 2.9 percent, +/- 0.2. Jobs in occupations BLS projects will experience relative declines in employment due to advanced technologies (see above) had a relative displacement rate of 3.7 percent, +/- 0.5, and all other jobs combined had a relative displacement rate of 3.6 percent, +/- 0.2. Jobs in production occupations had a relative displacement rate of 3.7 percent +/- 0.8, and all other jobs combined had a relative displacement rate of 3.1 percent, +/- 0.2. This section discusses the quantitative analysis methods we used to analyze geographic reliance on occupations susceptible to automation (as identified by Frey and Osborne; see above). We used OES data for these analyses. The OES survey is a federal-state cooperative effort between BLS and state workforce agencies, which collects information on occupational employment and wage rates for wage and salary workers in nonfarm establishments. The survey is based on a sample drawn from about 7.6 million in-scope nonfarm establishments in the United States that file unemployment insurance reports to the state workforce agencies. Using a stratified sample design, about 200,000 establishments are surveyed semiannually and employment estimates are based on six panels of data collected over a 3-year cycle. The final in-scope sample size when six panels are combined is approximately 1.2 million establishments. The OES survey includes all full- and part-time wage and salary workers in nonfarm industries, but excludes self-employed workers, owners and partners in unincorporated firms, household workers, and unpaid family workers. OES data provide occupational employment estimates by industry for the country as a whole, for individual states, and for more local geographic areas (e.g., metropolitan and nonmetropolitan areas). This report used data from the May 2017 Occupational Employment Statistics. To analyze what U.S. geographic areas rely more heavily on employment in occupations susceptible to automation, we used data from the May 2017 OES. For each local geographic area, we estimated how many jobs were in occupations identified as susceptible to automation by Frey and Osborne (see above) and how many jobs were in all other occupations. We also estimated how many jobs were in each group of occupations nationwide (using national-level data). We then calculated a location quotient for each local geographic area, which measures the proportion of each area’s jobs that were in occupations susceptible to automation compared to the national proportion of employment in these occupations. This measure depicts the extent to which a local geographic area relies on certain jobs for the employment of its population, relative to other areas. Based on their location quotients, we categorized and mapped 589 local geographic areas in the following three groups: Relatively High Concentration: Areas where the proportion of jobs susceptible to automation is at least 5 percentage points greater than the national average, and the difference is statistically significant at the 95 percent confidence level. This translates to an estimated location quotient of at least 1.1. Average or Relatively Low Concentration: Areas where the proportion of jobs susceptible to automation is within 5 percentage points above the national average or lower. Undetermined Reliance: Areas where the proportion of jobs susceptible to automation is undetermined. We classify an area’s proportion as “undetermined” if the estimated margin of error at the 95 percent confidence level is larger than 5 percentage points. We conducted one sided z-tests at the 95 percent confidence level to analyze each area’s estimated location quotient. The null hypothesis is that the area location quotient is less than or equal to 1.1 (i.e., the proportion of employment in the group of occupations in an area is 1.1 times the national proportion). The alternative hypothesis is that the area location quotient is greater than 1.1. Because estimated area employment proportions are based on a sample, we also restricted our tests to those areas that were reliable for our purposes by requiring that areas had sampling errors of no greater than 5 percentage points for a 95 percent confidence interval. According to BLS, employment estimates for individual occupations in individual local geographic areas may not be available in the public data for a variety of reasons, including for example, failure to meet BLS quality standards or to ensure the confidentiality of survey respondents. Because we aggregate data across multiple occupations, our methodology treats these cases as if employment in the given occupation in the given area was zero, which is not the case and which introduces imprecision into our analysis and the resulting location quotients. However, because ensuring confidentiality is a primary concern, we assume that most of these cases where data are suppressed would have relatively small numbers of jobs, and thus have minimal effects on our results. To test this assumption and to ensure the appropriateness of our methods, we compared the total number of jobs we analyzed across all local geographic areas to the total number of jobs reported at the national level (which do not have data suppressed). The total number of jobs analyzed across our local geographic areas was 5.5 percent lower than the total number of jobs reported at the national level, which we concluded was within an acceptable threshold to determine that the data were sufficiently reliable for our purposes and our analysis. In addition, according to BLS, because occupational employment estimates are rounded to the nearest 10 before publication, estimates of location quotients calculated from the public data will be subject to some rounding error, compared with location quotients calculated from the unrounded pre-publication data. In addition to the contact named above, Blake Ainsworth (Assistant Director), Michael Kniss (Analyst-in-Charge), Shilpa Grover, and John Lack made key contributions to this report. Also contributing to this report were James Bennett, Benjamin Bolitzer, Melinda Cordero, Holly Dye, Jonathan Felbinger, Sheila R. McCoy, Jean McSween, James Rebbe, Krishana Routt-Jackson, Benjamin Sinoff, Almeta Spencer, and Sonya Vartivarian.