For the 2020 Census, the Bureau intends to limit its per-household cost to not more than that of the 2010 Census, adjusted for inflation. To achieve this goal, the Bureau is significantly changing how it conducts the census, in part by re-engineering key census-taking methods and infrastructure. The Bureau’s innovations include (1) using the Internet as a self-response option; (2) verifying most housing unit addresses using “in-office” procedures rather than costly field canvassing; (3) in certain instances, replacing enumerator-collected data with administrative records (information already provided to federal and state governments as they administer other programs) and, (4) re-engineering field data collection methods. The Bureau’s various initiatives have the potential to make major contributions toward limiting cost increases. In October 2015, the Bureau estimated that with its new approach it can conduct the 2020 Census for a life-cycle cost of $12.5 billion, $5.2 billion less than if it were to repeat the design and methods of the 2010 Census (both in constant 2020 dollars). Table 1 below shows the Bureau’s estimated cost savings it hopes to achieve in 4 innovation areas. Sufficient testing, while important to the success of any census, is even more critical for the Bureau’s preparations for 2020. To help control costs and maintain accuracy, the 2020 Census design includes new procedures and technology that have not been used extensively in earlier decennials, if at all. While these innovations show promise for a more cost-effective head count, they also introduce new risks. As we have noted in our prior work, it will be important to thoroughly test the operations planned for 2020 to ensure they will (1) produce needed cost savings, (2) function in concert with other census operations, and (3) work at the scale needed for the national head count. The Bureau’s failure to fully test some key operations prior to the 2010 Census was a key factor that led us to designate that decennial as one of our high-risk areas. The 2016 test was the latest major test of NRFU in the Bureau’s testing program. In 2014, the Bureau tested new methods for conducting NRFU in the Maryland and Washington, D.C., area. In 2015, the Bureau assessed NRFU operations in Maricopa County, Arizona. In 2018, the Bureau plans to conduct a final “End-to-End” Test which is essentially a dress rehearsal for the actual decennial. The Bureau needs to finalize the census design by the end of fiscal year 2017 so that key activities can be included in the End-to-End Test. The Bureau plans to conduct additional research and testing through 2018 in order to further refine the design of the 2020 Census but recently decided to alter its approach. On October 18, 2016, the Bureau announced plans to stop two field test operations planned for fiscal year 2017 to mitigate risks from funding uncertainty. The Bureau said it would stop all planned field activity, including local outreach and hiring, at its test sites in Puerto Rico, North and South Dakota, and Washington State. The Bureau will not carry out planned field tests of its mail out strategy and NRFU in Puerto Rico. The Bureau also cancelled plans to update its address list in the Indians lands and surrounding areas in the three states. However, the Bureau said it will continue with other planned testing in fiscal year 2017, such as that focusing on systems readiness and internet response. Further, the Bureau said it would consider incorporating the stopped field activity elements within the 2018 End-to-End Test. The Bureau maintains that stopping the 2017 Field Test will help prioritize readiness for the 2018 End-to-End Test, and mitigate risk. Nevertheless, as we noted in our November 2016 testimony, it represents a lost opportunity to test, refine, and integrate operations and systems, and puts more pressure on the 2018 End-to-End Test to demonstrate that enumeration activities will function as needed for 2020. NRFU generally proceeded according to the Bureau’s operational plans. For example, the Bureau demonstrated procedures for quality assurance and training. On the other hand, according to preliminary 2016 Census Test data, there were 19,721 NRFU cases coded as non-interviews in Harris County, Texas and 14,026 in L.A. County, California, or about 30 and 20 percent of the test workload respectively. According to the Bureau, non-interviews are cases where no data or insufficient data were collected, either because enumerators made six attempted visits without success (the maximum number the Bureau allowed), or visits were not completed due to, for example, language barriers or dangerous situations. In such cases for the 2020 Census, the Bureau may have to impute attributes of the household based on the demographic characteristics of surrounding housing units as well as administrative records. According to Bureau officials, they are not certain why there were so many non-interviews for the 2016 Census Test and are researching potential causes. Bureau officials told us that they expect higher numbers of non-interviews during tests in part, because, compared to the actual enumeration the Bureau conducts less outreach and promotion. While the 2016 Census Test interview rate is not necessarily a precursor to the 2020 non-interview rate, because of its relationship to the cost and quality of the census, it will be important for the Bureau to better understand the factors contributing to it. Bureau officials hypothesized that another contributing factor could be related to NRFU methods used in the 2016 Census Test compared to earlier decennials. For the 2010 and prior censuses, enumerators collected information during NRFU using pencil and paper. Enumerators may have visited a housing unit more than the six maximum allowable visits to obtain an interview but did not record all of their attempts, thus enabling them to achieve a higher completion rate. For the 2020 Census, and as tested in 2016, the Bureau plans to collect data using mobile devices leased from a contractor, and an automated case management system to manage each household visit (see figure 1). The Bureau believes that this approach will provide a faster, more accurate, and more secure means of data collection. Unlike previous censuses and one prior test, enumerators in the 2016 Census Test did not have an assigned set of cases that they alone would work until completion. Instead, the Bureau relied on an enhanced operational control system that was designed to provide daily assignments and street routing of NRFU cases to enumerators in the most optimal and efficient way. At the same time, the mobile device and automated case management system did not allow an enumerator to attempt to visit a housing unit more than once per day, reopen a closed case, or exceed the maximum allowable six attempts. One factor we observed that may have contributed to the non-interview rate was that enumerators did not seem to uniformly understand or follow procedures for completing interviews with proxy respondents (a proxy is someone who is a non-household member, at least 15 years old, and knowledgeable about the NRFU address). According to the 2016 Census Test enumerator training manual, when an eligible respondent at the address cannot be located, the automated case management system on the mobile device will prompt the enumerator when to find a proxy to interview, such as when no one is home or the housing unit appears vacant. In such circumstances, enumerators are to find a neighbor or landlord to interview. However, in the course of our site visits, we observed that enumerators did not always follow these procedures. For example, we observed that one enumerator, when prompted to find a proxy, looked to the left and then right and, finding no one, closed the case. Similarly, another enumerator ignored the prompt to find a proxy and explained that neighbors are usually not responsive or willing to provide information about the neighbor and did not seek to find a proxy. Enumerators we interviewed did not seem to understand the importance of obtaining a successful proxy interview and many appeared to have received little encouragement during training to put in effort to find a proxy. Proxy data for occupied households are important to the success of the census because the alternative is a non-interview. In 2010 about one- fourth of the NRFU interviews for occupied housing units were conducted using proxy data. We shared our observations with Bureau officials who told us that they are aware that enumerator training for proxies needs to be revised to convey the importance of collecting proxy data when necessary. Converting non-interviews by collecting respondent or proxy data can improve interview completion rates, and ultimately the quality of census data. The Bureau told us it will continue to refine procedures for 2020. According to the Bureau, its plans to automate the assignment of NRFU cases have the potential to deliver significant efficiency gains. At the same time, improving certain enumeration procedures and communicating better could produce additional efficiencies by enabling the Bureau to be more responsive to situations enumerators encounter in the course of their follow-up work. Enumerators were unable to access recently closed incomplete cases. Under current procedures, if an enumerator is unable to make contact with a household member, the case management system closes that case to be reattempted at a later date, perhaps by a different enumerator; assuming fewer than six attempts have been made. Decisions on when re-attempts will be made—and by whom—are automated and not designed to be responsive to the immediate circumstances on the ground. This is in contrast to earlier decennials when enumerators, using paper-based data collection procedures, had discretion and control over when to re-attempt cases in the area where they were working. According to the Bureau, leaving cases open for re- attempts can undermine the efficiency gains of automation when enumerators depart significantly from their optimized route, circling back needlessly to previously attempted cases rather than progressing through their scheduled workload. During our test site observations, however, we found how this approach could lead to inefficiencies in certain circumstances. For example, we observed enumerators start their NRFU visits in the early afternoon as scheduled, when many people are out working or are otherwise away. If no one answered the door, those cases were closed for the day and reassigned later. However, if a household member returned while the enumerator was still around, the enumerator could not reopen the case and attempt an interview. We saw this happen at both test site locations, typically in apartment buildings or at apartment-style gated communities, where enumerators had clear visibility of a large number of housing units and could easily see people arriving home. Bureau officials acknowledged that closing cases in this fashion represented a missed opportunity and plan to test greater flexibilities as part of the 2018 End-to-End Test. Programming some flexibility into the mobile device—if accompanied with adequate training on how and when to use it—should permit some interviews to be completed without having to deploy staff to the same case on subsequent days. This in turn could reduce the cost of follow-up attempts and improve interview completion rates. Enumerators did not understand procedures for visits to property managers. Property managers are a key source of information on non- respondents when enumerators cannot find people at home. They can also facilitate access to locked buildings. Further, developing a rapport with property managers has helped the NRFU process, such as when repeated access to a secured building or residential complex is needed on subsequent days by different enumerators. In response to problems observed during the Bureau’s 2014 and 2015 Census tests and to complaints from property managers about multiple uncoordinated visits by enumerators, the Bureau’s 2016 Census Test introduced specific procedures to conduct initial visits to property managers in large multi-unit apartment buildings. The procedures sought to identify up front which, if any, units needing follow-up were vacant, eliminating the need for enumerators to collect this information from property managers with subsequent visits on a case-by-case basis. According to Bureau officials, the automated case management system was designed to allow for an enumerator to make up to three visits to property managers to remove vacant units. According to the Bureau, the 2016 Census Test demonstrated that vacant units could quickly be removed from the NRFU workload using these procedures in cases where a property manager was readily available; however, in other cases the procedures caused confusion. For example, whenever an initial visit was unsuccessful, all of the cases at that location—up until then collated into only one summary row of the enumerator’s on-screen case list—would suddenly expand and appear as individual cases to be worked, sometimes adding several screens and dozens of cases to the length of the list, which the enumerators we spoke with found confusing. Furthermore, without the knowledge of which units were vacant, enumerators may have unnecessarily visited these units and increased the cost and the time required to complete NRFU. During debriefing sessions the Bureau held, enumerators and their supervisors identified training in these procedures as an area they felt needed greater attention in the future. Bureau officials said that they are pleased that the test demonstrated their progress in automating case management at multi-unit locations, but at the same time, they recognize the need to better refine the initial property manager contact procedures and integrate multi-unit procedures into the training. During our field visits, we encountered several instances where enumerators had been told by a respondent or otherwise learned that returning at a specific time on a later date would improve their chance of obtaining an interview from either a household respondent or a property manager. According to the Bureau, while there was a mechanism for capturing and using this information, it was not uniformly available to the enumerators, nor did the enumerators always use the mechanism when appropriate. As a result, the Bureau’s 2016 Census Test and automated case management system did not have an efficient way to leverage that information. Attempting to contact non-responding households at times respondents are expected to be available can increase the completion rate and reduce the need to return at a later date or rely on proxy interviews as a source of information. The Bureau’s automated case management system used estimated hour- by-hour probabilities for the best time to contact people when making enumerator assignments. The estimation relied on various administrative records, information from other Bureau surveys that had successful contacts in the past, as well as area characteristics. The 2016 Census Test did not have a way to change or update these estimates when cases were subsequently reassigned. The assigned time windows were intended to result in more productive visits and reduce costs. When enumerators identified potentially better times to attempt a contact, they were instructed to key this information into their mobile devices. For example, one enumerator keyed in a mother’s request to come back Thursday afternoon when her kids were in camp, while others keyed in information like office hours and telephone contact numbers obtained from signs on the property they had seen for property managers. However, according to the Bureau this updated information went unused, and we met enumerators who had been assigned to enumerate addresses at the same unproductive time after they had written notes documenting other better times to visit. Another enumerator reported visiting a property manager who complained that the enumerator was not honoring the manager’s earlier request made during a prior enumeration attempt that an enumerator return during a specified time window. Such repeat visits can waste enumerator time (and miles driven), and contribute to respondent burden or reduced data quality when respondents become annoyed and may become less cooperative. We discussed our preliminary observation with Bureau managers at the test sites, who expressed frustration that the automated case management system did not allow them to use the locally-obtained data on when to contact people whom they found in enumerator notes in a way to affect future case assignment. Headquarters staff told us that while they have not fully evaluated this yet, they are concerned that providing local managers with too much flexibility to override the results of optimized case and time assignments would undermine the efficiency gains achievable by the automation. They also explained that enumerators were provided the capability to record what day or what time of day for follow-up. This information could have been used by the automated case management system to better target the timing of future assignments. However, they acknowledged that this procedure may not have been explained during enumerator training. Reviewing the enumerator training manual, we confirmed that there were no procedures to allow enumerators to systematically record what day or what time of day to follow-up at a housing unit. Bureau officials have said that this is another area they are looking into and plan to address. The key innovations the Bureau plans for 2020 show promise for controlling costs and maintaining accuracy, although there are significant risks involved. The Bureau is aware of these risks, and robust testing can help manage them by assessing the feasibility of key activities, their capacity to deliver desired outcomes, and their ability to work in concert with one another under operational conditions. Going forward, to help ensure a cost-effective enumeration, it will be important for the Bureau to improve its NRFU procedures by addressing the challenges identified during the 2016 Test, updating related training materials as needed, and completing these efforts in time to be included in the Bureau’s End-to-End Test scheduled for 2018. The challenges we observed include (1) reducing high non-interview rates, (2) difficulty accessing recently closed, incomplete cases, (3) the need to improve coordination with managers of multi-unit properties, and (4) the need to better leverage operational information collected by enumerators. Resolving these issues should help the Bureau improve its ability to collect quality data and reduce the cost of unnecessary follow-up visits during NRFU. We recommend that the Secretary of Commerce and Under Secretary for Economic Affairs direct the Census Bureau to take the following actions: 1. Determine the cause(s) for non-interviews experienced during the non-response follow-up operation and revise and test what, if any, changes need to be made to operational procedures, training, or both, including making contact with proxy respondents. 2. Revise and test operational procedures for accessing incomplete closed cases and revise and test training material to reflect when this flexibility to access incomplete closed cases should be used by the enumerator. 3. Revise and test operational procedures and relevant training materials for initial property manager visits to ensure procedures and training material are communicated to and understood by enumerators and their supervisors. 4. Revise and test procedures on how to better leverage enumerator- collected information on the best time or day to conduct interviews, and ensure enumerators are properly trained on these procedures. We provided a draft of this report to the Secretary of the Department of Commerce for comment. In its written comments, reproduced in appendix I, the Department of Commerce agreed with our findings and recommendations. The Census Bureau also provided technical comments that we incorporated, as appropriate. We are sending copies of this report to the Secretary of Commerce, the Counselor to the Secretary with Delegated Duties of the Undersecretary of Commerce for Economic Affairs, the Director of the U.S. Census Bureau, and interested congressional committees. The report also will be available at no charge on our website at http://www.gao.gov. If you have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. The GAO staff that made major contributions to this report are listed in appendix II. Robert Goldenkoff, (202) 512-2757 or goldenkoffr@gao.gov. In addition to the contact named above, Lisa Pearson, Assistant Director; Mark Abraham, Shea Bader, Richard Hung, Donna Miller, Ty Mitchell, Cynthia Saunders; A.J. Stephens, and Timothy Wexler made significant contributions to this report.

Summary: With a life-cycle cost of about $12.3 billion, the 2010 Census was the most expensive enumeration in U.S. history. To help control costs and maintain accuracy, the 2020 Census design includes new procedures and technology that have not been used extensively in earlier decennials, if at all. While these innovations show promise for a more cost-effective head count, they also introduce risks. As a result, it will be important to thoroughly test the operations planned for 2020. The objective of this report is to assess key NRFU operations performed during the 2016 Census Test to identify any lessons learned that could have a potential impact on pending design decisions for the 2020 Census. To assess NRFU operations GAO visited both test locations and observed enumerators conducting NRFU interviews, and reviewed relevant documents including the test plan and enumerator training manuals. The Census Bureau (Bureau) recently completed its 2016 Census Test in Los Angeles County, California, and Harris County, Texas. One primary focus of the test was to assess the methodology for non-response follow-up (NRFU), where enumerators personally visit households that do not self-respond to the census. GAO found that during the 2016 Census Test, NRFU generally proceeded according to the Bureau's operational plan. However, data at both test sites indicate that the Bureau experienced a large number of non-interviews. Non-interviews are cases where either no data or insufficient data are collected. Bureau officials are not certain why there were so many non-interviews for the 2016 Census Test and are researching potential causes. Going forward, it will be important for the Bureau to better understand the factors that contributed to the non-interview rate because of its relationship to the cost and quality of the census. GAO also found that refining certain enumeration procedures and training enumerators better could produce additional efficiencies by enabling the Bureau to be more responsive to situations enumerators encounter on the ground. For example, enumerators, by design, were unable to access on the mobile device recently closed, incomplete cases. Bureau officials acknowledged that closing cases in this fashion represented a missed opportunity and plan to test greater flexibilities as part of the 2018 End-to-End Test. Programming some flexibility into the mobile device—if accompanied with adequate training on how and when to use it—should permit enumerators to complete some interviews and reduce the cost of follow-up attempts. Further, enumerators did not always understand procedures for visiting property managers in multi-unit buildings. Specifically, the 2016 Census Test demonstrated that vacant units could quickly be removed from the NRFU workload where a property manager was readily available to provide that information; however, in other cases the procedures confused enumerators and they did not understand how to proceed. Without the knowledge of which units were vacant, enumerators may have unnecessarily visited some vacant units and thereby increased the cost of NRFU. During GAO's field visits, GAO encountered several instances where enumerators learned that returning at a specific time on a later date would improve their chance of obtaining an interview from either a household respondent or a property manager. However, the Bureau's 2016 Census Test and automated case management system did not have an efficient way to leverage that information. Attempting contact at non-responding households at times respondents are expected to be available increases the completion rate and reduces the need to return. GAO recommends the Secretary of Commerce direct the Bureau to: (1) determine causes for non-interviews, and revise and test what, if any, changes need to be made to operational procedures and training; (2) revise and test procedures and training on accessing closed cases, (3) revise and test procedures and training for initial property manager visits; and (4) revise and test procedures and training for how to use enumerator-collected data on the best time or day to conduct an interview. The Department of Commerce agreed with GAO's recommendations, and the Bureau provided technical comments that were incorporated, as appropriate.