NRFU is a field-based operation that the Bureau administers following the self-response period so that it can (1) determine the occupancy status of individual non-responsive housing units and (2) enumerate them. In most instances, the Bureau typically allows up to six enumeration attempts for each nonresponsive housing unit, or case. If the Bureau is unable to enumerate the housing unit in the field, it may have to impute attributes of the household based on the demographic characteristics of surrounding housing units as well as administrative records. Within the test site in Providence County, Rhode Island, the Bureau set up an area census office to administer field operations. Figure 1 provides an overview of the managerial hierarchy of the area census office. The area census office manager oversees day-to-day operations within the office and acts as a liaison with the Bureau’s New York Regional Census Center, which is a Bureau regional office with jurisdiction over the Providence area census office. Census field managers are to monitor operational progress and performance indicators to understand any areas of concern and shift resources as needed within the test site. Census field supervisors are to act as front-line supervisors for individual performance and payroll processes and receive procedural questions from enumerators, who conduct the count. The Bureau has another operation—Group Quarters—to enumerate those living or staying in a group facility that provides housing or services. Such facilities can include skilled nursing facilities, college and university student housing, and correctional facilities. Within the Group Quarters enumeration, the Bureau also enumerates places such as soup kitchens, homeless shelters, and other service-based enumeration facilities. Prior to Group Quarters enumeration in the field, the Bureau attempts to establish the facilities’ approximate population count and preferred enumeration method through the Advance Contact operation. These facilities can choose among methods including paper listing, where the facility provides a roster of residents as of census day to the Bureau, and in-person enumeration, where a team of enumerators count residents. For the 2020 cycle, the Bureau is also adding an “eResponse” option, tested on a small scale in 2016, whereby facility administrators can electronically submit enumeration data at a date of their choosing within operational time frames. The Bureau’s testing of the peak operations that we observed during the 2018 Census Test was intended to test collection of census data from those either not responding themselves via paper, telephone, or over the Internet or those living in group quarters. Prior to the start of NRFU during the 2018 test, roughly 45 percent of anticipated housing units in the test area of Providence County, Rhode Island, self-responded, leaving more than 140,000 remaining housing units to be attempted by NRFU itself, which took place between May 9 and July 31, 2018. The Bureau conducted a test of its Group Quarters enumeration from July 25 through August 24, 2018, including service-based enumeration. Both portions of Group Quarters fieldwork were preceded by the Advance Contact activity. Dates for the peak operations we observed during the 2018 test are listed in table 1. The Bureau’s operational plans for this phase of the fieldwork for the 2018 test incorporated two innovation areas that the Bureau hopes will produce savings for 2020. Reengineered field operations. For most of NRFU during the 2018 test, the Bureau relied on automated data collection methods, including a system-based, automated process for assigning work to enumerators, a smartphone-based application for collecting enumeration data in the field, and system-generated supervisory alerts. Use of administrative records. To help reduce costly NRFU visits during the 2018 test, the Bureau reviewed and, where appropriate, applied administrative records—information already provided to the government as it administers other programs, such as Social Security, the Selective Service, or the Special Supplemental Nutrition Program for Women, Infants, and Children—to determine the occupancy status of housing units and thus remove vacant housing units from the NRFU workload, as well as to provide population counts of households not responding. The Bureau also tested multiple operational features for the first time this decennial cycle under full census-like production conditions in 2018: NRFU Closeout. During the 2018 test, the Bureau tested how best to relax certain business rules and enumeration procedures late in the NRFU operation so that it can enumerate persistently non-responsive housing units. Examples of procedural modifications include increasing the maximum allowable number of enumeration attempts for each housing unit and manually assigning cases to the highest- performing enumerators. Office-Based Group Quarters Advance Contact. In the 2010 Census, the Bureau sent enumerator crews in person to each facility in advance of enumeration to establish the facility’s preferred method of enumeration and to obtain an approximate population count. In the 2018 Census Test, the Bureau implemented a new method for 2020 that instead involved clerical staff contacting facilities by telephone and updating the group quarters address list and enumeration information remotely to reduce expenses associated with field visits for its enumerator crews. The Bureau began the last phase of NRFU data collection in the 2018 test without having yet determined the procedures it would use for that critical phase. Bureau planning documentation from February 2018 described a late-operation “closeout” phase of NRFU that would attempt to resolve cases that had not yet responded. However, we found that the Bureau had not determined the procedural modifications this phase would involve, either in terms of rules enumerators followed or business rules for how cases were to be assigned. By late May, nearly 3 weeks into the operation, the Bureau issued a set of closeout procedures to census areas where most cases had either been completed or where at least four of the six allowable enumeration attempt day assignments had been made. The Bureau also placed a priority on having high-performing enumerators—in terms of their ability to complete cases—available to work these cases during this phase of the NRFU testing. Table 2 summarizes the chronology for when the Bureau implemented and documented procedural changes governing the transition from early to late-NRFU data collection, as well as the nature of those changes. In late June 2018, the Bureau began testing the third phase of NRFU data collection, what it referred to as the “final attempt” phase, with officials citing a high incidence of non-interviews during prior phases as the reason. However, the Bureau had also begun this phase’s data collection before it had established the procedural modifications it would be using. The modifications were intended to further increase the chances of enumerators completing cases in the field, such as by removing the limit on the number of attempts enumerators could make at each remaining case before NRFU ended. Standards for Internal Control in the Federal Government states that agencies should implement control activities by, for example, documenting policies. However, the Bureau did not determine procedures for the final attempt phase until after testing for this phase of NRFU had begun. Enumerators and census field supervisors thus began working closeout and final attempt cases without a standardized set of test procedures. Without determining the procedural changes the Bureau would be testing—or the business rules guiding when to make those changes—the Bureau was not well positioned to collect data to assess the alternatives it used during the test to inform planning for 2020. Bureau officials shared with us that they believed their automated case assignment approach is most effective during initial data collection but that it is less effective at targeting the toughest cases to resolve late in NRFU data collection. Yet, in part because the Bureau had not established when the transition from automated to manual case management would occur—or the business rules for determining when—some of the highest-performing enumerators were unavailable to receive assignments when the Bureau needed to begin the final attempt phase, according to the area census office manager. By not establishing the scope and timing of procedural changes for late-NRFU data collection in 2020, the Bureau may not be in a position to efficiently shift from its automated assignment approach to a manual one at the right time and position its most effective enumerators to receive assignments when needed. In November 2018, the Bureau provided a draft contact strategy for NRFU in 2020 that included an outline of a multi-phase strategy for late- NRFU data collection. By including multiple phases of (1) shifting away from a fully-automated case assignment process and (2) relaxing management controls to complete as much casework as possible in areas with continued high non-response rates, this strategy appears to follow what the Bureau ultimately implemented during the 2018 test. It will be important, however, for the Bureau to determine the business rules for procedural changes and their timing in advance so that it can maximize the value of NRFU in reducing the number of housing units that have to be imputed for the 2020 Census. Census field supervisors were not integrated into casework management. As described in the Bureau’s training and operational planning documents, census field supervisors were to be the primary points of contact in fielding and addressing enumerator questions. Census field managers—the next step above census field supervisors— were to focus their efforts on monitoring progress in completing the caseload, reviewing cases flagged by enumerators as problematic in one of a small number of pre-defined ways (e.g., dangerous addresses), and resolving significant performance issues. Among field supervisors’ key responsibilities, according to the Bureau’s plan for NRFU, were providing guidance to help enumerators understand procedural matters and to offer coaching and problem-solving support to enumerators who may need it. They also led enumerator training prior to the beginning of NRFU and generally were to train their specific team of enumerators. However, census field managers and enumerators indicated that census field supervisors were often not the primary actors involved in fielding and addressing enumerator questions. Instead, enumerators and census field managers reported having direct contact with each other over procedural questions. Moreover, seven of the nine enumerators participating in the Bureau’s operational debrief focus group who responded said they thought finding someone who could answer their questions was either difficult or very difficult. We found that census field supervisors went underutilized in part because the Bureau did not recruit and position them to assume front-line supervising and coaching responsibilities. As outlined in training documentation, the Bureau vested supervisory review authority (for special cases, such as resident refusals and language barrier issues) within census field managers, the area census office manager, clerks, and office operations supervisors instead of census field supervisors. Additionally, as part of the Bureau’s reengineered field operations for 2020, census field supervisors are given automated tools to monitor enumerators, and enumerators we observed told us that they generally did not interact in person with their supervisors apart from training. We believe that the combination of these factors resulted in census field supervisors having limited exposure to NRFU casework and any problematic situations enumerators might encounter. Officials also told us that the Bureau did not screen census field supervisors for their supervisory or coaching skillsets, though officials noted that this has been the practice in prior censuses, too. Rather, they hired census field supervisors based on their scores on the online enumerator training and because they reported an interest in supervising. Additionally, census field supervisors lacked access to certain data streams from the test that could have helped them answer or troubleshoot enumerator questions. According to two census field managers, the Bureau did not regularly share consolidated records of procedural changes with census field supervisors. Information technology (IT) and census field managers also noted that the Bureau did not share or compare observations between the census field supervisor hotline and the decennial IT hotline, even though enumerators could potentially call either or both with technical or procedural questions. As a result, without sharing how best to respond to similar questions across support lines, enumerators could receive different answers for related questions depending on which hotline they contacted. Standards for Internal Control in the Federal Government states that agency management should demonstrate a commitment to recruit, develop, and retain competent individuals. Management should establish expectations for competence in key roles and should consider the level of assigned responsibility and delegated authority when establishing expectations. Yet, the role the Bureau envisioned census field supervisors having was not aligned with the authority supervisors were given, the skills for which the Bureau hired them, or the access to information that they had for the 2018 test. When we raised this issue related to using census field supervisors, Bureau officials agreed and cited feedback they had received that census field managers felt inundated with the combination of the volume of supervisory review cases that flowed to them and with troubleshooting day-to-day enumerator questions. In October 2018, the Bureau provided documentation to us proposing a set of questions that they could use in screening applicants for the census field supervisor position to identify supervisory skills. Officials also said they were still evaluating options for granting census field supervisors more supervisory review authority. As the Bureau continues to learn from the 2018 test as part of its planning for 2020, it will be important to align census field supervisor roles with their authorities, skills, and information flows so that the Bureau does not underutilize a key portion of its field management chain. Doing so could also lessen the operational burden on higher-level census field managers. Enumerators did not receive training to address mid-operation issues. Prior to the start of 2018 NRFU testing, the Bureau trained enumerators with a series of online training modules and assessments and one full day of in-person training facilitated by census field supervisors. The training included modules on data stewardship requirements, payroll responsibilities, and procedural directions for conducting respondent interviews. However, officials acknowledged that when the Bureau implemented its closeout and final attempt phases of NRFU, it did not provide standardized training to enumerators on the rollout of procedural changes. Five enumerators we observed during these stages said they relied on informal communications from their census field supervisors or census field managers for guidance. The initial practice had been for enumerators to receive daily assignments and follow pre-specified case sequencing and routing based on the Bureau’s automated system. During the final attempt phase, enumerators were given discretion over the sequencing, routing, and number of attempts to make for cases that could be manually assigned, yet they were not given standardized training on how to handle this shift. During our field observations, some enumerators we spoke with said they were uncertain about core procedures. For example, enumerators were not consistently aware that they had some discretion in large multi-unit settings to deviate from the assigned sequence of their cases provided by the automated system. Enumerators we observed and spoke to were also not always clear on how to flag within their field enumeration application the commonly occurring cases with confusing address markings and numberings. For example, enumerators had the option of selecting a case outcome of “missing unit designation,” but they were not always sure whether this selection would capture the nuances of what they were seeing on the ground or how it differed from other selection options. Standards for Control in the Federal Government states that agencies should demonstrate a commitment to competence by, for example, tailoring training based on employee needs and helping personnel adapt to an evolving environment. Targeted informational training would help the Bureau ensure that staff understand mid-operation procedural changes, and the training could be an opportunity for the Bureau to address commonly-observed and persistent implementation issues that may be arising. By developing brief, targeted mid-operation training, either as formal modules, guidance, or other standardized job aids, such as “frequently asked questions” worksheets, the Bureau could better position itself to react nimbly to enumerator feedback. We have previously reported challenges the Bureau faces with its field work in other locations, such as connecting to the Internet during testing of address canvassing in rural West Virginia in 2017 and dealing with language barriers and other circumstances in unincorporated communities in southern Texas or with migrant and seasonal farmworkers in southern California during the 2000 Census. All challenges are not universal to all locations. Given that some of the enumeration challenges enumerators encountered in 2018 NRFU testing might not occur everywhere, and that some other areas of the country will have their own types of challenges, locally- or regionally-specific training or guidance may better address some needs. By relying solely on pre-NRFU training, the Bureau risks having little opportunity to course-correct with enumerators who may not have absorbed all of the training and are experiencing difficulty completing interviews or not collecting quality data. We observed and discussed with Bureau officials in real time several other implementation issues that occurred during the 2018 test. Bureau officials acknowledged these issues and, as of September 2018, were assessing them and developing mitigation strategies as part of their test evaluation process. These issues include: Training certification. Census field managers estimated that roughly 100 enumerators were unable to transmit their final test scores because the Bureau’s online learning management system had an erroneous setting. According to Bureau officials, this problem delayed the start of unsupervised work for these otherwise-qualified enumerators by an average of 2 days per enumerator and resulted in the attrition of some who were able to quickly find other work. Bureau officials told us they have fixed the system setting and are considering an alternative means to certify training, such as by having the option of trainees taking and verifying their final assessment as part of their final capstone day of classroom training. According to Bureau officials, development of this backup strategy will begin in December 2018. Assigning cases manually in batches. During the 2018 test, the Bureau’s automated case management system was not configured for non-Headquarters staff to manually assign multiple cases to an enumerator at once. Rather, according to officials, census field managers were faced with having to manually assign thousands of cases individually during latter stages of NRFU. According to field management, this problem presented an unexpected burden on them, delayed assignments of the hardest-to-count cases, and contributed to high- performing enumerators not receiving work timely and in some cases for days in a row. Officials told us that, as a work-around, the Bureau shifted responsibility for assigning cases to a headquarters official with access rights in the system to assign large numbers of cases at once. Bureau officials acknowledge the unsustainability of this work-around if needed at a national level and the importance of resolving this before the 2020 Census. As of October 2018, Bureau officials showed us system screenshots of how census field managers would be able to manually assign batches of cases and indicated that this functionality would be ready for the 2020 Census. Monitoring operational progress. The Bureau’s reporting on its progress in completing the NRFU casework for the 2018 test emphasized a process-oriented measure that overstated the extent to which the NRFU efforts were resulting in completed workload. In planning documentation, the Bureau listed the outcomes of interview attempts that it considered complete and thus not in need of further enumeration assignments. These outcomes—such as a full interview of the household or confirmation of a housing unit being vacant or nonexistent—would also result directly in reduction in the number of incomplete cases needing to have some of their missing data imputed by the Bureau later. Yet the daily Bureau progress report and “dashboard” the Bureau provided us for the 2018 test, which decennial leadership also identified as their primary monitoring report, did not reflect these pre-planned definitions of completed workload. Rather, as officials acknowledged, it included cases that the Bureau had unsuccessfully attempted to enumerate the maximum number of allowable times for the initial phase of NRFU being tested, even though those cases could still—and did— receive additional attempts during later phases of NRFU. Officials noted that the measure reported could be helpful during early stages of the operation in determining whether enough employees had been hired, or whether case assignments were being worked quickly enough. Figure 2 demonstrates the gap that arose during 2018 NRFU test implementation between the reported progress measure and the number of cases actually being completed. The totals reflected in the Bureau’s reported measure include those that either have to be re-worked in the field during the final attempt phase as discussed or have their data imputed after fieldwork had ended. By contrast, an outcome-based measure of operational progress, like the one the Bureau designed, would capture only those cases where the Bureau had completed enumeration of the nonresponding housing units and thus be a more accurate representation of the operation’s status. Bureau officials acknowledged the need to maintain measures that focus on process as well as outcomes—such as avoiding having to impute data for cases after field work—when measuring progress completing NRFU. They said that managers in the field and in Bureau headquarters had access to alternative measures and reports that more closely identified outcomes. The officials noted that the reporting mechanism expected to be used in 2020 was not fully available in time for the beginning of NRFU testing in 2018, so the reporting format and measures will likely differ. In addition, in October 2018, the Bureau provided a draft dashboard for 2020 that included greater detail on the number of cases that could still require work to enumerate. Such detail could help assist with determining when to transition to the final-attempt phase of the operation to address cases without sufficient information yet collected. Integrating key systems settings. At the beginning of NRFU test implementation, the Bureau’s case assignment and case sequencing systems were operating as if they were on time zones 4 hours apart. Bureau officials said that this resulted in enumerators receiving mismatched case assignment times, which hampered early NRFU production, and census field supervisors having to process erroneous “work not started” supervisory alerts. Bureau officials said they addressed the problem within the first week of the operation and that they would ensure that future updates and key settings would be coordinated across systems. Tracking employees and equipment. The Bureau used two different sets of employee identification numbers to track their payroll status and use of Bureau-issued equipment (e.g., smart phones), respectively, without cross-walking them. According to census field managers, this resulted in extra work when trying to monitor changes in enumerators’ employment statuses and whether enumerators had returned their equipment to the Bureau. The managers noted that office staff had to spend extra time comparing different lists of staff, while one manager developed a spreadsheet listing all staff by their two different identification numbers. Bureau officials said they considered this a priority issue to be resolved during final systems development for 2020 and had already developed a fix within their case management system so that the cross- walk between the two systems would be integrated within their management system. This would eliminate the need for manually reconciling the differing identification numbers. Having more enumerators work weekends. Until the latter stages of 2018 NRFU testing, the Bureau assigned cases to enumerators based on the alignment of the Bureau’s estimated probability of finding respondents at home at certain times and enumerator-reported work availability. Bureau officials told us that Saturdays are generally one of the best days to find a household member home to respond to the census. However, during the test, our analysis showed that Saturdays had the second- fewest number of enumerators assigned to cases of any day of the week. Bureau officials said that they would review whether the incentive structure for working on Saturdays should be altered and that they would examine ways to ensure that more enumerators are working on those days. This includes exploring the feasibility of hiring and assigning work to applicants who may only want to work weekends and being clearer with enumerators about what the expected peak enumeration hours are. Increasing electronic completion rates for Group Quarters. The Bureau hopes to reduce field costs for Group Quarters (such as skilled nursing homes, college and university student housing, and correctional facilities) by, for the first time, encouraging facilities to self-enumerate electronically, when possible. As previously discussed, clerical staff first establish facility enumeration preferences during Group Quarters Advance Contact, and enumeration (either in person or otherwise) takes place afterward. During the 2018 test, the Bureau reported that only 25 of the 75 facilities that selected the “eResponse” enumeration option during Advance Contact submitted responses by the enumeration deadline. As of September 2018, Bureau officials said they were still evaluating potential causes of the low response rate by Group Quarters facilities but noted that issues with the required format for the submission of response files may have prevented some submissions. Bureau officials also acknowledged the need to conduct more active follow-up with these facilities during the eResponse period to ensure a full and accurate count of Group Quarters facilities. We reported to the Bureau during the 2015 Census Test that information enumerators were typing into their case notes did not appear to be systematically used by their managers. We also reported that, during the 2016 Census Test, the Bureau was not reviewing case notes written by enumerators providing respondent information on better times-of-day for future NRFU visits to their housing unit, and enumerators did not always have this note-taking feature available. During the 2018 test, enumerators were trained to take notes and, when appropriate, identify special cases that would later require supervisory review. We also observed that enumerators appeared to be consulting case notes from prior enumerator visits when planning NRFU visits to the same housing unit. Enumerators can use markings within their automated interview instrument to describe certain types of cases (e.g., hearing barriers and dangerous situations), which the automated system would then route to receive supervisory review. Enumerators could select other case outcomes that the automated system would apply predetermined business rules to either reassign the cases (e.g., refusal, no one home) or treat them as completed (verified vacant or not a housing unit). However, we identified multiple scenarios in which enumerators had described cases in their case notes but for which the enumerators had not selected the corresponding case flag for the situation that would have resulted automatically in a supervisory review. One census field manager described discovering several dozen cases that had been inactive where enumerators had written case notes describing language barriers encountered but had not specifically marked the flag within the device for “language barrier.” Because these cases thus were not triggered for supervisory review, they were eligible to be reassigned by the Bureau’s standard automated system. As a result, the Bureau was not controlling for the requisite language skills in assigning the cases for subsequent enumeration attempts. The Bureau’s use of automated systems to apply business rules to efficiently manage field casework for 2020—including identifying which cases receive supervisory review—relies critically on field staff understanding how to reflect what they are seeing on the ground within the choices provided to them with which to flag cases in their interview device. The Bureau’s use of remote management as part of reengineered field operations also relies on enumerators knowing when and how to report issues to their supervisors. We observed multiple field scenarios that called these conditions into question, however. For example, enumerators we observed during NRFU told us that they indicated address listing issues in their case notes, such as if the unit designation was missing or incorrectly marked. Yet, these enumerators did not know how to flag such cases in their interview instrument to trigger supervisory review. According to Bureau officials, this type of address listing issue turned out to be a broadly experienced challenge within the test area. Additionally, during Group Quarters, an enumerator we observed received supplemental information about the number of residents at a neighboring facility after that facility had been enumerated. The enumerator made note of this discrepancy and included the original facility’s identifying information but was uncertain about how, if at all, to alert the supervisor about the discrepancy. Moreover, we saw little evidence that census field supervisors or managers were systematically reviewing case notes for the purpose of identifying either cases not being marked properly or for which the selected flags may not have been fully describing the case characteristics. For example, a census field manager confirmed that case notes recorded at Group Quarters facilities that were enumerated would not be reviewed during clerical processing, leaving the possibility—such as we observed—that if enumerators relied on case notes to communicate information about the accuracy of data collected, it would not be acted on. Bureau officials told us that reviewing all case notes could require more staff time than budgeted for, and changing the automation process to selectively present unflagged cases for supervisory review could necessitate requirements changes to systems whose development is already pressed for time. Standards for Internal Control in the Federal Government states that agencies should use quality information by, for example, processing reliable information to help it make informed decisions. Bureau enumerators can record useful local knowledge about their cases with their choice of case type flags and within their descriptive case notes. While the Bureau has anticipated a broad range of types of cases for enumerators to select from when documenting their casework, enumerators we observed did not uniformly understand those options, and the descriptors did not fully anticipate what enumerators were encountering. Improving training and guidance to field staff on the intended use of case notes and on alternative ways to communicate their concerns about cases, such as flags for different types of cases, can help ensure the Bureau has reliable data on the cases during its field operations relying on automated interviewing instruments. During the 2015 Census Test, we reported that certain technical and procedural problems that enumerators were encountering in the field were going unreported and that enumerators did not always know who to contact for assistance. We further noted that the Bureau was not systematically assessing or tracking the extent of these issues during testing, and we recommended that it enable such capture of information by training enumerators on where to record issues and whom to contact. The Bureau agreed with our recommendation. During the 2018 test, the Bureau had both an information technology (IT) hotline and a census-field-supervisor hotline established for technical and procedural questions, respectively. Yet, enumerators we observed told us they did not always report to their support lines the technical issues that they were easily able to resolve by, for example, turning their devices on and off to reset. This lack of reporting kept the Bureau from getting information on commonly-occurring challenges that might be useful real- time feedback in the testing environment. Moreover, a Bureau IT manager noted that the Bureau does not formally review and share observations and troubleshooting notes from IT hotline and census field supervisor hotline calls. Because enumerators may call either or both hotlines when having difficulty operating their Bureau-issued smart phone, operators of these hotlines could be unaware of the prevalence of or solutions to a given problem if the Bureau does not monitor troubleshooting information across the two operational silos. For the Bureau to be informed on any additional training needs or other operational decisions for 2020, it will need to continue to expand its efforts in collecting information on enumerator-reported problems per our 2015 recommendation. Depending on the size of a Group Quarters facility (e.g., skilled nursing facility, college and university student housing), the Bureau can use varying sizes of enumerator crews to conduct an onsite count. During the 2010 Census, we observed overstaffing during the service-based enumeration (e.g., homeless shelters and soup kitchens) portion of Group Quarters. While determining staffing levels at these facilities can be challenging, such overstaffing can lead to poor productivity and unnecessarily high labor costs. We recommended that the Bureau determine and address the factors that led to this overstaffing prior to 2020. The Bureau agreed with our recommendation. However, the Bureau has faced challenges determining the right staffing ratios in light of complications with the Advance Contact phase of Group Quarters. As previously noted, the Bureau used this phase to establish facilities’ enumeration method preferences. For the 2018 test, most Group Quarters facilities selected the facility-provided paper listing and the eResponse enumeration options. Therefore, the Bureau allocated a large share of its enumerator and census field supervisor workforce in the test area to the 44 known service-based enumeration facilities, which were restricted in terms of the enumeration options they could select and tended to select in-person enumeration. However, only 11 of these facilities responded to initial inquiries, so the Bureau had less work than anticipated for its enumerator crews. At multiple sites we observed in the test area, enumerators appeared either idle or underutilized. Moreover, several of the Group Quarters facilities we observed had changed their initial choice of enumeration method on the day of enumeration. Enumerator crews thus ran the risk of either being overstaffed (in the case of switching to a facility-provided paper listing) or understaffed (in the case of switching to an in-person enumeration). The Bureau’s Advance Contact activities have a potential benefit—if the Bureau can get accurate information on the method of enumeration and approximate population within a facility ahead of time, Bureau managers and enumerator crews can more proactively allocate resources and prepare for the count. Bureau officials said they are still assessing outcomes of Advance Contact to see if these gains were realized and may have completed the assessment by as early as January 2019. Doing so will help the Bureau determine appropriate staffing sizes and thus address our prior recommendation. According to preliminary data from the 2018 Census Test, the Bureau experienced similarly high rates of cases coded as non-interviews as it did during its last major field test of NRFU in 2016. Non-interviews are cases where enumerators collect no data or insufficient data from households either because enumerators made the maximum number of visits without a successful interview, or because of special circumstances like language barriers or dangerous situations. When this happens, the Bureau may have to impute the census data for the case, such as whether the housing unit is vacant or not, the population counts of the households, or demographic characteristics of their residents. In January 2017, we reported that, during the 2016 Census Test, the Bureau incurred what it considered high non-interview rates (31 and 22 percent across the two test sites, respectively, as the Bureau preliminarily reported at the time), and we recommended that the Bureau determine the causes of these rates. Using the same method to calculate the rate of non-interviews for the test as in 2016, the 2018 Census Test had similarly high non-interview rates— 33 percent of all NRFU cases. Bureau officials said they are still examining causes of these elevated non-interview rates and whether final attempts helped to mitigate the non-interview rate and will report out on what they learn as part of their comprehensive assessment of the test, planned for December 2018. A draft of the Bureau’s revised contact strategy for NRFU, provided in November 2018, indicates that as part of enumerator training in 2020 the Bureau will need to incorporate messaging that emphasizes the importance of obtaining sufficient data from interview attempts. Officials noted that any interim lessons learned from this assessment process would inform updates to the field enumeration contact strategies for 2020. Enumerators are directed to try and complete a NRFU case by interviewing a proxy for a household respondent, like a neighbor, after multiple failed attempts have been made to contact someone in the household for that case. We previously observed in the 2016 Census Test that enumerators did not seem to understand the procedures for conducting these interviews and, as a result, underutilized the interviewing method. In our January 2017 report, we therefore recommended that, as part of determining the causes of its non-interview rate, the Bureau revise and test any needed changes to proxy procedures and associated training. The Bureau agreed with our recommendation and subsequently developed automated supervisory performance alerts for census field supervisors and census field managers that would inform them when an enumerator was not following prompts to conduct proxy interviews for eligible cases. However, in implementing proxy interview procedures for the 2018 Census Test, the Bureau experienced a technical glitch resulting in some confusion among some enumerators and their supervisors about related procedures. Early in NRFU data collection for the test, a programming error within the field enumeration application was prompting enumerators to make more than the allowable three attempts to interview a proxy respondent. The Bureau reported promptly implementing a technical fix to this issue; yet, enumerators we observed reported receiving varying guidance from their supervisors on whether to abide by the erroneous prompts. While some of these enumerators appeared to understand the importance of attempting proxy interviews, some did not appear to understand Bureau guidance that enumerators should make no more than three attempts to interview a proxy respondent, and some appeared conditioned to follow the erroneous prompts. Proxy interviews can be a substantial portion of completed interviews during the census. In 2018 NRFU testing, interviews of proxy respondents accounted for 27 percent of all successful interview-based enumerations of occupied housing units—compared to 24 percent during the 2010 Census and 9 percent during the 2016 Census Test. Given the role that proxy interviews play in completing census data collection, it will be important for the Bureau to fully implement our recommendation so that enumerators are properly pursuing and conducting these interviews. Initial visits to property managers of multi-unit residences can help the Bureau identify vacant and occupied housing units before sending enumerators to individual units within the facilities. We have previously reported that property managers can also be a helpful source of information on respondents who are not at home, thereby making subsequent follow-up visits to individual units more productive. During the 2016 Census Test, we observed that enumerators were uncertain of how to handle individual cases within a multi-unit once they were unsuccessful in contacting a property manager initially. As a result, we recommended in January 2017 that the Bureau revise and test procedural and training modifications as needed to aid enumerators and their supervisors in these cases. The Bureau agreed with this recommendation and indicated that the evaluations of the 2018 test would inform its strategies for 2020. However, we observed a similar issue during the 2018 Census Test in that enumerators were unclear on what, if any, proxies to attempt if they were unsuccessful in finding the listed property manager. Additionally, we observed multiple enumerators leave voicemails with their contact information—not a central number—for the listed property manager, but it was unclear how these voicemails would produce a successful interview because, later, the automated system could reassign other enumerators to visit the manager. When we raised this concern with Bureau officials, they acknowledged that they need to continue to refine procedures for handling initial property manager visits for 2020. Previously, during the 2016 Census Test, we observed that enumerators were unable to re-open closed non-interview cases even if they happened upon the respondent in question soon after and nearby. We noted this inefficiency, since these cases would get re-assigned later, and in January 2017, we recommended that the Bureau revise and test procedures that would grant flexibility to enumerators to access cases in these circumstances. The Bureau agreed with our recommendation. For the 2018 Census Test, the Bureau provided a list in the field enumeration application of the cases that had been worked by the enumerator that day but that had not been submitted for processing or reassignment. Training for enumerators described this enumeration option, and enumerators were authorized to access these cases when needed, but not all enumerators we observed were consistently aware of how to do so. Enumerators we spoke with cited uncertainty over how to access these cases and whether enumerators were allowed to do so as considerations. Continuing to review the procedures and guidance to enumerators on this flexibility for completing interviews, consistent with our prior recommendation, will help the Bureau make better use of it in 2020. As we reported in 2017, the Bureau previously modified how it would treat some of the households that did not respond to the 2020 Census and that the Bureau’s use of administrative records had determined to be not occupied. The Bureau’s earlier testing had determined that the Bureau should require two—instead of just one—notices from the United States Postal Service that mail could not be delivered to these households before removing their addresses from the NRFU workload. After we provided a draft of this report to the Department of Commerce to obtain agency comments, Bureau officials provided us with findings from an evaluation of the 2018 Census Test. In the evaluation, Bureau officials observed that there were households for which they had received multiple notices from the United States Postal Service that mail was undeliverable but that Bureau enumerators recorded as occupied. While Bureau officials believe, based on their follow-up research, that these addresses may likely be vacant or not housing units, they are concerned about possible undercounting from not enumerating people who may be at these addresses. As of November 2018, the Bureau was considering adding one physical visit for each of these cases. Bureau officials said they are continuing to analyze these evaluation results and expect to document and include changes within its final operational plan for the 2020 Census due in January 2019. The 2018 Census Test offered the Bureau its last opportunity to test key procedures, management approaches, and systems under decennial-like conditions prior to the 2020 Census. As the Bureau studies the results of its NRFU and Group Quarters testing to inform 2020, it will be important that it address key program management issues that arose during implementation of the test. Namely, by not establishing the intended procedural changes for late-NRFU data collection ahead of time, the Bureau risked not getting the most out of NRFU to minimize the number of housing units having to have their information imputed by the Bureau later. Additionally, by not aligning the skills, responsibilities, and information flows for census field supervisors, the Bureau limited their role in support of enumerators within the reengineered field operation. The Bureau also lacks mid-operation training or guidance, which, if implemented in a targeted, localized manner, could further help enumerators navigate procedural modifications and any commonly- encountered problems when enumerating. Finally, without enumerators understanding how to use case notes and flags for various types of cases in their enumeration device and to report enumeration challenges to supervisors and managers, the Bureau may be unaware of field work issues that could affect the efficiency of its operations and the quality of its data. We provided near real-time feedback to the Bureau across a range of test implementation issues. Some, such as those related to staffing ratios for the Group Quarters operation, build on long-standing implementation issues that, if addressed, can contribute to the efficiency and effectiveness of 2020 field operations. Others, like not having NRFU progress measures that provide true indications of completed workload, are issues specific to this test that the Bureau is assessing as part of its 2018 Census Test evaluations. It will be important for the Bureau to prioritize its mitigation strategies for these implementation issues so that it can maximize readiness for the 2020 Census. We are making four recommendations to the Department of Commerce and the Census Bureau: The Secretary of Commerce should ensure that the Director of the Census Bureau determines in advance of Non-Response Follow-Up what the procedural changes will be for the last phases of its data collection and what the business rules will be for determining when to begin those phases, which cases to assign, and how to assign them. (Recommendation 1) The Secretary of Commerce should ensure that the Director of the Census Bureau identifies and implements changes to align census field supervisor screening, authorities, and information flows to allow greater use of the census field supervisor position to provide supervisory support to enumerators. (Recommendation 2) The Secretary of Commerce should ensure that the Director of the Census Bureau enables area census offices to prepare targeted, mid- operation training or guidance as needed to address procedural changes or implementation issues encountered locally during Non-Response Follow-Up. (Recommendation 3) The Secretary of Commerce should ensure that the Director of the Census Bureau improves training and guidance to field staff on the intended use of case notes and flags, as well as on alternative ways to alert supervisors and managers when case characteristics are not readily captured by those flags. (Recommendation 4) We provided a draft of this report to the Secretary of Commerce. In its written comments, reproduced in appendix II, the Department of Commerce agreed with our findings and recommendations and said it would develop an action plan to address them. The Census Bureau also provided technical comments and an update on their evaluation of the test, which we incorporated as appropriate. We are sending copies of this report to the Secretary of Commerce, the Undersecretary of Economic Affairs, the Acting Director of the U.S. Census Bureau, and the appropriate congressional committees. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report please contact me at (202) 512-2757 or goldenkoffr@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. This report examines (1) how peak field operations were implemented during the 2018 Census Test; and (2) the extent to which implementation issues raised in prior 2020 Census tests have been addressed, and what actions the Census Bureau (Bureau) is taking to address them. To address these objectives, we reviewed 2018 Census Test and 2020 Census operational planning and training documentation. We also reviewed our prior reports and documentation on prior census testing operations. Non-Response Follow-Up (NRFU) operations took place from May 8 through July 31, 2018, while Group Quarters took place from July 25, 2018 through August 24, 2018, with the service-based enumeration portion taking place July 25, 2018 through July 27, 2018. To review the Bureau’s test implementation and mitigation strategies for previously-identified implementation issues for peak operations, we visited Providence, Rhode Island, multiple times between May and August 2018 to observe enumerators, census field supervisors, and management operations. NRFU visits took place between mid-May and late-July 2018, while we also conducted two iterations of visits of Group Quarters in late July and early-August 2018. These multiple iterations both across and within operations enabled us to see how, if at all, implementation of procedures varied over time. It also enabled us to get direct feedback from Bureau field managers on how various phases of test operations were proceeding. These visits consisted of non- generalizable observations of field enumeration and office clerical work, as well as interviews with local managers. For each of these visits, we developed data collection instruments to structure our interviews and to cover topics that were pertinent to the given phase of the operation we were observing. We also observed debrief sessions with multiple levels of the Bureau’s field workforce following the field work. To translate our observations into actionable feedback for the Bureau, we shared high-level observations in near real-time to Bureau headquarters management overseeing the operations so that the Bureau could mitigate and adapt to known issues in a timely manner. We also discussed any mitigation or evaluation strategies developed in response to our observations with the cognizant Bureau headquarters officials. For objective two specifically, we reviewed Bureau test planning documentation and our work from prior tests to examine how, if at all, the Bureau planned to address known implementation issues. To gain insight into how implementation was proceeding when we were not directly observing test implementation, we received daily management progress reports from the Bureau throughout the NRFU operation testing that included information on the total number of NRFU cases, the final outcomes of each case, and the number of cases that the Bureau reported as completed for each day of the NRFU operation. We also received Periodic Management Reports that summarized high level outcomes of both the NRFU and Group Quarters workload. To fully understand the source of the Bureau’s daily progress reports, we requested and received all transactional data collected during NRFU production. We reconciled case totals and outcomes with the final numbers in the NRFU progress reports and then used these data to analyze the Bureau’s progress during NRFU production. We also received and analyzed Bureau payroll data on enumerator hours worked during NRFU operations. Specifically, we assessed the number of enumerators working each day, the number of enumerator’ hours paid each day, and the days of the week that were worked the most by enumerators. We found the Bureau’s transactional and payroll data sources to be sufficiently reliable for our reporting purposes. We conducted this performance audit from April to December of 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Ty Mitchell (Assistant Director), Devin Braun, Karen Cassidy, Joseph Fread, Robert Gebhart, Krista Loose, Kathleen Padulchick, Lisa Pearson, Kayla Robinson, Robert Robinson, and Cynthia Saunders made significant contributions to this report.

Summary: The cost of the decennial census has steadily increased over the past several decades, with self-response rates declining over the same period. The largest and costliest operation that the Bureau undertakes, NRFU is the Bureau's attempt to enumerate households not initially self-responding to the census. GAO was asked to review NRFU implementation during the 2018 Census Test as well as the Bureau's overall readiness for peak field operations, which cover the actual enumeration of residents. This report examines (1) how peak field operations, including NRFU, were implemented during the test; and (2) the extent to which prior test implementation issues have been addressed. GAO reviewed test planning and training documentation, as well as production and payroll data. At the test site, GAO observed and interviewed enumerators, field supervisors, and managers conducting peak operations. In preparation for the 2020 census, the Census Bureau (Bureau) set out to enumerate over 140,000 housing units during the 2018 Census Test at a site in Providence County, Rhode Island. The 2018 Census Test marked the Bureau's last chance to test enumeration procedures for peak field operations under census-like conditions before 2020. Implementation of this test identified the following concerns: The Bureau experienced operational issues during implementation of the Non-Response Follow-Up (NRFU) as part of the 2018 Census Test. For example, the Bureau had not finalized procedures for data collection during late phases of NRFU (e.g., after multiple attempts to interview had been made) until after the work had already started. As a result, enumerators and their supervisors did not have standardized procedures during the test, which made it difficult to evaluate the effectiveness of the test procedures. GAO also observed a range of other NRFU implementation issues during the test, such as the Bureau's use of progress reporting that overstates the number of NRFU cases not needing any additional fieldwork and the Bureau having fewer of its enumerators work Saturdays, which can be among the most productive interview days. The Bureau is taking steps to assess and mitigate these and other issues that GAO identified. The Bureau's field workforce was not fully prepared to face all of the enumeration challenges that arose during the test. For instance, the Bureau expects census field supervisors to provide front-line coaching to enumerators but did not screen these employees to ensure they had the needed skills. Moreover, it did not provide them with the authorities and information that would have helped them serve that role. As a result, we believe that supervisors did not have the casework expertise, information, or authority to help enumerators with procedural questions, and higher-level census field managers ended up providing direct support to enumerators. While the Bureau provided extensive online and in-person training to enumerators prior to NRFU fieldwork for the 2018 Census Test, the Bureau lacked any standardized form of mid-operation training or guidance as new procedures were implemented. GAO observed that during the test some enumerators continued to have questions and were uncertain about procedures. Developing targeted, location-specific training could help ensure that, in 2020, enumerators receive the guidance they need to collect census data consistently and in accordance with NRFU procedures. The Bureau has made progress addressing prior test implementation issues but still faces challenges. For example, the Bureau improved its collection of enumerator case notes, which reflect real-time knowledge gained during enumeration. However, enumerators did not always report cases using flags built in to their interviewing device that would benefit from supervisory review, such as for language barriers. Moreover, supervisors were not systematically analyzing case notes to identify cases not flagged properly. As a result, critical data on fieldwork challenges were not being communicated effectively to those who could analyze and use them. GAO recommends that the Bureau (1) determine procedures for late-NRFU data collection; (2) align census field supervisor screening, authorities, and information flows; (3) prepare for targeted mid-operation training or guidance as needed; and (4) improve training on reporting cases that need supervisory attention and alternative ways to communicate these cases. The Department of Commerce agreed with GAO's findings and recommendations, and the Bureau provided technical comments that were incorporated as appropriate.