Congress, GAO, the Department of Commerce Inspector General, and even the Bureau itself have all noted that the 2000 Census was marked by poor planning, which unnecessarily added to the cost, risk, and controversy of the national head count. In January 2003, we named the 2010 Census a major performance and accountability challenge because of our growing concern over the numerous obstacles to a cost-effective enumeration as well as its escalating price tag. More recently, we reported that while the Bureau’s preparations for the 2010 Census appeared to be further along than at a similar point during the planning cycle for the 2000 Census, considerable risks and uncertainties remained. Thus, it is imperative that the Bureau adequately test the various components of its design for the 2010 Census. A rigorous testing program provides at least four major benefits. First, testing allows the Bureau to refine procedures aimed at addressing problems encountered in past censuses. During the 2000 Census, for example, group quarters were sometimes counted more than once or counted in the wrong location; the wording of the race and ethnicity question confused some respondents, which in some cases resulted in lower quality data; and following up with nonrespondents proved to be costly and labor-intensive. A second benefit is that sound testing can assess the feasibility of new procedures and technologies, such as HHCs (see fig. 1), that have never before been used in a decennial census. Third, a rigorous testing program helps instill a comfort level among members of Congress and other stakeholders that the Bureau (1) has chosen the optimal design given various trade-offs and constraints and (2) has identified and addressed potential risks and will be able to successfully execute its plan. Such confidence building, developed through regular updates and open lines of communication, is essential for continuing congressional support and funding. And finally, proper testing early in the decade will help the Bureau to conduct a dress rehearsal in 2008 that fully assesses all aspects of the census design under realistic conditions. Because of various late requirement changes, certain procedures that were added after the 1998 dress rehearsal for the 2000 Census were not properly tested. As agreed with your offices, our objectives for this report were to (1) assess the soundness of the Bureau’s design for the 2004 census test and whether the Bureau implemented the test consistent with its plans, (2) review the quality of the Bureau’s IT security practices, and (3) identify initial lessons learned from conducting the test and their implications for the 2010 Census. To assess the soundness of the design we reviewed pertinent documents that described the Bureau's test and evaluation plans. We systematically rated the Bureau’s approach using a checklist of design elements that, based on our review of program evaluation literature, are relevant to a sound study plan. For example, we reviewed the Bureau's approach to determine, among other things, (1) how clearly the Bureau presented research objectives, (2) whether research questions matched the research objectives, and (3) the appropriateness of the data collection strategy for reaching the intended sample population. As part of our assessment of the Bureau’s test design, we also reviewed evaluations of the prior decennial census to determine the degree to which the new operations being tested addressed problematic aspects of the 2000 Census. However, we did not assess the Bureau’s criteria in selecting its objectives for the 2004 census test. To determine if the Bureau implemented the test consistent with its plans, we made multiple site visits to local census offices in Thomasville, Georgia; and Queens Borough, New York. During these visits, we interviewed local census office mangers and staff, observed various data collection activities, and attended weeklong enumerator training. We observed a total of 20 enumerators as they completed their daily nonresponse follow-up assignments—half of these were in southern Georgia, in the counties of Thomas, Colquitt, and Tift, and half were in Queens (see fig. 2 for maps of the test site areas). The results of these observations are not necessarily representative of the larger universe of enumerators. To evaluate the quality of the Bureau’s IT security practices, we assessed risk management documentation associated with IT systems and major applications for the 2004 census test. We based our determination on applicable legal requirements, Bureau policy, and leading practices described in our executive guide for information security management. We also interviewed key Bureau officials associated with computer security. To identify lessons learned from the 2004 census test, we met with officials from the Bureau’s Decennial Management Division regarding overall test plans and with officials from its Technologies Management Office about using HHCs. Bureau officials and census workers from both test locations also provided suggestions on improving census operations. We requested comments on a draft of this report from the Secretary of Commerce. On December 20, 2004, the Under Secretary for Economic Affairs, Department of Commerce, forwarded written comments from the Bureau (see app. I). We address these comments in the “Agency Comments and Our Evaluation” section at the end of this report. The Bureau designed a sound census test and generally implemented it as planned. However, in looking ahead, the Bureau’s planning and investment decisions could benefit from analyzing (1) the degree to which HHCs contributed to the Bureau’s cost containment goal and (2) the results of the targeted second mailing, an operation designed to increase participation by sending a follow-up questionnaire to nonresponding households. Future tests could also be more informative if the Bureau developed quantifiable productivity and other performance requirements for the HHCs and then used the 2006 test to determine whether the devices are capable of meeting those requirements. Collectively, these refinements could provide Bureau officials with better information to guide its IT and other design decisions, as well as refine future census tests. The design of the 2004 census test contained many components of a sound study (see table 1). For example, the Bureau identified test objectives, designed related research questions, and described a data collection strategy appropriate for a field test. The Bureau also developed evaluation plans for each of the test’s 11 research questions, and explained how stakeholders were involved with the design, as well as how lessons learned from past studies were incorporated. Although the Bureau plans to evaluate various aspects of the 2004 test, it does not currently plan to assess the impact that the HHCs and targeted second mailing had on cost savings and productivity. According to the Bureau, the census test was focused more on determining the feasibility of using the HHCs and less on the devices’ ability to save money. Likewise, the Bureau said it is not assessing the impact of the targeted second mailing because the operation is not one of its four test objectives for improving (1) field data collection using the HHC, (2) the coverage of undercounted groups, (3) questions about race and ethnicity, and (4) methods for defining special places and group quarters. These decisions might be shortsighted, however, in that the Bureau included the HHCs and targeted second mailing in the 2010 Census design, in part, to reduce staff, improve productivity, and control costs. For example, Bureau studies have shown that sending out replacement questionnaires could yield a gain in overall response of 7 to 10 percent from households that do not respond to the initial census mailing, and thus generate significant cost savings by eliminating the need for census workers to obtain those responses via personal visits. Thus, information on the degree to which the HHCs and second mailing contribute to these key goals could help inform future budget estimates, investment and design decisions, as well as help refine future census tests. Moreover, the feasibility of a targeted second mailing is an open question, as the Bureau has never before included this operation as part of a decennial census. Although a second mailing was part of the original design for the 2000 Census, the Bureau had abandoned it because it was found to be logistically unworkable. A Bureau official said that the second mailing was included in the 2004 test only to facilitate the enumeration process, and it would be better tested in a larger scale operation such as the 2008 dress rehearsal. However, we believe that it would be more prudent to assess the second mailing earlier in the census cycle, such as, during the 2006 test so that its basic feasibility could be assessed, any refinements could be evaluated in subsequent tests, and the impact on savings could be estimated more accurately. While the design of the 2004 test was generally sound, refinements could strengthen the next field test in 2006. Opportunities for improvement exist in at least two areas: ensuring that (1) the HHCs can meet the demanding requirements of field data collection and (2) management of the local census offices mirrors an actual enumeration as much as possible. With respect to the HHCs, because they replace the paper version of the nonresponse follow-up questionnaire the devices must function effectively. Further, this test was the first time the Bureau used the HHCs under census-like conditions so their functionality in an operational environment was unknown. Bureau officials have acknowledged that for the 2004 test they had no predefined indicators of success or failure other than if there was a complete breakdown the test would be halted. This is a very low standard. Now that the Bureau has demonstrated the basic functionality of the computers, it should next focus on determining the specific performance requirements for the HHCs and assess whether the devices are capable of meeting them. For example, the Bureau needs productivity benchmarks for the number of interviews per hour and per day that is expected per census worker. Durability measures, such as how many devices were repaired or replaced, should be considered as well. Assessing whether the HHCs can meet the requirements of nonresponse follow-up will help inform future design and investment decisions for whether or not to include the devices in the 2010 design. Ensuring that key positions in the local census offices are filled from the same labor pool as they would be in an actual decennial census could also enhance future census tests.