To obtain information on the number and kinds of issues identified by the FSCPE and bureau analysts and to determine how the bureau used the information developed during the Full Count Review program, we analyzed the work papers submitted by FSCPE members and other participants in the Full Count Review program. We also analyzed data from the bureau’s Count Review Information System, a database that the bureau used to track issues flagged during the review process. We did not independently verify the information it contained. To identify lessons learned for future improvements, we examined bureau training manuals, statements of work, process models, and other documents that described the objectives, processes, and decision-making criteria. We also reviewed the results of a survey the bureau conducted of FSCPE members that asked them to rate their experience with Full Count Review processes and tools, bureau staff, and the overall effectiveness of the Full Count Review program. In addition, we interviewed managers in the bureau’s Population Division and other officials responsible for implementing the Full Count Review program, as well as three FSCPE members. We performed our audit in Washington, D.C., and the bureau’s headquarters in Suitland, Maryland, between May 2001 and April 2002. Our work was done in accordance with generally accepted government auditing standards. On April 26, 2002, we requested comments on a draft of this report from the Secretary of Commerce. The Secretary forwarded the bureau’s written comments on June 11, 2002 (see app. II). We address them in the “Agency Comments and Evaluation” section of this report. Accurate census results are critical because the data are used to reapportion seats in the House of Representatives and for congressional redistricting. Moreover, census data remain an important element in allocating federal aid to state and local governments. With billions of dollars at stake, the data are scrutinized intensely for accuracy. To help ensure the accuracy of census data, the bureau conducted a number of quality assurance programs throughout the course of the census. One such program was the Full Count Review program, which was designed to rapidly examine, rectify if possible, and clear census data files and products for subsequent processing or public release. The bureau expected data analysts to identify data discrepancies, anomalies, and other data “issues” by checking the data for its overall reasonableness, as well as for its consistency with historical and demographic data, and other census data products. The Full Count Review program ran from June 2000 through March 2001. According to bureau officials, because the bureau could not complete the Full Count Review workload without a costly staff increase, some of the analysts’ work was contracted to members of the FSCPE, an organization composed of state demographers that works with the bureau to ensure accurate state and local population estimates. The bureau contracted with 53 FSCPE members who reviewed data for 39 states and Puerto Rico. Bureau employees reviewed data for the 11 remaining states and the District of Columbia without FSCPE representation in Full Count Review. Bureau and FSCPE analysts were to ensure that (1) group quarters were correctly placed or “geocoded” on census maps, and that their population counts and demographic characteristics appeared reasonable and (2) population counts of other areas were in line with population estimates. They were to describe each issue flagged and provide supporting documentation derived from bureau resources and/or resources of the respective state government. Additionally, bureau officials stated that staff from the regional offices reviewed demographic data from the 50 states, Puerto Rico, and the District of Columbia. They focused on identifying inconsistent demographic characteristics and did not necessarily concentrate on any one particular state or locality. The bureau reimbursed state governments for wages and expenses FSCPE members incurred. A separate set of employees from the bureau’s Population Division assessed issues identified by Full Count Review analysts based on (1) the adequacy of the documentation supporting each issue, and (2) whether or not they believed the issue to be resolvable through follow-up research by the bureau. Those issues deemed to have adequate documentation were classified as a “group quarters,” “housing unit,” or “household” or “other” issue. Bureau officials told us that the remaining issues could not be categorized because the nature of the issue could not be determined from the documentation. Bureau data show that after reviewing census data for 39 states and Puerto Rico, FSCPE members identified a total of 1,402 issues, or about 29 percent of the 4,809 issues collectively flagged during Full Count Review (see table 1). Since the bureau has yet to resolve most of these issues, it is not known whether they are necessarily errors. Table 1 also shows that group quarters issues were those most frequently identified by the bureau, accounting for 1,599 of the 4,809 issues identified (33 percent). Group quarters issues relate to suspected discrepancies in the population counts and locations of prisons, dormitories, nursing homes, and similar group living arrangements. Analysts also identified 479 housing unit issues (10 percent of the total), and 288 household issues (6 percent of the total). With housing unit issues, the count of occupied housing units differed from what analysts expected while household issues had population data for occupied residences that differed from what analysts expected. There were also 383 issues (8 percent) that the bureau classified as “other”. They contained questions concerning the demographic characteristics of the data such as age, race, and gender. The bureau was unable to classify 2,060 issues (43 percent). Bureau officials told us that in these cases, analysts did not provide sufficient documentation for the bureau to determine the nature of the issue. According to bureau officials, bureau analysts identified a larger number of issues than FSCPE members—and a far larger number of issues for which the bureau could not assign a type—because bureau analysts used an automated process that compared data from the 2000 Census to independent benchmarks such as the 1990 Census, and flagged any anomalies. This process alerted bureau officials that there were data discrepancies, but did not indicate their nature. By comparison, FSCPE members compared census data to administrative records and other data, and were better able to document specific issues. Examples of the three issue categories and how they were found include: Group quarters issues: Analysts noticed that the group quarters population count in a particular census tract of a large midwestern city appeared to be too high, while a neighboring tract had a correspondingly low group quarters population count. By comparing state administrative records to information obtained from bureau resources, analysts determined that bureau data had placed college dormitories in the wrong tract. Housing unit issues: An urban area had a large amount of redevelopment since the 1990 census. As part of this, several condominiums and apartment complexes were built which substantially increased the number of housing units in a particular census tract. However, when the analyst compared population data from the 1990 Census and 2000 Census, the 2000 Census did not appear to reflect this increase, and it was flagged. Household issues: Data from the 2000 Census appeared to accurately reflect the large amount of new house construction that had taken place within a specific census tract. However, because the population count differed from that indicated by other data sources, the analyst flagged it as an issue to avoid undercounting the population. Bureau officials told us that they used the Full Count Review program to identify systemic errors such as those that could be produced by software problems. None were found. The officials noted that the bureau generally did not use the Full Count Review program to resolve individual issues. According to bureau officials, the bureau corrected data for 5 of the 4,809 issues prior to the December 31, 2000, release of reapportionment data and the April 1, 2001, release of redistricting data. According to bureau officials, FSCPE members identified the five issues, all of which involved group quarters that were placed in the wrong locations, but the population counts were correct. They included (1) a military base in Nevada, (2) 10 facilities at a college in Wisconsin, (3) 9 facilities at a prison in New York City, (4) 14 facilities at a Washington prison, and (5) a federal medical center in Massachusetts. Bureau officials said that the bureau was able to correct these issues for two reasons. First, FSCPE analysts found them early in the Full Count Review program, while the bureau was processing a key geographic data file and was thus able to incorporate the corrections before the data were finalized. Second, the FSCPE analysts had thoroughly documented the issues and recommended how the bureau should correct the errors. The five errors did not require additional research or field verification. Bureau officials told us that they lacked the time to research the remaining issues, as well as field staff to inspect purported discrepancies prior to the release of the public law data. As a result, the bureau missed an important opportunity to verify and possibly improve the quality of the data, and instead the apportionment and redistricting data were released with more than 4,800 unresolved issues. Until these issues are resolved, uncertainties will surround the accuracy of the census data for the affected localities. Some of the issues might be resolved under the CQR program, which the bureau designed to respond to challenges to housing unit and group quarters population counts received from state, local, or tribal governments. However, as shown in table 2, of the 4,804 issues remaining after Full Count Review, 1,994 (42 percent) were referred to CQR, and of these, 537 (11 percent) were accepted for further investigation. The remaining 1,457 issues referred to CQR did not meet the bureau’s documentation requirements and consequently, the bureau took no further action on them (see app. 1 for the disposition of Full Count Review data issues by state). The overall results of the Full Count Review program and FSCPE members’ participation appear to be mixed. On the one hand, the bureau reported that the Full Count Review program was successful in that it met a number of performance goals.