Established in 1965, Head Start is a federally funded early childhood development program that served over 900,000 children at a cost of $6.8 billion in 2004. Head Start offers low-income children a broad range of services, including educational, medical, dental, mental health, nutritional, and social services. Children enrolled in Head Start are generally between the ages of 3 and 5 and come from varying ethnic and racial backgrounds. Head Start is administered by HSB within ACF. HSB awards Head Start grants directly to local grantees. Grantees may develop or adopt their own curricula and practices within federal guidelines. Grantees may contract with other organizations—called delegate agencies—to run all or part of their local Head Start programs. Each grantee or delegate agency may have one or more centers, each containing one or more classrooms. In this report, the term “grantee” is used to refer to both grantees and delegate agencies. Figure 1 provides information on the numbers of Head Start grantees, delegate agencies, centers and classrooms. Since the inception of Head Start, questions have been raised about the effectiveness of the program. In 1998, we reported that Head Start lacked objective information on performance of individual grantees and Congress enacted legislation requiring HSB to establish specific educational standards applicable to all Head Start programs and allowed development of local assessments to measure whether the standards are met. HSB implemented this legislation by developing the Child Outcomes Framework to guide Head Start grantees in their ongoing assessment of the progress of children. The Framework covers a broad range of child skill and development areas and incorporates each of the legislatively mandated goals, such as that children “use and understand an increasingly complex and varied vocabulary” and “identify at least 10 letters of the alphabet.” Since 2000, HSB has required every Head Start grantee to include each of the areas in the Framework in the child assessments that each grantee adopts and implements. The eight broad areas included in the Framework are language development, literacy, mathematics, science, creative arts, social and emotional development, approaches to learning, and physical health and development. Grantees are permitted to determine how to assess children’s progress in these areas. These assessments are to align with the grantee’s curriculum; as a result the specific assessments vary across the grantees. The assessments occur 3 times each year and generally involve observing the children during normal classroom activities. The results of the assessments are used for the purposes of individual program improvement and instructional support and are not aggregated across grantees or systematically shared with federal officials. The NRS, prompted by the April 2002 announcement of President Bush’s Good Start, Grow Smart initiative, builds on the 1998 legislation by requiring all Head Start programs to implement the same assessment, twice a year, to all 4- and 5-year-old Head Start participants who will attend kindergarten the following year. When President Bush announced this initiative in April 2002, it called for full implementation in fall 2003; as a result the NRS was developed and preparations for implementation occurred within an 18-month period. See figure 2. Shortly after the President announced this initiative, HSB hired a contractor to assist it in developing and implementing the NRS. The contractor, working closely with HSB, was responsible for the design and field testing of the NRS, including developing training materials to support national implementation of the reporting system by grantees. HSB also worked with the Technical Work Group and others throughout implementation of the NRS. The Technical Work Group includes 16 experts in such areas as child development, educational testing, and bilingual education. They advised HSB on the selection of assessments, the appropriateness of the assessments in addressing the mandated indicators, the technical merit of the assessments, and the overall design of the NRS. While the Technical Work Group members offered advice, the group members were not always in agreement with each other and HSB was not obligated to act on any of the advice it received. A list of the Technical Work Group members and their professional affiliations is included in appendix I. Through focus groups, teleconferences, and various correspondences, HSB officials communicated to Head Start grantees the purpose of the NRS and their plans for administering the assessment. Focus groups and discussions were held with various interested parties, including Head Start managers and directors and experts from universities and the public sector, on issues ranging from strengths and limitations of various assessment tools to strategies for assessing non-English speaking children. HSB also received input through a 60-day public comment period, from mid-April to June 2003. Another contractor developed a Computer-Based Reporting System (CBRS) for the NRS. Local Head Start staff use the CBRS to enter descriptive information about their grantees, centers, classrooms, teachers, and children, as shown in table 1, as well as to keep track of which children have been assessed. HSB analyzes the descriptive information from the CBRS in conjunction with the child assessment data to develop reports on the progress of specific subgroups of children. For example, HSB can report separately on the average scores of children enrolled in part-day programs and those enrolled in full-day programs. HSB, with assistance from the contractors, worked to ensure local staff received adequate training on administering the assessment and using the CBRS, and provided guidance on how to obtain consent from parents. Training and certification of all assessors was required so that all assessors would administer the NRS in the same way. Two-and-a-half day training sessions were held at eight sites throughout the U.S. and Puerto Rico during July and August 2003. Roughly 2,800 individuals completed the training, of which 484 were certified in both English and Spanish. In turn, these certified trainers held training sessions locally to train and certify additional staff who would be able to administer assessments. The development of educational tests is a science in itself, to which university departments, professional organizations, and private companies are devoted. Among the most important concepts in test development are validity and reliability. Validity refers to whether the test results mean what they are expected to mean and whether evidence supports the intended interpretations of test scores for a particular purpose. Reliability refers to whether or not a test yields consistent results. Validity and reliability are not properties of tests; rather, they are characteristics of the results obtained using the tests. For example, even if a test designed for 4th graders were shown to produce meaningful measures of their understanding of geometry, this wouldn’t necessarily mean that it would do so when administered to 2nd or 6th graders or with a change in directions allowing use of a compass and ruler. Test developers typically implement “pilot” tests that represent the actual testing population and conditions and they use data from the pilot to evaluate the reliability and validity of a test. This process generally takes more than 1 year, especially if the test is designed to measure changes in performance. In the remainder of the report, we will discuss how the focus of the NRS was determined and the assessment was developed, HSB’s response to problems in initial implementation as well as some implementation issues that remain unaddressed, and the extent to which the assessment meets the professional and technical standards to support specific purposes identified by HSB. The NRS assesses vocabulary, letter recognition, simple math skills, and screens for understanding of spoken English. As initially conceived by HSB, the NRS was to gauge the progress of Head Start children in 13 congressionally mandated indicators of learning. However, time constraints and technical matters precluded HSB from assessing children on all of the indicators and led HSB to consider, and eventually adopt, portions of other assessments for use in the NRS. The 18 months from announcing the Good Start, Grow Smart initiative, of which the NRS is a part, to implementing the assessment was not enough time for HSB to develop a completely new assessment. Therefore, HSB, with the advice of its contractor and the Technical Work Group, chose to borrow material from existing assessments. Concerns raised by Technical Work Group members and the contractor about the length and complexity of the assessment and the technical adequacy of individual components eventually led to limiting the areas assessed in the NRS, from 13 skills to 6. The six legislatively mandated skills that HSB targeted included whether children in Head Start: use increasingly complex and varied spoken vocabulary; understand increasingly complex and varied vocabulary; identify at least 10 letters of the alphabet; know numbers and simple math operations, such as addition and subtraction; for non-English speaking children, demonstrate progress in listening to and understanding English; and for non-English speaking children, show progress in speaking English. In April and May of 2003 an assessment that included 5 components covering the 6 skills was field tested with 36 Head Start programs to examine the basic adequacy of the NRS, as well as the method for training assessors, and the use of the CBRS. The field test also included a Spanish version of the NRS. Based on the field test, one component–-phonological awareness, or one’s ability to hear, identify, and manipulate sounds–-was eliminated. While this component examined an area that experts have linked to prevention of reading difficulties, the test used to assess it was problematic. HSB moved forward with the other components of the NRS. The four components of the NRS each measure one or more of the six legislatively-mandated indicators. The four components that comprise the NRS are from the following tests: Oral Language Development Scale (OLDS) of the Pre-Language Assessment Scale 2000 (Pre-LAS 2000), Third Edition of the Peabody Picture Vocabulary Test (PPVT-III), Head Start Quality Research Centers (QRC) letter-naming exercise, and Early Childhood Longitudinal Study of a kindergarten cohort (ECLS-K) math assessment. Some or all of each test was previously used for other studies, and the PPVT and letter naming were previously used in studies of Head Start children. Three of the four tests were modified from their original version, as shown in table 2. Figures 3 and 4 are examples from the letter naming and early math skills components of the NRS. Figure 5 is an example of the type of item used in the vocabulary (PPVT) component of the NRS. Here are some letters of the alphabet. GESTURE WITH A CIRCULAR MOTION AT LETTERS AND SAY: Point to all the letters that you know and tell me the name of each one. Go slowly and show me which letter you’re naming. INDICATE ONLY CORRECTLY NAMED LETTERS ON ANSWER SHEET. WHEN CHILD STOPS NAMING LETTERS, SAY: Look carefully at all of them. Do you know any more? KEEP ASKING UNTIL CHILD DOESN’T KNOW ANY MORE.

Summary: In September 2003, the Head Start Bureau, in the Department of Health and Human Services (HHS) Administration for Children and Families (ACF), implemented the National Reporting System (NRS), the first nationwide skills test of over 400,000 4- and 5-year-old children. The NRS is intended to provide information on how well Head Start grantees are helping children progress. Given the importance of the NRS, this report examines: what information the NRS is designed to provide; how the Head Start Bureau has responded to concerns raised by grantees and experts during the first year of implementation; and whether the NRS provides the Head Start Bureau with quality information. The Head Start Bureau developed the NRS to gauge the extent to which Head Start grantees help children progress in specific skill areas, including understanding spoken English, recognizing letters, vocabulary, and early math. Due to time constraints and technical matters, the Head Start Bureau adapted portions of other assessments for use in the NRS. Head Start Bureau officials have responded to some concerns raised during the first year of NRS implementation, but other issues remain. For example, the Head Start Bureau has modified training materials and is exploring the feasibility of sampling. However, it is not monitoring whether grantees are inappropriately changing instruction to emphasize areas covered in the NRS. Head Start Bureau officials have said NRS results will eventually be used for program improvement, targeting training and technical assistance, and program accountability; however, the Head Start Bureau has not stated how NRS results will be used to realize these purposes. Currently, results from the first year of the NRS are of limited value for accountability purposes because the Head Start Bureau has not shown that the NRS meets professional standards for such uses, namely that (1) the NRS provides reliable information on children's progress during the Head Start program year, especially for Spanish-speaking children, and (2) its results are valid measures of the learning that takes place. The NRS also may not provide sufficient information to target technical assistance to the Head Start centers and classrooms that need it most.