measuring reading skills with disadvantaged students, taking as an example the Ashaninkas; and fourth, on how reading tests can be used to guide policymaking and strengthen accountability in the education system.

# **5.1. Scope and limitations of the two tests to measure reading ability**

We found that the oral fluency test and the written comprehension test adequately measure students’ reading abilities, and there are no differences in the discrimination power between both tests. However, there are some issues related to test construction and administration that need to be discussed.

With regards to the tests’ construction, item and test analysis for both raw and Rasch scores of the written test revealed inadequate adjustment to psychometric assumptions in our sample: both scores have non-normal distributions, only moderate reliability, and lack unidimensionality; the first items are too easy for this sample, and the test needs more items of specific difficulty level to adequately measure all levels of reading abilities. In addition, the item analysis shows problems with several items, thought there are more problems using raw scores than with Rasch scores.

All of these problems could be explained in part by the limited number of items measuring second grade level abilities, as stated by curricular standards. The first items turned out to be too easy because they were measuring first-grade skills, while the other items that presented problems were actually measuring third and fourth grade skills.

However, the problems found pertain to this particular test (in this particular sample) and not to the written comprehension tests in general. The UMC’s ECE tests can be a powerful tool to identify at-risk students and schools, as well as to track progress at the school level over the years, which would yield the necessary information needed to design and implement targeted programs and identify and reward successful educational experiences. The Rasch model makes it possible to establish equivalences with other standardized tests that are reported using the Rasch model, which, for instance, would allow tracking the progress of a child or a school over time. This is of great relevance now that the census evaluation will be done on an annual basis. All of this, provided that test developers in UMC design the test to fit the Rasch model from the start; include a greater number of items in rotated tests forms; and have enough time to pilot the items in a sample to make sure the test is unidimensional and reliable with an Alpha coefficient of at least 0.70, and provides an adequate mapping of skills based on the difficulty of the items.

The oral test could not be analyzed psychometrically, but there are no evidences of problems in the construction of the fluency measure. There are, however, indications of validity constraints with both comprehension measures (Q&A and Recall). Their low correlation with fluency and written comprehension indicates that they may have low predictive and concurrent validity. It seems that the Q&A comprehension measure is not able to adequately discriminate comprehension skills because most children were able to answer all the questions correctly. This led to little variance in the distribution of scores, which in turn decreased its correlations with other variables such as fluency and written comprehension. Indeed, Q&A Comprehension correlates the lowest with the rest, and these correlations become insignificant when done at the school level, as variance reduces even more. This is consistent with the results found in the broader Junin study (Banco Mundial, 2008), which led the authors to decide not to use Q&A Comprehension results. This problem could be solved by using a longer passage and adding more questions of greater difficulty. For instance, Fuchs, Fuchs, Hosp and Jenkins (2001) suggest ten questions per 400-word passage.

45