Using a broader sample of 1,001 third-grade students who took the ECE test,<sup>29</sup> we calculated three alternative scales of transformed scores with Rasch. The first scale was done with WinBUGS and includes all 1,001 cases. The second scale was calculated with Winstep, which automatically excludes the extreme cases and it is the method used by MINEDU (N=980). Finally, a common criticism to the use of multiple choice items in educational assessment is that there is no provision in the model for guessing, so a third scale was calculated using the Birnbaum’s three-parameter logistic item response model to control for guessing (Baker & Kim, 2004; Birnbaum, 1968) but this model did not yield significant additional information. At the end, we decided to work with the first one because it provides information for all subjects and we found no need to control for guessing in this case (see Annex 7.5.a for more details).

After choosing the most appropriate Rasch scale, we analyzed the ECE test to evaluate which scores (raw or Rasch) would be better suited for comparison against the oral test. In theory, the Rasch model is more accurate, as it gives more information, has better psychometric properties and allows interpretation of the results from a pedagogical and curricular point of view. However, the benefits of using Rasch scores depend on the characteristics of the test and how it behaves in a given population. Unidimensionality is the most important premise for using the Rasch model. In this case, we find that the test is not unidimensional (Annex 7.5.c), which makes it unfit for using Rasch as a measurement model.<sup>30</sup> But even when the test cannot be fully adjusted to the Rasch model, transforming scores into a Rasch scale would yield useful information on the difficulty of items as well as provide more elements for interpretation based on curricular standards. For the purposes of this paper, such information is quite valuable.

Moreover, the Rasch scores correlate strongly and significantly with the raw scores (r=0.995, p<.0001), which means that, in this case, we can use almost indistinctively raw or Rasch scores when exploring the relationship between the ECE test and the oral fluency test. In terms of the psychometric properties of the written test (ECE) there are no significant differences between using raw or transformed scores. Both distributions of scores are non-normal, and have moderate internal reliability (Annex 7.5.c): in both cases the reliability coefficients for the sample of 1,001 students fall below the recommended threshold of 0.7, but they reach 0.74 for the sample of 475 students (which is the one finally used for the comparative analysis). Item analyses for both raw and Rasch scores indicate some problems with the construction of the test, specifically with regards to certain items (see Annex 7.5.b), but do not affect the behavior of the test as a whole.

Taking all these elements into account, we decided to use Rasch only for transforming scores and not as a measurement model (Bond & Fox, 2007; Wilson, 2004), which means that conclusions drawn from these analyses are suggestive but not definitive.

# **<mark>4. FINDINGS</mark>**

In general, we find that the reading performance of third grade students in Junin is below the expected level both in terms of fluency and comprehension (Table 5). On average, it is estimated that a third grade student in Junin can read about 57 words in one minute, slightly below the

> 29 To benefit from a larger dataset available, the calculation of Rasch scores was done using test results from all third-grade students who took the ECE test in the Junin study, totalling 1,001 (of which 475 also took the oral test). See Annex 7.4.b for a description of both samples and Annex 7.4.c for details on the adjustment factors.

> 30 This issue deserves further discussion, but it is beyond the purposes of this paper.

29