Each of these pairs is treated as one item, the same way the MINEDU did for ECE (assigning one point if at least one of the items in the pair is answered correctly). Thus, the test comprises 23 questions but only 19 items, and the score range goes from zero to 19.<sup>53</sup> The transformed scores were calculated using the Rasch model, which is what the MINEDU uses to analyze and report the results of its academic performance tests, and also the one used in international student performance assessments such as LLECE. Using a broader sample of 1,001 third-grade students who took the ECE test, we calculated three alternative scales of transformed scores using Rasch, setting the mean at 50 and created a scale from 0 to 100, with a standard deviation of 10. The first scale included all 1,001 cases and was calculated using WinBUGS software from a Bayesian perspective with the Markove chain Monte Carlo method (Kim, 2001), which estimates the abilities of all subjects, including extreme cases (those who answered all questions correctly or none).

MINEDU calculates ECE scores using the Rasch model<sup>54</sup> with Winstep, which automatically excludes the extreme cases because according to the authors of the estimation method these cases do not provide any information about the items (Wright & Stone, 1979). Thus, we calculated a second scale using this methodology, which included only 980 cases because 13 students scored 0, and 8 scored 19.

Finally, a common criticism to the use of multiple choice items in educational assessment is that there is no provision in the model for guessing, so a third scale was calculated using the Birnbaum’s three-parameter logistic item response model to control for guessing (Baker & Kim, 2004; Birnbaum, 1968) but this model did not yield significant additional information. The estimations of the subjects’ abilities obtained using this method were very similar to those obtained using the Rasch model, with a high correlation between the two (r=0.9518, p<0.001). The same was true for the estimations of the difficulty of the items (r=0.966, p<0.001). Furthermore, the guessing estimate, though positive for all items, is lower than the expected value of 0.33.

At the end, we decided to work with the first scale, which included all cases using the Rasch model, because when evaluating students for monitoring their progress, it is more important to have measures for all subjects than only for those who can provide information for the model (namely, the non extreme scores). The first scale was preferred also over the third one because it is a simpler model and provides similar information.

# **_b. Item analysis_**

To evaluate the quality of individual items of the test we used both classic statistics and statistics based on the Rasch model using STATA and Winsteps. To evaluate the raw scores we used the percentage of right answers, the item-test and item-rest correlations, and the point-biserial correlations. To evaluate the Rasch scores we used the Infit and Outfit indicators and their z values.

For the raw scores, an item is considered to behave poorly from a psychometric point of view if at least two of the following statements are true: (i) the percentage of right answers is under 20 or over 80; (ii) the non-response rate is higher than 15 percent; (iii) the item-test or item-rest

> 53 There is also the possibility of transforming those scores to percentages of right answers, which would take values from 0 to 100, but we preferred to keep them in their raw form for simplicity and because transforming values to percentages does not provide any additional information.

> 54

> For more information on Rasch modelling and its applications, see (Bond & Fox, 2007). Also see: (Wilson, 2004)

69