**Figure 14. Normal Q-Q Plots for Raw and Rasch Scores**

<!-- Start of picture text --> a. Rasch scores b. Rasch scores (excl. students with scores <= 15.23)<br>36.73142 50.05267 63.37392 38.85654 50.51175 62.16695<br>20 40 60 80 30 40 50 60 70<br>Inverse Normal Inverse Normal<br>Grid lines are 5, 10, 25, 50, 75, 90, and 95 percentiles Grid lines are 5, 10, 25, 50, 75, 90, and 95 percentiles<br>c. Fluency Scores d. Fluency Scores (excl. students with score=0)<br>5.846713 54.98073 104.1148 18.52683 60.45335 102.3799<br>-50 0 50 100 150 0 50 100 150<br>Inverse Normal Inverse Normal<br>Grid lines are 5, 10, 25, 50, 75, 90, and 95 percentiles Grid lines are 5, 10, 25, 50, 75, 90, and 95 percentiles<br>80 80<br>60 63.49 60 63.49<br>49.97797 49.98354<br>40 40<br>39.71<br>37.14<br>Scale using 1001 cases Rasch Scale using 1001 cases Rasch<br>20 20<br>150 150<br>100<br>105.8824 100<br>105.8824<br>50 53<br>50<br>56.16327<br>0 0 22<br>Fluency: Words read per minute Fluency: Words read per minute<br>0<br>-50<br><!-- End of picture text -->

# **_Reliability_**

To test the reliability of the ECE we use the Cronbach's Alpha coefficient for the raw scores (Table 25).<sup>56</sup> For the Rasch scores, we use the Winsteps person reliability index ( _pri_ ),<sup>57</sup> which is equivalent to the traditional "test" reliability of Cronbach's Alpha (Table 27). Results indicate that reliability of the test is similar for raw and Rasch scores, in both cases moderate: α =0.675 and _pri_ = 0.61, respectively (excluding extreme cases for the latter). These coefficients are below the lowest bar recommended (0.7), suggesting that the reliability of the written test is not good enough. Given these results, one would expect that the internal problems of the test would also affect its correlation to other measures of the same or related constructs. However, internal reliability for Sample B (which is the one being used for comparison) is higher and above the

> 56 Alpha is computed by correlating all the scores on individual items, with the overall score on the test. Tests with high reliability, i.e. those with high internal consistency, will achieve an alpha coefficient of 0.75 or more on a scale of 0 to 1 where a high score indicates high reliability. The minimum alpha for acceptable reliability is 0.7.

> 57 The person reliability index “indicates the replicability of person ordering we could expect in this sample of persons were given another a parallel set of items measuring the same construct” (Bond & Fox, 2007, p. 40). The person reliability tells us if the test discriminates the sample into enough levels. Low values indicate a narrow range of person measures, or a small number of items. To increase person reliability, one can test persons with more extreme abilities (high and low) or lengthen the test. Improving the test targeting may help slightly. An index of 0.9 discriminates into 3 or 4 levels; an index of 0.8 discriminates into 2 or 3 levels; and an index of 0.5 discriminates into 1 or 2 levels. On the other hand, low item reliability means that the sample is not big enough to precisely locate the items on the latent variable. Literature supporting or elaborating on this index is very limited (see WINSTEPS).

73