# **_6.2 Limitations_**

There are several important limitations to be mindful of when considering the implications of our results. We are examining only one country, one set of standards, and two tools. Results could differ substantially with different countries’ standards and tools. In particular, the types of items included in standards and tools appear to have an important relationship with consistency, so variation in the emphasis of standards, e.g., whether they focus on child development outcomes, a quality environment, child-teacher interactions, or other aspects of quality will shape consistency.

Our measures of consistency also embed a number of different issues. We cannot separate testre-test differences<sup>18</sup> from inter-rater reliability or that questions were, in some cases, asked or scaled differently across the tools. Enumerators were also only sometimes observing the same period, although visiting around the same time. Implementing QAS at scale, this problem is likely to be worse, as children and classes at different schools could be observed at time points that are even further apart. We also do not and cannot know whether the QAS tool or diagnostic tool data, or neither, are ultimately “right or wrong.” Although the underlying tools, such as the MELQO tools, have undergone extensive design, testing, and adaptation (Raikes et al. 2019), measuring ECD and quality are fundamentally difficult tasks (Burchinal 2018).

# **_6.3 Policy implications_**

Our findings on the limited consistency of QAS tool results have important implications for the design of QAS, and especially subsequent quality enhancement actions. One potential approach to QAS, which may be a good starting point for fiscally constrained countries working to initially develop a QAS system, would be to start the QAS as a national-level system, using a random sample of schools to identify quality problems, nationally. The diagnostic research tool and QAS tool data were most consistent at the national level in identifying similar strengths and weaknesses in KG quality in Egypt. Policy makers often do not know the specific quality issues their schools face, or the depths of the learning crisis, and so a nationally representative sample can be a good starting point for a QAS (Wiseman 2014). National quality enhancement actions, such as improvements to the curriculum or nation-wide training programs (e.g., teacher education and induction training) can then be revised to target key quality issues and enhance quality.

Where we did find higher levels of consistency was in the similar national results across tools, as there were similar national means on the QAS tool and diagnostic research tool. An important implication of this finding is that shorter, simpler monitoring tools (which are more feasible and affordable for governments to implement) are as good as more detailed research tools for measuring quality, on average, nationally. This has implications for sustainability in both bringing such data collection within the purview of governments (rather than research institutes or other organizations) and in financing data collection on quality. Since the training was one-

> 18 Another potential issue would be whether enumerators successfully observed and entered data for the exact same teacher, classroom, or child. Although there were detailed training instructions on identifying the class list, children, etc., together, there may have been deviations in the field. However, if these were major, we would expect much more consistency on the school level, which is not what we observe.

23