substantial consistency, suggesting these aspects of child development may perform relatively better as aspects of a QAS. Ten items had moderate consistency, including some infrastructure items, teacher’s supports and children’s multidisciplinary skills. Other more subjective aspects of the classroom environment, materials being present and used, and almost all the measures of pedagogy had lower consistency. On the class and school levels, summary measures, corresponding to levels of the QAS for different dimensions and overall, were correspondingly inconsistent.

Consistency on QAS dimensions, modeled at the classroom level, showed relatively weak relationships with community, school, class, and teacher characteristics. While it is still possible that measures are _consistently_ biased, these characteristics at least do not contribute to additional variability. There were also, importantly, consistent results whether an independent data collection firm staff member or MoETE supervisor was acting as the enumerator. This finding suggests that QAS data collection efforts can be embedded within ministries of education and achieve comparable reliability and validity to external evaluations. Such embedding has important implications for cost, feasibility, and sustainability of QAS systems in LMICs.

Simulations of targeting potential quality enhancement actions using the two different data show relatively low consistency as well. For instance, only 31% of classrooms identified as in the top 10% of classes overall in the QAS tool were also so identified in the diagnostic research tool. Results were somewhat more consistent with coarser targeting (targeting 25% rather than 10%, for instance). Results were not substantially more consistent on the school or district level; in some cases they were more consistent and in other cases less consistent. Identifying the bottom 10% or 25% of students in terms of letter recognition was the most consistent in the simulations, which follows from the high consistency of individual foundational skills items.

Taken together, these results suggest that QAS tools are often subject to substantial measurement error. Even on identical items, inconsistency was often substantial. Some of this inconsistency may be coming from different timing of data collection. However, timing will vary at scale as well since different schools would be observed on different days. Different choices in the design and implementation of tools for QAS may also yield fundamentally different decisions on where and how to intervene. For instance, different scales may have contributed to some differences in items that were not identical.

These findings emphasize the importance of QAS tools being carefully designed and tested before implementation, and further, that tools should be calibrated based on their ability to capture information with policy relevance. The results also underscore that QAS tools should not be designed or implemented for high-stakes, summative decisions about specific schools or teachers. Instead, QAS tools can, depending on feasibility and funding, inform national action or act as a formative starting point for quality enhancement. For example, one approach is to use a QAS tool to identify schools that should be followed up with additional observation, rather than making decisions based on QAS tool results alone. We discuss these options in greater detail below.

22