metadata: authors: 0: author: Seiden,Jonathan Michael
1: author: Hasan,Amer
2: author: Luna Bazaldua,Diego Armando
count: Pakistan
docna: 0: docna: Early Learning in South Punjab, Pakistan:
            Investigating Child Development and Classroom Quality
docty: Policy Research Working Paper
theme: Social Safety Nets,Mitigation,Gender,Human Development and Gender,Nutrition,Education Facilities,Skills Development,Social Protection,Social Development and Protection,Reproductive and Maternal Health,Environment and Natural Resource Management,Standards, Curriculum and Textbooks,Health Service Delivery,Labor Market Policy and Programs,Active Labor Market Programs,Child Health,Climate change,Adaptation,Teachers,Nutrition and Food Security,Education,Health Systems and Policies,Social protection delivery systems
lndinstr: 0: lndinstr: Investment Project Financing
lndinstr_key: 1353855
abstracts: cdata!: In Pakistan, learning poverty among
            primary school aged children is estimated to be as high as
            77 percent, but little data exist on early learning
            experiences. This paper describes the state of classroom
            quality in 1,395 classrooms and the early childhood
            development status of 8,249 children in a representative
            sample of 894 public schools in South Punjab, using two
            measurement tools: The Teach ECE classroom observation tool,
            which describes the structural and process quality features
            of classrooms, and the Anchor Items for the Measurement of
            Early Childhood Development Direct Assessment which reports
            on early learning and developmental outcomes of children
            aged 4 to 6 years. The paper finds key gaps in the
            foundational skills of young children and areas for
            improvement in both the physical classroom and teaching
            practices. In examining the relationships between teaching
            practices and early childhood development outcomes, the
            analysis finds a strong positive relationship across the
            areas of process quality and domains of childhood
            development. Children studying in a high-quality classroom
            have outcomes that are equivalent to having been in school
            nine months longer than children of similar ages in an
            average quality classroom, suggesting that a sharper focus
            on teaching quality may improve early childhood development
            outcomes and school readiness. The findings also show that
            after accounting for teaching quality, degrees and
            certification are not associated with early childhood
            development outcomes, but that classes taught by female
            teachers have better early childhood development outcomes.
display_title: Early Learning in South Punjab,
            Pakistan: Investigating Child Development and Classroom Quality
listing_relative_url: /research/2024/05/34314650/early-learning-south-punjab-pakistan-investigating-child-development-classroom-quality
EnvironmentalCategory: B
topicv3: Quality of Education m1326344 491,Preschool Education m1327265 1585
publishtoextweb_dt: 2024-05-07T00:00:00Z
dois: 10.1596/1813-9450-10764
no_of_pages: 55
author_keys: 0: author_key: 1914791
1: author_key: 807987
2: author_key: 1803322
model_extractions: Policy Research Working Paper 10764 Early Learning in South Punjab, Pakistan Investigating Child Development and Classroom Quality _Jonathan Seiden Amer Hasan Diego Luna Bazaldua_ Education Global Practice May 2024 A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.
Policy Research Working Paper 10764

# **Abstract**

In Pakistan, learning poverty among primary school aged children is estimated to be as high as 77 percent, but little data exist on early learning experiences. This paper describes the state of classroom quality in 1,395 classrooms and the early childhood development status of 8,249 children in a representative sample of 894 public schools in South Punjab, using two measurement tools: The Teach ECE classroom observation tool, which describes the structural and process quality features of classrooms, and the Anchor Items for the Measurement of Early Childhood Development Direct Assessment which reports on early learning and developmental outcomes of children aged 4 to 6 years. The paper finds key gaps in the foundational skills of young children and areas for improvement in both the physical classroom and teaching practices. In examining the relationships between teaching practices and early childhood development outcomes, the analysis finds a strong positive relationship across the areas of process quality and domains of childhood development. Children studying in a high-quality classroom have outcomes that are equivalent to having been in school nine months longer than children of similar ages in an average quality classroom, suggesting that a sharper focus on teaching quality may improve early childhood development outcomes and school readiness. The findings also show that after accounting for teaching quality, degrees and certification are not associated with early childhood development outcomes, but that classes taught by female teachers have better early childhood development outcomes.

This paper is a product of the Education Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at ahasan1@worldbank.org. A verified reproducibility package for this paper is available at http://reproducibility.worldbank.org, click **here** for direct access.

<!-- Start of picture text --> TRANSPARENT<br>ANALYSIS<br>G<br>R E<br>N<br>S<br>I P<br>Y E<br>K A<br>C A<br>R P<br>LI R<br>O E<br>O C<br>W R<br>H<br>S<br>P<br><!-- End of picture text --> _The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Early Learning in South Punjab, Pakistan: Investigating Child Development and Classroom Quality**<sup>**1**</sup>

Jonathan Seiden<sup>2</sup> Amer Hasan<sup>3</sup> Diego Luna Bazaldua<sup>2</sup>

## **Keywords: Early Childhood Development, Classroom quality, Pakistan, pre-primary, primary**

**JEL Codes: I20, I21**

> 1 The authors would like to thank the Programme Monitoring and Implementation Unit of the School Education Department of the Government of the Punjab for sharing the microdata used in this paper. Ghulam Dastgeer provided helpful clarifications on survey protocols. The authors would also like to thank Elizabeth Hentschel for the analysis reported in Table 1. All errors are our own. This work is a product of the staff of the World Bank with external contributions. The World Bank does not guarantee the accuracy of the data included in this work and does not assume responsibility for any errors, omissions, or discrepancies in the information, or liability with respect to the use of or failure to use the information, methods, processes, or conclusions set forth. The boundaries, colors, denominations, links/footnotes, and other information shown in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. The citation of works authored by others does not mean the World Bank endorses the views expressed by those authors or the content of their works.

2 Harvard Graduate School of Education and World Bank, Education Global Practice 3 World Bank, Education Global Practice
# **Introduction**

Pakistan is the fifth largest country in the world by population and home to over 53 million children under the age of eight (Pakistan Bureau of Statistics, 2017). Twenty million children are out of school (Ersado et al., 2023) and 77% of 10-year-olds are estimated to be unable to read and understand a simple, age-appropriate paragraph (World Bank, processed). Yet, little is known about young children’s school readiness or about the quality of their earliest educational experiences.

This study aims to fill this knowledge gap and documents the skills pre-primary-aged children can demonstrate and the quality of the classrooms they attend in South Punjab, Pakistan. The Punjab province is home to over half of the country’s population but there is widespread variation across the province in human development outcomes. As shown in **Table 1** , data from the most recent Multiple Indicator Cluster Surveys (MICS) study show that enrollment in early childhood education is far lower in South Punjab (27% of children enrolled) than in the rest of the province (36% of children enrolled) (Bureau of Statistics Punjab, Planning & Development Board, Government of the Punjab & UNICEF, 2018). Not surprisingly, the share of children developmentally on track is also markedly different – 52% of 3 and 4 year olds are reported by their parents as developmentally on track in South Punjab compared to 63% of children in the rest of the province (Bureau of Statistics Punjab, Planning & Development Board, Government of the Punjab & UNICEF, 2018).

_Table 1. Share of 3 to 4-year-old Children Enrolled in ECE and Share Reported to be Developmentally on Track_

||**%**<br>Mean|**Enrolled in ECE**<br>95% Confidence Interval|**% De**<br>Mean|**velopmentally on Track**<br>95% Confidence Interval|
|---|---|---|---|---|
|**South Punjab**|**27.32**|**(25.73, 28.92)**|**52.42**|**(50.67, 54.17)**|
|Male|27.85|(25.91, 29.78)|50.21|(47.91, 52.50)|
|Female|26.79|(24.57, 29.00)|54.72|(52.49, 56.96)|
|**Central & North Punjab**|**35.71**|**(34.69, 36.72)**|**62.55**|**(61.54, 63.55)**|
|Male|35.53|(34.18, 36.87)|60.67|(59.29, 62.05)|
|Female|35.90|(34.48, 37.31)|64.54|(63.16, 65.91)|
|**Punjab Overall**|**33.34**|**(32.48, 34.20)**|**59.69**|**(58.81, 60.57)**|
|Male|33.37|(32.36, 34.49)|57.73|(56.54, 58.92)|
|Female|33.30|(32.10, 34.50)|61.75|(60.57, 62.93)|

**_Source: MICS, Punjab 2017-2018_** (Bureau of Statistics Punjab, Planning & Development Board, Government of the Punjab & UNICEF, 2018)

The quality of pre-primary education plays an important role not just in early childhood development but potentially also in primary education (Akyeampong et al., 2023). Research shows that children who have high-quality early childhood education experiences have better cognitive and socio-emotional outcomes and that early school readiness skills are predictive of later schooling success (Duncan et al., 2007; Engle et al., 2011) – though whether this success is short2
term or long-term is far from a settled point in studies from low- and middle-income countries (See Akyeampong et al., 2023).

There is growing evidence of the benefits of quality pre-primary education as a costeffective catalyst for improvements in learning outcomes and long-term economic benefits later in life. Compared to other educational policies to improve foundational learning, global experts propose that investing in quality pre-primary education can help address learning inequities and reduce deficits by the time children enter primary school (Global Education Evidence Advisory Panel, 2020). This is confirmed by results from international large-scale assessment studies, which have shown that enrollment in pre-primary education results in higher average learning outcomes in reading, mathematics and science (Mullis et al., 2020; Mullis et al., 2023). Yet, even when investing in improving access to and quality of early childhood education can be seen as smart policy, most of the studies that have investigated this linkage come from high-income countries (Holla et al., 2021). Fortunately, more and more evidence on this linkage is being generated in low- and middle-income contexts to support the investment in quality early childhood education (Augsburg et al., 2022; Hasan et al., 2019; Nakajima et al., 2021).

Hence, this study provides a window into one of the less explored and more important policy aspects of young children’s education: what goes on in the early childhood education classroom in a lower-middle-income setting such as Pakistan – specifically in South Punjab – and how early childhood settings are associated with children’s cognitive and socioemotional development. It attempts to fill a knowledge gap by describing what relevant skills pre-primary aged children can do and the quality of classrooms in which they study. We also report on how children’s skills and classroom quality vary by student, teacher, and school characteristics and assess the correlations between these variables to understand the role that quality may play in supporting young children’s acquisition of school readiness skills. We use a large, representative dataset of public schools in South Punjab to answer the following core research questions:

- 1: What do pre-primary classrooms in South Punjab look like?

- 2: What is the structural and process quality of these classrooms?

- 2.1: How does quality differ by classroom and teacher factors?

- 3: What can children in pre-primary classrooms do?

- 4: How are children’s early learning and development related to individual, teacher, and class factors?

# **Data**

The data presented in this paper draw on the Baseline Survey on Early Childhood Care and Education (ECCE) carried out under the Education Component of the Punjab Human Capital Investment Project (PHCIP). Child-level data on the Anchor Items for the Measurement of Early Childhood Development: Direct Assessment (AIM-ECD DA) was collected from January to April 2022. AIM-ECD DA data was collected by 32 certified enumerators. Observations of classroom 3
# “
of 900 project schools. Within each school, all classes containing pre-primary-aged children 4-6 years were included for observation with Teach ECE. In most schools, a single _Katchi_ and a single Grade 1 class were observed. _Katchi_ classes are an informal grade before primary grade 1. A random sample of children aged 4-6 in attendance on the date of data collection was drawn by selecting five children from the _Katchi_ class and five children from the Grade 1 class when both were available. In cases where attendance was fewer than 10, all age-eligible children were asked to participate.

Out of 900 sampled schools, the survey contained 1,482 Teach ECE observations from classrooms in 899 schools (one school did not have an adult present) and attempted to administer AIM-ECD DA to 8,613 children. The survey excluded data from children who either did not assent to participate in AIM-ECD DA (n=45) or did not meet the quality check requirements (n=5) and reported a final working sample of 8,563 children. Despite attempting to interview 10 children per school, in 137 schools, fewer than 10 age-appropriate children were available and in four schools the number of children interviewed was 11. Additional cleaning and analysis of the data revealed 27 classrooms with repeated observations. In these cases, the first observation of the classroom was kept and the later observation was discarded. Six classrooms had no corresponding child data and two classrooms with child data had no corresponding quality observation. To allow for correlational analyses, we focus on classrooms with both child and classroom-level data. In addition to excluding observations due to data issues, we also focus on a substantive subset of the original data consisting exclusively of _Katchi_ and Grade 1 classes. This led to the exclusion of data from 3 higher-grade classrooms (which had pre-primary aged children) as well as 51 ECE classrooms, a new type of classroom with a dedicated curriculum in Punjab province. The subset of data remaining allows us to better reflect on the baseline status of pre-primary education in South Punjab before the widespread implementation of ECE. After removing these classrooms and corresponding children from our sample, we are left with a final analytical sample of 8,249 children (4,806 boys and 3,443 girls) studying in 1,395 classrooms (taught by nearly equal numbers of male and female teachers) in 894 schools.

The total number of schools and primary-aged enrollment in South Punjab as a whole, in the project schools, and in the analytical sample is shown below in **Table 2** . This approach creates a sample that is statistically representative of the 3,400 PHCIP schools with nearly 530,000 primary-aged children. While not statistically representative of the 11 districts as a whole, the general composition of the sample closely reflects the composition of districts, with Rahimyar Khan being under-represented in both the project and sample, and Rajanpur being slightly overrepresented.

5
_Table 2. Sample and project area by district compared to South Punjab_

|**District**|**N**|**umber of s**|**chools**|**Prim**|**ary enrol**|**lment**|
|---|---|---|---|---|---|---|
||**South**<br>**Punjab**|**PHCIP**|**Analytical**<br>**Sample**|**South**<br>**Punjab**|**PHCIP**|**Analytical**<br>**Sample**|
||2,145|473|126|269,425|71,282|17,888|
|**Bahawalnagar**|(13%)|(13.9%)|(14.1%)|(12.5%)|(13.5%)|(13.1%)|
||1,652|400|107|196,742|52,606|14,436|
|**Bahawalpur**|(10%)|(11.8%)|(12%)|(9.2%)|(10%)|(10.6%)|
||1,242|270|72|152,944|39,264|10,023|
|**Bhakkar**|(7.6%)|(7.9%)|(8.1%)|(7.1%)|(7.4%)|(7.4%)|
||1,567|194|45|203,814|31,322|6,967|
|**D.G. Khan**|(9.5%)|(5.7%)|(5%)|(9.5%)|(5.9%)|(5.1%)|
||929|235|63|101,302|33,400|8,677|
|**Khushab**|(5.7%)|(6.9%)|(7%)|(4.7%)|(6.3%)|(6.4%)|
||1,506|454|117|187,575|65,181|15,778|
|**Layyah **|(9.2%)|(13.4%)|(13.1%)|(8.7%)|(12.4%)|(11.6%)|
||750|241|63|110,317|38,601|10,146|
|**Lodhran **|(4.6%)|(7.1%)|(7%)|(5.1%)|(7.3%)|(7.5%)|
||1,201|282|80|144,850|44,758|12,465|
|**Mianwali**|(7.3%)|(8.3%)|(8.9%)|(6.7%)|(8.5%)|(9.2%)|
||1,731|372|96|266,126|67,497|17,512|
|**Muzaffargarh**|(10.5%)|(10.9%)|(10.7%)|(12.4%)|(12.8%)|(12.9%)|
||2,773|165|45|386,011|22,974|5,848|
|**Rahim Yar Khan**|(16.9%)|(4.9%)|(5%)|(18%)|(4.4%)|(4.3%)|
||942|314|80|129,564|60,853|16,441|
|**Rajanpur**|(5.7%)|(9.2%)|(8.9%)|(6%)|(11.5%)|(12.1%)|
|**Total**|**16,438**|**3,400**|**894**|**2,148,670**|**527,738**|**136,181**|

Source: Author’s calculations using the Punjab Annual School Census, 2021.

# **Tools**

The AIM-ECD DA (or AIM-ECD for short) was used to measure early childhood developmental outcomes for children aged 4-6 years old (Pushparatnam et al., 2021; World Bank, 2021). The Teach Early Childhood Education (Teach ECE) tool was used to conduct classroom observations to capture the quality of teaching practices and interactions between teachers and children (Ding et al., 2024; Pushparatnam & Ding, 2021). Each tool is described below.

The AIM-ECD DA is a tool that measures early childhood development through a set of activities that a trained enumerator engages in with a child. The AIM-ECD DA contains items in four critical domains of early development: Early Numeracy, Early Literacy, Executive Function, and Social-Emotional development and shares items from the Measure of Learning and Quality Outcomes Module on Early Learning and Development (MELQO MODEL) as well as the International Development and Early Learning Assessment (IDELA) (Pisani et al., 2018; Raikes et al., 2019).

6
The standard global version of the assessment comprises 93 items grouped in 14 subtasks (activities) chosen after analyzing a large cross-country dataset consisting of 24,533 children in 23 countries and identifying items with robust psychometric properties across contexts (Pushparatnam et al., 2021). In addition to the standard AIM-ECD subtasks items, during the contextualization process for Pakistan, a single subtask testing English letters was included so that children could be assessed on both their Urdu and English letter knowledge. **Figure 2** shows the AIM-ECD DA subtasks and the developmental domains to which each subtask belongs (World Bank, 2022).

While the AIM-ECD was not designed for specific use in Pakistan, the constructs it assesses are highly relevant according to both provincial and national sources. _Appendix A: Crosswalk between Student Learning Objectives and AIM-ECD Subtasks_ reviews the alignment of the assessment with the Single National Curriculum and finds good coverage (National Curriculum Council, 2020). _Appendix B: Crosswalk between Punjab Early Learning and Development Standards and AIM-ECD_ reviews the contents of Punjab-specific standards and the AIM-ECD assessment, again finding good matches for many constructs deemed important by the provincial education authorities (Quaid-e-Azam Academy for Educational Development & UNICEF, n.d.).

_Figure 2. AIM-ECD DA Subtasks and Domain Assignments_ <!-- Start of picture text --> AIM-ECD Overall<br>English Letter<br>(Early Learning and Identification<br>Development)<br>Early Early Executive Social English Letter<br>Numeracy Numeracy Functioning Emotional Identification<br>Head, Toes,<br>Listening Receptive Perspective<br>Knees,<br>Comprehensio Spatial Shoulders Taking/Empat<br>n Story Vocabulary Task hy<br>Urdu Letter Producing a Forward Digit<br>Identification Set Span<br>Initial Sound Simple Backwards<br>Addition and<br>Discrimination Subtraction Digit Span<br>Number<br>Name Writing Identification<br>Number<br>Comparison<br>Shape<br>Identification<br><!-- End of picture text --> The Teach ECE tool was developed to measure both structural and process aspects of classroom quality (World Bank, 2021). Structural aspects of quality refer to static aspects of quality that exist in the classroom, for example the quality and availability of learning materials, adequacy of the classroom environment, teacher-to-student ratio, and qualifications of the ECE 7
teacher. In contrast, process aspects of quality refer to the quality of the interactions that children experience in the classroom, e.g., the teacher’s promotion of critical thinking skills during learning activities, sensitivity to children’s needs, and the promotion of a positive classroom culture to enable development and early learning. In general, process quality elements, and the “quantity and quality of interactions between a young child and his or her primary caregivers, including ECE providers, are the most important factors in early development” (Burchinal, 2018, p. 3).

Teach ECE captures information regarding the process quality of classrooms during two 15-minute segments of class time. Observers collect information on whether the teacher is engaging the class in a learning activity and the proportion of the class that is actively engaged three times during each segment to measure the Time on Task of the classroom. Observers then code 28 behaviors indicative of quality during the lesson which fall under nine elements. These nine elements fall under three primary areas of process quality—Classroom Culture, Guided Learning, and Socioemotional Skills. **Figure 3** shows the structure of the Teach ECE measures of process quality (Ding et al., 2023; World Bank, 2021). In addition to the process quality of classrooms, observers also collect information regarding the structural aspects of quality, including the teacher’s level of education, teacher-child ratio, attendance and composition of enrolled children, language-match in the classroom, availability of learning materials, and safety and hygiene issues. Consistent with the literature emphasizing the importance of process quality in ECE, the majority of this paper focuses on these process quality scores from the Teach ECE classroom observation tool.

_Figure 3. Teach ECE Framework of Areas and Elements_ <!-- Start of picture text --> Quality of Teaching Practices Time on Task<br>Socioemotional<br>Classroom Culture Guided Learning Skills Time on Learning<br>Supportive Facilitation of<br>Learning Autonomy<br>Environment Learning<br>Positive Behavioral<br>Feedback Perserverance<br>Expectations<br>Social &<br>Critical Thinking Collaborative Skills<br>Checks for<br>Understanding<br><!-- End of picture text --> 8
# **_Training and Field Work_**

Comprehensive knowledge and practice training and examples are provided to AIM-ECD and Teach ECE enumerators before any field data collection activity. The training is a quality assurance activity to ensure that enumerators correctly interpret each item correctly and score the assessments in a reliable and consistent manner to ensure the fidelity of scores.

Enumerators participated in a four-day training course on Teach ECE and a three-day training on AIM-ECD consistent with the tool recommendations (World Bank, 2021, 2022). On the first day, the Teach ECE trainer explained the tool in detail, its intended measurement purposes and permitted uses, its teaching behaviors and elements. On the second and third days, enumerators practiced Teach ECE coding using practice videos of local teachers previously recorded in each location to maximize local adaptation, which were 20 minutes long to match the time they would be coding in the field. On the last day, enumerators went to schools to practice coding live and receive feedback from trainers. A similar structure is followed in the case of AIM-ECD, but with enumerators practicing the direct assessment administration with a child. Before enumerators are allowed to conduct classroom observations or child assessments in the field, they must pass a certification exam. In the case of Teach ECE, the exam involves scoring three videos within 1 point of the master codes (i.e., codes developed by Teach ECE experts after observing the same video) for at least 8 out of 9 elements for each of the three videos (80% of the time). In the case of AIM-ECD, the certification involves a reliability quiz and observation from the master trainer. This certification increases the quality assurance in the use of the tool and reduces potential scoring bias; at the same time, the exam is an excellent opportunity to clarify any questions regarding the scoring process.

# **Methods**

# **Calculation of Scores**

Both the AIM-ECD and Teach ECE contain many items that must be consolidated into aggregate scores for hypothesis testing. Below we describe the procedures used to calculate these aggregate scores. Overall, both instruments appeared to function well.

# **_AIM-ECD DA Score Calculation_**

We followed standardized guidance from the tool creators in order to calculate subtask, domain, and overall AIM-ECD scores (World Bank, 2022). Table 3 describes the sub-tasks in each domain as well as the number of items in each sub-task. In general, AIM-ECD scores are calculated on a 0-100% scale roughly equivalent to the percentage of items children scored correctly. For each **subtask** , the score is calculated by scoring each question (usually as correct/incorrect) and then dividing the total score by the maximum possible score. For each **domain** , the average is taken of all subtask scores of the four domains. Finally, the AIM-ECD Overall score is calculated by taking the average of the four domain scores. To ensure consistency of scoring and comparability with other datasets, the English Letters subtask was analyzed separately and not included in the calculation of the AIM-ECD DA Overall Score.

9
_Table 3. AIM-ECD DA domains, subtasks, and items_

|**Domain**|**Subtask**|**Number of items**|
|---|---|---|
|**Early**<br>|Listening Comprehension Story|5|
|**Literacy**|Letter Identification|20|
||Initial Sound Discrimination|3|
||Name Writing|1|
|**Early**|Receptive Spatial Vocabulary|4|
|**Numeracy**|Producing a Set|6|
||Simple Addition|3|
||Number Identification|20|
||Shape Identification|5|
|**Executive**<br>|Head, Toes, Knees, Shoulders Task|15|
|**Function**|Forward Digit Span|4|
||Backwards Digit Span|4|
|**Social-**<br>**Emotional**|Perspective Taking/Empathy|3|
|**English**<br>**Letters**|English Letter Identification|20|

For the more formal hypothesis tests used later in this report, we rely on the AIM-ECD Overall and domain scores. The advantage of this approach is that we are able to analyze and summarize a spectrum of ECD that captures many different skills in a single score. To further add meaning to our results and to concretize interpretations about what children can and cannot do, we also analyze a set of intentionally selected individual exemplar AIM-ECD items as described in Table 4.

_Table 4. Exemplar AIM-ECD Direct Assessment items_

|**Domain**|**Subtask**|**Exemplar Description**|
|---|---|---|
|**Early**|Letter Identification|Identified Urdu letter Alifpresented out of sequence: ا<br>Identified Urdu letter small ye presented out of<br>sequence: ی|
|**Literacy**|ListeningComprehension|Identified color of hat from story|
||Name Writing|Wrote at least some letters when asked to write name|
||Sound Discrimination|Identified word startingwith /s/ sound|
||Nb Idtifiti|Identified numeral 10presented out of sequence|
||umer encaon|Identified numeral 1presented out of sequence|
|**Early**|Producin a Set|Counted 6 objects correctly|
|**Numeracy**|g|Counted 14 objects correctly|
||Sh Idntifitin|Identified a circle amongshapes|
||ape ecao|Identified a triangle amongshapes|

10
||Simple Addition|Added 2+1=3|
|---|---|---|
||Spatial Vocabulary|Correctly identified the word describing a picture of a<br>ball under a box|
||Backward Digit Span|Could repeat 2 digits in reverse order|
|**Executive**|Forward Digit Span|Could repeat 3 digits in same order|
|**Functioning**|Head Toes Knees<br>Shoulder Task|Correctly followed instructions of a confusing task|
|**Social**<br>**Emotional**|Empathy Perspective<br>Taking|Could identify emotion crying girl was feeling|
|**English**|English Letter|IdentifiedEnglish letter A|
|**Letters**|Identification|Identified English letter Z|

# **_Teach ECE Score Calculation_**

As with the AIM-ECD DA, we followed the standardized guidance for calculating Teach ECE **behavior** , **element** , and **area** scores (World Bank, 2021). Behaviors are the individual items that comprise the Teach ECE assessment. When observing a class, the observer scores each of the 28 **behaviors** during two segments of an observation. Behaviors are coded in an ordinal scale as “Low,” “Medium,” and “High” (1, 3, or 5). To facilitate reporting and interpretation, more emphasis is placed on the coding of the ten **element** scores. Based on the behavior scores within each element, the observer scores each element on a scale of 1-5, with 1 indicating low quality and 5 indicating high quality. It is important to note that, unlike AIM-ECD DA, element scores are _not_ simply the average of behavior scores. While element scores are based on behavior scores, observers have flexibility of assigning slightly higher or lower element scores based on their judgment. Finally, we calculate the **area** scores and an overall Teach ECE score for each segment. The area score is calculated by taking the average of the elements that comprise that area and the Teach ECE Overall score is the average of the Classroom Culture, Guided Learning, and Socioemotional Support scores. Because of the structure of the assessment, there are essentially two sets of scores for each classroom observed—one from the first segment and one from the second segment. Segments are therefore the basic unit of analysis when analyzing Teach ECE scores. As such, when reporting percentages in relation to benchmarks, the actual interpretation is “percentage of segments” rather than “percentage of classes.” This paper primarily focuses on Area and Element scores as well as the overall Teach ECE score. In addition, it presents the results of selected exemplar items (i.e., scored behaviors) as shown in **Table 5** .

_Table 5. Exemplar behaviors from Teach ECE_

|**Area**|**Element**|**Exemplar Behavior**|
|---|---|---|
|**Time on Task**|Time on Task - Student|0.2 The teacher leads a learning activity and<br>most children are on task|
|**Classroom**<br>**Culture**|Supportive Learning<br>Environment|1.1 The teacher treats all children respectfully|

11
||Positive Behavioral<br>Expectations|2.1 The teacher sets clear behavioral<br>expectations for classroom activities and/or<br>routines|
|---|---|---|
||Facilitation of Learning|3.3 The teacher makes connections during the<br>day that relate to other concepts or children’s<br>daily lives|
|**Guided**<br>**Learning**|Checks for Understanding|4.3 The teacher adjusts teaching to the level of<br>the children|
||Feedback|5.1 The teacher provides specific comments or<br>prompts that help clarify children’s<br>misunderstandings|
||Critical Thinking|6.3 The children ask open-ended questions or<br>perform thinking tasks|
||Autonomy|7.3 Children volunteer to participate in the<br>classroom|
|**Socioemotional**<br>**Skills**|Perseverance|8.2 The teacher responds positively to children's<br>challenges|
||Social & Collaborative<br>Skills|9.2 Children collaborate with one another<br>through peer interaction|

# **_Reliability of Tools_**

The Cronbach’s alpha of the domains and overall scores for the AIM-ECD and Teach ECE are shown below in **Table 6** . Overall, the AIM-ECD demonstrates superb reliability with an overall alpha of 0.978 and excellent reliability for each domain. Overall Teach ECE segment scores were very good with a Cronbach’s alpha of 0.884, but individual domains were lower—while Guided learning demonstrated very good reliability (0.820), Socio-Emotional Skills was acceptable (0.771) and Classroom Culture was questionable/fair (0.651). These figures are similar, and slightly better, than reliabilities from a review of international Teach ECE data (Ding et al., 2023, 2024).

_Table 6. Cronbach’s Alpha of AIM-ECD Domain and Teach ECE Areas_

|**Tool**|**Domain/Area**|**Cronbach’s Alpha**|
|---|---|---|
|**AIM-ECD DA**|Early Literacy|0.938|
||Early Numeracy|0.964|
||Executive Functioning|0.943|
||Social Emotional|0.951|

12
||**Overall**|0.978|
|---|---|---|
|**Teach ECE**|Classroom Culture|0.651|
||Guided Learning|0.820|
||Socio-Emotional Skills|0.771|
||**Overall**|0.889|

# **Analysis**

Given the sampling design, results do not require weighting to be representative of the PCHIP project area. In addition, on reporting the results of composite items calculated as described above, we also present results based on the proportion of **segments** demonstrating a certain degree of quality and the proportion of **children** with different skills. Below, we outline the process used to set benchmarks needed to dichotomize these results on AIM-ECD and Teach ECE. In addition to the simple descriptive statistics and disaggregation, we also build a statistical model for inferential purposes. The details of this model-building process are outlined below.

# **_Quality Benchmarks, Exemplar Items, and Standardized Scores_**

As described above, both the AIM-ECD DA and Teach ECE generate continuous scores to describe child developmental outcomes and quality of teaching practices, respectively. AIMECD scores range from 0-100% while Teach ECE element and area scores range from 1-5.

These continuous scores are limited in their comprehensibility. To improve the communicability of our results, we also utilize benchmarks on Teach ECE to report on how much of the time (i.e., what proportion of segments) classrooms are demonstrating minimal quality during lessons. When reporting on the proportion of segments demonstrating minimum quality, we use the proportion of segments that received at least a “Medium” rating, i.e., an average rating of at least 3. Three represents the score achieved by a class with no extremely serious quality issues and thus is a reasonably interpretable benchmark. Teach ECE quality benchmarks are useful for creating concrete descriptives about the status of ECD in South Punjab but should not be interpreted as the result of best practice standards recommended for external use.<sup>5</sup> We do not set similar benchmarks for AIM-ECD that allow us to talk about the proportion of children demonstrating minimal developmental levels. Instead, we report the results from a set of exemplar items on the assessment to describe the proportion of children able to accomplish certain skills.

In addition to using quality benchmarks for Teach ECE and exemplar items AIM-ECD, we also transform AIM-ECD and Teach ECE continuous scores to improve interpretability of our multiple regression analysis. We transform both scores to a standardized variable with mean 0 and standard deviation 1 (i.e., Z-scores). This allows us to interpret coefficients from our models as differences in outcome standard deviations (for AIM-ECD scores) and in teaching quality (for

> 5 Standard setting is a complex normative process that requires buy-in from relevant stakeholders and experts, careful consideration of what children “should” be able to do at a given age or stage of education, and an examination of the consequences of these decisions (American Educational Research Association et al., 2014; Cizek & Bunch, 2007). A formal standard setting exercise to provide additional meaning to AIM-ECD DA and Teach ECE scores is currently under consideration and will be part of the public set of resources available for users of these tools.

13
Teach ECE scores). Transforming scores in this way deviates from standardized AIM-ECD and Teach ECE guidance, but doing so does not alter any statistical approaches (World Bank, 2021, 2022).

# **_Multiple Regression Analysis_**

In addition to the simple descriptive statistics and disaggregation by covariates that help answer research questions 1-3, we build a model to investigate the joint relationships between classroom quality and child outcomes. To accomplish this, we use the overall and domain-specific AIM-ECD scores discussed in **Calculation of scores** (above) to analyze the associations between AIM-ECD by various child and classroom factors. As described above in _Quality Benchmarks, Exemplar Items, and Standardized Scores_ , we standardize both the AIM-ECD and Teach ECE scores prior to fitting these models. Data management and statistical analysis was conducted using R 4.2.1 with the lme4 and tidyverse packages and in Stata 17.0 (Bates et al., 2015; R Core Team, 2021; StataCorp, 2019; Wickham et al., 2019).

Due to the sampling and data collection design, children are nested within classrooms as well as enumerators. This leads to a complex error structure that should not be ignored in analysis. Children studying in the same classroom may have non-independent errors given within-site homogeneity and exposure to the same teacher’s lessons. Similarly, enumerators may be systematically more or less lenient, leading to correlations within enumerators.

To account for this data structure, we use a mixed effects (random effects) model with children’s AIM-ECD scores nested in classrooms and also add a random effect to account for differences in enumerators with the covariance of enumerator and class intercepts constrained to zero (i.e., assuming that enumerator leniency across classrooms was not correlated with overall classroom averages). For each AIM-ECD domain or overall score, we fit the following model to predict the overall or domain standardized Z-score for child in classroom assessed by enumerator . Random effects at the enumerator and classroom allow us to examine associations both within and between classrooms and to partially control for differences by ii jj kk jj enumerator leniency. kk vv uu A A i 0C C 1E E E ii CC,jj CCE EE,kk<sup>+ ββ2A</sup> 0C C 00 01TAADD = ββT + ββ02 03<sup>GGMMGGMMee1eeii+ ββ3SSeexx</sup> 04<sup>iiMMuuM+ εεiijjkk</sup> ββ CC,jj = ββ + ββ05 AAT TTjj + ββ06 MMuuM iiAAM jj<sup>+ ββ</sup> 07 jj<sup>+ ββ</sup> 08<sup>iiAAM</sup> × GGMMGGMMee1 jj jj jj jj jj + ββ TTeeGGTThTTMM + ββ 1ETTeeGGTThTTxxEEE E + ββ TTeeGGTThSSeexx + ββ TTMMGGC ssGGssAA + uu where CCE EE,kk<sup>= ββ20 + vvkk</sup> ββ jj uu ~NN(0, θθ) �= 0 kk vv ~NN(0, Ψ) jj kk Given the above model, we are primarily interested in the coefficients given by TTCCvv�uu , vv 01—i.e., the estimate of the association of a given aspect of quality and child development conditional on the age and sex of a child and structural aspects of the classroom. ββ 14
# **Results**

# **Q1. What Do the Study Pre-Primary Classrooms in South Punjab Look Like?**

We begin our results with purely descriptive analysis of the composition of classrooms focusing on the characteristics of their teachers and the children enrolled in them. As shown in **Figure 4** , _Katchi_ classes are designed for children aged 4-5 years old and Grade 1 is designed for children about 6 years old and above. While these distinctions exist in theory, in practice age requirements are not followed. In addition, a new type of class, ECE class, has a two-year curriculum according to the Punjab ECCE policy. As described above in **_Sample_** , we do not include ECE classes in this analysis.

_Figure 4. Intended Age Range and Curriculum by Class Type_ <!-- Start of picture text --> Age<br>4 5 6+<br>Katchi No curriculum<br>ECE 2 year curriculum<br>Grade 1 Grade 1 curriculum<br><!-- End of picture text -->

**Table 7** below describes the composition of classes in the sample. Determining the grade/level of a classroom is not always straightforward given that many classes operate in a multi-graded fashion–i.e., multiple levels are taught in the same room by the same teacher. For analytical purposes, we classify grades according to the grade/level that was being taught at the time of observation and additionally record whether a classroom was multi-graded or not. Nearly four in ten classrooms were taught in a multi-graded fashion. In research question 4, we will address multigraded classes, but for the subsequent sections, we focus only on disaggregating by _Katchi_ and Grade 1, regardless of whether or not they were multi-graded.

_Table 7. Number of Classrooms by Grade and Multi-graded Status_

|**Grade**|**Single graded**|**Multi-graded**|**Total**|
|---|---|---|---|
|_Katchi_|394|295|689|
|Grade 1|451|255|706|
|**Total**|845|550|1,395|

# **_Enrollment and Attendance_**

Enrollment size varies substantially, with as few as 5 and as many as 136 children enrolled in a single class. The average classroom has about 29 children enrolled. As shown in **Figure 5,** _Katchi_ classrooms were much larger than Grade 1 classes, with the average _Katchi_ class enrolling 15
<!-- Start of picture text --> Mean Median<br>33.4<br>30<br>30-<br>24.44<br>®<br>2 21<br>5 20-<br>2<br>2<br><x<br>S<br>c<br>o<br>£<br>2<br>¢<br>Wig.<br>0 -<br>Katchi Grade 1 Katchi Grade 1<br>Type of classroom<br><!-- End of picture text --> <!-- Start of picture text --> 15% -<br>Qa<br>@<br>QQ<br>3 10%-<br>xe}<br><<br>@Q<br>2<br>@o<br>a<br>5% -<br>0% -<br>0% 25% 50% 75% 100%<br>Percent of children in attendance on date of observation<br>| At least 75% present , | Below 75% present<br><!-- End of picture text -->
<!-- Start of picture text --> Attendance Enrollment<br>6.83<br>6-<br>5.52<br>c<br>2 4.74<br>x3<br><<br>is}<br>‘Oo 4- 3.85 3.83<br>by 3.62<br>we}<br>E 3.15<br>c 2.98 2.98<br>o 2.69<br>DoO 2.49<br>io}<br>g 2.18<br><to- 1.85<br>1.51<br>1.38<br>1.12<br>0.37 909 0.46 0.38<br>O- 0.05——_0.03 ‘H 0.07=e0.05 |<br>1 ' ' 1 ' 1 1 ' 1 1 ' 1<br>Less than3 3 4 5 6 Greater than 6 Less than3 3 4 5 6 Greater than 6<br>Age<br><!-- End of picture text -->

Sex 7 Boys 7 Girls
<!-- Start of picture text --> im)<br>o<br>& Katchi-<br>iS)<br>£<br>0)<br>D<br>©<br><=<br>iS)<br>©<br>©<br>ow<br>2<br>c<br>a)<br>ae)<br>2<br>®<br>-lo)<br><<br>8<br>a<br>oO ne<br><!-- End of picture text --> Age of students 3 BH 4 | 5 i 6 i Greater than 6
<!-- Start of picture text --> 70%<br>66%<br>60% -<br>2<br>a<br>E<br>£<br><=4 40% -<br>S<br>hat£<br>€2 23%<br>g5 20%- Ta<br>a 16%<br>7%<br>0% -<br>None' PTC or' CT B.Ed.M.Ed' or<br>Teacher Certification Level<br>Teacher Gender Hi Female a Male<br><!-- End of picture text --> <!-- Start of picture text --> 42%<br>0.4-<br>37'<br>oO a<br>203<br>E<br>a<br>c<br>0<br>o<br>5 21% 22%<br>2@ 0.2 02- 20<br>:<br>5<br>a2£ a 14%<br>2 3%<br>0.1-<br>4YA%<br>3%3%<br>0.0- | =1%1%<br>reportedNone Lessone year than 1-2 years 3-5 years 6-10 years years10-19 20+ years<br>Teacher experience teaching pre-primary<br>Teacher Gender a Female H Male<br><!-- End of picture text -->
<!-- Start of picture text --> no_ ee<br>no<br>&<br>oO<br>-_fo}<br>oO<br>S<br>i<br>- ee<br>Percent of Teachers by Certification Level<br>Certification [J None §) etcorct I) Bea. ormed<br><!-- End of picture text -->
<!-- Start of picture text --> Katchi Grade1<br>100% - 96% 96%<br>92% 93%<br>90% 91%<br>81% 81%<br>£ 75%-<br>=&<br>‘©<br>><br>Cy<br>at<br>=]<br>E<br>6a 50%-<br>7)<br>& 41%<br>e 38%<br>2<br>c<br>°<br>Q<br>2<br>Oo 25%-<br>0%-<br>' 1 1 1 J ' ' ' ' '<br>Clean . Child-friendlyhand Separate Clean . Child-friendlyhand Separate<br>drinking vere washing toilets mean drinking Handwashing washing toilets ne<br>water & toilet for girls water y & toilet for girls<br>facilities facilities<br>WASH Resource<br><!-- End of picture text -->
<!-- Start of picture text --> Katchi Grade 1<br>100% -<br>=<br>Fo}<br>&<br>‘oO<br>> 75% -<br>nan<br>=<br>59%<br>><br>o<br>ae]<br>= 50% - 47%<br>2<br>179)<br>o<br>oDno 33%<br>cS) 27% 27%<br>c ., an 26% a 24% 25%<br>O 25%- 5 21%<br>= 18% 18% 19%<br>Qa<br>Q 9%<br>o 6% ae 7%<br>1% 3% 2% 1% 1% 1%<br>0% -<br>t 1 1 1 1 1 t ' 1 1 1 1<br>an n<br>2 s £ 2 s s<br>7) 2 oe o ® 2 @ n ow a 9 2<br>5 = az So Q $ 5 2 az 53 By ro<br>5 = >h eE s 9 5 & >h SE s 9<br>5D 2 aoot oOos o 2> D> F} aoOo = wos 29 a><br>5 2 = 38 3 g 5 = §S <3 @& 8<br>= < f us a = < 2 us 7)<br>* Ea = i ry<br>g g<br>Type of resource<br>IY) % of Classes with availability [JJ Estimated % of Children with availability<br><!-- End of picture text -->
<!-- Start of picture text --> 90%<br>75%- 13%<br>ay<br>Cc<br>3)<br>=<br>oO2 50%‘0 a4<br>o 42%<br>: 40%<br>2<br>5 33% 34%<br>6<br>o 26%<br>25% -<br>12%<br>a -<br>x & se SO xe N Ry & RR<br>PLE FGF KO _ gs? Sa SS Ss SF<br>°° SES CK KS OP & ORS Se s Fv<br>we Vs CE <P Ce <° “ Y o oe<br>& om s g<br>Teach ECE Element<br>Teach ECE Area i Classroom Culture | Guided Learning Bi Socioemotional Skills | Time on Task<br><!-- End of picture text -->
<!-- Start of picture text --> ron ore ne) LeeVo<br>The teacher treats all children<br>respectfully ae_ wv<br>The teacher sets clear behavioral<br>expectations for classroom activities - 52%<br>and/or routines<br>— The teacher makes connections during<br>Ee the day that relate to other concepts or - 42%<br>£ children's daily lives<br>a4ie} The teacher adjusts; teaching; to the _<br>a<br>uy The teacher provides specific comments<br>Oo or prompts that help clarify children’s - 47%<br>Ww misunderstandings<br>$<br>© The children ask open-ended questions or<br>Fe perform thinking tasks _ 10%<br>Children volunteer to participateclassroomin the _ 58%<br>The teacher responds positively to _ 58%<br>children’s challenges<br>Children collaboratethroughwithpeeroneinteractionanother_ 13%<br>1 1 1 1<br>0% 25% 50% 75%<br>Proportion of Segments<br>Teach ECE Area LJ Classroom Culture Oo Guided Learning || Socioemotional Skills a Time on Task<br><!-- End of picture text -->
<!-- Start of picture text --> 9<br>Time on Task 4 Po yy %<br>Supportive Learning _ 90%<br>Environment 91%<br>PositiveExpectationsBehavioral _ 42%43%<br>FacilitationLearningof _ 38%42%<br>UnderstandingChecks for _ 50%50%<br>Feedbackeeadbac! - a 33%34%<br>Criticritic a ll Thinking - 12%<br>Thinking — 12%<br>anyutonomy =Ha32%35%<br>Perserverance - 26%26%0,<br>CollaborativeSocialSkills& _ 7%7%<br>0% 25% 50% 75% 100%<br>Proportion of Segments<br>A i Classroom Culture i Socioemotional Skills _ Katchi<br>rea Hi Guided Learning i Time on Task Class Type i Grade 1<br><!-- End of picture text -->
<!-- Start of picture text --> 716%<br>Time on Task- 69%<br>65%<br>Supportive Learning _ ss aed<br>Environment Hi 83% °<br>Positive Behavioral<br>Expectations _ L—} 33% Ape°<br>Facilitation of °<br>Learning _ 27% 34% 45%<br>Checks for<br>Understanding _ 41%44% 54%<br>Feedback - by 35%<br>33%<br>13%<br>Critical Thinking - 11%<br>9%<br>Autonomy - 34%35%<br>30%<br>Perserverance - 22% 29%<br>19%<br>Social &<br>Collaborative _ ae<br>Skills %<br>0% 25% 50% 75% 100%<br>Proportion of Segments<br>Area a Classroom Culture [| Socioemotional Skills Bi None<br>i Guided, Learning; | Time; on Task Teacher Certificationoo. J PTC or CT<br>9 c0. or med<br><!-- End of picture text -->
<!-- Start of picture text --> Time on Task - 71%<br>Supportive Learning, SH 89%<br>Environment . 92%<br>PositiveExpectationsBehavioral _ 41%43%<br>FacilitationLearningof_ 36% 44%<br>UnderstandingChecks for _ 45% 55%<br>Feedbackcea - TT30% am<br>Critical Thinking - 10%<br>° ett 14%<br>Autonomy 33%<br>’ - Se 35%<br>Perserverance - as<br>| 32%<br>Social & _ 5%<br>Collaborative Skills 9%<br>0% 25% 50% 75% 100%<br>Proportion of Segments<br>A i Classroom Culture yy Socioemotional Skills i Female<br>rea Hi Guided Learning | Time on Task Teacher Gender i Male<br><!-- End of picture text -->
<!-- Start of picture text --> 71%<br>Time on Task - 82%<br>81%<br>Supportive Learning<br>Environment _ aan— atl20%, 9,°<br>Positive Behavioral 39%<br>Expectations ~ 46% 1 58%<br>cig gs 41%<br>g 26%<br>Checks for<br>Understanding _ 44956%51%<br>eedback - = H129%<br>13%<br>Critical Thinking - 10%<br>10%<br>Autonomy - 34%34%<br>30%<br>Z aa: 29%<br>erserverance - ta,<br>Social &_ : 8%<br>Collaborative Skills 4%<br>0% 25% 50% 75% 100%<br>Proportion of Segments<br>Area _ Classroom Culture i Socioemotional Skills [| Urdu<br> | Guided Learning i Time on Task Language Used C] Saraiki<br>|| Other<br><!-- End of picture text -->
Among the items in Early Literacy, the steepest age gradients are found in identifying Urdu letter _choti ye_ (66% of six-year-olds vs. 45% of four-year-olds) and writing letters of their name (60% of six-year-olds vs. 33% of four-year-olds). In the Early Numeracy domain, the steepest differences were found in identifying the numeral 10 (73% of six-year-olds vs. 47% of four-yearolds), and counting 6 and 14 objects, which had a difference of over 20 percentage points. Identifying English Letters also exhibited a large age difference with 58% of six-year-olds identifying letter Z compared to just 35% of four-year-olds. In contrast, other items show smaller differences between ages. On a listening comprehension item recalling details about the story, sixyear-olds were only 10 percentage points more likely to respond correctly than four-year-olds (61% compared to 51%). Similarly, the difference between scores by age on shape identification by age was small—with just 20% of six-year-olds and 14% of four-year-olds able to identify a triangle. While all items exhibited an age gradient, it was much less pronounced in the Social Emotional and Executive Functioning domains.

Focusing exclusively on the six-year-old children, who are either in Grade 1 or about to be, reveals significant gaps in the knowledge that these children possess. In Early Literacy, 6% could not identify the first letter of the Urdu alphabet, 34% could not identify the last letter of the Urdu alphabet, and 40% of six-year-olds could not write any letters of their name. In Early Numeracy, nearly one in four children could not identify the numeral 1 presented out of order, half of children could not count 14 objects correctly, 41% could not identify a circle, and 80% could not identity a triangle. In the Executive Functioning and Social Emotional domain, 11% of children could not remember and repeat a three-digit-sequence of numbers and just 17% of children could name an emotion that a picture of a girl crying was feeling.

The skills tested on the AIM-ECD Direct Assessment serve as the foundational skills upon which later learning is built. As such, these findings strongly suggest room for improvement in the outcomes of preschool-aged children in at least some districts in South Punjab.

29
<!-- Start of picture text --> iertton Ur ter Ait Ts.86% 94%<br>ye 45%<br>Identified color of hat from<br>story _ 51% Jo<br>shes<br>Wrote at least some letters<br>_ 46%<br>when asked to write name 33%<br>Identified word starting _ 53% ele<br>with /s/ sound 47%<br>Identified numeral 1 presented _ 69% we<br>out of sequence 60% °<br>Lik)<br>Identified numeral 10<br>_ 61%<br>presented out of sequence 47%<br>77%<br>Counted 6 objects correctly - 66%<br>54%<br>Counted 14 objects corecty- Tass;29% 52%<br>Identified a circle among _ 3% ae<br>shapes 51%<br>Identified a triangle among _ a<br>shapes 14%<br>75%<br>Added 2+1=3 - 67%<br>57%<br>Correctly understood spatial _ 69 oa<br>vocabulary for under 62%<br>77%<br>Identified English letter A - 69)<br>60%<br>58%<br>denied Engish cter2- ST35%<br>Could repeat 2 digits in _ 33% eas<br>revserse order 28%<br>Could repeat 3 digits in same _ 85%89%<br>order 79%<br>Correctly followed 65%<br>task 52%<br>Could . . . ‘ 17%<br>identify emotion crying _ 2%<br>girl was feeling 8%<br>0% 25% 50% 75% 100%<br>Percentage of children correctly responding<br>Age i 4 7 5 i 6 i Early Literacy i Executive Functioning<br>Domain i English Letters | Social Emotional<br>i Early Numeracy<br><!-- End of picture text -->
<!-- Start of picture text --> . . 94%<br>ye 57%<br>Identified color of hat storyfrom _ 56%61%<br>Wrotewhen ataskedleasttosomewrite lettersname _ 48% 55%<br>Identified word starting _ 58%<br>with /s/ sound 54%<br>Identified numeral 1 presented _ 73%<br>out of sequence 70%<br>presentedIdentifiedout ofnumeral sequence10 _ 64%66%<br>. 70%<br>Counted 14 object: tly - 43%<br>Identified a circle among _ 58%<br>shapes 5A%<br>Identified a triangle among _ 19%<br>shapes 17%<br>70%<br>Correctly understood spatial _ 73%<br>vocabulary for under 70%<br>. . 74%<br>. . 53%<br>Could repeat 2 digits in _ 36%<br>revserse order 35%<br>Could repeat 3 digits in same _ 87%<br>order 85%<br>Correctly followed 61%<br>task 60%<br>Could identify emotion crying _ 14%<br>girl was feeling 14%<br>0% 25% 50% 75% 100%<br>Percentage of children correctly responding<br>BH Early Literacy | Executive Functioning Sex I Boys Hi Girls<br>Domain BH English Letters | Social Emotional<br>Hi Early Numeracy<br><!-- End of picture text -->
The model building process used to arrive at our final model is described below in **Table 8** . We use Model 6 as our primary analytical model, which includes variables controlling for child’s age and sex, Grade 1 classroom, multi-graded classroom, the interaction between multi-graded and Grade 1 classroom, teacher’s sex, education, and experience, and classroom language. In addition to interactions between class type and multi-graded status, we tested a number of additional models including more complex interactions but found little of substance. As such, we do not report these models.

_Table 8. Model Building Process for AIM-ECD Overall Z-scores by Teach ECE Overall Z-scores_

||(1)|(2)|(3)|(4)|(5)|(6)|
|---|---|---|---|---|---|---|
|Teach ECE Overall Z-score|0.0887<sup>***</sup><br>(0.0162)|0.0892<sup>***</sup><br>(0.0155)|0.0860<sup>***</sup><br>(0.0142)|0.0777<sup>***</sup><br>(0.0138)|0.0759<sup>***</sup><br>(0.0140)|0.0755<sup>***</sup><br>(0.0140)|
|Child’s age (years)||0.232<sup>***</sup>|0.203<sup>***</sup>|0.198<sup>***</sup>|0.198<sup>***</sup>|0.197<sup>***</sup>|
|||(0.0131)|(0.0130)|(0.0129)|(0.0129)|(0.0129)|
|Child is female||0.0295|0.0205|0.0178|0.00631|0.00511|
|||(0.0203)|(0.0198)|(0.0197)|(0.0202)|(0.0202)|
|Multi-graded class|||-0.168<sup>***</sup>|0.0935<sup>*</sup>|0.0976<sup>**</sup>|0.0932<sup>*</sup>|
|<br>(vs. Single-graded)|||(0.0284)|(0.0376)|(0.0376)|(0.0376)|
|Grade 1 class|||0.566<sup>***</sup>|0.816<sup>***</sup>|0.820<sup>***</sup>|0.814<sup>***</sup>|
|(vs._Katchi_)|||(0.0276)|(0.0364)|(0.0374)|(0.0375)|
|Multi-graded x Grade 1 Class||||-0.554<sup>***</sup><br>(0.0541)|-0.553<sup>***</sup><br>(0.0543)|-0.546<sup>***</sup><br>(0.0543)|
|Teacher has teaching|||||-0.0238|-0.0257|
|certificate or ed. degree|||||(0.0428)|(0.0428)|
|Teacher has 3+ years pre-|||||-0.0126|-0.0140|
|primary experience|||||(0.0331)|(0.0330)|
|Teacher is female|||||0.0692<sup>*</sup>|0.0594<sup>*</sup>|
||||||(0.0281)|(0.0291)|
|Saraiki language class||||||-0.0271|
|<br>(vs. Urdu language class)||||||(0.0479)|
|Other language class||||||-0.156<sup>*</sup>|
|<br>(vs. Urdu language class)||||||(0.0691)|
|Constant|0.0663|-1.189<sup>***</sup>|-1.239<sup>***</sup>|-1.330<sup>***</sup>|-1.337<sup>***</sup>|-1.314<sup>***</sup>|
||(0.0605)|(0.0929)|(0.0922)|(0.0925)|(0.102)|(0.103)|
|N|<br>8249|<br>8249|<br>8249|<br>8249|<br>8249|<br>8249|
|Var u_k (Enum)|0.111|0.112|0.110|0.112|0.113|0.114|
|Var uj (Class)|0.308|0.274|0.200|0.181|0.180|0.179|
|_<br>Var e_ijk (Residual)|0.589|0.576|0.571|0.571|0.571|0.571|

Standard errors in parentheses ~ _p_ < 0.1, * _p_ < 0.05, ** _p_ < 0.01, *** _p_ < 0.001 32
Including the factors discussed in Q3 jointly makes several inferences abundantly clear. As expected, older children score higher on the AIM-ECD than younger children. An additional year of age is associated with a 0.197 standard deviation ( ) increase in AIM-ECD Z- score. However, we do not see any significant differences in overall development by child sex. In ββ= 0.197, EE< .001 addition to age, the grade level of the class explains a huge amount of variation in AIM-ECD scores. Children in Grade 1 classes performed 0.814 standard deviations ( ) higher than those in _Katchi_ classes. However, this difference is greatly attenuated in multi-graded classes. While multi-graded _Katchi_ classes have slightly higher ( ββ= 0.814, EE< .001 ) AIM-ECD Z- scores than children in single-graded _Katchi_ classes, children in multi-graded Grade 1 classes score -.546 SD worse than those in single-graded Grade 1 classes ( ββ= 0.093, EE< .05 ) . Classes taught in Urdu and Saraiki appear to have similar levels of achievement on average, but those taught in another language had -0.156 SD lower scores on average than Urdu classes ββ= −0.546, EE< .001 ( ) . Our model finds strong evidence about the importance of ECE teaching quality as shown ββ= −0.156, EE< .05 in **Table 8** and Figure 21. A one-standard deviation difference in Overall Teaching Quality is associated with a 0.076 standard deviation difference in AIM-ECD Z- scores ( ) , even after controlling for other factors (Model 6). After controlling for classroom quality, other ββ= 0.076, EE< .001 teacher-level factors that might be hypothesized to be related to outcomes are not significantly associated with AIM-ECD. In our final model, teachers with three or more years of experience and those with higher levels of certification did not have classes with higher levels of AIM-ECD scores. One factor appears to remain salient—teacher gender. Classrooms led by female teachers have 0.060 SD higher ( ) AIM-ECD Z-scores on average, even after controlling for other factors of quality. ββ= 0.060 EE< 0.05 Using association between age and AIM-ECD scores allows us to convert differences in Z-scores to differences by age. Given that a year of age is associated with a 0.197 SD difference in AIM-ECD scores, and that a 1 SD difference in Teach ECE scores is associated with a .076 SD difference in AIM-ECD scores, we can conclude that a child in a high-quality classroom (+2SD) is predicted to have an AIM-ECD score that is similar to a child who is 0.77 years (~9 months) older in an average quality classroom.

33
<!-- Start of picture text --> 4<br>©<br>fe)<br>re)<br>°<br>N 2<br>ic<br>)<br>><br>e)<br>Q0<br>O<br>Ud<br>=<br>xt<br>To} -.2<br>L<br>2<br>xo]<br>©<br>oO<br>-.4<br>4 5 6<br><!-- End of picture text --> Child's age <!-- Start of picture text --> Low Quality<br>Average Quality<br>High quality<br><!-- End of picture text -->
<!-- Start of picture text --> Early Literacy Early Numeracy<br>3 3<br>g2g2<br>BB<br>Ny 1 NC<br>no}®®no}<br>&<br>ne} 0 ne}& 0 ee<br>22<br>a-.1 Qa -1<br>-.2 -.2<br>a rs<br>Low Quality Average Quality High quality Low Quality Average Quality High quality<br>Executive Functioning Social-Emotional<br>3 3 : :<br>o2©2<br>fo)°fe)<br>N“i - 1 N“ics) - 1<br>®®<br>xo}&no} |<br>xo} 0 ao]& 0<br>22<br>a-.1 a -1<br>-.2 : : -.2 : :<br>rs a rs<br>Low Quality Average Quality High quality Low Quality Average Quality High quality<br><!-- End of picture text -->
<!-- Start of picture text --> Time on Task Classroom Culture<br>High quality oe High quality rm—<br>Average Quality ae Average Quality -———<br>Low Quality — Low Quality -——<br>rr rs a a a rr es a a a<br>-.2 -1 0 1 2 3 -.2 -1 0 1 2 3<br>Guided Learning Socioemotional Skills<br>Average Quality —— Average Quality pa<br>Low Quality —— : Low Quality — :<br>a a a a a rs a as as<br>-.2 -1 0 1 2 3 -.2 -1 0 1 2 3<br><!-- End of picture text -->
the quality of pre-primary schooling and children’s development outcomes in the early years. A brief recap of what the data indicate:

**Pre-primary classrooms are often large and composed of children of diverse ages.** Many children are studying in classrooms with age-levels that are not consistent with the assigned age range of the class. Large age ranges in a classroom can make it difficult for a single teacher to effectively engage the entire class in activities.

**Children’s absenteeism is a challenge.** About a quarter of classes had more than one in four children absent on the day of an announced visit. Ensuring children and teachers are present in schools is, of course, a prerequisite for learning. As such, better understanding the causes for absenteeism of pre-primary children is an important area for future research.

**Many children lack critical skills.** Many children lack the pre-academic and schoolreadiness skills necessary to succeed later on in life as evidenced by their performance on basic exemplar items such as number identification and two digit addition. We find the results for sixyear-old children particularly illustrative in this regard, with a third of 6-year-olds unable to identify the last letter of the Urdu alphabet, 41% unable to identify a circle on a piece of paper, and 23% unable to identify the numeral 1 out of sequence.

**Girls are not attending in equal numbers.** Even at the earliest ages and in the youngest classes, there are more boys than girls. The gap is larger for older children and in Grade 1. It appears that the enrollment gap in primary school starts extremely early, and there is no age at which there are equal numbers of boys and girls. Policy makers interested in girls’ education cannot wait to address the issue of non-enrollment of girls when children are older—it is something that must be addressed when children are as young as four years old.

**Girls in school can thrive.** Despite being underrepresented in pre-primary classes, they appear to be doing at least as well as boys in many early learning and development skills. Indeed, in the Early Literacy domain of AIM-ECD, we find that girls significantly outperform boys.

**An examination of structural quality reveals both areas of strength and of weakness.** Structural quality features of classrooms reveal many positive findings, such as a relative absence of safety hazards in classrooms and adequate water and sanitation facilities in most schools. However, we also find substantial material deficiencies. It is unusual for pre-primary children to be in classrooms with adequate levels of learning materials.  Learning resources are rare in classrooms—with just 25% of _Katchi_ classrooms having storybooks, and 27% with any sort of educational toys or blocks. Even when classrooms have such materials in the classroom, many times children are not able to access or use them.

**Multi-graded classrooms are a challenge for older children.** When teachers have multiple grades in the same classroom, it presents a challenge to target instruction to the right level, manage the classroom effectively, and ensure that all children are learning. According to our results, these factors appear to be especially challenging for Grade 1 classrooms, where multigrading was strongly associated with negative outcomes. We do not find the same challenge in _Katchi_ classrooms.

37
**Process quality indicators show substantial room for improvements in how teachers engage young students.** The findings from Teach ECE provide ample evidence of areas in both structural and process quality that would benefit from further investment. In more than one-fourth of segments observed, the teacher was not engaging children in a learning activity. Classroom Culture was largely positive, but in just one third of segments were teachers observed providing children with constructive feedback, in just 12% were children engaged in any kind of critical thinking activities, and in just one third of segments did children have any sort of autonomy over their participation in class activities.

**Teaching quality matters – more than teacher education and experience** . Despite low levels of teaching quality in many classrooms, we find strong evidence of its association with outcomes. In earlier analysis, we disaggregated levels of children’s learning and development by classroom and teacher factors, we found that children’s skills varied by teacher level of certification, with children in classes taught by teachers with higher education exhibiting stronger outcomes. We also found that classes with more highly educated teachers exhibited higher levels of quality. However, once we controlled for multiple factors simultaneously, we find consistent and positive associations between teaching quality (i.e., practices employed by teachers in the classroom) and outcomes, but that other teacher- and classroom-level factors matter less. Our results cannot be taken as causal evidence, as we discuss more in _Limitations,_ but these results suggest that that differences in outcomes by teacher certification may be due to the higher quality of classrooms taught by more certified teachers.

Given the low levels of quality observed in many classrooms, these results are encouraging as a potential path forward for further investment in both the material resources preprimary classes have access to and use as well as encouragement for teachers to engage in more evidence-based classroom practices.

**Female teachers perform** . As mentioned above, most teacher and classroom factors were not associated with outcomes after controlling for quality. One exception was teacher gender. We find a small but consistent advantage in AIM-ECD performance from classrooms led by a female teacher that are not explained by differences in quality or background.

# **Limitations**

This work is, to the best of our knowledge, the first to systematically examine the early learning and developmental status of children enrolled in ECE settings and the quality of such settings in selected districts of South Punjab, Pakistan. Its strengths lie in its ability to present rich data on both teaching quality and child outcomes using a large, representative sample of schools from a particularly disadvantaged setting in a low-income country. We must also acknowledge the limitations of this work. These limitations fall under three main categories: sampling, design, and interpretation.

# **_Sample_**

The sample of children included in this study is a representative sample of pre-primary aged children studying in the 3,400 PHCIP public schools and can be reasonably argued to be representative of the full population of public pre-primary aged children in the 11 districts in South 38
Punjab. However, as noted earlier, many pre-primary aged children in Pakistan are either not enrolled in school or are studying in private schools that were not included in this study. Further, given the sampling design, children who are chronically absent are less likely to be included. As such, the generalizability of the sample may be somewhat limited to this albeit large and important subset of public pre-primary aged children in South Punjab.

# **_Design_**

We have reported outcomes disaggregated by numerous factors and statistically tested associations between classroom quality and early learning and developmental outcomes. These findings represent the results of a cross-sectional analysis at a single point of time and do not utilize any quasi-experimental sources of exogenous variation. As such, the inherently correlational nature of our analysis prohibits us from making any causal claims. We attempt to address some sources of confounding in the model building process used to address Q4, but it is possible that there are additional unobserved factors that may be influencing results. This is particularly important given the literature on longer-term impacts of classroom quality on learning outcomes in primary education or better income in life. Further research utilizing stronger causal designs must be implemented in order to better understand these types of conclusions.

# **_Interpretation_**

Our exemplar items on AIM-ECD and Teach ECE are a useful framework to understand in concrete terms what the children in the study in South Punjab are and are not able to do and the levels of quality present in classrooms. However, these comparisons are inherently limited without a more robust normative interpretation. Our descriptive analysis is an empirical examination of what children are _able_ to do and what quality levels teachers demonstrate, but we do not report on what level they _should_ be able to achieve. Further work is required to define minimum levels of acceptable proficiency on the AIM-ECD and stronger definitions of minimum benchmarks on Teach ECE in order to better interpret the gaps that exist.

# **Conclusion**

This study is the first large-scale, in-depth study of pre-primary aged children’s outcomes and classroom quality in South Punjab, Pakistan. We find that while some classrooms demonstrate adequate levels of quality, there is substantial room for improvement. We find variable levels of achievement on a range of important school readiness and early learning and developmental skills and identify several factors that may be important to consider for policy.

The data also present some findings that run contrary to popular belief. These include the findings on teacher certification, the schooling outcomes of girls, and the strengths and weaknesses of being in a multigrade classroom – to name a few.

39