# 3. Methods

## 3.1 Data

The analyses described in this paper are drawn from two recent large-scale assessments of education: TIMSS 2019 and Saudi Arabia’s NALO 2018. TIMSS provides a robust nationally representative sample, while the NALO study was conducted to provide regionally representative information (requiring a much larger sample size) as well as national-level data. Although the paper focuses on the national level in the analyses described below, it should be noted that subsequent analyses at the regional level would be possible using NALO data.<sup>5</sup> In addition, there is a substantial degree of overlap between the content covered by the TIMSS and NALO contextual questionnaires, although some variables appear in one study but not the other, or they are presented in slightly different formats. The primary analysis, reported in depth in the next section, is conducted using TIMSS 2019 data. Given the high degree of overlap between the two studies’ focus on mathematics and science, at the same two grade levels (grades 4 and 8), data from NALO 2018 are used to supplement this primary analysis by drawing on variables of particular interest to Saudi Arabia that have no equivalents in TIMSS. In other words, NALO 2018 data are used as supplementary information to shed additional light on questions arising from the multilevel analysis of TIMSS data. For this reason, NALO data are described in text, with full tabulation of the NALO analyses presented in appendix A1.

### _3.1.1 TIMSS 2019_

TIMSS is a study of the International Association for the Evaluation of Educational Achievement (IEA). It assesses mathematics and science achievement at two grade levels, grades 4 and 8. TIMSS has been carried out every four years since 1995. In 2019, 64 countries participated in TIMSS, including Saudi Arabia. In addition to providing countries with robust data on mathematics and science achievement, TIMSS collects a wealth of contextual data from students, parents, teachers, and school principals. In Saudi Arabia, 5,453 grade 4 students (mean age 9.9 years; 49.6 percent male) across 220 public and private schools and 5,680 grade 8 students (mean age 13.9 years; 49.2 percent male) across 209 schools took part in TIMSS 2019. The TIMSS 2019 data were collected using a stratified two-stage cluster sample design, with a sample of schools selected randomly at the first stage and one or more classes of students selected per each of the sampled schools at the second stage (LaRoche, Joncas, and Foy 2020). The Saudi sample of schools was drawn systematically in order for the sampled schools to represent the populations of grade 4 and grade 8 students nationally, with representation from 13 regions and a balance between male and female schools. Implicit stratification methods were used also to ensure representation of the various school types (such as public versus private schools). The IEA requires high participation rates and adherence to standardized administration procedures for participating countries to be included in the international results. The IEA calculates and provides sampling weights (to ensure that the final sample of participating students can be generalized to the national populations of grade 4 and grade 8 students) and plausible values<sup>6</sup> for mathematics and science scores (to ensure accurate population-level estimates of achievement) in order to facilitate appropriate analyses, taking the complex nature of the data into account.

> 5 Representative regional-level data from NALO could be exploited to examine the varying availability of resources and variability in practices across the different regions of Saudi Arabia, and how region-level differences are related to differences in achievement and the gender gap.

> 6 Plausible values are generated by imputing a set of values (five values in the case of TIMSS) representing ‘plausible’ estimates of student achievement based on their responses to the assessment and background variables. Plausible values are not suitable for reporting individual-level results, but at the population level the use of plausible values facilitates the calculation of appropriate standard errors for complex survey designs such as those used by TIMSS where each student is administered only a small subset of the items in the assessment.

7