## _3.1.2 NALO 2018_

Saudi Arabia’s NALO is administered by the Education and Training Evaluation Commission (ETEC), an independent government agency responsible for school evaluation, accreditation, and assessment, among other responsibilities. In 2018, the domains assessed by NALO were mathematics and science in grades 4 and 8, which means that the data from NALO 2018 are closely aligned to TIMSS 2019 both in terms of the target domains and grade levels. In NALO 2018, 27,985 grade 4 students (50.2 percent male) across 964 government, private, and Quran schools completed tests of mathematics and science, as did 30,157 grade 8 students (49.6 percent male) across 939 schools. The schools that took part in NALO were sampled using procedures similar to those used in TIMSS.

# 3.2 Statistical analysis

Prior to the main analysis, descriptive statistics were computed, and relevant statistical tests were conducted to provide a comprehensive overview of the gender differences across the contextual variables of interest to this study. The levels of statistical significance along with the relevant effect sizes for each of these differences are reported. The phi ( _φ_ ) and the Cramer’s V ( _φc_ ) effect size measures were used for the contextual categorical variables for 2x2 contingency tables and for contingency tables larger than 2x2, respectively. The Cohen’s _d_ effect size measure was used for the contextual continuous variables (Fritz, Morris, and Richler 2012). Cohen (1988) guidelines were used in conjunction with Hattie (2009) guidelines for the interpretation of the effect sizes. The IEA International Database Analyzer (IDB Analyzer) and the Statistical Package for the Social Sciences (SPSS) were used to compute descriptive statistics for TIMSS and NALO, respectively.

Next, multilevel linear regression analysis was used to explore the different factors that are associated with boys’ underperformance in Saudi Arabia as well as all students’ achievement in mathematics and science in grades 4 and 8 more generally. The multilevel linear regression models draw on the TIMSS 2019 data, with NALO 2018 data used to supplement the analysis by describing data that are not available in TIMSS. Four separate families of models were constructed: grade 4 mathematics, grade 4 science, grade 8 mathematics, and grade 8 science. All plausible values of achievement and sampling weights were used in all the analyses as per the relevant guidelines by von Davier et al. (2009) and Rutkowski et al. (2010), respectively. Assumptions necessary for conducting the multilevel linear regression analysis (i.e., linearity, homogeneity of variance, normality of errors) were checked and met, and parameters for the models were estimated using the maximum likelihood estimation with robust standard errors ( _SEs_ ) (Muthén and Muthén 2017). Multilevel linear regression analysis was performed using Mplus 8 software.

Along with mathematics and science achievement scores, variables included in the multilevel models are drawn from the student and home questionnaires (level 1 of the analysis) as well as the teacher and school principal questionnaires (level 2 of the analysis). Variables were selected for inclusion in the models _a priori_ , based on the previous literature on student achievement and the gender gap in education in particular, as discussed in section 1 of this paper, in addition to the expected theoretical or policy relevance of these variables to the question of gender differences in performance in the Saudi context. As far as possible, each model was constructed using the same set of variables, notwithstanding some slight differences arising from the selection of variables related specifically to mathematics or science instruction and some differences between the grade 4 and 8 questionnaires.

A hierarchical approach was followed, whereby conceptually similar variables were entered into each step of each model in blocks, as shown in table 1. The first step of each model included student gender only. By including the student gender variable into the model alone, the difference in achievement between boys and girls, after controlling for the clustering of the data within classes/schools, could be observed. Next, different blocks of variables were entered into the model one by one to allow both for the examination of their contribution in explaining the difference in achievement between boys and girls 8