metadata: authors: 0: author: Yi Chang,Andres
count: World
docna: 0: docna: Test Score Gap Robustness to Scaling : The
            Scale Transformation Command
docty: Policy Research Working Paper
subtopic: Educational Sciences,Inequality,Health Care Services Industry,Health Service Management and Delivery,Education for Development (superceded),Educational Populations,Education For All
teratopic: Industry,Health, Nutrition and Population,Education,Poverty Reduction
abstracts: cdata!: Social scientists frequently rely on the
            cardinal comparability of test scores to assess achievement
            gaps between population subgroups and their evolution over
            time.  This approach has been criticized due to the ordinal
            nature of test scores and the sensibility of results to
            order-preserving transformations, which are theoretically
            plausible. Bond and Lang (2013) document the sensitivity of
            measured ability to scaling choices and develop a method to
            assess the robustness of changes in ability over time to
            scaling choices. This paper presents the scale
            transformation command, which expands the Bond and Lang
            method to more general cases and optimizes their algorithm
            to work with large data sets. The program assesses the
            robustness of an achievement gap between two subgroups to
            any arbitrary choice of scale by finding bounds for the
            original gap estimation. Additionally, the program finds
            scale transformations that are very likely and unlikely to
            benchmark against the results obtained. Finally, the program
            also allows the user to measure how much gap growth
            coefficients change when including controls in their specifications.
repnme: repnme: Test Score Gap Robustness to Scaling : The
            Scale Transformation Command
display_title: Test Score Gap Robustness to Scaling
            : The Scale Transformation Command
listing_relative_url: /research/2019/08/31344304/test-score-gap-robustness-scaling-scale-transformation-command
topicv3: Inequality under Law m1326002 925,Quality of Education m1326344 491,Education m1327048 470,Health Services Delivery m1327294 1305,Psychiatry, Psychology m1504686 3613
publishtoextweb_dt: 2019-08-19T00:00:00Z
historic_topic: Industry,Health, Nutrition and Population,Education,Poverty Reduction
no_of_pages: 16
author_keys: 0: author_key: 1388215
model_extractions: Policy Research Working Paper 8986 Test Score Gap Robustness to Scaling The Scale Transformation Command _Andres Yi Chang_ Human Development Global Practice August 2019
Policy Research Working Paper 8986

# **Abstract**

Social scientists frequently rely on the cardinal comparability of test scores to assess achievement gaps between population subgroups and their evolution over time. This approach has been criticized due to the ordinal nature of test scores and the sensibility of results to order-preserving transformations, which are theoretically plausible. Bond and Lang (2013) document the sensitivity of measured ability to scaling choices and develop a method to assess the robustness of changes in ability over time to scaling choices. This paper presents the scale transformation command, which expands the Bond and Lang method to more general cases and optimizes their algorithm to work with large data sets. The program assesses the robustness of an achievement gap between two subgroups to any arbitrary choice of scale by finding bounds for the original gap estimation. Additionally, the program finds scale transformations that are very likely and unlikely to benchmark against the results obtained. Finally, the program also allows the user to measure how much gap growth coefficients change when including controls in their specifications.

This paper is a product of the Human Development Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The author may be contacted at ayichang@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
**_The scale transformation Command_**

2

# **Test Score Gap Robustness to Scaling: The** **_scale_** **_~~t~~ ransformation_ Command**

Andres Yi Chang World Bank Group Washington, D.C. ayichang@worldbank.org

**Keywords:** STATA, scale ~~t~~ ransformation, test scores, measurement, achievement gaps, robustness to scaling, psychometrics

**JEL Classification:** C15, C49, C87, I29

# **1 Introduction**

With enrollments at a historical high for low- and middle-income countries, the focus in education has shifted away from enrollment and toward improving the quality of education (World Bank World Development Report, 2018). Countries around the world are focusing on both increasing learning and closing substantial gaps across sub-populations, particularly in favor of disadvantaged groups. These gaps have been extensively documented in the United States, where a large literature has identified stubborn test score gaps by race (Bertrand and Pan 2013; Fryer and Levitt 2010), socio-economic status (Reardon 2011, 2013) and sex (Hanushek and Rivkin 2006; Clotfelter et al. 2009; Fryer and Levitt 2004, 2006, 2013).

Gap measurement is essential to make inferences about equitable learning. What is surprising though, despite the seemingly large gaps across sub-groups, is that different studies often come to different conclusions. For instance, Fryer and Levitt (2004) first noted that a substantial black-white test score gap that had received extensive attention in the literature (see Coleman et al. 1981; Kaufman and Kaufman 1983; Krohn and Lamp 1989; Phillips et al. 1998; Phillips 2000) disappeared after controlling for covariates. Additionally, they found that over the first 4 years of school, blacks lose about -.10 standard deviations per year compared to other races. Bond and Lang (2013) then demonstrated that even this seemingly rock-solid racial gap in the United States is in fact, not robust to arbitrary scale transformations. That is, transformations that minimize or maximize learning by weighting test-items differently, while still preserving order, create lower and upper bounds that are too wide to be informative or definitive.

These findings contribute to an ongoing debate around test score measurement and, in particular, on whether the assumptions that justify cardinality are satisfied in most tests (Ballou 2009; Ho 2009; Bond and Lang 2013; Jacob and Rothstein 2016). Three alternatives have been proposed. The first is to argue that the test at hand has been constructed in a manner that allows for cardinal measures of ability; this is the approach of Rasch measurement and Domingue (2014) provides statistical methods to verify the assumptions required for cardinality. The second is to give up on cardinality entirely and focus only on ordinal comparisons. Reardon et al.’s (2017) _HETOP_ , Williams’s
_Yi Chang_ 3 (2011) _oglm_ , among others, as well as other built-in Stata commands, are designed to help users follow this approach.

A third approach, which this paper focuses on, is to assess the robustness of findings to arbitrary scaling choices as proposed by Bond and Lang (2013). Specifically, their method builds on the idea that any monotone transformation of an ordinal scale is rank preserving and is therefor an equally valid measure of the underlying variable. They present an optimization method that searches for monotone transforms that minimize and maximize test score gaps, thus allowing researchers to provide bounds to their estimates of test score evolution across sub-groups over time. Here, I extend their technique to more general cases and optimize their algorithm to work with large data sets. This allows us to simulate multiple monotone transforms several times and bound differences by choosing transformations that maximize or minimize differences between sub-groups.

# **2 Robustness to Scaling**

The ordinal nature of test scores implies that any monotonic transformation to a test score scale is potentially valid. We see this concept applied when colleges arbitrarily put more weight on grades from “harder” classes for admission purposes or when, within a particular test, a teacher decides that one question is worth more points because it is more “challenging”. For this reason, when comparing test scores, we can only ascertain that someone with a high test score performed better than someone with a low one. It is much more difficult to know by how much. Standardizing test scores, which is the most common scale transformation, does not solve this issue. In fact, any analysis that relies on comparing standardized test scores implicitly assumes that test scores are interval scales. This assumption means that, in a scale from 0 to 100 points, the difference in underlying knowledge between individuals who scored 0 and 10 points is the same as that of those who score 90 and 100 points. In reality, this is seldom the case as there is no theoretical justification for this assumption: most exams have questions with varying level of difficulty and utilize scales that are arbitrary in their mean, range and distribution. Without interval scale properties, mean differences and standard deviations can be distorted depending on the weight each question receives.

Even when we equate test scores using Item Response Theory (IRT), most of the time, the underlying _θ_ ability parameters obtained are ordinal in nature and defined only up to a linear transformation (see Lord (1975); Ballou (2009) for a detail explanation). Only in the very special case where all questions are equally informative (i.e. Rasch model), IRT scales can be interpreted as interval scales. Nonetheless, this assumption is rarely fulfilled.

An assumption that is more reasonable and appropriate in most settings is that test scores are ordinal. Unlike, temperature and distance where the magnitude of a given difference corresponds to the same underlying measurement at different parts of the distribution, test scores differences of the same “magnitude” can mean very different things at different points of the distribution. The only inference that we can make
**_The scale transformation Command_**

4 between someone with a score of 90 points and another person with a score of 100 is that the individual with a score of 100 demonstrated a larger amount of underlying knowledge or ability than that of the individual who scored 90.

In their 2013 paper, Bond and Lang noted that the current literature does not adequately ensure that results are robust to scaling transformations and devised a methodology to find bounds for estimates measuring differences in test scores across sub-groups and over time, such as the black-white test score gap in the United States. In particular, they created an algorithm that finds the upper and lower bounds for the estimates of differences in test scores across two sub-groups at two points of time. In order to do so, they transform test scores using a 6th-degree polynomial monotonic transformation and optimize over the test score gap growth (i.e. difference in test scores between two sub-groups in _t_ 0 minus the difference in _t_ 1) to make it as large or small as possible.

In the next section, I describe the step-by-step methodology developed by Bond and Lang and how the `scale` ~~`t`~~ `ransformation` command extends their methodology for general use to a wide range of applications in this literature.

# **3 The scale** **~~t~~ ransformation Command**

## **3.1 Bond and Lang Methodology**

The Bond and Lang (2013) methodology tests the robustness to scaling of sub-group differences using arbitrary monotone transformations of an underlying ability or test score distribution. Relying on 6th-degree polynomial transformations, the original code tries to minimize or maximize the differences in test score growth between any two sub-groups. These growth-minimizing and growth-maximizing transformations present lower and upper-bounds respectively of the differences between sub-groups.

### **Step-by-step methodology**

1. Define the objective function as follows

_GapGrowth_ = _α_ (1 _,t_ =1) _− α_ (1 _,t_ =0) _,_ (1) where _α_ (1 _,t_ ) is the difference in test scores between groups at time _t_ (see Step 2d for more details on how to obtain _α_ (1 _,t_ )). Note that test scores need to be already vertically equated. That is, they should be completely comparable and mapped into a single scale regardless of the time in which they were captured.

2. Optimize the following function over the defined objective:

- a. Take the original test scores and scale them down to be between 0 and 1 (for computational speed).

- b. Transform original test scores ( _s_ ) using a 6th-degree polynomial monotonic transformation _T_ ( _s_ ), optimizing over the gap growth by choosing 5 coeffi-
_Yi Chang_ 5 cients ( _β_ 2 _, β_ 3 _, β_ 4 _, β_ 5 and _β_ 6) and 1 constant ( _c_ ) for the polynomial transformation:

- _T_ ( _s_ ) = _β_ 1( _s−c_ )+ _β_ 2( _s−c_ )<sup>2</sup> + _β_ 3( _s−c_ )<sup>3</sup> + _β_ 4( _s−c_ )<sup>4</sup> + _β_ 5( _s−c_ )<sup>5</sup> + _β_ 6( _s−c_ )<sup>6</sup> (2)

- c. Standardize transformed test scores in both periods (i.e. _t_ 0 and _t_ 1) together. The original Bond and Lang code standardizes test scores separately, which could artificially increase or decrease the resulting gap. By jointly standardizing both years, we ensure that they remain vertically equated and that the resulting gap is only due to the transformations.

- d. Run an OLS regression separately for each period using a subgroup dummy and controlling for other desired variables. Given this set-up, the coefficient on the subgroup dummy can be interpreted as the difference between each sub-group.

_T_ ( _s_ ) _t_ = _α_ (0 _,t_ ) + _α_ (1 _,t_ ) _GroupDummy_ + _θtControls_ (3)

- e. Find gap growth between _t_ 0 and _t_ 1, as defined in Step 1.

3. Run Step 2 multiple times from different random starting values, checking that transformations preserve order (monotonic rule).

# **3.2 Improvements**

The process described in section 3.1 is complex and computationally demanding. Most of the original public code is written in Mata, which might restrict wider use among researchers. Furthermore, the public code was written for the specific estimation in their paper. Thus, it makes some assumptions that might not necessarily extrapolate to other data and might affect the speed or magnitude of the results. This left space for two areas of improvement: (a) making the methodology more accessible to Stata users, and (b) improving the precision, flexibility and efficiency of the methodology.

In terms of accessibility, the `scale transformation` command, together with this paper and its documentation will allow any Stata user to perform these robustness checks in one line of code.

Methodologically, the original process also sacrificed precision for efficiency. For instance, the parameter _β_ 1 was not optimized and the monotonicity rule used did not check every single plausible value but rather checked monotonicity in small equallyspaced intervals. The `scale` ~~`t`~~ `ransformation` command improves on these aspects as well as on computational speed.

Specifically, the original methodology has been improved by computing more accurate estimates (in quad precision) using QR decomposition tools in Mata. The monotonicity rule has also been enhanced to be more accurate and flexible with 3 variants: (a) a default rule that checks for monotonicity at every possible score value up to 4 decimals within the plausible range in a transformed scale between 0 and 1; (b) a quick
**_The scale transformation Command_**

6 rule, which only checks for monotonicity at the unique scores values present in the sample data; and (c) a theoretical rule that checks for monotonicity at every possible value using an external file. The latter is of particular value if test scores come from a larger data set that has scores for multiple years (which are not included in the optimization) or includes scores for sub-groups not being analyzed. Furthermore, other options such as time-specific controls and weights have been included, as well as options to specify the number of times the program will run (yielding different results), the optimization methodology to be used at certain stages of the process, and the maximum number of iterations before the optimization cycle stops and returns results as if convergence was achieved.

Another important improvement of the `scale transformation` command is that it is robust to transformations that fully change the sign of the gap regardless of its initial value. Since the optimization problem for the maximization or minimization gap growth options is complex and does not have a closed-form solution, in some specific cases the maximization could yield the solutions for the minimization problem (and vice versa). The `robust(` _integer_ `)` option allows the program to use the first _K_ = _integer_ iterations to check that the sign of the optimization is correct.

Finally, other robustness checks used by Bond and Lang have been added to the program. In particular, the `scale` ~~`t`~~ `ransformation` command allows to search for transformations that maximize or minimize the correlation between initial and final test scores by either looking at the binary correlation between them or the resulting R-square from regressing initial test scores on final test scores. These transformations allow the user to benchmark a likely or unlikely transformation and compare it to the results obtained when maximizing or minimizing the gap. Likewise, the program allows users to optimize over the absolute value of the difference between the Gap Growth calculated with and without controls. This additional check helps users to measure how much Gap Growth coefficients could change when including controls to their specifications.

# **3.3 Description**

The `scale` ~~`t`~~ `ransformation` command finds a monotonic transformation for a test score scale, which optimizes a specific objective function. The optimization object includes scores at two points of time for two comparison groups (e.g. rich vs poor). This program performs a grid search from multiple random initial parameters to find the desired values. All the transformations found by this command are theoretically plausible given the ordinal nature of test scores.

A 6th-degree polynomial monotonic transformation is used because it provides flexibility to approach numerous continuous functions. Nevertheless, monotonicity checks need to be applied as this type of transformation does not necessarily preserve order. Furthermore, the monotonicity restriction introduces a discontinuity into the objective function that creates multiple local maxima and minima. These characteristics yield a complex optimization problem that does not have a closed-form solution.

The `scale` ~~`t`~~ `ransformation` program takes advantage of the Mata [M-5] **optimize**
_Yi Chang_ 7 function and performs a grid search to find multiple solutions. The results are reported in a data set format that includes the values of the evaluated objective functions, the resulting parameters of each optimization iteration and the initial parameters that yielded that result.

To install the command on your system, open Stata and type: `. ssc install scale transformation` .

You can also find the latest version of the program at: `https://github.com/andresyichang/scale transformation` .

# **3.4 Syntax**

`scale transformation ,` <u>`type(`</u> _integer_ `)` <u>`score1(`</u> _varname_ `)` <u>`score2(`</u> _varname_ `)` � <u>`compgroup(`</u> _varname_ `)` <u>`controls(`</u> _varlist_ `)` <u>`controls1(`</u> _varlist_ `)` <u>`controls2(`</u> _varlist_ `)` <u>`weights(`</u> _varname_ `)` <u>`iterations(`</u> _integer_ `)` <u>`maxoptiterations(`</u> _integer_ `)` <u>`singhmethod(`</u> _integer_ `)` <u>`bounddown(`</u> _integer_ `)` <u>`boundup(`</u> _integer_ `)` <u>`monotonicity(`</u> _integer_ `)` <u>`monofile(`</u> _filename_ `)` <u>`timeroff save(`</u> _filename_ `)` <u>`seed(`</u> _integer_ `)` <u>`robust(`</u> _integer_ `)` <u>�</u>

# **3.5 Main Options (Required)**

`type(` _integer)_ indicates the type of optimization and the objective function to be performed. The options are as follows:

1. Gap Growth Maximization

2. Gap Growth Minimization 3. Correlation Maximization 4. Correlation Minimization 5. R-squared Maximization 6. R-squared Minimization 7. Controls Explanation Maximization `score1(` _varname_ `)` specifies the variable that contains scores at (initial) period 1. In order to improve processing speed, this variable is automatically scaled to be between 0 and 1 prior to running the program. The transformation used for this process is the same for variables score1 and score2, and does not affect the resulting estimations in any way.
**_The scale transformation Command_**

8 `score2(` _varname_ `)` specifies the variable that contains scores at (final) period 2. In order to improve processing speed, this variable is automatically scaled to be between 0 and 1 prior to running the program. The transformation used for this process is the same for variables score1 and score2, and does not affect the resulting estimations in any way.

# **3.6 Secondary Options**

<u>`compgroup(`</u> _varname_ `)` specifies the variable that contains the group classification to be analyzed and from which the program will compare the top and bottom groups. This variable needs to be numeric. The program will take the lowest and highest values (i.e. groups) and compare them, while controlling for all other groups in the middle at the same time. Note that the variable compgroup should only take on integers greater or equal to 0 (all missings will be ignored). Only allowed when optimizing Gap Growth or Controls Explanation (i.e. type 1, 2 or 7).

<u>`controls(`</u> _varlist_ `)` include varlist as controls at both period 1 and 2. Variables should be numeric as they will be used directly in regressions. Only allowed when optimizing Gap Growth or Controls Explanation (i.e. type 1, 2 or 7).

<u>`controls1(`</u> _varlist_ `)` include varlist as controls at period 1 only. Variables should be numeric as they will be used directly in regressions. Only allowed when optimizing Gap Growth or Controls Explanation (i.e. type 1, 2 or 7).

<u>`controls2(`</u> _varlist_ `)` include varlist as controls at period 2 only. Variables should be numeric as they will be used directly in regressions. Only allowed when optimizing Gap Growth or Controls Explanation (i.e. type 1, 2 or 7).

<u>`weights(`</u> _varname_ `)` uses varname as inverse probability weights. If not specified, weights are set to 1.

<u>`iterations(`</u> _integer_ `)` specifies number of times that the program will find optimal values using unique random-generated initial parameters. The default is set to `iterations(1000)` . The more iterations, the longer it will take the program to run and finish. Note that by default, if the program is stopped before ending, the results will not be stored. It is recommended that you test your program with a low number of iterations.

<u>`maxoptiterations(`</u> _integer_ `)` specifies maximum number of iterations before the optimization program stops and return results as if convergence was achieved. The default is set to `maxoptiterations(25)` , which in practice should be enough to achieve ’convergence’ with a fair degree of accuracy. This limit is necessary as the monotonicity restriction and complexity of the objective functions cause the optimization program to run indefinitely otherwise. See [M-5] **mf** **~~o~~ ptimize** **~~m~~ axiter** for more details.

<u>`singhmethod(`</u> _integer_ `)` specifies what the optimizer should do when, at an iteration step, it finds that H is singular. The default is set to singhmethod(1) for ”hybrid” but can be changed to 2 for ”modified Marquardt algorithm”. See [M-5] **mf** **~~o~~ ptimize** **~~s~~ ingularH**
_Yi Chang_ 9 for more details.

<u>`bounddown(`</u> _integer_ `)` specifies lower bound for the random-generated initial parameters for the optimization. This program creates random initial values and then uses them to find all the different local maxima or minima, accordingly. The default is set to `bounddown(-1500)` .

<u>`boundup(`</u> _integer_ `)` specifies upper bound for the random-generated initial parameters for the optimization. This program creates random initial values and then uses them to find all the different local maxima or minima accordingly. The default is set to `boundup(1500)` .

<u>`monotonicity(`</u> _integer_ `)` specifies the type of monotonicity check to be applied. The default is set to `monotonicity(1)` for ”Standard”, which checks for monotonicity at every possible score value (up to 4 decimals within the plausible range) in the original scale that was transformed to be between 0 and 1. When less precision is needed, monotonicity can be set to 2 for “Sample”, which only checks for monotonicity at the unique scores values present in the sample data. If the test scores come from a larger data set that have scores for multiple years, it is possible to do a more thorough check (but more time consuming) by putting together an external file with all theoretical or observed plausible scores where you want the program to check for monotonicity. For the latter, you must set monotonicity to 3 for “External” and include the filename in `monofile(` _filename_ `)` .

<u>`monofile(`</u> _filename_ `)` specifies the file to be used when monotonicity type is set to 3 for ”External”. This option should only be used with `monotonicity(3)` . You can use .dta, .csv, .xls or .xlsx files. Make sure that .csv, .xls and .xlsx files have the name of the variable on the row.

<u>`timeroff`</u> turn off the timer. The default is set to include the time (in minutes) that the program took to run, which is reported only if the program finishes.

<u>`save(`</u> _filename_ `)` save optimization results using filename once program is done.

<u>`seed(`</u> _integer_ `)` set seed for replication, where 0 _<_ = integer _<_ = 2,147,483,647. If missing, the seed is randomly generated and included in the header of the program. Always save the log file to ensure you are able to recover the seed number for replication.

<u>`robust(`</u> _integer_ `)` indicates the program to use the first _K_ = _integer_ iterations to check that the sign of the optimization is correct. Since the optimization problem for the maximization or minimization Gap Growth options (i.e. type 1 or 2) is complex and does not have a closed-solution, in some specific cases the maximization could yield the solutions for the minimization problem (and vice versa). In order to address this issue, the user could activate the robust option to use the first _K_ = _integer_ (where 20 _<_ = _K <_ = 300; _K_ = _even_ ) iterations (in addition to _N_ specified by the iterations option) to check for a possible change of sign in Max/Min Gap Growth optimization results. Half of the _K_ iterations are used to the maximization solution while
**_The scale transformation Command_**

10 the other half _K_ iterations are used for the minimization problem. Once the correct approach is selected, the program keeps the solutions for the correct specification and runs the remaining _N_ iterations indicated in the `iterations()` option.

## **3.7 Optimization Objectives**

The `scale transformation` command finds a 6th-degree polynomial monotonic transformation for a test score scale that optimizes one of the following:

1. Gap Growth: gap difference between the bottom and top groups from the compgroup variable in period 2, minus the same gap difference in period 1. In other words, it is the difference between the regression coefficient of the top compgroup indicator (controlling for groups in the middle) in period 2, minus the same regression coefficient of the top compgroup indicator (also controlling for other groups in the middle) in period 1. In both, the coefficient of the top compgroup indicator measures the difference between the top and bottom groups. Note that the Gap Growth is positive when the Gap widens and negative when the Gap shrinks (always with respect to the Gap in period 1). Thus, the Gap Growth will always be negative when the coefficient changes signs between period 1 and 2 (regardless of whether it is from + to -, or from - to +). Conversely, it will be positive when the gap widens, for instance, from -.1 to -.3;

2. Correlation: coefficient from regressing test scores in period 1 on test scores in period 2 (no controls allowed); 3. R-squared: from regressing test scores in period 1 on test scores in period 2 (no controls allowed); or 4. Controls Explanation: measures how much Gap Growth coefficients change when including controls in `controls1(` _varlist_ `)` and `controls2(` _varlist_ `)` - not in `controls(` _varlist_ `)` . This allows to test the explanatory power of only a subset of desired control variables. That means that, if you want to measure how much Gap Growth coefficients change by introducing time-invariant controls, you must include these variables in both `controls1(` _varlist_ `)` and `controls2(` _varlist_ `)` . The Controls Explanation term is estimated by taking the distance (i.e. absolute value of the difference) between the Gap Growth calculated without variables included in `controls1` and `constrols2` , minus the Gap Growth calculated with these two controls varlists. To make the interpretation easier, both Gap Growths coefficients - with and without controls - are included in the results.

# **4 Resulting Output**

## **4.1 Example**

In this section, we use the `timss` ~~`t`~~ `estscores.dta` ancillary data set file to find the Gap Growth maximization solution for test scores between year 1 and 2, comparing males
<!-- Start of picture text --> GlFleDataEdt EditorViewrows)Data - [Untitled]Took - ox<br>gas fe a) v-<br>tw. i<br>obit [ J<br>=a =a = a ea = ee x TeeBt letach<br>rs . . . . . . + 38.970212 -325.509282 599.09734saree ue cate112.97602 ea -1424.098eee 012.74774decantecee -120.101j|eaase [|Sec) Mteomiee<br>3 68752935 8500951.2 19550521 25176778 -8384590.5 -377167.5 712280.41 -.1509461 -998.20917 382.0521 12474915 -1458.8124 399.06216 754.65643 791.74609 |g py<br>54 =: 27 :F 23 EZ 7_ :E += 1169.8243689.75195 -569.90527999.2793 68.958015616.1552 579.929021152.0438 -1139.7943-45.424345 758.26648103.43911 -445.92679-1201.5419 | be|b?<br>6 +6897979 -S1805.18 206889.43 85112.771 234586.04 -122684.51 46830.111 .21258351 692.27509 715.07086 374.93167 266.05246 -605.56091 238.15776 868.1499 ia bs:<br>8 : : : : : : : + 670,32507 741.10284 148,88612 -248.94681 -1228.5475 315.0404 -656.20361 | we<br>9 : : : - : : : + 995.17462 -966.36139 1158.7749 -1911.179 115.791 €44.73279 -1268.7124 | intbT<br>10 ‘ 5 5 < Fi : Fi + 7376.82166 -1342,0963 71.957176 -324.57217 1132.1536 953.63586 -1333.572 | mtb?<br>11 62046187 32.756008 1257.0352 -#29.92018 935.05759 8754.6019 .08702618 -75151.712 32.756008 1257.0352 -829.92022 935.69274-1271.765 429.4967 _766.42651 a<br>12 6851018 4804311.7 13041236 55255176 1.445e+08 51261142 6395256.8 31159448 322.06852 767.10315 10278666 954.08605 -1320.0433 648.61108 74506515 |ntbs<br>a3 : ; 5 7 E : : + 470,00652 1240.1121 1200.8855 -602.50954 94.425602 44..462987 -1450.4161 | @ initbs<br>1516“4 ::: 7:: r.. ::: :: :: F:: +++ -731,35742799.209041376.7255 -1260.0149-1402.5431672,$7797 -1245.2069-1185.975¢9.327011 -1142.7782-705.20416659.04155 -612.11401-310.9761008.02149 538.05884210.26232146.0199 -044.25917-867.40161580.5025 ||| usPinte igismpos<br>718 :3 :p :E :7 :F ". :F ++ 7296.564457418-66739 -1145.9005-1978.9674 625.120311236.9536 -1017.2505-1242.7751 1222-0761-308.55228 -1005.6263-245.0954 ~79.572342100.7124 avaiableaie |a<br>19 = é ‘ = E , : + -1424.012 -706.5781 576.49463 732.4502 -1410.4358-907.4621sa1.09277_| [Mme ot<br>20 69573202 1102062.7 1629028 -1142306.9 794959.63102063.45957629.51 .690775S6 -599.25474 229.6698 -1212.0749 -1296.2605 -1004.7808422.52438 1957.5984 | |r dou<br>21 69083772 1.705e+08 © 4.145e+08 ©. E55e+08 4.501e+08 -4.005e+08 2.244e+08 04286195 -1273.5139 646.45551 -1288.3158 573.61652 -894.58423 -332.68869 -1142.4724 |Format %10.09<br>2 . : . . . . . + 700.79742 892.4116 -969.93629 -682.47473 $05.44897 | 1155.2128 -130.90827 | [ht<br>23. .21006293 -1193.4755 -11036.201 227.2552 503.76974 2.729e+08 -111.67814 -2.586e10 -1193.4755 644.10681 -229.60178 674.4885 335.48074 403.3064 997.31073 (5 Data |<br>24 65186156 162559.42419614.64 591554.51 635750.24 -235162.59 161949.07 16364163 -1126.4531 940.4574 217.52652 1258-5411 907.58075 44404959 S21.08276 hier<br>25 69008137 295045.9¢ 338834.2 317295.2 -23788704 -4506607.3 577944.76 4815186 -354.15881 744.21893 30.147589 738.54944 738.3222 714.92676 _791.97278 |Notes<br>26 z . F z E 5 E + -1374,653 910.9969 -245.00269 -1209.6971 -1291.0752 -545.04254 954.54291 || wisble15<br>27 z z E z E 5 5 + 642,18646 -1044.0369 -356.86188 -211.99652 115.1077 1468.79 -3.895102 | (oS<br>28 . : . : : . . 1 1183,¢106 -3.70¢79¢5 -1202.928 402.57416 216.71965 351.4473 ~aa15.0882 | Menorsa<br>29 = .68989564 45039487 = 1.166e+08 5.1736+08 1.855e+08 -66776747 29903572 .19055157 17.254559 -129.90472 1015.5053 -576.34753 -237.06885 1299.6484 948.56325 sSeredt<br>30 69240777 96168557 2.477408 4.5170408 3.4130408 -2.3740+08 1.746008 16467885 -69.69006 187.9828 -551.79022 886.92611 -1090.1384 -209.92022 -951.29101<br><!-- End of picture text -->
**_The scale transformation Command_**

12 maximization (i.e. `type(1)` ), we select the transformation that yields the highest Gap Growth from the resulting data set, which is equal to .695732. Then, we use the corresponding coefficients and constant and apply the 6th-degree polynomial transformation in equation (2). Figure 2 plots the original versus transformed scores on the left and the test score gap evolution by sex between year 1 and 2. This particular Gap Growth maximization makes the gap in year 1 as small as possible while, at the same time, making the gap in year 2 as large as it can.

Note that the original gap magnitude was .343418, about half the size of the transformed gap. These results are particular to this specific scenario and, in practice, the algorithm might behave differently depending on the data and the objective function at hand.

## Figure 2: Original vs Transformed Scores

<!-- Start of picture text --> Original vs Transformed Scores Original vs Transformed Gap Evolution<br>Max Gap Growth by Sex 1<br>4<br>2 .5<br>0<br>0<br>−2<br>−.5<br>−4<br>−4 −2 0 2 4<br>Transformed Scores<br>−1<br>Scores Y1 Scores Y2 1 2<br>kdensity Transformed Scores Y1 kdensity Transformed Scores Y2 Years<br>45 degree line<br>Original Gap Evolution Transformed Gap Evolution<br>Original Gap Growth=.343417677879334<br>Max Gap Growth=.6978828 Note: Top and bottom lines for each color represent sex groups<br>Original Scores Test Scores<br><!-- End of picture text -->

# **4.2 Interpretation**

Regardless of the optimization object chosen, the resulting data set has a similar structure. Each row contains the results for one simulation. The `obj` variable contains the optimization object, while the rest of the variables contain the parameters of the 6th-degree polynomial transformation that achieves that object and the initial random starting values for these parameters. This format allows the user to not only have the full distribution of results but also access the necessary parameters to replicate the desired scale transformation if needed.

For the Gap Growth maximizing and minimizing options, one should be aware that the program finds the most extreme transformations, some of which might be extremely unlikely (although theoretically plausible). Thus, for instances where the bounds are too wide, it might be useful to benchmark against likely transformations (such as those found by the `scale` ~~`t`~~ `ransformation` program that maximize correlation or R-squared) and explore other properties of test scores that could help us discard some of the most extreme and very unlikely transformations. For instance, Ho and Yu (2015) present values for skewness and kurtosis from 330 United States state-level examinations. Based on this, one could look at the moments of the resulting distributions and discard those
_Yi Chang_ 13 that are extremely rare and unlikely to happen, thus narrowing the space between the resulting bounds.

# **5 Conclusion**

One simple step that researchers can take in order to address some the challenges of scaling and the ordinal nature of test scores is to test the robustness of their results to changes in the test score scales. The `scale` ~~`t`~~ `ransformation` command allows anyone to implement several robustness checks to scaling decisions, particularly in the context of analyzing achievement gaps between sub-groups in a panel setting. These checks build on the methodology developed by Bond and Lang (2013) and allow users to find bounds to the original gap estimation or transformations that are very likely or unlikely so as to benchmark against other results obtained. In practice, these are some of the essential checks that should be done when comparing test score averages over time and across sub-groups.

When using the `scale` ~~`t`~~ `ransformation` command, the researcher recognizes the ordinality of test scores and tests the robustness of her/his results to arbitrary monotonic transformations. This approach is more defensible on a theoretical basis and has the potential to discern real gaps from results that are only significant due to the arbitrary choice of scales made or taken as given.