_Yi Chang_ 3 (2011) _oglm_ , among others, as well as other built-in Stata commands, are designed to help users follow this approach.

A third approach, which this paper focuses on, is to assess the robustness of findings to arbitrary scaling choices as proposed by Bond and Lang (2013). Specifically, their method builds on the idea that any monotone transformation of an ordinal scale is rank preserving and is therefor an equally valid measure of the underlying variable. They present an optimization method that searches for monotone transforms that minimize and maximize test score gaps, thus allowing researchers to provide bounds to their estimates of test score evolution across sub-groups over time. Here, I extend their technique to more general cases and optimize their algorithm to work with large data sets. This allows us to simulate multiple monotone transforms several times and bound differences by choosing transformations that maximize or minimize differences between sub-groups.

# **2 Robustness to Scaling**

The ordinal nature of test scores implies that any monotonic transformation to a test score scale is potentially valid. We see this concept applied when colleges arbitrarily put more weight on grades from “harder” classes for admission purposes or when, within a particular test, a teacher decides that one question is worth more points because it is more “challenging”. For this reason, when comparing test scores, we can only ascertain that someone with a high test score performed better than someone with a low one. It is much more difficult to know by how much. Standardizing test scores, which is the most common scale transformation, does not solve this issue. In fact, any analysis that relies on comparing standardized test scores implicitly assumes that test scores are interval scales. This assumption means that, in a scale from 0 to 100 points, the difference in underlying knowledge between individuals who scored 0 and 10 points is the same as that of those who score 90 and 100 points. In reality, this is seldom the case as there is no theoretical justification for this assumption: most exams have questions with varying level of difficulty and utilize scales that are arbitrary in their mean, range and distribution. Without interval scale properties, mean differences and standard deviations can be distorted depending on the weight each question receives.

Even when we equate test scores using Item Response Theory (IRT), most of the time, the underlying _θ_ ability parameters obtained are ordinal in nature and defined only up to a linear transformation (see Lord (1975); Ballou (2009) for a detail explanation). Only in the very special case where all questions are equally informative (i.e. Rasch model), IRT scales can be interpreted as interval scales. Nonetheless, this assumption is rarely fulfilled.

An assumption that is more reasonable and appropriate in most settings is that test scores are ordinal. Unlike, temperature and distance where the magnitude of a given difference corresponds to the same underlying measurement at different parts of the distribution, test scores differences of the same “magnitude” can mean very different things at different points of the distribution. The only inference that we can make