however, it was learned that there was in fact considerable variation in the program’s impact. The average of this variation may have been close to zero, but for certain groups the program had worked quite well, for others it had had no impact, while for still others it had been detrimental. Who were these different groups, and what was it about them that led to such variable outcomes? A companion qualitative process evaluation<sup>17</sup> was able to discern that the key difference was the quality of implementation received by different groups, the level of support provided to them by managers and political leaders, and variations in the nature and extent of local-level inequalities (which in turn shaped which groups were able to participate, and on what terms). The administrative rules and implementation guidelines provided to all groups were identical, but in this case a qualitative process evaluation was able to document the ways and places in which variable fidelity to them yielded widely different outcomes (albeit with no net impact). Moreover, the qualitive data was able to discern subtle positive effects from the program that reliance on the quantitative survey instrument alone would have missed.

# 2. Measurement becomes conflated with management

<mark>An extension of the above point is that undue emphasis on quantitative data can lead to measurement becoming a</mark> _<mark>substitute for</mark>_ <mark>rather than a</mark> _<mark>complement to</mark>_ <mark>management. This is evident when only that which is quantifiable receives any significant form of managerial attention, an outcome made possible when the easily quantifiable becomes the measure of success, in turn becoming the object of management’s focus, typically to the exclusion of all else. As Wilson (1989: 161) famously intoned in this classic study of bureaucratic life, “[w]ork that produces measurable outcomes tends to drive out work that produces immeasurable outcomes” (Wilson 1989: 161).</mark> <mark>In one sense this is hardly surprising; the need for managers to make decisions on the basis of partial information is difficult and feels risky, so anything that claims to fill that gap and bypass the perceived uncertainty of subjective judgement will be readily welcomed. “The result”, Simon Caulkin argues, “both practically and theoretically, is to turn today’s management into a technology of control that attempts to minimise rather than capitalise on the pesky human element.”</mark><sup>18</sup> <mark>And in public administration, a managing-it-by-measuring-it bias can mean that, over time, the bulk of organizational resources end up neglecting the “pesky human element” of change processes, despite the fact that it is this element which is often central to attaining any transformational outcomes managers are seeking.</mark> This dynamic characterizes key aspects of the Saving One Million Lives (SOML) initiative, an ambitious health sector reform program launched by the Government of Nigeria. The original goal of SOML was to save the lives of one million mothers and children by 2015; to this end,

> both qualitative and econometric methods is the capacity of the former to identify factors that were not anticipated in the original design (see Rao 2022). In either case, Ravallion’s (2001) injunction to “look beyond averages” when engaging with complex phenomena is worth being heeded by all researchers (and those that interpret the researchers’ findings), no matter their disciplinary or methodological orientation.

> 17 On the use of mixed methods in process evaluations, see Rogers and Woolcock (forthcoming).

> 18 <u>https://www.treasurers.org/hub/treasurer-magazine/decision-making-how-make-most-data</u>

10