metadata: authors: 0: author: Mahony,Christopher Brian
1: author: Manning,Matthew
2: author: Wong,Gabriel
count: World,Ukraine
docna: 0: docna: Weighting Justice Reform Costs and Benefits
            Using Machine Learning and Modern Data Science
docty: Policy Research Working Paper
abstracts: cdata!: Can the impact of justice processes be
            enhanced with the inclusion of a heterogeneous component
            into an existing cost-benefit analysis app that demonstrates
            how benefactors and beneficiaries are affected? Such a
            component requires (i) moving beyond the traditional
            cost-benefit conceptual framework of utilizing averages,
            (ii) identification of social group or population-specific
            variation, (iii) identification of how justice processes
            differ across groups/populations, (iv) distribution of costs
            and benefits according to the identified variations, and (v)
            utilization of empirically informed statistical techniques
            to gain new insights from data and maximize the impact for
            beneficiaries. This paper outlines a method for capturing
            heterogeneity. The paper tests the method and the
            cost-benefit analysis online app that was developed using
            primary data collected from a developmental crime prevention
            intervention in Australia. The paper identifies how
            subgroups in the intervention display different behavioral
            adjustments across the reference period, revealing the
            heterogeneous distribution of costs and benefits. Finally,
            the paper discusses the next version of the cost-benefit
            analysis app, which incorporates an artificial
            intelligence-driven component that reintegrates individual
            cost-benefit analysis projects using machine learning and
            other modern data science techniques. The paper argues that
            the app enhances cost-benefit analysis, development
            outcomes, and policy making efficiency for optimal
            prioritization of criminal justice resources. Further, the
            app advances the policy accessibility of enhanced, social
            group-specific data, illuminating optimal policy orientation
            for more inclusive, just, and resilient societal
            outcomes&#x2014;an approach with potential across
            broader public policy.
display_title: Weighting Justice Reform Costs and
            Benefits Using Machine Learning and Modern Data Science
listing_relative_url: /research/2023/05/34064703/weighting-justice-reform-costs-benefits-using-machine-learning-modern-data-science
topicv3: Social Inclusion m1326031 421,Criminal Justice m1326993 1014,Access to Justice m1327477 942
publishtoextweb_dt: 2023-05-18T00:00:00Z
dois: 10.1596/1813-9450-10449
author_keys: 0: author_key: 1357648
1: author_key: NA
2: author_key: NA
model_extractions: Policy Research Working Paper 10449 Weighting Justice Reform Costs and Benefits Using Machine Learning and Modern Data Science _Chris Mahony Matthew Manning Gabriel Wong_ Social Sustainability and Inclusion Global Practice May 2023
Policy Research Working Paper 10449

# **Abstract**

Can the impact of justice processes be enhanced with the inclusion of a heterogeneous component into an existing cost-benefit analysis app that demonstrates how benefactors and beneficiaries are affected? Such a component requires (i) moving beyond the traditional cost-benefit conceptual framework of utilizing averages, (ii) identification of social group or population-specific variation, (iii) identification of how justice processes differ across groups/populations, (iv) distribution of costs and benefits according to the identified variations, and (v) utilization of empirically informed statistical techniques to gain new insights from data and maximize the impact for beneficiaries. This paper outlines a method for capturing heterogeneity. The paper tests the method and the cost-benefit analysis online app that was developed using primary data collected from a developmental crime prevention intervention in Australia. The paper identifies how subgroups in the intervention display different behavioral adjustments across the reference period, revealing the heterogeneous distribution of costs and benefits. Finally, the paper discusses the next version of the cost-benefit analysis app, which incorporates an artificial intelligence-driven component that reintegrates individual cost-benefit analysis projects using machine learning and other modern data science techniques. The paper argues that the app enhances cost-benefit analysis, development outcomes, and policy making efficiency for optimal prioritization of criminal justice resources. Further, the app advances the policy accessibility of enhanced, social group-specific data, illuminating optimal policy orientation for more inclusive, just, and resilient societal outcomes—an approach with potential across broader public policy.

This paper is a product of the Social Sustainability and Inclusion Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at cmahony@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Weighting Justice Reform Costs and Benefits Using Machine Learning and Modern Data Science**

Chris Mahony<sup>1*</sup> , Matthew Manning<sup>2*</sup> , Gabriel Wong<sup>3*</sup>

**Keywords:** justice reform; cost-benefit analysis; machine learning; data science; justice processes; heterogeneity

* This paper exclusively reflects the views of the authors and does not necessarily represent those of the World Bank nor its Board of Directors. The authors thank Varalakshmi Vemuru, Jose Cuesta, Rohini Srihari, James Brumby, Barry Maher, Patrick Barron, and Aly Rahim for useful feedback and suggestions. All remaining errors are solely the responsibility of the authors.

1 World Bank Group; Peloria P.B.C; 2 Department of Social and Behavioral Sciences, City University of Hong Kong; 3 ANU Centre for Social Research and Methods, The Australian National University
# **Introduction**

The joint United Nations-World Bank flagship report, 'Pathways for Peace' (United Nations and World Bank, 2018), cites grievances surrounding social group-specific exclusion from access to justice and security as one of four arenas of social contest that inform the risk of violence.<sup>1</sup> The World Bank has developed analytical tools for identifying the political economy and efficacy of governance and justice reform including the relationship between justice sector policy and programmatic approaches and the World Bank’s twin goals of reducing extreme poverty and driving shared prosperity.<sup>2</sup> To achieve the above goals, a comprehensive economic framework is required for tracing how public policy processes or their substantive consequences vary across individuals and communities. Currently in this space, however, cost-benefit analysis (CBA) relies on the average treatment effect (ATE), which does not adequately unpack the contextual variations that moderate economic outcomes across affected communities or sub-groups within communities.

This paper considers the policy question specific to the justice sector: Can the impact of justice processes be enhanced with the inclusion of a heterogenous component into an existing CBA APP that demonstrates how benefactors and beneficiaries are affected? The development of such a framework requires: (i) moving beyond the traditional cost-benefit conceptual framework of utilizing averages (i.e. ATE); (ii) identification of variation or differences between groups or populations (e.g. individual actor attributes, political structures, institutions); (iii) identification of how justice processes differ across groups/populations; (iv) distribution of costs and benefits according to the identified variations (i.e. heterogenous impact specific to the nature of justice policy/intervention or the context in which the policy/intervention occurs); and (v) utilization of empirically informed statistical techniques to gain new insights from data and maximize impact to beneficiaries.

> 1 The three other ‘arenas’ of societal contestation associated with social group specific grievances surrounding exclusion include access to political power, access to land and resources, and access to public services. See United Nations & World Bank. (2018). Pathways for Peace: Inclusive Approaches to Preventing Violent Conflict. World Bank. Washington DC.

> 2 See analytical approaches employed by the World Bank: https://www.worldbank.org/en/topic/governance/brief/justice-rights-and-public-safety. And preceding approaches: Justice diagnostics

> (https://documents1.worldbank.org/curated/en/803711468338346161/pdf/437070WP0Box3210March0200701P UBLIC1.pdf), Governance diagnostics

> (https://web.worldbank.org/archive/website00818/WEB/DIAGNOST.HTM).

2
Individually, these points are not novel, but it is their integration that provides new empirical opportunities.

The Pathways for Peace report takes important steps in identifying the delivery of education, health care, water, sanitation, justice and security as “the glue” binding state and society (Milliken and Krause, 2002) via the basic minimum citizens expect in order to accept state authority (Gilley, 2009). However, the report acknowledges the complexity of the relationship between service delivery and legitimacy (Brinkerhoff et al., 2012; Sacks and Larizza, 2012; Stel and Abate, 2014; Mcloughlin, 2015; Fisk and Cherney, 2017). State legitimacy depends on public expectations, which are informed by prior experiences (Nixon et al., 2017), geography, identity, and culture (Sturge et al., 2017). The report acknowledges the Sustainable Development Goals (SDG 16.3) to promote the rule of law and ensure equal access to justice for all (United Nations and World Bank, 2018). The report emphasizes that development approaches must identify _how_ , not only _why_ , justice processes and outcomes discriminate, or are perceived to discriminate, against certain groups (United Nations and World Bank, 2018). Beyond establishment of frameworks identifying how processes treat groups differently, development actors must grapple with how to most equitably deploy finite resources to change that unequal treatment. For example, decisions regarding the equitable deployment of finite resources may be influenced by static (e.g. ethnicity) and dynamic (e.g. socio-economic status) indicators that recognize group differences within a community. Incorporating this data into CBA enables identification of how justice processes treat different groups and the consequences (economic and social) for those groups.

Identifying and communicating the dynamic interactions of factors associated with the individual and net costs and benefits (including their inclusivity) of policy change increases scope for informed citizen participation – enhancing social inclusion and process legitimacy. Social inclusion and process legitimacy constitute two of four (social resilience and social cohesion being the others) clusters of factors comprising the concept of ‘social sustainability’, which is associated with increased per capita income and reduced poverty (Cuesta Leiva et al (2022).<sup>3</sup>

> 3 We note the absence of consensus on the concept or definition of social sustainability. However, we acknowledge the four Cuesta Leiva et al (2022)-identified clusters of identified social sustainability factors. One of those factors is ‘process legitimacy’, described as the extent to which policy and implementation processes’ engagement with local norms and values, and their reconciliation of opposing viewpoints drive perceptions of

3
To take forward a framework that also appropriately prioritizes the use of finite resources, the costs and benefits of policy options must be properly weighed within the context of the “dynamic interaction of actors, institutions, and structural factors over time” (United Nations and World Bank, 2018). Tools necessary to achieve this outcome must drive optimal justice sector decisions in a systematic, data-informed, and resource-efficient way, including drawing on World Bank analytical products. These products include legal needs surveys, Functional Reviews, Institutional and Expenditure Reviews, Justice for the Poor dispute resolution analytics, the Doing Business survey (which includes a Quality of Judicial Process Index), The Women, Business and the Law index, and the Data and Evidence for Justice Reform platform. By drawing together data from all analytical processes, scope for comparison of various justice-related datasets is enabled.<sup>4</sup> Most importantly, the analytical approaches assist in identifying the various elements of justice processes whose costs and benefits may then be traced.

Early CBA tools, such as the Washington State Institute of Public Policy's (WSIPP) Benefit-Cost Tool (Aos and Drake, 2010) and the Manning Cost Benefit Tool (MCBT) (College of Policing, 2022), were developed to support an evidence-based policing approach. More recent developments have been undertaken by Manning and Wong, which represents an extension of the above-mentioned MCBT (now adopted by the College of Policing UK), are beginning to incopporate machine learning and artificial intelligence, including the development of CBT APP (Manning, 2018) and ‘Smart’ CBTs (Manning et al., 2018), take important steps towards robust and time-sensitive analytical methods. These tools (currently in various stages of development), which have been validated using a range of crime data,<sup>5</sup> provide a framework with systematic data management capacity that enables user input decisions as sufficiently fair, credible, and acceptable. A second is ‘social inclusion’ which is the extent to which access to basic services and markets as well as political, social, and cultural spaces are enabled in order for participation in society with agency and dignity (Cuesta Leiva 2022). To measure the four (social inclusion, social resilience, social cohesion, and process legitimacy) components of ‘Social Sustainability’ they cite, (1) for ‘social inclusion’, labor participation, financial inclusion, public service accessibility, and political participation; (2) for ‘social resilience’, the ability to save, income diversity, shock-responsive assets, food security and migratory mobility; (3) for ‘social cohesion’, interpersonal and institutional trust, violence levels, and democratic and civic participation, and; (4) for ‘process legitimacy’, rule of law measurements, corruption levels, separation of power, objectivity and equality (including accessibility) of law/regulation application, and physical security.

4 See analytical approaches employed by the World Bank: https://www.worldbank.org/en/topic/governance/brief/justice-rights-and-public-safety.

> 5 The current version of MCBT, with crime-related data as examples, can be found at:

> <u>https://dmm.anu.edu.au/CD-AV/. New examples are regulalry uploaded to demonstrate the capability of the tool</u> to be adopted in different contexts. For access, please contact <u>matthew.manning@anu.edu.au.</u>

4
support and economic analysis. The tools can capture heterogeneity across social groups informing justice reform investment decisions that best manage and mitigate social group specific grievances while maximizing economic consequence. The ‘Smart CBT’, once fully developed, will move beyond orthodox techniques by developing a database system that securely stores and de-identifies project data. That data is then redeployed using a range of machine learning and data science techniques.

In the context of justice reform investment, the Smart CBT, coupled with the proposed addition to the tool outlined in this paper, will be enhanced with the inclusion of new features: (i) analysis identifying methods of reform; (ii) policy adjustment; and/or (iii) capacity enhancement that optimizes the economic benefits to a society as a whole.

The first challenge is to what extent the enhanced tool discussed above selects the most efficient option/s available while accounting for heterogenous treatment effects? Traditionally, CBA focuses on average benefits (e.g. average treatment effects (ATE)) and average costs. These metrics are not always adequately disaggregated among groups/communities, undermining scope for evaluation of interventions’ effects on horizontal inequalities (social inclusion). Heterogeneous treatment effects and/or costs are most often present but unaccounted for, potentially diminishing legitimacy, particularly for social groups that experience disproportionaly negative effects. The retention of big data enables greater capacity to undertake more ‘comprehensive’, rather than ‘narrow’ CBA. This means that efficiency potential is hidden in both the efficacy and implementation of public and private programs, if policies can be targeted at those who, net of costs, benefit the most and/or are most vulnerable. Here, we speak in terms of disproportionate social group access to public goods such as health, electricity, justice or other services (United Nations and World Bank, 2018). However, the primary contribution enabled by this work is the potential for advanced CBA to support ex-ante policy and programmatic analysis alongside orthodox Development Impact Evaluations and/or Poverty and Social Impact Analyses.

This paper explores the utility of the CBA APP for a justice sector case. However, contextual factors identified in this or other smart CBAs could be more readily drawn upon to test other interventions across the policy spectrum, including for heterogenous impact. The development of such capacity, it is hypothesized, would significantly reduce human and material cost obstacles associated with repeated individual CBAs for different justice processes or other public policy areas. Re-using the CBA APP would re-vet contextual data representativieness alongside enabling more efficient CBA across the public policy spectrum.

5
In the following sections we consider the state of the literature and practice of costbenefit analysis before introducing a systemtic method for weighing costs and benefits, including with disaggregation to different social groups. We then introduce the case and method for testing this approach, including in identifying heterogeneous distribution of costs and benefits. We then introduce an AI-driven component that reintegrates individual CBA projects, enhancing, we argue, CBA, development outcomes, and policy making efficiency for optimal prioritization of criminal justice resources. We discuss our results and the potential for public policy application to better illuminate social group specific costs and benefits, and identify optimally inclusive and just policy settings.

# **1. Cost-Benefit Analysis Literature**

The literature outlining existing knowledge and data gaps that have historically constrained reliable cost-benefit analysis in the justice field is scattered. We recognize that the literature on justice sector-specific CBA is developing, with much scope for advance as the data illuminating human behavior becomes more representative, reliable, and accessible. Amartya Sen notes the historically narrow nature of analysis solely through market values, which omit social choices that enable freedom of valuation and increased informational inputs (Sen, 2000). Harley et. al define narrow CBAs, which they observe as more commonly employed, as comprising “direct tangible benefits and costs” resulting from the programmatic or policy change (Harley et al., 2019).<sup>6</sup> Harley et. al define ‘comprehensive’ CBA as accounting for direct tangible benefits and costs plus “a more extensive accounting of the indirect economic benefits to all those affected”, including “benefits to individuals, the justice system, the economy and society” (Harley et al., 2019). Diverse datasets that capture heterogeneity, when drawn upon via modern artificial intelligence techniques, empower data management systems and development practitioners to deliver more comprehensive CBAs.

Development actors play an important role in identifying optimal methods for justifying and prioritizing scarce justice sector resources that benefit society at large while accounting for disproportionate costs and benefits (both objective and subjective in nature). Importantly, this role must also fully consider social implications including intangible costs and benefits. The World Bank is enhancing and upholding a high level of transparency and integrity with regards to safeguarding investment that is efficient and transparent. CBA tools

> 6 Their consideration of CBA is specific to Legal Aid.

6
have an important role to play in advancing the prevention of integrity crimes. Approaches, reforms, and enforcement capacity relating to improper conduct such as improprieties surrounding tendering processes (particularly public sector contracts), bribery, and financial sector integrity (Financial Action Task Force (on Money Laundering) (FATF) compliance), for example, may best be prioritized via objective observation of costs and benefits.

# **_Engaging the economics of access to justice_**

Like policy settings relating to FATF, a CBT may also be employed to identify broader justice sector policy cost-benefit implications. Evidence regarding human decision making in developing country contexts is scarce (World Bank, 2017). The World Bank observes three principals informing understanding of human cognition in these contexts. Firstly, that most judgments and choices are made automatically, not deliberatively. Second, people often “think socially – they act and think dependent on what others around them do and think. Third, individuals share a common perspective on making sense of the world based on persons in the society around them” (World Bank, 2017). Banerjee and Duflo (2011) identify the constrained capacity of the poor to take a longer view of their lives because of the extra mental energy used just to address the most basic of needs, like shelter, within a heirachy of needs (McLeod, 2007). Looking beyond a heirachy of needs and relying on randomized control trials in 18 countries, Banerjee and Duflo note that the constrained mental energy of the poor limits their ability to even envisage a future. Failure in criminal or civil justice processes exaggerates exclusion by increasing obstacles to low-level needs like shelter. The opportunity cost for disproportionately time-poor households of pursuing justice outcomes exaggerates barriers to accessing justice processes and just outcomes (Vickery, 1977). Inability to access justice processes also exaggerates other inequalities that render the poor time-poor. The wealthy enjoy chlorinated water, regular and directly bank-deposited salaries often with automatically deducted pension and health care contributions. The poor expend time and scarce resources to organize and engage justice processes to obtain these items (Banerjee and Duflo, 2011). A CBT needs to understand and respond to the logic of how poor people cope individually and collectively in specific contexts. Existing measures of access to justice (e.g. the World Justice Project Rule of Law Index) (The World Justice Project, 2019) alongside data collected by governments and development actors, may be incorporated into a CBT. Using CBA tools that incorporate heterogenous outcomes is important in order to determine how resources directed to eliminate barriers to justice can be efficiently and effectively allocated.

7
Data in fragile and conflict-affected states is scarce. However, legal aid programs enjoy particular success in situations where political space for other forms of justice sector support is low. A review of 199 legal aid studies found only ten that considered legal aid programs in ‘not free’ countries, indicating the scarcity of data and understanding of legal aid programs under repressive regimes (Goodwin and Maru, 2017). Scarce data demands further research in relation to legal aid programs’ medium and long-term effects, their sustainability, their development effect beyond the immediate legal need (including changes in agency), and the use of accountability mechanisms (including human rights commissions, ombuds offices, and public interest litigation) (Goodwin and Maru, 2017). We know little about legal empowerment, despite the multitude of forms it might take. Scarce knowledge surrounding the environments where legal aid programs are successful (at national or community level) demands research that codes according to demography, social norms, power differentials and other political circumstances (Goodwin and Maru, 2017). It also requires consideration of programmatically specific stakeholders and risks across the many forms of legal aid programming relating to many interactions between citizens and formal justice systems, informal justice systems, and formal and informal administrative processes.

The Commission on Legal Empowerment for the Poor found that “...four billion people around the world are robbed of the chance to better their lives and climb out of poverty, because they are excluded from the rule of law” (Albright and de Soto, 2008). We know that there is a link between the strength of the Rule of Law and levels of violence. The cost of violence, is rarely provided as justification for Rule of Law programming. Similarly, the specific economic benefits of empowerment programs are rarely calculated, let alone cited as justification for government interest in their support. Governments (and non-state actors), particularly those that have fought to capture the state, may perceive bottom-up Rule of Law programming as undermining their control over resource distribution, revenue collection and other rule of law-related levers of power (Hackmann, 2012). Persuading governments that bottom-up Rule of Law programming drives growth for all, elevates normative pressure while also appealing to realist self-interest. Increases in income among the bottom 40 percent alongside increases in equality of opportunity has a greater positive impact on growth than that of other income quintiles (Narayan et al., 2013). Inequality drives lower growth and diminished market efficiency, to a significant extent as a result of diminished equality of opportunity for its most productive asset – its people (Stiglitz, 2012).

8
For development, modeling informs us that extreme poverty will lower to 3 percent by 2030 rather than between 6.7 and 4.7 percent if the bottom 40 percent’s income increases 2 percent faster than a country’s mean (Lakner et al., 2014). For development practitioners and for governments, these findings are critical to our approach to policy planning, and our selection, design and implementation of interventions.

We know that the ‘bottom 40’ disproportionately spend their income, and thereby increase the velocity of money in an economy causing an increased multiplier of secondary effects, including increases in government’s tax revenue (Kahn, 1931; Rivas, 2003). We know that growth, disproportionately experienced by the top quintile, conversely, also correlates to capital flight constraining growth’s multiplier effect and deepening inequality (Collier, 2006). Increased spending by the ‘bottom 40’ also increases aggregate demand in the economy for goods and services provided by the upper income quintiles. Identifying multiplier effects, because of their secondary nature, are less specific to the nature of the supported justice reform. Additional economic activity, more income, and more spending in line with a community’s size and spending patterns, is driven, in a self-reinforcing cycle by surplus income or savings, particularly at the individual and household level (Cavallari et al., 2014). Legal aid CBAs more commonly focus on tangible benefits and costs directly from legal aid provision. However, specific legal aid intervention CBAs more commonly incorporate external (health and quality of life) effects (benefits and costs), despite the difficulty in allocating monetary value to those effects. Intangible benefits such as sense of empowerment, enhanced social clout or increased trust in (and engagement of) government are rarely incorporated. Development of robust Information Management Systems and utility of artificial intelligence techniques better enable their incorporation.

An important question to consider in the context of the the aforementioned social and economic patterns regarding justice reinvestment is, for example, “Can we determine or estimate the average cost per-day to an economy of a social harm or good, such as pre-trial detention, including on a government’s tax base?” Making such an estimate requires considering the cost to the person detained, to their employer, to their dependents, and/or to the persons on whom they depend (who may have to undertake unproductive activity such as time off from work to visit the place of detention). Incarceration holds societal costs in that it lowers rates of educational attainment, increases unemployment, weakens community bonds, and increases adult criminality (Thornberry, 1987; Sampson and Laub, 1997; Thornberry and Krohn, 2001).

9
Estimating an average cost per day of pre-trial detention would allow us to use the total number of days of pretrial detention to estimate the total loss of income and total loss of tax revenue. The first step is to collect data on pre-trial detention and its consequences (Sarkin, 2009). We might also hypothesize that this loss is of greater effect because of the increased likelihood of expenditure by those disproportionately held in pre-trial detention – those in the ‘bottom 40’. This approach would exclude consideration of inequality-informing variables that pre-trial detention affects: the skills, training and education of the workforce to which detained persons are less exposed (Piketty, 2014). It also excludes the potential relationship to violence of the discontent that is often associated with detention (Thurston and Lebovich, 2013; Dufka, 2016).

The logic of economics, however, does not always inform political decisions about Rule of Law programming, including efforts to drive down average time spent in pre-trial detention. Political actors experience a diversity of pressures, including incentives to harm the economy of a particular territory, for example a region where perceived adversary groups reside (Toft, 2005; Carment et al., 2009). In attempting to persuade governments of the merits of economic interdependence among arguably competing domestic groups, international development actors, employing analytical approaches, should identify and sensitize key commercial stakeholders linked to policy makers.

An important part of the economic rationale for enhancing criminal justice efficacy is to build a nation’s wealth, as well as to stimulate/increase its GDP growth rate. The World Bank has quantitatively identified that equitable advancement in state justice systems also improves economic development. To drive economic development, a 1 percent increase in rule of law institutions’ value will increase intangible capital by 83 percent (Hamilton, 2006). Education and justice system investments, therefore, “...are the most important means of increasing the intangible components of total wealth” (Hamilton, 2006). For low-income countries, therefore, identifying optimally sustainable growth includes identification of policy orientation that maximizes deployment of rents from land and natural resources into physical and human capital (Hartwick, 1977).

As violence tends to affect population subgroups disproportionately, it is important for an economic justification to be informed by how people interact under conditions of socio-economic hardship.<sup>7</sup> Scarce econometric explanations are available of social

> 7 See the following sections on interventions and their political economy.

10
interactions, especially amongst persons in low socio-economic environments; in particular in identifying the level of territorial control exercised by state and non-state actors (Gangopadhyay et al., 2014). For the purposes of determining the normative and economic utility of criminal justice interventions, an inadequate understanding of social conformity and deviance within peer groups diminishes our capacity to identify environments most receptive to criminal justice interventions and where those interventions will have the greatest impact (on violence and socio-economic development) (Gangopadhyay et al., 2014).

# **_Weighing risk of intervention capture or loss of community support_**

Development actors aspire to comprehensive data regarding the direct and indirect functional and developmental effect of programs. To consider law enforcement effectiveness, we must turn to our knowledge of what form of deterrence is most effective. Young males, for example, respond to certainty of punishment rather than severity (Cooter and Ulen, 2008). Cooter and Ulen cite Lee and McCrary’s (Lee and McCrary, 2005) observation that the certainty of punishment, not severity of sentence, may act as a greater deterrence mechanism; therefore, informing greater investment in police presence and detection capacity than prisons. The risk that support for criminal justice and security sector reform is captured by a government or non-government actors is informed by the inclusivity of stakeholder engagement and our capacity to foster inclusivity in reforms themselves (Hellman et al., 2000). Where the presence of security actors is increased, trust between the police and the ‘policed’ – including perceptions of police legitimacy – is critical. Ensuring demographic inclusivity and demographic reflection of policed communities in deployed personnel, as well as connection between the police, community leadership and trading communities has also been found to enhance community support (Baker and Scheye, 2007).

Conversely, low public perceptions of police integrity and legitimacy drive poor civic engagement with the police and increased criminality (Cooter and Ulen, 2008). Deterrence, therefore, is not only dependent upon police presence but also upon “...the support of good people to help the police [i.e third party involvement] and other legal officials” (Cooter and Ulen, 2008). The personal sacrifice of time, effort, convenience or safety by ‘cooperating people’ increases police effectiveness and certainty of deterrence (Cooter and Ulen, 2008). The cost per person illuminates the circular causal chain of an increasing number of people diminishing each individual’s cost of assisting, and in turn increasing willingness to assist among the community (because of group pressure and lower perceived cost) (Cooter and Ulen, 2008). These issues focus our attention on the dynamic of inter-personal trust, requiring 11
that we draw on not just psychology, but also from sociology and anthropology to determine a situation’s civic trust in authority (Hoff and Stiglitz, 2016). Alternatively, diminished trust in the legitimacy or integrity of the enforcing entity may dissuade public assistance, deepening both that entity’s inability to confront violence as well as public reluctance to assist (Rossmo, 2018) . The emergence of online language metadata enables scope to measure sentiment about related topics, the circumstances that affect them, and how sentiment about topics affects both sentiment about law enforcement and incidents of violence. Measuring the relationship of online language sentiment about violence-relevant topics has already been shown to not only relate to change in violence, but to also enable significant accuracy in predicting its change (Mahony et al., 2019).

# **_Contexts where state control of territory is contested_**

In the future, capacity may emerge to forecast when, for example, criminal networks or the state are consolidating or ceding control over communities to one another and how strong a shift in control over territory might be.<sup>8</sup> Emerging data sources, such as geolocated online language data may assist in indicating change in phenomena such as which entity exercises effective authority over specific territory.

Once we have identified the mechanisms associated with decline in criminal conduct, we can use available data to identify the most efficient use of resources to advance that decline. For example, because more _certain_ (not longer) punishment is more heavily associated with decline in youth violent conduct than severity of sentence, redirecting money away from incarceration and prisons and toward (evidentially-informed) community policing constitutes a more efficient allocation of scarce resources to reduce youth violence (Lee and McCrary, 2005). This is known in the criminological literature as justice reinvestment. Using data procured from existing World Bank and other justice sector analytical products may assist in maximizing identification of the specific elements of community policing capacity that demand prioritized support. Identifying armed groups’ deterrence or incentives when designing justice interventions may be even more complex. To consider prioritization of policy in such circumstances is to consider that, firstly, criminal organizations’ control of

> 8 Our understanding of local context informs the degree of control over local civic life (criminal consolidation) and the nature of relations between non-state actors and the state (proximity between armed actors and the state). Where civic groups play important roles in mediating relations with the state, an opening to employ those groups as sensitizing agents exists. Where closer relations between the state and the armed actors exist, the state should take the lead in sensitizing the armed actors as to the intended focus of punitive state action. See: Enrique Desmond Arias. (2013). The impacts of differential armed dominance of politics in Rio de Janeiro, Brazil. Studies in Comparative International Development. 48:3, 263-284, p. 265.

12
urban spaces reflects rule of law’s breakdown as well as variant emerging micro- and locallevel political and security orders and dispute resolution processes. Secondly, localized orders are embedded in, stem from, and affect wider political and economic institutions and practices central to development. Thirdly, variance in local armed dominance enables and constrains reforms and stakeholder mobilization behind them (Lee and McCrary, 2005). Prioritizing, sequentially, the economic security and peace building incentives of vulnerable armed actors, dilutes the risk of their remobilization and enhances prospects for consolidating sustainable peace and the accompanying benefits (rather than costs).

Considering the knowledge and gaps outlined above, we: (i) outline a method for capturing heterogeneity based on earlier empirical studies undertaken on this issue; and (ii) demonstrate the method and enhanced APP using primary data from a study conducted in 2006 on the costs and outcomes associated with a developmental crime prevention intervention undertaken in Australia; (iii) apply our method to develop CBA outcomes that incorporate the heterogenous distribution across intervention subgroups; (iv) discuss how the method we developed in this paper is incorporated into the enhanced APP; (v) discuss future developments of the enhanced APP that will incorporate modern data science techniques. Overall, the method we describe, test, and apply to the enhanced APP can capture the complexity of prevention and will assist policy makers in producing evidence sensitive to contextual social group-specific variation.

# **2. A Systematic method for weighing costs and benefits**

# **_Accounting for heterogeneity_**

The main purpose of CBA is to inform policy makers on the economic viability of publicly funded programs. As briefly highlighted above, the benefits identified and measured are based on parameter estimates of the average effect of a given program or intervention, which are then compared to the associated average costs (Boardman et al., 2017). However, substantial heterogeneity often exists in the benefits and/or costs across individuals and groups. As such, the traditional CBA approach may hide valuable information about which social groups benefit the most net of costs.

The Kaldor–Hicks efficiency criterion is commonly applied in CBA and states that a publicly funded program can be justified when the overall benefits outweigh the costs. This criterion is in contrary to the earlier Pareto criterion. It requires that no one will suffer from any change under the proposed intervention conditions. Theoretically, a program under the 13
Kaldor-Hicks efficiency criterion can be defended even if it produces undesirable or negative outcomes for some individuals or groups, so long as the overall benefits are greater than the overall costs to society (Manning, 2008) . However, we propose that the conceptual foundation of CBA (Kaldor-Hicks Criterion) does not adequately address the issue of heterogeneity in program impacts. We argue that an important consequence of accounting for heterogeneity is that international development assistance-driven justice sector gains, net of costs, disproportionately target those in extreme poverty and those in the bottom 40% of the population.

# **_Average and quartile treatment effects_**

The availability of data now makes obtaining average treatment effect (ATE) easier, particularly when random control trials (RCT) are employed. In short, the average treatment of the treated group (ATT) equals the ATE (for the population). The RCT lends itself to a simple nonparametric analysis of the treatment effect, given by 1 0 where, 1 is, for example, the average number of episodes of political violence (using the ATT = ATE = YY<sup>�</sup> −YY<sup>�</sup> Armed Conflict Location and Event Database (ACLED)) for the experimental group, while YY<sup>�</sup> 0 is the average number of episodes of political violence for the control group. The average effect of the intervention of interest is simply the difference between the two averages. As YY<sup>�</sup> stated above, however, the shortcoming of ATE is our lack of understanding of the distribution of effects. In addition, the mechanisms that make the prevention of conflict (economically) successful.

It is inherently difficult to measure the policy impact for each individual, since, by definition, the counterfactual for each individual under the policy may be unobserved depending on the possibility for appropriate matching of experimental and control groups. Often, even when these groups are matched, the heterogeneity of policy impact is buried in the average effect. We can, however, disaggregate the population based on certain characteristics or key factors and compute the distribution of outcomes within the population (Heckman et al., 1997). A sign of heterogeneity is revealed by an uneven distance across individuals. Although we cannot fully appreciate the distribution of effects, we can nevertheless gain useful knowledge about whether there exist heterogeneous effects across groups in the population.

14
It is well understood, as discussed above, that group differences within a community can be observed among a range of static and dynamic factors. But unless we can observe these differences, we have to rely on the ATE of our policy, which does not fully appreciate the unique costs and benefits felt by each group. For example, a defensive policy aimed at reducing domestic terrorism and violence may have significant distributional costs and benefits across the population and, therefore, relying on the ATE could overestimate the effects on some in the population but underestimate the effects on others. So, if one of our variables of interest was social dominance orientation (SDO) (intellectualized as an individual’s preference for inter-group hierarchies within a social system or group-based discrimination), we must capture the levels of SDO that exist across our population. We note here that individuals with high SDO are more likely to support public policies that promote or recreate social hierarchies, while people with low SDO favor more equality-based policies (Pratto et al., 1994). Of course, there are many other variables that we can use to study the heterogenous effects on such a policy, which allow us to separate individuals and groups within the population that are affected by the policy and thus allowing for a comprehensive understanding of the economic implications of our policy decisions. The question now is, how can we disaggregate the population according to the observed and sometimes unobserved differences?

# **_Quantile treatment effects_**

There exists a number of methods to capture heterogeneous effects. To capture such variation we outline, in what is a standard approach, the method used by Kristensen et al (Kristensen et al., 2017). The authors apply a method that, like us, attempts to monetize the economic benefits associated with an intervention. In addition, the authors clearly outline how they overcome the shortfalls of traditional CBAs, which focus exclusively on ATEs. In this study the authors employ a randomized control-trial experiment to examine the heterogenous impact of the Danish return-to-work program. The authors successfully demonstrate: (i) the benefits of estimating quantile treatment effects (QTE) (a well-established technique used widely in the social sciences and also included in a range of statistical programs<sup>9</sup> ); and (ii) the use of the efficiency potential (what we call potential efficiency gain) to assess the net benefits resulting from the reallocation of resources sensitive to the QTE. Kristensen and colleagues, based on their results, propose a screening system to triage participants in order to

> 9 See for example,  Frölich, M. and Melly, B. (2010). Estimation of quantile treatment effects with Stata. Stata Journal. 3:10, 423-357.

15
create effeciencies and ultimately maximize overall program effects by taking into account subgroup differences.

The aforementioned static and dynamic factors, which affect treatment outcome, can be measured either categorically or continuously. Categorical divisions of the population could be based on some naturally formed social groups (e.g. ethnicity and gender). The impact of a policy may be moderated by these factors where the policy may unevenly affect different groups within the population. For continuous latent variables (e.g. level of SDO), which are measured along a continuum, we could use confirmatory factor analysis (CFA) to study and describe the relationships between a set of observed variables (e.g. based on Likert scale items) and the specific set of continuous latent variables of concern (Fox, 2010; van der Linden, 2016). This is especially helpful when no pre-established indexes are available. This allows us to create a combined measure capturing the level or degree of an unobserved latent characteristic or factor. For example, when there are no reliable indicators that accurately classify individuals in the community with respect to level of SDO, an evaluation team may decide to incorporate certain assessment criteria (measured continuously) and assess the uniformity of these criteria in measuring SDO levels (the unobserved latent variable). Following Bitler et al. (2006), consider two distributions where we have our experimental and control groups, respectively, _F_ 1 and _F_ 0, and define quantile treatment effects (QTE) as ∆ (0), where is the _q_ th quantile of distribution _Ft_ . qq qq qq qq **_Accounting for heterogeneity_** = yy (1) − yy yy (tt) Policy outcome differences within the distribution of the continuous latent variable will indicate whether the outcome effects are stronger for some individuals than for others. Figure 1 shows the cumulative distribution for hypothetical treatment and control groups in an RCT of a given policy.

16
**Figure 1. Distribution of number of violent incidents, treatment and control groups**

<!-- Start of picture text --> 60<br>50<br>40<br>30<br>20<br>10<br>0<br>0.5 0.8 1.1 1.4 1.8 2.2 2.5 3.0 3.3 3.7<br>Average daily income (USD)<br>Treated Control<br>Reduced number of violent incidents<br><!-- End of picture text --> The horizontal difference in the distributions indicates the impact of the policy, and this difference corresponds to the QTE shown in Figure 2. A horizontal line for the ATE is presented in Figure 2, which indicates no difference across the continuum. Figure 2 clearly indicates that the hypothetical policy has varying impacts across the distribution. Given the positive relationship between the policy impact and the latent continuous variable, positive effects are found from about the 10th percentile onward. The comparison between ATE and QTE, shown in Figure 2, reveals substantial heterogeneity in the treatment effect.

Specifically, violence prevention programs for domestic violence (our hypothetical example) are found to be more effective among households with higher income.

17
**Figure 2. Average Quantile Treatment Effects**

<!-- Start of picture text --> 50<br>40<br>30<br>20<br>10<br>0<br>0 10 20 30 40 50 60 70 80 90<br>-10<br>Percentile (Average daily income)<br>QTE Lower limit Upper limit ATE<br>Reduced number of violent incidents<br><!-- End of picture text --> If we were able to perfectly screen individuals and only allocate treatment to individuals with a treatment effect above the average cost level, (in the absence of individualized cost data), that is individuals above the break-even line shown in Figure 3, then, we would be able to improve the overall net benefit of the policy. In this example, there is room to improve the net benefit of the policy for the population concerned. We define this as the potential efficiency gain (PEG). That is, the increased benefits net of costs that would accrue to society, if it is possible to perfectly screen individuals to those who benefit at least as much as the costs involved.

PEG is given by: where ‘BC maximized’ refers to the scenario where the benefit-cost ratio is maximized due to a selective application of treatment/policy (i.e. treatment only applied to those individuals with a positive net benefit), and ‘BC baseline’ refers to the baseline where the treatment/policy has been adopted to everyone as a universal treatment option. The reader should note that there may be heterogeneity in benefits across individuals not captured by the quantile methods employed – this may be a result of variation of policy impact based on other unobserved moderators in the population. Hence, the PEG is not a maximum benefit 18
<!-- Start of picture text --> : | y<br>a!y<br>>10 —_—<br>ee |<br>-10 !<br>~ i 79 100<br>Quantile<br>cl 95% -------- Benefit<br><!-- End of picture text -->
Given that perceptions of exclusion from access to justice are associated with increased risk of violence, ‘exclusion’ may constitute a covariate that moderates policy impact. The World Bank, for example, recently concluded the establishment of the WEI, which may be employed as data indicating selectivity of the provisional benefits (and costs). The WEI represent, across 181 countries from 1900-2018, denial of individuals’ access to services or participation in governed spaces based on their identity or belonging to a particular group. The indicators measure exclusion across four dimensions: exclusion from civil liberties; exclusion from access to public services; exclusion from access to state jobs; and exclusion from access to state business opportunities. Exclusion in these dimensions is grouped by five indices: socio-economic group; by gender; by rural/urban location; by political group; and by social group (including caste, ethnicity, language, race, region, religion, migration status, or some combination thereof). The WEI, therefore allow for exclusion, for example by gender, within a country or across a region, for example the Middle East across the four dimensions (Pillai, 2019). Of particular relevance to justice processes are the dimensions of exclusion of social groups in their access to civil liberties and the state service of justice. However, the efficacy of equal access to other state services, to state jobs, and state business opportunities may inform, in particular, identification of social group-specific exclusion in CBA modeling of justice processes to resolve equal access in those dimensions.

A further covariate example that may moderate policy impact may be ‘governance’. The World Bank’s WGI reports aggregate and individual governance indicators for over 200 countries and territories from 1996, for six governance dimensions: Voice and Accountability; Political Stability and Absence of Violence; Government Effectiveness; Regulatory Quality; Rule of Law; and Control of Corruption. These aggregate indicators, based on over 30 individual data sources, combine the views of a large number of enterprise, citizen, and expert survey respondents.

As data emerges specific to _how_ different justice process exclude social groups, frameworks can be developed and vetted to inform what constitutes optimal social (such as inclusion) and economic net benefits. The United Nations and World Bank, on this issue specific to processes of accountability for atrocity crimes, note: “Weighing the equality of accountability processes against the imperative to bring perpetrators to book is critical to the challenge of advancing stabilization and justice in conflict-affected environments under SDG 16. Accountability processes may exacerbate 20
grievances related to specific social groups if they are perceived to discriminate between groups. How and why the real or perceived unequal treatment of social groups actually occurs varies from one process to another. Frameworks to identify how accountability processes treat groups differently can help to identify ways in which to preempt spoilers and mitigate risks of conflict.” (Mahony, 2016; United Nations and World Bank, 2018)<sup>10</sup>

# **3. Method**

In this paper we use the enhanced CBA APP to process the cost and benefit data. We adopt the QTE method as described above. The enhanced APP proposes to overcome the shortcomings of existing cost-benefit tools (e.g. the Manning Cost-Benefit tool versions 1 and 2 (Manning et al., 2016; Manning and Wong, 2016a; Manning and Wong, 2016b)), with the inclusion of the heterogeneity component which allows us to: (i) identify variation or differences between groups or populations (e.g. individual actor attributes, political structures, institutions); (ii) identify how justice processes differ across groups/populations; and (iii) distribute costs and benefits according to heterogenous impact.

Figure 4 illustrates the enhanced CBA APP. In this study, we demonstrate three of the six interacting modules (Modules 1, 2 and 3). A full discussion of the six modules included in the enhanced CBA APP is provided by Manning et al. (2018).

> 10 The UN and World Bank cite a framework that breaks down the jurisdictional and functional elements of criminal justice processes that advance or undermine equality before the law.

21
<!-- Start of picture text --> Module 1 Module 2 Module 3<br>@u sersC =» Input Heterogeneous CB Output of<br>G,( )\ =» cost-benefit, effects =» calculations => .CB analysis:.<br>(| (CB) data Relationship (incl. graphics)<br>Module 4 y Linked open data 7 y Module 5<br>— —_—- Geolocation <a<br>User input. = - Population > aN Calculated<br>database - Indexes CBA result<br>ane database<br>Machine learning<br>User input support Improved CB analysis<br>- Data imputation - Feature selection Module 6<br>- Anomaly detection - Feature engineering<br>- Correlation detection<br><!-- End of picture text -->
affected social groups (e.g. beneficiaries) if the moderator is categorical in nature (e.g. SES). If the moderator is continuous in nature, then the evaluator can employ a range of methods, for example QTE as described above, to generate patterns of benefits across the distribution of individuals or groups along the continuum.

In module 3, the enhanced CBA APP provides the outputs of the economic analysis. In summary, the APP allows for cost-feasibility analysis (i.e. comparing the overall costs of the project against the budget), cost-savings analysis (i.e. comparing the costs of the project against the savings generated from avoided crimes), cost-effectiveness analysis (i.e. comparing the costs of the project against the number of units of output such as the number of crimes prevented) and CBA (i.e. comparing the costs of the project against the overall benefits – avoided crimes and other benefits such as enhanced safety). Tables and plots are also included to display, for example, net costs, net benefits, cost-benefit ratio, net benefits by bearers and potential efficiency gain (PEG - as described above).

# **_Data employed to test modules 1 to 3 using the enhanced CBA APP_**

We employ data from the Pathways to Prevention Project (Homel et al., 2006) to test modules 1 to 3 of the enhanced CBA APP. The Pathways to Prevention project was established over twenty years ago as an early intervention, developmental crime prevention initiative focussed on the transition to school in one of the most disadvantaged urban areas in Queensland, Australia (Homel et al., 2001). Here, the goal was to provide positive pathways to individuals at-risk of later learning and behavioral problems that may eventually lead to crime and deviance. The project comprised two components: (i) a preschool intervention program (PIP), which was a child-focused and school-based set of activities; and (ii) a family independence program (FIP), which was a family-focused and community based set of activities run by Mission Australia (Manning et al., 2006).

In this paper, we focus exclusively on the PIP component. PIP aimed to enhance the participant’s readiness to succeed at school. Through a sequence of structured small-group interactions with either specialist teachers or program staff, PIP aimed to develop a participants: (i) communication skills by introducing more abstract language, complex vocabulary and appropriate grammar formats as part of the preschool experience, and (ii) social skills to improve a participants ability to better interpret social interactions, overcome emotions that are unproductive (e.g. anger and anxiety) and develop strategies for dealing with problems that often occur during exchanges with peers (Manning et al., 2006).

23
To undertake our CBA, we use cost data derived from Manning et al. (2006) and outcome data collected from the chief investigator of the project, Professor Ross Homel. Project costs were estimated separately for three distinct stages: development, implementation, and evaluation. The cost analysis method employed and cost estimates are described in the study undertaken by Manning and colleagues (2006). Part of our CBA also requires us to estimate potential costs of a negative outcome – in this case the cost of dealing with behavioral problems – or the avoided costs if the intervention produces outcomes that reduce the probability of a child needing future behavioral management intervention.

Manning and colleagues (2006) estimated the costs of four behavioral management alternatives (grouped into two categories) that may be required if a child presents with behavioral problems in the early years of education. Category 1 consists of programs developed to help improve the behavior of children with borderline or less challenging behavioral problems, while Category 2 consists of programs aimed at helping those children with more severe or extreme behavioral problems (see Table 1). Homel and colleagues (2015) measured difficult and challenging behavior using the Rowe Behavior Rating Inventory (RBRI). The RBRI is a validated teacher checklist used to assess the level of children’s difficult behavior (Rowe and Rowe, 1995). In this paper we classify children who received a score of 20 to 29 as displaying a low level of poor behavior and thus only requiring little assistance (i.e. Alternative 1). We coded children receiving a score greater than 30 but less than 40 as requiring more assistance (i.e. Alternative 2). Children who received a score greater than 40 were identified as having more severe or extreme behavioral problems and fell into Category 2 type programs (RBRI 40 - 49 – Alternative 3; RBRI ≥50 – Alternative 4).

24
**Table 1: Behavioral management programs**

|**Category 1**||
|---|---|
|Alternative 1|Pathways Communication Program (RBRI<br>score 20-29)|
|Alternative 2|School district behavioral management<br>teams (Inala Cluster)- known as Behavior<br>Support Team, Corinda District (RBRI 30-<br>39)|
|Category 2<br>||
|Alternative 3|Pathways Social Skills Program (RBRI 40-<br>49)|
|Alternative 4|Behavioral School (Tennyson Special<br>School) (RBRI 50+)|

The costs of the alternative programs estimated by Manning and colleagues (2006) are presented in Table 2.

**Table 2. Cost of alternative behavioral management alternatives**

|**Intervention**|**Budget Cost**|**No. of Participants**|**Per participant**<br>**cost**|
|---|---|---|---|
|Alternative 1|$47,861.41|125|$382.89|
|Alternative 2|$236,312.93|145|$1,629.74|
|Alternative 3|$13,999.93|100|$139.99|
|Alternative 4|$417,460.32|21|$19,879.06|

# **_Measuring heterogenous impact of PIP_**

In terms of our analysis of potential heterogenous outcomes, we use scores derived from the same data used by Homel and colleagues (2015). As described in brief above, RBRI forms a total measure of behavioral adjustment ranging from 12 (positive adjustment) to 60 (poor adjustment). In the longitudinal study undertaken by Homel and colleagues (2015) there were five measures of behavioral adjustment across the reference period, one for each academic year. Our model specifies behavioral adjustment for the fifth assessment period as the dependent variable, behavioral adjustment for the first baseline period as a control, and the 25
exposure to PIP as an independent variable. The model also analyzes the heterogenous impact of PIP on behavioral adjustment through a series of regression analyses according to the number of siblings of the program participants (i.e. our diversity variable). We chose number of siblings as a diversity variable as it has been shown that children that have many siblings tend to display an increased odds of adverse developmental outcomes (de la Rochebrochard and Joshi, 2013).

In practice, then, compliance at Time 1 has been partialled out, leaving only the difference between Time 1 and Time 5 to be explained by the independent variable in the model (Cohen and Cohen, 1983). Using the _rnorm()_ function of the R Studio (R.Studio Team, 2020), the coefficients drawn from these analyses were applied to simulate nine datasets, each with 1,000 samples, with a child having no siblings to 8 siblings. For more details of the simulation please refer to Peng (2022).

# **4. Results**

Our regression results show that PIP led to statistically significant poor behavioral adjusted outcomes for participants with no siblings (β = 10.45, p < .001), 5 siblings (β = 9.10, p < .001), 7 siblings (β = 12.27, p < .001) and 8 siblings (β = 10.83, p < .001) as revealed by the change in RBRI score post intervention. The PIP, however, was estimated to be beneficial to children with 1 (β = -7.00, p < .001) or 2 siblings (β = -2.09, p < .001), but had no statistically significant effect on children with 3 (β = 0.57, p > .1), 4 (β = 2.01, p > .1) or 6 siblings (β = - 0.15, p > .1).

Based on these results, we further estimate the number of individual children who potentially may be triaged into the aforementioned alternative treatment and behavioral management programs (as described in Table 1). The difference in proportion of children with an RBRI score of 20-29 (category 1), 30-39 (category 2), 40-49 (category 3) and 50-60 (category 4) were observed between the intervention group (with PIP) and control group (without PIP). Such a difference in proportion was applied to the actual data collected by Homel et al. (2015), suggesting either a potential reduction or increase in the number of children who may require future behavioral management interventions. The estimated number of children who require additional behavioral management interventions is based on the percentage of children sorted into one of the four RBRI score categories before and after the intervention. This is calculated by applying the percentages derived from the simulated dataset to the actual number of participants in the Homel dataset. We present these results in 26
Table 3 where we show, for example, that the estimated number of children requiring Alternative 1 (i.e. additional behavioral management intervention) with one sibling is 1.92, Alternative 2 (-2.44 (negative sign signifying a reduction in children requiring intervention)), Alternative 3 (-1.44), and Alternative 4 (-1.44).

We then applied the outcome data with the cost data described above into the enhanced CBA APP. Since the provision of PIP did not affect the RBRI score of some of the subgroups (non-statistically significant effect), no benefits could be attributed to those groups. As such, the economic benefits for these groups were not estimated (see Table 3 column 3, row 5). For other groups, however, we found positive and significant benefits (resulting in a positive avoided cost), and also negative and significant benefits (resulting in additional future treatment cost). The results presented in Table 3 show the number of children under each category (i.e. by number of siblings) multiplied by the corresponding per participant costs of the management programs to generate an estimate of the costs, benefits, and net benefits of PIP (i.e. the avoided costs of behavioral management). As discussed above, PIP was most beneficial for those PIP participants that had 1 or 2 siblings. Table 3 reveals the net benefits to PIP participants with 1 or 2 siblings of $ $18,234.39 and $74,049.29, respectively.

**Table 3. Estimated CBA results according to the no. of siblings**

|No. of<br>siblings|Costs<br>(AUD)|Benefits<br>(AUD)|Net benefit<br>(AUD)|Estimated<br>no. of<br>children<br>requiring<br>Alternative<br>1|Estimated<br>no. of<br>children<br>requiring<br>Alternative<br>2|Estimated<br>no. of<br>children<br>requiring<br>Alternative<br>3|Estimated<br>no. of<br>children<br>requiring<br>Alternative<br>4|
|---|---|---|---|---|---|---|---|
|0|$4,611.92|-$20,928.98|-$25,540.90|-2.00|1.20|-1.00|1.00|
|1|$13,835.76|$32,070.15|$18,234.39|1.92|-2.44|-1.44|-1.44|
|2|$2,8824.5|$102,873.79|$74,049.29|-7.42|-9.42|0.74|-4.27|
|3|$15,565.23|$-|-$15,565.23|-|-|-|-|
|4|$10,376.82|$-|-$10,376.82|-|-|-|-|
|5|$6,917.88|-$27,133.43|-$34,051.31|-1.80|5.00|-1.40|1.00|
|6|$6,341.39|$-|-$6,341.39|-|-|-|-|
|7|$1,729.47|-$2,876.68|-$4,606.15|-1.00|2.00|0.00|0.00|
|8|$4,611.92|-$4,646.32|-$9,258.24|-1.00|3.00|1.00|0.00|

The above economic estimates (derived from the enhanced CBA APP) are consistent with the findings of our regression analyses, both in strength and direction, where there were positive estimated benefits of the PIP intervention for children with 1 or 2 siblings and 27
<!-- Start of picture text --> 80k ° —— Relationship<br>' —*— Lower limit<br>60k ' —*— Upper limit<br>' --@- Break-even line<br>=w 40k 'H<br>tdc H<br>a20k °<br>2 ‘<br>2'<br>0<br>—20k<br>e1<br>—40k<br>0 1 2 3 4 5 6 7 8<br><!-- End of picture text -->
outcomes specific to social groups, to context and to specific intervention elements. Manning et al. (2016) provide a detailed discussion on these elements. However, the current version of the enhanced CBA APP is capable of more than what we have presented here. Below we describe three additional modules that are currently in various stages of development, testing and implementation.

Module 4 (as shown in Figure 4) improves upon earlier economic tools, such as the MCBT, by drawing upon and exploiting machine learning (ML) techniques. Here, we incorporate a ‘User Input Database’ module, where all data entered by users into the APP is stored as one set of records per project. The enhanced APP will allow the user to store their input data on their server, providing a single data resource for the ML module (i.e. Module 6). Here, we intend to provide source code to allow users to manage their data on their own server, utilizing their own security protocols. Specifically, since Module 1 identified and established a relationship between moderators and outcome, module 4 then captures those relationships allowing the system to learn from these relationships and identify moderators that are relevant to outcomes concerned. This allows for the creation of a more comprehensive database and the possibility of subsequent ML, as described below in Module 6 (e.g. imputation of missing data).

Linked to Module 4, Module 5 will provide code for users to build a database to store the calculated results after analysis. Storing the calculated results is crucial because it provides the ability to model the relationships between all input-relevant cost and benefit data (Module 1) and the output of CBA (Module 3). Therefore, the database in this module of the workflow diagram stores the benefits (that could also be weighted using a harm index) and analysis data, enabling the system to map the relations between input and output, and exploit this to learn and improve CBA over time.

Finally, Module 6 integrates ML techniques to achieve two main goals, namely: (i) to provide input support to the user by imputing missing values (currently in testing stage), identifying potentially erroneous values, and make suggestions about relevant contextual factors; and (ii) to improve the analytical capabilities of CBA by usefully reducing the number and types of variables to minimize user effort (e.g. time-consuming data entry) and develop better estimates (e.g. cost savings; crimes avoided), based on what the system learns from earlier projects.

29
As with all policy and programmatic interventions, the moral and commonly aspirational arguments for justice interventions must be enhanced by collection of data representing how different intervention designs enhance or undermine the social groups’ equality before and after access to justice. Sucha a granular measure of the equality of social groups under specific justice policy settings could enhance measurement of this component of process legitimacy. Further, the legitimacy of justice interventions can be enhanced by better quantifying and comparing (and communicating) their economic and societal benefits, particularly where this can be done in such a way to demonstrate disproportionate impact for excluded social groups. Justice data collection and analysis, which employs broader societal data, as well as data generated from existing justice sector analytical methods, is a prerequisite for a ‘business case for justice’. An enhanced CBA APP, as described above, serves these objectives by improving the accuracy of justice sector resource allocation for maximum economic and societal outcomes while targeting society’s most vulnerable and excluded social groups.

Traditional CBAs commonly seek to weigh anticipated policy or intervention benefits against a policy reform or an intervention’s cost to determine overall societal benefit. CBAs demand resources and time due to the many challenges of quantifying anticipated costs and benefits, idiosyncratic to the nature of policies and interventions in the diverse complexity of contexts in which they occur (Prescott, 2010). Drawing on multiple sources of societal and justice process data in a systematic way, allows governments and stakeholders more exhaustive identification of interventions’ cost-benefits. It also enables systematic intervention comparison across a range of metrics beyond monetary benefits (such as crimes avoided and level of safety improvement) and to identify which interventions have greatest benefit for the most vulnerable social groups. Future utilization of an enhanced CBA APP, which includes the tested heterogenous impact component and future machine learning capacity, will enable significant contextual factors identified in previous CBAs to be tested in future intervention CBAs.

The enhanced CBA APP confronts the human and material resource problem of repeated individual CBAs across different justice processes. Broader societal data (e.g. geographic, economic, demographic, climate, security) as well as data specific to civil, criminal, and informal justice sector capacity can be retained and employed by the APP across (where appropriate) justice processes. Identified intervention costs are disaggregated by cost bearer (e.g. criminal justice system, local government, business, and society) and 30
intervention outcomes (either positive or negative) may be qualified by moderators, including, via the use of Governance and Exclusion indicators, the social groups that benefit and the societal governance issues perceived as most demanding attention.

The enahnced CBA APP retention and AI-driven vetting of data integrity component will enable re-vetting of data’s representativeness over time and alert researchers/users of the APP to potentially erroneous values (making suggestions about relevant contextual factors), and significant factors for data input narrowing. For development actors considering budget support or programmatic operations, enhanced capacity to determine the costs and benefits of policy and programming, particularly for the most vulnerable, enhances operational credibility, legitimacy, government process ownership, and development outcomes.

We know that a 1 percent increase in rule of law institutions’ value increases intangible capital by 83 percent – the most significant long-term development investment societies can make (Hamilton et al., 2005). We know that since education and justice system investments constitute the most significant components of intangible capital, identifying optimal approaches to enhancing justice processes is critical for economic development (Hamilton et al., 2005). Since rule of law and institutional confidence also constitute critical components of ‘social cohesion’ and ‘process legitimacy’, which are critical to social sustainability (Cuesta et al., 2022), identifying justice policy settings that enhance justice accessibility and associated outcomes also advance social sustainability. The enhanced CBA APP takes the first steps towards enabling government and stakeholder-led use of artificial intelligence to advance our analytical capacity to drive these inclusive justice objectives.

Further, the potential of this methodological approach is more significant when considered across broader public policy and when drawing on emergent data sources and methods. Application of this approach when drawing more fully on data representing components of social sustainability (social inclusion, social resilience, social cohesion, and process legitimacy) may further enhance, via cost-benefit identification, context-specific exivdence of social sustainability. Development and dissemination of evidence identifying optimal policy settings based on weighting of costs and benefits in turn enhances informed public understanding, civic participation, and civic inclination to access justice processes (Mansuri and Rao, 2022; Innes and Booher, 2004).

The results demonstrate significant efficacy of this approach and suggest exploration of its application to other justice and broader public policy processes.

31