metadata: authors: 0: author: Bridges,Kate
1: author: Woolcock,Michael
count: Nigeria
docna: 0: docna: Implementing Adaptive Approaches in Real
            World Scenarios : A Nigeria Case Study, with Lessons for
            Theory and Practice
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Adaptation to Climate Change,Health Service Management and Delivery,Family Planning Research
teratopic: Environment,Industry,Health, Nutrition and Population
abstracts: cdata!: How does adaptive implementation work in
            practice? Drawing on extensive interviews and observations,
            this paper contrasts the ways in which an adaptive component
            of a major health care project was implemented in three
            program and three matched comparison states in Nigeria. The
            paper examines the bases on which claims and counterclaims
            about the effectiveness of these approaches were made by
            different actors, concluding that resolution requires any
            such claims to be grounded in a fit-for-purpose theory of
            change and evaluation strategy. The principles of adaptive
            development may be gaining broad acceptance, but a complex
            array of skills, expectations, political support, empirical
            measures, and administrative structures needs to be deftly
            integrated if demonstrably positive operational results are
            to be obtained, especially when undertaken within
            institutional systems, administrative logics, and political
            imperatives that are predisposed to serve rather different purposes.
geo_reg_key: 517191,1927320
repnme: repnme: Implementing Adaptive Approaches in Real
            World Scenarios : A Nigeria Case Study, with Lessons for
            Theory and Practice
display_title: Implementing Adaptive Approaches in
            Real World Scenarios : A Nigeria Case Study, with Lessons
            for Theory and Practice
listing_relative_url: /research/2019/06/31171432/implementing-adaptive-approaches-real-world-scenarios-nigeria-case-study-lessons-theory-practice
topicv3: Education, Skills Development and Labor
            Market m1326135 1484,Health Workforce m1326573 631,Contraceptives m1327726 1453,Climate Change Adaptation m1327767 574,Health m1504661 3480
publishtoextweb_dt: 2019-06-18T00:00:00Z
historic_topic: Environment,Industry,Health, Nutrition and Population
no_of_pages: 37
author_keys: 0: author_key: 166689
model_extractions: Policy Research Working Paper 8904

# Implementing Adaptive Approaches in Real World Scenarios

A Nigeria Case Study, with Lessons for Theory and Practice _Kate Bridges Michael Woolcock_ Development Economics Development Research Group June 2019
Policy Research Working Paper 8904

# **Abstract**

How does adaptive implementation work in practice? Drawing on extensive interviews and observations, this paper contrasts the ways in which an adaptive component of a major health care project was implemented in three program and three matched comparison states in Nigeria. The paper examines the bases on which claims and counterclaims about the effectiveness of these approaches were made by different actors, concluding that resolution requires any such claims to be grounded in a fit-for-purpose theory of change and evaluation strategy. The principles of adaptive development may be gaining broad acceptance, but a complex array of skills, expectations, political support, empirical measures, and administrative structures needs to be deftly integrated if demonstrably positive operational results are to be obtained, especially when undertaken within institutional systems, administrative logics, and political imperatives that are predisposed to serve rather different purposes.

This paper is a product of the Development Research Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at k8bridges@gmail.com and mwoolcock@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Implementing Adaptive Approaches in Real World Scenarios: A Nigeria Case Study, with Lessons for Theory and Practice**

Kate Bridges Michael Woolcock<sup>1</sup>

**Key words:** Institution building, Development effectiveness, Public sector reform, Legitimacy **JEL codes:** D73 (Bureaucracy), H83 (Public Administration), O17 (Institutional Arrangements)

> 1 The findings, interpretations, and conclusions expressed in this report are entirely those of the authors. They do not represent the views of the International Bank for Reconstruction and Development (World Bank) and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Thanks to Rachel Ort, Asmeen Khan, Benjamin Loevinsohn and colleagues in the Nigeria Country Office for their active support and helpful feedback on earlier (even longer!) versions of this paper; remaining errors of fact and interpretation are solely ours. Email addresses for correspondence: k8bridges@gmail.com; <u>mwoolcock@worldbank.org</u>
# 1. The challenge to operationalize “doing development differently”

Pointing out that something is not working is hard. Explaining why something is not working is harder. Developing and operationalizing alternatives that are sound, supportable, implementable and (ultimately, demonstrably) ‘better’ is typically harder still.

For many years now, the traditional, orthodox approach to development – one focused on transferring ‘best practice’ solutions and with limited assessment of endogenous constraints or supports – has been critiqued as a donor‐driven endeavor, too often concerned with providing uniform solutions rather than crafting context‐specific (‘best fit’) responses to locally nominated and prioritized problems. Traditional models, grounded as they are on the assumption that complex development challenges can be solved if sufficient money and expert technical advice are provided, work reasonably well for the types of problems for which they were designed, but have borne little fruit as their reach has expanded into more complex domains, such as public sector reform, which has frequently generated a plethora of institutions that _look_ deceptively like those of their OECD counterparts but have little of the underlying functionality (Andrews 2013). The fact that a variety of countries have achieved a measure of inclusive economic development by way of _contextualized_ institutional solutions, rather than copying the forms of others, confirms that a great deal of development consists of complex challenges for which we do not have ready‐made answers.<sup>2</sup> What we do know is that transferring formal, institutional solutions from OECD countries to developing counterparts seldom succeeds in generating functional success.

As Booth (2018) lays out, institutions comprise both formal and informal elements, such that if we only ever transfer the more visible, formal elements we end up with structures, laws and organizations that do not have the supportive base necessary for them to be functional.<sup>3</sup> Moreover, institutions are shaped and impacted by idiosyncratic patterns of politics that cannot be replicated elsewhere,<sup>4</sup> as well as by the way power is distributed among various individuals and groups in society.<sup>5</sup> This unavoidable uncertainty within complex systems makes the wholesale transference of prefabricated solutions into countries like Nigeria not only unrealistic but potentially harmful, if in so doing it weakens domestic problem‐solving procedures or de‐legitimizes the very idea of reform.

The observations of complexity and isomorphism clearly apply in the Nigerian context. Analysis of even successful cases of functional change in Nigeria show that “reforms in Nigeria are reversible, rarely considered complete or predictable in their path” and that “[r]esults never follow best practice in terms of technical content and often represent small steps toward better development outcomes” (World Bank 2015, 5). Political economy analysis (PEA) confirms the salience of politics and leadership in the Nigerian context, highlighting the importance of state leadership and of ensuring that the federal level actors allow sufficient autonomy for the states to act (Porter and Watts 2017). In other words, like every other country, Nigeria has a slew of existing rules, often “below the waterline”, that will condition the success or failure of any reform attempt. A development approach that ignores these rules can expect little traction.

> 2 See, among many others, Chang (2007), Rodrik (2007), Booth and Cammack (2013), Kelsall (2013), Booth (2015), Buntaine, Parks and Buch (2017) and World Bank (2017).

> 3 The foundational work is North (1990, 2003), but many others – before and since – have developed this thesis: e.g., Future State (2005, 2010), Carothers and De Gramont (2013), Unsworth (2015), and Yanguas and Hulme (2015).

> 4 E.g., Bates (1989, 2014), Leftwich (2007), North et al. (2009), IPPG (2010), Putzel and Di John (2012), North et al. 2013).

> 5 See Khan (1995), Khan and Sundaram (2000), Shirley (2008), Khan (2010), Pincus and Robinson (2014), Hickey et al. (2015), Hough and Grier (2015), and Khan (2017).

2
Aside from presenting and analyzing the failures of the old orthodoxy, practitioners have had some success in building analytical consensus around the core principles of its proposed replacement (or complement). Terms such as ‘Doing Development Differently’, ‘Thinking and Working Politically’, ‘Problem Driven Iterative Adaptation’, ‘Development Entrepreneurship’, ‘Agile Development’ and the like all describe approaches that present a conscious shift from the traditional model of best practice transfer, and which are similar enough to be grouped under the umbrella term “adaptive development”.<sup>6</sup> Emerging as a potential new orthodoxy, adaptive development emphasizes the importance of clearly identifying and understanding the nature of the problem being addressed (in particular its political economy factors) and taking small, incremental steps and adjustments towards a long‐term goal. Adaptive programming is predicated on the notion that ‘solutions’ to complex development problems can only emerge through implementation itself; they are essentially impossible to identify at the outset of a program (Kauffman 2016).

The relevance of adaptive principles is established in recent analytical work conducted by the World Bank in Nigeria, particularly in a recent Governance Approach note, which commits the World Bank Nigeria Country Team to “move away from best practice to best fit” and to “focus on function— that is, what works and why—rather than templates and generalized approaches copied from other operations and countries” (World Bank 2017). The righthand column of Table 1 highlights explicit commitments that the Country Management Unit of the World Bank office in Nigeria has made regarding implementation of its projects. As can be seen, these commitments correlate strongly with at least five of the six principles that are recognized as being at the heart of adaptive programming (Teskey 2017).

Though the rationale for a shift is well‐established, and although alternative models have emerged, the task of implementing these models, and demonstrating their utility, is still in its very nascent stages. For the average frontline staff member, big questions remain: How does one effectively operationalize adaptive development in the midst of systems, mindsets, skillsets and incentives that have largely been built for its solution‐driven predecessor? How does one do development differently in real world development contexts? And do these new approaches do what they claim they can do? Do they actually fix problems that matter?

The study from which this paper is drawn was undertaken to explore such questions. This paper is specifically written for operational staff in large development agencies, who accept the rationale for a more adaptive approach and wish to better understand how these can be actualized in country contexts. As such, details pertaining to the empirical findings – for those who wish to consult them – are largely consigned to annexes, with the main text providing an extended summary of the key findings and a detailed discussion of their implications for policy and practice. The research itself constitutes an analysis of the experience of a ‘Doing Development Differently’ initiative in Nigeria, which involved the piloting of adaptive assistance in eight Nigerian states, as part of World Bank support to the Saving One Million Lives (SOML) Program, itself Nigeria’s first attempt to implement a project via a Program for Results (PfR)<sup>7</sup> framework. The Terms of Reference (ToRs) for the Technical Assistance (TA) firm hired to implement the adaptive approach state that “the most successful

> 6 In certain quarters ‘adaptive development’ itself is a noun rather than an adjective (in the fields of child psychology, for example, as well as international development). As exemplified in the vibrant online discussion group #adaptdev (see <u>https://groups.google.com/forum/#!forum/adaptdev), however, the term has become, if</u> only by default, an encompassing expression for the non‐mainstream approaches to implementation used by those working within mainstream development agencies.

> 7 The original goal of SOML was to save the lives of one million mothers and children by 2015. The program has since been extended for five years as part of the Federal Government of Nigeria’s (hereafter FGON) National Strategic Health Development Plan (NHSDP). The stated objective is to dramatically improve the coverage of six interventions (discussed below) that currently suffer from poor access and utilization. The World Bank has committed USD 500 million equivalent to support this initiative, to be delivered via the Program for Results instrument over a five‐year period.

3
approaches to improving development outcomes and improvements in service delivery in developing countries combine three key ingredients: working in problem‐driven and politically informed ways; being adaptive and entrepreneurial and supporting change that reflects local realities; and is locally led.” With a view towards drawing out emergent lessons for the implementation of flexible, adaptive technical assistance, this paper aims to do three things: (i) document the nature of the adaptive approach undertaken (specifically in relation to those “three key ingredients”); (ii) draw some conclusions about the impact of that approach; and (iii) present some guidance for individuals and groups that wish to operationalize adaptive programming in their own contexts.

# Methodology

The analysis on which the conclusions presented here rest took place in three phases, between May 2018 and May 2019: Phase one of the fieldwork entailed conducting exploratory research in three of the TA‐ receiving states: Kogi, Abia and Anambra. It sought to document the nature of the tasks actually performed by the TA firm – Adaptive Development Group (hereafter ADG)<sup>8</sup> – and to make a preliminary identification of the contextual processes and organizational dynamics driving observed impacts. Interviews, focus group discussions, and participant‐observations were undertaken with various stakeholders in the three states, and were complemented with an assessment of reports from the TA firm, SOML project documentation, and other literature relevant to the PfR and to the TA component.

Building on phase one, the aim of the second phase was to further determine impact and relevance of the provided TA, specifically by comparison of findings from TA states vis‐à‐vis otherwise ‘comparable’ states that did not receive TA from ADG. The strategy of this phase was to visit three states that did not receive TA – Enugu, Akwa Ibom and Rivers – but that had similar political, social, financial and geographical contexts to the three TA states already visited. These decisions, and choices regarding where specifically to visit in each state, were made on the basis of extensive consultations with Nigeria‐based colleagues and government counterparts. Within each state, the team met with various SOML stakeholders to ask about the ways in which they have understood the SOML PfR, designed workplans and implemented interventions.

In each state and during the two fieldwork phases the team sought to meet with the following stakeholders:

- State Commissioner for Health

- Permanent Secretary of Health

- SOML Program manager and program officers from the SOML Program Management Unit

- Local Government chairpersons and Heads of Department (HoD) for Health

- Director of the State Primary Health Care Development Agency (SPHCDA)

- A Primary Health Care Facility in which SOML interventions have taken place

In addition, there were interviews undertaken with World Bank staff, ADG staff and stakeholders from the Federal Ministry of Health in Abuja. The final phase of the research entailed analyzing fieldwork data and integrating it with material collected from the two field visits, preparing papers for different audiences, incorporating feedback on them and securing the necessary clearances for their release.

> 8 This is a pseudonym, deployed here to protect the actual identity of the Africa‐based consulting company that implemented the technical assistance (TA) aspects of the ‘Saving One Million Lives’ project.

4
TABLE 1: ADAPTIVE PRINCIPLES REFLECTED IN BANK DOCUMENTATION

|**Seven Principles of adaptive**<br>**programming (Teskey 2017)**|**Commitments highlighted in the Nigeria Governance Note that reflect**<br>**adaptive principles (World Bank 2017)**|
|---|---|
|1) Context is everything|“We will continue to base operations on a solid understanding of<br>technical challenges, but we will also strive to understand local contexts<br>and feasible reform processes as opposed to filling technical gaps or<br>deficiencies in our clients’ current approaches.”|
|2) There is no singular blueprint<br>for development – rather<br>flexible, responsive, adaptive<br>programming|“There will be a presumption in favor of flexibility and adaptation in<br>designing and implementing Bank interventions.”<br>“...we will be looking to scale up our support to local pockets of efficacy<br>where the likelihood of sustainable reforms is greatest.”|
|3) Best fit not good practice|“The Nigerian context calls for flexibility over the lifetime of projects<br>with<br>regard<br>to<br>technical<br>approach,<br>counterparts,<br>phasing,<br>communications, geographical scope and scale, and so on. Accordingly,<br>we will encourage flexible project design and encourage and reward<br>project restructuring based on solid supervision and implementation<br>facilitation.”|
|4) Enabling, not doing|“While recognizing the importance of leadership for reform, we will<br>expand the range of stakeholders included in the design and<br>implementation of our engagements, with a view to creating broader<br>support and sustainability for reforms.”<br>“Ownership of approaches will be tested throughout the delivery cycle<br>and, where it does not exist, exit strategies will be employed.”|
|5) Real‐time learning|“Information—baselines, evidence, and feedback—will be a central<br>feature of our work as we test and tailor approaches.”<br>“Our approach will be based on problem solving and unpacking<br>assumptions behind hypotheses.”<br>“We will supplement our focus on rules, legislation, manuals, and formal<br>institutional designs with a focus on implementing these rules and<br>institutional designs,etc.”|
|6) Long‐term<br>commitments<br>with staff continuity|n/a|

The paper proceeds as follows. Following this introduction, section 2 lays out further details regarding the contexts within which ‘adaptive’ implementation work was undertaken within a national health project in Nigeria. Sections 3 and 4 provide an extended summary of the key findings, providing the empirical and analytical foundations for section 5, which explores some specific implications of these findings for policy, practice and theory. Section 6 concludes. (Detailed tabular presentations of the empirical results in each of the six states visited are presented in the annexes.)

# 2. Contexts: Doing ‘adaptive’ work in a national health project

The SOML PfR in Nigeria has as its primary goal helping states achieve specific improvements in six pre‐defined health interventions (the ‘six pillars’): 5
1. Immunization – Penta 3 Coverage (U1)

2. Nutrition – Vitamin A Supplementation (U5) 3. HIV/AIDS – Prevention of Mother to Child Transmission (PMTCT) of HIV 4. Family Planning – Contraceptive Prevalence Rate (CPR): Modern Contraceptives 5. Maternal Health – Skilled Birth Attendance (SBA) 6. Malaria – Utilization of Insecticidal Treated Nets (ITN) This definition of success formed the boundaries in which ADG, the TA firm, was expected to then undertake a process of locally‐driven problem‐solving within the states. ADG was required to work with states to identify, analyze and ‘fix’ the wide range of problems that cumulatively contribute to low coverage in the above areas, and therefore low health outcomes among the Nigerian population. Since low coverage in the six different indicators is unsurprisingly caused by a wide range of factors, often very specific to the peculiar context and personalities present in the state, ADG was inevitably engaging with a wide range of very diverse problems and causes.<sup>9</sup> In documenting the approach undertaken, we find that the TA provided by ADG exhibited aspects of all three of the “key ingredients” stipulated in the firm’s Terms of Reference: “[1] working in problem‐driven and politically informed ways; [2] being adaptive and entrepreneurial; and [3] supporting change that reflects local realities and is locally led”.

The problem construction and deconstruction process facilitated by ADG appears to have been locally‐anchored and sincerely appreciated by state officials. This is despite the fact that, in our view, the scope of problems ADG was tasked with fixing were large and diverse to the point of being unmanageable. Officers from the three State SOML Program Management Units (PMUs) that ADG were assisting narrate how the firm constantly pressed them to justify their proposed interventions on the basis of how it would affect the six outcomes, thus inculcating a far greater attention to data than usual. While it is apparent that the process ADG facilitated for identifying the causes of low coverage in the indicators was more thorough and nuanced than the traditional solution‐focused approach, the identified causes and problems themselves were not actively captured on paper or routinely revisited in the states. Nevertheless, with the help of their data capture platform, PIGEON, officers from the PMU and within local government were being alerted on a week‐to‐week basis of low reporting levels or unexpected results, and immediately following up on them directly with local government counterparts and with health facilities (often with advice from ADG as well<sup>10</sup> ). In this way, problem‐solving was made an active feature of ongoing implementation efforts.

9 A small snapshot of the sort of problems that have been identified as requiring “fixing” is reflected in the following list: People use the bed nets we distribute for fishing so that they can catch fish to eat and sell; Health workers have not been paid for the past six months so they do not come to the facilities for work; The roof of the health facility has fallen in so no‐one wants to go there; Basic equipment and drugs are unavailable in the health facilities, so people do not see the value in going there; Women do not use contraceptives or use them and do not report it; Religious leaders are opposed to the use of contraceptives by their members; Some health facilities are inaccessible in the rainy months; Health workers charge people for services that should be free, thus dissuading them from using them; Some people in the community do not trust immunizations, believing there is a plot by another identity group to infect the local people with a virus; Traditional birth attendants are more accessible, particularly for rural communities than skilled birth attendants in the health facilities and they offer more services (they will cook you pepper soup); There is no transport available for taking pregnant women to secondary health facilities; Health workers do not come to work because of security issues and doctors having been kidnapped in the past; It is culturally problematic for health workers to come into a person’s home, particularly if they are of a different gender, thus limiting the ability to monitor or advise on mosquito bednet utilization; People get upset if we test them for HIV but then we do not have drugs available for treatment so some health workers avoid testing.

10 Like the DHIS data, which are collected at the facility rather than household level, the PIGEON data are known to have a poor correlation with state level household survey data; as such, their use by ADG (as providing an 6
ADG placed a particularly strong emphasis on the identification and management of political and non‐technical drivers of the problem. The consultancy was co‐designed by Bank colleagues from the outset _not_ to be driven by technical best practice but rather to provide what some staff felt had been missing in previous TA, namely politically savvy local knowledge.<sup>11</sup> Interviewees were unanimous in identifying politics and personalities as _the_ main determinant of smooth workplan implementation in the Nigerian states. In both the TA and non‐TA states we assessed, there was strong evidence of powerful individuals either positively or negatively affecting the smooth functioning of SOML (see Annexes 8.1 and 8.2=3). Accordingly, during state planning workshops in early 2016, ADG drew participants’ attention to power dynamics in their states, helping participants to identify “key influencers” in their states – such as the first lady, traditional rulers, religious leaders, private sector actors, civil society, trade unions etc. – and the importance of managing them. ADG encouraged the state PMUs (SOML’s Programme Management Units) to consider the “electoral value” within all their initiatives, prompting them to opportunistically access the political will of certain key players. ADG reportedly pushed the PMUs to be better at highlighting good results, using data visualizations and capturing stories of success in a bid to maintain buy‐in.

Beyond simply applying a political lens and operating in a politically savvy manner, ADG also saw their role as one of advocating for the PMU and managing political interferences directly in an effort to create (and protect) space for implementers. In Kogi and Anambra, ADG made active, sustained attempts to manage problematic interference and was largely successful. In Abia, similar attempts were made, but the power of an interfering individual was so significant that success was only realized after the program had already been seriously undermined (see Abia snapshot, Annex 8.1).

We needed to uncouple the political from the technical. Civil servants in Nigeria are so cowed, so scared to push back on the interference. But when those interfering elements see that we are engaged as well, they think twice...” – _Interview, liaison officer, ADG_ This “political hustling” goes beyond the typical facilitative role that would be expected of a firm supporting adaptive programming and makes the Nigeria work particularly unique.

Regarding the second ingredient – “being adaptive and entrepreneurial” – there is some evidence that an emphasis on problem diagnosis, planning and PfR sensitization left limited time for cycles of learning and adaptation. Though the SOML and ADG’s assistance began in January 2016, it was early 2017 before any actual implementation of solutions began in any of the states. Partly this is due to a delay in SOML funds reaching the states<sup>12</sup> ; ADG had anticipated funds reaching states by March 2016 and implementing workplans from then, on the back of momentum gained and political authorization obtained during the preceding months. But due to the delay they instead entered what they describe as a “holding pattern.... all dressed up with nowhere to go” (interview, ADG).<sup>13</sup> Once accurate snapshot of health outcomes) has been questioned, particularly by Bank staff. Federal level staff also questioned why there was a need for what they see as “a parallel system to DHIS”. But at the state level, the impression given was that PIGEON functioned as a rudimentary problem warning system that became invaluable in establishing a culture of close monitoring, performance management and follow up.

11 Bank staff weighted the evaluation of the proposals with a particular view towards political and technical competence and then insisted on face to face interviews in order to test these skills. Reportedly, the weighting given to ADG was largely down to the support they expressed for a performance pay arrangement (which Bank staff took as a “sign of their commitment to a paradigm shift”) and their evident local and political knowledge. 12 Funds were made available for draw down in the states much later than initially anticipated (for a variety of reasons, including Bank delays as well as bureaucratic and political obstacles within the states), which had the effect of delaying anticipated implementation by over a year in some cases.

13 For the next year ADG continued engagement, mainly by focusing on the setting up of a data capture platform – known as PIGEON – geared towards ensuring weekly data collection once implementation began. They describe their task during this period as “preaching, handholding and maintaining a presence”. This was manly done via phone and email since at this stage they had no liaisons based in the states.

7
implementation finally got going, in early to mid‐2017, there was evidence of adaptation and of innovation (see Annex 8.3 for comparison of innovation across the visited states). The workplans are clearly understood to be flexible documents by the TA states and this fact appears to have left room for some “muddling” and iteration in the states during implementation. But the basis on which micro‐ adjustments were encouraged by ADG was not always apparent. In the absence of well‐documented theories of change, it does not seem that adaptation involved methodical reassessment of initial assumptions or was always carried out in a way that required clear justifications. State implementers also reflected that they felt the need for better data analysis skills to help them adapt in accordance with the evidence.<sup>14</sup> Regarding the third ingredient – “supporting change that reflects local realities and is locally led” – ADG conveyed that they felt it was within the implementer’s capability to identify challenges and to come up with the most appropriate solutions to address these. While ADG did recommend the possible application of certain solutions based on their own knowledge of what had worked well in other contexts, they appear not to have seen their role primarily as experts offering solutions.<sup>15</sup> Rather, they envisaged their role as being to create space for a latent creativity and innovation to emerge. Accordingly, even though they gave training to assist with problem identification and asked questions that helped PMUs to refine their analysis, the onus was placed on program officers responsible for each of the interventions<sup>16</sup> to be the ones to actually identify the critical drivers relating to their theme. Decisions about strategy and prioritization seem to have been made by certain individuals (typically program officers responsible for the different thematic areas) largely on the basis of their intuitive (‘gut’) sense of what was achievable. (As one of ADG’s liaison officers put it, “The states actually know what to do and how to do it. But they need space and confidence...they need the leeway to do what they do. You have to clear the way. Our role was to create space.”) It is our impression that after the MICS 2016 survey results came out, highlighting the negative performance of the eight TA states against the six outcome indicators thus far (in our section on impact, below, we question whether this was an appropriate means of measuring the success of the TA), ADG began to operate somewhat more out of a pressure to achieve results than to ensure a locally‐driven process. Having begun with an emphasis on very state‐specific, tailor‐made solutions, it appears that towards the later part of their contract, ADG began to push for more standardized solutions, most likely reflecting their increasing desperation to improve results (and thus obtain a performance payment as well as a potential extension of their contract with the federal government). Apparently, Bank staff also gave explicit guidance that ADG should ‘do more’ to support expansion of two particular activities: Maternal and Neonatal Child Health (MNCH)<sup>17</sup> weeks and Quality Implementation Supervision Support (QISS).<sup>18</sup> ADG thereafter placed various “liaison officers” in the

> 14 Though public sector staff in the TA states did refine interventions based on data analysis, they admitted finding it difficult to drill down into what is driving failure of certain interventions and expressed concern that they were sometimes misinterpreting data.

> 15 One of ADG’s liaison officer notes that in two states there were proposals to do costly launches of the SOML program, which would have used about N 10 million of project funds. Rather than saying they should not do it, ADG’s approach was to ask the states: How far will this move any of your indicators and are there other places it could be put that would move them more? The response by the states was to revise the workplans and to incorporate the launch into an actual rolling out of activities.

16 In most states there is one program officer for each of the SOML thematic areas e.g. a malaria officer, nutrition officer, immunizations officer etc.

> 17 States are expected to conduct week‐long campaigns twice a year that are targeted to cover a range of immunizations, nutrition objectives and other interventions for mothers and children.

18 QISS refers to regularized visits conducted by state and local government staff to monitor the range of equipment, infrastructure and services in operation at health facilities across the state. Teams of government staff visit each facility with a checklist and submit their findings back to local government representatives and the SOML PMU.

8
states towards the end of the contract, with explicit ToRs to strengthen implementation of these specific solutions, rather than to continue the locally‐driven problem‐solving process more generally.

Part adaptive facilitator, part political hustler, the firm also wore a more traditional technical assistance hat at times, devoting considerable effort to sensitization, training and advisory tasks typical of more solution‐driven approaches. A large part of the firm’s work in the state was devoted to sensitization on the mechanisms and objectives of the SOML PfR (which ADG insists took up far more of their energy than originally anticipated), setting up and training on data capture solutions (specifically PIGEON) and strengthening the roll‐out of some pre‐selected health interventions (as encouraged by the Bank), such as the QISS and MNCH weeks.

In short, the firm’s approach might be best described as a hybrid, or merged TA approach, combining elements of old and new, with an accompanying tension, messiness and innovation all part of the package.

# 3. Findings 1: How context conditions the approach

The hybridity of the firm’s approach should not surprise us, nor should the fact that the approach exhibited weaknesses. Sometimes the obvious requires stating: adaptive approaches do not take place in perfectly insulated environments but rather in complex contexts comprising (by definition) competing incentives, varied skills, divergent mindsets and inherently imperfect mechanisms.

Firstly, the quality of the adaptive process is necessarily conditioned by the firm facilitating that process. Wild et al (2017, 25) note that “whether or not programmes are genuinely doing development differently depends on who is implementing them” and that “[s]ervice providers or implementing partners need teams and staffing with the right capacities, skills and experience to be able to manage programmes in new ways”. The scope of problems ADG was tasked with fixing was large and diverse to the point of being unmanageable; indeed, to some degree we consider it a reflection of their inexperience that they took on a performance‐based contract with these payment terms.<sup>19</sup> The firm also struggled to support effective monitoring of outcomes, with Bank staff expressing serious concerns about ADG’s ability to collect credible data and carry out quality data analysis. In the absence of the firm carrying out a household survey (as had been recommended in the ToRs), it was very difficult for Bank staff to monitor their progress. Our analysis of ADG’s reports also indicates that their reporting skills were weak – language was often convoluted and opaque, and there was little reflection on problems fixed, adaptations made, or assumptions revised.<sup>20</sup> The reports tend to follow a more traditional structure, focusing on inputs delivered. A failure to methodically capture collectively identified problems and their causes appears to have led to limited reassessment of assumptions and reduced the quality of ongoing learning. Adaptation was certainly encouraged and incidents of it during implementation are evident, but it only got going latterly, shortly before the TA itself ended altogether.

Some Bank staff and federal level actors questioned whether ADG had sufficient technical understanding and expertise to carry out their assistance role. According to the federal level actors (with whom ADG had an unfortunately antagonistic relationship), ADG implemented technical solutions in the states that were ineffective and not based on a good understanding of what interventions would have the best results. Partly, this critique reflects a more traditional best‐practice

> 19 It is worth noting that the other bidding firms had substantially higher cost proposals, suggesting they did not judge the performance payment aspect worth the risk.

> 20 To some extent this is because the Bank staff also struggled with how to monitor and evaluate this work and found it difficult to give ADG a very clear or manageable framework to guide their reporting.

9
mindset, unaccustomed to this notion of finding locally‐driven, hybrid solutions.<sup>21</sup> Our own analysis confirms some instances where the firm could have been stronger at offering a variety of technical solutions as options for implementers to consider – options that were in keeping with the existing conditions and problems as identified by local stakeholders. That said, the very novelty of adaptive implementation – in Nigeria and elsewhere – means that, by definition, every firm that takes on this challenge for the first time is bound to encounter serious difficulties. (This does not absolve ADG of reasonable criticism, but it also means that their genuine efforts in these challenging circumstances should be fully appreciated.) Aside from the capabilities of the firm conditioning the quality of the approach, there are elements of the World Bank’s organizational environment – its skillsets, incentives, instruments and resources – that either support or undermine a more flexible, problem‐driven model. For example, although state level officials testify to the fact that the PfR instrument made possible a far more innovative, locally‐empowered and responsive way of working, it is also true that the instrument itself is still extremely novel in Nigeria and required significant socialization. Field work confirms that various agents in the states have faced significant challenges, delays and unintended effects as they have tried to understand and navigate this new performance‐based system that they have hitherto never encountered. The initial state advances were themselves seriously delayed and there continue to be challenges passing PfR payments through the state system: at the end of a year of implementation, an average of only 33% of funds had been disbursed in total. Drivers of this slow disbursement include bureaucratic obstacles (such as imposition of and subsequent required training for the accounting system, REMITA), infighting between the state and primary health care providers (the SPHCDAs) and rank opportunism and interference, as exemplified by the Abia case (Annex 8.1). In this context, ADG’s TA was required to be not only an agent of socialization into the brave new world of adaptive implementation _but also_ into PfR. Delays in fund disbursement contributed to a misalignment of TA and state implementation,<sup>22</sup> a somewhat unbalanced emphasis on planning, and what was perceived by various actors as an abrupt (and unfortunate) exit of ADG’s support.

World Bank staff and federal level officials have also been learning and adjusting to the PfR instrument as they go, and they readily refer to aspects that, given a second chance, could be designed differently. For example, although the SMART survey, on whose results performance payments were to be made, was expected to take place every year, in practice it was used to form a 2015 baseline but then did not take place in 2016 or 2017 and has only recently been undertaken in 2018.<sup>23</sup> The use of an alternative survey tool in 2016 raised significant outcry and has affected perceptions of overall fairness.<sup>24</sup> The fact that this 2016 survey, against which performance was measured and payments 21 There is a possibility that what federal level officials saw as low technical capacity, ADG saw as a case of letting state implementers lead in the design of hybrid, state‐relevant solutions.

22 Initially it was planned that the TA would take place in a fairly linear model – an initial preparation, sensitization and planning phase that would include obtaining political buy‐in and would result in completion and acceptance (by government) of state‐level action plans in eight states; an immediate implementation phase with constant monitoring, adaptation and learning over a yearlong period; and a completion phase characterized by reflection on results and capturing of lessons learned. This was originally expected to be completed by June 2017. Due to funding delays, however, the firm adjusted its efforts, such that the initial planning phase spanned a year‐long period, certain activities had to be repeated due to loss of momentum, and implementation and lessons‐learning phases were collapsed into six months.

23 Results from the 2018 survey round – reportedly completed in June 2018 ‐ are still not available at the time of writing.

24 In 2016 an alternative household survey – the Multiple Indicator Cluster Survey (MICS) – was used to determine progress on the six coverage areas and performance payments were awarded on the basis of that survey’s results. While some federal level government officials and Bank staff argue that there is a strong correlation between the MICS and the SMART surveys, the outcry generated among state officials after revelation of MICS results demonstrated that complementarity of the surveys was not a matter of consensus. Given that the MICS does not in fact measure one of the six key indicators – Vitamin A coverage – the states that 10
made, took place in August 2016 – i.e., before the states had begun implementation – also generated controversy. Stakeholders disagree as to whether it was reasonable to expect that states could have initiated some activities even without these funds, but all agree that the level of delay was unanticipated and that funds were expected to have been in the states much earlier.

Regarding Bank monitoring of the TA, the SOML supervision budget covered two weeks of staff time supervising the TA, though no _in‐country_ staff had a direct monitoring responsibility.<sup>25</sup> Initially, the firm was encouraged to carry out a mini household survey to check progress mid‐way through the assignment, but this did not take place.<sup>26</sup> The Bank TTL for the TA (from the Governance practice and based out of country), who acted as the contact point for ADG, evaluated their deliverables (sharing them with health sector colleagues) and managed their payments, confirms that monitoring was made additionally difficult by the fact that adaptive assistance, by its nature, is tough to report on or assess. In the absence of a reliable means of viewing progress on indicators or a reporting system that captured evidence of problems solved or capabilities enhanced, Bank staff began to question what was actually happening on the ground – but had no means of verification other than the word of the firm itself and their fairly convoluted, opaque reports.<sup>27</sup> Missions to the states (and especially rural areas) are logistically difficult<sup>28</sup> and the TA firm’s communication style has also been challenging.<sup>29</sup> As a result of growing concerns, Bank staff began to push the firm to implement solutions, specifically some solutions that it felt were technically appropriate. These solutions were dutifully implemented across the range of contexts in which ADG was operating, and as a result (as noted above) the TA began to resemble a more orthodox approach. Although there was certainly an incentive for states to try and track how their chosen interventions impacted the six indicators, the focus on those state‐level aggregate measures meant that examples of local level or discrete problems being addressed tended to be ignored, or at most viewed as a means towards an end and not as an end in and of themselves. In a determined and arguably well‐intended effort to “get out of the kitchen”, Bank and federal staff may have inadvertently neglected the chance to observe and learn from successful, local level performance improvements.

The political currency of adaptive development is such that authorization and engagement for such an approach is often more passive than active, and in this particular case the loss of key Bank staff during implementation of the adaptive TA had a negative effect on internal authorization. While the commitments outlined in the Governance Approach note (World Bank 2017) are certainly ‘adaptive’, it is our view that they reflect the commitment of a core governance team, rather than the Country Office more broadly. And although passive authorization may be sufficient for programs that already achieve wide organizational buy‐in, that passivity is often insufficient in large aid facilities, where “the context, the inheritance, strongly favours doing development pretty much as it was done previously” (Teskey and Tyrell, 2017). Even where staff are in principle remarkably open to adaptive programming, they will likely struggle to resist the incentives aligned against it – e.g. pressure to pre‐ program, to demonstrate linear results and to deliver best practice solutions. In the absence of felt they had expended particular effort on this indicator were particularly frustrated with the results. The outcry was such that it was eventually agreed to advance _all_ states 25% of the possible performance payment, regardless of actual results.

25 Attempts were made to address this – by inviting locally‐based staff to join supervision trips – but they were unsuccessful.

26 The main reason seems to be the cost associated with such a survey.

27 The firm for its part, appears to have been reporting to the best of its ability but genuinely struggled with how to report on the vast scope of problems that they were engaged in tackling with a wide range of stakeholders. 28 Testament to this point, the logistics required for the state visits conducted during this review took many weeks to organize, required lengthy visa processes, security clearances, armed police escorts and, needless to say, significant cost.

29 The key ADG point person has a loquacious, somewhat confounding communication style, which, given that he has been the key point of contact, meant factual details were sometimes hard to come by.

11
significant ‘unlearning’, upskilling and consensus‐building having taken place, it is likely that authorization will fluctuate and there will be a continual (perhaps inexorable) reversion to a more traditional approach. In our case study, this manifested as progressively less active engagement in the TA component by non‐core staff and increased pressure from Bank staff towards the firm for implementation of specific targeted solutions.

In addition, the Nigerian states themselves also present highly varied political contexts, capabilities, personalities and donor presence, all of which have the potential to significantly affect the impact of the SOML and of the TA component. (Annex 8.2 summarizes these factors while Annex 8.3 contains a brief side‐by‐side comparison of the six states, giving some indication of how these contextual divergences have significantly altered the way SOML has operated in each of the states.) We argue that even if the selection criteria for the eight TA states are relatively clear,<sup>30</sup> there remain many differences (observable and unobservable) between the states, a number of which are likely to contribute to divergence in health outcomes and which necessitate a tailoring of the TA approach.

# 4. Findings 2: In the midst of messiness, impact nonetheless

Despite hybridity and the various challenges highlighted above, there are strong indications that the TA _was_ indeed successful in solving discrete problems and in stimulating positive shifts in organizational behaviors and strategies. We find there is evidence that the TA states have been somewhat successful in fixing various performance‐related problems, albeit at a more discrete level than the six aggregate outcome level indicators can possibly capture. Although this current assessment of the six states is not sufficiently wide or deep to claim with certainty that the adaptive TA was ‘successful’, it is certainly the case that stakeholders in the two TA states where implementation is well underway (Kogi and Anambra<sup>31</sup> ) were able to talk compellingly about a range of specific problems they had identified and context‐specific ways they had discovered of resolving them. By contrast, stakeholders in the non‐TA states that have made some progress with implementation (Akwa Ibom and Rivers) refer mainly to standardized interventions like QISS and MNCH weeks when asked about interventions that have had impact.<sup>32</sup> (See Annex 8.3 for side‐by‐side comparisons.) Not only do the states that received TA exhibit more evidence of functional problems being fixed than their non‐TA counterparts, but they also exhibit more evidence of enhanced problem‐ solving capabilities and behaviors generally. TA states demonstrated greater appreciation and pursuit of coordination across stakeholders, were undertaking more _collective_ problem solving, were more cognizant of the necessity of managing higher level “authorizing” relationships, and demonstrate a greater understanding and appreciation of the non‐technical (or informal) drivers of the problems they are trying to fix. Along with a more nuanced understanding of the problem, TA states appear to 30 Eight states were selected to be recipients of the TA from ADG. According to Bank and Government staff, the aim was to select states that were already quite capable at health implementation, and that might thus act as “pace setters” for the others by providing a positive demonstration effect showcasing the value of adaptive‐ style TA while also motivating other states to improve their own performance (interviews, World Bank, Federal officials). Implementation ‘capability’ of the states appears to have been judged solely on the basis of how well states were performing on the six SOML indicators prior to the program’s commencement. Sixteen top performers were identified on this basis, covering four different regional zones of Nigeria. (The divergence in performance between Northern and Southern states is so marked that all 16 of the sample group come from the center or the South.) Two states were then randomly selected from each zone to be part of the TA complement, making eight states in total: Abia, Anambra, Delta, Edo, Imo, Kogi, Osun, and Oyo.

31 Recall that the other TA state – Abia – experienced a derailment in its program due to excessive political interference (see Annexes 8.2 and 8.3).

32 The only contextualized, successful solution that we identified in the non‐TA states was in Akwa Ibom, where they were piloting “husband schools”.

12
have selected solutions that show a greater appreciation of the context in which they operate, have favored more innovative solutions, and seemed more confident about experimenting and adapting during implementation than their non‐TA counterparts. TA states have made greater attempts to improve collection and utilization of what data are available and are actively using them to ensure that interventions link to SOML objectives. TA states also demonstrate a noticeably higher level of understanding regarding the nature and mechanisms of the PfR instrument.

In short, our analysis of a hybrid adaptive programming approach in Nigeria finds that such approaches _can_ make important gains in complex areas and on problems that have hitherto appeared intractable. This is despite the fact that such approaches will tend to exhibit all the tension, messiness and hybridity that one would expect of attempts to introduce a novel approach into a development sector such as health (where changing behavior, especially intimate behavior, is widely recognized as deeply challenging). On the other hand, we recognize that these gains are not irreversible, and in keeping with the views of ADG and various recipients of the TA, we believe that the TA would have benefitted from a longer running time – too much was expected from too few too soon. We note that the Program Support Units (PSUs), which have since been hired to support SOML in the states, have been given ToRs that are far more orthodox (i.e., solution‐driven) in their approach, and are highly unlikely to enhance the behavioral shifts that have been made thus far.

# 5. Implications and applications: Some guidance for the intrepid

For the development staff who are disenchanted with the old orthodoxy, eager to operationalize adaptive principles and ready for the adventurous (and at times messy) ride, we conclude by offering some guidance and reflections that we hope will be helpful and useable. This guidance is organized according to four core challenge areas associated with implementation of adaptive programming: (1) Identifying the problem to be fixed; (2) Monitoring and evaluation; (3) Selection and management of the firm; and (4) Maintenance of internal authorization and support. It fleshes out some practical lessons in relation to each of them. This guidance is by no means exhaustive; rather, we have focused on aspects that we think are currently under‐addressed in the existing literature on the subject.

## 5.1 Identifying the problem

**Start by facilitating the selection of a good problem.** Note the word _facilitating_ . Often donors create projects to fix problems that _they_ care about, rather than ones that matter to the actual implementers or beneficiaries. Often the problems that are selected are actually “the absence of a specific solution” (e.g., there is no functioning Anti‐Corruption Bureau; staff salaries need to be rationalized; compliance with reporting is low) rather than something that constitutes a specific issue of service delivery. In the Nigerian case, the fact that the metrics of success (and by inference, the problem) were already predetermined and framed at a very high, aggregate level, (i.e., the six state‐level health outcome indicators) made the scope of challenges that the firm had to deconstruct virtually unmanageable, and possibly not prioritized by the implementers they were working with.

It does not work when outsiders analyze the problem on behalf of those who will act to solve it. It works when those in the insider PDIA teams construct and deconstruct the problem (whether they do this ‘right’ or ‘wrong’). The insiders must own the process, and the outsiders must ‘give the work back’ to the rightful owners. – Matt Andrews<sup>33</sup> To be clear: we are not saying that the selection of the six indicators was ‘wrong’ – these are eminently sensible goals that the elected government has determined it wants to achieve – but rather that the measure of ADG’s success should have been far narrower and more locally proscribed, focused on the extent to which it helped implementers solve problems that the implementers themselves (in partnership with recipients) have discerned. At the aggregate level, movement in the six health

> 33 <u>https://buildingstatecapability.com/2017/03/20/initiating‐pdia‐start‐by‐runningand‐then‐run‐some‐more/</u>

13
indicators, whether positive or negative, is a function of a vast assemblage of interlocking factors, only a small fraction of which could ever be clearly attributable to ADG.

For proponents of adaptive development, beginning an intervention with collective identification of a clear, locally relevant problem is considered critical to building support for the program and for ensuring that whatever solutions are developed are relevant, legitimate and practical. In an effort to assist that problem construction process, Andrews et al. (2017) provide three pointers for what constitutes a ‘good problem’ as well as some tools for how to get there: TABLE 2: PRACTICAL STRATEGIES FOR IDENTIFYING A GOOD PROBLEM

|1.<br>A good problem**cannot be**<br>**ignored and matters**to key<br>change agents.|To ensure that whatever problems identified do matter sufficiently to Nigerians,<br>teams can try to ensure that in constructing a problem statement, the<br>statement should be able to answer five questions: ‘What is the problem?’,<br>‘Why does it matter?’, ‘To whom does it matter?’, ‘Who needs to care more?’<br>and ‘How do we get them to give it more attention?’ In answering these<br>questions teams can arrive at a problem statement that is clearly focused on<br>important service delivery failures and also enables easy construction of an<br>aspirational goal of problem solved.|
|---|---|
|2.<br>A good problem**can be broken**<br>**down**into easily addressed causal<br>elements.|Tools for deconstruction such as the “5 Why Technique” and “fishbone<br>diagram” can be used to identify the various causal elements of a problem,<br>including those that are typically “below the waterline” (such as the more<br>cognitive or normative drivers that form behavior). This_deconstruction_of our<br>problem will likely show that there is not any one cover‐all solution but instead<br>a multitude of small solutions that will need tailoring to each causal dimension.<br>It will also raise questions such as “Where do I start in trying to solve the<br>problem? What do I do? How do I ensure that all causal strands are addressed?”.|
|3.<br>A good problem**allows real,**<br>**sequenced, strategic responses**|Each identified cause and sub‐cause should be treated as an essentially<br>separate—albeit connected—point of engagement, each offering a different<br>opportunity for change (“space for change” or “readiness”). Teams can ask<br>therefore—for each sub‐causal strand—what the authorizing environment<br>looks like and where**authority**for intervention will come from, whose<br>**acceptance**is needed to move ahead, and what kinds of**abilities**are needed to<br>make real progress. This will enable teams to determine which areas will<br>warrant activities that grow the change space, whilst others will allow more<br>aggressive reform or policy adjustment because the change space is already<br>perceived as sufficient.|

**Defining and deconstructing the problem requires savvy facilitation.** The pointers and tools mentioned above are undoubtedly helpful and practical, but it bears repeating that one should never underestimate the difficulty that is often to be found in constructing and deconstructing problems. In the absence of savvy facilitation, actors struggle to decide which level of a problem to focus on, how far to break it down into its causes, sub‐causes and so on as well as which aspects are ‘contributors’ or fresh separate ‘problems’ in and of themselves.<sup>34</sup> Most development professionals are trained and conditioned to think in terms of providing solutions. Often, when one asked staff of MDAs the question “What is it you want to fix?”, the answer was framed in terms of the absence of a specific solution, rather than in terms of specific service delivery issues that matter to communities. Government counterparts have been taught the language of ‘best practice’ after years of learning that speaking in these terms is what keeps donor funds flowing. It is often difficult to sufficiently ‘remove hierarchy in

34 Within the adaptive community perhaps more can be done to refine and expand existing toolsets, but there is also clearly a need to work on identifying and building the particular skill‐sets that are required of facilitators who can assist these kinds of construction and deconstruction processes and establishing networks of such individuals who can assist in‐country teams. Some of these skillsets may be teachable, but some may be innate.

14
the room’ such that stakeholders can be honest about where they see breakdowns in the implementation process or where there is tension about the diagnosis itself. In such contexts, it requires savvy facilitation to help actors peel back the necessary layers until they identify the elemental (and actionable) issues at stake.

**Do not get ‘stuck’ in the problem or planning phase.** Identifying and deconstructing the problem is a necessary foundational phase. Plans of action are also important. But in the Nigeria case there is a sense that work floundered somewhat in this stage, that ‘planning’ continued for too long and that action was overly tied to incoming funds rather than existing resources.<sup>35</sup> The result was frustration, loss of momentum, and implementers being distracted from the daily work of _doing_ . Teams need to move quickly from analysis and plans into action and learning if momentum is to be maintained. An adaptive approach is predicated on the notion that you cannot plan everything out in advance in any case; no team can achieve perfect understanding before starting the work. Given the flexible, adaptive nature of this approach, it is useful to see planning as a continuous exercise rather than a one‐off event, manifested in a single document. The BSC PDIA team, for example, initiates planning in what they call a ‘Launchpad Event’, that takes all of a day and involves their facilitators working with a number of internal teams to tease apart existing problems. That process generates a one‐page action agenda that is intentionally short and simple and includes the following information.

- a description of the problem, and why the problem matters

- the causes of the problem

- what the problem might look like solved (and especially what this kind of result would look like in the time period within which they are working; usually 6 months)

- what the ‘indicative’ results might involve at the 4‐month and 2‐month marks (working backwards, the facilitators ask ‘Where would you need to be to get to the 6‐month ‘problem solved’ result?)

- fully specified next steps (where the teams identify what they will do in the next two weeks and what they plan to do in the two weeks after that)

- what is assumed in terms of authorization of the next steps, acceptance of these next steps, and abilities to do the next steps (they want teams to specify their assumptions, so they can track and learn where they are right and wrong and adapt accordingly in future steps).<sup>36</sup>

World Bank teams can learn from these approaches and in doing so shift partners’ and implementers’ emphasis from lengthy planning to iterative, action‐oriented learning.

# 5.2 Monitoring and evaluating an adaptive approach

**Adaptive approaches, given their novelty, need to make monitoring front and center of implementation.** The novelty of adaptive programming and the extent to which it is still poorly incentivized in major development institutions ensures that passive or distant management of an adaptive approach is likely to result in derailment, especially in contexts where a complex mix of disparate motivations and capabilities prevails. Aside from being committed to adaptive principles themselves, task managers responsible for adaptive programs should anticipate the need for both minor and major course corrections through the lifetime of the program and will need to be on hand to guide and navigate these. This role is unlikely to be fulfilled by the occasional ‘supervision mission’. As one experienced practitioner recently put it, “designing, contracting and implementing programs

> 35 There may be a sense in which the provision of external funds to undertake adaptive work is inherently problematic, since it complicates incentives and drive, adding an additional ‘carrot’ into existing motivations. If what is desired is that stakeholders implement solutions based on existing capability and a drive to fix the problem, then the provision of funds can risk diluting ownership and momentum. This is evidently the case when those funds are delayed, though it is hard to know whether it would still have had a negative impact on ownership if they had been timely. This is an area for continued research.

> 36 <u>https://buildingstatecapability.com/2017/03/27/active‐and‐adaptive‐planning‐versus‐set‐plans‐in‐pdia/</u>

15
to work in this way is time consuming and challenging – often involving swimming against the tide of conventional practice and expectations” (ODI 2016, 9).<sup>37</sup> It is unrealistic to imagine that such work can be managed out of country or with negligible supervision budgets. In the Nigeria case, Bank staff found that management of this adaptive component required far more of their time than had been budgeted for and required smarter monitoring tools than those they had at their disposal.

**The success or failure of adaptive programming cannot be measured with tools designed to measure traditional approaches.** Ladner (2015, 3) points out that “standard performance monitoring tools are not suitable for highly flexible, entrepreneurial programs as they assume that how a program will be implemented follows its original design”. A traditional method of monitoring will typically track a program’s progress in achieving predetermined milestones and indicators of success against an assumed path to change. In a project where the causal connections are well‐established and where we ‘know what works’, this approach makes sense. But for a project where we expect to learn much about what does and does not work during implementation, the attempt to formalize milestones and process indicators at the start can risk derailing the iterative approach needed for success (Preskill et al 2014, USAID 2014, Andrews et al. 2013, Pritchett et al. 2013). As Ladner (2015, 3) puts it, “Investing significant resources up front to establish baselines, identify indicators, and set up data collection protocols could inadvertently ‘lock in’ a theory of change and prevent program teams from exploring other options or seizing on opportunities when they emerge.”

**Success measures should focus on the targeted problem, and not an aggregate indicator far removed.** Proponents of adaptive programming insist that the adaptive approach is a better way of solving problems and building implementation capability than the typical solution‐driven, best practice model. _This claim should be put to the test._ It is legitimate and necessary to examine whether an adaptive approach actually fixed the performance problems it targeted and whether the capabilities of individuals, teams and organizations were improved such that they are better equipped to resolve problems as they emerge in the future. Of course, the critical factor here is that the problem being measured _must_ correlate to the one that the adaptive approach was actually working on, and not one that is many miles down the line of one’s Theory of Change assumptions. In the Nigeria case, the firm was being judged on the attainment of state‐wide aggregate health outcomes. In reality, it was working on issues as discrete as ambulances arriving late or doctors not turning up to work. We may _assume_ that the fixing of these discrete, local level issues will progressively combine to shift state‐ wide health outcome indicators, but that is an assumption that requires testing. As noted above, a vast assemblage of factors must come together to shift a state‐wide indicator, and even if they ultimately do, the trajectory of change in such complex interventions is rarely linear (Woolcock 2013) and the specific contribution of the ‘adaptive’ component will be almost impossible to empirically isolate. In the meantime, solving local, discrete but functional problems _matters_ – in its own right and because failing to do so creates a blockage point, or ‘binding constraint,’ on the larger system’s overall functionality. What should be measured is the extent to which such constraints are actually being ‘unblocked’ and ‘unbound’.

**Measuring whether improved problem‐solving is taking place will require more continuous and more ‘relational’ ways of monitoring.** Drawing on insights from the technology sector – where principles of ‘agile’ or ‘adaptive’ technology development are well‐established – Wild et al (2017, 25) note that an adaptive program by its nature will most likely require “more continuous and close involvement between both sides from the start (rather than the classic model where a funder hands over what is contracted, and only interacts if there is a problem).” As such, more ‘relational contracts’ become relevant. Teskey and Tyrel (2017) highlight the utility of donors participating in regularized formal and informal Review and Reflection (R&R) points, during which the contractor (and, we would insist, the implementers too) demonstrates how politics, interests, incentives and institutions were

> 37 Staff involved in the SOML PfR have likewise acknowledged how much more of their work and input the TA management required beyond initial anticipation and budget.

16
systematically considered in problem selection and design, and justifies why certain choices were made to stop, drop, halt or expand any activity or budget during implementation. The Harvard BSC team highlights light‐touch methods, such as their PDIA “check ins”, which include a series of probing questions to assist teams in capturing learning and maximizing adaptation.<sup>38</sup> Their more detailed “SearchFrame” is offered as a tool that “fosters a structured iterative process that is both well suited to addressing complex problems and meeting the structural needs of formal project processes”.<sup>39</sup> Drawing from the lessons of tech start‐ups, others recommend tools such as Strategy Testing<sup>40</sup> (Ladner 2015) as a means of evaluating process and method with a view to improved problem‐solving and iteration. Wild et al 2017 give an example of an adaptive logframe, drawn from DfID experiences, that sets out a set of clear objectives at the outcome level, and focuses monitoring of outputs on the quality of the agreed‐upon rapid‐cycle learning process.

**Document the process.** Whichever method of measuring problem solving is chosen, we recommend that documentation of that process be undertaken. In the Nigeria case, the absence of documented problem construction and deconstruction – or of problems solved, or solutions tried and abandoned – has left implementers with very little ability to re‐evaluate their assumptions or adjust their implementation in a methodical manner that foregrounds learning. In Nigeria, the absence of such information also left the World Bank with very little confidence as to what was actually taking place on the ground. We do _not_ recommend that this documenting take the form of long donor reports, for which implementers have to spend inordinate hours preparing. Instead it should be action‐oriented, capturing ongoing discussions and reflections. It can be the regularized updating of a project’s fishbone diagrams (showing problem targeted, and causes identified), as well as the answers to key questions asked at regular R&R check‐ins: What was done? What was learned? What is next? What are your concerns? In this way it covers the basic facts of action as well as initiating a process of reflection. Note that expert facilitation will be necessary to help ensure that participants do ask deep questions about their lessons.<sup>41</sup>

**Bank teams may wish to consider arms‐length monitoring.** The average Bank TTL is not given the time or scope to invest considerable energy in R&R with various project partners. Neither do they necessarily have the skills and ability to facilitate honest learning and reflection sessions with their partners. Until that changes (and it will certainly not change overnight), the type of relational monitoring we are talking about may be best handled by a firm that has the skillsets and experience necessary to facilitate thoughtful R&Rs, to document lessons and feed them back in synthesized form to donors. If the adaptive endeavor is small scale, then the outsourcing may be to a single consultant, practiced in the skills required; if the adaptive endeavor spans a country portfolio, then hiring a firm for the monitoring and quality assurance task would seem appropriate. Either way, monitoring needs to be budgeted as a core element, and not as an afterthought.

> 38 <u>https://buildingstatecapability.com/2017/04/11/initiating‐action‐the‐action‐learning‐in‐pdia/</u> and <u>https://buildingstatecapability.com/2017/05/07/motivating‐teams‐to‐muddle‐through/</u> and <u>https://buildingstatecapability.com/2017/03/27/active‐and‐adaptive‐planning‐versus‐set‐plans‐in‐pdia/</u>

> 39 <u>https://buildingstatecapability.com/2016/06/06/searchframes‐for‐adaptive‐work‐more‐logical‐than‐ logframes/</u>

40 Strategy Testing (ST) is a monitoring system that The Asia Foundation developed specifically to track programs that are addressing complex development problems through a highly iterative, adaptive approach.

41 Andrews notes for example how “someone may say ‘we tried to get Mr X to work with us, and he did not respond positively, so we learned that he does not want to work with us.’ We would follow up by asking, ‘Why do you think Mr X did not respond?’ Often this leads to a new set of questions or observations about contexts in which work is being done (including, very importantly, the politics of engagement). In the example, for instance, the ‘why’ question raised discussion about how people engage in the government (and if the team reached out to Mr X in the right manner) and the politics of the context (the interests of Mr X and how these might be playing into his non‐response).” 17
# 5.3 Selecting and managing the firm

**Select service providers with the right combination of skillsets for the work and the context.** We think the World Bank team did well to select a firm to carry out the TA that had experience navigating Nigeria’s political realities. Clearly politics and personalities are key drivers of challenges in the Nigerian health sector, and an ability to think and work politically will therefore be critical to effective implementation. On the other hand, the firm displayed some serious weaknesses in its ability to provide a range of technical solutions that could be relevant to the identified context. They also appear to have had few tools at their disposal to assist with problem construction, deconstruction and adaptation and seemed to struggle with documentation and reporting in a manner that was light touch and responsive. A balance needs to be struck between technical expertise, political networks and skills and experience in rolling out and reporting on adaptive approaches. Where the skills do not already exist, they need to be identified and addressed (whether by the Bank, the firm or a intermediary manager). Teskey and Tyrel (2017) include some helpful suggestions of the sort of organizational capabilities that might be assessed for and that can help the donor gain confidence of the contractor’s ability to facilitate a truly adaptive, iterative methodology:

- Technical and analytical tools for problem construction and deconstruction

- Systems for developing multiple theories of change and action

- Systems of embedded learning and monitoring that ensure they are capable of testing their theories of change

- A management structure that delegates high levels of discretion, allowing those on the ground to make micro‐adjustments

- High numbers of staff with social connections and insider knowledge

- A budget management system which allows flexibility and transfers between streams based on learning

**Consider longer‐term ‘partnerships’ with service providers.** Teskey and Tyrel (2017) insist that the levels of trust required to allow learning from failure as well as the space necessary for adaptive work requires the relationship between donor and contractor to be one of ‘partnership’ rather than principal‐agent. We would tend to agree with this, while adding that such relationships will necessarily take time and active cultivation. It is our view that the skillsets required for effective implementation of adaptive programming (e.g., an ability to facilitate joint problem construction, to support consensus‐building among diverse stakeholders, to provide a range of solutions with understanding of preconditions that drive their success, to provoke learning and analysis, etc.) seldom exist ready‐ made in most of the contexts that we wish to use them. ADG had elements of these skillsets but also had various deficiencies (not surprisingly, given that both PfR and ‘adaptive implementation’ had not previously been attempted at this scale in Nigeria). But if donor organizations seek to work only with those who excel in all these aspects, they will likely have no partners to work with. We therefore recommend seeing the development of adaptive service providers as a medium‐term but eminently worthy investment objective. In future relationships of this type, the Bank may wish to build in an initial set of monitoring and/or performance milestones that are associated with the service provider’s program and operational systems for designing and delivering adaptive TA (see bulleted list above).

**Put in place obvious incentives for being frank about and learning from failure.** If there exists a commitment to adaptation, testing and experimentation, then there _must_ of necessity exist an expectation and appreciation of the fact that failures will occur. Failures are critical to experimentation and learning and are a necessary step on the path to crafting fit‐for‐purpose solutions. Relationships between donors and contractors seldom foster honest discussion about failure, however, since donors are typically understood to reward only success stories and punish failures. Although ADG was told to foster experimental solutions, their rewards – e.g., potential continuation of their contract by the Nigerian government and their performance payments – were contingent not on learning from failure but on aggregate indicators being moved. It would have been difficult for them to honestly share the

18
aspects of their approach that were not working with their World Bank counterparts, even though this was rhetorically encouraged. This links back to our discussion above, about how to best evaluate success.

**Performance‐based payments for service providers of adaptive assistance may be appropriate in certain contexts but chosen measures of success must be tied to the problems being targeted.** The utilization of a performance payment for ADG in this case was problematic, since the measure of performance (improvements in state level indicators of health coverage) was too far removed from the actual performance problems that ADG’s assistance was directly targeting. That disconnect resulted in a clash between attempts to attain performance payments and attempts to promote a locally‐driven program. But it is conceivable that a performance payment could have been appropriate if what had actually been measured was discrete problems fixed or some other evidence of active, ongoing learning and an increase in problem‐solving abilities (refer to previous section on monitoring). Teams need to be thoughtfully experimental in this regard. Performance‐based payments will require savvy design if they are to avoid creating perverse incentives and ensure that focus remains on actually fixing problems rather than gaming the system.

# 5.4 Maintaining internal authorization

**Do not underestimate the potential costs of piloting adaptive approaches.** The perceived ‘failure’ (at least by some) of the adaptive component within the SOML PfR has likely had costs for future adaptive approaches in the Nigerian context. The costs are both relational (e.g., bridges burned with a bold service provider who has found the costs of adaptive programming not worth the benefits) and reputational (e.g., federal actors have judged the approach a ‘failure’ and appear less inclined to consider adaptive approaches going forward; some Bank actors appear to reflect similar sentiments).<sup>42</sup> Regardless of whether we believe these costs could have been better mitigated (or that the perceptions of failure are misguided), it is important that actors who wish to implement adaptive approaches do recognize that they operate in a contested space and that the attempt to shift the status quo may on occasion engender some resistance (even hostility). Implementation therefore needs to be as carefully considered, thoughtfully implemented and methodically evaluated as possible. Effective messaging and communication is key, particularly in highlighting stories of success when they occur and acclimatizing colleagues to the inherent messiness of a more adaptive approach. Proponents may not have many chances to ‘get it right’ and can risk burning significant political capital with rushed or poorly resourced and ill‐communicated efforts. More broadly, it is also important to be able to discern the difference between ‘failure’ that stems from honest efforts to explore possible solutions to vexing problems (i.e., short‐term failure which is the ‘price one pays’ to find the longer‐ term solution) versus failure that is a result of professional incompetence versus apparent failure that comes from trying to resolve a problem that, it turns out, is just too large or too entrenched. Few in _any_ branch of development implementation and evaluation have adequately engaged with these key issues (Rao et al 2017 is a notable exception).

## **Prioritize and sustain active authorization and support’; passive is insufficient.** The prevalence of powerful, systemic incentives for best practice, solution‐driven programming in large donor

42 It is clear that within the Bank, the PIU and the states, there were very varied expectations about the role of the TA and its impact. There are some indications that this divergence in expectations has had an undermining effect on appetite for adaptive approaches at the federal level and within the Bank; certain actors are fairly unequivocal in their view that the ‘adaptive development’ approach has failed in this case. Given the novelty of the approach, it was not just state actors who needed to maintain authorization; Bank staff needed to maintain this internally and with managers of the trust fund; ADG needed to do it with federal actors; the PIU needed to do it within Federal Government, etc. The Bank in particular had an important role to play in managing the relationships between Government counterparts and the contracting firm, and in the case of SOML might have done more to delineate responsibilities between these actors in a manner that may have enabled more productive engagement between them.

19
organizations (Bridges and Woolcock 2017; Bain 2016) is such that even if an adaptive approach is initially agreed by World Bank managers and co‐TTLs, these individuals will inevitably face pressures to revert to a traditional approach as the program continues. The ability of the program to resist such pressure will be primarily down to the presence of powerful, active authorizers capable of making the case for continued implementation. In such cases, passive buy‐in by critical implementing and managing staff of adaptive programs does not appear to be sufficient to withstand existing institutional recalcitrance. Adaptive teams need to not only identify active authorizers but must work hard to maintain and build that authorization throughout implementation, seeking alternative support when staff turnover leaves them bereft of a critical backer.

**Recognize the role of ‘intrapreneurs’; cultivate space for them and ensure long‐term staffing arrangements.** In their review of DfID’s adaptive attempts, Wild et al (2017) note that adaptive approaches tend to “cluster” around specific contexts and individuals. Denney and McLaren (2016, 31) refer to the need for internal ‘champions’ within the donor organization who can advocate for the adaptive approach, as well as close working relationships between the donor and implementing team; flexible and potentially longer‐term funding and staffing arrangements; and a recognition that trajectories of change and program results cannot always be predicted. Faustino and Booth (2015, 27) reflect on the critical role of “development _intra_ preneurs” who “maneuvered through the bureaucratic procedures and approvals from senior management, finance and contracts officers and desk officers... to create program modalities that supported iterative, politically informed programming.” Other commentators have highlighted the importance of long‐term staffing arrangements for adaptive programs (Teskey and Tyrell 2017; ODI 2016), observing the extent to which such programming benefits from the kind of relational management and embedded knowledge that only comes with longer term placement. This review likewise finds that the role of individual “intrapreneurs” or “mavericks” (Bain 2016) is critical in making adaptive approaches possible in big institutions, and in ensuring they are well‐implemented. Relatedly, the long‐term staffing of specific individuals over the course of such a program’s implementation cycle is recommended for program momentum and broader support. The novelty and counter‐organizational‐culture nature of adaptive programming is such that for the near future, specific individuals are going to be particularly critical to the success of programs. In practical terms, this means that organizations wishing to undertake such an approach need to think carefully about the human resources they have available to devote to the endeavor, and even more so if they are not adopting an arms‐length arrangement (see earlier recommendation) to manage the TA. They would do well to prioritize the placement of national, in‐ country individuals on the adaptive team, since these will typically reside in‐country for far longer than their international counterparts. And they need to ensure that these teams have sufficient reputational clout (or at least some powerful supporters/allies to call upon) within their organization so that they can defend their approach in the face of potential internal resistance.

**More generally, if organizations are committed to adaptive methods, they need to carefully think about the longer term ‘unlearning’ and upskilling that it may require of their organizational staff.** Much of the discussion here revolves around the outsourcing of adaptive facilitation to partners and firms, who then provide support to World Bank clients in a manner that seeks to be problem‐driven, locally led, adaptive and iterative. But given that a great number of Bank staff, particularly those based in country, have _direct_ responsibility for supporting government(s) in the resolution of complex problems, the elephant in the room, so to speak, is “why are we so focused on _outsourcing_ adaptive facilitation rather than building such capabilities within our existing teams, among Bank staff who are meant to be supporting the strengthening of Government problem‐solving capabilities on an almost daily basis?” This would be no small feat of course, particularly for a cadre of professionals whose identity is that of a detached ‘expert’, and who have been trained in the delivery of solutions with confidence (and arguably hubris). A move from a solution‐providing to problem‐driven orientation requires that ‘experts’ learn to be ‘facilitators’, adept in prompting the deconstruction of existing challenges in a manner that harnesses local knowledge and fosters creativity. It requires the

20
abandonment of “we know what works” and an embrace of experimentation, ‘muddling through’ and ‘good enough’ solutions. The unlearning, upskilling and consensus‐building required may not even be attainable by everyone; various commentators have reflected on the difficulty of teaching development professionals to “think and work politically”, suggesting that a good deal of the skill required may be innate (Marquette 2014, Rocha Menocal 2014, Carothers and Gramont 2013). Rocha Menocal likens it to learning a new language, in the sense that “natural ability or predisposition matter, but so does the learning process itself”.<sup>43</sup> We do not have conclusive advice on this broader challenge, but we cannot help but be excited by the possibility of a country office that sees adaptive programming not simply as a case of subcontracting other entities to run facilitative TA, but also as an upskilling of whole Bank teams to be more humble, more problem‐driven and experimental about the manner in which they support government problem‐solving.

# 6 Conclusion: Pragmatism, hope, reasoned persistence

This work takes time, it takes perseverance and it requires trust, and the task of attacking some of the most challenging areas in government is frustrating but absolutely worth it with each breakthrough. – _<u>Poobalan 2017</u>_ In conclusion then, our message is both pragmatic and hopeful. Though we might like to imagine adaptive approaches being implemented by perfectly equipped teams, funded by unequivocally committed donors and implemented in wholly conducive contexts, the reality, at least in the near future (but most likely any future), is going to be far more hybrid, messy and imperfect. What makes this review hopeful is that despite all that imperfection – be it funding delays, mixed motivations, inflexible instruments, political interference, passive authorization or destabilizing staff losses – positive impact still appears to be possible. Clearly there is much that remains to be learned about these approaches and a great deal more effort must be put in to demonstrating their utility, but it certainly seems realistic that with each attempt at adaptive programming, Bank staff can improve on their ability to manage such processes for increasingly greater impact.

# 7 Bibliography

Andrews, Matt (2013) _The Limits of Institutional Reform in Development: Changing Rules for Realistic Solutions_ . Cambridge: Cambridge University Press.

Andrews, Matt, Tim McNaught and Salimah Samji (2018) ‘Opening adaptation windows onto Public Financial Management reform gaps in Mozambique’ Harvard University, Center for International Development Faculty Working Paper No. 341.

- Andrews, Matt, Lant Pritchett and Michael Woolcock (2013) ‘Escaping capability traps Through Problem Driven Iterative Adaptation (PDIA)’, _World Development_ 51(11): 234‐244.

- Andrews, Matt, Lant Pritchett and Michael Woolcock (2017) _Building State Capability: Evidence, Analysis, Action_ . Oxford: Oxford University Press.

- Bates, Robert (2014) ‘The new institutionalism’, in Sebastian Galiani and Itai Sened (eds.) _Institutions, Property Rights, and Economic Growth: The Legacy of Douglass North_ . Cambridge, UK: Cambridge University Press, pp. 50‐65.

- Booth, David (2012a) _Development as a Collective Action Problem: Addressing the Real Challenges of African Governance._ London: ODI/Africa Power and Politics.

> 43 <u>https://oxfamblogs.org/fp2p/parlez‐vous‐politics‐or‐why‐working‐politically‐is‐like‐learning‐a‐language/</u>

21
- Booth, David (2012b) ‘Working with the grain and swimming against the tide: barriers to uptake of research findings on governance and public services in low‐income Africa’, _Public Management Review_ 14(2): 163‐180.

- Booth, David and Diana Cammack (2013) _Governance for Development in Africa: Solving Collective Action Problems_ . London: Zed Books.

- Booth, David and Victoria Chambers (2014) ‘Towards politically smart, locally led development: the SAVI programme in Nigeria.’ Discussion Paper. London: Overseas Development Institute.

- Booth, David and Sue Unsworth (2014) ‘Politically smart, locally led development.’ Discussion Paper. London: Overseas Development Institute.

- Booth, David (ed.) (2015) _Developmental Regimes in Africa: Synthesis Report_ . London: Overseas Development Institute.

- Booth, David (ed.) (2016) _Politically Smart Support to Economic Development: DFID Experiences_ . London: Overseas Development Institute.

- Booth David (2018) ‘When are we going to stop giving solution‐driven aid?’ Valedictory lecture delivered at the Overseas Development Institute, London, 23 April.

- Bridges, Kate and Michael Woolcock (2017) ‘How (not) to fix problems that matter: assessing and responding to Malawi’s history of institutional reform.’ Policy Research Working Paper 8289. Washington, DC: World Bank.

- Bridges, Kate and Michael Woolcock (2018) ‘Elephants in the adaptive development room: lessons from efforts to operationalize an adaptive development approach in the Nigerian health sector’ Mimeo, World Bank. Available upon request.

- Buntaine, Mark T., Bradley C. Parks and Benjamin P. Buch (2017) ‘Aiming at the wrong targets: the domestic consequences of international efforts to build institutions’ _International Studies Quarterly_ 61(2): <mark>471–488</mark>

- Carothers, Thomas and Diane De Gramont (2013) _Development Aid Confronts Politics: The Almost Revolution_ . Washington, DC: Brookings Institution Press.

- Chang, Ha‐Joon (2007) ‘Understanding the relationship between institutions and economic development: some key theoretical issues’ in Ha Joon Chang (ed.) _Institutional Change and Economic Development_ . London: Anthem Press/UN University Press, pp. 17‐33.

- Dasandi, Niheer, Heather Marquette and Mark Robinson (2016) ‘Thinking and working politically: From theory building to building an evidence base’. Birmingham: DLP.

- Denney, Lisa and Rebecca McLaren (2016) ‘Thinking and working politically to support developmental leadership and coalitions: The Pacific Leadership Program’. Birmingham: DLP.

- Faustino, Jaime and David Booth (2014) ‘Development entrepreneurship: how donors and leaders can foster institutional change’. Working Politically in Practice Series. London: ODI.

- Future State, Centre for the (2005) ‘Signposts to more effective states: responding to governance challenges in developing countries.’ Brighton, UK: Institute of Development Studies.

- Future State, Centre for the (2010) ‘An upside down view of governance.’ Brighton: Institute of Development Studies.

- Hickey, Sam, Kunal Sen and Badru Bukenya (eds.) (2015) _The Politics of Inclusive Development: Interrogating the Evidence_ . Oxford: Oxford University Press.

- Hough, Jerry F. and Robin Grier (2015) _The Long Process of Development: Building Markets and States in Pre‐industrial England, Spain, and their Colonies_ . Cambridge, UK: Cambridge University Press.

- IPPG (2010) ‘Beyond institutions: institutions and organizations in the politics and economics of growth and poverty reduction’. Manchester, UK: University of Manchester, Improving Institutions for Pro‐Poor Growth Programme.

- Kauffman, Stuart A. (2016) _Humanity in a Creative Universe_ New York: Oxford University Press.

Kelsall, Tim (2016). ‘Thinking and working with political settlements’. London: ODI.

22
- Khan, Mushtaq (1995) ‘State failure in weak states: a critique of new institutionalist explanations’ in John Harriss, Janet Hunter and Colin M. Lewis (eds.) _The New Institutional Economics and Third World Development_ . London: Routledge, pp. 71‐86.

- Khan, Mustaq H. and Jomo K. Sundaram (eds.) (2000) _Rent, Rent‐Seeking and Economic Development: Theory and Evidence in Asia_ . Cambridge: Cambridge University Press.

- Khan, Mushtaq (2010) ‘Political settlements and the governance of growth‐enhancing institutions’. London: SOAS, University of London.

- Khan, Mushtaq (2017) ‘Political settlements and the analysis of institutions’, _African Affairs_ 117(469): 636‐655.

- Ladner, Debra (2015) ‘Strategy testing: an innovative approach to monitoring highly flexible aid programs’. Working Politically in Practice Case Study 3. San Francisco: The Asia Foundation.

- Laws, Ed and Heather Marquette (2018) ‘Thinking and working politically: Reviewing the evidence on the integration of politics into development practice over the past decade’ _,_ TWP Community of Practice. <u>https://twpcommunity.org/wp‐content/uploads/2018/04/Thinking‐and‐working‐ politically‐reviewing‐the‐evidence.pdf</u>

- Leftwich, Adrian (2007) ‘The political approach to institutional formation, maintenance and change: a literature review essay’. Discussion Paper 14. Manchester: Improving Institutions for Pro‐Poor Growth.

- Marquette, Heather (2014) ‘Do donors have realistic expectations of their staff when it comes to ‐

- ‘thinking and working politically?’. DLP Opinions, 9 June. <u>http://www.dlprog.org/opinions/do donors‐have‐realistic‐expectations‐of‐their‐staff‐when‐it‐comes‐to‐thinking‐and‐working‐</u> ‐

- <u>politically .php</u>

- North, Douglass C. (1990) _Institutions, Institutional Change and Economic Performance_ . Cambridge: Cambridge University Press.

- North, Douglass C. (2003) ‘The role of institutions in economic development.’ Discussion Paper 2003.2. Geneva: UN Economic Commission for Europe.

- North, Douglass C., John J. Wallis and Barry R. Weingast (2009) _Violence and Social Orders: A Conceptual Framework for Interpreting Recorded Human History_ . Cambridge, UK: Cambridge University Press.

- ODI (2016) ‘Doing Development Differently: Who we are, what we’re doing, what we’re learning.’ London: ODI.

- Pincus, Steven C.A. and James A. Robinson (2014) ‘What really happened during the Glorious Revolution?’, in Sebastian Galiani and Itai Sened (eds.) _Institutions, Property Rights, and Economic Growth: The Legacy of Douglass North_ . Cambridge, UK: Cambridge University Press, pp. 192‐222.

- Piron, Laure‐Hélène, with A. Baker, L. Savage, and K. Wiseman (2016) ‘Is DFID getting real about politics? A stocktake of how DFID has adopted a politically‐informed approach (2010‐2015)’. London: DFID.

- Porter, Doug and Michael Watts (2017) ‘Righting the resource curse: institutional politics and state capabilities in Edo state, Nigeria’ _Journal of Development Studies_ 53(2): 249‐263

- Preskill, H., S. Gopal with K. Mack and J. Cook (2014) ‘Evaluating complexity: Propositions for ‐

- improving practice.’ Boston: FSG. http://www.fsg.org/publications/evaluating <u>complexity</u>

- Putzel, James and Jonathan Di John (2012) ‘Meeting the challenges of crisis states.’ London: London School of Economics/Crisis States.

- Rao, Vijayendra <mark>, Kripa Ananthpur and Kabir Malik (2017) ‘The anatomy of failure: an ethnography of a randomized trial to deepen democracy in rural India’</mark> _<mark>World Development</mark>_ <mark>99(11): 481‐497</mark>

- Rocha Menocal, A. (2014) ‘Getting real about politics: from thinking politically to working differently’. London: ODI.

- Rodrik, Dani (2007) _One Economics, Many Recipes: Globalization, Institutions, and Economic Growth_ . Princeton, NJ: Princeton University Press.

23
- Shirley, Mary M. (2008) _Institutions and Development: Advances in New Institutional Analysis_ . Cheltenham, UK: Edward Elgar.

- Teskey, Graham (2017) ‘Thinking and working politically: are we seeing the emergence of a second orthodoxy?’ ABT Associates, Governance Working Paper Series

- Unsworth, Sue (2015) ‘It’s the politics! Can donors rise to the challenge?’ in Alan Whaites et al (ed.) _A Governance Practitioner’s Notebook_ . Paris: OECD.

- Wild, Leni, David Booth and Craig Valters (2017) ‘Putting theory into practice: How DFID is doing development differently’. London: ODI.

Williamson, Tim (2015) ‘Change in challenging contexts: How does it happen?’. London: ODI Woolcock, Michael (2013) ‘Using case studies to assess the external validity of complex development interventions’ _Evaluation_ 19(3): 229‐248

- World Bank (2017) _World Development Report 2017: Governance and the Law_ . Washington, DC: World Bank.

Yanguas, Pablo and David Hulme (2015) ‘Barriers to political analysis in aid bureaucracies: From principle to practice in DFID and the World Bank’ _World Development_ 74(10): 209‐219 24
# 8 Annexes

## 8.1 Snapshot of Abia state: The primacy of politics and its impact on performance

Of the three TA‐receiving states we visited, Abia experienced the most detrimental political interference. Although Anambra and Kogi both experienced interference, they appear to have created some space for successful operation within that. For Abia, space has been highly constrained. Their case provides an excellent example of **how political interference can derail program performance.** Although the initial $1.5 million for SOML implementation was released by the Bank in June 2016, it was only accessible to the PMU from June 2017. The PMU team and counterparts undertook implementation between June and August 2017 on the basis of workplans devised with ADG. Then, in September, it became apparent that the then Permanent Secretary of Health had unilaterally changed three signatories to Remita (Remita is the account approval software). There are six signatories in total, though only three required for approval), apparently replacing them with persons thought to be more amenable to “doing his bidding” (interview, anonymous). In order to effect this change, the PS in question is said to have forged the Governor’s signature. Finding themselves no longer able to access the account, members of the PMU raised the issue with staff of the Auditor General, who in turn raised it with the Governor. This initiated a feud with the Governor and the PS.

The issue is complicated by the fact that the sitting Governor, who would ordinarily be far more powerful than the PS, has been recently contesting his election in court (which resulted in him being removed for a short period and then brought back) and is thus preoccupied with an existing power struggle. Moreover, the PS in question is said to have considerable political clout and commands a large number of potential votes. In addition, the power of the health commissioner in Abia has been reduced by the fact that, owing to circumstances we do not understand, the Governor suspended all commissioners for 5 months. The result has been that the PS is far more powerful in relation to both the Governor and Commissioner than would normally be the case.

ADG made use of its political capital and reputational clout to engage directly with these various players in an effort to prevent the program going off track and to return power to the PMU. They describe the PS as “frankly criminal” and characterized the change of signatories as him “wanting his cronies to sign the cheque”. Meetings with him were unable to effect change. ADG was surprised that although the Governor was made aware of the PS forging the Governor’s signature, he did not fire him. After engaging directly with the Governor, ADG learned that this was due to the fact of the Governor considering it too politically sensitive to remove him by firing. Instead, the Governor told ADG they would wait and that he would gradually remove him from the program. In ADG’s words “He asked us to lie down and be quiet. We were to trust him that eventually the changes would be made”.

True to the Governor’s promise, the PS was eventually transferred, though this was done during a mass transfer of PSs in general, presumably to reduce the chance of political fallout. A number of Directors that had been appointed by the unruly PS remain in position. The PS transfer happened in November 2017. Unfortunately, as one of his parting moves, and apparently as revenge against the PMU members who had alerted AGF to the signatories issue, the PS wrote a letter requiring that the then Program Manager be removed from the PMU. Staff are currently waiting on the AGF to write to the new PS, requiring him to update the account verifier and signatories. The removed PM is hopeful that such an update might lead to his position being returned, though our own assessment is that this is unlikely.

### _What has been the effect of all this on implementation?_

In Abia we met with both old and new PMU members (it seems that in addition to removing the PM formally, a number of other PMU members have been informally side‐lined). It was very apparent that implementation was only recently begun by the new team and in contrast to Kogi and Anambra, they 25
have little sense of any impact yet. When asked about whether they’d improved on any indicators, they indicated that it is too early to tell. Moreover, we saw less sign of the team making decisions based on evidence or a detailed understanding of the problem. Members of the old PMU claim that tasks have deviated significantly from the previous workplan (for example the QISS component was removed on the basis that it was too expensive, though it has since been reinstated) and that the emphasis has been on large procurements that they believe will have little bearing on the indicators. We were told that four vehicles have been procured at inflated costs for example, two of which are being used by individuals and not for the program at all. We also noted that some procurements the team discussed with us are actually related to the advancement of general health objectives (for example procuring generators) and do not seem to have been made with a clear attention to their impact on the six indicators.

We found that the current PMU does have technically capable people in it, some of whom benefitted from the ADG TA. We also found that their discussion of future interventions showed some appetite for innovation and some attention to outcomes, though this was not so pronounced as their Kogi and Anambra counterparts. Current interventions, such as the installment of a data monitoring chart in all the health facilities, show some promise and an appetite for innovation, but in general, we noted that most of their interventions are still in planning phase, with very little actually delivered. Given that the budget is supposedly over 80% spent, this suggests that a large part of expenditure may have been diverted to inputs that were not directly relevant to SOML.

Unlike the other states, Abia was unable to present us with any current DHIS data on their performance against the six indicators. We also noted that the new PS is fairly unaware of SOML and, unlike his Abia and Anambra counterparts, did not seem to know the various indicators.

# 8.2 Factors that contribute to divergence in outcome indicators between states

|**Existing levels of state**<br>**funding**|The 36 Nigerian states have very different levels of existing revenue available to them and<br>therefore different levels of additional resource to draw on in order to supplement SOML<br>funds. It is apparent that whilst the delay in SOML funds arriving in some states meant<br>those states were unable to begin any substantial interventions, there are other, more<br>financially viable states who were able to continue complementary activities even in the<br>absence of fresh resources. A state like Rivers which has a higher Internally Generated<br>Revenue (IGR) and receives a greater federal allocation than any other state, has drawn on<br>a large bank of resources to supplement SOML efforts, building health infrastructure,<br>improving health worker remuneration and heavily investing in equipment at health<br>facilities. The simple fact of state indebtedness is likely to have a major impact on health<br>interventions since it directly impacts health worker salaries and therefore motivation.|
|---|---|
|**Presence and capability of**<br>**the private sector**|As indicated, the ADG states were selected from a group of 16 states that began with a<br>higher baseline on the six coverage indicators; apparently this was considered a rough<br>proxy for existing levels of “capability”. These states were judged to be higher in existing<br>capability and lower in existing donor activity. But this is a crude measure of capability. It<br>has since been pointed out that higher‐than‐average baselines probably reflected the<br>capability of the private more than the public sector in many of these states (interview,<br>WB). In practice, there is likely to be wide divergence between capability of the public<br>sector across the states and there is little hard evidence to support the assertion that the<br>pool from which ADG states were selected have a higher capability to start with. And<br>assuming that the private sector_is_a major driver of improved coverage, then the extent<br>to which the private sector is supported, enabled and capable (a fact which is largely<br>beyond the scope of SOML) becomes a major factor in the ability of states to improve the<br>six indicators.|

26
|**Levels**<br>**and**<br>**quality**<br>**of**<br>**existing donor support**|In addition to having varied levels of financial capacity, the 36 states have highly varied<br>levels of implementing partner (IP) support. The six states covered as part of this review<br>are reportedly considered to all be “under‐resourced in TA” (Interview, WB),<sup>44</sup>but in<br>practice, there are still significant divergences between them. The details of which IPs are<br>operating in the state, on which thematic areas and with how much support clearly varies<br>significantly. For example, we noted that the indicator in which Enugu currently has the<br>best rates of reporting and performance (according to the very imperfect data available)<br>is in Penta 3 coverage, despite the fact that_none_of their SOML funds have been utilized<br>towards Penta 3 thus far. Instead, that aspect continues to be almost completely financed<br>and driven by an IP. Other states that do not have the presence of such an IP are clearly at<br>a major disadvantage. Bank staff likewise confirmed that one state advanced in its PMTCT<br>indicator because “[an INGO] ran the whole program”. Referring to TA and non‐TA states<br>is thus something of a misnomer. A state may have had no ADG presence but still have<br>been able to draw on considerable donor resources and TA.|
|---|---|
|**Status**<br>**of**<br>**PHCUOR**<br>**implementation**|‘Primary Health Care Under One Roof’ refers to a policy of the FGON to shift all PHC‐related<br>financial resources, employees, programs and facilities from Local Government to the<br>SPHCDA. The SPHCDA/B in each state is required to integrate all Primary Health Care (PHC)<br>activities hitherto run by the Local Governments, on the principle of “three ones” – one<br>plan, one management and one monitoring and evaluation system – as a means of<br>improving primary health care implementation.<sup>45</sup>This shift has happened to varying<br>degrees in the different states and in some states, has been a major source of blockage,<br>internal conflict and funding delays<sup>46</sup>(interview, WB). State officials insist that a major<br>reason for resistance and delay with the PHCUOR has been that it entails local government<br>players losing out on the personal gains they were previously able to make when PHC funds<br>flowed through their structures.<br>The transfer of PHC staff from Local Government to the SPHCDA has particularly important<br>implications for the ability of the state to manage staff performance and remuneration.<br>This transfer has been completed in only one of the states that we visited, and it was<br>immediately noticeable that this state has been able to harmonize salaries and clear<br>arrears to thegreat advantage of the health sectorgenerally.|
|**Political/ security context**|The political economies of the states vary enormously and are a major factor in successful<br>implementation. As our comparison of the three TA states – Abia, Anambra and Kogi –<br>demonstrates, politics appears to be a major contributor to performance of the SOML<br>program in each state. In Abia, despite similar levels of assistance, orientation and<br>resources, the program has been severely hampered by the opportunistic interference of<br>powerful individuals (see Bridges and Woolcock 2018, Annex D). By contrast, a non‐TA<br>state like Rivers has a highly supportive political context for SOML, with a Governor who is<br>committed to funding major improvements in the health sector from the state budget.<br>Stakeholders confirm that north‐western states,which are said to exhibit far less negative|

44 IPs have understandably focused their energies on states with the highest rates of poverty and lowest attainment in human development outcomes, which has meant that Southern states tend to have considerably less donor presence than their Northern counterparts. 45 <mark>According the PHCUOR guidelines, the SPHCDAs are expected to be autonomous of the State Ministries of Health and to only report to the State Governors through the State Commissioners for Health. They are to be the sole employers of all human resources for PHC and managers of all PHC‐related financial resources, programs and facilities in a state.</mark> Funding for PHC activities in a state is supposed to be dedicated and requires “Basket funding” which means that all PHC funds from all sources (including development partners) are pooled into one “basket” and distributed to various programs and components of PHC based on an integrated operational plan adopted by all stakeholders. This is to be embedded in the legislations establishing the SPHCDAs in each state. 46 According to the National Health Act of Nigeria 2014 states are expected to have functional State Primary Health Care Boards as one of the prerequisites for accessing their allocations from the Basic Health Care Provision Fund.

27
||political interference, are showing faster levels of improvement in SOML indicators<br>(interviews, WB, ADG).<br>Levels of insecurity and conflict also vary greatly between the states and are likely to have<br>major impact on SOML performance. Our six selected states are generally less conflict‐<br>ridden than some of their Northern counterparts but still have considerable divergences<br>between them. It should be noted that the effect of conflict or insecurity on the indicators<br>can go both ways. On one hand we find that health facilities in Rivers have historically been<br>affected by kidnappings of health workers for example, which will clearly have a negative<br>effect on performance; in other states, the presence of Internally displaced peoples camps<br>for example is said to have enabled easier targeting of interventions.|
|---|---|
|**Survey timing**|The timing of surveys should theoretically benefit those whose interventions coincided<br>fortuitously and disadvantage those whose did not. Bank staff indicated that state officials<br>should have been aware of the dates when surveys were to take place in their states. States<br>were apparently notified of a three‐month window during which surveys would take place.<br>Some staff pointed out that savvy teams would seek to “game the system” by timing their<br>MNCH weeks for example, so that they took place shortly before the survey was<br>undertaken, for maximum effect.<sup>47</sup>Our interviews with states however indicated that they<br>had little to no awareness of when the surveys were taking place and that they certainly<br>did not time their activities to take fullest advantage of this. It is not clear whether this is<br>because they were not given the information about dates or whether they were given it<br>too late to shift plans. Either way, it is apparent that indicator performance varies<br>significantly in relation to the timing of certain interventions (see graphs in Anambra<br>snapshot for example; Bridges and Woolcock 2018, Annex D), and if any states were lucky<br>or savvy enough to time their interventions with the survey, this may have had significant<br>effect.|

> 47 Though other staff indicate this may be overstated, since Vitamin A coverage is measured over the last 6 months and immunization over the last 9‐12 months.

28
# 8.3 The state of the states

As part of this review, visits were made to six states, covering three regional zones, three of which received the TA from ADG and three of which did not. In each of the states we made a rough assessment of the political context, progress in implementation and fund utilization, problem identification, range of solutions arrived at and success in improving outcomes. Our findings are briefly summarized in Table 3.

## **TABLE 3: HIGHLIGHTED CHARACTERISTICS OF THE SIX VISITED STATES**

|**State**|**Summary**|**Political support / interference**|**Status of interventions and fund**<br>**utilization**|**Range of solutions**|**Examples of problems fixed**|**Performance on the six**<br>**key indicators (2016)**|
|---|---|---|---|---|---|---|
||<br> <br> <br>|<br> <br> <br>|**_TA states_**<br>||<br> <br>||
|Kogi|One<br>of<br>the<br>stronger<br>states.<br>Advanced<br>in<br>implementation<br>with<br>strong<br>evidence<br>of<br>innovation,<br>adaptation<br>and impact. Benefitting<br>from<br>a<br>somewhat<br>supportive context, and<br>actively engaging with<br>political<br>realities<br>for<br>greater effect.|Kogi<br>has<br>had<br>a<br>**somewhat**<br>**supportive political environment.**<br>Governor was up for re‐election<br>and has been eager to demonstrate<br>gains in health service delivery<br>since this was a key campaign<br>promise. This has not translated<br>into regularized payment of health<br>worker salaries however and salary<br>arrears were running at 5 months<br>during our visit. Commissioner<br>highly supportive and well‐versed<br>in SOML objectives. There were<br>some issues that the team describe<br>as “internal competitive rivalry”<br>rather than political interference,<br>with certain figures wanting to<br>chair committees or be part of the<br>PMU – ADG helped manage these.|**Sustained implementation since**<br>**March**<br>**2017.**<br>Began<br>implementation of workplan in<br>March 2017.<br>A good number of interventions<br>well underway, including drug<br>procurements and distributions,<br>various trainings, sensitizations,<br>advocacy<br>campaigns<br>and<br>supervision visits.|**Wide range of solutions from**<br>**the typical (QISS and MNCH**<br>**weeks) to the innovative.**E.g.,<br>introduction of food incentives –<br>“pluses” – to improve levels of<br>trust and usage of PHCs; use of<br>private clinics as distribution<br>points for Vit A; engagement of<br>community volunteers selected<br>_by the community,_thereby<br>allowing<br>greater<br>access<br>to<br>people’s houses for purposes of<br>giving advice and monitoring<br>utilization; resurrection of ward<br>management committees.<br>Also evidence of engagement<br>with<br>political<br>realities<br>e.g.<br>coordinating net distributions<br>with Governor campaign in order<br>to<br>increase<br>resources<br>and<br>impact.|Engagement<br>of<br>locally<br>selected<br>community<br>volunteers appears to have<br>increased bed‐net utilization<br>as well as awareness of PHC<br>services and greater use of<br>PHC facilities.<br>Reinvigoration<br>of<br>ward<br>committees appears to have<br>had<br>direct<br>impact<br>on<br>community ownership and led<br>to<br>infrastructure<br>improvements being financed<br>by community members.<br>MNCH weeks have positively<br>impacted<br>Vit<br>A<br>and<br>immunization rates.|**Improved.**One of only 2<br>TA states that improved<br>performance against its<br>baseline according to the<br>MICS<br>2016<br>survey.<br>Though PMU confirms<br>these<br>improvements<br>happened before any<br>SOML<br>implementation<br>was underway.<br>Although Kogi improved<br>in 3 of the 5 indicators<br>measured by MICS and<br>had a sum increase of<br>12.9% overall, it had<br>regressions of ‐33% and ‐<br>3.7% in Penta 3 and in<br>CPR respectively.|

29
|**State**|**Summary**|**Political support / interference**|**Status of interventions and fund**<br>**utilization**|**Range of solutions**|**Examples of problems fixed**|**Performance on the six**<br>**key indicators (2016)**|
|---|---|---|---|---|---|---|
|Abia|Implementation severely<br>hampered<br>by<br>malign<br>political<br>interference.<br>Very few interventions<br>undertaken so far and<br>little evidence of impact<br>despite<br>significant<br>expenditure.<br>Slowly<br>getting back on track,<br>with<br>some<br>innovative<br>plans in the pipeline but<br>impact of interference<br>likely to be felt for some<br>time.|**Highly**<br>**unsupportive**<br>**political**<br>**environment.**Direct interference<br>by a politically powerful civil<br>servant<br>has<br>led<br>to<br>financial<br>irregularities (including accusations<br>of<br>fraud),<br>suspension<br>of<br>procurement, staff transfers/ side‐<br>lining and a prolonged delay in<br>implementation<br>of<br>SOML<br>interventions.<br>The<br>political<br>currency of the interfering person<br>in question was such that efforts by<br>ADG to address the interference<br>were unsuccessful. Matters have<br>begun to be resolved in early 2018,<br>but<br>transferred<br>staff<br>appear<br>unlikely to be reinstated.|**Significantly**<br>**delayed**<br>**implementation and minimal**<br>**activities**<br>**despite**<br>**significant**<br>**expenditure.**The initial $1.5<br>million<br>for<br>SOML<br>implementation<br>was<br>only<br>accessible to the PMU from June<br>2017. The PMU team and<br>counterparts<br>undertook<br>implementation between June<br>and August 2017, but this<br>implementation<br>was<br>then<br>suspended<br>after<br>political<br>interference from the PS at the<br>time. It has only been reinitiated<br>since February 2018 after a<br>lengthy,<br>politically<br>charged<br>process to remove the PS. Very<br>few interventions are actually<br>underway despite the fact that a<br>large percentage of SOML funds<br>(reportedly 80%) have actually<br>been spent (indications that<br>expenditure has been poorly<br>linked<br>to<br>outcomes<br>and<br>emphasising<br>equipment<br>purchases that benefit specific<br>individuals).|**Very few solutions have been**<br>**implemented.**Even standard<br>interventions like the QISS have<br>only just begun implementation.<br>Proposed interventions, such as<br>the<br>installment<br>of<br>a<br>data<br>monitoring chart in all the health<br>facilities, show some promise<br>and an appetite for innovation<br>though whether such solutions<br>will<br>achieve<br>implementation<br>remains to be seen.|In<br>the<br>absence<br>of<br>implementation, no evidence<br>of specific problems being<br>fixed.|**Regressed.**<br>Results<br>according<br>to<br>the<br>5<br>indicators<br>that<br>were<br>captured during the 2016<br>MICS survey show Abia as<br>having regressed in all<br>except<br>one<br>indicator<br>(SBA). Their sum total<br>across<br>the<br>indicators<br>showed a regression of ‐<br>47.6%.|

30
|**State**|**Summary**|**Political support / interference**|**Status of interventions and fund**<br>**utilization**|**Range of solutions**|**Examples of problems fixed**|**Performance on the six**<br>**key indicators (2016)**|
|---|---|---|---|---|---|---|
|Anambra|Well‐advanced<br>implementation and wide<br>range of innovative and<br>standard solutions as well<br>as good evidence of<br>impact at the local level.<br>Broader<br>context,<br>specifically the delay in<br>implementation<br>of<br>PHCUOR,<br>likely<br>to<br>negatively affect state‐<br>wide impact of SOML<br>interventions.|**Partly**<br>**supportive**<br>**political**<br>**environment.**<br>Direct<br>political<br>interference has not been a<br>problem for SOML in the state. But<br>there<br>has<br>been<br>significant<br>resistance to the PHCUOR initiative<br>in the state, such that unions went<br>to court to try and prevent the<br>integration. In the absence of<br>integration,<br>local<br>government<br>remains responsible for payment of<br>health workers, salaries remain in<br>arrears and the state actors find it<br>difficult<br>to<br>manage<br>the<br>performance of the health workers.<br>The PMU confirms that this inability<br>to motivate the health workers has<br>been its greatest obstacle.|**Implementation is fairly well**<br>**advanced, having begun in early**<br>**2018**after funds were made<br>available for draw down in the<br>state.<br>A good number of interventions<br>are well underway, including<br>drug<br>procurements<br>and<br>distributions, various trainings,<br>sensitizations,<br>advocacy<br>campaigns<br>and<br>supervision<br>visits.|**Wide range of solutions from**<br>**the typical (QISS and MNCH**<br>**weeks) to the innovative**e.g.<br>conducting<br>SOML<br>Health<br>Intervention Days (SHIDs) –<br>which were used to “mop up”<br>people missed during MNCH<br>weeks with household visits;<br>distribution of vitamin A cards<br>and extensive sensitization via<br>outreaches; engagements with<br>traditional birth attendants and<br>extensive<br>advocacy<br>with<br>community influencers; training<br>of community youths to hang<br>nets and advocacy with teachers<br>to influence the same etc.|Initiated an MoU with 60<br>private facilities that included<br>a provision to give these<br>facilities some resources and<br>some stipends on the basis<br>that they would regularly<br>supply data on coverage of the<br>6 indicators. This has greatly<br>improved reporting rates and<br>improved coordination with<br>these facilities.<br>Development of Vitamin A<br>cards along with outreaches<br>regarding their use has helped<br>ensure tracking and treating of<br>persons missed out during<br>MNCH weeks.<br>Initial success in reaching key<br>community<br>influencers<br>regarding<br>the<br>use<br>of<br>contraception.|**Regressed.**Anambra had<br>a sum total of ‐4.1%<br>reduction<br>from<br>its<br>baseline according to the<br>2016 MICS survey. It<br>made small gains in<br>Penta 3 and SBA but<br>declines in the other 3.|

|**_Non‐TA states_**|
|---|

31
|**State**|**Summary**|**Political support / interference**|**Status of interventions and fund**<br>**utilization**|**Range of solutions**|**Examples of problems fixed**|**Performance on the six**<br>**key indicators (2016)**|
|---|---|---|---|---|---|---|
|Enugu|Very<br>delayed<br>implementation and low<br>expenditure. Momentum<br>appears low in the team.<br>Little to no evidence of<br>thorough<br>problem<br>analysis or of ability to<br>link inputs compellingly<br>with<br>outcomes.<br>Somewhat unimaginative<br>solutions, few of which<br>have been implemented.<br>Major issues of data<br>completeness.<br>Overly<br>reliant on implementing<br>partners.|**Elements**<br>**of**<br>**interference.**<br>Stakeholders in Enugu tend to insist<br>that the political context is highly<br>supportive. But deeper questioning<br>reveals evidence of interference<br>that has affected performance. For<br>example, the SOML account for<br>Enugu was suspended for some<br>months due to an irregular transfer<br>of SOML funds (15 million naira)<br>into the state account that was<br>initiated by the Governor. It is also<br>apparent that additional layers of<br>approval have been added to the<br>standard approval process in Enugu<br>such that the commissioner and<br>Governor have more input in the<br>workplans than ordinarily required,<br>which leads to additional delays in<br>the process. It is unclear what the<br>drivers of these interferences are,<br>but it is clear that they have slowed<br>down<br>implementation<br>of<br>the<br>program substantially.<br>PHCUOR has not yet been fully<br>implemented, and there continue<br>to be issues with health worker<br>salary payments and performance<br>management.|**Expenditure has been majorly**<br>**delayed in Enugu and shows**<br>**little sign of improvement.**<br>Roughly 25% of funds utilized<br>thus far, 50% of which have gone<br>to<br>“crosscutting<br>expenses”.<br>Examples of funds utilization<br>include<br>laptops,<br>plumbing,<br>vehicles, data bundles, training<br>etc. This is partly due to an initial<br>account suspension that was the<br>result of funds being transferred<br>out of the SOML account on the<br>direction of the Governor in<br>contravention of program policy.<br>On re‐activation, they then faced<br>the<br>issue<br>of<br>the<br>existing<br>accountant retiring and the<br>replacement<br>not<br>yet<br>being<br>“activated” on remitta, meaning<br>that no transactions can take<br>place. This has been the case<br>since<br>the<br>previous<br>officer<br>departed suddenly in December<br>2017 (over seven months ago).<br>Standard interventions – like<br>QISS visits, which have been<br>running for many months in Kogi<br>and Anambra – are not yet<br>started<br>in<br>Enugu.<br>Staff<br>repeatedly<br>referred<br>to<br>the<br>actions<br>of<br>implementing<br>partners, rather than the state,<br>when<br>asked<br>about<br>what<br>interventions were taking place.|**Proposed solutions are fairly**<br>**run of the mill**– MNCH weeks,<br>re‐trainings and procurement of<br>drugs ‐ and do not seem to<br>demonstrate<br>significant<br>contextualization<br>or<br>deep<br>understanding<br>of<br>causes.<br>Members of the TWG seemed<br>fairly uninterested/ unpracticed<br>in discussing the drivers of<br>existing problems and were<br>eager to move directly to<br>solutions.<br>PMU<br>members<br>admitted that they are not clear<br>on what is required by a<br>“bottleneck analysis”.|**Implementation**<br>**has**<br>**not**<br>**advanced**<br>**sufficiently**<br>**to**<br>**confirm any problems yet**<br>**fixed.**<br>According to existing DHIS<br>data,<br>which<br>were<br>acknowledged<br>to<br>be<br>unreliable and incomplete,<br>most indicators appear to be<br>regressing as do reporting<br>rates. State officials explain<br>the regression as being the<br>result<br>of<br>implementing<br>partners existing.<br>Implementation that is taking<br>place<br>is<br>driven<br>by<br>implementing partners. For<br>example, improvements in<br>coverage of Penta 3, appear to<br>be driven completely by IPs.<br>The PMU has dedicated 0% of<br>funds to that area so far, and<br>yet it is the area where there is<br>best reporting and signs of<br>increased coverage.|**Regressed.**Enugu had a<br>sum total of ‐12.1%<br>reduction<br>from<br>its<br>baseline according to the<br>2016 MICS survey. We<br>are not privy to the<br>breakdown.|

32
|**State**|**Summary**|**Political support / interference**|**Status of interventions and fund**<br>**utilization**|**Range of solutions**|**Examples of problems fixed**|**Performance on the six**<br>**key indicators (2016)**|
|---|---|---|---|---|---|---|
|Akwa Ibom|Powerful<br>commissioner<br>creating a very supportive<br>context<br>for<br>improvements<br>in<br>the<br>sector.<br>Implementation<br>well<br>underway<br>but<br>expenditure<br>seems<br>mainly to be contributing<br>to sector‐wide objectives<br>rather than specific SOML<br>objectives. Issues of weak<br>problem diagnosis, data<br>analysis and an over‐<br>reliance on implementing<br>partners. Low levels of<br>understanding regarding<br>the PfR instrument and its<br>implications<br>for<br>procurement, contracting<br>etc.|**Somewhat**<br>**supportive**<br>**political**<br>**context**<br>**but**<br>**bureaucratic**<br>**obstacles.**The Governor in Enugu is<br>reportedly committed to improving<br>health service delivery. He has<br>appointed a Commissioner for<br>health who was former Chief<br>Medical Officer of a major private<br>company – Exxon Mobil – and has<br>been given space and resources to<br>rejuvenate the health sector. The<br>Commissioner appears supportive<br>and committed to SOML objectives.<br>The fact that PHCs still fall under<br>the remit of Local Government<br>(PHCUOR<br>having<br>not<br>been<br>completed), but the commissioner<br>has proven powerful/ respected<br>enough to enforce a level of<br>discipline<br>(he<br>regularly<br>visits<br>facilities unannounced, has fired<br>doctors, initiated prosecutions for<br>fraud and reward programs, etc.)<br>and has succeeded in completely<br>clearing a backlog of salary arrears.<br>So, despite PHCUOR not yet being<br>fully implemented, motivation and<br>performance of health workers are<br>said to have improved significantly.|**Above**<br>**average**<br>**expenditure**<br>**with an emphasis on equipment**<br>**and infrastructure.**60% of SOML<br>funds<br>utilized<br>thus<br>far.<br>A<br>significant<br>amount<br>of<br>expenditure is being allocated to<br>inputs that are not directly<br>related to SOML goals e.g.<br>Essential<br>drugs,<br>generators,<br>refrigerators,<br>HIV<br>test<br>kits,<br>Infrastructure.<br>A number of standard SOML<br>interventions have not been able<br>to run, apparently due to “lack of<br>funds”.<br>MNCH<br>weeks<br>for<br>example did not run in 2016 and<br>have not yet run in 2018. Highly<br>reliant<br>on<br>implementing<br>partners<br>e.g.<br>UNICEF<br>for<br>supplying vitamin A.<br>We noted pronounced confusion<br>about the nature of the PfR, with<br>members of the PMU very<br>unclear<br>about<br>procurement<br>requirements.|**Evidence of some innovative**<br>**solutions, but many solutions**<br>**that do not seem clearly linked**<br>**to the problem or goal.**One<br>innovative solution is the piloting<br>of “husband schools”: in an<br>effort to target men, who have<br>significant<br>decision‐making<br>power in the household, short<br>term ‘schools’ are set up in a<br>community that men are invited<br>to attend, for instruction on<br>immunizations, contraceptives,<br>skilled birth attendance etc.<br>But it seems that high levels of<br>expenditure on infrastructural<br>needs and large equipment<br>purchases (neither of which link<br>specifically to SOML goals) have<br>squeezed<br>out<br>other<br>interventions.<br>The<br>team<br>consistently reflected on a lack<br>of funds and a reliance on<br>implementing<br>partners.<br>And<br>other solutions, such as training<br>of TBAs and procurement of nets<br>do not appear to be being done<br>with deep thought regarding the<br>foundational problem and its<br>drivers.|The one example that the<br>PMU<br>could<br>give<br>of<br>an<br>intervention having impact<br>was the ‘husband schools’.<br>Apparently<br>DHIS<br>data<br>demonstrate improvements in<br>CPR and in SBA in the areas<br>where the schools have been<br>piloted.<br>As with other states, they<br>appear generally unaware of<br>the impact of other inputs.<br>Data are incomplete and not<br>considered reliable.|Akwa Ibom had a sum<br>total of 24% reduction<br>from<br>its<br>baseline<br>according to the 2016<br>MICS survey. We are not<br>privy to the breakdown.|

33
|**State**|**Summary**|**Political support / interference**|**Status of interventions and fund**<br>**utilization**|**Range of solutions**|**Examples of problems fixed**|**Performance on the six**<br>**key indicators (2016)**|
|---|---|---|---|---|---|---|
|Rivers|Implementation has been<br>slow to start but now<br>moving.<br>Standard<br>interventions<br>for<br>the<br>most part, with some<br>isolated<br>examples<br>of<br>innovation<br>and<br>little<br>evidence of impact as of<br>yet. Strong indications<br>that any improvements<br>on SOML indicators thus<br>far have been down to a<br>particularly<br>conducive<br>political environment.|**Highly supportive political context.**<br>Health worker unions are powerful<br>in Rivers and prior to the initiation<br>of SOML there was a health worker<br>strike that lasted one year. The<br>current Governor – nicknamed “Mr<br>Projects” – came to power on the<br>back of a campaign to deliver major<br>improvements in the health sector<br>and<br>address<br>health<br>worker<br>remuneration.<br>Considerable<br>amounts have since been spent<br>from the state budget raising and<br>harmonizing salaries, building and<br>upgrading health facilities (e.g. 14<br>secondary<br>health<br>facilities<br>completely<br>renovated),<br>and<br>harmonizing<br>administrative<br>structures. The Commissioner for<br>Health refers to the Governor as<br>“The friend of the health sector”<br>and narrates how health receives<br>the second highest expenditure in<br>the state (after works).<br>Rivers is the only state in our<br>sample where PHCUOR has been<br>implemented and the responsibility<br>for PHCs now lies with the State<br>Primary health care facility rather<br>than Local Governments. Salaries<br>are reportedlyallpaid on time.|**Implementation slow to start**<br>**but**<br>**now**<br>**moving**<br>**fairly**<br>**smoothly.**Despite money having<br>arrived earlier in Rivers than the<br>other states (around January<br>2017), there was a major delay in<br>accessing funds due to incorrect<br>designation of the account as a<br>naira rather than dollar account.<br>Implementation<br>eventually<br>began in June 2017. By August<br>2017 an MNCH week had been<br>run, with a positive impact on<br>the data. Another round has<br>been run in January 2018.<br>Evidence<br>of<br>trainings,<br>procurement of drugs, nets and<br>equipment<br>ongoing.<br>Some<br>standard interventions – like<br>QISS – still have not been<br>initiated.<br>TCG<br>not<br>meeting<br>regularly due to limited funds.<br>And orientations on SOML still in<br>process despite 2 years into<br>program.<br>Complaints that procurement<br>and approval processes lead to<br>long delays in interventions.<br>Little<br>evidence<br>of<br>adapting<br>workplans.|**Most solutions follow typical**<br>**model but some evidence of**<br>**innovation.**The majority of<br>solutions seem focused on re‐<br>trainings, procurement of drugs<br>and equipment, immunization<br>campaigns etc. But also some<br>interesting attempts to shift<br>behavior in rural communities<br>e.g. one intervention involves<br>the<br>training<br>of<br>community<br>members who have used SBAs or<br>CPR as “interpersonal change<br>agents” on the basis that<br>individuals are more likely to<br>listen to their peers than an<br>outsider. Also experiments with<br>results‐based payments e.g. the<br>retired health workers receive<br>incentives if they succeed in<br>bringing 20 women for ANC.|The one area where the team<br>is confident that they have had<br>an impact so far is with MNCH<br>weeks.<br>With<br>the<br>other<br>interventions there does not<br>appear to be any certainty/<br>evidence as yet of a positive<br>impact.|Rivers<br>had<br>a<br>21.5%<br>improvement<br>on<br>its<br>baseline according to the<br>2016 MICS survey. We<br>are not privy to the<br>breakdown.<br>Given<br>that<br>no<br>implementation<br>had<br>begun by the time these<br>results came in, they<br>seem to be testament to<br>the impact of the broader<br>context, specifically the<br>Governor’s<br>efforts<br>to<br>address health worker<br>performance<br>and<br>improve<br>health<br>infrastructure.|

34
35