of announcements need to be carefully managed- Public announcements as to the tangible results delivered through the aid program are essential. However, the EINRIP experience demonstrates the adverse effect this can have if targets are not accurate. Targets should be based on reasonable assumptions, careful analysis and ultimately viable plans. During interviews conducted by the evaluation team it was widely acknowledged that the original targets regarding scope and completion time were prematurely set and unachievable. Although EINRIP fell short of delivering on the original ‘annouceables’ it did ultimately deliver, with some delay, Agree in principle, but not appropriate in this case Although we that DFAT should do its utmost to provide advice to portfolio ministers to influence announcements and ensure that they are achievable, there is insufficient information within the report or supporting documents reviewed to support the assertion that the announcement made at the time was not managed appropriately.
against the outcomes contained in the loan agreement. Communicating success- EINRIP has produced better quality roads than has been delivered by DGH using GoI funds or with financing from the MDBs. This is well recognised by those closely associated with the project but the extent of its broader demonstration effect is unclear. The EINRIP M&E team prepares high quality annual reports which document many of the projects achievements. Although these are shared with GoI, especially DGH, and the MDBs, none of the officials interviewed by the Evaluation Team was aware of these reports. A clearer strategy and /or effort is required in order to increase awareness of the benefits associated with Agree Given the scale of the EINRIP program and the relatively innovative manner in which it was implemented it is important that we continue to communicate the program’s achievements. We will continue to look for opportunities to do this with regular advocacy with the Indonesian Government and also a standalone event to celebrate the completion of the program.
the EINRIP approach. This might include a short (2-4 page) annual summary perhaps launched at a conference or workshop organized by DFAT Jakarta. Loans combined with grants can be highly effective- Loan financing is typically not a feature of Australian aid. Whilst some other bilateral donors are active in this area, DFAT has considered the possibilities extensively but as yet not committed to the ongoing use of concessional loans as a form of aid. There are clearly a range of factors to be considered by the Australian Government before changing its policy position on loans and the aid program. However, in EINRIP there is a positive example of the additional benefits that can Agree The combination of loans and grants was highly effective in the delivery of this infrastructure program and welcomed by our GoI counterparts. We see value in further exploring this modality in the Indonesian context given Indonesia’s middle-income status. The use of grants with loans also provides Australia with greater opportunity to influence the upfront preparation and ongoing monitoring of infrastructure programs thereby reducing risk. Nonetheless, the future use of loans by DFAT
be derived when loans and grants are used together. is a broader policy issue for the department to consider, rather than a specific lesson for the Indonesia program. Lesson / Disagree Applicability of lesson to current or future investments in Indonesia Cost benefit analysis is an inadequate for project selection- The use of cost benefit analysis to guide the selection of infrastructure projects is necessary but not sufficient to ensure appropriate projects are supported. As outlined in chapter 6, development benefits do vary from project to project and a selection process which is able to take a fuller account of Partially agree The here appears to be one of transparency rather than limitations of cost benefit analysis. The review acknowledges that the EIRR of 15 percent was not met in all cases, which suggests that other factors influenced the selection of projects – the real question here is whether these factors led to the correct roads being selected? Multi-criteria analysis would have provided greater transparency and
logical factors which cannot easily be monetised is desirable. The use of analysis is one approach which facilitates the systematic consideration of economic factors alongside other important criteria to guide enabled more rigorous selection of projects that did not meet the simplistic EIRR threshold. analysis still has a role in lower value, less complex projects that do not warrant the research and analysis necessary for a comprehensive multi-criteria analysis. The three “E’s” (engineering, education and enforcement) are critical to road safety- Reducing road accidents and deaths, particularly as speeds increase, is extremely unlikely to occur through the use of engineering measures alone. These measures need to be accompanied by education and enforcement elements to Agree While we agree that good engineering coupled with education and enforcement is the best approach to improving road safety, this lesson also raises philosophical questions about the scope and limitations of infrastructure projects and the role of local partners as ‘owners’ of the infrastructure. Although we could (and did) engineer solutions to road safety issues on EINRIP roads,
bring about a reduction in traffic accidents including fatalities. Doubtless, this would either complicate, or simply be beyond the scope of many aid funded road construction projects. As an alternative it may be appropriate to include design features that deliberately slow traffic at selection locations in recognition of the limited attention provided to road user education and enforcement in Indonesia and many other developing countries. the issues of education and enforcement were not constrained just to EINRIP. The users of these roads and those enforcing safety standards extended beyond the scope of the project. We agree that the road safety lesson from EINRIP is really that better roads lead to more severe accidents as vehicle speeds increase and DFAT Posts supporting road investments should consider whether complementary road safety programs are appropriate. Often this will be the case and will be considered in the work of our new infrastructure program, KIAT. The potential for infrastructure to impact on men and women differently is not well understood- All parties Agree We are seeking to ensure that gender considerations are central to the work of KIAT, and the ODE Evaluation has been provided to the KIAT team
interviewed by the evaluation team supported the idea that EINRIP roads should benefit men and women equally. Many assumed this to be the case believing that roads are generally, if not always, gender neutral. However, the absence of any detailed consideration to gender issues in EINRIP suggests the project was more likely gender blind, than gender neutral. Whilst specialist gender expertise is now more often a feature of current Australian aid investments, a short case study or guidance piece which helps DFAT officers and their counterparts better appreciate why gender matters in as key guidance material. We are already working closely with all of our development partners to advocate for a greater focus on gender in our current and future infrastructure programs. We also understand that the Infrastructure Policy Section is currently working on a guidance note to advise on gender aspects of infrastructure programs.
Effectiveness: Was EINRIP effective in the achievement of intended outcomes? Evaluation question / sub question Evidence required Data collection method and source Analytical approach To what extent do the physical infrastructure and capacity gains established through EINRIP reflect those outlined at project inception? (E1) The degree to which completed infrastructure reflects the original and revised scope of work Document review: To trace evolutions in scope against final deliverables (project completion report, loan agreement, annual reports, AQC reports) Map planned outcomes against actuals The extent to which roads meet original estimates relating to travel times, traffic volumes, accident rates and vehicle operating and maintenance costs Document review: Assess validity of initial assumptions against achievements (baseline & monitoring surveys, EIRR calculations) Categorise and tabulate KPI data The extent to which innovations demonstrated in the design, planning and delivery of EINRIP have been taken up by DGH Stakeholder interviews: To understand current practice in Indonesian road construction (DFAT, DGH, World Bank, ADB, engineering consultants) Collate and synthesise feedback from stakeholders What were the positive and negative, intended and unintended consequences associated with EINRIP? (E2) The extent to which intended benefits were delivered (as above) Feedback from stakeholders as to the appropriateness of roads and the EINRIP approach Stakeholder interviews: To capture perspectives of DGH and other institutional stakeholders Document review: To capture perspectives of road users and communities (social surveys) Synthesise feedback from stakeholders Synthesise evidence from communities
How effective was the EINRIP M&E program in guiding the successful implementation of the project and demonstrating the of the EINRIP approach to stakeholders? (S2) The degree to which issues identified through various M&E process have been subsequently addressed and resolved Stakeholder interviews: Identify a sample of issues and probe during interviews as to whether these have been addressed. (DFAT, DGH) Document review: Examine annual documents (AQC reports, M&E surveys) to test for recurrent issues Test utility of EINRIP M&E The degree to which data and analysis generated by EINRIP M&E processes has been influential in DGH Stakeholder interviews: Solicit views as to the lessons learned through EINRIP and how these have been communicated (DGH) Test influence of EINRIP M&E Efficiency: Does EINRIP represent for money, both relative to the quality of roads constructed and the institutional gains made? Evaluation question / sub question Evidence required Data collection method and source Analytical approach Was the modality (mix of loans and grant TA) relevant to intended EINRIP outcomes? (VFM1) The extent to which intended outcomes (stated and additional) reflect inputs The extent to which grant-funded activities directly support loan outcomes Document review: Identify between grant and loan-funded activities (project completion report, loan agreement, annual reports) Map links between grant and loan funded streams Stakeholder interviews: Solicit views as to the complementarity of grant and loan-funded activity. (DGH, DFAT) Synthesise feedback from stakeholders What are the early indications of the sustainability of both EINRIP-supported institutional capacity and completed roads and bridges? (VFM2) The extent to which EINRIP project roads, particularly those completed some time ago, are in good repair Document review: Examine assessments on road conditions (annual reports) Field Visit: Verify condition of a sample of EINRIP roads Synthesise data on the status of EINRIP roads Test for resolution of identified faults The extent of maintenance work required on EINRIP project Document review: Identify maintenance expenditure on EINRIP and non-EINRIP roads Compare expenditure levels
Stakeholder interviews: To understand DGH approach to road maintenance Describe road maintenance arrangements The degree to which EINRIP supported capacity-building measures remain relevant and in use As per question S1 above. Do EINRIP represent value for money? Taking into account any increases in traffic volumes, have the additional costs associated with project preparation and higher quality standards been effectively offset against levels of required maintenance expenditure and other economic benefits (e.g. reduced vehicle operating cost)? (VFM3) The extent to the planned and actual benefits from EINRIP roads exceed costs Document review: Examine feasibility study and 2015 of costs and benefits Document review: Identify project cost Stakeholder interviews and document reviews: Identify typical road construction cost in Indonesia Categorise roads according to net Present value & benefit-cost ratio Apportion grant costs to roads Compare EINRIP costs against Indonesian norms Update economic evaluation Total EINRIP costs including both loan and grant elements Road construction cost data for roads Data on maintenance expenditure, vehicle operating travel times, accident rates The extent to which the EINRIP approach influenced the construction/upgrading of other roads As per S1 above Cross cutting issues: Did ENRIP road packages take adequate account of cross-cutting issues? Evaluation question / sub question Evidence required Data collection method and source Analytical approach Main question Did the design and implementation of EINRIP take adequate account of needs of women as road users and facilitate their Whether the needs of women were taken into account explicitly or implicitly The extent to which the strategy faithfully implemented Document review: To assess the consideration of gender issues in planning and reporting (social safeguards plan, loan agreement, M&E manual, social surveys, annual reports, AQC reports) Synthesise treatment of gender issues
involvement in project delivery? Stakeholder interviews: To triangulate evidence gathered and gauge perceptions as to the priority placed on gender Synthesise and triangulate feedback from stakeholders Were other key safeguard and cross-cutting issues, such as resettlement, road safety, environment and HIV/AIDS, given appropriate attention? The extent to cross-cutting issues were identified and suitable strategies developed in the planning stage Whether these strategies were faithfully implemented, monitored and reported against The extent to which strategies were adapted to address any emerging issues Document review: to assess the consideration of resettlement, environment, road safety and HIV/AIDs in planning and reporting (social safeguards plan, loan agreement, M&E manual, social surveys, annual reports, AQC reports) Synthesise treatment of each of the four key cross-cutting issues Stakeholder interviews: to triangulate evidence gathered and gauge perceptions as to the priority placed on key issues Synthesise and triangulate feedback from stakeholders