Table 3: Component 1 key findings and recommendations Findings Recommendations Level of urgency Prime Responsibility The extent to which the EOPO is achieved will depend upon maximising EOCO contribution as well as encouraging the contribution of other access-related initiatives (see §2.4.3 and 2.4.4) R1: DFAT and MoEC should reaffirm their commitment to the current EOPO. *** DFAT and MoEC R2: Critically appraise coordination of school construction with other access-related initiatives within MOEC with a view to maximising the extent of EOPO achievement. *** DFAT and MoEC (with SSQ and POM support) is evidence of sub-optimal location of some selected sites and improving site selection is both possible and desirable (see §2.4.3) R3: Further analysis of site selection outcomes should be undertaken to provide a robust basis to continue to improve the selection process. *** POM (with MoEC and SSQ support) R4: DFAT and MoEC should strengthen the site selection process by refining factors of JSE demand and supply in the selection criteria, and ensuring a strict application of these criteria. *** DFAT and MoEC (with SSQ support) A number of lessons on construction processes are being learned through the EP investment (see §2.4.5) R5: Lessons learned from EP implementation (e.g. regarding monitoring process) should be synthesised and disseminated to interested parties (e.g. MoEC and other GoI institutions/agencies involved in construction processes), should there be merit or demand. * MoEC and SSQ NB: Red (***) - high urgency; orange (**) - medium urgency; green (*) - low urgency
What does this mean for the EP? The bottom-line A summary table of component performance by DAC criteria is presented in Table 5. The ratings scale used with the DFAT QRS (QAI and ICR) has been adopted. Recommendations for improvement and, in particular, on increasing the likelihood of achieving EOPO 2 are made in Table 6, below.
6: Component 2 key findings and recommendations Findings Recommendations of urgency Prime Responsibility Given delays to the implementation of the GA, the intended program outcomes may only be achieved after 2016 (see §3.4.3) R6: DFAT-MoEC assign sufficient resources to C2 so as to ensure timely delivery of GA results, or should readjust the EOPO and implementation strategy accordingly. *** DFAT and MoEC (with SSQ support) GoI’s political commitment to the PD system remains unclear (see §3.4.1) R7: MoEC should lobby and advocate for the PD system to be included in strategic documents (e.g. the RPJMN and the upcoming Renstra), thereby enhancing political commitment to its implementation. *** MoEC (with DFAT support through ATOG and GOG; and SSQ support) Unknown ability and willingness of central, provincial and/or district government to finance the system or pay for services provided by it the GA (see §3.4.1 and §3.4.5) R8: Institutional Partners must better understand the financial capacity and willingness of district, provincial and central government levels to fund PD services, and develop a workable sustainability strategy. *** MoEC and DFAT (with POM and SSQ support) NB: Red (***) - high urgency; orange (**) - medium urgency; green (*) - low urgency
Table 9: Component 3 scorecard DAC Criterion 2013 Justification Comments Relevance 4 Most factors of Madrasah performance are covered in the EP design. Weak consideration/ focus on community engagement. Expected changes at systemic level are not well delineated. Factors influencing Madrasah performance have been defined based on a preliminary analysis of Phase 2 activities. Block grant socialisation is thought to be instrumental in stimulating community engagement. Efficiency 5 Implementation is contracted to the SSQ managing contractor; there are no substantial delays and disbursements are on track. Efficiency of support could be increased through targeting madrasah with lower initial standards. The efficiency of the outsourcing model could be examined as part of a VFM assessment. Effectiveness 4 Tangible and observable incremental gains are evident at Madrasah level, including in some non-participating Madrasah. In the absence of clear systemic change, the EP benefits may be restricted to individual Madrasah. Results of the BAN S/M accreditation process might soon highlight the extent of the effectiveness of Madrasah support. Impact TE It is still too early to confidently assess the impact of EP activities – the first phase of support has just been finalised in December 2013. Sustainability 2 Lack of clarity about achievements and benefits to be sustained, and absence of strategy compounded by project modality. UPPAM mandate inconsistent with level of GoI resources. Uncertainty about BAN-S/M resources and capacity to include Madrasah in the accreditation process beyond EP life. With the of the second half of the EP period, considerations about the sustainability of EP achievements and their benefits become crucial. Table 10: Component 3 key findings and recommendations
Table 12: Component 4 scorecard DAC Criterion 2013 Justification Comments Relevance 4 Investment in policy processes remains highly relevant but ACDP’s relevance will be undermined if/where ‘windows of opportunity’ are missed. Efficiency 3 Not infrequent replacement of consultants Delays to submissions Variable quality of reporting Effectiveness 4 The long lifecycle of many ACDP activities is seemingly inconsistent with the political realities of decision-making Nevertheless, examples of evidence utilization can be cited, e.g. ACDP 001. ACDP should consider options to better bundle findings, i.e. so evidence is provided by theme and not just by report-by-report Impact TE Likelihood is unknown and depends on a number of factors. As such, it is very difficult to provide an accurate, defensible score The likelihood of a number of ACDP outputs having impact-level benefits is very uncertain but should one have a direct, significant and meaningful effect on a key policy the impact of the ACDP could be significant. ACDP’s six-monthly reports benefit from improved tracking/reporting of post-report measures and results. POM’s upcoming evaluation study will support understanding of policy processes and ACDP’s impact in that regard. Sustainability 4 Sustained benefits will accrue should key report recommendations be implemented. Insufficient routine capture, tracking and reporting of changes to the policy environment completion of individual activities limits the extent to which POM can state with confidence that (specific) sustained benefits are likely. The modality – the ACDP delivery vehicle – will not be sustained and is not expected to out-live the EP. The extent to which benefits (e.g. changes to policy content and subsequent implementation) are likely to continue after ACDP closure depends, in part, on the extent to which ACDP and its individual activities have successfully ushered in improved understanding of and attitudinal change and created ‘space’ for meaningful policy-related discussion about the issues.
Table 13: Component 4 key findings and recommendations Findings Recommendations Level of urgency Prime Responsibility The ACDP remains insufficiently agile and responsive to the needs of its client (see §5.4.2) R14: ACDP should continue to revise and diversify its product offering (activity such that it is responsive and timely, and offers value for money. ** ACDP (with guidance from the ATOG) ACDP processes and events offer DFAT opportunity to engage in more strategic dialogue with key GoI officials but this potential is under-exploited (see §5.4.3) R15: DFAT should better articulate how, when and why it seeks to gain strategic value from its investment in the ACDP. ** DFAT (with ACDP support as required) EP implementing partners (e.g. SSQ and POM) have a patchy understanding of upcoming ACDP events (see §5.4.3) R16: DFAT/ACDP should make a concerted effort to identify cross-component links and to flag relevant upcoming events such that ACDP becomes a more integral part of the EP and that synergies are exploited. * DFAT and ACDP NB: Red (***) - high urgency; orange (**) - medium urgency; (*) - low urgency
Less than satisfactory Satisfactory 1: Very poor quality; needs major overhaul 4: Adequate quality; needs some work 2: Poor quality; needs major work to improve 5: Good quality 3: Less than adequate quality; needs significant work 6: Very high quality TE: Too early to rate
4 Sustainability To what extent are EP benefits likely to be sustained beyond the life of the EP? Has sustainability of EP benefits been adequately considered in the design of the interventions? Have specific constraints to sustainability been identified? Do partner institutions demonstrate a level of ownership? Has the level of funding to sustain benefits been considered and is it likely to be forthcoming? 5 Beneficiaries How have EP interventions impacted upon beneficiaries? Have the direct and indirect beneficiaries of EP interventions been identified? What are the expected and unexpected impacts on these beneficiaries? Which potential beneficiaries are excluded? EOPO 6 Enrolment in Junior Secondary Education To what extent have EP interventions contributed to increase enrolment in participating district? What is the evidence of increased JSE enrolment in participating districts? How do EP interventions contribute to increase in JSE enrolment in participating districts? Are EP interventions influencing motivation and incentives for student enrolment? Have EP interventions improved district capacity to plan and supply access to JSE? Has the EP contribution been sufficient to justify the investment? 7 Management of schools and Madrasah To what extent has management of schools and Madrasah improved in participating districts? What is the evidence of improved management of schools and Madrasah in participating districts? How do EP interventions contribute to improvement in schools and Madrasah management? What’s the role of improving P and S competencies? Has the EP contribution been sufficient to justify the investment?
8 Quality of Madrasah service provision To what extent has the quality of Madrasah service provision improved (in line with NES) in targeted provinces? What is the evidence of improved quality of Madrasah service provision in targeted provinces? How do EP interventions contribute to improved quality of Madrasah service provision in targeted provinces? How have SNIP and Block grants improved the quality of Madrasah service provision? What are the key factors of changes in community engagement or perception Madrasah quality? Has the EP contribution been sufficient to justify the investment? 9 Utilisation of analytical work in education sector policy, planning and budgeting To what extent have policy-makers utilised analytical work to inform education sector policy, planning and budgeting? What is the evidence of increase utilisation of analytical work to inform education sector policy, planning and budgeting? How do EP interventions contribute to increase utilisation of analytical work to inform education sector policy, planning and budgeting?? Has the EP contribution been sufficient to justify the investment? Social Inclusion 10 Gender To what extent has the EP promoted gender equality in the Indonesian education sector? Is it apparent gender-related issues the EP is seeking to tackle? Are the targets and targeting measures appropriate? What are we learning about the different impacts of EP interventions on women and men, girls and boys? Are EP interventions facilitating increased women’s empowerment and voice in decision-making and leadership?
11 Disability To what extent has the EP promoted disability inclusive education in Indonesia? Is it apparent which disability-related issues the EP is seeking to tackle? Are GoI and GoA policies related to disability being reflected in EP workstreams. Are the targets and targeting measures appropriate? What are we learning about the different impacts of EP interventions on people/children with disabilities? 12 Poverty To what extent has the EP targeted poor people? Given GoA’s policy focus on poverty reduction: How have different components targeted poor people? How successful has the EP been in targeting poor people? Who is capturing the benefits? Is that in line with expectation? Are the targets and targeting measures appropriate? Value for Money 13 for Money To what extent does the EP represent Value for Money? What do we mean by “value for money” in the EP? Are there intra-component differences? How is the EP tracking against agreed metrics? What related considerations do we need to be mindful of, e.g. sustainability, comparability, relativity, risk, and broader GoA policy objectives?
Yaya Kardiawarman Senior Professional Development Advisor, SSQ Alison Atwell International Adviser Provincial and District Development, SSQ Abdul Munir Component 3 Manager, Madrasah Accreditation, SSQ Tjipto Prakosa Data Analyst & Training Advisor, SSQ Mokhamad Iksan Sub-National Institutional Capacity Development Advisor, SSQ John Virtue Education Sector Governance & Capacity Development Advisor, ACDP Alan Prouty Team Leader (Operational Management Specialist), ACDP David Harding Senior Education Advisor, ACDP Basilius Bengoteku Education Sector Research and Capacity Planning Advisor, ACDP Abdul Malik Education Sector Research, Information & Accountability Advisor, ACDP Bambang Indriyanto Director, Center for Policy Research, MoEC Karen Taylor Operations Manager, EPOS Sarah Gray Public Affairs Specialist, EPOS Joanne Dickinson English Language Learning Specialist, EPOS
Annex VII – ECBP M&E Standards Standard 3: Initiative Progress Reporting No Feature of Progress Report EP-APPR Compliance 3.1 There is an executive summary that communicates the key information required for QAI reporting See Executive Summary 3.2 The relevant aspects of the context are adequately described Each component section has an individual section on Context 3.3 There is a reflection on the continuing relevance of the expected end-of-program outcomes Each component section has an individual section on Relevance 3.4 An assessment of the adequacy of progress toward sustained end-of-program outcomes is described Assessment of progress against DAC criteria are scored for each component 3.5 The quality, reach and coverage of key outputs or deliverables for the reporting period are described Achievements are listed for each component 3.6 The adequacy of progress implementing the annual plan is described n/a (implementing partners provide reporting against annual plans) 3.7 A reflection on the adequacy of planned inputs to meet the expected end-of-program outcomes is provided Each component section has an individual section on Efficiency and Effectiveness 3.8 The adequacy of progress against the budget is assessed Not done 3.9 The efficiency and effectiveness of key management or implementation system is assessed or demonstrated Each component section has an individual section on Efficiency and Effectiveness and one section is dedicated to Governance and Management 3.10 The report achieves a fair balance between reporting of positive or negative issues or achievements Quasi-independence of POM lends itself to this 3.11 The report provides credible evidence of claims made Considerable evidence is presented to justify claims 3.12 Important lessons are summarised Findings and associated recommendations are summarised in each section