1.6.2.2 Types of evaluation Table 2: Summary of types of evaluation across government 1 Type of evaluation Covers Timing Diagnostic Evaluation This is preparatory research (often called ex-ante evaluation) to ascertain the current situation prior to an intervention and to inform intervention design. It identifies what is already known about the issues at hand, the problems and opportunities to be addressed, causes and consequences, including those that the intervention is unlikely to deliver, and the likely effectiveness of different policy options. This enables the drawing up of the theory of change before the intervention is designed At key stages prior to design or planning Design evaluation Used to analyse the theory of change, inner logic and consistency of the programme, either before a programme starts, or during implementation to see whether the theory of change appears to be working. This is quick to do and uses only secondary information and should be used for all new programmes. It also assesses the quality of the indicators and the assumptions. After an intervention has been designed, in first year, and possibly later Implementation evaluation Aims to evaluate whether an intervention’s operational mechanisms support achievement of the objectives or not and understand why. Looks at activities, outputs, and Once or several times during the intervention
amount to the total quantity of outcome generated, to produce a “cost per unit of outcome” estimate (e.g. cost per additional individual placed in employment); and • cost-benefit analysis (CBA), which goes further in placing a monetary value on the changes in outcomes as well (e.g. the value of placing an additional individual in employment).1 Evaluation synthesis Synthesising the results of a range of evaluations to generalise findings across government, e.g. a function such as supply chain management, a sector, or a cross-cutting issue such as capacity. DPME will undertake evaluation synthesis based on the evaluations in the national evaluation plan and do an annual report on evaluation After a number of evaluations are completed This draws from the Mexican experience, Castro et al (2009): “Mexico’s M&E System: Scaling Up from the Sectoral to the National Level”, ECD Working Paper Series, Independent Evaluation Group, Washington, World Bank. Figure 1: Evaluation Pyramid
The change in client behaviour, systems or institutional performance because of the combined output strategy and key assumptions. The value, benefit and return on the investment. Outputs The actual deliverables. What the operation can be held accountable for producing. Output indicators that measure the goods & services finally delivered by the operation. People, events, processes, sources of data. Supervision & monitoring system for validating the operation’s design. (Output-to-Purpose) Risks regarding design effectiveness. Activities The main activity clusters that must be undertaken in order to accomplish the Outputs. Inputs/Resources Budget by activity. Monetary, physical & human resources required to produce the outputs. People, events, processes, sources of data. Monitoring system for validating implementation progress. (Activity-to-Output) Risks regarding implementation & efficiency. CHAPTER 2: INSTITUTIONALISATION OF THE PERFORMANCE MONITORING AND EVALUATION SYSTEM Presidential PME processes: Improving Government performance In 2005 Cabinet approved an implementation plan to develop a Monitoring and Evaluation system for use across government to encompass: Monitoring, implementation, effectiveness, and validation; Evaluation, impact and process evaluation; Early warning through proactively identifying blockages; Verification which validates integrity of data; Data collection using existing capacities; Analysis and research-driven assessments; and Reporting which is appropriate & customised to target groups.
SUB-PROGRAMME: DISTRICT MANAGEMENT 3.5.1 PROVINCIAL STRATEGIC OBJECTIVES INDICATORS AND ANNUAL TARGETS FOR DISTRICT MANAGEMENT TABLE DHS3: PROVINCIAL STRATEGIC OBJECTIVES, PERFORMANCE INDICATORS AND ANNUAL TARGETS FOR DISTRICT HEALTH SERVICES BUDGET SUB PROGRAMME: DISTRICT MANAGEMENT STRATEGIC GOAL 4:RE-ENGINEER PRIMARY HEALTH CARE TO CREATE ACCESS TO QUALITY SERVICES. Strategic objective Performance indicator Strategic Plan target Means of verification / Data Source Audited/ Actual performance Estimated performance Medium term targets 2010/11 2011/12 2012/13 2013/14 2014/15 2015/16 2016/17 2.1 Improve service delivery programmes Number of Ward Based Outreach Teams appointed 356 teams Letters of Appointment & PHC Reports New indicator (No Data) 47 42 70 144 268 354 Households (OHH) registration visit coverage (annualised) 100% DHIS & Service Registers New indicator (No Data) New indicator (No Data) New indicator (No Data) 16% (127 750 / 823 316) 26% (216 000 / 823 316) 49% (402 000 / 823 316) 9% (77 566 / 823 316) School ISHP coverage 100% DHIS, School Health Services reports & DoBE New indicator (No Data) 72.6% 91.14% 70% 95% 100% 100% of ISHP quintile 1 and quintile 2 schools visited by school health teams 95% (883/929) DHIS, School Health Services reports & DoBE 43% 60% (542/903) 91.1% (823 / 903) 50% (481/961) 60% (557/929) 80% (745/929) 95% (882/929) School Grade R screening coverage (annualised) 100% DHIS, School Health Services reports & DoBE New indicator (No Data) New indicator (No Data) 2.99% 12.93% (4 171 / 32 251) 17.78% (5 735 / 32 251) 24.9% (8 029 / 32 251) 31.12% (10 036 / 32 251)
Table 7: Example of reporting framework (logframe) on performance indicators with baseline data, quarterly targets and performance according to Treasury guidelines. Programme / Subprogramme / Performance Measures Audited Outcome for 2012/13 as per Annual Report Target for 2013/14 as per Annual Performance Plan (APP) 1st Quarter Planned output as per APP 1st Quarter Actual output - validated 2nd Quarter Planned output as per APP 2nd Quarter Actual output - validated 3rd Quarter Planned output as per APP 3rd Quarter Actual output - validated 4th Quarter Planned output as per APP 4th Quarter Actual output - validated Actual output for 2013/14 Comments for the 4th Quarter QUARTERLY OUTPUTS Programme 2: District Health Services ● Provincial PHC Expenditure per uninsured person R 410 R 514 R 514 R 489 R 514 R 451 R 514 R 469 R 514 R 439 R 462 Target achieved ● PHC Utilisation rate 2.5 3.0 3.0 2.4 3.0 2.2 3.0 2.1 3.0 2.4 2.3 Refer to narrative report ● PHC Utilisation rate under 5 years 3.8 visits 3.8 visits 3.8 visits 4.2 visits 3.8 visits 3.5 visits 3.8 visits 3.3 visits 3.8 visits 3.9 visits 3.7 visits Target achieved ● PHC supervisor visits rate (fixed clinic/CHC/CDC) 86.0% 100.0% 100.0% 87.5% 100.0% 72.4% 100.0% 66.2% 100.0% 77.5% 227.7% Refer to narrative report ● Complaint Resolution within 25 working days rate 78.3% 85.0% 85.0% 72.2% 85.0% 30.1% 85.0% 98.4% 85.0% 83.4% 48.7% Refer to narrative report