responsibility for the implementation and management of the SSFGS has been delegated to the TOC team. There is still a significant reliance by schools on the support, advice and guidance provided by the TOC team. No other person in MESC has a complete understanding of the detail and complexity of the scheme. With the exception of the Team Leader, the TOC team is no longer employed. The Team Leader now has responsibility for both the primary and secondary school grants schemes, and does not have the time to provide all the support that is necessary for primary schools. Further, her contract is due to finish in July 2016. Reliance on a single person for management responsibility means that the whole scheme is vulnerable. The establishment of 12 School Review Officer positions has been approved and budgeted for, but the appointment process for the positions has not begun. Of equal concern is the need for leadership of the SSFGS. It is too large a task to be subsumed into the role of the ACEO of the Schools Operation Division to which position the SROs will be responsible. Recommendations have been made elsewhere to help ensure that the risk to sustainability outlined above is managed. 5 Evaluation Conclusions This section summarises the conclusions of the Evaluation, on the analysis in the previous chapter. Without doubt, schools have benefited significantly from the SSFGS (the Scheme). The analysis of data collected against the Development Assistance Committee criteria (relevance; effectiveness; efficiency; impact; and sustainability) found that the Scheme generally worked well, provided value for money, and supported schools in achieving their educational objectives. The benefits include that: Financial barriers have been reduced though not eliminated for all parents. Over 200 children previously not attending school are now enrolled. Principals, in association with their school committees, have become financial managers as well as professional leaders. Training has been provided and most principals are now competent financial managers.
The SSFGS gives schools direct access to resources needed to support teaching and learning, unlike the use of parental fees in the period before the introduction of the scheme, where funds were spent mainly on repairs and maintenance. Principals report that the introduction of the scheme since 2010 has supported better teaching materials and student learning in schools. The devolution of responsibility for financial management of schools has brought benefits such as improved self-management of schools, although the current highly centralised management of schools by MESC (for example, centralised control of teacher appointments) prevents principals from exercising more autonomy and introducing initiatives (such as more flexible use of staffing) that could potentially improve school performance. Schools should continue to encourage parents to enrol children in school, and to keep children enrolled throughout the school year. A strong point of the scheme has been that new students have been enrolled at all levels in Samoan primary schools each year. A focus on student attendance and on numbers of students attending school needs to be maintained. The SSFGS has led to improved accountability by schools for the use of school funds, and better communication with parents and school communities about how the funds are used. In general, school grant funds are being spent on the desired educational objectives. The latest figures (for the financial year July 2014 to June 2015) indicate that 25% of expenditure (nearly WST4 million) has been spent on teacher materials and student materials, 27% on school administration, 30% on fixed assets, 12% on consumer tax (CT) and 6% on repairs and maintenance. Training of principals as school leaders and as managers of grant funds is critical, and must be regular and ongoing. Leadership by the principal is a key factor in determining effective school performance. The links between school improvement plans, identified priorities for the school, the use of grant funds, and the Minimum Service Standards are important dimensions of effective school planning. Teaching principals in primary schools in (especially in larger schools) now have a heavy administrative workload which adds to the pressures of the position. Principals need ongoing support to carry out their essential functions of managing the performance of staff and establishing benchmarks to assess student achievement. Ongoing support is also needed to assist principals with the application and interpretation of the Minimum Service Standards. Professional development of staff (principals and teachers) is essential in order to improve teacher quality and the implementation of the new school curriculum.
Anticipated key deliverables and delivery dates are as follows: No. Outputs Description Due date Phase One 1 Briefing The evaluation team will participate in a briefing (via teleconference) with the Steering Group (MFAT, DFAT and GOS) to discuss objectives, plans and expectations of the evaluation. tbc 2 Document Review (Team) Desk Based activity – Literature and documentation review to establish understanding of context and challenges and identify information which needs to be collected during the in country component to address the scope of services and terms of reference of the evaluation. This will form part of the final Evaluation Report. Key documents will be provided by MFAT, DFAT, MESC, MOF. tbc 3 Develop and Finalise Evaluation Plan (Team) A detailed work plan for the overall evaluation, which includes an Evaluation Plan prepared in accordance with Terms of Reference and submitted for approval by the Steering Group (MFAT, GOS, DFAT). tbc Two 4 In Country Briefing (Team) Evaluation team will host a briefing with the Steering Group - MFAT, DFAT, GOS and the local consultant to go over any last minute changes and or questions. tbc 5 In Country Mission (Team) Field work including travel: Site visits to schools under the SSFGS programme (Upolu and Savaii) Before 31 2015 6 Presentation/ Workshop Present initial key findings & conclusions to key stakeholders (Aide Memoire) tbc 7 Draft Evaluation Report Draft report circulated to stakeholders for comments within two weeks of the Aide Memoire tbc 8 Final Evaluation Report Acceptance/approval by the Steering Group (MFAT, DFAT and tbc
Minimum Service Standards (MSS) are outlined, and the MSS are linked to the SSFGS. A section of the programme design document describes how the monitoring, evaluation and reporting on the scheme will be undertaken. Other dimensions such as risk mitigation, and programme sustainability are described in detail. 6es cover School Level Financial Procedures and Accountability, the Monitoring and Evaluation Framework, and a table that sets out percentages of primary students identified as being at risk. The Evaluation will cover the intention and logic of the design and its implementation, and will comment on the thinking and logic that underpinned the design and delivery of the programme. The linkages and relationships between the various participants in the programme will be examined in this context. This evaluation has been designed to enable the collection of relevant information across the evaluation objectives in order to assess the extent to which the goal, objectives, outcomes and outputs of the partnership have been achieved. 3.3 Evaluation Methodology The methodology to be used in this Evaluation was set out in detail in the response to the Request for Quote submitted by Vince Catherwood & Associates Ltd. The evaluation will be conducted by a team of three people: Vince Catherwood (Team Leader), Lester Taylor, and Nimera Taofia (Samoan counterpart). The evaluation will be based on available documentation (document reviews) and on interviews with stakeholders including MESC Principals, and School Committee members. Details of the evaluation techniques and strategies are included in this Evaluation Plan, and the Evaluation Plan will be included as Appendix 2 in the final report. The evaluation methodology will cover all the dimensions listed in the terms of reference under “Evaluation Methodology”. These are elaborated in more detail in the following sections. We propose that the conduct of the evaluation is undertaken in two main phases, based on the structure of the Evaluation Deliverables set out in the terms of reference on page 11. Each of these two phases includes a number of specific components. Phase One Phase One is an initial planning, scoping and analysis phase. This phase includes • A Preliminary Briefing (via teleconference) with the Steering Group. The purpose of the preliminary briefing is to discuss objectives, plans and expectations of the evaluation. • A Document Review. The document review will examine and analyse available literature and reports, including the key documents provided by Development Partners and by Ministries in Samoa such as the
The following table identifies potential risks for the evaluation process, and suggests mitigation strategies for managing those risks. Table 6: Risks and Risk Management Strategies The table below sets out risks that have been identified and the risk management strategies that are proposed to mitigate those risks. Risk Risk Management Strategy Limitations of documents/ reports in providing information Difficult to access documents Bias in selection of documents Lack of comprehensive coverage L M M Seek missing documents from sponsor or partners. If unavailable, plug any gaps through interviews with appropriate stakeholders who can provide the necessary information. Slippage in the tight timeline will affect timely delivery of outputs H M M Appoint a liaison counterpart from MESC to assist the Evaluation Team with retrieval of information and organisation of school and other interviews Unavailability of key stakeholders for interview (e.g. absence overseas, illness) M M M Use telephone interviews if face-to-face interviews not possible. Seek alternatives who could provide a similar perspective, in discussion with NZHC and Steering Group. Unwillingness by participants to L H H Use an informed consent process to ensure participants have an understanding of the purpose and benefits of the evaluation. Work with peers in-country to achieve full participation. Evaluation design faulty or inadequate, leading to premature conclusions L H H Undertake pilots of interview schedules. Review design if necessary. Ensure communication with NZHC and the Steering Group is frequent and effective
Interviews do not provide sufficient information M M M Reinterview if necessary. Redesign interview schedule following pilot. Ensure interview notes fully reflect feedback. Seek alternative sources of information if possible. Breach of individuals’ privacy or confidentiality L L H Use informed consent procedures to ensure participants are no individual will be identified in reports Relationships damaged through evaluation process (e.g. through adverse findings that may be disputed) M H H Use of experienced consultants should ensure issues with potential for conflict can be handled sensitively. Triangulate findings to ensure accuracy. Communicate sensitivities to NZHC and Steering Group. Unanticipated delays (e.g. getting responses to invitations from Head Teachers) affect timing of field work interviews and /or delivery of draft or final reports M H L Use programme management skills to ensure programme to timeline. Report to NZHC and Steering Group to discuss alternative strategies if slippage unavoidable. Delays in feedback to consultants from MESC, DPs or NZ Aid Programme L M L Effective communication of response deadlines. Timely reminders. Discuss alternative strategies with NZHC and Steering if necessary. Impact of natural disasters M H H Seek advice from NZHC and Steering Group if completion of evaluation is compromised 5.4 Governance Arrangements The Evaluation will be conducted the supervision of an Evaluation Steering Group. The Evaluation Steering Group will be responsible for overall governance of the evaluation. The Evaluation Steering Group consists of the following people: Perenise Stowers, Development Programme Co-ordinator, Education, MFAT; Ingrid Van Principal Evaluation Manager, MFAT; Vena-Liz Upton, Senior Programme Manager, (Education) DFAT;
A combination of some or all of these?_____________________________________ Do you have any other suggestions for how payments for school operations can be made more sustainable in the longer term?
Following the interview programme, the information and data gathered was collated and analysed. From this analysis an aide memoire was developed. A workshop for invited stakeholders to present the preliminary findings of the evaluation for discussion and comment was held on Friday 4 December 2015. The main conclusions and recommendations of the evaluation were
Appendix 7: Monitoring and Evaluation Framework A MESC document entitled Monitoring, Evaluation and Reporting for the Samoa School Fee Grant Scheme sets out the full advice on monitoring, evaluation, research, reporting and sustainability. The specific part of this document relating to the monitoring and evaluation framework for the SSFGS is reproduced below. Objective Indicator Source of Information Responsibility Reporting/Timing Increased primary enrolment and retention SSFGS schools see rolls rise and dropouts decrease National enrolment and retention figures rise Schools admissions books; discussions with principals and school committees. National statistics TOC/schools TOC quarterly reports Primary school performance improved against Minimum Service Standards All SSFGS schools record at least three improvements against MSS each year Self assessment by principal and school committee against standards TOC analysis of school reporting Principals TOC Annual School Performance Report TOC reports; annually Output/target Indicator Source of Information Responsibility Reporting/Timing Compliance in Schools Financial reporting by all schools is regular, School financial records maintained Cash books Principals Principals quarterly / six monthly reports