Budget AUD3 m AUD1.9 m AUD 1.9 m AUD1.9 m (DRR) AUD1.9 m (UXO) Figure 2: LANGOCA portfolio attributes Evaluation Objectives This review conformed to the standard AusAID practice of commissioning independent evaluations at the midterm of initiatives. The stated objectives of the IPR were to: Assess the overall performance of LANGOCA through two levels of analysis: (a) at the level of the Program’s overall objectives; and (b) at the level of the specific objectives of the funded activities; Recommend actions necessary to improve the performance of LANGOCA up to its completion date; Provide insights and lessons learned from the LANGOCA experience for consideration of future programming with NGOs, including options to strengthen the role of NGOs in the Australian development assistance program in Laos. The evaluation team achieved the objectives by reviewing key documents and obtaining the perspectives of relevant stakeholders about the program’s relevance, effectiveness, efficiency, impact, sustainability, monitoring and evaluation (M&E), gender and inclusive development, and analysis and learning. The IPR was conducted by a team engaged by AusAID. Two field teams comprised an independent M&E Specialist, an NGO representative appointed by the Australian Council for International Development (ACFID), two independent gender specialists, three AusAID staff (two from Vientiane Post, one from Canberra) and two GoL representatives. Scope & Methods In line with the requirement for a rapid evaluation, the IPR team used qualitative methods: specifically, key informant interviews; focus group discussions; field observations, and; document reviews. Documents were provided by AusAID and the NGOs as requested. Interviews were arranged by AusAID and the NGOs at locations convenient for the interviewees. A question guide (Appendix B) assisted dialogue with interviewees and assisted consistency of data collecting between the two field teams. Six broad classes of stakeholder were interviewed concerning the performance of LANGOCA: AusAID: program management staff and technical advisers in Canberra and Vientiane. GoL: GoL partners at national, provincial and district level. ANGO: Australia-based program management staff. Informed third parties: donor representatives and civil society representatives in Vientiane. Implementing partners: Laos-based management and technical staff from partner NGOs. structured
finalizing the review report after receiving comments and feedbacks. The second team member (potentially an ACFID-nominated NGO representative) The team member will have: (i) demonstrated experience in the monitoring, evaluation and assessment of development assistance activities in particular NGO program assessment and performance evaluation; (ii) demonstrated understanding of the socio, political and economic situation in Laos, including in respect to the GoL’s policies pertaining to civil society organizations and NGOs; (iii) strong teamwork skills; and (V) sound cross-culture knowledge. The team member will be responsible for: finalizing his or her own international travel for in-country mission; liaising with the team leader on the team’s work program and meetings schedule prior to the mission commencing in-country; working with the team leader on initial planning and review of relevant documentation as listed below; working with other team members on specific tasks during the mission; cooperating with the team leader to present and discuss the mission’s Aide Memoire at the workshop; and working with the team leader to finalize the review report after receiving comments and feedbacks. Local consultant/interpreter This member will have (i) demonstrated understanding of the socio, political and economic situation in Laos, including in respect to the GoL’s policies pertaining to civil society organizations and NGOs; (ii) demonstrated strong knowledge in institutional capacity building/development and community development, particularly in terms of the role of civil society organizations in a country like Laos. This team member will be responsible for interpretation services and assisting the team leader to undertake in-country activities leading to the final production of the review, under the direction of the team leader. AusAID Canberra participant An AusAID Canberra officer from the Community Partnership Section will support the review team to facilitate discussions relating to AusAID’s policies and guidelines on cooperation with NGOs and feed-in lessons learned from similar reviews in other countries. AusAID Vientiane participant/s This officer/s will provide local knowledge and necessary support to the review team as required. Government of Laos (GOL) participant This government officer will most likely come from the Ministry of Foreign Affairs’ Department of International Organizations. 8. REPORTING REQUIREMENTS
The review team will produce the following papers: An Issues Paper to be developed in phases 1 and 2, discussed with LANGOCA NGOs, and sent to AusAID Vientiane for circulation to relevant stakeholders before the in-country visit commences; An Aide Memoire for the end-of-mission workshop; A draft Review Report (electronically) in a format outlined in the Issues Paper. The draft Review Report will be marked as draft and will have the revision date on the cover. The team leader should submit the draft Review Report to AusAID Vientiane. Review Report (2 hard copies and electronically) in a format outlined previously in the Issues Paper. The team leader should submit the final report to AusAID Vientiane within five working days of receiving comments and feedback, in writing, on the draft review. The team leader is responsible for preparing and submitting the above documents in consultation with the other team members. AusAID will have ownership of all documentation. 9. TIME LINES Activities and Mile Stones April 2011 May 2011 June 2011 In-Australia Phase Desk review in Australia x Issues paper draft 1 in Australia x Consultations in Australia x Issues paper draft 2 for Laos x In-Laos mission Briefing with AusAID x Vientiane-based meetings x x Field visits (at least 4 NGO activity sites) x x x x Aide memoire drafting x End-of-mission Workshop x In-Australia Mid-term review report writing and finalization x x 10. READING DOCUMENTS The review team should consider all relevant documentation, including the following:
for area considered one of 47 poorest Effectiveness Constructive process of village participatory planning fosters ownership Integrated ‘livelihoods approach’ to UXO work has development integrity Integration of LANGOCA project within CARE’s wider provincial program adds value Nuanced approach to MRE which segments community risk behaviours and enables targeted messages Appropriate ‘partnership’ approach to training delivery with relevant GoL actors encourages technical quality Beneficiaries report use of practical training methods including games and demonstrations Some ethnic language speaking staff ‘Problem-based’ approach to participatory village planning engenders CARE-dependency compared with strengths-based approach which empowers autonomy There may be scope for further refining of training methods to ensure full use of interactive and adult-appropriate approaches, especially given high levels of illiteracy; also develop post-training support/mentoring systems Questionable/circular design logic: UXO clearance improve livelihood opportunities reduce risk of UXOs; weak causal links between some activities and objectives/purpose (e.g. GFS reduced vulnerability to UXO) Wide array of seemingly discrete activities makes management difficult, community engagement piecemeal and fragments design coherence Efficiency Have cleared more than planned 20 ha UXO contaminated fields Provincial program appears professionally managed and organised Work plan proactively accommodates wet/dry season engagements Work plan behind schedule, but making progress Some delays attributed to known factors (e.g. remoteness) that should be factored into planning Wide array of activities requires higher level of staffing Uncertain value for money/effort of fish pond activity given extent of effort for single family benefit Impact Coffee should yield good source of long-term income Wheelbarrow distribution appreciated; created immediate positive impact on women’s exertion GFS providing year-round clear water for household use Evidence of ‘diffusion of innovation’ (coffee nurseries, fish ponds) Kitchen gardens appear to be an undervalued activity. May be important in cases where farmers substitute rice for coffee (nutritional value of kitchen garden overshadowed by economic value of coffee?) Kitchen gardens should promote appropriate alternative starch food to mitigate risks of coffee substitution for upland rice production Diffuse array of activities may erode potential project impact
Met Village(s) 70 Beneficiary target 3 PDMC, 3 DDMCs, 60 VDMCs Villages, households Total budget (AUD) AUD2 million Current % spent Approx 55% Life of project (months) Will require extension Current % past History of engagement Project aligned under LANGOCA DR objective Significant disruption (staff reassigned for 6 mo) but also benefits of Ketsana Oxfam engaged in Ta Oy before LANGOCA; other locations new Evaluation Criteria Relative Strengths Relative Weaknesses Relevance Strong engagement with District partners PRAs used to develop comprehensive picture of community needs Project addressing range of felt needs prioritised by target communities Target villages in poorest 47/142 districts prioritised by GoL Inclusion of mine education in relevant areas Limited donor consciousness among communities (and to a lesser extent district staff) SRI only relevant for paddy farmers (tend to be wealthier) Beneficiaries essentially defined at village level (although useful wealth-ranking process in PRA) Effectiveness Broadly expected to achieve objectives (DM capacity, food security, health/wellbeing) Livelihood/resilience approach to DM relevant Possible synergies between some activities (SRI, rice banks, irrigation, root crops, kitchen gardens) Appropriately working through GoL partners (a function of staffing limitations); training, coordination, GoL reporting Design balances need for project impact (service delivery) and longer-term strengthening of enabling environment (capacity building) Objective hazard mapping to identify UXO priorities Weak causal linkages between some activities and objective (DM); although broadly relevant to rural livelihoods Ambitious design: target villages & thematic foci Wide array of activities fragments design coherence High number of target villages (70) reduces project focus No linkages with similar work by CARE, SCA & WV Variable integration of activities Narrow focus on training rather than more elaborate capacity building PRA process constructive but still grounded in problem-based analysis; strengths-based approach shown to be more empowering Efficiency From broad perspective: significant outputs from relatively little inputs Decentralising operations to field offices Staff with relevant qualifications Wide array of activities increases management complexity High staff turnover has affected progress and performance Difficulty engaging staff in
(health, agriculture) remote areas Will require no-cost-extension to achieve design targets Some pervasive/known factors cited as reasons for delay (remoteness, wet season, ethnic communities) Impact Some reported diffusion of SRI innovation Work with village midwives having direct impact on maternal health & cultural practices GFS and latrines having direct impact on health and female exertion levels Provincial Vice Governor reported that villages with VDMCs were better prepared and more responsive to Ketsana; planning to roll out VDMCs in all villages Use of hybrid seeds in remote communities risks unintentionally eroding food security if farmers retain/plant seeds Wide target area reduces potential for deep engagement with communities and therefore likely to foster less impact than more focussed projects Some technical/site-level issues noted (e.g. open drainage from water point in village) Sustainability Disaster committees integrated into existing leadership structures Project activities implemented through district government partners Evidence of ownership of activities (initiated by communities) Established conservation committees in water catchment areas No environmental protection strategy or assessments; no soil conservation methods promoted in agricultural activities Community representatives note good progress but cite insufficient capacity to proceed without Oxfam support Provincial government committed to VDMC model, but has no confirmed GoL resources No village graduation/exit strategy Uncertain sustainability of village volunteer model (HIV, health etc.) Inclusive development Community members (women and men) demonstrate basic appreciation for gender equality principles Work with village midwives having direct impact on women’s health and on cultural practices concerned with child birth Awareness raising on HIV/AIDS relevant given rapid road development to Vietnam border; active female & male HIV volunteers Gender analysis carried out for project activities No explicit/current strategy to promote/address disability within project activities Extent of gender equality awareness variable across villages (very strong to nascent) Gender analysis needs to be extended to monitoring & evaluation that identifies qualitative impact for women and girls and people with disabilities, both intended & unintended. M&E Village PRA provided valuable No systematic identification of
in Laos where Government involvement and commitment is crucial. Capacity building objectives seen as relevant by many stakeholders, including the NDMO. Resilience promotion also seen as important community need to reduce vulnerability to disaster and poverty reduction. usually small scale such as windstorm damage or animal disease but cause significant impact in very poor communities. However, project team should try to not be constrained by the disaster lens and deepen the efforts already begun on increasing resilience Beneficiaries very broadly defined and analysis of beneficiary circumstances somewhat limited. However this is being addressed by SCA’s proposed refocusing on women headed households and people with a disability. Effectiveness Objectives currently on track to being met. SCA stated that to date they have focused on capacity building component and over next two years will focus more at community level, but that overall they feel that they are heading in the right direction to meet the objective of promoting community resilience. Notable achievements include work with children’s groups and facilitation of multi-sectoral Government planning processes around DRR. DRR plans developed at the same time as the District Socio-Economic plans and seem quite well integrated as a result. This in turn strengthens both partnerships and activities. Government counterparts see main value of NGO partnership as being capacity building. NGO also acts as intermediary between Government and community. SCA staff expressed opinion that involvement of an NGO brings rigour/systems to No particular evidence that the LANGOCA program level design translated into more effective project work on the ground. Struggling partnership with ADPC. In 2010, ADPC could not provide technical assistance requested by SCA for 6 months. Partnership may need to be reviewed. Build on the livelihood and income generating activities that exist and strengthen these possibilities (e.g. irrigation, Job’s Tears production, chili market) Stakeholders would have appreciated the DRR timeframe to be in line with the socio-economic timeframe (i.e. through to 2015). This may be something for AusAID to consider for future programming.
There was evidence at all levels (within communities, children’s groups, DDMC, PDMC), of high levels of confidence. This was commonly attributed to the project. In the long-term, the involvement of and investment in children may prove to be significant. Where water systems were observed, systems were in place to ensure they could be maintained (water fees) Some DDMC members recognise their own persisting capacity limitations and have requested specific additional support, such as in agriculture. Overall, partner level capacity building efforts are primarily focused on men. However, this is due to the composition of the DDMC and PDMC, which are beyond the control of the project. Beyond funding, the efforts of a dedicated staff is most likely required if DRR efforts are to be sustained. DDMC members all have sectoral responsibilities and may not have the time or commitment to push DRR activities in the absence of SCA staff. Constant turnover of Government staff also reduces likelihood of sustainability of activities at Government level. Focus on youth is a good thing for future sustainability Inclusive development Discussions with project team suggest that comments from MEF processes about improving gender have been taken on board. SCA has lined up GDG to do gender and will follow up with training for PDMC and DDMC and their wives (capacity assessment is currently in process). Government partners are well versed in the rhetoric of gender work (though much less so the practice!). Project team provides good model for gender balance and equality (team is headed up by a woman and contains more women than men). There is little evidence that women are effectively involved in community level decision making (not just within the project, but more generally.) Project team cited learnings from CARE/Oxfam gender activities that were very basic (e.g. the need for separate FGD with men and women). There are no specific strategies for ensuring that people with disability can meaningfully participate in project activities. Although efforts were made to request the support of Handicap International in this area, the orientation so far received sounds very
baseline, it was not clear if this was being used to track progress over time Without the beneficiary consultations introduced by the MEF, it is not clear what system SCA uses to monitor outcomes at the project level, for example it was not clear whether the water systems implemented were being monitored in terms of: where taps are located; accessibility; women’s privacy; use of the water; for latrines, whether these were being used and kept clean; and for hygiene, whether this had resulted in behaviour change or health impacts, etc. Learning 5 SCA project team appear very responsive to suggestions and recommendations at all levels. Following the 2010 AEW and observed weaknesses around gender and disability, SCA have put in noticeable efforts to improve in these areas. Challenges in the area of learning relate to the extent to which LANGOCA is really flexible enough to accommodate such learning. Further dialogue with AusAID may be useful. Although project staff were able to cite examples of interactions with other LANGOCA NGOs, there is little evidence of any real cross learnings from other projects. From the point of view of the project, LANGOCA as a program of 4 agencies appears to be something of an artificial construct. H: ANGO World Vision Australia Project Name Integrated UXO Action Project