GoTL Contribution to monitoring of NAP-GBV and information relevant to implementation of GoTL programming Partners Information for learning and improvement KEY EVALUATION QUESTIONS AND SUB-QUESTIONS The KEQs were initially developed during the October 2017 workshop with further development in the April 2018 workshops. They provide an organising framework for Nabilan’s monitoring and evaluation approach, and will be used as a structure for progress reports. The KEQs and sub-questions are listed in Table 4 below. Table 4. Key Evaluation Questions and Sub-questions Evaluation Criteria Key Evaluation Questions Sub-Questions Effectiveness KEQ1 How effective was the Nabilan program in ensuring women and children who experience violence have access to adequate and responsive services? 1.1 To what extent are service provider staff using their knowledge and skills for improved service delivery? 1.2 To what extent did Nabilan contribute to high quality service delivery? 1.3 To what extent did Nabilan contribute to inclusive service delivery? 1.4 How has Nabilan strengthened referral pathways? 1.5 How has Nabilan contributed to improving the justice sector response to VAWC? Effectiveness KEQ 2 To what extent are individuals, communities and institutions taking action to reduce violence against 2.1 Is Nabilan effectively supporting partner agencies to undertake quality prevention programming? 2.2 How has Nabilan contributed to the independent movement’s advocacy on gender equity and ending VAWC? 2.3 To what extent has Nabilan influenced increased GOTL dialogue and action on preventing VAWC? 2.4 Is Nabilan effectively supporting Change Ambassadors to influence their programs on quality prevention approaches? 2.5 To what extent has Nabilan supported individuals, communities and institutions to recognise VAWC as a problem? 2.6 To what extent has Nabilan supported communities to take independent action to prevent VAWC? 2.7 To what extent has the evidence from the CBA influenced prevention programming by other agencies? Sustainability KEQ 3 What key factors will contribute to the sustainability of the Nabilan program outcomes beyond the end of the program? 3.1 To what extent is GOTL resourcing prevention activities and essential services for and children impacted by violence? How did Nabilan contribute to this? 3.2 Is there evidence of change of community attitudes and practices relating to VAWC?
Annex 1: Monitoring Evaluation and Learning Framework KEQ and Relevant outcome Indicators Corresponding PAF Indicator Data Collection Method Baseline Target KEQ1 How effective was the Nabilan program in guaranteeing women and children who experience violence have access to adequate and responsive services? EOPO1: Women and children who experience violence have access to quality and inclusive services Number of new clients supported by Nabilan partners. Disaggregation: gender, age, municipality, disability, type of case PAF Indicator 10: Number of additional people who receive services Partner reports Available - 2017 Number of new clients steady compared with 2017 levels Number of services provided by Nabilan partners Disaggregation: type of service PAF Indicator 12: Number of additional services provided Partner reports Available - 2017 Number of services provided steady compared with 2017 levels 1.1 To what extent are service provider staff using their knowledge and skills for improved service delivery? IO1.1: Service providers use increased skills, knowledge and sensitivity to deliver services ImO1.1 Key service provider staff have improved knowledge and skills Number of Nabilan partner and staff completing training (Certificate III in Social Services, Medical Forensic Examiner Training, 4R). Disaggregation: gender, disability, service. Training reports Available - 2017 Certificate III: 20 in 2018; 20 in 2019; 20 in 2020; 20 in 2021 4R: 30 in 2018; 45 in 2019; 45 in 2020; 45 in 2021 Medical forensic examiner training: 15 new examiners by end of 2019 Proportion of Nabilan-trained participants who Indicator 8: Number of people who Training reports (pre- and Available - 2017 At least 50% of participants in each training
demonstrate improved knowledge/skills. Disaggregation: gender, disability, service apply improved skills to deliver better quality services assessment, Certificate III workplace assessment) demonstrate improved knowledge/skills at end of training 1.2 To what extent did Nabilan contribute to high quality service delivery? IO1.1: Service providers use increased skills, knowledge and sensitivity to deliver services ImO1.1 Key service provider have improved knowledge and skills Case reviews showing increase in quality at service provider-level Indicator 9: Number of service delivery improvements at service level Case Management Assessment Tool ALFeLa Case Audit Available - 2017 By June 2019, at least 70% of cases meet minimum standards. By June 2021, at least 70% cases demonstrate high quality. Instances of Nabilan contributing to improved quality of support and legal services Indicator 7: Instances of improved service delivery at systems Stories of significant change N/A N/A 1.3 To what extent did Nabilan contribute to inclusive service delivery? IO1.1: Service providers use increased skills, knowledge and sensitivity to deliver services Instances of Nabilan contributing to partners delivering inclusive services for marginalized clients Indicator 7: Instances of improved service delivery at systems Stories of significant change N/A N/A Proportion of clients supported by Nabilan partners that are vulnerable Partner reports Available - 2017 Maintain or increase in proportion of clients with disabilities (and other vulnerable clients) compared with 2014 baseline 1.4 How has Nabilan strengthened referral pathways? IO1.2: Women and children experiencing violence are referred to appropriate services Number of women and children being referred to partner services. Disaggregation: referral source (from partners or Partner reports Available - 2017 5% increase in referrals from outside the referral network by 2020 10% increase in number of
ImO1.2 network service providers have knowledge and skills to respond appropriately to women and children who experience violence outside the referral network (DPOs; health workers; community leaders; self-referred)) referrals made between service providers 1.5 How has Nabilan contributed to improving the justice sector response to VAWC? IO1.4: The justice sector improves its response to VAWC ImO1.4 Partners and service providers advocate for improved justice sector response to VAWC Evidence of appropriate charging and sentencing in the formal justice system: number of DV cases resulting in suspended sentence with monitoring obligation; number of DV cases being charged as serious assault or mistreatment of a spouse; number of DV cases resulting in fine Court monitoring Available - 2017 Compared against 2017 baseline: increase in number of DV cases resulting in suspended sentence with monitoring obligation; increase in number of DV cases being charged as serious assault or mistreatment of a spouse; decrease in number of DV resulting in fine Number of cases of VAWC reaching the courts monitored by JSMP Disaggregation: case type; special features such as Court monitoring Available - 2017 Number of cases holding steady against 2017
defendant has disability, use of medical evidence; represented by ALFeLa Evidence of Nabilan partners and service providers advocating for improved justice sector response to VAWC Partner reports JSMP newsletter and thematic reports N/A N/A KEQ 2 To what extent are individuals, communities and institutions taking action to reduce violence against women? Individuals, communities and institutions take action to reduce violence against women and children Selected members at each level of the ecological model (individuals, communities, institutions) can demonstrate examples of action taken to reduce violence Change Ambassador survey Change Ambassador interviews Partner reports CBA monitoring information: media reports, meeting minutes No baseline data available 20% of respondents to Change Ambassador Survey and CBA monitoring survey can demonstrate appropriate examples of action taken to reduce violence 2.1 Is Nabilan effectively supporting partner agencies to undertake quality prevention programming? IO2.1 Quality VAWC prevention initiatives which target individuals, communities and institutions are implemented Number of prevention activities reaching: individuals, families/relationship level, communities, institutions, society Number of people reached by each prevention activity. Disaggregation: gender, disability, Partner reports Available - 2017 At least one activity at each level.
ImO2.2 Partner have knowledge & skills to implement effective prevention approaches location (when available) Effectiveness for partner prevention programming Lancet Effectiveness Scale Available – 2017 Each partner has an annual score increase of at least 5%, or reaches score of 28 by 2022. 2.2 How has Nabilan contributed to independent women's movement's advocacy on gender equity and ending VAWC? ImO2.1 The Independent women’s movement is supported to promote gender equity and is empowered to advocate for social change Instances of Nabilan support to the independent women's movement to advocate on preventing VAWC Indicator 14: Instances of women’s leadership opportunities and empowerment Stories of significant change N/A N/A 2.3 To what extent has Nabilan influenced increased GOTL dialogue and action on preventing VAWC? ImO2.3 Selected GoTL agencies have improved knowledge to implement effective prevention approaches Instances of Nabilan supporting Government agencies to integrate effective prevention messaging in their own programs Indicator 18: Instances of improved policy and implementation; Indicator 19: Instances of evidence available for decision makers Stories of significant change N/A N/A 2.4 Is Nabilan effectively supporting Change Ambassadors to influence programs to integrate quality prevention approaches? ImO2.4 Change Ambassadors have the knowledge and are supported to influence the integration of quality prevention approaches Examples of Change Ambassadors integrating quality prevention approaches into their own programs Change Ambassador interviews N/A N/A Number of Change Ambassadors who continue to Change Ambassador survey No baseline data available 20 by end 2019 have continued to
participate in the network participate in the network. Proportion of Change Ambassadors with improved knowledge and confidence to advocate for the integration of effective prevention messaging into their own programs Change Ambassador survey No baseline data available At 70% of Change Ambassadors have improved knowledge and feel confident about integrating effective prevention messaging into their programs. 2.5 To what extent has Nabilan supported individuals, communities and institutions to recognise VAWC as a problem? IO2.2 Individuals, communities and institutions recognise VAWC as a problem Proportion of participants displaying improved knowledge and attitudes related VAWC Disaggregation: gender Reported separately for: * Participants in primary prevention initiatives * Target groups of other Nabilan-supported prevention activities PAF Indicator 13: Number of people that demonstrate improved attitudes to women and girls Prevention program monitoring Partner reports (these contain pre- and of participants in prevention initiatives) Baseline – for each prevention activity. TBC 2.6 To what extent has Nabilan supported communities to take independent action to prevent VAWC? IO2.3 Communities in Suku Letefoho and other focus areas are taking independent action to prevent VAWC Completion of all four phases of SASA! in Suku Letefoho by July 2020. Note: SASA! methodology requires community members to a threshold of knowledge, attitudes, skills, and behaviour before moving onto each phase. This will be assessed through qualitative and CBA monitoring 2018: Baseline for knowledge & attitudes (awareness), support skills and behavior in relation to power and violence against women: 50% women and 40% men think VAW is normal in a relationship; Completion of II (Awareness) between July 2018 – June 2019; Completion of Phase III (Support) between July – December 2019; Completion of Phase III (Action) between Jan – July 2020. Target will combine analysis
are generally not effective and do not promote relationships. Quantitative and qualitative data is required. 2017: Baseline for tolerance compiled and analysed from Activity Outcome Tool data gathered in Suku Letefoho from May to 15 August 2017: score for knowledge was 1.4 and score for attitude was 1.5 (out of a highest possible score of 4). Number of activities conducted by community mobilisers to prevent VAWC in suku Letefoho. CBA monitoring 0 prior to 2017 Phase 1: minimum 2 activities per CM per month Phase 2: minimum 4 activities per CM per month; Phase 3 & 4: TBC Number of collective action taken by community members in Suku Letefoho to prevent VAWC. Indicator 13: Number of people that demonstrate improved attitudes to women and girls CBA monitoring 0 prior to 2017 At least 3 instances of collective action taken by 2020 2.7 To what extent has the evidence from the CBA influenced prevention programming ImO2.5 Learning from the Community Based Approach in Suku Letefoho is documented Instances where other agencies have adopted the key approaches and learning from the CBA pilot. Activity tracking N/A N/A
by other agencies? analysed and used by Nabilan and other organisations. KEQ 3 What key factors will contribute to the sustainability of the Nabilan program outcomes beyond the end of the program? Principle: Sustainability Addressing KEQ3 and its sub-questions will be a key focus of the mid-term review. To answer these questions, the mid-term review will synthesise data from a range of sources, including key informant interviews, reviews of existing documents and literature, existing data collected to answer KEQs 1 and 2, Mid-term review N/A N/A 3.1 To what extent is GOTL resourcing prevention activities and essential services for women and children impacted by violence? How has Nabilan influenced this? IO1.3: Relevant Ministries use evidence to allocate funding of essential services ImO1.3 Partners advocate for increased GoTL funding of essential VAWC services Instances that reflect GoTL recognition of the need for sustainable funding for services Partner reports Meeting minutes and public statements from GoTL Mid-term review N/A 1 MoU with MoH signed; 1 PRPF held each year; funding discussed at coordination meetings with UN agencies, government and donors Evidence of changed funding portfolio (%) provided by Nabilan/other donors/government to for Nabilan partners GoTL budget allocations Available - 2017 N/A Evidence of Nabilan partners advocating to GoTL for increased funding Partner reports N/A N/A 3.2 Is evidence of changes in community attitudes and practices to Indicators 13 and 23 above CBA monitoring Mid-term review Refer to Indicators 13 and 23 above Refer to Indicators 13 and 23 above
relating to VAWC? 3.3 Is there evidence of strengthened local capacity and systems to continue implementation of quality services and prevention approaches? Organisational Institutional Strengthening assessment ratings Nabilan Partner Civil Society Organisations (CSOs) Capacity Assessment Grid Available - 2017 By 2022 each organisation is meeting at least 80% of all OIS milestones across all nine categories to Indicators 5-8 above Case Management Assessment Tool Case Audit Mid-term review Refer to Indicators 5-8 above Refer to Indicators 5-8 above 3.4 To what extent is the CBA approach likely to be replicable in the context of Stakeholder perceptions of appropriateness of the CBA approach in the context of Timor-Leste Mid-term review N/A N/A KEQ 4 Are there indications that the Nabilan programming model has contributed, or could contribute, to changing social norms relating to VAWC in Timor-Leste? Principle: Addressing KEQ4 and its sub-questions will be a key of the mid-term review. To answer these questions, the mid-term review will synthesise data from a range of sources, including key informant interviews, reviews of existing documents and literature, existing data collected to answer KEQs 1 and 2. The Nabilan Endline survey will provide additional data against this KEQ. Mid-term review Endline survey N/A 4.1 In Nabilan and partners supported focal areas, what evidence is
13. Non-acceptance of program research findings by key stakeholders Limited value placed on research findings, leading to challenges in uptake of findings in developing evidence based strategies. 2 3 M During Phase 1 Nabilan established a Research Reference Group to ensure research methodologies are relevant to the context of Timor-Leste, while continuing to draw on proven research methodologies used globally. Although research will be more limited in Phase 2 due to funding constraints, any research findings will continue to be presented with positive messaging that is relevant to the local context. The strategic use of research and engagement with key stakeholders on research design and findings aim to mitigate this risk. Nabilan Team Leader, Pillar Coordinators, M&E team 14. Difficulty in getting traction within other DFAT sector programs Impacts on the extent that the Australian Government can support a holistic approach to ending violence against women and children (current best practice) and maximise its investment in ending VAWC. 3 3 M Nabilan’s new Change Ambassadors Program will introduce a new approach to leveraging GoA funded programs. Early indications are that a number of DFAT programs including GfD, TOMAK and Workforce Development are keen to participate in this approach. Historically, the Foundation has attempted to use the Australian Government Gender Action Plan developed in early 2014 as a basis for securing the involvement of other sector programs. In addition, the Foundation attempted to establish regular policy dialogues per the recommendation of the Technical Advisory Group, in order to support coordination between Nabilan and other programs. Nabilan has met with sector programs outside of the policy dialogues. This approach will be continued.. Nabilan Phase 2 will continue to encourage all DFAT Programs to use positive messaging on VAWC. Australian Government Nabilan Team Leader Institutional Risks 1. Financial mismanagement by grantees Loss of program funds, loss of confidence in the Nabilan Program, or the end of a relationship with a partner impacts on program delivery. Impacts on clients if funding an organisation becomes not viable due to financial mismanagement. 3 2 M The Foundation’s due diligence and financial management systems, including close monitoring and capacity building for grantees, protect against financial mismanagement. Audits are conducted and any issues raised and followed up. The Foundation has placed increased resources on management of the grants to assist with monitoring and capacity building of partners. The Foundation will ensure close communication with Nabilan Team Leader, Finance Manager, Pillar Coordinators