Annex 8: Evaluation Framework Evaluation Questions Types of answers/ Evidence Needed Methods for data collection e.g., records, structured observation, Key informant interviews, mini-survey Sampling or selection Approach (If one is needed) Data Analysis methods e.g.… Method Data sources PROGRAM DESIGN Is there any evidence that programs (HIV, TB, malaria and maternal and child health) have incorporated community based approaches through CHWs into their program strategies? Analytic description of the technical approaches/ strategies and objectives of the programs (HIV, TB, malaria and maternal and health); evidence of CSS involving established community health workers Document/ Literature review National Strategic & grant documents, Project theory of change, M&E plan, performance (activity, quarterly, annual, and other relevant) reports; work plans, budgets, financial reports; survey reports None Content analysis and interpretation of qualitative findings Is there evidence to suggest that CHWs have participated in and contributed to the three disease control program, and MNCH? Analytic descriptions of the entire national and district coordination structures (up to lowest service delivery level), quality of the relationship that exists whether it’s also supportive and mutually beneficial Analytic description of supportive mechanism for the work of the CHWs, data recording and reporting systems as well as methodologies used in aggregation and forward transmission to the centre; evidence of use of information generated to influence management decisions and refine implementation Document Review of program documents, periodic performance reports, minutes of performance review meetings Focus Group Discussions with partners, stakeholders, and target groups at district, health facility and Community level. Key Informant interviews (PR, MOHCW (Directorate of Nursing), ZAN and HSS CCM subcommittee) and external informants (partners, Disease specific Policy, Strategic Plan, and implementation guidelines Grant documents District annual performance reports Key informants from technical side and stakeholders at all relevant levels especially the community Sampling of districts and communities to be decided upon during development of protocol Sampling of levels health facilities and communities will be done to achieve a balance between the districts and between good and poor performing health facilities/communities. Content analysis and Grounded theory approach for analysis and interpretation of qualitative findings Consultative meetings, selection, analysis, feedback and verification of ‘Significant Change’ stories
stakeholders and target groups at central, district, health facility, and community level). SWOT analysis Is there evidence to suggest that the CHW component as currently designed can be sustained beyond the Global Fund investments? Analytical Description and critique of Project Theory of Change including 1. design and approaches 2. The decentralized setting of health service delivery. 3. Role of Ministry/Programs & recipient districts in this process and during implementation Analytic description of the changing political and social environment and the extent to which districts/communities can develop and implement comprehensive and integrated wok plans for HIV, TB, Malaria & other diseases. The extent to which services are provided in an integrated manner in grant supported districts. Establish linkages and/ or main streaming of GF reporting systems with the GoZ HMIS Establish the readiness/preparedness of the district coordination structures to plan for Desk review of program documents and reports Focus Group Discussions as in No. 2 above Key informant interviews as above SWOT analysis Most Significant Change Technique involving project stakeholders Grant documents as above District annual performance reports Key informants at National, Province, District and Community Sampling of districts to be decided upon during development of protocol Sampling of Lower levels health facilities and communities will be done to achieve a balance between the districts and between good and poor performing health facilities/communities. Content analysis and Grounded theory approach for analysis and interpretation of qualitative findings Consultative meetings, selection, analysis, feedback and verification of ‘Significant Change’ stories
prevention activities against the three diseases beyond GF investments Establish the extent to which GF Round 8 HSS grant has strengthened health facility based and community based referral/service networks that benefit the vulnerable groups Client perspective of the services offered by GF R8 HSS supported health facilities and communities PROGRAM IMPLEMENTATION Have the community health approach interventions been implemented as intended? Adherence to work plan: service package, delivery approaches and adherence to national guidelines/procedures, target populations and subgroups Comparison of the actual program results against the targets prescribed in the Project Theory of change/M&E Plan/Performance framework (annually) Compare access and utilization of services - pre and post Project intervention Analytic description of the factors that hindered or facilitated the achievement of program results Describe lessons learnt and best practices Suggest relevant recommendations Desk review of program documents and reports Review the baseline and periodic performance information collected quarterly and annually against the project theory of change and M&E/Performance Framework Key informant interviews as SWOT analysis Strategic and grant documents as in No. 1 Above. District annual performance reports including HMIS (Plus DQA where necessary) for major indicators Key informants from the Headquarters, District Field staff, District leaders, disease focal persons, Health facility staffs, Community Health Workers) Sampling of districts to be decided upon during development of protocol and data collection tools and after budgetary considerations Sampling of Lower levels and health facilities will be done to achieve a balance between the districts and between good and poor performing health facilities and communities. Content analysis and interpretation of qualitative findings Analysis of performance trends (routine HMIS and program/grant targets) in the sampled districts Is there adequate capacity/skills among the CHW cadre Compare the actual program results against the targets prescribed in the Desk review of program documents and reports
Project Theory of change/M&E Plan (annually) Compare access and utilization of services - pre and post Project intervention Compare trends in performance of the CHW – pre and post implementation; available supportive mechanisms for the CHWs, and type and frequency of logistics provided to support work of the CHWs Analytic descriptions of all the district/community coordination structures strengthened by the (including involvement of partners and other stakeholders at national, district and community levels; meetings, communication, involvement of communities and beneficiary populations in various project implementation stages) Analytic description of the factors that hindered or facilitated the achievement of program results Describe lessons learnt and best practices Suggest relevant recommendations Review the baseline and periodic performance information collected annually against the project theory of change and M&E Framework Key informant interviews as above SWOT analysis District annual performance reports including HMIS (Plus DQA where necessary) for major disease indicators Key informants from the above stated protocol/data collection tools; and after budgetary considerations Sampling of Lower levels and health facilities will be done to achieve a balance between the districts and between good and poor performing health facilities. (Time and distance to play role in the final sample size) Analysis of performance trends (routine HMIS and program/grant targets) in the sampled districts/communities What are the facilitating and inhibiting factors in the field implementation of the CHW program (support from community, assistance from health facilities etc)? PROGRAM RESULTS
What is the overall perception of community members and community leaders of the strengths and weaknesses of CHW program? What has changed in the community due to presence? Analytic description of the overall perceptions of the key “supply side” actors in Community Health Systems strengthening component, the strengths and weaknesses of the CHW program Key informant interviews with the actors stated in the question SWOT analysis Key informant from national, district, sub national and community levels as stated in the question Sampling to be agreed alongside that of the districts Content analysis and interpretation of qualitative findings What have been the results of the community interventions on HIV, TB and malaria program coverage, and access to services (diagnosis and treatment, referral and follow-up services, including linkages between care with institutional care and provision of care and support services)? Comparison of the actual program results against the targets prescribed in the Project Theory of change/M&E Plan (annually) Compare access and utilization of services - pre and post Project intervention Analytic description of the factors that hindered or facilitated the achievement of program results Describe lessons learnt and best practices Desk review of program documents and reports Review the baseline and periodic performance information collected annually against the project theory of change and M&E Framework Key informant interviews as in No. 1 above SWOT analysis Grant documents as in No. 1 Above. District annual performance reports including HMIS (Plus DQA where necessary) for major malaria indicators Key informants from National, Province, District, Health Facility and community level Sampling as Sampling of Lower levels and health facilities will be done to achieve a balance between the districts and between good and poor performing health facilities. (Time and distance to play role in the final sample size) Content analysis and interpretation of qualitative findings Analysis of HMIS and other relevant performance trends in the sampled districts/communities Is the CHW program supported through Global Fund grant cost-effective? What is the ratio of direct to indirect costs for maintaining each CHW; what is the cost Determination of the direct and indirect costs of the CSS component Determination of clients serviced by the CHW at national and local level as selected in the sample. Literature review Review approved budgets, mapping CSS related costs and collate and analyse expenditure reports Comparison with the cost of alternative Proposal and Strategic documents, Project theory of change, M&E plan, activity, quarterly and annual reports; work None Content analysis and interpretation of costs and other quantitative findings Analysis of HMIS and other relevant performance trends in