Intervention design Evaluation design Evaluation implementation 1. Our decision to make an intervention is based on a robust assessment and specification of the need to make an intervention. 2. We are clear about the difference we wish to make through our intervention. 3. We have reason to believe that the intervention we propose to deliver will produce that difference. 4. We are clear about the nature of the data required to demonstrate that we have made a difference. 5. We are clear about the methods we will employ to collect that data. 6. We know how we will analyse the data we collect to produce conclusions. 7. We know how we will use our conclusions. 8. We have assessed and committed the resources required to deliver the evaluation. Robust and routine evaluation will make it more possible to establish if, and how, particular interventions make a difference or not, and help identify what specifically should be done to prevent, and respond more effectively to, prejudice, unlawful discrimination, and identity-based violence and harassment.
1 |Introduction Evaluation should be simple. It should be as simple as asking: ‘Is what we do making a difference?’ Unfortunately, evaluation is more complex than that. If it was that simple, then there would be no need for this guidance. Imagine you are an architect. It may not be wise to accept a brief that simply said ‘build a house’. You would want a little more detail and clarity than that. If you don’t have that clarity, then it’s quite likely that the house you build wouldn’t quite meet everyone’s expectations. Similarly, with evaluation, it may not be wise to accept a brief that simply instructed you to answer the question, ‘is what we do making a difference?’ Unless all the components are clearly and properly described, then the final evaluation may not meet anyone’s expectations, and may not even be an evaluation at all. The aim of this guidance is to support you to incorporate ‘good enough’ evaluation processes into the planning stages of new interventions. An ‘intervention’ is any activity designed and carried out with the aim of tackling, preventing or reducing prejudice, discrimination, identity-based violence or harassment. There are many forms of evaluation. This guidance is concerned with evaluation processes that allow you to measure the difference your work makes to tackling prejudice, discrimination, identity-based harassment or violence in Britain. Knowing if and how your work makes a difference is important. If your work isn’t making a measurable difference, why do it? This guide is intended to support improved self-evaluation; that is, evaluation that is planned and conducted by the same organisation that is delivering the intervention. It includes several lessons that can also be applied to commissioning an external evaluation. The guidance has been designed to be applied to the evaluation of project-level interventions. However, it should also be useful for partnerships managing or coordinating broader programme-level interventions.
5 |A minimum standard for evaluation The Equality and Human Rights Commission report ‘Prejudice and unlawful behaviour: exploring levers for change’ (Abrams et al., 2016) applied a set of criteria as a method of assessing and scoring the quality of the evaluation processes of 24 different interventions. The scores achieved by the 24 interventions studied ranged from 15 per cent to 71 per cent. The variation in the quality of evaluation processes that were reviewed show just how challenging it can be to undertake a ‘good enough’ evaluation. We have designed these principles following analysis of the barriers and challenges to enable you to achieve that ‘good enough’ evaluation. By ‘good enough’ we mean an evaluation that is proportionate and realistic, but can also provide evidence that is reliable enough for the Commission and others to use to inform their work. Table 1. Eight principles for evaluating anti-prejudice projects Intervention design Evaluation design Evaluation implementation 1. Our decision to make an intervention is based on a robust assessment and specification of the need to make an intervention. 2. We are clear about the difference we wish to make through our intervention. 3. We have reason to believe that the intervention we propose to deliver will produce that difference. 4. We are clear about the nature of the data required to demonstrate that we have made a difference. 5. We are clear about the methods we will employ to collect that data. 6. We know how we will analyse the data we collect to produce conclusions. 7. We know how we will use our conclusions. 8. We have assessed and committed the resources required to deliver the evaluation.
the extent of a problem, you should be able to produce a reasonable baseline from which to measure any change arising from your activity. Principle 2: We have clarity about the difference we wish to make By applying Principle 1, you should have a clear idea of the nature of the need for change. If you are vague about the nature of the change you want, then you will find it impossible to measure if you have achieved it. The next step is to set out your vision of the nature of the change that’s needed in an outcome statement. This represents outcomes in a ‘theory of change’ approach. Outcomes and outputs are frequently confused. Charitable think tank New Philanthropy Capital (Kazimirski et al., 2016) define outputs as ‘products, services or facilities that result from an organisation’s or project’s activities’, and outcomes as ‘the changes, benefits, learning or other effects that result from what the project or organisation makes, offers or provides’. Outcome statements are essentially a description of the desired change. For example, ‘organisations delivering anti-prejudice interventions have increased capacity to evaluate the change they make’ is a statement of outcome. For the purposes of evaluation, outcomes should be specific enough to be measurable, and for change to be reasonably attributable to the intervention. Broader outcomes of societal change and ‘world peace’ tend to be unsuitable for project design, or for evaluation at project level. They might be worthwhile having in a statement as a reminder of our long-term aspiration, but won’t be achievable by, or attributable to, a single intervention. Outcomes for anti-prejudice interventions should, as far as possible, specify the protected characteristic that they intend to affect. It can be tempting to focus on less specific ideas of equality and prejudice, but change in these concepts can be more difficult to demonstrate than change in a specific circumstance associated with a specific protected characteristic. While these broader changes in values or attitudes may be desirable, it is exceptionally difficult to generate baseline data on existing levels of non-specific prejudice, and therefore difficult to confidently measure change. If you want to have a manageable evaluation process, it is wise to choose fewer outcomes. This may mean prioritising the outcomes that you wish to invest in aluating. The New Philanthropy Capital ‘four pillar’ approach to planning evaluation
Qualitative data is more open to differing interpretations, and therefore offers a greater risk of inaccurate interpretation, or of the evaluation being perceived to be subjective or biased. It also means that qualitative data can fail one of the key tests for a good success measure – that it should be reliably and consistently interpreted by different parties. The Evaluation Center’s ‘Qualitative evaluation checklist’ (Patton, 2003) further notes that the ‘quality of qualitative data depends to a great extent on the methodological skill, sensitivity, and integrity of the evaluator ... Generating useful and credible qualitative findings through observation, interviewing, and content analysis requires discipline, knowledge, training, practice, creativity, and hard work.’ In this respect, the data produced through qualitative methods may be regarded as potentially persuasive and useful in influencing others, particularly at reporting stage, but they present substantial challenges in producing data that is able to provide a robust evaluation of change from a baseline for your success measure. Principle 6: We know how we will analyse the data we collect to produce conclusions Another area where self-evaluation processes frequently stall is in the analysis of the data collected. Organisations can quite competently design interventions, set outcomes and success measures, and use appropriate methods to collect the data, and then do very little with it. Analysis can take time, but the key to efficient and effective data analysis is making the process manageable. That’s achieved by only looking at data that allows you to form conclusions. It’s not uncommon to invest time in gathering data you can’t or won’t use in your evaluation. To prevent this, you can ask yourself a few simple questions: ‘What will I do with this information?’ If the answer is ‘nothing’ or ‘I don’t know’, you may be better not collecting it. ‘How will I analyse the data I collect?’ If the answer is ‘I don’t know’ or ‘with great difficulty’, then you are either collecting data that doesn’t meet the relevance test for success measures, or data that is qualitative in nature. Data that allows you to form conclusions about the change from your baseline is what you need for evaluation; that means data that allows you to measure difference, and that means quantitative data.
Nesta. 2016. [ONLINE] [accessed: 18 July 2017]. Nesta (2011a), ‘’. [accessed: 18 July 2017]. Nesta (2011b), [ONLINE]. [accessed: 18 July 2017]. Patton, M. Q. (2003), ‘’, University of Western Michigan. [accessed: 18 July 2017]. Peer, E. and Gamliel, E. (2011), ‘’, Practical Assessment, Research and Evaluation Journal, vol. 16, no. 9. [accessed: 18 July 2017]. Project Implicit. [2011]. . [accessed: 31 July 2017]. Puttick, R. and Ludlow, J. (2013), ‘’. Nesta. [accessed: 18 July 2017].