Purpose Clarity of purpose is integral to implementing a Return on Investment Framework. This maintains focus and helps to specify the parameters. Studies have used ROI for various reasons. These include business improvement through supporting new technologies and improving workforce efficiency, workplace health and safety learning (Brown, et al 2015) and as part of funding agreements (IPP, 2012). Key questions Why are stakeholders measuring ROI? How will ROI information be used? ROI Evaluation Framework The purpose of this paper is to highlight the issues involved with structuring an evaluative ROI framework that aims to: Develop a practical, coherent and transparent methodology Measure on the basis of selective and useful variables and Produce the most compelling evidence possible based on what data are available. A review of the issues in measuring ROI in the literature helps to identify the key elements that constitute an Evaluation Framework. These are represented in the diagram in Table 1. Table 1 ROI Evaluation Framework elements Context, scope and purpose, as discussed in the first section of this report, form the foundation of the framework. The other components are presented separately in the following sections along with steps and methods to guide the process. Guiding principles The first step is to develop guiding principles to follow and apply a consistent and standard frame of judgement to the ROI evaluation. These include the following: The ROI model/method adopted: Must be customised, fit for purpose and add-value. Requires an overarching clarity of purpose Measures factors that matter Measures factors that are specific, in context and within an identified group. The implementation of the ROI model considers: Is it practical and useful?
What are the direct costs? What are the indirect costs? Who pays for the training? Do the costs differ by industry? Over what period of time are the costs calculated? Are intangible costs measurable? How can we measure intangible costs? Can or should intangible costs be converted into monetary value? Training benefits Benefits are defined in various ways, arise at different points in time and are sometimes difficult to quantify (OECD, 2008). There are two categories of benefits – market and non-market. In the workplace, for example, these refer to job related and non-job related outcomes. Table 5 illustrates some of these benefits for individuals, organisations and the economy. Market benefits are directly measurable and relative to the stakeholder group. The main benefit of TVET that influences pre-tax earnings of individuals is improved employability (Long and Shah, 2008) and for the economy increased participation in the workforce. Non-market benefits are not so easily quantifiable. Employee social and well-being aspects or business workplace literacy, safety and workforce flexibility are more difficult to measure. Outcome measures tend to have an economic focus, excluding community and personal outcomes that are more difficult to quantify (OECD 2008). A model that takes both market and non-market benefits is recommended (CEDEFOP, 2013). Table 5 Benefits of TVET training by stakeholder Individuals Employer Economy Job related Higher employability Employment Higher salaries Higher savings levels Improved working conditions Professional mobility Productivity (highly skilled) Market Productivity Sales & profitability Customer service and satisfaction. Occupational health & safety Quality product & services Saving on material & capital costs. Tangible benefits Higher employability Increased participation in the workforce Decrease in unemployment levels Productivity gains Higher skilled workforce Non-job related Higher education pathway Pathway to further study Improved self esteem Communication skills Engagement Improved problem solving Improved health & wellbeing Improved economic standards of living Non-market Motivated workforce Improved organisational climate and culture. Increased literacy in workplace Employee skill gains Employee well-being Employee workplace practices Intangible benefits Improved health Improved environment Reduced national crime Increased social cohesion Increased social inclusion Strengthened social capital Active citizenship Technological change adaptation
Table 6 Benefits of Training for Employees and Employer Benefits Employee Employer Tangible Improved employee pay Improved language & literacy Improved technical skills Increased use of new technologies Improved workplace practices and procedures. Increased productivity and efficiency Increased sales and profitability Improved product quality & services Improved customer service and satisfaction levels. Improved OHS Intangible Social and well-being Improved self-confidence & morale Reduced stress Improved motivation Improved work ethic Improved physical and mental health Job satisfaction Better management and employee workplace relations More co-operation among employees Reduced internal conflicts Developing a learning culture Supporting social cohesion and inclusion Source: Adapted from Barker 2001 and Moy and McDonald 2001 in Brown et al 2015 Workplace Literacy Pays, ACER Note: A detailed list of benefits is presented in Appendix 7. Time duration - Benefits may arise at different points in time and may extend well beyond the final offering. The payback period could be projected one to five years. The period of time data points may cover - during a training program, at its completion or long after the event (OECD, 2008). Table 7 illustrates a comparison of the short and long term benefits to the individual, organisations and the economy. Medium to long term benefits such as mobility or the capacity to upgrade skills later in life are more difficult to quantify (OECD, 2008). The value-chain of benefits needs to be considered to fully capture more accurate returns of training. Table 7 Short and long term benefits of training by stakeholder Benefits Individual Employer Economy Short-term benefits Employment opportunities Increased earning levels Work satisfaction Higher productivity from trained workforce. Saved costs from recruiting external skilled workers Improved quality of products and services. Improved customer satisfaction levels. Reduced reliance on welfare Social cohesion Long-term benefits Greater employee flexibility and mobility Lifelong learning Reduced employee turnover. Improved safety record. Better workplace relations. Productivity gain from educated workforce. Increase in tax income from higher earnings. Source: Costs and Benefits in Vocational Education and Training 2008 OECD Source: Adapted from Barker 2001 and Moy et al 2001. Key questions What are the tangible benefits? What are the intangible benefits? Should intangible benefits be converted to monetary value? (e.g. well-being) How can intangible benefits be monetised/quantified? What are the short, medium and long term benefits of training? When are the most important data collection points?
SROI Case Studies – Employment creating social enterprises ENTERPRISE AIMS KEY LESSONS LIMITATIONS Food Connect Brisbane Food distribution To understand and value the impact on its stakeholders given its growth plans – Forecast SROI. (Funding agreement with investor) The SROI process and analysis enabled FCB to objectively and independently validate the impact they created and intend to create. Lack of understanding of what constitutes a “good result” made it difficult to compare with organisations or benchmarks. SROI was resource intensive due to limited data collected in the past. STREAT Assists homeless & disadvantaged youth find long-term and meaningful employment. Forecast SROI analysis to develop a measurement/evaluation framework to identify data to collect and the social, economic and environmental impact. SROI analysis confirmed the value of STREAT’s employment support approach and created baseline data for their measurement and evaluative framework. It considered outcomes in isolation and did not fully capture their Due to the forecast nature it was difficult to engage with some stakeholders and they were excluded. SROI analysis did not represent the full potential impact. People Power Cleaning (PPC) Provides employment to refugees and migrants at risk of mental health issues who face employment barriers. To understand the costs associated with providing support to marginalised employees. (As part of a funding agreement with investor) Identified the most significant support costs were training and on-job supervision. Provided a credible and transparent justification for funding an additional employee. No limitations indicated. Tasty Fresh Community Catering Provides support, training and job opportunities for underprivileged women. Part of funding agreement with investor. To understand the costs associated with providing support to marginalised employees. Identified the real costs associated with achieving the social mission, many of which had been underestimated and not accurately factored into forward projections. The timing of the analysis was not ideal. The results would be more useful had the process been conducted when there was more clarity around forecast growth and revenue targets. Source: Adapted from Investing in Partnerships (IIP) 2012, Social Return on Investment. Lessons Learned in Australia prepared for Investing in Impact Partnership. Appendix 4 (cont’d) SROI Case Studies – Employment creating social enterprises
ENTERPRISE AIMS KEY LESSONS LIMITATIONS Livingin Constructions Constructs affordable buildings /landscapes for families in need while employing people excluded from the labour market. Part of investor funding agreement. To understand the additional costs incurred, and value created from employment. Provide management and investor with the range and value of outcomes and costs. Stakeholder feedback was a critical component. Illustrated key driver of social value was through increased employee social interaction. Gained clarity on outcomes, values and costs. Informed what data to collect and evaluate in the future and how to reduce costs. SROI analysis was useful but could have been more powerful if it incorporated all elements of their model i.e. both the universal housing design and employment support programs. Sandgate Enterprise Economic Development Provides jobs for people excluded from the labour market. Part of investor funding agreement. Learn about the methodologies, insights and improvements. Significant social value was created for employees/ centre. SROI highly valued by management/corporate customers due to analysis integrity. No limitations indicated. Source: Adapted from Investing in Partnerships (IIP) 2012, Social Return on Investment. Lessons Learned in Australia prepared for Investing in Impact Partnership. Appendix 5 Summary of results from Kirkpatrick/Phillips Levels of Evaluation ROI Model for Returns to investment of Workplace Literacy (Australia) INDUSTRY (size) DESCRIPTION DATA COMPLETENESS & QUALITY RETURNS/IMPACT Aged Care (Small) Improving documentation to enhance organisational culture Monetised changes in supervisor time across all data collection points. 117.5% - savings through reduced documentation errors. Aged Care (Medium) Reducing turnover and improving employee engagement Systems and data capture not ideal for this purpose. Anecdotal and important part of orientation – improving documentation and maximising call on government funding.
Building and Construction (Large) Supporting workforce engagement and worker advancement Large, complex and multi-faceted program made data collection difficult. Difficult to quantify across entire program – sub program identified $192,600 in savings. Manufacturing (small) Supporting Lean manufacturing and enhancing worker engagement Data-driven culture. Uses multiple indicators to monitor change. 132% return from Manufacturing (medium) Supporting workplace health and safety learning Multi-layered program produced measurable sub-set of data 163% return from error reduction and saved supervision time. Utilities (medium) Supporting new technologies and improve workforce efficiency ROI calculated based on unit improvements in service orders. 102% at one site based on efficiency improvements. Source: Adapted from Brown, J, Taylor, M, McKenzie, P and Perkins, K 2015 Investing in Workforce Literacy Pays: Building Employer Commitment to Workplace Language, Literacy and Numeracy Programs. ACER. Appendix 6 TVET costs, benefits and impact Study (Year) TVET Costs/Benefits and Impact AWPA 2013 Literature review Positive effects on business productivity and profitability. Higher returns where training is highly specific; accomplished quickly and related to new technology or processes. The industry of the business and quality of training may influence the outcomes. CEDEFOP (2013) Literature review Education benefits: Improved health, improved environment, reduced national crime and drug use, democratisation and human rights. Adaption to technological change. Produce higher skilled workforce. Initial TVET – better health, membership of organisations and job satisfaction. Non-market benefits of self-esteem, confidence and opportunities for advancement. Size of organisation, industry and type of training impact on the return. Independent Economics (2013) Costs to Individual: Tuition costs, foregone earning opportunities Benefits to Individual: Higher employability, productivity, pathways to further study