Table 3: The department’s Evaluation Maturity Matrix Principle Beginning Developing Embedded Leading Integrated Awareness of the benefits of evaluation is low. Evaluation is seen as a compliance activity and threat. Fear of negative findings and recommendations leads to a perception of ‘mandatory optimism’ regarding program performance. Insufficient resources allocated to evaluation activities. Evaluation and performance measurement skills and understanding limited, despite pockets of expertise. Appreciation of the benefits of evaluation improving. Evaluation is being viewed as core business for the department, not simply a compliance activity. A culture of evaluative thinking and continual improvement is introduced and communicated across the department. Skills in performance measurement and evaluation developed through targeted training and guidance materials. Evaluation website and guidance materials developed. The role of the Evaluation Unit is widely communicated. Unit seen as the authoritative source for advice. A culture of evaluative thinking and continual improvement is embedded across the department, with lessons learnt being acted upon. Evaluation is seen as an integral component of sound performance management. General evaluation skills widespread. Improved skills and knowledge in developing quality performance measures. Evaluation Unit team members have high order skills and experience which are leveraged by the department. Evaluation Unit team members hold and are encouraged to Evaluations motivate improvements in program design and policy Demonstrated commitment to continuous learning and improvement throughout the agency. Department is recognised for its evaluation and performance monitoring expertise, and innovative systems and procedures.
Developing further expertise in the Evaluation Unit. undertake formal qualifications in evaluation and related subjects. Fit for Purpose Frequency and quality of evaluation is lacking. Guidelines for prioritising and scaling evaluation activity are used. Priority programs are evaluated. Evaluations use fit for purpose Evaluation effort is scaled accordingly. Specialist and technical skills well developed to appropriate methodologies. Data holdings and collection methods are insufficient or of poor quality. Planning at program outset improves data holdings and collection methods. Developing skills and knowledge in applying robust research and analytical methods to assess impact and outcomes. Quality of evaluations is improving. A range of administrative and other data is used in the assessment of performance. Robust research and analytical methods are used to assess impact and outcomes. Evaluations conform to departmental standards. The department continually develops and applies robust research and analytical methods to assess impact and outcomes. Evaluation and performance measurement conform to recognised standards of quality. Timely Effort and resources are allocated in an ad hoc and reactive manner with little foresight. Developing performance information at the Evaluation activity is coordinated. An evaluation plan is in place and regularly monitored. Strategically significant and The department employs strategic criteria to prioritise evaluation The department’s approach to evaluation and performance planning is seen as the exemplar. based
inception of a program is ad hoc and of variable quality. risky programs are prioritised. Planning for evaluation and performance monitoring is being integrated at the program design stage. All programs are assessed for being Evaluation Ready. effort. Evaluation plans are updated annually and progress is monitored on a regular basis. Planning for evaluation and performance measurement is considered a fundamental part of policy and program design. All programs have program logic, performance and evaluation plans in place. All programs have been signed off and are Evaluation Transparent Findings and recommendations held in program and policy areas. No follow up on the implementation of Findings and viewed as an opportunity to identify lessons learnt. Evaluations are available in the completed evaluations library to improve the dissemination of lessons learnt and inform policy development. Findings widely disseminated and drive better performance. Website and guidance materials are a valuable resource for staff. Evaluation findings and reports are published where appropriate. Findings are consistently used to optimise delivery and have influence outside the department. Independent Independent conduct and There is an improved level of All evaluations include a level Evaluations conducted by
Inputs and participation Is either not comprehensive or is inaccurate in relation to inputs, stakeholders. Omits staffing and/or administered funding. Lists government under participation (unless the program targets government as the beneficiary). Inputs section includes staffing. Inputs section includes formal external inputs where the department is not the sole funder. Funding for inputs is broken down by administered and departmental, where known. Inputs section includes a clear timeframe for funding, either across the lifetime of the program or other clear timeframes. Participation section identifies target recipients for the program. The focus is on beneficiaries, not deliverers of it, such as government. If many participants, these are grouped into logical subgroups. Includes in-kind inputs where relevant. Clarifies target market — distinguishes between primary and secondary beneficiaries. Participation is represented so as to align with activities and outcomes. Includes all stakeholders impacted, not just program participants. Concise. Activities and/or outputs Too much detail on generic administration processes such as for granting programs. Outputs are confused with or substitute for outcomes. Activities don’t link to outputs and outcomes. Identifies who does what to whom. Separates Commonwealth and participant activities as necessary. Shows ordering of key activities and links to outcomes. Activities/outputs are directly related to objectives and can be monitored and assessed. Uses action verbs to identify activities. Outcomes are informed by evidence and experience / lessons learnt.
External factors and assumptions Not included or not clearly identified. Not supported by evidence. Key external factors and assumptions identified. Assumptions supported by evidence/theory of change and Informed by lessons learnt. Assumptions comprehensively state the conditions required for the program to function effectively. Notes: 1 SMART Specific, Measurable, Attainable, Relevant, and Time-bound Source: Department of Industry, Innovation and Science (2017)