in November) will be less likely that co-regulatory arrangements will meet or exceed their individual targets prior to November. Assess liability and notify liable parties Statements of advice (liability) sent to liable parties in May and July, with final in November. Adequate. Liable parties are sent relevant notifications at specified Additional information and support is also provided to liable parties by phone and email. Assess and approve co-regulators Approved arrangements Adequate. Co-regulatory arrangements were assessed according to the Regulations and, as appropriate, approved. As new co-regulatory arrangements have sought to join the scheme, they have been similarly assessed and approved. Assess and approve co-regulators’ quarterly and annual reports (through PSO) Approved reports Adequate. Co-regulatory arrangements’ quarterly and annual reports are assessed and approved. As identified in our response to KEQ 2a on page 24, delays occur when the arrangements’ annual reports do not tally with the separate figures provided to DoEE (by the co-regulator arrangements themselves). These delays are sometimes exacerbated by PS Online, in which co-regulatory arrangements may experience problems entering or uploading data. Engage and collaborate with key partners Consultative /reference group Quarterly visits to Conference papers and presentations Not adequate. DoEE does engage with key stakeholders (as discussed under KEQ 4 on page 30) but not through all of the planned outputs articulated here. A formal consultative /reference group does not exist. DoEE liaises with and territory governments on an ad hoc basis (approximately annually), and there is no regular liaison with local government. Engagement with co-regulatory arrangements is generally undertaken on a quarterly basis, including site visits to regulators regulators regulators co- co-
See discussion of KEQ 5b (page 34) and Effectiveness and Impact (page 70) Medium term outcomes 7 Increased awareness and knowledge about scheme performance Outcome has been achieved. DoEE publishes the co-regulatory arrangements’ annual reports and its own annual outcomes reports detailing scheme performance. See discussion of KEQ 4a (page 30) Co-regulatory arrangements’ annual reports and DoEE annual outcomes reports Stakeholder interviews and survey responses 8 Broader awareness of e-waste recycling collection services Outcome has been achieved. E-waste recycling collection services are well used, and stakeholders indicated that there was a demand for additional services. See discussion of KEQ 5b (page 34) and Effectiveness and Impact (page 70) Collection service statistics in co-regulatory arrangements’ annual reports and DoEE annual outcomes reports Stakeholder interviews and survey responses 9 & 10 Increased compliance with the scheme by arrangements and liable parties. Liable parties comply with regulations Outcome has been achieved. Co-regulatory arrangement and liable party awareness of, and compliance with, scheme obligations is consistently high. See discussion of KEQ 4a (page 30) and 5b (page 34) Liable compliance statistics and statements in DoEE annual outcomes reports Stakeholder interviews and survey responses 11 Co-regulatory arrangements achieve of recycling targets as they increase to 80% and recovery targets Insufficient evidence to assess whether outcome has or has not been achieved. While co-regulatory arrangements are currently meeting their targets (except for one in 2016-17), there is insufficient evidence to assess whether co-regulatory arrangements will continue to N/A
(Further, we note the term “broader use” is vague and therefore difficult to assess) Long term outcomes 17 Reduced (by weight) to landfill, specifically the hazardous materials found in e-waste Outcome is in process of being achieved. Relevant targets have been achieved each year. Statements and recycling statistics in co-regulatory arrangements’ annual reports and DoEE annual outcomes reports 2014-15 Operational Review issues paper 2015 RIS Stakeholder interviews and survey responses 18 Increased recovery of reusable materials in a safe, scientific and environmentally sound manner Outcome is in process of being achieved. Statements and recovery statistics in co-regulatory arrangements’ annual reports and DoEE annual outcomes reports 2014-15 Operational Review issues paper 2015 RIS Stakeholder interviews and survey responses 19 Increased recycling, including by households and business, as a result of access to fair and equitable industry recycling schemes available nationally Outcome is in process of being achieved. Statements and recycling statistics in co-regulatory arrangements’ annual reports and DoEE annual outcomes reports 2014-15 Operational Review issues paper 2015 RIS Stakeholder interviews and survey responses 20 Increased efficiency, quality and coverage Outcome is in process of being achieved. Access to collection, recycling and Statements in co-regulatory arrangements’ annual
Appendix E: Evaluation Framework Evaluation Question Sub-Questions Measures Data source or method Appropriateness 1. To what extent has the program design proven to be adequately matched to identified needs? 1a. Have the needs identified during the original development of the NTCRS changed over time? 1b. What evidence is there that the NTCRS program design meets or is likely to meet identified needs? analyses. Synthesis of evaluation research findings. Document review – Department documents Regulatory Impact Statements Possible source: Co-regulators may have consumer behaviour surveys/studies. ACIG to check if are willing to share or if DoEE can provide. Efficiency 2. How efficiently has the NTCRS been managed and implemented? 2a. To what extent were available resources used to the best effect? Summative assessment of evaluation findings concerning program delivery. Evaluation synthesis of costs versus outputs achieved 3. How efficient has the NTCRS been in delivering its intended outcomes? 3a. To what extent are the costs of program delivery justifiable against its results? 3b. To what degree is the scheme cost-effective compared to like programs or other options that address the same needs? Program costs and benefits. Note: the evaluation will be limited to whatever cost and benefit data is currently available from the Department. Comparison of program costs and benefits with other programs (to the extent that they are available). 2014-15 Operational Review of the NTCRS Departmental budget and administrative data Regulatory Impact Statements Evaluation synthesis of evidence on and benefits Effectiveness and Impact 4. Were communication and education activities adequate? 4a. How were the target audiences for education and information determined and to what extent were they reached? Degree of reach to audience; website analytics trends; communications materials created and disseminated. Awareness of program objectives, penalties, liabilities (obligations under the Act) by liable parties and Department documents and interviews Stakeholder survey Stakeholder interviews Stakeholder survey 5. Were the activities and their resourcing adequate to ensure delivery of the planned outputs? 5a. Were the activities that were delivered adequate in achieving the planned outputs? 5b. Were the resources provided adequate to provide all the planned outputs? Program management’s assessment of program implementation. Program management’s assessment of program implementation. Department documents and interviews Department documents and interviews