Table 1: Evaluation Questions and Data Matrix Key Evaluation Question Information Requirements Data Sources Method of data collection or retrieval What were the strengths and limitations of the program design? Description of the coaching intervention and activities undertaken by coaches, quality of content, design (coaching model) - Observation at 6 services (last coaching - Secondary data - coaching reports - Coaches (x 4) and coaching advisor -Educators (16) and 6 managers across 8 services - Fieldwork - Document analysis - Individual interviews - Individual interviews and informal conversational interviews What changes in educators’ practice resulted from the coaching program? of a wide range of stakeholders about the value of coaching and of this specific intervention (self-report from educators and coaches reports) -4 coaches plus overall coaching advisor (Gowrie Victoria) - educators participating in and end of program survey and/or interviews - Interviews and coaching reports -End of program survey and interviews at service visits (x 8) How did the program influence and improve processes and practices at the service level? What were the most significant changes that occurred in services participating in the program? What is the likely sustainability of these changes? Managers and/or directors of participating services Coaches Case impact profiles and review of services rated through NQS - Interviews - Coaching reports and interviews with coaches 5. What are the lessons learned what is required to support early childhood services in Victoria? How does coaching compare with other professional development options? What strengthened and inhibited effective implementation of the coaching program? -Costing information from Gowrie Victoria - Coaches - Service Impact profiles (x3) re implementation of the coaching program - Retrieval from Gowrie Victoria records - Interviews (individual and group) - Interviews and service visit
The table below outlines the original model and the coaching program model offered in 2011–2013. 2: Comparison of coaching programs Features Pilot coaching program (2010-2011) Coaching program (2011-2013) Number of services 92 54 Selection of services Expression of interest and criteria for selection Specified criteria of service need for coaching Expression of Interest by services Types of services involved 4 Family Day Care 4 Out of School Hours Care 1 Kindergarten cluster/OSHC 32 Kindergarten 1 Long Day Care and Kindergarten 44 Long Day Care 3 Preschool 1Multifunctional Aboriginal Children's Service (MACS) 36 LDC 11 Kindergarten 4 LDC/Kindergarten 1 LDC/Occasional care 1 OSHC 1 LDC/OSHC Timeframe of coaching process 16 months 16 months Program 6 visits 2 consultancy visits and 6 coaching visits Number of Coaches 6 4 Preparation and support offered to Coaches Professional development offered to coaches in approved learning frameworks and NQF (2011) monthly coach meeting All four coaches had been engaged in the project so no further professional development in this area was required -Monthly coaching meetings Working Relationship Work with change agents in service and other educators where possible Work primarily with a nominated educational leadership team to enhance spread across service, build leadership capability and foster sustainability. (observe and provide feedback) to as many other educators as possible in time/scope of visits of Visits 1. Relationship building 2. Practice principles audit 3. Reflective practice and Practitioner Inquiry Projects 4. Linking learning frameworks to NQS quality areas 1 and 5 5. Quality Improvement Plan 6. Review and directions Consultancy Visit (x2 days): -identify leadership and governance issues in the service - leadership capacity - initiate a quality improvement plan (QIP) - Coaching Visits (x6). Specific tools used when judged appropriate by the coach Materials and other resources offered to Monthly contact Rural network meetings Practice principles audit Coaches maintained the resources used in the pilot in the coaching program. New elements included:
work with educators. A collaborative approach was valued highly and could contribute to better outcomes for educators, services and the children in their care. There is potential for leadership and coaching skills of educational leaders to be further enhanced through professional development options, and further targeted coaching or mentoring. Further assessment of the commitment or needs of educational leaders in this area may be beneficial. A toolkit of coaching strategies and materials, including a needs analysis tool could be a useful resource for educational leaders seeking to build additional leadership and coaching capability. Lessons Learned 5.Creating Sustainable Change and Improvement There were differences in the level of support for, and knowledge of, the VEYLDF among approved providers, directors and/or managers according to educational leaders and coaches interviewed in this evaluation. Coaches and managers reported that low levels of knowledge and support were a barrier to progressing shifts in practice. This was particularly highlighted by educators trying to get additional resources, or in some cases fix or maintain equipment to an appropriate standard. Coaches noted that coaching had little influence on service quality if the manager or director resisted change. They noted that progress in some services could only be made following a change in leadership. In circumstances and contexts where directors and managers do not support the intent or philosophies underpinning the VEYLDF, it is doubtful that gains made at the service level will be maintained unless there is another impetus for maintaining quality. The introduction and ongoing assessments that form part of the NQS are likely to maintain a focus on service quality, but there will need to tailored support and consequences for services not meeting service standards. Lesson Learned 6. Optimum Conditions for Coaching Coaching undertaken under the right conditions – where there is commitment by the approved provider, manager, educational leadership team and educators – is likely to be effective in changing practice. Where there are multiple barriers or intractable leadership, coaching may not have much impact, and sustainable change is unlikely. Providing face-to-face coaching support is comparatively costly relative to other support interventions and the investment required will produce better outcomes if the service is engaged in a multi-faceted program of reform. Lesson Learned 7. Sustainability of Practice Changes through Networks of Support