Figure CHSD Evaluation Framework Level I Impacton, and outcomes for, consumers (clients, families, carers, friends, communities) EVALUATION HIERARCHY What did you do? PROGRAM / PROJECT DELIVERY How did it go? PROGRAM / PROJECT IMPACT Can you keep going? PROGRAM / PROJECT SUSTAINABILITY What has been learnt? PROGRAM / PROJECT CAPACITY BUILDING Are your lessons useful for someone else? PROGRAM / PROJECT GENERALISABILITY Who did you tell? DISSEMINATION Outcomes, indicators and measures to be developed for each cell as relevant Describe what was implemented and, if necessary, contrast to what was planned Impact on consumers and carers Sustainability assessment Capacity building assessment Generalisability assessment Dissemination log Level 2 Impact on, and outcomes for, providers (staff, formal carers, professionals, volunteers, organisations) EVALUATION HIERARCHY What did you do? PROGRAM / PROJECT DELIVERY How did it go? PROGRAM / PROJECT IMPACT Can you keep going? PROGRAM / PROJECT SUSTAINABILITY What has been learnt? PROGRAM / PROJECT CAPACITY BUILDING Are your lessons useful for someone else? PROGRAM / PROJECT GENERALISABILITY Who did you tell? DISSEMINATION Outcomes, indicators and measures to be developed for each cell as relevant Describe what was implemented and, if necessary, contrast to what was planned Impact on professionals, volunteers, organisations Sustainability assessment Capacity building assessment Generalisability assessment Dissemination log 3 Impact on, and outcomes for, the system (structures, processes, networks, relationships) EVALUATION HIERARCHY What did you do? PROGRAM / PROJECT DELIVERY How did it go? PROGRAM / PROJECT IMPACT Can you keep going? PROGRAM / PROJECT SUSTAINABILITY What has been learnt? PROGRAM / PROJECT CAPACITY BUILDING Are your lessons useful for someone else? PROGRAM / PROJECT GENERALISABILITY Who did you tell? DISSEMINATION Outcomes, indicators and measures to be developed for each cell as relevant Describe what was implemented and, if necessary, contrast to what was planned System level impacts, including external relationships Sustainability assessment Capacity building assessment Generalisability assessment Dissemination log Evaluation approach While the overall EBPAC initiatives were seeking to have an impact at each of the three levels identified in the framework, this was not necessarily the case for individual projects. In some instances, not all cells within the CHSD Evaluation Framework were relevant. For example, individual projects may have been aiming to have an impact at one, two or all
activities with the lowest ratings were newspaper article and radio interview. Given the small numbers involved in how many times some of these activities were recorded the result should be interpreted with some caution. Summary Those dissemination activities that were most likely to be used, followed up and considered most effective by projects appeared to be: Presentation at a conference (national or state/territory or local) Story in a magazine or newsletter (local, industry or professional) Information provided on a website Brochures, leaflets or posters in a health and community setting Peer reviewed journal article Project newsletter The main purpose of dissemination activities was for awareness and understanding with relatively fewer activities aimed at promoting action. DISCUSSION This report has detailed the overall impact and outcome of the latest round of EBPAC projects, and raises important insights into facilitating evidence-based practice within a complex and increasingly competitive aged care sector. What was achieved? There is no doubt that the Australian Government has provided significant investment to improve the delivery of evidence-based practice for aged care residents. This latest round of EBPAC projects has involved several thousands of aged care workers nationally. It builds on the earlier EBPAC rounds which, at the time of writing that evaluation report, we noted was the ‘most comprehensive, coordinated approach to implementing evidence-base practice in residential aged care’ within Australia. The outcomes of these projects have been promoted widely through the Aged Care Quality Agency’s Better Practice conferences and through its website. The residential aged care sector has subsequently been systematically equipped with resource kits and training and consultancy initiatives derived from five of the thirteen Rounds 1 and 2 projects: Better Oral Health in residential aged care: 396 workshops were delivered nationally, with 4,885 people trained from 2,809 aged care homes, multi-purpose services and Indigenous flexible care services (2009-10) Dementia Dynamics Toolkit: distribution of free toolkits to all residential aged care facilities, supported by the delivery of 30 workshops nationally (2014-2015)
Fourteen out of 17 participants (82%) thought that the use of evidence-based practice had improved in their facility. Only participant felt that it had not improved and two were not sure. Thirteen participants provided feedback the use of EBP in their facility. The majority of feedback related to the use of the toolkit by staff. Participants indicated that the use of the toolkit had been helpful in promoting a consistent documented approach. This has promoted confidence and teamwork, as this participant states: Our staff are more confident in what they are doing and RNs have a better understanding of advanced care planning and pain management. The EoL pathway encourages team work. Some participants noted gaining expertise through avenues other than the toolkit. The Joanna Briggs Institute was mentioned but also local specialist palliative care services have also been accessed for training and information support. Have the knowledge and skills of aged care workers improved following implementation? Participants who answered this question (n=16) mostly indicated that the knowledge and skills of aged care workers at their facility had improved following implementation (n=14; 82%). One participant felt that skills and knowledge had not improved and one was not sure. All participants who felt there had been improvements also provided details. Most participants felt that there had been an increase in the knowledge and skills of aged care workers. Evidence for these improvements was clear in some cases, with positive feedback from residents and families: We have had extremely positive feedback from families about palliation. The toolkit helps this and the training we offer staff is important too. Palliative care is a key focus for us. We are building a caring team. Some participants discussed how staff confidence had also improved as a result of the improvements in skills and knowledge, for example: Compared to 12 months ago we are more certain as to what to do with our palliative patients. Now staff are more confident and they are more independent and less reliant on RN advice. Some participants felt that improvements had occurred at all levels of staff, including RNs, ENs, AINs and other staff. Several participants gave examples of staff within their facility that’s had improved their knowledge and skills. …we have nominated AIN champions too and provided education for all. This is across all disciplines - and hospitality staff. We use the toolkit as an educational resource.