Scope of this review In scope Out of scope Those data quality variables which relate to the movement and transformations of data through the data chain which begins with Clinical Information Systems (CIS) and ends at the Australian Institute of Health and Welfare (AIHW). These include: CIS data definitions CIS data extract for the Clinical Tool (CAT) Data transformations to calculate nKPI values Clinical data quality or consistency issues due to individual health services' own data practices- i.e. the quality of data in the CIS themselves nKPI indicator definitions The interpretation of nKPI data by AIHW nKPI data submissions from health services in the Northern Territory using the PCIS system High-level findings The review has found no evidence of system-wide technical problems affecting nKPI data quality. Compared with the later Healthy for Life data collections (2010-2011), nKPI health services are achieving higher levels of compliance more quickly, with comparable levels of data validity. While the collection of nKPIs to date can be regarded as a solid start, the review identifies a number of areas of focus for improving future nKPI data collections. These include recommendations to improve the: Accuracy and completeness of data for specific indicators nKPI functionality of specific clinical information systems Tools provided to health services to preview and improve their nKPI data A key determinant of future nKPI data quality will be enhancing the ongoing capability and capacity of reporting health services to capture, clean, manage and interpret their patient data. While nKPI reporting compliance is high, low levels of engagement in parts of the Aboriginal health sector suggest that challenges remain in moving the nKPI collection from a reporting compliance activity to an embedded CQI activity. The structure of this report To aid the reader, the structural flow of this report is illustrated below in Figure 1.
According to the AIHW’s METeOR specification of the the intended objectives of the indicators are to: Indicate the health issues pertaining to the regular client population of Indigenous-specific primary health care services (especially those of maternal health, early childhood and the detection and prevention of chronic diseases) Outline the extent to which government-funded primary health care services collect, record and review pertinent data on these issues, and Reveal changes in health risks or outcomes that may be driven by the quality of care that government-funded services provide to their clients. Of those objectives, items 1 and 2 can be reviewed even at this early stage in the life of the nKPI collection, while item 3 will only emerge as further data builds up over time. Therefore, this review will consider objectives 1 and 2 only. Dimensions of data quality The DQF describes seven dimensions of data quality. These dimensions will be more or less appropriate or important for any particular data set, depending on its objectives, characteristics and circumstances of collection, and the scope and purpose of the data quality review process. Figure 2: Dimensions of data quality The seven dimensions are defined as follows: Dimension Description Relevance to nKPIs Relevance to current nKPI data quality review Institutional Environment This dimension refers to the institutional and organisational factors which may have a significant influence on the effectiveness and credibility of the agency producing the statistics. Consideration of High Aboriginal primary health care services vary greatly in size, staffing levels, degree of remoteness, mix of health services provided, approaches to data collection High Many health service environmental factors have a bearing on the collection of data for the nKPIs, including: Capability and capacity of staff in data entry, cleansing and management The clinical information
the institutional environment associated with a statistical product is important as it enables an assessment of the surrounding context, which may influence the validity, reliability or appropriateness of the product. and management, and choice of information systems system at the health service The rate of staff turnover at the health service Relevance This dimension refers to how well the statistical product or release meets the needs of users in terms of the concept(s) measured, and the population(s) represented. Consideration of the relevance associated with a statistical product is important as it enables an assessment of whether the product addresses the issues most important to policy-makers, researchers and to the broader High The right concepts must be measured for the nKPIs to be useful. For the nKPIs, the concepts and definitions have been developed by the Technical Working Group, and documented and maintained by the AIHW Low The concepts measured and population represented by the nKPIs are outside the scope of the nKPI data quality review
Australian community. Timeliness Timeliness refers to the delay between the reference period (to which the data pertain) and the date at which the data become available High If data is not made available quickly to those who need it, its value may be reduced Medium Relevant aspects include: The availability of information to health services through OCHREStreams AIHW turnaround times for commentary on health service data AIHW turnaround times on acceptance of health service data AIHW turnaround times for development of health service final reports Accuracy Accuracy refers to the degree to which the data correctly describe the phenomenon they were designed to measure. This is an important component of quality as it relates to how well the data portray reality, which has clear implications for High If nKPI data is not sufficiently accurate, it is less able to fulfil its objectives High Relevant aspects of accuracy include: Coverage: consistent application of the agreed client definition for submitted data Non-response: the extent to which data are missing from responses Response: the extent to which
review these data items. Interpretability refers to the availability of information to help provide insight into the data. Information available which could assist interpretation may include the variables used, the availability of metadata, including concepts, classifications, and measures of accuracy. High Ease of interpretability may have a bearing on levels of nKPI take up and local use within health services for quality improvement High Current sources of nKPI interpretability information for health services and other stakeholders are: The AIHW METeOR specification for the nKPIs The AIHW national nKPI report (the first national report is scheduled for publication in May 2014) In addition, participating health services receive a report prepared by the AIHW following the acceptance of their data for each six monthly collection. Accessibility Accessibility refers to the ease of access to data by users, including the ease with which the existence of information can be ascertained, as well as the suitability of the form or medium through which information can be accessed. High If data cannot be readily accessed by authorised its value may be reduced Medium Relevant aspects include: The ability of health services to view and retrieve their own submitted data through OCHREStreams The ability of health services to initiate, respond to requests for,
Reporting health services are able to include comments to AIHW in reports, either at individual indicator level or at global level. Analysis of de-identified comments suggests the following: There is no evidence from health services’ perspective of system-wide issues in the nKPI collection process from CIS through the CAT to OCHREStreams. There have been concerns noted with the MMeX submission to OCHREStreams, however, these are well known and there are promising developments that suggest future improvements in this area which are outlined below. Many health services comment that they are working to get better at their data capture and management practices in order to improve their nKPI submission for future collections A high proportion of comments lodged to the AIHW are issues that are attributable to health services’ own clinical practices and/or their use of their Clinical Information System. Services’ contact with Improvement Foundation help desk As part of the arrangements for operating and maintaining OCHREStreams, the Improvement Foundation provides a help desk function to assist health services on a wide range of topics. Based on help desk data provided by the IF, analysis indicates that only 1.7% of total contacts with the OCHREStreams help desk during the last two years were attributable to data submission questions or problems which could not be immediately resolved by the help desk. In these cases, the IF strongly recommended that the service include a comment about the problem to the AIHW. The low proportion of data quality calls to the help desk suggests that system data quality is not perceived as a major concern by health services. Improvements across process of care indicators While the nKPI set is a new collection, to date some of the indicators have been reported up to four times by some health services (June 2012, Dec 2012, June 2013, Dec 2013). In the nKPI: First National Results (June 2012 to June 2013), the AIHW reports overall improvements in nKPIs covered by multiple reporting periods, namely: The proportion of babies with birth weight recorded MBS health assessments for adults Clients with Type 2 diabetes who received a Team Care Arrangement Recording of smoking and alcohol status Additionally, an encouraging number of health services are reporting complete, or near-complete, results for various process of care indicators, for example: 28 organisation achieved 100% recording for weight The top 25% of organisations recorded birth weight for 88% or more of their clients 8 organisations achieved 100% recording for smoking status
Medium/ Long) (Low/ Medium/ High) overall nKPI data quality Medium/ Large) 2= Med 3= Low 34 Timeliness of data extraction Improvement Foundation to increase health service awareness of the importance of timely data extraction the CAT scheduler (for services) 21 Medium Low Small 2 35 Use of Exception Reporting AIHW to monitor health service access to exception reporting, as indicator data from health service systems improves, with a view to phasing out completely over time Long Low Medium 2
34 Timeliness Improvement Foundation to increase health service awareness of the importance of timely data extraction using the CAT scheduler (for non-Communicare services) Low Improvement Foundation 35 Exception reporting AIHW to monitor health service access to exception reporting, as indicator data from health service systems improves, with a view to phasing out completely over time Low AIHW 7 Indicators that rely on pathology results Improvement Foundation to request that the Data Expert Group works towards a coordinated approach by software vendors for more complete capture of non-numeric pathology data Medium Improvement Foundation 13 Alcohol status (PI16) Improvement Foundation to provide input to support the development by software vendors of a more intuitive approach to reviewing and maintaining alcohol status in patient records Medium Improvement Foundation 27 Engagement OCHREStreams Advisory Group to seek ideas from Affiliates and key health services on ways to increase and sustain service engagement. Medium OCHREStreams Advisory Group 28 Engagement OCHREStreams Advisory Group to promote awareness by health services (and affiliates) of how nKPI data is being and will be used Medium OCHREStreams Advisory Group 29 Engagement OCHREStreams Advisory Group to consider how NACCHO and Affiliates can become Medium OCHREStreams Advisory Group