† Percentage of valid responses (not including “not applicable”) from direct patient care staff ‡ Percentage of total responses from direct patient care staff Table 3: Responses (%) to items omitted from final factor analysis Teamwork items †Disagree strongly †Disagree slightly †Neutral †Agree slightly †Agree strongly ‡Not applicable *Where I work, it is difficult to speak up if I perceive a problem with patient care. 6 12 4 23 55 2 *I am frequently unable to express disagreement with the senior clinical staff here. 9 13 14 27 37 6 Briefing staff on handovers between shifts / periods of work (i.e. to plan for possible contingencies) is important for patient safety. 0.4 0.3 3 12 84 15 Safety climate items *Staff frequently disregard rules or guidelines (e.g. hand-washing, treatment protocols / clinical pathways, etc.) that are established for the area where I work. 7 10 8 22 53 2 *Where I work, it is difficult to discuss errors. 4 11 9 30 46 1 † Percentage of valid responses (not including “not applicable”) from direct patient care staff ‡ Percentage of total responses from direct patient care staff *These items are negatively worded Table 4: Mean score for all items in each factor (scale) Factor Mean score 95% CI Teamwork factor 1 Input into decisions and collaboration with other staff 4.25 4.21 – 4.30 Teamwork factor 2 Information handover 4.24 4.19 – 4.29 Safety Climate factor 1 Attitudes to safety within own team; capacity to learn from errors 4.10 4.05 – 4.14 Safety Climate factor 2 Overall confidence in safety of organisation 3.68 3.61 – 3.75
(ii) Confirmatory factor analyses run separately on subgroups Subgroup 2 CFI TLI RMSEA SRMR Primary care 122 on 43df 0.858 0.818 0.116 0.075 Hospital 83 on 43df 0.938 0.921 0.075 0.051
Initially, an analysis of patterns and trends in reporting was undertaken to explore whether higher reporting rates were associated with particular hospital characteristics or types of incident (Table 1). Secondly, the relationship between reporting rates and other safety data was explored (Table 2). Table 1: Analysis of patterns and trends in reporting Analyses Datasets used Trend in reporting rates over time Number of reports to NRLS per month per hospital, provided by the NPSA Relationship with type/severity of incidents reported Number of reports to NRLS per month per hospital categorised by incident type and severity, provided by the NPSA Relationship with hospital size and characteristics Healthcare Commission definitions of hospital type and size.[20] Data on patient age, gender, length of stay, and proportions of emergency and day-case admissions for 2004-5, obtained from the Hospital Episodes Statistics (HES) website[21] Table 2: Analysis of relationships between reporting rates and other safety data Analyses Datasets used Relationship with safety culture data Proportion of staff at each hospital giving positive responses to the 2004 and 2005 NHS Staff Survey questions on safety culture and incident reporting, obtained from the Healthcare Commission website[18] Risk management ratings per hospital awarded by the NHS Litigation Authority, ranging from 0 (lowest standard) to 3 (highest standard); data for December 2005 were obtained from the NHS Litigation Authority website[19] Relationship with reporting rates to other systems Number of reports of Methicillin-Resistant Staphylococcus Aureus (MRSA) bacteraemias during 2004-5, obtained from the Department of Health website[16] Number of reports about medical devices to the Medicines and Healthcare Products Regulatory Agency (MHRA)[17] during 2005, obtained from the MHRA via the NPSA Relationship with outcome data Hospital Standardised Mortality Ratios (HSMRs) for each hospital for 2004-5, obtained from the Dr Foster website[22] Hospital-level data on Patient Safety Indicators, measures of safety and quality which can be calculated using HES data. These indicators were adapted by the Healthcare Commission and the NPSA from those developed by the Agency for Healthcare Research and Quality (AHRQ) in the US;[23] data were provided by the NPSA.
error/near miss they saw (2004 survey data, regression coefficient 0.04, 95% CI 0.01 to 0.06, p = 0.005, 2005 survey data, regression coefficient 0.04, 95% CI 0.01 to 0.07, p = 0.015). There was an additional correlation in the 2005 data in which as the number of staff indicating that they had a moving and handling injury increased there was a decrease in the number of reported overall incidents per 100 bed days per year (correlation coefficient 95% CI -0.090 to -0.002, p = 0.041). A similar non-significant trend was found in the 2004 data. The NHS Litigation Authority (NHSLA)[19] conducts risk management assessments within hospitals. Organisations can choose whether to be assessed for Level 1, 2 or 3, with Level 3 being the highest (Table 4), and receive a financial incentive for achieving a higher Level. We explored the relationship between reporting rates and NHSLA risk management ratings. The 10 hospitals achieving Level 3 had significantly higher reporting rates than hospitals at Level 1 or 2 (Table 4), although there was no significant difference in reporting rates between hospitals at Levels 1 and 2. Table 4: Relationship between reporting rates and NHS Litigation Authority (NHSLA) risk management ratings NHSLA risk management rating Interpretation of ratings Number of hospitals Mean reporting rate per 100 bed days (95% CI) Significant differences in reporting rates Level 0 Not achieved Level 1 1 3.89 (n/a) Between Levels 1 and 3: p = 0.026 Between Levels 2 and 3: p = 0.037 Level 1 Effective risk management systems and processes established 88 0.82 (0.68 to 0.96) Level 2 Systems described for Level 1 have been implemented 49 0.84 (0.61 to 1.06) Level 3 Organisation is monitoring compliance with systems and acting on findings 10 1.36 (1.06 to 1.66) n/a – not applicable Relationship between reporting rates and other safety and quality data
Appendix 1: COPD review criteria ……………………………………………………...101 Appendix 2: Heart failure review criteria ……………………………………………….111 Appendix 3: Validity of review criterion questionnaire (COPD) ……………………..119 Appendix 4: Screen shot of a holistic review data collection page …………………126 Appendix 5: Reviewer training scenarios to assist in recognising variation in care quality using holistic review ……………………………………………………………..127 Appendix 6: Example of a reviewer feedback report ……………………………......130 Appendix 7: COPD – correlations between holistic mean overall scale scores and outcome variables ……………………………………………………………………….153 Appendix 8: Heart Failure – correlations between holistic mean overall scale scores and outcome variables …………………………………………………………………..154 Appendix 9: COPD – correlations between holistic mean phase scale scores and outcome variables ………………………………………………………………………..155 Appendix 10: Heart Failure – correlations between holistic mean overall scale scores and outcome variables …………………………………………………………………..156 Appendix 11: COPD – correlations between holistic mean criterion scores and outcome variables ………………………………………………………………………..157 Appendix 12: Heart Failure – correlations between mean criterion scores and outcome variables ………………………………………………………………………..158
There are relatively few admissions per hospital per year of new cases of heart failure or COPD and much of the diagnostic work-up is undertaken in the primary care or outpatient setting. We therefore chose to study cases of admission for an exacerbation of either of these two tracer conditions and excluded admissions for diagnostic workup. These working definitions for data collection were: Definition of an exacerbation of COPD: “An exacerbation is a sustained worsening of the patient’s symptoms from their usual stable state which is beyond normal day-to-day variations, and is acute in onset. Commonly reported symptoms are worsening breathlessness, cough, increased sputum production and change in sputum colour.”[31] Definition of an exacerbation of heart failure: "An exacerbation of heart failure is a sustained worsening of the patient's symptoms from their usual stable state which is beyond normal day-to-day variations, and is acute in onset. Commonly reported symptoms are worsening breathlessness, tiredness and swelling of the feet and/or ankles."[23] Choosing the number of case notes for review In choosing the number of case notes for review we were unable to use prior hypotheses to assist in determining how many case notes would be required for the reliability studies. We considered using van Belle’s method[32] of calculating the number of events (e.g. identified opportunities for error as being event = 20 times the number of parameters) from which to assist with this calculation, but subsequently found that it was practically impossible to model the range of opportunities for error presenting in these complex care pathways. In addition it was decided that the study was more likely to find variations in care rather than identifiable adverse events and that in health care there could be very large opportunities for error. We therefore took a pragmatic decision to select approximately 50 case notes per condition per hospital. This number also fitted with the custom and practice of the Royal College of Physicians CEEu in which about 60 case notes per hospital form the basis for review in national clinical audits and this number of case notes had previously provided sufficient data for studies of inter-rater reliability.[24]
(%) (%) (%) Doctors (n = 273 reviews) Number of completed rating scales 269 (98.5) 265 (97) 267 (97.8) 272 (99.6) 1073 (97.4) Missing data 4 (1.5) 8 (3) 6 (2.2) 1 (0.4) 19 2.6 Non-clinical audit (n = 271 reviews) Number of completed rating scales 260 (96) 261 (96.3) 263 (97) 227 (83.8) 1011 (93.3) Missing data 11 (4) 10 (3.7) 8 (3) 44 (16.2) 73 (6.7) Clinical (n = 357 reviews) Number of completed rating scales 341 (95.5) 332 (93) 326 (91.3) 353 (98.9) 1352 (94.7) Missing data 16 (4.5) 25 (7) 31 (8.7) 4 (1.1) 76 (5.3) Table 4: Completion rates for Heart Failure holistic reviews
For the COPD reviews, across all of the phases of care and overall care, for both conditions, there were statistically significant differences between the types of comments made by the three types of staff for COPD, with the medical staff using the highest level codes across all phases of care and overall care (Table 11). There is therefore some evidence to suggest that doctors are more likely to use explicit codes than the other staff groups. Table 11: Types of comment made by different staff types for COPD (“highest level” comment per review) Admission Initial management Pre-discharge Overall Coding bands or audit Doctor (nurse or other) audit Doctor or audit Doctor or audit Codes 1-5 (Little or no comment about care and little of no judgement) 5% 14% 41% 1% 12% 38% 4% 13% 47% 4% 16% 47% Codes 6-8 (Limited comment about care and implied judgement) 15% 32% 56% 15% 36% 48% 27% 53% 48% 4% 15% 7% Code 9-13 (explicit judgement of care [9] or more sophisticated comments about care with explicit judgements and views) 81% 54% 4% 84% 52% 13% 68% 34% 6% 93% 69% 46% Number of reviews (and reviewers): 258 (6) 357 (8) 271 (7) 273 (6) 356 (8) 271 (7) 273 (6) 355 (8) 271 (7) 273 (6) 352 (8) 267 (7) Chi-square: 338.0 (p < 0.001) 306.2 (p < 0.001) 313.1 (p < 0.001) 190.3 (p < 0.001) For heart failure reviews, while there are still statistically significant differences between the three groups, the differences between the groups are less strong, with similarities the doctors and nurses for phases of care and little difference between the groups for overall care (Table 12).
COPD 268 (3) 34.12 16.7 5 - 105 8.53 4.17 1.25 – 26.25 All reviews 881 (20) Heart Failure 309 (21) 24.1 10 5 - 50 12.96 5.84 2.75 – 24.75 Heart Failure Other Clinical 177 (3) 29 21 10 - 180 10.54 7.80 3.33 – 60.00 Heart Failure 63 (7) 27.1 9.5 10 - 50 6.78 2.37 2.50 – 12.50 All reviews 493 (27) Table 14: Resource used during the criterion review process, by staff type Staff type No. valid reviews Missing data () Mean review time in minutes (SD) SD time per review Time per review Min - Max Mean total cost per review (£) SD cost per review (£) Cost per review (£) COPD Doctors 254 (0) 19.5 7.2 6 - 50 9.16 5.21 2.50 – 32.50 COPD Nurses/ Other Clinical 335 (1) 15.4 5.8 3 - 45 5.57 2.16 1.00 – 15.00 COPD 243 (1) 30 26.2 10 - 280 6.00 2.52 1.33 – 18.75 All reviews 832 (2)
16, the reviewers were using implicit judgements to score care in Table 15 but there were explicit standards set through the review criteria in Table 16. Judgement based holistic scale scoring tends to show larger ‘tails’ at both ends of scoring than does criterion scoring, which is unlikely to be influenced by outcome (Figures 8 – 11). For ‘poor’ care at the individual case level, reviewers using holistic methods can be very critical, as some of our textual data shows, and the reverse is the case for good care. The criterion based review data that are presented in Table 16 are also shown in Figures 10 and 11. Table 16: Relationship between the mean criterion score (scaled to 100) and mortality group Staff type Condition Higher mortality (n=4) N; mean (SD) Lower mortality (n=5) N; mean (SD) Mean 95% interval of the difference P-value Doctors COPD 187; 77.33 (10.47) 67; 77.99 (7.85) 0.67 -5.63 to 6.97 0.796 COPD 120; 74.26 (12.90) 124; 74.78 (9.31) 0.52 -9.09 to 10.13 0.895 Clinical COPD 189; 73.63 (11.98) 147; 76.02 (9.73) 2.39 -5.54 to 10.33 0.499 All staff COPD 496; 75.18 (11.77) 338; 75.96 (9.28) 0.78 -3.27 to 4.83 0.691 Doctors Heart Failure 163; 71.41 (14.65) 150; 77.90 (9.23) 6.49 1.52 to 11.46 0.016* Heart Failure 10; 79.00 (7.74) 61; 74.96 (8.72) -4.04 -20.39 to 12.31 0.399 Clinical Heart Failure 135; 80.0 (5.62) 42; 80.65 3.86) 0.65 -2.98 to 4.27 0.664
Percentage of COPD patients who die in hospital within 28 days -0.290 0.215 Weak HSMR from Dr. Foster (3 year mortality) -0.135 0.569 Very weak HSMR from Dr. Foster (1 year mortality) -0.095 0.691 Very weak Incidents to NPSA per 100 bed days -0.067 0.778 Very weak SMR for deaths in low mortality HRG’s 0.070 0.771 Very weak COPD finished consultant episodes 0.422 0.072 Moderate COPD bed days 0.387 0.102 Weak COPD mean length of stay -0.146 0.550 Very weak COPD mean age 0.108 0.658 Very weak Star rating (0 worst to 3 best) 0.035* 0.882 Very weak Use of resources HCC -0.054* 0.821 Very weak Patient’s experience -0.365 0.114 Weak Quality of services -0.083 0.729 Very weak Percentage of patient’s with acute MI receiving thrombolysis 0.350 0.142 Weak Existing national targets -0.054* 0.821 Very weak New national targets 0.073* 0.759 Very weak NHS staff survey Q25a: Seen errors in the past month (% yes) -0.236 0.316 Weak NHS staff survey Q27b: Encouraged to report errors (mean) -0.330 0.156 Weak NHS staff survey Q27e: Trust takes action to ensure does not happen again (mean) 0.038 0.872 Very weak NHS staff survey Q24a: Know how to report (% yes) -0.248 0.292 Weak NHS staff survey Q24b: System for reporting (% yes) -0.203 0.390 Weak NHS staff survey Q24b: System for reporting (% no) -0.103 0.664 Very weak NHS staff survey Q24b: System for reporting (% DK) 0.238 0.312 Weak NHS staff survey Q22e: Care of patient/service -0.454 0.044 Moderate
(4.00 – 5.00) (4.00 – 5.00) (3.3 – 5.0) Min - max 3.0 – 5.0 3.0 – 6.0 2.0 – 6.0) phase quality of care rating Mean (SD) 4.5 (0.6) 5.0 (0.7) 3.8 (1.1) Median (IQR) 5.0 (4.0 – 5.0) 5.0 (5.0 – 5.0) 4.0 (3.0 – 5.0) Min - max 3.0 – 5.0 3.0 – 6.0 1.0 – 5.0 Overall phase quality of care rating Mean (SD) 4.4 (0.5) 4.8 (0.9) 4.0 (0.9) Median (IQR) 4.0 (4.0 – 5.0) 5.0 (4.5 – 5.0) 4.0 (3.0 – 5.0) Min - max 4.0– 5.0 1.0 – 6.0 2.0 – 5.0 Box and whisker plots of holistic scale score data The box and whisker plots compare the median quality of care scale rating and the inter-quartile range, for each reviewer for each phase of care. Figure 13.1: Holistic phase quality of care rating Admission phase Initial management phase Pre-discharge phase Total score
Quality of care scores were assigned to the criterion in the admission, initial management and pre-discharge phases. The method used was similar to that in the Phase One study, whereby each time a criterion was met or done or the reviewer selected the not done for a valid reason option a score of 1 was given. If a reviewer selected not recorded, the review item did not receive a score as this option presumes the care was not provided. The mean and median quality of care scores are presented in table 13.5. Table 13.5: Criterion based quality of care scores Phase of care Clinical reviewer (Nurse Doctor 1 reviewer Doctor 2 reviewer Admission Phase (out of 11) Mean (SD) 9.2 (1.2) 10.5 (0.6) 7.8 (2.3) Median (IQR) 9.0 (9.0 – 10.0) 11.0 (10.0 – 11.0) 8.0 (6.0 – 10.0) Min – Max 6.0 – 11.0 9.0 – 11.0 3.0 – 11.0 Initial Management Phase (out of 6) Mean (SD) 5.2 (0.9) 5.8 (0.4) 4.6 (1.2) Median (IQR) 5.0 (5.0 – 6.0) 6.0 (6.0 – 6.0) 5.0 (4.0 – 5.0) Min – Max 2.0 – 6.0 5.0 – 6.0 1.0 – 6.0) Pre-discharge phase (out of 16) Mean (SD) 13.5 (1.3) 15.2 (1.9) 10.1 (2.8) Median (IQR) 13.0 (13.0 – 14.5) 16.0 (15.0 – 16.0) 10.0 (8.0 – 12.0) Min – Max 11.0 – 16.0 5.0 – 16.0 4.0 – 16.0 Total score (sum of all phases, max 33) Mean (SD) 28.0 (2.3) 31.2 (2.0) 22.6 (4.5) Median 29.0 32.0 22.6