Intelligent Monitoring NHS acute hospitals

Intelligent Monitoring NHS acute hospitals Statistical methodology July 04

Contents. Introduction. Analysis of cross-sectional data using z-scores. Z-scores. Over-dispersion 4 3. Cross-sectional analysis of raw counts data 6 3. Poisson events 6 3. Negative binomial distributions 7 4. Analysis of time series 9 4. Introduction 9 4. CUSUMs of z-scores 0 4.3 CUSUMs of rare events data 3 5. Aggregate scoring methods 4 5. SUS Data Quality: Trust level aggregate scoring 4 5. Electronic Staff Records (ESR) data 7 6. Analyses carried out by external organisations 7. Further reading Intelligent Monitoring: NHS acute hospitals - statistical methodology Page of

. Introduction This document describes, in some detail, the statistical methods we have used to analyse the data that supports the new risk assessment model. This analysis is relevant to the majority of indicators, but not all, as some are already analysed by external organisations. Indicators that are analysed externally are summarised in section 6. Our general approach for this early version of the model is to assess risk by comparing a trust s observed outcomes with others. Where appropriate, we account for the relative sizes of trusts and, in several cases, their variable case mix. We use two types of analysis: Cross-sectional: Assessing risk by comparing trust outcomes over a fixed period of time, for example, annual rates of emergency readmissions, annual reporting of patient safety incidents. Previous values or trends are not accounted for. Time series analysis: Assessing risk by analysing series of reported outcomes over time so that significant deterioration can be detected more quickly.. Analysis of cross-sectional data using z-scores. Z-scores With cross-sectional data we measure the deviation of observed values from an expected or target value. Where we can transform the data into a standard normal distribution we generate z-scores which reflect the number of standard deviations. Suppose the trust value for an indicator is y, and it has an expected or target value t, we can express the deviation of the indicator from the expected value as a z- score, defined as: y t z = s 0 where s 0 is the standard deviation of y if the trust s observed outcomes were randomly distributed about t. Here z is referred to as the unadjusted z-score. Under a null hypothesis that a trust s true level of outcomes is exactly the same as the expected value, z has mean 0 and standard deviation, and if we assume normality, then p-values 0.05 and 0.00 correspond to z = ±.96 and z = ± 3.0 respectively, which corresponds very closely to and 3 standard deviations from the mean. Intelligent Monitoring: NHS acute hospitals - statistical methodology Page of

The default expected values against which a trust is compared are calculated by comparing rates observed for an individual trust against the national rates. However, for some items we standardise by case mix (for example, by age and sex) in order to compare observed outcomes against what you would expect if the rate for each patient was the same as for similar patients over the whole country. Often the raw data are not normally distributed, in which case we use one of the following appropriate transformations... Z-scores from proportions Assume an observed proportion y = r/n, with an expected or target proportion p. The observed proportion is transformed to render it more normally distributed by applying an arcsine transformation to the square root of the observed proportion: Y = arcsin r n The expected value can be approximated by: T = arcsin p and the standard deviation (s) is approximated by: s = n Hence the transformed unadjusted z-score: z = Y T s = n arcsin r arcsin p n.. Z-scores from standardised ratios This method is used when comparing an observed value against an expected value derived using indirect standardisation. We assume a standardised ratio y = O/E based on an observed count O and an expected count E. A square root transformation is applied to the standardised ratio (y): Y = O E which has an expected value approximately equal to one. Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 3 of

Under appropriate Poisson assumptions, the standard deviation approximates to: s = E Thus, the transformed unadjusted z-score is given by: z = Y s = O E..3 Z-scores from ratios of counts We assume a ratio indicator of the form y = O /O, where O and O are both counts, and an average or target ratio t. In order to deal with zero/low counts we add 0.5 to all observations, and, noting that a log transformation reduces positive skewness, the transformed indicator becomes: Y = log e O + 0.5 O + 0.5 with an expected value approximately equal to and a standard deviation: T = log e (t) O s = (O + 0.5) + O (O + 0.5) Thus the transformed, unadjusted z-score becomes: z = Y T s = log e [(O + 0.5) (O + 0.5) ] O (O + 0.5) + O (O + 0.5) If either O or O is much bigger than the other, say when one represents a population, it will have a negligible impact on the score.. Over-dispersion Many z-scores are likely to be over-dispersed, that is their true variances are greater than one, which may be because of insufficient benchmarking or the presence of common-cause factors that render the Poisson model inadequate. Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 4 of

The consequence is that analyses may pick up statistically significant differences that are not of practical importance. When considering an outcome based on an average or expected level, it may then be reasonable to accept as inevitable a degree of between-trust variability and we therefore seek to identify trusts that deviate from this distribution, rather than deviating from a single measure. In order to do this we must estimate the degree of over-dispersion (see Section..). When estimating over-dispersion it may be better to do so using techniques that avoid undue influence of outlying trusts, such as winsorisation (see Section..). The significance of observed deviations then takes into account both the precision with which the indicator is measured within each trust (i.e. the sample size), and the estimated between-trust variability... Winsorisation Winsorisation is the process of transforming outliers in statistical data. In this context it involves shrinking in extreme unadjusted Z-scores to the value of a selected percentile. This is done by:. Ranking trusts according to their unadjusted Z-scores.. Identifying Z q and Z -q, the 00q% most extreme high and low unadjusted Z-scores, where q may be, for example, 0.. 3. Setting the lowest 00q% of unadjusted Z-scores to Z q and the highest 00q% of Z-scores to Z -q. These are the winsorised statistics. This process retains the same number of Z-scores, but protects our estimation of over-dispersion from the influence of actual outliers... Estimating over-dispersion In calculating an adjusted Z-score for an indicator, we estimate the over-dispersion factor phi (φ) as follows: φ = n z i where n is the number of trusts for a data item and z i is the winsorised z-score for the ith trust. (Note: this is just the observed variance of the winsorised z-scores about zero.) Under a null hypothesis that all units only exhibit random variability around the expected value, nφ has an approximate χ n distribution. This can therefore be used as a standard test of heterogeneity. n i= Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 5 of

..3 Calculating adjusted Z-scores We then use the resulting over-dispersion factor to calculate an adjusted Z-score for each observation. The over-dispersion model we use is an additive random effects model. This model assumes that each trust has its own true underlying level t i, and that for non-standard trusts t i is distributed with mean t 0 and standard deviation,τ. In other words the null hypothesis is represented by a distribution rather than a single point. A standard method of moments estimate for τ is: nφ τ (n ) = n n i= w i i= w i n j= w j Where w i = s and nφ is the test for heterogeneity. (s is as calculated in section. with the appropriate transformation.) If nφ (n ) then the adjusted Z-scores are given by: z i = (z i t 0 ) s + τ where z i is equal to the winsorised data value. Otherwise, if φ < (n ), τ is set to zero, complete homogeneity is assumed and no adjustments are necessary. 3. Cross-sectional analysis of raw counts data 3. Poisson events In some instances, for example, when monitoring never events, observations may be sufficiently infrequent that it is not possible to generate sufficiently robust z- scores. Where there is no evidence of over-dispersion, we can assume the events are Poisson distributed and establish levels of risk based on the probability any observed outcome could have happened by chance (the p-value). Suppose X is a random variable representing the number of events reported at a trust over a given period of time and that λ represents the expected number of events, based on national reported rates. If n events are observed, then a p-value can be expressed as: Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 6 of

n p(x > n) = p(x n) + 0.5 p(x = n) = λi i! e λ (where the latter term of the formula is used as a mid-p-value). i=0 + λn n! e λ These p-values can then be used in correspondence with given thresholds of significance to define levels of risk. 3. Negative binomial distributions 3.. Fitting a model to the data For some events the Poisson assumptions may not provide a good fit to the data, in which case a negative binomial distribution may be more appropriate. Also, for some indicators there may be a disproportionate number of zeros among the trustlevel values, necessitating a zero-inflated model. Some of our count-based indicators are correlated with the volumes of patients seen by trusts, and so our assessment of risk needs to identify trusts that have unusually high counts compared with what would be expected given their patient volumes (measured in either bed-days or total patient contacts, both scaled by 00,000). Other indicators consist of negative and positive comments, the numbers of which are often correlated. Here, our assessment of risk needs to identify trusts with unusually high counts of negative comments compared with what would be expected given their count of positive comments. The negative binomial distribution is expressed as: f(y i x i ) = Γ(y i + θ) y i! Γ(θ) θ θ + μ i θ μ i y i, yi = 0,,, θ + μ i Where µ i is the conditional mean, and θ is a positive gamma distribution parameter used to determine the conditional variance. The zero-inflated negative binomial model has two parts a negative binomial count model as above, and a logistic regression model for predicting excess zeros. Additional zeros occur with the probability ϕ i, as determined by: φ i = f(t) = et e t + = + e t where f is the logistic function and t is typically a linear function of one or more explanatory variables. For a given count indicator, we determine which probability distribution Poisson, negative binomial, zero-inflated Poisson, or zero-inflated negative binomial is the best fit to the data, by modelling the raw count as a function of the most Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 7 of

appropriate comparator bed-days, total patient contacts, or count of positive comments. If Y is the count indicator and X the comparator, our model seeks to determine: Pr (Y = y X = x) with estimated parameters μ and θ. If there are no zeros, then we fit only the negative binomial model. If there are zeros, we fit each of the chosen distributions in turn. The measure of the fit of each model is expressed as a log-likelihood. The ratio of the log-likelihoods of the two models is approximately chi-square distributed with degrees of freedom equal to df df, where, df is the degrees of freedom for the zero-inflated model (which is more complex) and df is the degrees of freedom for the ordinary negative binomial model. A statistically significant likelihood ratio test indicates that one model is a better fit than the other. 3.. Identifying extreme values Given the model of best fit, we iteratively establish levels of risk based on the probability that the most extreme outcome could have happened by chance (the p- value). Each model iteration comprises a series of steps, as follows: i. First we condition on Y being positive, such that: Pr(Y = y X = x, Y > 0) = Pr (Y = y X = x) Pr (Y > 0 X = x) = { p 0(x)}f(y x) f(0 x) This conditioning means that we do not need to be overly concerned about the point-mass at zero for the zero-inflated models as we are primarily interested in non-zero counts. ii. Next, we find the trust-level p-values. For example, for trust j: p j = Pr (Y = y X = x j, Y > 0) y y j iii. We find the smallest p-value across all trusts (p min ), and use this to calculate the group-level p-value (p g ), which accounts for multiple comparisons: p g = ( p min ) M This is the probability that none of a sample size of M is more extreme than p min. It is a measure of how much of an outlier p min actually is. Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 8 of

iv. If p g is small ( 0.0) then we conclude that the count associated with p min is too high to have come from the fitted model. The value of p g is therefore used in correspondence with given thresholds of significance to define the level of risk for the trust with p=p min. This trust is then removed from the dataset, and steps i through iv are repeated. If p g is large (> 0.0) then the trust with p=p min is not an outlier, and we stop iterating the model. 4. Analysis of time series 4. Introduction Our approach to time series data is to use statistical process control (SPC) in order to detect sudden or persistent deviations from a reference value. Our favoured SPC technique is the Cumulative Sum or CUSUM. The CUSUM is a sequential hypothesis testing technique by which evidence in favour of outcomes occurring at the reference rate (the null hypothesis, H 0 ) is continually weighed up against evidence that a change has occurred (the alternative hypothesis, H ). CUSUM control charts are plots of the cumulative log likelihood ratio between these two hypotheses. They are also constrained not to fall below zero. This means that, if a CUSUM is designed to detect series of outcomes that are worse than expected, it cannot build up credit for series of good outcomes. If the CUSUM exceeds a predefined threshold, or control limit, then the hypothesis of a change (H ) is accepted in favour of the null (H 0 ) and this constitutes an alert or signal. After each alert, the CUSUM is reset to zero so that if any changes subsequently occur there is time for them to take effect. Alternatively, if poor outcomes persist then a further signal is likely to occur at a later time. Control limits have to be set to guard against too many false alarms occurring as a result of random variation, but not be set at too a high a value that it becomes very difficult to detect any differences in mortality. A CUSUM is illustrated in Figure. Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 9 of

Figure : CUSUM control chart CUSUM value 8 7 6 5 4 3 Control limit Alert signalled 0 3 4 5 6 7 8 9 0 Time period Mathematically, if C t denotes the CUSUM value and w t the log likelihood ratio at time t then: C C 0 t = 0 = max { C + w,0} The values w t are called the CUSUM weights. Note that the CUSUM will rise when the weight is greater than zero, and since these are derived from log likelihood ratios, this corresponds to when H is more likely than H 0. t t 4. CUSUMs of z-scores Outcomes are first converted to z-scores, as described in section. To allow for any over-dispersion the z-score distribution is modelled with a hierarchical structure: if z kt represents the z-score for trust k at time t, z kt θ θ k k ~ ~ N( θ, σ ) k N(0, τ ) i.e. a trust s z-scores are normally distributed about a local mean for that trust, θ k, with standard deviation, σ. These trust mean values are themselves normally distributed about zero with standard deviation, τ. 4.. Estimating variances Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 0 of

Estimates for the local means θ k are derived from the z-scores over a specified time period (t = to T, say): T ˆk = zkt / t= θ To estimate the variances we exclude the trusts with the most extreme local means (the top and bottom 0%) and use the formulae: T σ = N T k = t= ( z N( T ) ˆ kt θ k ) T N ( zkt zt ) t= k = τ = σ ( N ) T where N denotes the number of trusts and zt is the mean z-score over period t. 4.. Hypothesis tests To set a null hypothesis we set a value for the local mean that is in the upper part of its probability distribution: H : θ = 0 k γ τ where γ can be interpreted as a tolerance factor for the mean. We are thus allowing the expected value for a trust to be greater than zero (see Figure ). This is tested against the alternative hypothesis: H : θ = γ τ + k γ σ Figure : Assumed distribution of z-scores for testing higher than expected mortality Intelligent Monitoring: NHS acute hospitals - statistical methodology Page of

Distribution of z-scores under the null hypothesis centered on the national mean (zero) plus a tolerance factor to allow for overdispersion Distribution of z-scores under the alternative hypothesis -4-0 4 6 8 0 We can transform our z-scores into standard normal variates under the null hypothesis by calculating: * z γ τ z = kt kt σ and our hypothesis test becomes: H H 0 * : θ = 0 k * : θ = δ k where δ = γ 4..3 The CUSUM for standard normal data With standard normal data and a hypothesis test as above, the CUSUM weights have values: w * δ t = δ zkt This means that CUSUM will increase if and only if z kt > δ 4..4 Deriving control limits and other stopping rules from steady state p- values For such a CUSUM it is possible to estimate steady-state p-values: Intelligent Monitoring: NHS acute hospitals - statistical methodology Page of

where: p =, = λ e = γ e x x, λ e κx, if x = 0 if 0 < x x ' if x > x ' λ = e λ = 0.073δ + 0.03 κ = e 0.735( δ 0.39).30( δ.3) +.04 δ γ = 0.986(0.56 ) + 0.008δ ' x = 0.70δ +.053δ 0.0 +.098(0.073δ + 0.03) 0.074 Knowing the p-value is necessary for setting a stopping rule based on the false discovery rate (FDR) and also enables constant limits to be set that correspond to pre-specified tails of the null distribution of the CUSUM. We typically use the limit that corresponds to the upper 0.% tail of the steady state distribution. 4.3 CUSUMs of rare events data 4.3. CUSUM formulae CUSUMs are also used for analysing series of rare events. Let λ 0 and λ denote the expected or target frequency under the null and alternative hypotheses, respectively. If we assume Poisson events and there are n t events observed at time period t, then, under the null hypothesis, the likelihood is: n λ λ 0 0 L0 = e n! and the likelihood ratio between the alternative and null hypotheses is: giving a log likelihood ratio: w t L L = n t 0 λ = e λ nt nt 0 ( λ λ ) 0 λ log ( λ λ0 ) λ 0 which become the CUSUM weights. With λ = λ 0 : w t = n t log ( ) λ0 Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 3 of

(Note that if the ratio of observed to expected, n t / λ 0 > / log() ~.44 the CUSUM will increase.) 4.3. Setting the control limit In practical applications of the CUSUM there are several rules for determining where the control limit should be. Commonly they refer to the rate of true and false alarms that would occur, or the expected time to a true or false alarm. Calculating true and false alarm rates associated with different limits is generally not straightforward and often requires either simulating large numbers of CUSUMs or modelling the CUSUM as a Markov Chain. Because the size of the organisation affects the value of λ 0, it will also have an impact on the appropriate position of the control limit. Generally, the larger the value of λ 0, the higher the control limit that is appropriate to ensure each organisation is evaluated equitably. In the current programme this method has been used for never events reported by the Strategic Executive Information System (STEIS), and the control limits were determined so that the probability of a false alarm in any given month is 0.5%. 5. Aggregate scoring methods In this section we describe the methods used to create aggregate indicators for assessing Secondary Uses Service (SUS) data quality and electronic staff records (ESR) indicators. 5. SUS Data Quality: Trust level aggregate scoring A total of 40 items are currently used by CQC each of which measures data quality errors for specific aspects of patient records. Information on errors is currently provided for A&E, Admitted Patients and Outpatients records. Data quality of trust returns to the HSCIC is assessed separately for Inpatients, Outpatients and Accident and Emergency records. Valid data fields are analysed as percentage correct data submission for A&E, Outpatient and Inpatient records. Please see Table for detail of SUS DQ data fields used. Table : SUS DQ numerator/ denominator fields Numerator (each Denominator: Specialty items numerator field processed as the number of correct data returns): APC NHS Number APC CDS SUSAPC0 Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 4 of

APC Treatment Function APC CDS APC Main Specialty APC CDS APC Registered GP Practice APC CDS APC Postcode APC CDS APC Org of Residence APC CDS APC Commissioner APC CDS APC Primary Diagnosis APC CDS (with discharge) APC Primary Procedure APC CDS (with discharge) APC Ethnic Category APC CDS APC Site of Treatment APC CDS (The mean percentage correct data returns for valid fields pertaining to inpatient records) OP NHS Number OP CDS SUSOP0 OP Treatment Function OP CDS OP Main Specialty OP CDS (The mean OP Registered GP Practice OP CDS percentage correct data returns for valid OP Postcode OP CDS fields pertaining to OP Org of Residence OP CDS outpatient records) OP Commissioner OP CDS OP First Attendance OP CDS OP Attendance Indicator OP CDS OP Referral Source OP CDS OP Referral Received Date OP CDS OP Attendance Outcome OP Attendances OP Priority Type OP CDS OP Primary Procedure OP Attendances OP Ethnic Category OP CDS OP Site of Treatment OP CDS A&E NHS Number A&E CDS SUSA&E0 A&E Registered GP Practice A&E CDS A&E Postcode A&E CDS (The mean A&E Org of Residence A&E CDS percentage correct data returns for valid A&E Commissioner A&E CDS fields pertaining to A&E Attendance Disposal A&E CDS accident and A&E Patient Group A&E CDS emergency records) A&E First Investigation A&E CDS A&E First Treatment A&E CDS A&E Conclusion time A&E CDS A&E Ethnic Category A&E CDS A&E Departure time A&E CDS A&E Department type A&E CDS Table : SUS DQ numerator/ denominator fields Rather than treat these as 40 individual items we have created a single trust level aggregate score for the identified number of data quality errors. Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 5 of

Due to the variation in total number of cases reported between each of the three specialty areas, it was decided that a simple aggregation of total data would not allow for accurate risk rating. The alternate approach used for the purpose of generating a trust level risk rating was as follows: An aggregate risk score generated for each of the three individual areas. Trust level risk rating generated through use of a rules based alerting model utilising the three individual risk scores generated for each area. The different steps of our analysis are described in table. Table : Trust aggregate scoring risk model for SUS data quality measure Step Description: For each specialty area: Process numerator and denominator fields as specified within Table. Z-score analysis conducted on items SUSAPC0, SUSOP0 and SUSA&E0. A risk score rating for each of the specialty areas is assigned using the following criteria: No evidence of risk: Z-score <.0 Risk: Z-score.0-.9 Elevated risk: Z-score >3.0 3 Weighting applied to risk ratings for each of the specialties: No evidence of risk: 0 Risk: Elevated risk: 4 The trust level risk rating is calculated using the following: a) Where data is available for A&E, inpatients and outpatients: Sum (risk ratings generated for all service areas) / (number of services with data available x ) No risk identified: where this result is <0.3 Risk: where this result is 0.3 and <0.5 Elevated risk: where this result is 0.5 b) Where data is available for inpatients and outpatients only: Sum (risk ratings generated for all service areas) / (number of services with data available x ) No risk identified: where this result is <0.5 Risk: where this result is 0.5 and <0.75 Elevated risk: where this result is 0.75 Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 6 of

5. Electronic Staff Records (ESR) data From the data provided a total of 5 individual items relating to staffing were created, each of which was allocated into one of the following six categories: Sickness Registration Turnover Stability Support/Supervision Staff Ratios. A full list of items is shown in table 3. Table 3: Electronic Staff Record item list and parameters Area Sickness Common area parameters Abs Rate Person Dim.User Person Type LIKE 'Employee%' (((Abs Rate Staff Group Dim.Staff Group = 'Medical and Dental') AND (Abs Rate Assignment Dim.Contracted Wte <=.)) OR ((Abs Rate Staff Group Dim.Staff Group <> 'Medical and Dental') AND (Abs Rate Assignment Dim.Contracted Wte BETWEEN 0.05 and ))) Item Code Item Name Parameters ESRSICK0 ESRSICK0 ESRSICK03 ESRSICK04 Proportion of days sick due to back problems in the last months Proportion of days sick due to stress in the last months Proportion of days sick in the last months for Medical and Dental staff Proportion of days sick in the last months for Nursing and Midwifery staff Abs Rate Abs Dim.Attendance Reason LIKE 'S Back Problems' Abs Rate Abs Dim.Attendance Reason LIKE 'S0 Anxiety/stress/depression/other psychiatric illnesses' Abs Rate Staff Group Dim.Staff Group LIKE 'Medical and Dental' Abs Rate Staff Group Dim.Staff Group LIKE 'Nursing and Midwifery Registered' NVL(Asg Organisation Type,'X') <> 'Hospice' ESRSICK05 Proportion of days sick in the last months for other clinical staff Abs Rate Staff Group Dim.Staff Group IN ('Add Prof Scientific and Technic', 'Additional Clinical Services', Allied Health Professionals', 'Healthcare Scientists') NVL(Type Of Contract,'X') NOT IN 'Bank', 'Locum', 'Honorary' ESRSICK06 Proportion of days sick in the last months for nonclinical staff Abs Rate Staff Group Dim.Staff Group IN ('Estates and Ancilliary', 'Administrative and Clerical', 'Students') Registration Wfc Assignment Dim.Primary Flag = 'Y' Wfc Assignment Dim.Status LIKE 'Active Assignment' Wfc Assignment Dim.Asg Type Of Contract IN ('Permanent', 'Fixed Term Temp') ESRREG0 Proportion of Medical and Dental staff that hold an active professional registration Wfc Staff Group Dim.Staff Group LIKE 'Medical and Dental' Wfc Profession Registration.Registration Body IN ('General Dental Council', 'General Medical Council' ) Wfc Profession Registration.Registration Number IS NOT NULL Abs Rate Person Dim.User Person Type LIKE 'Employee%' Wfc Profession Registration.Expiry Date + 56 >= :Effective Date Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 7 of

Area Common area parameters Wfc Fact.Contracted WTE for Assignment > 0 Item Code Item Name Parameters ESRREG0 Proportion of Nursing and Midwifery staff that hold an active professional registration Wfc Staff Group Dim.Staff Group LIKE 'Nursing and Midwifery Registered' Wfc NMC Registration.Registration Body LIKE 'Nursing and Midwifery Council' Turnover (((Turnover Staff Group Dim.Staff Group = 'Medical and Dental') AND (Turnover Fact.Asg Contracted Wte <=.)) OR ((Turnover Staff Group Dim.Staff Group <> 'Medical and Dental') AND (Turnover Fact.Asg Contracted Wte BETWEEN 0.05 and ))) ESRTURN0 ESRTURN0 Turnover rate (leavers) for Medical and Dental staff Turnover rate (leavers) for Nursing and Midwifery staff Wfc NMC Registration.Registration Number IS NOT NULL Wfc NMC Registration.Expiry Date >= :Effective Date Turnover Staff Group Dim.Staff Group = 'Medical and Dental' Turnover Assignment Dim.Asg Type Of Contract = 'Permanent' Turnover Assignment Dim.Staff Group LIKE 'Nursing and Midwifery Registered' Turnover Assignment Dim.Asg Type Of Contract IN ('Fixed Term Temp', 'Permanent') ESRTURN0 3 Turnover rate (leavers) for other clinical staff Turnover Assignment Dim.Staff Group IN ('Add Prof Scientific and Technic', 'Additional Clinical Services', Allied Health Professionals', 'Healthcare Scientists') ESRTURN0 4 Turnover rate (leavers) for all other staff Turnover Assignment Dim.Staff Group IN ('Estates and Ancilliary', 'Administrative and Clerical', 'Students') Stability Turnover Assignment Dim.Asg Type Of Contract IN ('Fixed Term Temp', 'Permanent') ESRSTAB0 Stability Index for Medical and Dental staff Turnover Staff Group Dim.Staff Group = 'Medical and Dental' Turnover Assignment Dim.Asg Type Of Contract = 'Permanent' Turnover Workforce Movement Dim.Date Of Joining Org <= First day in period ESRSTAB0 ESRSTAB0 3 Stability Index for Nursing and Midwifery staff Stability Index for other clinical staff Turnover Assignment Dim.Staff Group LIKE 'Nursing and Midwifery Registered' Turnover Assignment Dim.Staff Group IN ('Add Prof Scientific and Technic', 'Additional Clinical Services', Allied Health Professionals', 'Healthcare Scientists') ESRSTAB0 4 Stability Index for all other staff Turnover Assignment Dim.Staff Group IN ('Estates and Ancilliary', 'Administrative and Clerical', 'Students') Support/ Supervision Abs Rate Person Dim.User Person Type LIKE 'Employee%' ESRSUP0 Ratio of Band 6 Nurses to Band 5 Nurses Wfc Staff Group Dim.Staff Group = 'Nursing and Midwifery Registered' AND Wfc Staff Group Dim.Job Role NOT IN ('Midwife', 'Midwife - Specialist Practitioner', 'Midwife - Manager', 'Midwife - Consultant') Wfc Grade Dim.AfC Band = 'Band 5' NVL(Asg Organisation Type,'X') <> 'Hospice' NVL(Type Of Contract,'X') NOT IN 'Bank', 'Locum', 'Honorary' ESRSUP0 Ratio of Charge Nurse/ Ward Sister (Band 7) to Band 5/6 Nurses Wfc Grade Dim.AfC Band = 'Band 6' Wfc Staff Group Dim.Staff Group = 'Nursing and Midwifery Registered' AND Wfc Staff Group Dim.Job Role NOT IN ('Midwife', 'Midwife - Specialist Practitioner', 'Midwife - Manager', 'Midwife - Consultant') Wfc Grade Dim.AfC Band IN ('Band 5', 'Band 6') Wfc Grade Dim.AfC Band = 'Band 7' (((Wfc Staff Group Dim.Staff Group = 'Medical and Dental') AND ESRSUP03 Proportion of all ward staff who are registered nurses Wfc Staff Group Dim.Staff Group = 'Nursing and Midwifery Registered' (Numerator) Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 8 of

Area Common area parameters (Wfc Fact.Contracted WTE for Assignment <=.)) OR ((Wfc Staff Group Dim.Staff Group <> 'Medical and Dental') AND (Wfc Fact.Contracted WTE for Assignment BETWEEN 0.05 and ))) Item Code Item Name Parameters ESRSUP04 Ratio of Consultants to Junior Doctors (Wfc Staff Group Dim.Staff Group = 'Additional Clinical Services' AND Wfc Staff Group Dim.Job Role IN ('Healthcare Assistant', 'Health Care Support Worker', 'Helper/Assistant')) OR Wfc Staff Group Dim.Staff Group = 'Nursing and Midwifery Registered' (Denominator) Wfc Staff Group Dim.Staff Group = 'Medical and Dental' AND Wfc Grade Dim.Grade Code IN ('KC', 'KP0', 'KP0', 'KP03', 'KP04', 'KP05', 'KP06', 'KP07', 'LC0', 'LD', 'MC0', 'MC', 'MC', 'YC5', 'YC5', 'YC53', 'YC54', 'YC55', 'YC56', 'YC57', 'YC58', 'YC59', 'YC60', 'YC6', 'YC6', 'YC63', 'YC64', 'YC65', 'YC66', 'YC67', 'YC68', 'YC69', 'YC70', 'YC7', 'YC7', 'YC73', 'YK5', 'YK5', 'YK53', 'YK54', 'YK55', 'YK56', 'YK57', 'YK58', 'YK59', 'YK60', 'YK6', 'YK6', 'YK63', 'YK64', 'YK65', 'YK67', 'YK69', 'YK70', 'YK7', 'YK73', 'YL5', 'YL5', 'YL53', 'YL55', 'YL56', 'YL57', 'YL6', 'YL63', 'YL69', 'YL70', 'YL7', 'YL73', 'YM5', 'YM5', 'YM53', 'YM54', 'YM55', 'YM56', 'YM57', 'YM58', 'YM59', 'YM60', 'YM6', 'YM6', 'YM63', 'YM64', 'YM65', 'YM66', 'YM67', 'YM68', 'YM69', 'YM70', 'YM7', 'YM7', 'YM73', 'ZL8') OR Wfc Staff Group Dim.Job Role = Consultant (Consultants) Wfc Staff Group Dim.Staff Group = 'Medical and Dental' AND Wfc Grade Dim.Grade Code NOT IN ('KC', 'KP0', 'KP0', 'KP03', 'KP04', 'KP05', 'KP06', 'KP07', 'LC0', 'LD', 'MC0', 'MC', 'MC', 'YC5', 'YC5', 'YC53', 'YC54', 'YC55', 'YC56', 'YC57', 'YC58', 'YC59', 'YC60', 'YC6', 'YC6', 'YC63', 'YC64', 'YC65', 'YC66', 'YC67', 'YC68', 'YC69', 'YC70', 'YC7', 'YC7', 'YC73', 'YK5', 'YK5', 'YK53', 'YK54', 'YK55', 'YK56', 'YK57', 'YK58', 'YK59', 'YK60', 'YK6', 'YK6', 'YK63', 'YK64', 'YK65', 'YK67', 'YK69', 'YK70', 'YK7', 'YK73', 'YL5', 'YL5', 'YL53', 'YL55', 'YL56', 'YL57', 'YL6', 'YL63', 'YL69', 'YL70', 'YL7', 'YL73', 'YM5', 'YM5', 'YM53', 'YM54', 'YM55', 'YM56', 'YM57', 'YM58', 'YM59', 'YM60', 'YM6', 'YM6', 'YM63', 'YM64', 'YM65', 'YM66', 'YM67', 'YM68', 'YM69', 'YM70', 'YM7', 'YM7', 'YM73', 'ZL8') AND Wfc Staff Group Dim.Job Role <> Consultant (Junior Doctors) ESRSUP05 Ratio of band 7 Midwives to band 5/6 Midwives Wfc Grade Dim.AfC Band IN ('Band 5', 'Band 6') AND Wfc Staff Group Dim.Job Role IN('Midwife', 'Midwife - Specialist Practitioner', 'Midwife - Manager', 'Midwife - Consultant') Wfc Grade Dim.AfC Band = 'Band 7' AND Wfc Staff Group Dim.Job Role IN ('Midwife', 'Midwife - Specialist Practitioner', 'Midwife - Manager', 'Midwife - Consultant') Staff Ratios Abs Rate Person Dim.User Person Type LIKE 'Employee%' ESRRAT0 Ratio of all medical and dental staff to occupied beds Wfc Staff Group Dim.Staff Group LIKE 'Medical and Dental' Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 9 of

Area Common area parameters (((Wfc Staff Group Dim.Staff Group = 'Medical and Dental') AND (Wfc Fact.Contracted WTE for Assignment <=.)) OR ((Wfc Staff Group Dim.Staff Group <> 'Medical and Dental') AND (Wfc Fact.Contracted WTE for Assignment BETWEEN 0.05 and ))) Item Code Item Name Parameters ESRRAT0 ESRRAT03 ESRRAT04 Ratio of all nursing staff to occupied beds Ratio of all other clinical staff to occupied beds Ratio of all midwifery staff to births Wfc Staff Group Dim.Staff Group = 'Nursing and Midwifery Registered' Wfc Staff Group Dim.Staff Group IN ('Add Prof Scientific and Technic', 'Additional Clinical Services', Allied Health Professionals', 'Healthcare Scientists') Wfc Staff Group Dim.Job Role IN ('Midwife', 'Midwife - Specialist Practitioner', 'Midwife - Manager', 'Midwife - Consultant') NVL(Asg Organisation Type,'X') <> 'Hospice' NVL(Type Of Contract,'X') NOT IN 'Bank', 'Locum', 'Honorary' Each of these 7 items was processed individually as proportional data with a risk rating based on z-score analysis (see section..). From these we used a rulebased approach to create a single, aggregate score for each of the six areas. The stages of this approach are described in table 4. Table 4: Trust aggregate scoring risk model for Electronic Staff Record indicators Step Description: Process each individual item (specified in Appendix B) and z-score accordingly. A risk score rating for each of the items is assigned using the following criteria: No evidence of risk: Z-score <.0 Risk: Z-score.0-.9 Elevated risk: Z-score >3.0 3 Weighting applied to risk ratings for each of the items: No evidence of risk: 0 Risk: Elevated risk: 4 (for ESRREG) 4 (for ESRSIC ESRSTAB ESRSTAFF ESRSUP The aggregate risk rating is calculated for each of the six areas identified using the following: Risk: Sum (risk ratings for each of the items within the areas) greater than. Elevated risk: Sum (risk ratings for each of the items within the areas)/ (risk ratings generated for all service areas)/ (number of services with data available * ) >0.5 The aggregate risk rating is calculated for each of the six areas identified using the following: Risk: Sum (risk ratings for each of the items within the areas) greater than. Elevated risk: Sum (risk ratings for each of the items within the Intelligent Monitoring: NHS acute hospitals - statistical methodology Page 0 of

ESRTO) areas)/ (risk ratings generated for all service areas)/ (number of services with data available * ) = >0.5 6. Analyses carried out by external organisations The previous sections describe the analysis that has been carried out by CQC where appropriate. Several indicators, such as the HSMR or SHMI, are already analysed by external organisations, and in such cases we report the results of the external analysis. Indicators that are analysed externally are shown in table 5 below. Table 5: Indicators Imperial College mortality outliers: CCS diagnosis groups and procedures Hospital Standardised Mortality Ratio (HSMR) Deaths in low-risk diagnosis groups Summary Hospital Mortality Indicator (SHMI) Adult cardiac surgery audit Source of information Imperial College Dr Foster Unit Dr Foster Intelligence Dr Foster Intelligence Health and Social Care Information Centre Society for Cardiothoracic Surgery Summary of analysis CUSUM analysis using case-mix adjustments similar to the Dr Foster HSMR. Each patient outcome is a Bernoulli random variable with a risk of death determined by factors present on admission, thus enabling the CUSUM to change with each individual patient. Limits are set so that the probability of a false alarm over a year is of the order of 0.%. Cross-sectional analysis of annual data with a 99.8% control limit to determine outliers. Case-mix adjustments. Cross-sectional analysis of annual data with a 99.8% control limit to determine outliers. Crude rates. Cross-sectional analysis of annual data with a 95% control limit to determine outliers, after adjusting for overdispersion. Case-mix adjustments. Cross-sectional analysis of outcomes over a three-year period. A 99.9% control limit is used to determine outliers Intelligent Monitoring: NHS acute hospitals - statistical methodology Page of

Inpatient survey CQC/Picker Institute Europe The Picker Institute Europe run this on behalf of CQC, and calculate modified z- scores for trusts which take into account not just the distribution of trust-level results but also the sample size within the trust. We have used these modified z-scores to determine Risk and Elevated Risk flags for these indicators. 7. Further reading CQC z-scoring and CUSUM methodology Spiegelhalter D J, Sherlaw-Johnson C, Bardsley M, Blunt I, Wood C, Grigg O. Statistical methods for healthcare regulation: rating, screening and surveillance. J Roy Statist Soc A 0; 75: -47. Cross-sectional analyses using z-scores and funnel plots Spiegelhalter D J. Funnel plots for comparing institutional performance. Stat Med 005;4:85-0. CUSUM: steady state values and control limits Grigg O A, Spiegelhalter D J. The null steady-state distribution of the CUSUM statistic. Technical report. 007.Medical Research Council Biostatistics Unit, Cambridge. http://www.mrcbsu.cam.ac.uk/bsusite/publications/preprints/cuspval_ogds.pdf. Jones H E, Ohlssen D I, Spiegelhalter D J. Use of the false discovery rate when comparing multiple healthcare providers. J Clin Epidemiol 008;6:3-40. Intelligent Monitoring: NHS acute hospitals - statistical methodology Page of