Understanding Standardised Mortality

class: title-slide

## Understanding Standardised Mortality Ratios (SMRs)

.pull-left[

### SHMI and HSMR

### <img src="./assets/img/crop.png" alt="Mainard icon" height="40px" /> Dr Chris Mainey

.pull-right[

<img src="understanding_SMRs_files/figure-html/funnel-1.png" width="100%" />
]

.footnote[Presentation and code available: **https://github.com/chrismainey/understanding_standardised_mortality**]

???

Say hi - introduce myself

I'm Patient Safety Lead - System Analysis and Delivery with NHS England and Improvement.

I'm analyst by background, specialised in statistical modelling and and machine learning methods.  My previous role was with UHB - HED, providing HES-based analytical tools to ~50 NHS trust and other organisations. Specialised in statistical productions, mainly risk adjustment and prediction focused.

Today we are going to talk about monitoring mortality, the construction of standardised ratios, the common SMRs in use SHMI and HSMR, cross-sectional and longitudinal monitoring, and issues.

A bit of theory, a bit of anecdote.  Some R scripts available to show some of the functions.

---

# Measuring death

### Why do we do it?

+ 'Smoke alarm for the quality of care' - <a id='cite-keoghKeoghReviewHospital2013'></a>(<a href='#bib-keoghKeoghReviewHospital2013'>Keogh, 2013</a>)
+ Tacit assumption that high mortality is bad

### Does it work?

+ __Yes:__ 
 + Increases power of comparison by reducing confounding.
 + Monitoring them has effects on hospital's vigilance and culture <a id='cite-jarmanMonitoringChangesHospital2005'></a><a id='cite-wrightLearningDeathHospital2006'></a>(<a href='https://doi.org/10.1136/bmj.330.7487.329'>Jarman, Bottle, Aylin, et al., 2005</a>; <a href='https://doi.org/10.1258/jrsm.99.6.303'>Wright, Dugdale, Hammond, et al., 2006</a>)

+ __No:__ 
 + Poor proxy of avoidable death: <a id='cite-girlingCasemixAdjustedHospital2012'></a>(<a href='https://doi.org/10/f4fr3b'>Girling, Hofer, Wu, et al., 2012</a>)
 + Case-mix adjustment can exaggerate biases it tries to address: <a id='cite-deeksEvaluatingNonrandomisedIntervention2003'></a>(<a href='#bib-deeksEvaluatingNonrandomisedIntervention2003'>Deeks, Dinnes, D'Amico, et al., 2003</a>)

---
class: middle

### Desceptive argument:
___"...of course we should be monitoring deaths"___:  - but is this too simplistic a view?  Is it a measure of 'quality?'

---
class: inverse middle

# Crude Mortality

---
## Crude mortality Rate

.pull-left[

+ Count the numbers of deaths?   - Not a rate
{{content}}
]

.pull-right[
<img src="understanding_SMRs_files/figure-html/crudeplot-1.png" width="100%" />

<img src="understanding_SMRs_files/figure-html/crudetime-1.png" width="100%" />
]

+ Give it some sort of scale:  proportion
  + In hospital, patients, discharges, bed-days etc.
  + Often adjusted to a larger standard, e.g. per thousand

`$${Crude\ Rate(p)} = \frac{\Sigma Deaths}{n}$$`

{{content}}
--

+ __Strengths:__
 + Easy to calculate
 + Directly linked to real deaths
 
{{content}}
--

+ __Weaknesses:__
 + Not really comparable across organisations
 + Case-mix confounds rate

---
class: inverse middle

# Standardising mortality
### Aim: reduce confounding and increase power of comparison

---
## Direct standardisation

+ Take our data and map them to a common population/structure.

+ Example:  Age-standardisation:
  + Calculate age-specific rates in groups (e.g. 10-year bands)
  + Identify a relevant standard population in corresponding groups. E.g. [European Standard Population]( https://www.opendata.nhs.scot/dataset/standard-populations)
  + Multiply age-specific rates by standard population bins
  + Sum and adjust to desired multiplier (e.g. per 100, per 100,000 etc.)

+ Commonly used in public health cases, such as cancer incidence and mortality rates.

--
+ __Strengths:__
 + Directly comparable between units/group/sites/countries
 + Does not require statistical model

+ __Weaknesses:__
 + Harder to relate to local/observed numbers
 + Challenging to do for anything more than age and sex

---
## Indirect standardisation

+ Compare our data to expected averages

+ E.g Calculate average rate, per-patient, across the dataset
 + Per trust, calculate 'expected rate': `$average\ rate * n$`
 + Present as grouped ratios of Observed / Expected

+ Commonly uses a regression model
  + Predict risk of event, based on case-mix factors (predictors)

+ __Strengths:__
 + Directly comparable between units/group/sites/countries
 + Usually require statistical model, e.g. regression
 
{{content}}
--

+ __Weaknesses:__
 + Usually requires a statistical model, e.g. regression to calculate 'expected rate'
 + Susceptible to more forms of bias <a id='cite-iezzoniRisksRiskAdjustment1997'></a>(<a href='https://doi.org/10/c2trv4'>Iezzoni, 1997</a>; <a href='#bib-deeksEvaluatingNonrandomisedIntervention2003'>Deeks, Dinnes, D'Amico, et al., 2003</a>)
 + Can be challenging to understand what changes in rates mean

---
## Example SMR:
.pull-left[

```r
# Load 'medpar' dataset from COUNT package.
data("medpar")

# build logistic regression for risk of death
mod1 <- glm(died ~ los + factor(type) + age80
 , data=medpar
 , family = "binomial")

# Predict risk of death back into data frame.
medpar$pred <- predict(mod1, type="response")

# SMRs
medpar %>% 
  group_by(provnum) %>% 
  summarise(Observered = sum(died),
            Expected = sum(pred),
            SMR = sum(died) / sum(pred))
```
]

.pull-right[

```
## # A tibble: 54 x 4
## provnum Observered Expected SMR
## <labelled> <int> <dbl> <dbl>
## 1 030001 16 19.2 0.832
## 2 030002 19 20.7 0.916
## 3 030003 2 1.77 1.13 
## 4 030006 23 25.8 0.893
## 5 030007 2 4.44 0.451
## 6 030008 5 9.24 0.541
## 7 030009 4 5.34 0.749
## 8 030010 22 17.9 1.23 
## 9 030011 12 12.6 0.955
## 10 030012 12 7.48 1.60 
## # ... with 44 more rows
```
]

__SMR = 1: observed=predicted, >1: observed>predicted, <1: observed <predicted__
---
class: inverse middle
# Common (indirectly standardised) SMRs:

### Summary Hospital-level Mortality Indicator (SHMI)

### Dr Foster Hospital Standardised Mortality Ratio (HSMR)

---
## Hospital Standardised Mortality Ratio (HSMR)

+ Work in USA as early as 1970s demonstrated ability to calculate theses metrics.

+ Prof. Sir Brian Jarman and others adapted these methods to English health care system, data coding standards and structures.

+ Methods were heavily impacted and applied in aftermath of Bristol and Mid-Staffs enquiry.

+ Controversy on some issues:
 + Commercial exploitation of method by Dr Foster Intelligence - accused of 'black box methods'
 + University of Birmingham and others published criticism
 + Imperial rebutted criticism, and won NIHR funding to build national monitoring system
 + Until recently, sent alerts to CQC and trusts for high mortality.
 
__Key References:__
+ Original paper: <a id='cite-jarmanExplainingDifferencesEnglish1999'></a>(<a href='https://doi.org/10/fkkfm9'>Jarman, Gault, Alves, et al., 1999</a>)
+ Birmingham's criticism: <a id='cite-mohammedEvidenceMethodologicalBias2009'></a>(<a href='https://doi.org/10/bv4sh8'>Mohammed, Deeks, Girling, et al., 2009</a>) 
+ Paul Taylor's long-form article on history and controversy: <a id='cite-taylorStandardizedMortalityRatios2014'></a>(<a href='https://doi.org/10/f5pxfw'>Taylor, 2014</a>)

---
## Summary Hospital-level Mortality Indicator (SHMI)

With growing controversy, then NHS Medical Director, Prof. Sir Bruce Keogh, commissioned a review.

Recommended creating a new, NHS owned and transparently published indicator, however:
+ Changed the remit to in-hospital or within 30-days of discharge
+ Applied to all acute activity (except still birth)
+ Cruder case-mix model deliberately to avoid controversial measure such as:
  + Palliative care coding
  + No adjustment for deprivation - political context
  + Co-morbidity score 'binned' rather than continuous to reduce change of gaming
  + Fewer, larger diagnosis groups

__Key References:__
+ Review: [National Quality Board (in national web archives)](https://webarchive.nationalarchives.gov.uk/ukgwa/20120503091458/http://www.dh.gov.uk/en/Healthcare/Qualityandproductivity/Makingqualityhappen/NationalQualityBoard/DH_102954)
+ Sheffield paper: <a id='cite-campbellDevelopingSummaryHospital2012'></a>(<a href='https://doi.org/10/gb3r9t'>Campbell, Jacques, Fotheringham, et al., 2012</a>)
+ NHSD SHMI: <a href="www.digital.nhs.uk/SHMI">www.digital.nhs.uk/SHMI</a>

---
class: center
## Case-mix factors:

---

# Criticisms (HSMR and/or SHMI)

+ Link to quality of hospitals unclear, 
 + For: <a id='cite-cecilWhatRelationshipMortality2020'></a><a id='cite-cecilInvestigatingAssociationAlerts2018'></a>(<a href='https://doi.org/10.1177/1355819619847689'>Cecil, Bottle, Esmail, et al., 2020</a>; <a href='https://doi.org/10.1136/bmjqs-2017-007495'>Cecil, Bottle, Esmail, et al., 2018</a>)
 + Against: <a id='cite-lilfordUsingHospitalMortality2010'></a><a id='cite-blackAssessingQualityHospitals2010a'></a>(<a href='https://doi.org/10/frq62g'>Lilford and Pronovost, 2010</a>; <a href='https://doi.org/10.1136/bmj.c2066'>Black, 2010</a>)

+ They do not directly relate to avoidable death <a id='cite-hoganAvoidabilityHospitalDeaths2015'></a>(<a href='https://doi.org/10/f4fr3b'>Girling, Hofer, Wu, et al., 2012</a>; <a href='https://doi.org/10/gb3swm'>Hogan, Zipfel, Neuburger, et al., 2015</a>)

+ Single number does not convey nuance

+ Insensitive to who patients who survived

+ Susceptible to 'gaming' - <a id='cite-hawkesHowMessageMortality2013'></a>(<a href='https://doi.org/10.1136/bmj.f562'>Hawkes, 2013</a>)

+ Covid-19 pandemic, these models assume stability

+ Case-mix adjustment fallacy (<a href='https://doi.org/10/bv4sh8'>Mohammed, Deeks, Girling, et al., 2009</a>)

+ Constant risk fallacy <a id='cite-nichollCasemixAdjustmentNonrandomised2007'></a>(<a href='https://doi.org/10/bv4sh8'>Mohammed, Deeks, Girling, et al., 2009</a>; <a href='https://doi.org/10/d7f9jh'>Nicholl, 2007</a>)

+ Potential for Simpsons paradox <a id='cite-marang-vandemheenSimpsonParadoxHow2014'></a>(<a href='https://doi.org/10.1136/bmjqs-2014-003358'>Marang-van
de
Mheen and Shojania, 2014</a>)

---
class: inverse middle

# Cross-sectional

---

# Comparison at single point in time

.pull-left[
+ Both HSMR and SHMI report on the final year of their modelling period.

+ Snapshot of performance against expected

+ League-tables are bad, as measure is relative <a id='cite-goldsteinLeagueTablesTheir1996'></a><a id='cite-lilfordUseMisuseProcess2004'></a>(<a href='https://doi.org/10/chf9kj'>Goldstein and Spiegelhalter, 1996</a>; <a href='https://doi.org/10/c97xd2'>Lilford, Mohammed, Spiegelhalter, et al., 2004</a>)

+ SPC principles applied in using funnel plot <a id='cite-spiegelhalterFunnelPlotsComparing2005'></a>(<a href='https://doi.org/10/fq7z8t'>Spiegelhalter, 2005a</a>)

+ Overdispersion
]

.pull-right[
![](./understanding_SMRs_files/figure-html/funnel-1.png)
]
---

# Overdispersion

.pull-left[

___Overdispersion___, where conditional variance is greater than conditional mean, occurs when:

1. Aggregation / Discretization
1. Mis-specified predictors/model
1. Presence of outliers
1. Variation between response probabilities (heterogeneity)

]

__Repeated measures (correlation) __

+ Regression assumes all points independent
+ Sampling from same organisations repeatedly
+ Clustered: - local means

.pull-right[
![](./assets/img/ints.gif)
]

---

# Dealing with overdisperion

+ Ignore it: use-case dependent.  False alarm rate too high here, as error is underestimated

+ Improve the model with more information: Some room for this.

+ Build a model with clustered assumption <a id='cite-maineyStatisticalMethodsNHS2020'></a>(<a href='https://discovery.ucl.ac.uk/id/eprint/10094736/'>Mainey, 2020</a>):
 + Quasi-likelihood methods (with multiplicative scale factor) <a id='cite-wedderburnQuasiLikelihoodFunctionsGeneralized1974'></a>(<a href='#bib-wedderburnQuasiLikelihoodFunctionsGeneralized1974'>Wedderburn, 1974</a>)
 + Compound distribution model: beta binomial <a id='cite-skellamProbabilityDistributionDerived1948'></a>(<a href='#bib-skellamProbabilityDistributionDerived1948'>Skellam, 1948</a>)
 + Random-intercept model: model 'within' and 'between' variance

+ Apply tools based on meta-analysis methods: <a id='cite-spiegelhalterStatisticalMethodsHealthcare2012'></a>(<a href='https://doi.org/10/dqpqpk'>Spiegelhalter, Sherlaw-Johnson, Bardsley, et al., 2012</a>)
 + Designed to summarise studies fo different size
 + Akin to hospitals of different sizes
 + Additivity assumption - more like random-intercept than scale factor

---
class: inverse middle

# Longitudinal

---

# How?

.pull-left[
+ Can't simply plot in XmR chart, as risk-adjustment forms denominator (and overdispersion)

]

.pull-right[

]

+ Can use the observed and predicted in risk-adjusted control chart

{{content}}
--

+ Common is 'risk-adjusted CUSUM'
  + Continuous log-likelihood ratio test

{{content}}
--

`$$C_t = max(C_{t-1} + w_t, 0)$$`

+ `$C$` CUSUM value at time-point `$t$` (e.g. a monthly at a trust)
+ `$w$` is a weighting, in this case the log-likelihood ratio (observation v.s. England) to calculate the CUSUM weight/value ( `$C$` ) at time point ( `$t$` )

---
# Differences in CUSUM methods

Until recently, mortality outlier programme and Imperial college sent monthly alerts to Trusts.

.pull-left[
### CQC - (aggregated)
+ Data are transformed to z-scores
+ Overdispersion adjustment based on additive model (<a href='https://doi.org/10/dqpqpk'>Spiegelhalter, Sherlaw-Johnson, Bardsley, et al., 2012</a>)
+ Can convert average run-length to FDR, and set threshold <a id='cite-griggNullSteadystateDistribution2008'></a><a id='cite-carequalitycommissioncqcNHSAcuteHospitals2014'></a>(<a href='https://doi.org/10/bgvkdx'>Grigg and Spiegelhalter, 2008</a>; <a href='#bib-carequalitycommissioncqcNHSAcuteHospitals2014'>Care Quality Commission, 2014</a>)
+ Global trigger (5.48) - set to marginal 0.01% FDR.
+ Applicable to other indicators and groupings, subject to same transformation
]

.pull-right[
### DFI/Imperial - (person-level)
+ Binomial assumption and threshold set through simulation of average run-length to false positives. <a id='cite-bottlePredictingFalseAlarm2011'></a>(<a href='https://doi.org/10/cbr4rq'>Bottle and Aylin, 2011</a>)
+ Unique to each trust / group / reporting period.
+ Intractable to calculate each month, and authors fitted a set of descriptive equations give a decent approximation for conditions where mortality rate 30% or lower.
+ Formula can then be solved through optimisation methods and give threshold value for each group.

]

---
# Problem with CUSUM charts

+ A common criticism of CUSUMs is that the are opaque and hard to interpret
+ What does the CUSUM value mean?

.pull-left[
### Variable Life-Adjusted Display (VLAD)

+ Originally used to visualise surgical outcome more intuitively <a id='cite-lovegroveMonitoringPerformanceCardiac1999'></a>(<a href='https://doi.org/10/dtsvbg'>Lovegrove, Sherlaw-Johnson, Valencia, et al., 1999</a>)
+ Can actually add limits to plot using cusums <a id='cite-sherlaw-johnsonMethodDetectingRuns2005'></a>(<a href='https://doi.org/10/dpvrft'>Sherlaw-Johnson, 2005</a>)

`$$Vn=\sum\limits_{i=1}^ny_n -  \sum\limits_{i=1}^n X_n$$`
]

.pull-right[

.footnote[Chart sourced from https://www.ouh.nhs.uk/about/trust-board/2018/january/documents/MRG2017.149a-shmi-update.pdf]

]

---
class:  inverse
## Summary

+ Mortality monitoring is common, but it's use as a global measure (rather than specific conditions) is unclear.

+ Not directly linked to avoidable deaths

+ Crude mortality can be sensibly used in some cases, but confounded by case-mix

+ Case-mix adjusted mortality is usually done by indirect standardisation, with regression model

+ SMRs as grouped sums of observed / 'expected' (or 'predicted')

+ HSMR was first national measure in UK - narrow scope and extensive case-mix adjustment

+ SHMI is NHS-owned indicator - wider scope and less case-mix adjustment

+ Criticisms remain of both -  and SMRs broadly

+ Cross-sectional comparisons usually by funnel plot, longitudinal with cusums and vlad.

__Worth knowing history and limitations of indicators before using them__

---
class: references

### References (1)

<cite><a id='bib-blackAssessingQualityHospitals2010a'></a><a href="#cite-blackAssessingQualityHospitals2010a">Black, N.</a>
(2010).
&ldquo;Assessing the Quality of Hospitals&rdquo;.
In: BMJ 340, p. c2066.
DOI: <a href="https://doi.org/10.1136/bmj.c2066">10.1136/bmj.c2066</a>.</cite>

<cite><a id='bib-bottlePredictingFalseAlarm2011'></a><a href="#cite-bottlePredictingFalseAlarm2011">Bottle, A. and P. Aylin</a>
(2011).
&ldquo;Predicting the false alarm rate in multi-institution mortality monitoring&rdquo;.
In: The Journal of the Operational Research Society 62.9, pp. 1711&ndash;1718.
ISSN: 01605682, 14769360.
DOI: <a href="https://doi.org/10/cbr4rq">10/cbr4rq</a>.</cite>

<cite><a id='bib-campbellDevelopingSummaryHospital2012'></a><a href="#cite-campbellDevelopingSummaryHospital2012">Campbell, M. J., R. M. Jacques, J. Fotheringham, et al.</a>
(2012).
&ldquo;Developing a summary hospital mortality index: retrospective analysis in English hospitals over five years&rdquo;.
In: BMJ 344, p. e1001.
DOI: <a href="https://doi.org/10/gb3r9t">10/gb3r9t</a>.</cite>

<cite><a id='bib-carequalitycommissioncqcNHSAcuteHospitals2014'></a><a href="#cite-carequalitycommissioncqcNHSAcuteHospitals2014">Care Quality Commission</a>
(2014).
NHS acute hospitals: Statistical Methodology.
Care Quality Commission (CQC).</cite>

<cite><a id='bib-cecilWhatRelationshipMortality2020'></a><a href="#cite-cecilWhatRelationshipMortality2020">Cecil, E., A. Bottle, A. Esmail, et al.</a>
(2020).
&ldquo;What Is the Relationship between Mortality Alerts and Other Indicators of Quality of Care? A National Cross-Sectional Study&rdquo;.
In: Journal of Health Services Research &amp; Policy 25.1, pp. 13&ndash;21.
ISSN: 1355-8196.
DOI: <a href="https://doi.org/10.1177/1355819619847689">10.1177/1355819619847689</a>.</cite>

<cite><a id='bib-cecilInvestigatingAssociationAlerts2018'></a><a href="#cite-cecilInvestigatingAssociationAlerts2018">Cecil, E., A. Bottle, A. Esmail, et al.</a>
(2018).
&ldquo;Investigating the Association of Alerts from a National Mortality Surveillance System with Subsequent Hospital Mortality in England: An Interrupted Time Series Analysis&rdquo;.
In: BMJ Quality &amp;amp; Safety 27.12, p. 965.
DOI: <a href="https://doi.org/10.1136/bmjqs-2017-007495">10.1136/bmjqs-2017-007495</a>.</cite>

<cite><a id='bib-deeksEvaluatingNonrandomisedIntervention2003'></a><a href="#cite-deeksEvaluatingNonrandomisedIntervention2003">Deeks, J. J., J. Dinnes, R. D'Amico, et al.</a>
(2003).
&ldquo;Evaluating Non-Randomised Intervention Studies.&rdquo;
In: Health technology assessment (Winchester, England) 7.27, pp. iii-173.
ISSN: 1366-5278.</cite>

<cite><a id='bib-girlingCasemixAdjustedHospital2012'></a><a href="#cite-girlingCasemixAdjustedHospital2012">Girling, A. J., T. P. Hofer, J. Wu, et al.</a>
(2012).
&ldquo;Case-Mix Adjusted Hospital Mortality Is a Poor Proxy for Preventable Mortality: A Modelling Study&rdquo;.
In: BMJ Quality &amp; Safety 21.12, pp. 1052&ndash;1056.
DOI: <a href="https://doi.org/10/f4fr3b">10/f4fr3b</a>.</cite>

<cite><a id='bib-goldsteinLeagueTablesTheir1996'></a><a href="#cite-goldsteinLeagueTablesTheir1996">Goldstein, H. and D. J. Spiegelhalter</a>
(1996).
&ldquo;League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance&rdquo;.
In: Journal of the Royal Statistical Society. Series A (Statistics in Society) 159.3, pp. 385&ndash;443.
ISSN: 09641998, 1467985X.
DOI: <a href="https://doi.org/10/chf9kj">10/chf9kj</a>.</cite>

<cite><a id='bib-griggNullSteadystateDistribution2008'></a><a href="#cite-griggNullSteadystateDistribution2008">Grigg, O. and D. Spiegelhalter</a>
(2008).
&ldquo;The null steady-state distribution of the CUSUM statistic&rdquo;.
In: Technometrics : a journal of statistics for the physical, chemical, and engineering sciences.
DOI: <a href="https://doi.org/10/bgvkdx">10/bgvkdx</a>.</cite>

<cite><a id='bib-hawkesHowMessageMortality2013'></a><a href="#cite-hawkesHowMessageMortality2013">Hawkes, N.</a>
(2013).
&ldquo;How the Message from Mortality Figures Was Missed at Mid Staffs&rdquo;.
In: BMJ : British Medical Journal 346, p. f562.
DOI: <a href="https://doi.org/10.1136/bmj.f562">10.1136/bmj.f562</a>.</cite>

<cite><a id='bib-hoganAvoidabilityHospitalDeaths2015'></a><a href="#cite-hoganAvoidabilityHospitalDeaths2015">Hogan, H., R. Zipfel, J. Neuburger, et al.</a>
(2015).
&ldquo;Avoidability of Hospital Deaths and Association with Hospital-Wide Mortality Ratios: Retrospective Case Record Review and Regression Analysis&rdquo;.
In: BMJ 351.
DOI: <a href="https://doi.org/10/gb3swm">10/gb3swm</a>.</cite>

---
class: references

### References (2)

<cite><a id='bib-iezzoniRisksRiskAdjustment1997'></a><a href="#cite-iezzoniRisksRiskAdjustment1997">Iezzoni, L. I.</a>
(1997).
&ldquo;The Risks of Risk Adjustment&rdquo;.
In: JAMA 278.19, pp. 1600&ndash;7.
ISSN: 0098-7484 (Print) 0098-7484 (Linking).
DOI: <a href="https://doi.org/10/c2trv4">10/c2trv4</a>.</cite>

<cite><a id='bib-jarmanMonitoringChangesHospital2005'></a><a href="#cite-jarmanMonitoringChangesHospital2005">Jarman, B., A. Bottle, P. Aylin, et al.</a>
(2005).
&ldquo;Monitoring Changes in Hospital Standardised Mortality Ratios&rdquo;.
In: BMJ (Clinical research ed.) 330.7487, pp. 329&ndash;329.
ISSN: 1756-1833.
DOI: <a href="https://doi.org/10.1136/bmj.330.7487.329">10.1136/bmj.330.7487.329</a>.</cite>

<cite><a id='bib-jarmanExplainingDifferencesEnglish1999'></a><a href="#cite-jarmanExplainingDifferencesEnglish1999">Jarman, B., S. Gault, B. Alves, et al.</a>
(1999).
&ldquo;Explaining differences in English hospital death rates using routinely collected data&rdquo;.
In: BMJ 318.7197, pp. 1515&ndash;1520.
ISSN: 0959-8138 1468-5833.
DOI: <a href="https://doi.org/10/fkkfm9">10/fkkfm9</a>.</cite>

<cite><a id='bib-keoghKeoghReviewHospital2013'></a><a href="#cite-keoghKeoghReviewHospital2013">Keogh, B.</a>
(2013).
Keogh Review on Hospital Deaths Published - NHSUK.</cite>

<cite><a id='bib-lilfordUseMisuseProcess2004'></a><a href="#cite-lilfordUseMisuseProcess2004">Lilford, R., M. A. Mohammed, D. Spiegelhalter, et al.</a>
(2004).
&ldquo;Use and Misuse of Process and Outcome Data in Managing Performance of Acute Medical Care: Avoiding Institutional Stigma&rdquo;.
In: Lancet 363.9415, pp. 1147&ndash;54.
ISSN: 0140-6736.
DOI: <a href="https://doi.org/10/c97xd2">10/c97xd2</a>.</cite>

<cite><a id='bib-lilfordUsingHospitalMortality2010'></a><a href="#cite-lilfordUsingHospitalMortality2010">Lilford, R. and P. Pronovost</a>
(2010).
&ldquo;Using Hospital Mortality Rates to Judge Hospital Performance: A Bad Idea That Just Won't Go Away&rdquo;.
In: BMJ 340, p. c2016.
ISSN: 1756-1833 (Electronic) 0959-535X (Linking).
DOI: <a href="https://doi.org/10/frq62g">10/frq62g</a>.</cite>

<cite><a id='bib-lovegroveMonitoringPerformanceCardiac1999'></a><a href="#cite-lovegroveMonitoringPerformanceCardiac1999">Lovegrove, J., C. Sherlaw-Johnson, O. Valencia, et al.</a>
(1999).
&ldquo;Monitoring the Performance of Cardiac Surgeons&rdquo;.
In: The Journal of the Operational Research Society 50.7. Publisher: Palgrave Macmillan Journals, pp. 684&ndash;689.
ISSN: 01605682, 14769360.
DOI: <a href="https://doi.org/10/dtsvbg">10/dtsvbg</a>.</cite>

<cite><a id='bib-maineyStatisticalMethodsNHS2020'></a><a href="#cite-maineyStatisticalMethodsNHS2020">Mainey, C.</a>
(2020).
&ldquo;Statistical methods for NHS incident reporting data&rdquo;.
London.
URL: <a href="https://discovery.ucl.ac.uk/id/eprint/10094736/">https://discovery.ucl.ac.uk/id/eprint/10094736/</a>.</cite>

<cite><a id='bib-marang-vandemheenSimpsonParadoxHow2014'></a><a href="#cite-marang-vandemheenSimpsonParadoxHow2014">Marang-van
de
Mheen, P. J. and K. G. Shojania</a>
(2014).
&ldquo;Simpson's Paradox: How Performance Measurement Can Fail Even with Perfect Risk Adjustment&rdquo;.
In: BMJ Quality &amp;amp; Safety 23.9, p. 701.
DOI: <a href="https://doi.org/10.1136/bmjqs-2014-003358">10.1136/bmjqs-2014-003358</a>.</cite>

<cite><a id='bib-mohammedEvidenceMethodologicalBias2009'></a><a href="#cite-mohammedEvidenceMethodologicalBias2009">Mohammed, M. A., J. J. Deeks, A. Girling, et al.</a>
(2009).
&ldquo;Evidence of Methodological Bias in Hospital Standardised Mortality Ratios: Retrospective Database Study of English Hospitals&rdquo;.
In: BMJ 338, p. b780.
ISSN: 1756-1833 (Electronic) 0959-535X (Linking).
DOI: <a href="https://doi.org/10/bv4sh8">10/bv4sh8</a>.</cite>

<cite><a id='bib-nichollCasemixAdjustmentNonrandomised2007'></a><a href="#cite-nichollCasemixAdjustmentNonrandomised2007">Nicholl, J.</a>
(2007).
&ldquo;Case-Mix Adjustment in Non-Randomised Observational Evaluations: The Constant Risk Fallacy&rdquo;.
In: Journal of Epidemiology and Community Health (1979-) 61.11, pp. 1010&ndash;1013.
ISSN: 0143005X, 14702738.
DOI: <a href="https://doi.org/10/d7f9jh">10/d7f9jh</a>.</cite>

<cite><a id='bib-sherlaw-johnsonMethodDetectingRuns2005'></a><a href="#cite-sherlaw-johnsonMethodDetectingRuns2005">Sherlaw-Johnson, C.</a>
(2005).
&ldquo;A Method for Detecting Runs of Good and Bad Clinical Outcomes on Variable Life-Adjusted Display (VLAD) Charts&rdquo;.
In: Health Care Management Science 8.1, pp. 61&ndash;65.
ISSN: 1572-9389.
DOI: <a href="https://doi.org/10/dpvrft">10/dpvrft</a>.</cite>
---
class: references

### References (3)

<cite><a id='bib-skellamProbabilityDistributionDerived1948'></a><a href="#cite-skellamProbabilityDistributionDerived1948">Skellam, J. G.</a>
(1948).
&ldquo;A Probability Distribution Derived from the Binomial Distribution by Regarding the Probability of Success as Variable Between the Sets of Trials&rdquo;.
In: Journal of the Royal Statistical Society. Series B (Methodological) 10.2, pp. 257&ndash;261.
ISSN: 00359246.</cite>

<cite><a id='bib-spiegelhalterFunnelPlotsComparing2005'></a><a href="#cite-spiegelhalterFunnelPlotsComparing2005">Spiegelhalter, D. J.</a>
(2005a).
&ldquo;Funnel plots for comparing institutional performance&rdquo;.
In: Statistics in Medicine 24.8, pp. 1185&ndash;202.
ISSN: 0277-6715 (Print) 0277-6715 (Linking).
DOI: <a href="https://doi.org/10/fq7z8t">10/fq7z8t</a>.</cite>

<cite><a id='bib-spiegelhalterStatisticalMethodsHealthcare2012'></a><a href="#cite-spiegelhalterStatisticalMethodsHealthcare2012">Spiegelhalter, D., C. Sherlaw-Johnson, M. Bardsley, et al.</a>
(2012).
&ldquo;Statistical methods for healthcare regulation: Rating, screening and surveillance&rdquo;.
In: Journal of the Royal Statistical Society: Series A (Statistics in Society) 175.1, pp. 1&ndash;47.
ISSN: 09641998.
DOI: <a href="https://doi.org/10/dqpqpk">10/dqpqpk</a>.</cite>

<cite><a id='bib-taylorStandardizedMortalityRatios2014'></a><a href="#cite-taylorStandardizedMortalityRatios2014">Taylor, P.</a>
(2014).
&ldquo;Standardized Mortality Ratios&rdquo;.
In: International Journal of Epidemiology 42.6, pp. 1882&ndash;1890.
ISSN: 0300-5771.
DOI: <a href="https://doi.org/10/f5pxfw">10/f5pxfw</a>.</cite>

<cite><a id='bib-wedderburnQuasiLikelihoodFunctionsGeneralized1974'></a><a href="#cite-wedderburnQuasiLikelihoodFunctionsGeneralized1974">Wedderburn, R. W. M.</a>
(1974).
&ldquo;Quasi-Likelihood Functions, Generalized Linear Models, and the Gauss-Newton Method&rdquo;.
In: Biometrika 61.3, pp. 439&ndash;447.</cite>

<cite><a id='bib-wrightLearningDeathHospital2006'></a><a href="#cite-wrightLearningDeathHospital2006">Wright, J., B. Dugdale, I. Hammond, et al.</a>
(2006).
&ldquo;Learning from Death: A Hospital Mortality Reduction Programme&rdquo;.
In: Journal of the Royal Society of Medicine 99.6, pp. 303&ndash;308.
ISSN: 0141-0768.
DOI: <a href="https://doi.org/10.1258/jrsm.99.6.303">10.1258/jrsm.99.6.303</a>.</cite>