Chief Analyst Interview - The Health Economics Unit

class: title-slide

# Chief Analyst Interview

### Examples of previous analytical projects

### <img src="./assets/img/crop.png" alt="Mainard icon" height="40px" />  Dr Chris Mainey

.footnote[Presentation and code available: **https://github.com/chrismainey/chief_analyst_presentation**]

.art_cap[R generative art - inspired by Antonio Sánchez Chinchón - @aschinchon]

???

__End 30 seconds__!

Say hi - introduce myself

I'm Patient Safety Lead - System Analysis and Delivery with NHS England and Improvement. I'm an experienced analyst, data scientist and statistician, used to leading NHS data teams.  I'm a senior fellow in the NHS-R community, I contribute to the wider analyst community, and keen to help the profession move forward.

Thanks for the opportunity to and interview today.
As the brief was to present two projects that demonstrate my data science expertise.  I'm committed to using modern techniques, and embracing open source approaches.  In part to demonstrate this, the presentation is written in Rmarkdown, wit visualisation and code on GitHub.

---

# Two examples of my work:

<br><br>
### 1. Developing analytical methods for NHS incident reporting data

<br><br>
### 2. Re-imagining standardised mortality ratio (SMR) surveillance in HED

???

__End: 1:00__

I'm aiming to illustrate to problems that I was tackling, give concise summaries of the approaches, highlighting the methods and tool sets used.

---
class: inverse center middle

# Developing analytical methods for NHS incident reporting data

---
## Analysing Incident Data

+ My PhD Project: <a name=cite-maineyStatisticalMethodsNHS2020></a>([Mainey, 2020](https://discovery.ucl.ac.uk/id/eprint/10094736/)) thesis available [here](https://discovery.ucl.ac.uk/id/eprint/10094736/)

+ Patient Safety movement in England has encourage national incident reporting (NRLS)
 + Routed in visible failures in care such as Mid Staffs and Bristol
 + Harvard Medical Practice study <a name=cite-leapeNatureAdverseEvents1991></a>([Leape Brennan, et al., 1991](https://doi.org/10/fhvzbs)), 'Too Err is Human' <a name=cite-kohnErrHumanBuilding2000></a>([Kohn Corrigan, et al., 2000](#bib-kohnErrHumanBuilding2000)), 'An organisation with a memory' <a name=cite-donaldsonOrganisationMemory2002></a>([Donaldson, 2002](https://doi.org/10/gfz5w4))

???
Patient safety movement is reasonably new, established in the 80s through the like of Harvard medical practice study, and the reading, or clinical audit of patient notes and reflections on harm.  Often 'imported' from high-risk industries, and incident reporting was ported over, but fairly different in healthcare.

+ National Reporting and Learning System (NRLS) is the national repository for incident data

### Commonly, with a national dataset there is a desire to use it for benchmarking

__End 2:30__

+ Desire to compare organisations, and often to 'point the finger', but this is actually at odds with the Patient Safety discipline - about encouraging reporting to learn.

+ Still desire to understand reporting patterns, but clinical review is main way to use signal.

+ Desire to compare behaviour across organisations safety culture
  + Little use in most quantitative measures of incident reporting
  + Change over time?

+ Main signal is free-text:
  + Clinical review
  + NLP / text mining
  
--

### HED wanted an NRLS 'module' for comparing trusts / survellieance

???

### My project was to examine statistical modelling approaches for these data.

---
## Project overview

.left-column[
+ __"Ground-clearing" exercise - Literature review__

+ Strengthened view on free-text signal
  + Data completeness a huge issue
  + Little-to-no useful parameters for modelling
  + No obvious orthogonal outcome
  + Reporting is highly organisation-specific

]

???

__End 3:30__

To quote Andi when he presented on last week's 'huddle', 'it doesn't get more exciting than a systematic literature search,' but this was the place to start.
Deeply challenging review, as it was an ill-defined question: 'how has NRLS been used for analysis?'

+ 'No orthogonal':
   Not immediately clear if high reporting is good or bad.  It could be that high reporting is a function of a mature organisation that recognises incidents and learns form them, or it could mean that lots of incidents occured.
   
+ NLP methods

+ __Only quantitative approach is 'counting'__
  + Created a 'panel' dataset - per organisation, per month
  + Other predictors from HES: APC, OP & AE
  + Multi-level model assumption due to clustering
  + Overdispersion modelled various ways

.right-column[
+ __Output approaches for __
  + How do we use risk-adjusted output to form indicators?
  + Visualisation of organisations
    + Cross-sectional
    + Longitudinal

<br>

]

+ __Natural Language Processing__
  + Examining how to create a useful dataset
  + Visualisation of processes
  + Identify topics in the data
  + Predictive modelling of harm level from text

---
## Analytical methods (1)

### Data preparation
+ `SQL Server Database` - ETL
+ Required manual `SAS` correction for 'delimiting' problems
+ Panel created using a 'calendar table', join all tables and aggregate per month, per trust.

???

__End 5:00__

+ Had to start all the way back with how do we get extracts, as NRLS is not static.  Needed system to chose mose recent updated.

+ Problems with delimiters - required diagnosing and manual SAS process

+ ETL and warehousing

+ Extraction into reporting tables as 'counts per day'
 + HES isn't per day, it's episodes.  So per day, is a spell happening?
 + Investigated different methods of bed-day calculation, depending on which you chose, you can call day cases zero time in hospital.

### Predictors / feature engineering

+ Predictors examined as raw counts, and proportions, as well as 'offset' denominators
+ Functional forms examined - e.g. percentile representation, multi-collinearity issues, and splines.

### Regression-based models:

+ Poisson regression <a name=cite-poissonRecherchesProbabiliteJugements1837></a>([Poisson, 1837](#bib-poissonRecherchesProbabiliteJugements1837)) (`R`) as basis, in `R`
+ Mixture/compound distribution models <a name=cite-sellersFlexibleRegressionModel2010></a>([Sellers and Shmueli, 2010](https://doi.org/10/cn6mjv))

---

## Analytical methods (2)

.pull-left[

___Overdispersion___, where conditional variance is greater than conditional mean, occurs when:

1. Aggregation / Discretization
1. Mis-specified predictors/model
1. Presence of outliers
1. Variation between response probabilities (heterogeneity)

]

__Repeated measures (correlation) __

+ Regression assumes all points independent
+ Sampling from same organisations repeatedly
+ Clustered: - local means

.pull-right[
![](./assets/img/ints.gif)
]

???

__End 6:00__

+ The big issue here is overdispersion.  For distributions with fixed variance assumption (Poisson and Binomial), there isn't room in the mathematical definition for anything other than perfect.

+ That is to say that, the model assumes it's a perfect fit, and it's not. Mainly from:

---

## Analytical methods (3)

.pull-left[
###Advances on Poisson regression

+ Generalized Additive Models (GAM):<a name=cite-woodGeneralizedAdditiveModels2006></a>([Wood, 2006](#bib-woodGeneralizedAdditiveModels2006)) 
  + smoothed functions of predictors

+ Tree methods: 
  + "Boosting" <a name=cite-friedmanStochasticGradientBoosting2002></a>([Friedman, 2002](https://doi.org/10/fxb956))
  + "Bagging" & Random Forests <a name=cite-breimanBaggingPredictors1996></a><a name=cite-breimanRandomForests2001></a>([Breiman, 1996](https://doi.org/10/fm3fwj); [Breiman, 2001](https://doi.org/10/d8zjwq))

+ Neural Networks <a name=cite-hastieElementsStatisticalLearning2009></a>([Hastie, Tibshirani, and Friedman, 2009](#bib-hastieElementsStatisticalLearning2009))
]

.pull-right[

### Use of models

+ Indirectly standardised ratio

+ Overdispersion adjusted z-scores <a name=cite-spiegelhalterHandlingOverdispersionPerformance2005></a>([Spiegelhalter, 2005b](https://doi.org/10/cdmxpn))

+ Funnel plots <a name=cite-spiegelhalterFunnelPlotsComparing2005></a>([Spiegelhalter, 2005a](https://doi.org/10/fq7z8t))

+ Control charting  <a name=cite-woodallUseControlCharts2006></a>([Woodall, 2006](https://doi.org/10/gjzktt))

]

???

_Ends 7:30__

+ GAMS - probably one of my favourite methods as they are good at characterising the relations and reducing the noise (bias - variance trade-off)

+ Tree are a completely different type of regression model that involves recursively splitting data at points that explain the most variance.  Give similar results to MLE, becuse both describing the data, but some differences.  Need to control for bias and over-fitting, so bootstrapping ('bagging') and boosting (iteratively refitting weighted on error).

+ Neural networks, at simplest are cascade of rsimple models, stacking in parallel with 'layers' in sequence.  So called 'deep learning' indicates networks with more layers.  Fitted a simple feed-forward network.  Little advantage to these on small datasets.

---

## Natural Language Processing (NLP)

Key that the free-text description contains real signal, but can't manually read ~500,000 per year

.pull-left[
+ Quantitative analysis of text

+ Text processing / preparation `tidytext`, `quanteda`

+ Visualisation - wordclouds, TF-IDF

+ Topic Modelling - LDA <a name=cite-bleiLatentDirichletAllocation2003></a>([Blei Ng, et al., 2003](#bib-bleiLatentDirichletAllocation2003))

+ Using LDA predictions to predict harm level

+ __Class-imbalance__

]

.pull-right[
<p style="text-align:center;">
<img src="./assets/img/wordcloud.png" alt="Wordclouds showing stage of NRLS cleaning" width = "400" height="350">
</P>
]

???

'Proverbial elephant elephant in the room'
The key issue here is that there are ~500,000 reports per year.
National Clinical review team of 3 -5 depending on project, so they focus on severe harm and death.
With routing research and other work, maybe a further 2% are systematically reviewed.
This means 98% are not reviewed and, to my mind, 'begging' for some methods of finding a signal.

This is where text methods come in.

Class imbalance is the major issue here:  that is where there are different size classes (or groups) to predict.
As said before Severe and and Death account for >1%.
Naturally poorer at predicting these, but interesting finding what that Random Forest methods performed better than others against class imbalance ~83% accuracy, and Naive Bayes performed well for minority classes by not majority.

---
## Outputs

+ HED interactive 'module' allowing - NHSEI prevented launch of this, but techniques used to improve several other parts of HED.

+ Recommendations for modelling shared with organisations:

+ NHSD now investigating some of methods for SHMI.
 + NHSEI simple method of 'under-reporting' greatly advanced by these models.
 + NLP methods have been recognised as way forward to make better use of data.
 
+ In moving to NHSEI patient safety team, I am starting to implement these methods for national team.
 
+ Creation of `FunnelPlotR` R package, available on CRAN and donated to NHS-R community.

???

__End 8:00__

---
class: inverse center middle

# Re-imagining standardised mortality ratio (SMR) surveillance in HED

---

## Re-imagining SMR surveillance

+ When comparing hospital mortality, it is common to adjust for "case-mix" to reduce confounding.

+ Organisation conflate 'excess' predicted deaths with 'preventable' deaths, and much academic debate

.pull-left[

+ Commonly measures are:
  + Summary Hospital-level Mortality Indicator (SHMI) <a name=cite-campbellDevelopingSummaryHospital2012></a>([Campbell Jacques, et al., 2012](https://doi.org/10/gb3r9t))
  + Dr Foster Hospital Standardised Mortality Ratio (HSMR) <a name=cite-jarmanExplainingDifferencesEnglish1999></a>([Jarman Gault, et al., 1999](https://doi.org/10/fkkfm9))

+ Commonly make the cross-sectional comparisons with Funnel Plot

+ __Unanswered question:__
  + 'Is trust "X" getting worse'?

]

.pull-right[

]

???

First point:  reduces confounding, but sometimes increases bias and is at the expense of complexity.

---

## Monitoring SMRs over time
.pull-left[
### Risk-adjusted cusum charts

+ Two differing approaches:
 + CQC outliers programme
 + Dr Foster's cususm alerts

+ Organisations don't understand as 'diagnostic test'

]

.pull-right[

]

???

For methods with more of a time dimension, one might consider forecasting of survival methods, in general.

Dr Foster Intelligence created an industry or 'alerting' trusts to risks - based on the methods they CQC

The diagnostic test, the same in any machine learning prediction model, is essential.  In epidemiology and machine learning we might construct a 2x2 table, or confusion matrix, with our predictions v.s. truth.  Assess quality of model, but there are always some false positive and negatives.  There is a trade-off between 'sensitivity' and 'specificity':  You can catch more with a lower threshold, but you'll get more false positives.  Extreme of this is to treat every data point as positive.
This has resource implications - requiring clinical review and often highlighting data quality or statistical issues.
--

+ Unclear why there were differences:
  + Aggregation
  + Trigger limit
  + 'Window' size

+ __Both claimed false discovery rate (FDR) of 0.1%__

---

### Developing methods

.pull-left[
### CQC
+ Data are transformed to z-scores
+ Global trigger (5.48) - 
  + Can convert average run-length to FDR, and set threshold <a name=cite-griggNullSteadystateDistribution2008></a><a name=cite-carequalitycommissioncqcNHSAcuteHospitals2014></a>([Grigg and Spiegelhalter, 2008](https://doi.org/10/bgvkdx); [Care Quality Commission, 2014](#bib-carequalitycommissioncqcNHSAcuteHospitals2014))
+ Applicable to other indicators and groupings, subject to transformation
+ Tested in R, translated in `SQL` procedures for calculation against all trusts and diagnosis groups
]

.pull-right[
### DFI
+ Binomial assumption and threshold set through simulation of average run-length to false positives. <a name=cite-bottlePredictingFalseAlarm2011></a>([Bottle and Aylin, 2011](https://doi.org/10/cbr4rq))
+ Unique to each trust / group / reporting period.
+ Intractable to calculate each month, and authors fitted a set of descriptive equations.
+ Wrote program to solve equations for each group with non-linear optimiser (`R`)
+ Translated into similar `SQL` process, with call to `R`

]

---

## Delivery and presentation

### Technical

+ Takes too long!
  + Iterative ~ 160 organisations \* 260 diagnosis groups \* 2 methods.
  + Tested multiple languages and data structures, designs
  + SQL 'quirky update' - best performance

### Display

+ Important to place charts together to see trigger times
  + Different aggregations
+ Balance of detail and simplicity
+ Linked to email alerts and dashboard summary
+ Patient details - audit

__Educate trusts on false-positive argument and build confidence, and skills to investigate.__

---
class: references

### References (1)

<a name=bib-bleiLatentDirichletAllocation2003></a>[Blei, D. M., A. Y.
Ng, and M. I. Jordan](#cite-bleiLatentDirichletAllocation2003) (2003).
"Latent dirichlet allocation". In: _J. Mach. Learn. Res._ 3, pp.
993-1022. ISSN: 1532-4435.

<a name=bib-bottlePredictingFalseAlarm2011></a>[Bottle, A. and P.
Aylin](#cite-bottlePredictingFalseAlarm2011) (2011). "Predicting the
false alarm rate in multi-institution mortality monitoring". In: _The
Journal of the Operational Research Society_ 62.9, pp. 1711-1718. ISSN:
01605682, 14769360. DOI: [10/cbr4rq](https://doi.org/10%2Fcbr4rq).

<a name=bib-breimanBaggingPredictors1996></a>[Breiman,
L.](#cite-breimanBaggingPredictors1996) (1996). "Bagging predictors".
In: _Machine Learning_ 24.2, pp. 123-140. ISSN: 1573-0565. DOI:
[10/fm3fwj](https://doi.org/10%2Ffm3fwj).

<a name=bib-breimanRandomForests2001></a>[Breiman,
L.](#cite-breimanRandomForests2001) (2001). "Random Forests". In:
_Machine Learning_ 45.1, pp. 5-32. ISSN: 08856125. DOI:
[10/d8zjwq](https://doi.org/10%2Fd8zjwq).

<a name=bib-campbellDevelopingSummaryHospital2012></a>[Campbell, M. J.,
R. M. Jacques, J. Fotheringham, et
al.](#cite-campbellDevelopingSummaryHospital2012) (2012). "Developing a
summary hospital mortality index: retrospective analysis in English
hospitals over five years". In: _BMJ_ 344, p. e1001. DOI:
[10/gb3r9t](https://doi.org/10%2Fgb3r9t).

<a name=bib-carequalitycommissioncqcNHSAcuteHospitals2014></a>[Care
Quality
Commission](#cite-carequalitycommissioncqcNHSAcuteHospitals2014)
(2014). _NHS acute hospitals: Statistical Methodology_. Care Quality
Commission (CQC).

<a name=bib-donaldsonOrganisationMemory2002></a>[Donaldson,
L.](#cite-donaldsonOrganisationMemory2002) (2002). "An organisation
with a memory". In: _Clinical Medicine (London, England)_ 2.5, pp.
452-7. ISSN: 1470-2118 (Print) 1470-2118. DOI:
[10/gfz5w4](https://doi.org/10%2Fgfz5w4).

<a name=bib-friedmanStochasticGradientBoosting2002></a>[Friedman, J.
H.](#cite-friedmanStochasticGradientBoosting2002) (2002). "Stochastic
Gradient Boosting". In: _Computational Statistics & Data Analysis_ 38,
pp. 367-378. DOI: [10/fxb956](https://doi.org/10%2Ffxb956).

<a name=bib-griggNullSteadystateDistribution2008></a>[Grigg, O. and D.
Spiegelhalter](#cite-griggNullSteadystateDistribution2008) (2008). "The
null steady-state distribution of the CUSUM statistic". In:
_Technometrics : a journal of statistics for the physical, chemical,
and engineering sciences_. DOI:
[10/bgvkdx](https://doi.org/10%2Fbgvkdx).

<a name=bib-hastieElementsStatisticalLearning2009></a>[Hastie, T., R.
Tibshirani, and J.
Friedman](#cite-hastieElementsStatisticalLearning2009) (2009). _The
Elements of Statistical Learning : Data Mining, Inference, and
Prediction_. 2nd ed. Springer Series in Statistics. New York,
NETHERLANDS: Springer. ISBN: 978-0-387-84858-7.

<a name=bib-jarmanExplainingDifferencesEnglish1999></a>[Jarman, B., S.
Gault, B. Alves, et al.](#cite-jarmanExplainingDifferencesEnglish1999)
(1999). "Explaining differences in English hospital death rates using
routinely collected data". In: _BMJ_ 318.7197, pp. 1515-1520. ISSN:
0959-8138 1468-5833. DOI: [10/fkkfm9](https://doi.org/10%2Ffkkfm9).

<a name=bib-kohnErrHumanBuilding2000></a>[Kohn, L. T., J. M. Corrigan,
and M. S. Donaldson](#cite-kohnErrHumanBuilding2000) (2000). "To Err is
Human: Building a Safer Health System". In: _To Err is Human: Building
a Safer Health System_. Ed. by L. T. Kohn, J. M. Corrigan and M. S.
Donaldson. Washington (DC).

<a name=bib-leapeNatureAdverseEvents1991></a>[Leape, L. L., T. A.
Brennan, N. Laird, et al.](#cite-leapeNatureAdverseEvents1991) (1991).
"The Nature of Adverse Events in Hospitalized Patients". In: _New
England Journal of Medicine_ 324.6, pp. 377-384. DOI:
[10/fhvzbs](https://doi.org/10%2Ffhvzbs).

<a name=bib-maineyStatisticalMethodsNHS2020></a>[Mainey,
C.](#cite-maineyStatisticalMethodsNHS2020) (2020). "Statistical methods
for NHS incident reporting data". London. URL:
[https://discovery.ucl.ac.uk/id/eprint/10094736/](https://discovery.ucl.ac.uk/id/eprint/10094736/).

---
class: references

### References (2)

<a name=bib-poissonRecherchesProbabiliteJugements1837></a>[Poisson, S.
D.](#cite-poissonRecherchesProbabiliteJugements1837) (1837).
_Recherches sur la probabilité des jugements en matiére criminelle et
en matière civile: précédées des règles générales du calcul des
probabilités_. Bachelier.

<a name=bib-sellersFlexibleRegressionModel2010></a>[Sellers, K. F. and
G. Shmueli](#cite-sellersFlexibleRegressionModel2010) (2010). "A
flexible regression model for count data". In: _Annals of Applied
Statistics_ 4.2, pp. 943-961. ISSN: 1932-6157. DOI:
[10/cn6mjv](https://doi.org/10%2Fcn6mjv).

<a name=bib-spiegelhalterFunnelPlotsComparing2005></a>[Spiegelhalter,
D. J.](#cite-spiegelhalterFunnelPlotsComparing2005) (2005a). "Funnel
plots for comparing institutional performance". In: _Statistics in
Medicine_ 24.8, pp. 1185-202. ISSN: 0277-6715 (Print) 0277-6715
(Linking). DOI: [10/fq7z8t](https://doi.org/10%2Ffq7z8t).

<a
name=bib-spiegelhalterHandlingOverdispersionPerformance2005></a>[Spiegelhalter,
D. J.](#cite-spiegelhalterHandlingOverdispersionPerformance2005)
(2005b). "Handling over-dispersion of performance indicators". In:
_Quality and Safety in Health Care_ 14.5, pp. 347-351. DOI:
[10/cdmxpn](https://doi.org/10%2Fcdmxpn).

<a name=bib-woodGeneralizedAdditiveModels2006></a>[Wood, S.
N.](#cite-woodGeneralizedAdditiveModels2006) (2006). _Generalized
Additive Models: An Introduction with R_. Taylor & Francis. ISBN:
978-1-58488-474-3.

<a name=bib-woodallUseControlCharts2006></a>[Woodall, W.
H.](#cite-woodallUseControlCharts2006) (2006). "The Use of Control
Charts in Health-Care and Public-Health Surveillance". In: _Journal of
Quality Technology_ 38.2, pp. 89-104. ISSN: 0022-4065. DOI:
[10/gjzktt](https://doi.org/10%2Fgjzktt).