Risk Stratification Tools

class: title-slide

# Risk Stratification Tools

### Deputy Director Specialist Analytics Interview

### <img src="./assets/img/crop.png" alt="Mainard icon" height="40px" />  Dr Chris Mainey

.footnote[Presentation and code available: **https://github.com/chrismainey/risk_stratification_model_presentation**]

.art_cap[R generative art - inspired by Antonio Sánchez Chinchón - @aschinchon]

???

__End 30 seconds__!

Say hi - introduce myself

???
Presentation on github with references and code, share afterwards

---
class: inverse center middle

## How risk stratification models could be used to improve health outcomes in the NHS

???

How you think risk stratification models could be used to improve health outcomes in the NHS

---

## What is risk-stratification?
+ Stratification (grouping or sorting) of population according to the risk of an event, usually determined by expert opinion, rule sets or statistical estimation of frequency based on correlated predictors.

???
These tools use relationships in historic population data to estimate the use of 
health care services for each member of a population. Risk stratification tools can be useful both 
for population planning purposes (known as “risk stratification for commissioning”) and for 
identifying which patients should be offered targeted, preventive support (known as “risk 
stratification for case finding”).

### Benefits for health outcomes

+ Quantified individual risk can allow for more personalised care in many settings

+ Understanding distribution of needs
  + Commissioning the right sized services to meet needs
  + Planning and delivering service in a way that is accessible or appropriate to the population

+ Monitoring: 
  + Allow closer monitoring of high risk patients (e.g. diabetes patients and foot-care).
  + Change in risk can itself be a trigger for intervention
  + Quality / Outcome monitoring

---
class: inverse center middle

## How to evaluate whether the risk stratification model(s) are reliably predicting population vs. individual risks

???

How you would evaluate whether the risk stratification model(s) was reliably predicting population vs. individual risks

---

> _"...Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad..."_ (Box, 1976)

### Assessing models

+ Models generally built for global accuracy (may or may not be population)

+ Understanding metrics:
  + AUC / R<sup>2</sup> global proportion of variance explained
  + MSE / RMSE / MAE - average errors per observation
  
--

### Validation in the population

+ Sub-group / marginal analysis
  + Understand where fit is poor
  + Class imbalance
  
+ Population weighting to assess fit e.g.(van
Alten Domingue, et al., 2022)

---
class: inverse center middle

## Considerations for deployment of risk stratification models across a complex health and care system

???
Considerations for deployment of risk stratification models across a complex health and care system

---

> _"...The right balance between productivity and equity is a value judgement. The decision will 
rest with ICBs, but a sensible one will engage with its population, their representatives 
and its staff and explore the trade-offs they may be willing to make..."_ (Wyatt, 2022)

.pull-left[
### Context
+ Awareness system you are working in and limitations
+ e.g. National VTE risk scoring
  + Built for Acute providers
  + How do we use it in mental health settings.

<br><br>
### Validation
+ Is this appropriate for your specific population / system?
+ Has it been tested in you population?
]

.pull-right[
### Monitoring

+ Change of usecase, e.g.:
  + Charlson score (Charlson Pompei, et al., 1987)
  + NEWS2 for COVID-19 patients (Carr Bendayan, et al., 2021)
+ Model drift / retraining
+ Interaction with other metrics

### Communication / engagement

+ Stakeholder engagement
+ Model, scope and limitations clear?
+ How can you take feedback?

]

---
class: inverse center middle
.pull-left[
<img src="https://imgs.xkcd.com/comics/curve_fitting.png" alt="XKCD Curve-fitting" height="600px" />
]

.pull-right[

__Source:__ xkcd: A webcomic of romance, sarcasm, math, and language

'Curve-Fitting Methods and the messages they send'

https://xkcd.com/2048/
]

---
class: references

### References

Alten, S. van, B. W. Domingue, T. Galama, et al. (2022). "Reweighting
the UK Biobank to Reflect Its Underlying Sampling Population
Substantially Reduces Pervasive Selection Bias Due to Volunteering".
In: _medRxiv_. DOI: 10.1101/2022.05.16.22275048. <URL:
https://www.medrxiv.org/content/early/2022/05/16/2022.05.16.22275048>.

Box, G. E. P. (1976). "Science and Statistics". In: _Journal of the
American Statistical Association_ 71.356, pp. 791-799. DOI:
10.1080/01621459.1976.10480949. <URL:
https://www.tandfonline.com/doi/abs/10.1080/01621459.1976.10480949>.

Carr, E., R. Bendayan, D. Bean, et al. (2021). "Evaluation and
Improvement of the National Early Warning Score (NEWS2) for COVID-19: A
Multi-Hospital Study". In: _BMC Medicine_ 19.1, p. 23. ISSN: 1741-7015.
DOI: 10.1186/s12916-020-01893-3. <URL:
https://doi.org/10.1186/s12916-020-01893-3>.

Charlson, M. E., P. Pompei, K. L. Ales, et al. (1987). "A New Method of
Classifying Prognostic Comorbidity in Longitudinal Studies: Development
and Validation". In: _Journal of Chronic Diseases_ 40.5, pp. 373-83.
ISSN: 0021-9681 (Print) 0021-9681 (Linking). DOI: 10/c4rqj6.

Wyatt, S. (2022). _Strategies to reduce inequalities in access to
planned hospital procedures_. <URL:
https://www.midlandsdecisionsupport.nhs.uk/wp-content/uploads/2022/05/Strategies-to-reduce-inequalities-in-access-to-planned-hospital-procedures_20220429iv.pdf>.