- Education
- Evaluating the impact...
- Evaluating the impact of healthcare interventions using routine data
CCBY Open access
Practice Essentials BMJ 2019; 365 doi: https://doi.org/10.1136/bmj.l2239 (Published 20 June 2019) Cite this as: BMJ 2019;365:l2239 Read the full collection
- Geraldine M Clarke, senior data analyst1,
- Stefano Conti, senior statistician2,
- Arne T Wolters, senior analytics manager1,
- Adam Steventon, director of data analytics1
- Correspondence to G Clarke Geraldine.clarke{at}health.org.uk
What you need to know
Assessing the impact of healthcare interventions is critical to inform future decisions
Compare observed outcomes with what you would have expected if the intervention had not been implemented
A wide range of routinely collected data is available for the evaluation of healthcare interventions
Interventions to transform the delivery of health and social care are being implemented widely, such as those linked to Accountable Care Organizations in the United States,1 or to integrated care systems in the UK.2 Assessing the impact of these health interventions enables healthcare teams to learn and to improve services, and can inform future policy.3 However, some healthcare interventions are implemented without high quality evaluation, in ways that require onerous data collection, or may not be evaluated at all.4
A range of routinely collected administrative and clinically generated healthcare data could be used to evaluate the impact of interventions to improve care. However, there is a lack of guidance as to where relevant routine data can be found or accessed and how they can be linked to other data. A diverse array of methodological literature can also make it hard to understand which methods to apply to analyse the data. This article provides an introduction to help clinicians, commissioners, and other healthcare professionals wishing to commission, interpret, or perform an impact evaluation of a health intervention. We highlight what to consider and discuss key concepts relating to design, analysis, implementation, and interpretation.
What are interventions, impacts, and impact evaluations?
A health intervention is a combination of activities or strategies designed to assess, improve, maintain, promote, or modify health among individuals or an entire population. Interventions can include educational or care programmes, policy changes, environmental improvements, or health promotion campaigns. Interventions that include multiple independent or interacting components are referred to as complex.5 The impact of any intervention is likely to be shaped as much by the context (eg, communities, work places, homes, schools, or hospitals) in which it is delivered, as the details of the intervention itself.6789
An impact is a positive or negative, direct or indirect, intended or unintended change produced by an intervention. An impact evaluation is a systematic and empirical investigation of the effects of an intervention; it assesses to what extent the outcomes experienced by affected individuals were caused by the intervention in question, and what can be attributed to other factors such as other interventions, socioeconomic trends, and political or environmental conditions. Evaluations can be categorised as formative or summative (table 1).
Table 1
Impact evaluations
View this table:
Approaches such as the Plan, Do, Study, Act cycle11, which is part of the Model for Improvement, a commonly used tool to test and understand small changes in quality improvement work12 may be used to undertake formative evaluation.
With either type of evaluation, it is important to be realistic about how long it will take to see the intended effects. Assessment that takes place too soon risks incorrectly concluding that there was no impact. This might lead stakeholders to question the value of the intervention, when later assessment might have shown a different picture. For example, in a small case study of cost savings from proactively managing high risk patients, the costs of healthcare for the eligible intervention population initially increased compared with the comparison population, but after six months were consistently lower.14
This article focuses on impact evaluation, but this can only ever address a fraction of questions.15 Much more can be accomplished if it is supplemented with other qualitative and quantitative methods, including process evaluation. This provides context, assesses how the intervention was implemented, identifies any emerging unintended pathways, and is important for understanding what happened in practice and for identifying areas for improvement.16 The economic evaluation of healthcare interventions is also important for healthcare decision making, especially with ongoing financial pressures on health services.17
What are the right evaluation questions?
An effective impact evaluation begins with the formulation of one or more clear questions driven by the purpose of the evaluation and what you and your stakeholders want to learn. For example, “What is the impact of case management on patients’ experience of care?”
Formulate your evaluation questions using your understanding of the idea behind your intervention, the implementation challenges, and your knowledge of what data are available to measure outcomes. Review your theory of change or logic model2122 to understand what inputs and activities were planned, and what outcomes were expected and when. Once you have understood the intended causal pathway, consider the practical aspects of implementation, which include the barriers to change, unexpected changes by recipients or providers, and other influences not previously accounted for. Patient and public involvement (PPI) in setting the right question is strongly recommended for additional insights and meaningful results. For example, if evaluating the impact of case management, you could engage patients to understand what outcomes matter most to them. Healthcare leaders may emphasise metrics such as emergency admissions, but other aspects such as the experience of care might matter more to patients.523
What methods can be used to perform an impact evaluation?
Randomised control designs, where individuals are randomly selected to receive either an intervention or a control treatment, are often referred to as the “gold standard” of causal impact evaluation.24 In large enough samples, the process of randomisation ensures a balance in observed and unobserved characteristics between treatment and control groups. However, while often suitable for assessing, for example, the safety and efficacy of medicines, these designs may be impractical, unethical, or irrelevant when assessing the impact of complex changes to health service delivery.
Observational studies are an alternative approach to estimate causal effects. They use the natural, or unplanned, variation in a population in relation to the exposure to an intervention, or the factors that affect its outcomes, to remove the consequences of a non-randomised selection process.25 The idea is to mimic a randomised control design by ensuring treated and control groups are equivalent—at least in terms of observed characteristics. This can be achieved using a variety of well documented methods, including regression control and matching,26 eg, propensity scoring27 or genetic matching.28 If the matching is successful at producing such groups, and there are also no differences in unobserved characteristics, then it can be assumed that the control group outcomes are representative of those that the treated group would have experienced if nothing had changed, ie, the counterfactual. For example, an evaluation of alternative elective surgical interventions for primary total hip replacement on osteoarthritis patients in England and Wales used genetic matching to compare patients across three different prosthesis groups, and reported that the most prevalent type of hip replacement was the least cost effective.29
Assessing similarity is only possible in relation to observed characteristics, and matching can result in biased estimates if the groups differ in relation to unobserved variables that are predictive of the outcome (confounders). It is rarely possible to eliminate this possibility of bias when conducting observational studies, meaning that the interpretation of the findings must always be sensitive to the possibility that the differences in outcomes were caused by a factor other than the intervention. Methods that can help when selection is on unobserved characteristics include difference-in-difference,30 regression discontinuity,31 instrumental variables,18 or synthetic controls.32Table 2 gives a summary of selected observational study designs.
Table 2
Observational study designs for quantitative impact evaluation
View this table:
Observational studies are often referred to as natural (for natural or unplanned interventions), or quasi (for planned or intentional interventions) experiments. Natural experiments are discussed to evaluate population health interventions.41
What’s wrong with a simple before-and-after study?
Before-and-after studies compare changes in outcomes for the same group of patients at a single time point before and after receiving an intervention without reference to a control group. These differ from interrupted time series studies, which compare changes in outcomes for successive groups of patients before and after receiving an intervention (the interruption).
Before-and-after studies are useful when it is not possible to include an unexposed control group, or for hypothesis generation. However, they are inherently susceptible to bias since changes observed may simply reflect regression to the mean (any changes in outcomes that might occur naturally in the absence of the intervention), or influences or secular trends unrelated to the intervention, eg, changes in the economic or political environment, or a heightened public awareness of issues.
For example, a before-and-after study of the impact of a care coordination service for older people tracked the hospital utilisation of the same patients before and after they were accepted into the service. They found that the service resulted in savings in hospital bed days and attendances at the emergency department.42 Reduced hospital utilisation could have reflected regression to the mean here rather than the effects of the intervention; for example, a patient could have had a specific health crisis before being invited to join the service and then reverted back to their previous state of health and hospital utilisation for reasons unconnected with the care coordination service.
Various tools are available to evaluate the risk of bias in non-randomised designs due to confounding and other potential biases.4344
Where can I find suitable routine data?
Healthcare systems generate vast amounts of data as part of their routine operation. These datasets are often designed to support direct care, and for administrative purposes, rather than for research, and use of routinely collected data for evaluating changes in health service delivery is not without pitfalls. For example, any variation observed between geographical regions, providers, and sometimes individual clinicians may reflect real and important variations in the actual healthcare quality provided, but can also result from differences in measurement.45 However, routine data can be a rich source of information on a large group of patients with different conditions across different geographical regions. Often, data have been collected for many years, enabling construction of individual patient histories describing healthcare utilisation, diagnoses, comorbidities, prescription of medication, and other treatments.
Some of these data are collected centrally, across a wider system, and routinely shared for research and evaluation purposes, eg, secondary care data in England (Hospital Episode Statistics), or Medicare Claims data in the United States. Other sources, such as primary care data, are often collected at a more local level, but can be accessed through, or on behalf of, healthcare commissioners, provided the right information governance arrangements are in place. Pseudonymised records, where any identifying information is removed or replaced by an artificial identifier, are often used to support evaluation while maintaining patient confidentiality. See table 3 for commonly used routine datasets available in England.
Table 3
Commonly used routine datasets available in the NHS in England
View this table:
Healthcare records can often be linked across different sources as a single patient identifier is commonly used across a healthcare system, eg, the use of an NHS number in the UK. Using a common pseudonym across different data sources can support linkage of pseudonymised records. Linking into publicly available sources of administrative data and surveys can further enrich healthcare records. Commonly used administrative data available for UK populations include measures of GP practice quality and outcomes from the Quality and Outcomes Framework (QOF),52 deprivation, rurality, and demographics from the 2011 Census,53 and patient experience from the GP Patient Survey.54
Are there any additional considerations?
It is essential to consider threats to validity when designing and evaluating an impact evaluation; validity relates to whether an evaluation is measuring what it is claiming to measure. See Rothman et al55 for further discussion.
Internal validity refers to whether the effects observed are due to the intervention and not some other confounding factor. Selection bias, which results from the way in which subjects are recruited, or from differing rates of participation due, for example, to age, gender, cultural or socioeconomic factors, is often a problem in non-randomised designs. Care must be taken to account for such biases when interpreting the results of an impact evaluation. Sensitivity analyses should be performed to provide reassurance regarding the plausibility of causal inferences.
External validity refers to the extent to which the results of a study can be generalised to other settings. Understanding the societal, economic, health system, and environmental context in which an intervention is delivered, and which makes its impact unique, is critical when interpreting the results of evaluations, and considering whether they apply to your setting.56 Descriptions of context should be as rich as possible.
Often, the impact of an intervention is likely to vary depending on the characteristics of patients. These can be usefully explored in subgroup analyses.57
Clear and transparent reporting using established guidelines (eg, STROBE58 or TREND59)to describe the intervention, study population, assignment of treatment, and control groups, and methods used to estimate impact should be followed. Limitations arising as a result of inherent biases, or validity, should be clearly acknowledged.
Around the world, many interventions designed to improve health and healthcare are under way. An evaluation is an essential part of understanding what impact these changes are having, for whom and in what circ*mstances, and help inform future decisions about improvement and further roll out. There is no standard, ‘‘one size fits all’’ recipe for a good evaluation: it must be tailored to the project at hand. Understanding the overarching principles and standards is the first step towards a good evaluation.
Further Resources
See The Health Foundation. Evaluation: what to consider. 201560 for a list of websites, articles, webinars and other guidance on various aspects of impact evaluation, which may help locate further information for the planning, interpretation, and development of a successful impact evaluation.5 23 55
Education into practice
What interventions have you designed or experienced aimed at transforming your service? Have they been evaluated?
What types of routine data are collected about the care you deliver? Do you know how to access them and use them to evaluate care delivery?
What resources are available to you to support impact evaluations for interventions?
Footnotes
Contributors GMC, SC, ATW and AS designed the structure of the report. GMC wrote the first draft of the manuscript. SC wrote table 2. ATW wrote table 3. AS and GMC critically revised the manuscript for important intellectual content. All authors approved the final version of the manuscript.
Competing interests We have read and understood BMJ policy on declaration of interests. All authors work in the Improvements Analytics Unit, a joint project between NHS England and the Health Foundation, which provided support for work reported in references of this report.133760
Provenance and peer review: This article is part of a series commissioned by The BMJ based on ideas generated by a joint editorial group with members from the Health Foundation and The BMJ, including a patient/carer. The BMJ retained full editorial control over external peer review, editing, and publication. Open access fees and The BMJ’s quality improvement editor post are funded by the Health Foundation.
Patient and/or members of the public were not involved in the creation of this article.
This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.
References
- ↵
- Davis K,
- Guterman S,
- Collins S,
- Stremikis G,
- Rustgi S,
- Nuzum R
. Starting on the path to a high performance health system: Analysis of the payment and system reform provisions in the Patient Protection and Affordable Care Act of.The Commonwealth Fund, 2010, https://www.commonwealthfund.org/publications/fund-reports/2010/sep/starting-path-high-performance-health-system-analysis-payment.
- ↵
- NHS NHS Long Term Plan
- ↵
- Djulbegovic B
. A framework to bridge the gaps between evidence-based medicine, health outcomes, and improvement and implementation science. J Oncol Pract2014;10:200-2. doi:10.1200/JOP.2013.001364pmid:24839282
- ↵
- Bickerdike L,
- Booth A,
- Wilson PM,
- et al
. Social prescribing: less rhetoric and more reality. A systematic review of the evidence. BMJ Open2017;7:e013384.
- ↵
- Campbell M,
- Fitzpatrick R,
- Haines A,
- et al
. Framework for design and evaluation of complex interventions to improve health. BMJ2000;321:694-6. doi:10.1136/bmj.321.7262.694pmid:10987780
- ↵
- ↵
- ↵
- ↵
- Pawson R,
- Tilley N
. Realistic evaluation.Sage, 1997.
Smith J, Wistow G. (Nuffield Trust comment) Learning from an intrepid pioneer: integrated care in North West London. https://www.nuffieldtrust.org.uk/news-item/learning-from-an-intrepid-pioneer-integrated-care-in-north-west-london
- ↵
- Improvement NHS
. Plan, Do, Study, Act (PDSA) cycles and the model for improvement.Handb Qual Serv Improv Tools, 2010.
- ↵
Academy of Medical Royal Colleges. Quality Improvement—training for better outcomes. 2016. https://www.aomrc.org.uk/wp-content/uploads/2016/06/Quality_improvement_key_findings_140316-2.pdf
- ↵
Lloyd T, Wolters A, Steventon A. The impact of providing enhanced support for care home residents in Rushcliffe. 2017. http://www.health.org.uk/sites/health/files/IAURushcliffe.pdf
- ↵
- Ferris TG,
- Weil E,
- Meyer GS,
- Neagle M,
- Heffernan JL,
- Torchiana DF
. Cost savings from managing high-risk patients. In: Yong PL, Saunders RS, Olsen LA, editors. The healthcare imperative: lowering costs and improving outcomes: workshop series summary.Nat Acad Press (US); 2010:301. https://www.ncbi.nlm.nih.gov/books/NBK53910/
- ↵
- Greenhalgh T,
- Papoutsi C
. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med2018;16:4-9.
- ↵
- Moore GF,
- Audrey S,
- Barker M,
- et al
. Process evaluation of complex interventions: Medical Research Council guidance. BMJ2015;350:h1258. doi:10.1136/bmj.h1258pmid:25791983
- ↵
- Drummond M,
- Weatherly H,
- Ferguson B
. Economic evaluation of health interventions. BMJ2008;337:a1204. doi:10.1136/bmj.a1204pmid:18824485
- ↵
- Lorch SA,
- Baiocchi M,
- Ahlberg CE,
- Small DS
. The differential impact of delivery hospital on the outcomes of premature infants. Pediatrics2012;130:270-8. doi:10.1542/peds.2011-2820pmid:22778301
- Martens EP,
- Pestman WR,
- de Boer A,
- Belitser SV,
- Klungel OH
. Instrumental variables: application and limitations. Epidemiology2006;17:260-7. doi:10.1097/01.ede.0000215160.88317.cbpmid:16617274
- ↵
- Center for Theory of Change
- ↵
- Davidoff F,
- Dixon-Woods M,
- Leviton L,
- Michie S
. Demystifying theory and its use in improvement. BMJ Qual Saf2015;24:228-38. doi:10.1136/bmjqs-2014-003627pmid:25616279
- ↵
Gertler PJ, Martinez S, Premand P, Rawlings LB, Vermeersch CMJ. Impact evaluation in practice. The World Bank Publications. 2017. https://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/Impact_Evaluation_in_Practice.pdf
- ↵
Cochrane A. Effectiveness and efficiency: random reflections on health services. London; 1972. https://www.nuffieldtrust.org.uk/research/effectiveness-and-efficiency-random-reflections-on-health-services
- ↵
- Portela MC,
- Pronovost PJ,
- Woodco*ck T,
- Carter P,
- Dixon-Woods M
. How to study improvement interventions: a brief overview of possible study types. Postgrad Med J2015;91:343-54. doi:10.1136/postgradmedj-2014-003620reppmid:26045562
- ↵
- ↵
- Austin PC
. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res2011;46:399-424. doi:10.1080/00273171.2011.568786pmid:21818162
- ↵
- ↵
- Pennington M,
- Grieve R,
- Sekhon JS,
- Gregg P,
- Black N,
- van der Meulen JH
. Cemented, cementless, and hybrid prostheses for total hip replacement: cost effectiveness analysis. BMJ2013;346:f1026.
- ↵
- ↵
- Venkataramani AS,
- Bor J,
- Jena AB
. Regression discontinuity designs in healthcare research. BMJ2016;352:i1216. doi:10.1136/bmj.i1216pmid:26977086
- ↵
- Abadie A,
- Gardeazabal J
. The economic costs of conflict: a case study of the Basque country. Am Econ Rev2003;93:113-32doi:10.1257/000282803321455188.
- McNamee R
. Regression modelling and other methods to control confounding. Occup Environ Med2005;62:500-6, 472. doi:10.1136/oem.2002.001115pmid:15961628
- Ho DE,
- Imai K,
- King G,
- Stuart EA
. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Polit Anal2007;15:199-236doi:10.1093/pan/mpl013.
- Sutton M,
- Nikolova S,
- Boaden R,
- Lester H,
- McDonald R,
- Roland M
. Reduced mortality with hospital pay for performance in England. N Engl J Med2012;367:1821-8. doi:10.1056/NEJMsa1114951pmid:23134382
- ↵
Stephen O, Wolters A, Steventon A. Briefing: The impact of redesigning urgent and emergency care in Northumberland 2017. https://www.health.org.uk/sites/health/files/IAUNorthumberland.pdf
- Bernal JL,
- Cummins S,
- Gasparrini A
. Interrupted time series regression for the evaluation of public health interventions: a tutorial.Int J Epidemiol2016, 46:348-55.
- Donegan K,
- Fox N,
- Black N,
- Livingston G,
- Banerjee S,
- Burns A
. Trends in diagnosis and treatment for people with dementia in the UK from 2005 to 2015: a longitudinal retrospective cohort study. Lancet Public Health2017;2667:1-8.
- ↵
- Craig P,
- Cooper C,
- Gunnell D,
- et al
. Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. J Epidemiol Community Health2012;66:1182-6. doi:10.1136/jech-2011-200375pmid:22577181
- ↵
- ↵
- Sterne JA,
- Hernán MA,
- Reeves BC,
- et al
. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ2016;355:i4919. doi:10.1136/bmj.i4919pmid:27733354
- ↵
- Chen YF,
- Hemming K,
- Stevens AJ,
- Lilford RJ
. Secular trends and evaluation of complex interventions: the rising tide phenomenon. BMJ Qual Saf2016;25:303-10. doi:10.1136/bmjqs-2015-004372pmid:26442789
- ↵
- Powell AE,
- Davies HT,
- Thomson RG
. Using routine comparative data to assess the quality of health care: understanding and avoiding common pitfalls. Qual Saf Health Care2003;12:122-8. doi:10.1136/qhc.12.2.122pmid:12679509
NHS Digital. Hospital Episode Statistics. https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/hospital-episode-statistics
NHS Digital. Data Access Request Service (DARS) https://digital.nhs.uk/services/data-access-request-service-dars
NHS Digital. Secondary Uses Service (SUS) https://digital.nhs.uk/services/secondary-uses-service-sus
Medicines and Healthcare Regulatory Agency and National Institute for Health Research (NIHR). Clinical Practice Research Datalink(CPRD). https://www.cprd.com
Office for National Statistics. Deaths. https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths
NHS Digital. Mental Health Services Data Set. https://digital.nhs.uk/data-and-information/data-collections-and-data-sets/data-sets/mental-health-services-data-set
- ↵
NHS Digital. Quality Outcomes Framework (QOF) https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/general-practice-data-hub/quality-outcomes-framework-qof
- ↵
- Office for National Statistics
. 2011 Census. https://www.ons.gov.uk/census/2011census
- ↵
- NHS England
. GP Patient Survey (GPPS). https://www.gp-patient.co.uk/
- ↵
- Rothman KJ,
- Greenland S,
- Lash T
. Modern Epidemiology.Lippincott Williams & Williams, 2005.
- ↵
- Minary L,
- Alla F,
- Cambon L,
- Kivits J,
- Potvin L
. Addressing complexity in population health intervention research: the context/intervention interface. J Epidemiol Community Health2018;72:319-23.pmid:29321174
- ↵
- Sun X,
- Briel M,
- Walter SD,
- Guyatt GH
. Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. BMJ2010;340:c117.
- ↵
- von Elm E,
- Altman DG,
- Egger M,
- Poco*ck SJ,
- Gotzsche PC,
- Vandenbroucke JP
. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med2007;147:573-7.
- ↵
- Des Jarlais DC,
- Lyles C,
- Crepaz N
. the TREND. Improving the reporting quality of nonrandomized evaluations: the TREND statement. Am J Public Health2004;94:361-6.
- ↵
The Health Foundation. Evaluation: what to consider. 2015. https://www.health.org.uk/publications/evaluation-what-to-consider
Back to top
