Workstream 2 – assessing the feasibility of efficient data methods for use in future implementation research

The accuracy of clinical outcome data on VTE events and bleeding is key to any future study of RAM performance and/or validation. However, any such studies are likely to require a large sample size, given the potential for low event rates and small absolute risk impact. The practical challenges and costs of individual patient contact and routine follow-up in such large studies may be prohibitive.

In this study, we sought to pilot the use of efficient methods alongside routine practice to determine the feasibility of a future implementation study of VTE risk assessment tools in hospital inpatients. Our main aim was to compare the accuracy and completeness of available coding data, local registry data and efficient data collection methods to determine clinically relevant VTE and bleeding outcomes against case note review by trained research assistants. We also collated further information on delivery methods for VTE risk assessment during routine care and the potential for granular data collection on individual risk variables, which could allow efficient RAM comparison in future work. Finally, we explored the ethical and regulatory acceptability of a data-enabled approach to any future potential implementation study.

We conducted this study in accordance with international in accordance with Enhancing the QUAlity and Transparency Of health Research (EQUATOR) guidelines. A completed STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) reporting checklist was used throughout to inform design, conduct and analysis of this observational cohort study and has been submitted with the report.

Methods

A multicentre observational, cohort study with prespecified feasibility criteria. We approached four sites to participate in this study: Salford Royal NHS Foundation Trust, Manchester University Hospitals NHS Foundation Trust, the Northern General Hospital in Sheffield and St Thomas’ NHS Foundation Trust. All sites agreed to participate, approve and deliver this feasibility study over a 12-month period from August 2020 to 2021 inclusive.

Population

We identified a consecutive, unselected cohort of general medical and surgical patients requiring hospitalisation at each site during the calendar year from 1 January to 31 December 2019. We chose 2019 because of concern that patients admitted during the COVID-19 pandemic might have been assessed for use of thromboprophylaxis differently to prior guidelines. We collated data on all risk assessments performed prospectively at the point of hospital admission, then scrambled episodes into randomly assorted batches of 50 (‘A’ batches) to ensure diversity in speciality presentation and mitigate seasonal bias. We excluded paediatric patients (age < 16), anyone requiring critical care admission (defined as level 2 care or above) and pregnant/post-partum patients due to differential risk, as outlined in the wider study protocol.

Study design

For each patient episode in each batch, we extracted basic speciality data using business intelligence and VTE risk assessments completed prospectively by clinical teams during the provision of routine care. Risk assessments were captured differently at each site, including the use of paper proformas, dichotomous output on electronic prescribing record, or through a detailed structured note within the electronic healthcare record. Example images/screenshots for each site can be found in Appendix 3, Figure 28. All four sites recommended the Department of Health tool in local guidelines, to facilitate initial VTE risk assessment.

Research assistants at each site were subsequently employed to undertake retrospective case note review for each patient episode through shared primary and secondary care electronic health records (EHRs). We extracted descriptive data on relevant clinical outcomes such as the subsequent diagnosis of HAT, major bleeding and CRNMB events as per internationally agreed definitions.79,80 Data extractors were trained in identification of these variables/outcomes and followed a detailed workflow diagram (see Appendix 3, Figure 29). Data abstractors were blinded to batch allocation, International Classification of Disease, Tenth Revision (ICD-10) coding/HAT database entries and the final analysis plan.

Following case note review, we collated multiple routine data sources for each patient episode to determine the accuracy of these methods in identifying relevant clinical outcomes, identified by research assistants. Data sources for interrogation were identified a priori and included the following: ICD-10 diagnostic codes judged relevant to thrombosis or bleeding by the project management group (PMG) (see Appendix 3, Table 30); Emergency Care Data Set/Systematised Nomenclature of Medicine (SNOMED) codes relevant to thrombosis or bleeding; and local HAT database entries. HAT database entries are co-ordinated by local thrombosis committees and include a contemporary register of all patients diagnosed with acute VTE (using radiology/pathology data), who are subsequently identified as HAT following local root cause analysis. All data sources were interrogated for the duration of hospital stay and up to 90 days post discharge, for each patient episode. Data sources were obtained through routine local business intelligence requests or direct approach to local coding teams. HAT database entries were obtained through local thrombosis committees, where feasible.

Given the potential for very low numbers of outcome events in the wider study population, we augmented the sample with likely positive thrombosis and bleeding cases. We obtained positive cases through ICD-10 coding identification for VTE events (V batches) and bleeding events (B batches) in accordance with the previously selected code list (see Appendix 3, Table 30) and local HAT database entries (H batches) within the relevant study period. Positive cases were scrambled, batched and reviewed in accordance with the general protocol.

The following feasibility criteria were proposed within our original protocol to determine success and accuracy of efficient data methods to identify relevant clinical outcomes:

1. ethical and regulatory approval for proposed methods
2. proportion of outcome events identified by routine data sources that are confirmed by record review (target 100%)
3. proportion of cases with no outcome event identified by routine data sources that have an event identified on record review (target 0%)
4. proportion of each RAM and individual risk variables collectible using efficient data methods (target 90%)
5. proportion of inpatients with data collected (target 90%).

In addition, we proposed using this sample of patients to estimate key parameters for any future implementation study, including the primary outcome event rates for HAT and major bleeding.

Statistical analysis

The accuracy of routinely recorded HAT and bleeding events was compared against direct case note review data for the cohort. Case note review determination of events was assumed to be the gold standard. Data are presented in contingency tables with sensitivity, specificity, positive and negative predictive values along with confidence intervals calculated using the Wilson score method.139

The primary analysis included patients identified from all sources (A, V, B or H batches) with bleeding and HAT assumed to have not occurred unless coded as such in the relevant data, or detected following case note review. In addition, two pre-planned sensitivity analyses were undertaken:

1. Inclusion was limited to cases identified in routine case review (i.e. ‘A batch patients’ only), with exclusion of all augmented sample cases.
2. Inclusion limited to participants for whom bleeding or HAT was definitively recorded.

We used case note review identification of HAT and bleeding events as the reference standard and report contingency tables for diagnostic test characteristics of all preidentified efficient data sources. We took a conservative approach and interpreted missing or unknown end points as ‘no event’.

The proposed sample size was reliant on an initial iterative developmental phase, including database refinement, user acceptability testing and site trial. We originally planned to identify 3000 inpatients across 4 hospitals during a 12-month study period within the 2-year project plan, dependent on appointment of research assistants and time required for outcome ascertainment. This sample size was designed to allow key parameters to be estimated with a high degree of precision across the whole cohort (standard error < 1%).

Ethical aspects

The study received a favourable opinion from the Proportionate Review Sub-committee of the London – West London & Gene Therapy Advisory Committee (GTAC) Research Ethics Committee (REC) and approval from the HRA and Care Research Wales (HCRW) on 18 September 2019 (reference 19/LO/1303, IRAS project ID 262220).

Participating sites identified members of the clinical care team (research nurses or assistants predominately) to access patient records and extract clinical data using a predesigned and protected Microsoft Excel database with embedded macro function, hosted at site. All data subsequently underwent local de-identification following completion and were exported to an independent team of statisticians at the Clinical Trials Research Unit (CTRU) in Sheffield, for collation and analysis. The REC approved this process and, as researchers outside the clinical care team only used anonymised data, Clinical Advisory Group approval was not required. We were able to access HES through the sites, and we were, therefore, able to undertake the original proposed analysis without involving NHS Digital.

All aspects of the data collection process, export, analysis and oversight were regularly reviewed by the internal PMG including CTRU representation, and an external Trial Steering Committee (TSC), throughout the duration of the project.

Patient and public involvement

Representatives of two PPI groups, Thrombosis UK and Sheffield Emergency Care Forum (SECF) joined the research team and were involved in developing the initial proposal and undertaking the wider study.

The SECF is a patient and public representative group with an interest in emergency care research. The forum has provided PPI for many emergency care research projects over the last 10 years.140 Thrombosis UK is a charity that aims to identify, inform and partner with the NHS, healthcare providers and individuals to work to improve prevention of VTE and the management and care of VTE events (see https://www.thrombosisuk.org/).

The PPI members were involved in determining the study design and ensuring that the proposal addressed the needs of patients and the NHS, while respecting the needs of potential participants. Their input regarding the importance of providing thromboprophylaxis for potential participants of any prospective cohort study and the need for such a study to yield reliable findings was instrumental in determining our approach to answering the research question. The PPI members also provided input at project management meetings and, where required, in day-to-day running of the project. The members used meetings and surveys of their wider PPI groups to enhance PPI in the project.

In relation to this particular substudy, the PPI members had specific roles in ensuring the acceptability of methods explored in the feasibility study. The efficient methods required to deliver an implementation study involve use of patient data without consent. We planned our methods so that they are compatible with ethical principles and data protection regulation, and would be acceptable to our PPI representatives. However, the feasibility study involved piloting these methods and the potential to adapt the methods to ensure efficiency. The PPI representatives were involved in project management, monitoring of data collection and specifically considered the acceptability of any proposed changes to the methodology throughout the project.

Results

Baseline sampling and augmentation

Over the 12-month feasibility study period we successfully set up and delivered the project across four sites. Patients (2115) were identified and selected for analysis with an original hospital admission occurring in the calendar year 2019. Of these, 107 patient episodes were ineligible due to being pregnant or post-partum women (n = 49); admitted to a critical care environment of level 2 or above (n = 38); children aged under 16 (n = 13) or for unrecorded reasons (n = 7) leaving 2008 episodes for analysis. All episodes were suitable for data extraction and comparison to routine data sources. Ethical and regulatory approvals were secured for all sites. All sites failed to meet their sample target of 750 for reasons mainly related to the SARS-CoV-2 pandemic, including redeployment of research staff to clinical care, delayed local approvals secondary to prioritised pandemic research and a longer than anticipated time for individual outcome ascertainment per case review. Data were missing or the outcome unknown for fewer than 2% of participants for each outcome/site.

Patient episodes showed an even balance of medical and surgical cases, but with a focus on emergency (73.7%) rather than elective (25.8%) admissions. A broad range of subspeciality interests were represented within the cohort. Median length of stay was 3 days [interquartile range (IQR) 3–8] and mean length of stay 7.75 days [standard deviation (SD) 16.5]. Speciality groups with frequencies and percentages are shown in Table 13. The vast majority of patient episodes (1809, 90.1%) were taken from ‘A’ batches, through patients identified as having a VTE risk assessment. The total sample was augmented by 45 (2.2%) patients with potential bleeding events (one B batch) and 154 (7.7%) patients with potential VTE events (one V and two H batches). Sites contributed evenly to the sample with the exception of Manchester; numbers at this site reflect a significant delay to institutional approval during the pandemic, with this research deprioritised. Site and batch numbers are shown in Table 14.

Accuracy of efficient data methods compared to case note review for ascertainment of key clinical outcomes

The accuracy of routine data sources for HAT and major bleeding events compared to case note review are shown in Table 15. The proportion of outcome events identified by routine data sources that were confirmed by case note review was 71% (95% CI 63% to 79%). The proportion of cases with no HAT outcome event identified by routine data sources that had an event identified on record review (target 0%) was 3% (95% CI 2 to 4) for ICD-10/SNOMED coding and 2% (95% CI 1 to 2) for local HAT database entries.

The sensitivity was 62% (95% CI 54% to 69%) for the use of ICD-10/SNOMED coding data to detect HAT events and 81% (95% CI 75% to 87%) for local HAT database entries. Sensitivity by individual site ranged from 45% (95% CI 28% to 63%) to 72% (95% CI 61% to 82%) using ICD-10/SNOMED coding and 68% (95% CI 51% to 84%) to 94% (95% CI 87% to 100%) using local HAT database entries. The sensitivity of ICD-10/SNOMED coding to detect major bleeding events identified by case note review was 38% (95% CI 27% to 50%). Sensitivity by individual site ranged from 22% (95% CI 0% to 49%) to 56% (95% CI 37% to 75%).

These sensitivity values imply moderate rates of missed HAT and major bleeding events through the use of efficient data methods in isolation. Specificity values were > 90% for both outcomes at all sites, implying reasonable accuracy when an event was identified by efficient data methods.

Sensitivity analyses and estimation of key parameters for future implementation work

A sensitivity analysis was conducted using only patient episodes obtained through ‘A’ batches, to remove augmentation of the sample and mitigate bias. The sensitivity of efficient data methods to detect key outcomes identified at case note review remained poor. These results are summarised in Table 16. This analysis also facilitated estimation of key parameters for any future implementation study. We found the HAT event rate on case note review to be 29/1809 (1.6%, 95% CI 1.0 to 2.2) and the major bleeding event rate to be 45/1809 (2.5%, 95% CI 1.8 to 3.2) within this large cohort of hospitalised patients receiving risk assessment and thromboprophylaxis in the context of routine care.

Key feasibility outcomes

Regarding key feasibility outcomes, we achieved ethical and regulatory approvals and the design of the study was acceptable to all sites. The proportion of HAT events identified by routine data sources that were confirmed by record review (target 100%) was 71% (95% CI 63% to 79%) for ICD-10/SNOMED coding and 100% (95% CI 97% to 100%) for local HAT database entries. The proportion of cases with no HAT outcome event identified by routine data sources that had an event identified on record review (target 0%) was 3% (95% CI 2% to 4%) for ICD-10/SNOMED coding and 2% (95% CI 1% to 2%) for local HAT database entries. The proportion of outcome major bleeding events identified by routine data sources that were confirmed by record review (target 100%) was 20% (95% CI 13% to 27%) for ICD-10/SNOMED coding. The proportion of cases with no major bleeding outcome event identified by routine data sources that have an event identified on record review (target 0%) was 2% (95% CI 1% to 3%) for ICD-10/SNOMED coding.

We were unable to use efficient data methods to collect individual risk variables in three out of four sites, due to use of either paper proformas or dichotomous risk assessment without granular data. As such we did not meet the feasibility target of using routine data to collect individual variables for 90% of patients. We were able to collect outcome data in 1745/2008 (87%) of inpatients (target 90%). This was mainly due to inability at the Manchester site to access the local HAT database. Excluding this issue, the other three sites all managed to collect relevant outcome data in > 98% of patient episodes.

Summary and key findings

• In this multicentre cohort study, routine and efficient data methods used to identify HAT events and major bleeding events performed poorly when compared to direct case note review by trained research staff.
• We were able to demonstrate ethical and regulatory approvals for our chosen design across four NHS sites.
• We were unable to demonstrate overall feasibility of an efficient design and did not meet several prespecified criteria set out in the original research proposal: Proportion of outcome events identified by routine data sources that are confirmed by record review (target 100%); VTE 71%, bleeding 20%: Criteria not met.Proportion of cases with no outcome event identified by routine data sources that have an event identified on record review (target 0%); VTE 3%, bleeding 2%: Criteria not met.Proportion of inpatients with data collected (target 90%); 87% across all sites, 98% across three sites: Partially met.Proportion of each predictor variable recorded (target 90%); only routinely recorded at one site: Criteria not met.Proportion of each risk assessment tool completed using available data (target 90%); only routinely recorded at one site: Criteria not met.
• These findings suggest a future data-enabled study using efficient methods would significantly under-report events, leading to inaccurate RAM evaluation.
• In this multicentre cohort of hospitalised inpatients, the HAT event rate was 1.6% (95% CI 1.0% to 2.2%) and the major bleeding event rate was 2.5% (95% CI 1.8% to 3.2%).