Workstream 1 – systematic review

A systematic review of the literature was undertaken to determine the comparative accuracy of individual RAMs for predicting the risk of developing VTE in hospital inpatients. Sections of this chapter have been reproduced from Pandor et al. under licence CC BY 4.0.17

This review was undertaken in accordance with the general principles recommended in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement18 and was registered on the International Prospective Register of Systematic Reviews (PROSPERO) database (CRD42020165778).

Methods

Eligibility criteria

We sought studies evaluating RAMs which could be applied to a general inpatient population (medical, surgical or trauma), rather than disease-specific models. All primary validation studies (as derivation studies may give an overoptimistic assessment of model performance measures) that evaluated the accuracy (e.g. sensitivity, specificity, concordance statistic) of a multivariable RAM (or scoring system) for predicting the risk of developing VTE were eligible for inclusion. We selected studies that included validation of the model in a group of patients that were not involved in model derivation. This involved either splitting the study cohort (internal) or using a new cohort (external). The study could have reported derivation of the model, but we only used the validation data to estimate accuracy. The study population consisted of hospital inpatients including those who required medical care, undergoing any surgery (excluding day surgery) or received care following an injury. Studies that primarily focused on children (aged under 16 years), women admitted to hospital for pregnancy-related reasons and any patient admitted to a level 2 or above critical care environment (e.g. patients requiring more detailed observation or intervention including support for a single failing organ system or postoperative care and those ‘stepping down’ from higher levels of care) were excluded. These patient groups have VTE risk profiles that differ markedly from the general inpatient population, making the use of a generic model inappropriate.

Data sources and searches

Potentially relevant studies were identified by searching the following electronic databases and research registers:

• Ovid MEDLINE^® Epub Ahead of Print, In-process and Other Non-indexed Citations, Ovid MEDLINE^® Daily, MEDLINE and Versions^® (OvidSP) 1946 to February 2021
• EMBASE (OvidSP) 1974 to February 2021
• Cochrane Database of Systematic Reviews (www.cochranelibrary.com/) 1996 to February 2021
• Cochrane Central Register of Controlled Trials (www.cochranelibrary.com/) 1898 to April 2017
• ClinicalTrials.gov (US NIH) 2000 to February 2021
• International Clinical Trials Registry Platform (WHO) 1990 to February 2021

The search strategy used free text and thesaurus terms and combined synonyms relating to the condition (e.g. VTE in medical inpatients) with risk prediction modelling terms. No language restrictions were used. However, as the current review updated three previous systematic reviews,8,9,10 searches were limited by date from 2017 (last search date from earlier reviews)8 to February 2021. Searches were supplemented by hand-searching the reference lists of all relevant studies (including existing systematic reviews); forward citation searching of included studies (using the Web of Science Citation Index Expanded and Conference Proceedings Citation Index – Science) to identify articles that cite the relevant articles; contacting key experts in the field; and undertaking targeted searches of the World Wide Web using the Google search engine. Further details on the search strategy can be found in Appendix 1.

All identified citations from the electronic searches and other resources were imported into and managed using the EndNote bibliographic software (version X8; Clarivate Analytics, Philadelphia, PA, USA).

Study selection process

The inclusion of potentially relevant articles was undertaken using a two-step process. First, all titles were examined for inclusion by one reviewer Katie Sworn (KS) and any citations that clearly did not meet the inclusion criteria (e.g. non-human, unrelated to VTE inpatients) were excluded. Second, all abstracts and full-text articles were then examined independently by two reviewers (KS and AP). Any disagreements in the selection process were resolved through discussion or if necessary, arbitration by a third reviewer (SG) and included by consensus.

Data abstraction and quality assessment strategy

Data relating to study design, methodological quality and outcomes were extracted by one reviewer (KS) into a standardised data extraction form and independently checked for accuracy by a second [AP or Michael Tonkins (MT)]. Any discrepancies were resolved through discussion or if necessary, arbitration by a third reviewer (SG) and included by consensus. Where multiple publications of the same study were identified, data were extracted and reported as a single study.

The methodological quality of each included study was assessed using PROBAST (Prediction model Risk Of Bias Assessment Tool).19,20 This instrument evaluates four key domains: patient selection, predictors, outcome and analysis. Each domain is assessed in terms of risk of bias and the concern regarding applicability to the review (first three domains only). To guide the overall domain-level judgement about whether a study is at high, low or an unclear (in the event of insufficient data in the publication to answer the corresponding question) risk of bias, subdomains within each domain include a number of signalling questions to help judge with bias and applicability concerns. An overall risk of bias for each individual study was defined as low risk when all domains were judged as low; and high risk of bias when one or more domains were considered as high. Studies were assigned an unclear risk of bias if one or more domains were unclear and all other domains were low. Further details on the PROBAST tool can be found in Appendix 1, Table 20.

The methodological quality of each included study was independently evaluated by two reviewers (KS and AP). Any discrepancies were resolved through discussion or if necessary, with involvement of a third reviewer (MT). Blinding of the quality assessor to author, institution or journal was not considered necessary.

Data synthesis and analysis

We were unable to perform meta-analysis due to significant levels of heterogeneity between studies (participants, inclusion criteria, clinical condition) and variable reporting of items. As a result, a pre-specified narrative synthesis approach21,22 was undertaken, with data being summarised in tables with accompanying narrative summaries that included a description of the included variables, statistical methods and performance measures [e.g. sensitivity, specificity and C-statistic (a value between 0.7–0.8 and > 0.8 indicated good and excellent discrimination, respectively; and values < 0.7 were considered weak23)], where applicable. All analyses were conducted using Microsoft Excel 2010 (Microsoft Corporation, Redmond, WA, USA).

Patient and public involvement

Patients and the public were not involved in the design of this systematic review. However, all patient representatives reviewed the most widely validated risk models identified, to comment on the appropriateness of criteria, acceptability and potential for face-to-face and remote (patient-led) completion.

Results

Quantity and quality of research available

The literature searches identified 6355 citations. Of these, 51 studies15,24–73 investigating 24 unique RAMs met the inclusion criteria. A flow chart describing the process of identifying relevant literature can be found in Figure 1. A total of 60 full-text articles were excluded as they did not meet all the pre-specified inclusion criteria. The majority of the articles were excluded primarily for not using a RAM for predicting the risk of developing VTE, having no useable or relevant outcome data or an inappropriate study design (e.g. derivation study, reviews, commentaries or editorials). A full list of excluded studies with reasons for exclusion is presented in Report Supplementary Material 1.

FIGURE 1 Study flow chart (adapted from free online resources hosted by the EQUATOR network https://www.equator-network.org). (Figure not reproduced here; see the original article.)

Description of included studies (design and patient characteristics)

The design and participant characteristics of the 51 included studies15,24–73 that provided data on the comparative accuracy of RAMs for predicting VTE in patients admitted to hospital is summarised in Table 1. All studies were published between 2003 and 2020 and were undertaken in North America (n = 24),26,27,35–42,45,49,50,54–61,67,70,71 Asia (n = 13),31,32,44,46–48,62–65,69,72,73 Europe (n = 9),15,25,28–30,33,51,53,68 the Middle-East (n = 2),24,65 South America (n = 1),34 Australia (n = 1)43 and one study was intercontinental.52 Sample sizes ranged from 7042 to 1,099,09345 patients in 37 observational cohort studies [11 prospective15,24,25,30,31,34,46,53,54 (5 of which were multicentre) and 26 retrospective26–29,35,36,38–43,45,48,51,52,55–57,60,61,64,65,67,70,71 (16 of which were multicentre) in design]. Sample sizes in 14 case–control studies32,33,37,44,47,49,50,59,62,63,68,69,72,73 (4 of which were multicentre) ranged from 14863 to 19,21759 patients.

The vast majority of studies evaluated VTE risk assessment in hospital inpatients who required medical care (n = 21),15,28–30,33,34,38,39,47,49,51,53,59,60,63,68,69,71–73 were undergoing surgery (n = 15)26,27,35,37,40,42,45,48,50,54,58,61,65,67,70 or were a mixed medical and surgical cohort (n = 4).25,31,32,36 The remaining studies focused on patients receiving care for trauma (n = 4),41,43,57,64 cancer (n = 4),24,44,56,62 stroke (n = 1),46 burn injuries (n = 1)55 and sepsis (n = 1).66 The mean age ranged from 45 years41 to 76 years52 (not reported in 29 studies)15,25–27,32,33,35,37,40,42–47,49,50,54,57–60,63,64,67,68,72,73 and the proportion of female subjects ranged from 17%42 to 81%61 (not reported in 12 studies).25–27,35,37,45,50,54,57,58,60,63

The majority of studies (n = 37)15,24,26,28–34,38,40,42–49,51,54,57–60,62,64–66,68,69,72,73 defined the VTE end point [deep-vein thrombosis (DVT) and or pulmonary embolism (PE)] as being objectively confirmed. Of the remainder, three studies36,56,71 had no objective confirmation of VTE and 11 studies25,27,35,37,41,50,55,61,63,67,68 did not report the methods for diagnosis confirmation. In terms of VTE risk period, half of the studies (n = 23)15,24–27,28,30,37–40,42,45,46,49,51–54,58,59,61,71 utilised the RAMs to predict the occurrence of VTE within 3 months of the index hospitalisation. The remaining studies did not report the VTE risk period. The reported incidence of VTE ranged widely from 0.3%39,70 to 27.9%48 depending on definition, study design and study participants (e.g. medical, surgical or trauma).

The studies included in this review evaluated 24 validated unique RAMs. The most widely evaluated models were the Caprini RAM (22 studies),24,26,31–35,37,38,40,42,44,45,47,48,50,51,61,62,65,72,73 Padua Prediction Score (16 studies),15,29,30,32,33,36,39,47,50,51,65,66,68,69,72,74 International Medical Prevention Registry on Venous Thromboembolism (IMPROVE) models (8 studies),29,30,33,39,49,51,52,59 the Geneva risk score (4 studies)28,29,30,33 and the Kucher score (4 studies).33,39,68,71 A summary of their associated characteristics and composite clinical variables is provided in Appendix 1, Table 21. Our patient and public involvement (PPI) group reviewed the most widely validated RAMs and identified all as appropriate and acceptable, in the context of face-to-face risk assessment. They did not feel patient-led completion of any RAM in isolation, to be feasible.

Statistical methods varied significantly between studies. Most studies reported the discrimination of the RAMs using a combination of the C-statistic and sensitivity or specificity.

Thromboprophylaxis was employed in about half (n = 25) of the studies, with the proportion receiving thromboprophylaxis ranging from 3.8% to 100%. It was not employed in three studies, and 23 studies did not report on thromboprophylaxis use. The use of thromboprophylaxis may lead to underestimation of predictive accuracy if a given RAM were to predict VTE events that were subsequently prevented by thromboprophylaxis.

Risk of bias and applicability assessments of included studies

The overall methodological quality of the 51 included studies15,24–73 is summarised in Table 2 and Figures 2 and 3. The methodological quality of the included studies was variable, with most studies having high or unclear risk of bias in at least one item of the PROBAST tool. The main sources of potential bias were related to the following domains: (1) patient selection factors, such as retrospective data collection, incomplete patient enrolment or unclear criteria for patients receiving VTE prophylaxis; (2) predictor and outcome bias arising from inappropriate inclusion of predictors within RAMs, unclear methods of outcome definition, low event rates and missing predictor or outcome data; (3) analysis factors, such as small sample sizes, inappropriate handling of missing data and failure in reporting relevant performance measures such as calibration.

FIGURE 2 PROBAST assessment summary graph – review authors’ judgements on risk of bias. (Figure not reproduced here; see the original article.)

FIGURE 3 PROBAST assessment summary graph – review authors’ judgements on applicability. (Figure not reproduced here; see the original article.)

Assessment of applicability to the review question led to the majority of studies being classed either as high (n = 35)24,25,31,32,34,36,37,40–51,54–57,61–70,72,73 or unclear (n = 12)15,26,29,30,33,35,52,53,58,60,71 risk of inapplicability. These assessments were generally related to patient selection (highly selected study populations, e.g. single pathologies, single site settings), predictors (inconsistency in definition, assessment or timing of predictors) and outcome determination.

Quantitative data synthesis (summary of results)

As there were a reasonable number of studies to compare, a summary of the C-statistics for studies involving medical, surgical and trauma patients, respectively, is presented in Figure 4, with the results grouped by RAM. Results of other hospital inpatients are presented in Appendix 1, Table 22. C-statistics varied markedly between these studies and between models, with no RAM performing obviously better than other models. In studies evaluating a single model, C-statistics23 were sometimes weak (< 0.7; 10 studies with 17 data points), often good (0.7–0.8; 17 studies with 20 data points) and a few were excellent (> 0.8; 5 studies with 5 data points). There was marked heterogeneity between multiple studies evaluating the same model. Studies evaluating multiple (more than 3) models33,39 tended to report weak accuracy across all the models (C-statistic < 0.7; 2 studies with 16 data points).

FIGURE 4 C-statistics by model for studies involving (a) medical, (b) surgical and (c) trauma inpatients. ACS NSQIP, American College of Surgeons National Surgical Quality Improvement Program; CI, confidence interval; NR, not reported; RAP, risk assessment profile. (Figure not reproduced here; see the original article.)

Table 3 shows the sensitivity and specificity at various thresholds for studies involving medical, surgical and trauma patients, respectively, with the results grouped by RAM. Interpretation was again limited by marked heterogeneity, which was exacerbated when different thresholds were reported by different studies evaluating the same model. Model accuracy was generally poor, with high sensitivity usually reflecting a threshold effect, as evidenced by corresponding low specificity (and vice versa).

Summary of key findings

• A number of RAMs have been evaluated for potential VTE risk stratification of hospitalised adult patients.
• In general, external validation studies have poor designs and limited generalisability.
• Available data suggest that RAMs have generally weak predictive accuracy for VTE.
• Use of VTE prophylaxis varied across the study populations and was not reported for many studies.
• There is insufficient evidence and too much heterogeneity to recommend the use of any particular RAM for predicting the risk of developing VTE in hospital inpatients.