Assessment of clinical effectiveness

Systematic review methods

Identification of studies

Comprehensive electronic searches were conducted to identify relevant reports of published studies. Highly sensitive search strategies were developed, including index terms, free-text words, abbreviations and synonyms, to combine biomarkers and AKI. The electronic databases MEDLINE (via Ovid), EMBASE (via Ovid), Web of Science Core Collection, Health Technology Assessment (HTA) Database, Cumulative Index to Nursing and Allied Health Literature, and Cochrane Central Register of Controlled Trials were searched, with no restriction on date or publication type. Full details of the search strategies are presented in Appendix 1. The searches were undertaken between 17 May and 10 June 2019.

In addition, we searched the following sources for ongoing or unpublished studies: ClinicalTrials.gov (www.clinicaltrials.gov/), the World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP) search portal (https://apps.who.int/trialsearch/) and the WHO Global Index Medicus (www.who.int/library/about/The_Global_Index_Medicus/en/) (all of these websites were accessed on 10 June 2019). Furthermore, websites of relevant professional organisations and health technology agencies, as well as appropriate clinical experts, were consulted to obtain any additional potentially relevant reports. The reference lists of all included studies were perused to identify further potentially relevant studies. We also considered evidence provided by the manufacturers of the biomarkers included in this assessment (i.e. Astute Medical, Inc.; Abbott Laboratories; and BioPorto Diagnostics A/S).

Inclusion and exclusion criteria

Inclusion and exclusion criteria for each of the clinical effectiveness questions considered in this assessment are summarised in Table 2. Only those studies that fulfilled these criteria were deemed suitable for inclusion.

Study selection and data extraction

A screening checklist was developed to assist study selection and data extraction (see Appendix 2, Figure 25). The data extraction form is provided in Appendix 3. One reviewer (CR) screened the titles and abstracts identified by the search strategies for inclusion or exclusion. A second reviewer (MI) double checked all non-selected citations. As a lot of relevant information was not available from the titles or abstracts of the reports identified by the literature searches (e.g. information about the immunoassay used and type of analyses), our selection approach was overinclusive. Full-text copies of all potentially relevant reports were retrieved and assessed for inclusion by one reviewer (MAM, MI or CR). A second reviewer (MAM, MI or CR) double checked 20% of the reports. Any disagreement was resolved by discussion or referred to a third reviewer (MB).

One reviewer (MAM, MB, MI, AP or CR) extracted data from each eligible study using a form developed and piloted for the purpose of this assessment. If multiple publications of the same cohort of participants were identified, the publication with the most complete or suitable data set was considered as the primary source of information. Any uncertainty related to the data extraction process was discussed among reviewers and resolved by consensus.

From each study, the following data were extracted:

• characteristics of studies – first author, year of publication, study centre, country, inclusion and exclusion criteria, method of participant enrolment
• characteristics of study participants – age, sex, target condition, setting, number of participants enrolled, number of participants analysed, number excluded from analysis, main reasons for exclusion
• characteristics of the biomarkers (e.g. manufacturer, detection method, threshold, timing of the measurement)
• characteristics of the reference standard (i.e. creatinine and urine output criteria for AKI)
• outcome data –
  • data on the diagnostic performance of the biomarkers for detection of AKI [absolute number of true-positive, false-positive, false-negative and true-negative cases; sensitivity and specificity values; area under the curve (AUC) calculated from the receiver operating characteristic (ROC) plot]
  • data on the prediction of development of AKI, worsening of AKI, mortality, need for RRT and CKD, as provided by the study authors [e.g. AUC values, odds ratio (OR) or hazard ratio (HR), duration of follow-up]
  • data on the clinical utility of the biomarkers (impact of the use of the biomarkers on clinical outcomes), as reported by study authors [e.g. number of events and number of participants for each relevant binary outcome; mean, standard deviation (SD) and number of participants for each relevant continuous outcome].

Assessment of risk of bias

Validated tools were used to assess the risk of bias of the included studies according to their study design. We used the Quality Assessment of Diagnostic Accuracy Studies, version 2 (QUADAS-2) tool21 to assess the risk of bias of studies assessing the diagnostic and prognostic accuracy of the biomarkers under investigation. The QUADAS-2 tool consists of four domains: patient selection, index test, reference standard, and flow and timing. Each domain is assessed in terms of having a ‘low’, ‘high’ or ‘unclear’ risk of bias, and the first three are also assessed in terms of concern regarding ‘low’, ‘high’ or ‘unclear’ applicability.

We used the Prediction model Risk Of Bias ASsessment Tool (PROBAST),22 which is structured into four domains (participants, predictors, outcome and analysis) to assess the risk of bias and applicability of prediction model studies.

A single reviewer (MAM, MB, MI, AP or CR) assessed the risk of bias of each of the included studies. Any uncertainty was discussed among reviewers and resolved by consensus.

No other types of study design were identified.

Data synthesis and analysis

For each assay, for each study, we calculated sensitivity, specificity and prevalence values from the reported numbers of true-positive, false-positive, false-negative and true-negative cases. If studies did not provide 2 × 2 data, these were derived from the sensitivity and specificity estimates, if given. We entered diagnostic data into Review Manager software (RevMan version 5.3, The Cochrane Collaboration, The Nordic Cochrane Centre, Copenhagen, Denmark) to produce forest plots of sensitivity and specificity estimates, together with their 95% CIs.

When appropriate, we performed a meta-analysis of each pair of sensitivity and specificity estimates from each included study for each relevant assay. As reported threshold levels for a positive test differed across studies, we conducted random-effects meta-analyses using the hierarchical summary receiver operating characteristic (SROC) model23,24 implemented in Stata^® (metandi command)25 (StataCorp LP, College Station, TX, USA) to estimate summary values for sensitivity and specificity. The model takes into account both of these measures of accuracy and their correlation, assumes that accuracy and thresholds vary between studies, and incorporates both within- and between-study variability. We constructed a SROC plot using the hierarchical model, produced sensitivity and specificity summary estimates, and hence a summary operating point, and calculated the 95% confidence and prediction regions. In accordance with the Stata requirements, we performed meta-analyses when data from four or more studies were available. For studies that reported multiple thresholds, we selected only one threshold to be included in the analysis. We performed separate meta-analyses for each biomarker, clinical setting, mode of sampling (urine or plasma) and type of patient population (adults or children). To inform the economic model, we also performed separate meta-analyses for each biomarker across all clinical settings.

For each biomarker, heterogeneity was assessed by visual inspection of the forest plots of sensitivity and specificity estimates and of the size of the prediction region in the SROC curve plots.

When possible, we performed meta-analyses of AUC values using a random-effects model to measure the performance of each biomarker for the prediction of each relevant outcome (i.e. AKI, mortality, RRT and CKD). We assessed the proportion of between-study variation in the area under ROC curve due to heterogeneity, rather than sample error, using the prediction interval. We considered an AUC of > 0.70 as indicative of a useful risk predictor.

Stata version 15.0 was used for all statistical analyses. Graphs were made using either Stata or RevMan version 5.3.

Patient and public involvement

There was no patient and public involvement in this study. The study was conducted as part of the NICE Diagnostics Assessment Programme (DAP). We did not deem it necessary to involve further patient representatives and laypeople as a range of stakeholders, including members of the public and national groups representing patients and/or their carers, are already involved as participants in the DAP process for each individual assessment.

Results of the assessment of clinical effectiveness

Results of the literature searches

The literature searches identified 6379 records; 86 additional records were identified in either trial registers (i.e. EU Clinical Trials Register, ICTRP, ClinicalTrials.gov) or other literature collections (i.e. HTA Database, WHO Global Index Medicus), for a total of 6465 retrieved records. After de-duplication, 2348 records were screened for relevance. Of these, 1050 were considered to be potentially relevant and were selected for full-text assessment. Four articles could not be obtained. Of the 1046 records retrieved and assessed in depth, 71 met the inclusion criteria. After excluding secondary or multiple publications, we selected 56 studies for inclusion in the systematic review of effectiveness. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram in Figure 1 shows the flow of studies through the selection process. The bibliographic details of the studies retrieved for full-text assessment and subsequently excluded, together with the main reasons for their exclusion, are presented in Appendix 5, Table 26.

FIGURE 1 The PRISMA flow diagram of selected studies. (Figure not reproduced here; see the original article.)

Overview of included studies

A list of all included studies can be found in Appendix 4. General characteristics of the 56 included studies and their associated references are provided in Table 3 for the adult population and in Table 4 for the child population. Further study characteristics are provided in Appendix 6, Table 27. The majority of studies were cohort studies. In 46 studies, data were collected prospectively;17–30,34,37,45–48,50–54,56–59,61–63,65–70,72–75,77–81,83,84,86–91,93 in one study, data were collected prospectively but analysed retrospectively;26 in one study, data were collected retrospectively;82 and in eight studies information on data collection was unclear.32,33,35,49,55,60,64,92 Fifty-three studies provided suitable data on the use of the biomarkers for detection or prediction of AKI in critically ill patients admitted to hospital,26–30,32–35,37,45–56,58–66,68–70,72–75,77–84,87–93 11 studies provided suitable data for prediction of mortality in critically ill patients at risk of AKI46,49,53,59,62,70,72,75,82,83,87 and four studies provided suitable data for prediction of need for RRT.46,72,75,87 No studies provided suitable data for prediction of CKD.

No randomised controlled trials (RCTs) or controlled clinical trials were identified; no studies provided data on the incremental value of the use of the biomarkers compared with standard clinical care.

Of the 56 included studies, 36 involved a single centre26–28,30,35,45,46,48–50,53–55,58–61,63,66,68–70,73,74,79,80,82,83,86–93 and 13 involved multiple centres.29,33,34,37,52,56,57,65,72,77,78,81,84 Seven studies did not provide this information.32,47,51,62,64,67,75 Twenty-eight studies were conducted in Europe (four in the UK, six in Germany, three in Italy, three in Spain, two in Greece, two in Denmark, one in the Netherlands, one in France, one in Belgium, one in France and Belgium, one in Finland, one in Norway, one in Switzerland and one in several European countries); 15 in North America (12 in the USA, two in the USA and Canada and one in Canada); nine in Asia (three in Japan, three in the Republic of Korea, two in Thailand and one in China); two in North America and Europe; and one in Australia. One study did not provide clear information on the geographical location.

NGAL was the most commonly studied biomarker (41/56 studies; 37 studies used urine NGAL assays and four used plasma NGAL assays). NephroCheck was assessed in eight studies. Seven studies provided data on more than one assay (six studies on urine NGAL and plasma NGAL assays and one study on NephroCheck, urine NGAL and plasma NGAL assays). Among the NGAL studies, 24 used the ARCHITECT urine NGAL platform and 20 used the BioPorto urine NGAL assay. All 11 plasma NGAL studies used the BioPorto Diagnostics assay. No studies used the NGAL Alinity i platform.

Of the 56 included studies, 46 enrolled adults only, eight enrolled children only and two enrolled both adults and children. The total number of participants was 17,967, of whom 16,247 were adults (average age ranged from 49 to 77 years) and 1720 were children (average age ranged from 1 day to 5 years). Of the 46 studies that focused on adults only, 12 assessed patients after cardiac surgery, four assessed patients requiring non-surgical cardiac care, one assessed patients undergoing major abdominal surgery, one assessed patients undergoing hepatobiliary surgery, 16 assessed patients admitted to intensive care units (ICUs), five assessed patients with liver disease (mainly cirrhosis), two assessed patients with sepsis, two assessed patients with CKD and three assessed patients admitted to EDs. Of the eight studies that focused on children, six assessed children (including neonates) undergoing cardiac surgery and two assessed children admitted to a paediatric ICU or neonatal ICU. The two studies that included both adults and children assessed patients undergoing cardiac surgery. For the purpose of the clinical effectiveness and cost-effectiveness analyses, the participants were grouped into three categories according to the clinical setting reported in the included studies: patients undergoing cardiac surgery, patients undergoing major non-cardiac surgery and patients admitted to critical care (mixed patient population). The critical care group includes critically ill patients presenting to the ED and participants admitted to the ICU or considered for critical care for various medical conditions or after surgery (but the studies did not specify which type of surgery or provide separate results for medical and surgical ICU participants).

Study quality

The risk of bias of studies assessing the accuracy of NephroCheck and NGAL assays in identifying people at risk of developing AKI was assessed using the QUADAS-2 tool. The results are summarised in Figure 2 and in Appendix 7, Table 28.

FIGURE 2 Risk-of-bias assessment of studies assessing the diagnostic performance of the biomarkers using the QUADAS-2 tool. (Figure not reproduced here; see the original article.)

Eleven studies (20%) did not report sufficient information to determine whether or not a selection of patients could have introduced bias; these studies were assessed as having an unclear risk of bias.32,33,35,49,50,54,55,60,72,82,94 The remaining studies were judged to be at low risk of bias for the patient selection domain.

The main potential source of bias across studies relates to blinding. Most studies (98%) were assessed as having an unclear risk of bias for the conduct and interpretation of the index test because of insufficient information or lack of clarity regarding whether or not the biomarker results were interpreted without knowledge of the reference standard results (see Figure 2). The studies that used NephroCheck were judged to be at low risk of bias with regard to the interpretation of the test because all of them used a common threshold. However, because of the differences in threshold level observed across studies and the lack of a common threshold, the risk of bias for NGAL studies was judged to be unclear. Although some studies alluded to the blinding to patients’ clinical information of personnel performing the biomarker measurements, it was unclear whether or not the personnel were indeed blinded to serum creatinine measurements (reference standard). With regard to whether or not the reference standard, its conduct or its interpretation may have introduced bias, two studies (4%) were judged to be at unclear risk of bias because baseline serum creatinine levels were determined by reviewing records of previous 12-month measurements.54,56 The remaining studies (96%) were judged to be at low risk of bias for the reference standard domain.

Two studies (4%) were judged to be at high risk of bias in terms of the patient flow (e.g. attrition) because > 50% of the participants were excluded from the analysis62 or because the reporting of the patient selection and flow was poorly detailed.50 Four studies (7%) were at unclear risk.54,57,64,92 The remaining studies (89%) were considered to be at low risk of bias regarding the patient flow domain.

Across studies, there was no major concern that the patient population and the conduct and interpretation of reference standard were not applicable to the review question. We observed an expected variation between studies in terms of characteristics of the index tests (biomarker assays) and clinical protocols. In particular, applicability of the index test to the review question was judged to be unclear in many studies, mainly because of the variation with regard to the biomarker thresholds and timing of sample collections.

The risk of bias of studies assessing the role of NephroCheck and NGAL assays for prediction of relevant clinical outcomes (i.e. worsening of AKI, mortality and need for RRT) was assessed using the PROBAST tool.22 The results are summarised in Table 5.

Twelve prediction studies were assessed for risk of bias and applicability.32,46,49,53,55,59,70,72,75,82,83,87 Three of these studies (25%) reported insufficient information to determine whether or not selection of patients could have introduced bias; these studies were judged to be at unclear risk of bias.32,55,72 The remaining studies were judged to be at low risk of bias for this domain. No studies were judged to have made predictor assessments without the knowledge of outcome data; therefore, the risk of bias for the predictors domain was judged to be unclear for all studies. The risk of bias in the outcome domain was rated as unclear for all studies, mainly because of inadequate information to assess whether or not outcomes were determined without knowledge of predictor information. The risk of bias for the analysis domain was rated as unclear in 58% of studies and as high in 42% of studies.

The overall risk of bias was considered to be unclear for most studies (70%), mainly because these studies were assessed as being at high risk of bias in the analysis domain. The remaining studies were judged to be at unclear risk of bias.

Most studies were judged to be at low risk of bias for applicability to the review question in each of the domain categories. Overall, applicability was judged to be at low risk of bias for 75% of the studies and at unclear risk of bias for the remaining studies. In general, there was no major concern that the studies were not applicable to the research questions of this assessment. Summaries of the results are shown in Figures 3 and 4, and individual study-level results are presented in Appendix 8, Table 29.

FIGURE 3 Risk-of-bias assessment of studies that assessed the role of biomarkers for prediction of relevant clinical outcomes using the PROBAST tool. (Figure not reproduced here; see the original article.)

FIGURE 4 Applicability of prediction studies to the research questions using the PROBAST tool. (Figure not reproduced here; see the original article.)

Accuracy of the NephroCheck and neutrophil gelatinase-associated lipocalin assays for identifying acute kidney injury

We were able to extract or derive 2 × 2 data from 33 studies that assessed the performance of the NephroCheck, ARCHITECT urine NGAL and BioPorto urine and plasma NGAL assays for identifying AKI in critically ill hospitalised patients. These studies are summarised in the following sections.

The summary estimates of accuracy and the SROC plots are provided separately for each assay, clinical setting, mode of sampling and type of patient population (adults and children). We also present analyses across all settings. Studies that could not be combined in a meta-analysis (fewer than four) are summarised narratively.

NephroCheck urine assay: adult population

A summary of the diagnostic data for the seven studies26,27,29,30,33,34,36 that assessed the use of NephroCheck for detection of AKI in adults is presented in Table 5.

Cardiac surgery

Two studies26,27 assessed the use of NephroCheck for detection of AKI in patients after cardiac surgery (data from a total of 500 participants were available for the analyses). Both studies used the same cut-off point (0.3 ng/ml²/1000). The study by Cummings et al.26 assessed a total of 400 cardiac patients soon after ICU admission. The sensitivity and specificity values were 0.31 (95% CI 0.09 to 0.61) and 0.78 (95% CI 0.74 to 0.82), respectively. The study was consistent with other cardiac surgery cohorts, but showed a low prevalence of AKI (4%). Only 14 participants developed KDIGO stage 2 and 3 AKI. The study by Oezkur et al.27 analysed 100 patients immediately after cardiac surgery. The reported sensitivity and specificity values were 0.60 (95% CI 0.36 to 0.81) and 0.89 (95% CI 0.80 to 0.95), respectively. The prevalence of AKI was 19%. Table 5 shows a summary of the diagnostic data for the two studies.

No suitable NephroCheck data in other post-surgical settings (major non-cardiac surgery) were available from the included studies.

Critical care: mixed population

Five studies (2279 participants in total) assessed the use of NephroCheck for detection of AKI in hospitalised patients admitted to ICU or critical care for various clinical reasons. The cut-off point used was consistent across studies (0.3 ng/ml²/1000). Table 5 shows a summary of the diagnostic data for the five studies.29,30,33,34,36 Sensitivity values ranged from 0.64 to 0.92; specificity values ranged from 0.46 to 0.56. The summary estimate of sensitivity was 0.83 (95% CI 0.72 to 0.91) and that of specificity was 0.51 (95% CI 0.48 to 0.54). The SROC plot with 95% confidence region for the summary operating point and 95% prediction region is presented in Appendix 9, Figure 27. The confidence and prediction regions indicate a greater degree of heterogeneity in sensitivity estimates than in specificity estimates between studies. Specificity estimates were low but reasonably homogeneous (see Appendix 9, Figure 26).

Figure 5 shows the forest plots of sensitivity and specificity estimates for all NephroCheck studies (2778 patients in total) across clinical settings. Sensitivity values ranged from 0.31 to 0.92 and specificity values ranged from 0.46 to 0.89. Summary estimates for sensitivity and specificity were 0.75 (95% CI 0.58 to 0.87) and 0.61 (95% CI 0.49 to 0.72), respectively. Figure 6 shows the SROC plot with 95% confidence region for the summary operating point and 95% prediction region. The confidence and prediction regions are large, indicating considerable heterogeneity between studies. Across studies, estimates of specificity were generally low, apart from two studies that showed higher estimates. Visual inspection of the forest plot and SROC plots shows that the study by Cummings et al.26 is an outlier, with a trend very different from that of the other studies.

FIGURE 5 Forest plots of sensitivity and specificity for NephroCheck studies: all clinical settings. FN, false negative; FP, false positive; TN, true negative; TP, true positive. (Figure not reproduced here; see the original article.)

FIGURE 6 The SROC plot for NephroCheck studies: all clinical settings. HSROC, hierarchical summary receiver operating characteristic. (Figure not reproduced here; see the original article.)

There were no studies assessing the use of NephroCheck in children, as this biomarker is recommended for adult use only (people aged ≥ 21 years).

ARCHITECT urine neutrophil gelatinase-associated lipocalin assay: adult population

Cardiac surgery

Two studies37,48 provide test accuracy data on the use of the ARCHITECT urine NGAL assay for detection of AKI in patients who underwent cardiac surgery (Table 6). The multicentre cohort study by Parikh et al.37 assessed a total of 1219 adults after cardiac surgery. The sensitivity and specificity values for the first urine sample collected soon after ICU admission were 0.46 (95% CI 0.33 to 0.59) and 0.81 (95% CI 0.79 to 0.83), respectively. The prevalence of AKI in the study was 5%, similar to that observed previously in the Cummings et al.26 study, which assessed the role of NephroCheck in 400 participants in the same clinical setting. The single-centre study by Thanakitcharu et al.48 assessed 130 patients immediately after cardiac surgery. The sensitivity and specificity values for the urine sample collected immediately after surgery were 0.74 (95% CI 0.49 to 0.91) and 0.6 (95% CI 0.51 to 0.70), respectively. The prevalence of AKI in the study was 35%.

No suitable ARCHITECT urine NGAL assay data in other post-surgical settings (major non-cardiac surgery) were available from the included studies.

Critical care: mixed population

Four studies (1998 patients in total) assessed the use of the ARCHITECT urine NGAL assay for the detection of AKI in patients admitted to an ICU or critical care for various clinical reasons. Cut-off values varied across studies (see Table 6). In three studies, urine NGAL levels were reported as ng/ml (per ml of urine), whereas, in one study, urine NGAL levels were normalised by units of urine creatinine (per mg of creatinine). Prevalence of AKI ranged from 6% to 36% across studies. Table 6 shows a summary of the diagnostic data, as reported by the six studies. Sensitivity values ranged from 0.63 to 0.78 and specificity values ranged from 0.58 to 0.81. The summary estimate of sensitivity was 0.70 (95% CI 0.63 to 0.76) and that of specificity was 0.72 (95% CI 0.63 to 0.80). The forest plot of sensitivity and specificity estimates and the SROC plot with 95% confidence region for the summary operating point and 95% prediction region are presented in Appendix 9 (see Figures 28 and 29). The large confidence and prediction regions of the SROC plot indicate considerable heterogeneity in estimates of accuracy across studies, especially for specificity. The analysis appears to be dominated by the Nickolas et al.56 study, the largest study, which shows a small number of true-positive cases and, subsequently, low sensitivity.

Figure 7 shows the forest plots of sensitivity and specificity estimates for all ARCHITECT urine NGAL studies (3347 patients in total) across all clinical settings. Sensitivity values ranged from 0.46 to 0.78 and specificity values ranged from 0.58 to 0.81. Summary estimates for sensitivity and specificity were 0.67 (95% CI 0.58 to 0.76) and 0.72 (95% CI 0.64 to 0.79), respectively. Figure 8 shows the SROC plot with 95% confidence region for the summary operating point and 95% prediction region. The confidence and prediction regions are large, indicating heterogeneity between studies.

FIGURE 7 Forest plots of sensitivity and specificity for the ARCHITECT urine NGAL assay for the detection of AKI in adults: all clinical settings. FN, false negative; FP, false positive; TN, true negative; TP, true positive. (Figure not reproduced here; see the original article.)

FIGURE 8 The SROC plot for the ARCHITECT urine NGAL assay studies: all clinical settings (adult population). HSROC, hierarchical summary receiver operating characteristic. (Figure not reproduced here; see the original article.)

BioPorto urine neutrophil gelatinase-associated lipocalin assay: adult population

Cardiac surgery

One study65 assessed the use of the BioPorto urine NGAL assay for the detection of AKI in a total of 398 patients who had undergone cardiac surgery (Table 7). Urine NGAL levels were normalised by units of urine creatinine (with a cut-off point of 98 µg/g creatinine). The sensitivity and specificity values for the urine sample collected 6 hours after surgery were 0.78 (95% CI 0.71 to 0.84) and 0.48 (95% CI 0.41 to 0.54), respectively. The prevalence of AKI in the study was 41%.

Non-cardiac surgery

One study66 assessed the use of the BioPorto urine NGAL assay for detection of AKI in 131 patients undergoing hepatobiliary surgery (see Table 7). The urine NGAL cut-off point was 92.85 ng/ml. The sensitivity and specificity values for the urine sample collected 12 hours after surgery were 0.78 (95% CI 0.52 to 1.00) and 0.80 (95% CI 0.73 to 0.87), respectively. The prevalence of AKI in the study was 8%.

Critical care: mixed population

Six studies68,69,72–74,78 (1442 patients in total) assessed the use of the BioPorto urine NGAL assay for the detection of AKI in patients admitted to an ICU or critical care for various clinical reasons (see Table 7). Some studies reported absolute levels of urine NGAL and other levels normalised to urine creatinine. The threshold varied across studies. The prevalence of AKI ranged from 5% to 49% across studies. Sensitivity values ranged from 0.58 to 0.90 and specificity values ranged from 0.70 to 1.00. The summary estimate of sensitivity was 0.72 (95% CI 0.61 to 0.80) and that of specificity was 0.87 (95% CI 0.66 to 0.96). The forest plots of sensitivity and specificity estimates and the SROC plot with 95% confidence region for the summary operating point and 95% prediction region are presented in Appendix 9 (see Figures 30 and 31). The confidence and prediction regions of the SROC plot are large, indicating heterogeneity between studies, especially for specificity.

Figure 9 shows the forest plots of sensitivity and specificity estimates for the eight studies (1971 patients in total) assessing the BioPorto urine NGAL assay for the detection of AKI in adults across all clinical settings. Sensitivity values ranged from 0.58 to 0.90 and specificity values ranged from 0.48 to 1.00. Summary estimates for sensitivity and specificity were 0.73 (95% CI 0.65 to 0.80) and 0.83 (95% CI 0.64 to 0.93), respectively. The SROC plot with the 95% confidence region for the summary operating point and the 95% prediction region, is shown in Figure 10. The confidence and prediction regions are large, indicating considerable heterogeneity between studies.

FIGURE 9 Forest plots of sensitivity and specificity for the BioPorto urine NGAL assay studies for the detection of AKI in adults: all clinical settings. FN, false negative; FP, false positive; TN, true negative; TP, true positive. (Figure not reproduced here; see the original article.)

FIGURE 10 The SROC plot for the BioPorto urine NGAL assay studies for the detection of AKI in adults: all clinical settings. HSROC, hierarchical summary receiver operating characteristic. (Figure not reproduced here; see the original article.)

Urine neutrophil gelatinase-associated lipocalin assays (ARCHITECT and BioPorto): critical care

Ten studies (3441 patients in total) assessed urine NGAL assays (both ARCHITECT and BioPorto) for the detection of AKI in patients admitted to critical care. Sensitivity values ranged from 0.58 to 0.90 and specificity values ranged from 0.58 to 1.00. Summary estimates for sensitivity and specificity were 0.71 (95% CI 0.64 to 0.77) and 0.82 (95% CI 0.67 to 0.90), respectively. The forest plot of sensitivity and specificity estimates and the SROC plot with 95% confidence region for the summary operating point and 95% prediction region are shown in Appendix 9 (see Figures 32 and 33). The prediction region in the SROC plot is large, especially for specificity, indicating heterogeneity across studies.

Urine neutrophil gelatinase-associated lipocalin assays (ARCHITECT and BioPorto): across all settings

Figure 11 shows the forest plots of sensitivity and specificity estimates for all 14 studies assessing the urine NGAL assays (both ARCHITECT and BioPorto) across all clinical settings (5319 patients in total). Sensitivity values ranged from 0.46 to 0.90 and specificity values ranged from 0.48 to 1.00. Summary estimates for sensitivity and specificity were 0.71 (95% CI 0.64 to 0.76) and 0.78 (95% CI 0.67 to 0.87), respectively. Figure 12 shows the SROC plot with 95% confidence region for the summary operating point and 95% prediction region. The prediction region is large, indicating heterogeneity across studies.

FIGURE 11 Forest plots of sensitivity and specificity for all urine NGAL assays (both ARCHITECT and BioPorto) for the detection of AKI in adults across all clinical settings. FN, false negative; FP, false positive; TN, true negative; TP, true positive. (Figure not reproduced here; see the original article.)

FIGURE 12 The SROC plot for all urine NGAL assays (both ARCHITECT and BioPorto) for the detection of AKI in adults: all clinical settings. HSROC, hierarchical summary receiver operating characteristic. (Figure not reproduced here; see the original article.)

BioPorto plasma neutrophil gelatinase-associated lipocalin assay: adult population

No suitable data in any post-surgical setting (cardiac surgery or major non-cardiac surgery) were available from the included studies.

Critical care: mixed population

Four studies (771 patients in total) assessed the use of the BioPorto plasma NGAL assay for the detection of AKI in patients admitted to ICU or critical care for various clinical reasons (Table 8). Cut-off points varied across studies The prevalence of AKI ranged from 13% to 38% across studies. Figure 13 shows the forest plots of sensitivity and specificity. Sensitivity values ranged from 0.58 to 0.93 and specificity values ranged from 0.23 to 0.85. The summary estimate of sensitivity was 0.76 (95% CI 0.56 to 0.89) and that of specificity was 0.67 (95% CI 0.40 to 0.86). Figure 14 shows the SROC plot with 95% confidence region for the summary operating point and 95% prediction region. Confidence and prediction regions are large and are greater for sensitivity than for specificity. Although this indicates the presence of heterogeneity across studies, it is worth observing that all studies and their confidence and prediction regions are positioned in the left side of the graph, above the diagonal of no effect. It is worth paying attention to the Itenov et al.81 study, which shows a high sensitivity estimate and a very low specificity estimate.

FIGURE 13 Forest plots of sensitivity and specificity for the BioPorto plasma NGAL assay for the detection of AKI in adults: critical care setting. FN, false negative; FP, false positive; TN, true negative; TP, true positive. (Figure not reproduced here; see the original article.)

FIGURE 14 The SROC plot for the BioPorto plasma NGAL assay studies for the detection of AKI in adults: critical care setting. HSROC, hierarchical summary receiver operating characteristic. (Figure not reproduced here; see the original article.)

Table 9 presents a summary of the diagnostic data for the seven urine NGAL assay studies that assessed AKI in children. All but one study assessed children who underwent cardiac surgery. Across studies, the age of the paediatric population ranged from 1 day to 8 years.

ARCHITECT urine neutrophil gelatinase-associated lipocalin assay: child population

Cardiac surgery

Five studies (887 children in total) assessed the use of the ARCHITECT urine NGAL assay for the detection of AKI among children who had undergone cardiac surgery (see Table 9). The cut-off point used to define a positive test and the timing of biomarker measurements varied across studies. The prevalence of AKI ranged from 17% to 55% across studies. Figure 15 shows the forest plots of sensitivity and specificity estimates across studies. Sensitivity values ranged from 0.42 to 0.83 and specificity values ranged from 0.43 to 0.92. The summary estimate of sensitivity was 0.68 (95% CI 0.53 to 0.80) and that of specificity was 0.79 (95% CI 0.63 to 0.89). Figure 16 shows the SROC plot with 95% confidence region for the summary operating point and 95% prediction region. The confidence and prediction regions are very large, indicating considerable heterogeneity between studies.

FIGURE 15 Forest plots of sensitivity and specificity for the ARCHITECT urine NGAL assay for the detection of AKI in children: cardiac surgery setting. FN, false negative; FP, false positive; TN, true negative; TP, true positive. (Figure not reproduced here; see the original article.)

FIGURE 16 The SROC plot for the ARCHITECT urine NGAL assay studies for the detection of AKI in children: cardiac surgery setting. HSROC, hierarchical summary receiver operating characteristic. (Figure not reproduced here; see the original article.)

Critical care: mixed population

One study91 assessed the use of the ARCHITECT urine NGAL assay for the detection of AKI in 324 children admitted to ICU or critical care for various clinical reasons (see Table 9). The cut-off point was 126 ng/ml. The prevalence of AKI in the study was 35%. The sensitivity and specificity values for the urine sample collected at ICU admission were 0.77 (95% CI 0.60 to 0.90) and 0.85 (95% CI 0.74 to 0.92), respectively.

BioPorto urine neutrophil gelatinase-associated lipocalin assay: child population

Cardiac surgery

One study65 assessed the use of the BioPorto urine NGAL assay for the detection of AKI in 323 children who underwent cardiac surgery (see Table 9). Urine NGAL was measured using a concentration normalised by units of creatinine. The sensitivity and specificity values for the urine sample collected 6 hours after surgery were 0.77 (95% CI 0.69 to 0.84) and 0.47 (95% CI 0.40 to 0.54), respectively. The prevalence of AKI in the study was 39%.

Urine neutrophil gelatinase-associated lipocalin assays (both ATCHITECT and BioPorto) across all clinical settings: child population

Figure 17 shows the forest plots of sensitivity and specificity estimates for the seven studies (1310 children in total) assessing urine NGAL assays (both ARCHITECT and BioPorto) for the detection of AKI among children across all clinical settings. Sensitivity values ranged from 0.42 to 0.83; specificity values ranged from 0.43 to 0.92. Summary estimates for sensitivity and specificity were 0.71 (95% CI 0.60 to 0.80) and 0.76 (95% CI 0.61 to 0.86), respectively. Figure 18 shows the SROC plot with 95% confidence region for the summary operating point and 95% prediction region. The confidence and prediction regions are very large, indicating considerable heterogeneity between studies.

FIGURE 17 Forest plots of sensitivity and specificity for all urine NGAL assays (ARCHITECT and BioPorto) for the detection of AKI in children across all clinical settings. FN, false negative; FP, false positive; TN, true negative; TP, true positive. (Figure not reproduced here; see the original article.)

FIGURE 18 The SROC plot for all urine NGAL assay (ARCHITECT and BioPorto) studies for the detection of AKI in children: all clinical settings. HSROC, hierarchical summary receiver operating characteristic. (Figure not reproduced here; see the original article.)

Accuracy of NephroCheck, ARCHITECT neutrophil gelatinase-associated lipocalin and BioPorto neutrophil gelatinase-associated lipocalin assays for the detection of acute kidney injury in critically ill patients

For both adults and children, the accuracy of NephroCheck, ARCHITECT urine NGAL and BioPorto urine and plasma NGAL assays for detection of AKI in each clinical setting is shown in Table 10. The table displays either the AUC estimates as reported by individual studies or the AUC summary estimates together with the corresponding prediction intervals when pooling of AUC estimates was feasible. Associated forest plots of the AUC meta-analyses are presented in Appendix 10 (see Figures 34–50). For the adult population, the AUC summary estimates ranged from 0.62 for the BioPorto urine NGAL assay to 0.74 for the BioPorto plasma NGAL assay in the cardiac surgery setting, and from 0.72 for the BioPorto urine and plasma NGAL assays to 0.76 for the ARCHITECT urine NGAL assay in the critical care setting. For the child population in the cardiac surgery setting, the AUC summary estimates ranged from 0.80 for the ARCHITECT urine NGAL assay to 0.88 for the BioPorto urine NGAL assay. All AUC summary estimates had relatively large 95% prediction intervals, indicating heterogeneity between studies. The forest plots in Appendix 10 (see Figures 34–50) show that variation is present both between and within studies.

For each biomarker, Table 11 shows the AUC for the detection of AKI compared with that of serum creatinine or conventional clinical assessment, as reported by the individual studies that provided this information. AUC values varied across studies. In the majority of cases, the reported AUC indicated a slightly better performance of the biomarkers than that of serum creatinine or conventional clinical assessment for the detection of AKI. However, in a number of cases, serum creatinine or conventional clinical assessment appeared to perform better than the biomarkers under assessment.

Role of biomarkers in predicting worsening of acute kidney injury mortality and need for renal replacement therapy

Table 12 displays the AUC and the pooled AUC estimates with corresponding 95% CIs for the NephroCheck, ARCHITECT urine NGAL and BioPorto urine and plasma NGAL assay studies for the prediction of worsening of AKI, mortality and RRT in each clinical setting for both adults and children. Only a limited number of studies were available for AUC meta-analyses. Associated forest plots are presented in Appendix 11 (see Figures 51–55). In the critical care setting (adult population), the AUC values reported in individual studies ranged from 0.66 for the BioPorto plasma NGAL assay to 0.71 for the BioPorto urine NGAL assay for worsening of AKI, and from 0.55 for the BioPorto plasma NGAL assay for prediction of 90-day mortality to 0.75 for the ARCHITECT urine NGAL assay for prediction of 30-day mortality. One study reported an AUC of 0.70 for BioPorto plasma NGAL for prediction of need for RRT. The AUC summary estimate (pooling of 2 studies) for worsening of AKI in the critical care setting was 0.65 (95% CI 0.43 to 0.82) for ARCHITECT urine NGAL. AUC summary estimates ranged from 0.62 for BioPorto urine NGAL for 90-day mortality to 0.68 for the BioPorto plasma NGAL assay for in-hospital mortality, with a 95% CI of 0.58 to 0.73. The AUC summary estimate (two studies) for prediction of RRT in critical care for the BioPorto urine NGAL assay was 0.74 (95% CI 0.49 to 0.89).72,75 In the cardiac surgery setting (adult population), AUC values from individual studies ranged from 0.68 for the ARCHITECT urine NGAL assay to 0.78 for NephroCheck for prediction of need for RRT, with a 95% CI of 0.57 to 0.84.

Appendix 12, Table 30, presents the AUC with 95% CI or the OR with 95% CI for the addition of the biomarkers to existing clinical models for the prediction of AKI, mortality and need for RRT. It is worth noting that the statistical models differed between studies and often were not sufficiently detailed. In particular, although most of the adjusting predictors were specified, information on the potential candidate variables was missing. In general, the number of events was small, given the number of predicting variables, even for AKI outcomes. Overall, the addition of biomarkers to the clinical models improved risk prediction of newly developed AKI or worsening of AKI, and mortality. However, only a limited amount of data were available for each biomarker in each clinical setting, thereby restricting any generalisable interpretation.

Interpretation of clinical effectiveness evidence

The results of the meta-analyses of sensitivity and specificity estimates suggest that the biomarkers under investigation (i.e. NephroCheck, ARCHITECT urine NGAL and BioPorto urine and plasma NGAL assays) may have a role in the detection of AKI in critically ill patients. However, owing to the considerable clinical and statistical heterogeneity observed across studies and the limited number of studies available for certain clinical settings and/or type of biomarker, these results should be interpreted with caution and require further evidence to substantiate them. Furthermore, the threshold level for NGAL varied considerably across studies. However, as optimal NGAL thresholds for detection of AKI in various clinical settings have yet to be established, we decided to pool results across studies with similar characteristics, despite this obvious limitation. For the adult population, we were able to conduct meta-analyses for studies that assessed patients in the critical care (mixed population) setting and for studies across all clinical settings. There were too few studies assessing patients after cardiac surgery or major non-cardiac surgery. The NephroCheck test had the highest pooled sensitivity (0.83), but the worst pooled specificity (0.51), whereas the ARCHITECT and BioPorto urine NGAL tests had slightly lower pooled sensitivity estimates (0.70 and 0.72, respectively), but better pooled specificity estimates (0.72 and 0.87, respectively). The BioPorto urine NGAL assay pooled sensitivity was similar to that of the BioPorto plasma NGAL assay (0.72 vs. 0.76, respectively), but the pooled specificity was better for the BioPorto urine NGAL assay than for the BioPorto plasma NGAL assay (0.87 vs. 0.67, respectively). The biomarkers had a similar performance across all clinical settings (the NephroCheck test pooled sensitivity and specificity were 0.75 and 0.61, respectively; the ARCHITECT urine NGAL assay pooled sensitivity and specificity were 0.67 and 0.72, respectively; the BioPorto urine NGAL assay pooled sensitivity and specificity were 0.73 and 0.83, respectively; and the BioPorto plasma NGAL assay pooled sensitivity and specificity were 0.76 and 0.67, respectively), with the BioPorto plasma NGAL assay showing the highest sensitivity (0.76) and the BioPorto urine NGAL assay showing the highest specificity (0.83).

With regard to the observed low specificity of the NephroCheck test, we do not know with certainty whether this is because of the relatively poor performance of the biomarker or the fact that serum creatinine is an imperfect reference standard for assessing kidney injury.

We also noted that, when a study had a small number of AKI events (low prevalence), the relationship observed between sensitivity and specificity estimates became quite different from that of studies for which prevalence was higher.

For the child population, we were able to conduct meta-analyses for the five ARCHITECT urine NGAL assay studies that assessed children who underwent cardiac surgery. The pooled sensitivity was 0.68 and the pooled specificity was 0.79. Too few studies were available for the other assays or clinical settings. When we combined all urine NGAL studies (ARCHITECT and BioPorto) across all settings (seven studies), we obtained similar estimates of accuracy (sensitivity 0.71; specificity 0.76).65,84,87–91

For the prediction of relevant clinical outcomes, only a limited number of studies were available for each biomarker in each clinical setting; this hampered the possibility of performing pooled analyses. Furthermore, the details of the methodology used for the statistical analyses were insufficient, especially for older studies. The more recent studies appeared to use some of the PROBAST22 recommendations and terminology, but they were still far from satisfactory, as demonstrated by the results of the PROBAST assessment (see Figures 3 and 4). Moreover, although information on the adjustment strategies and the process of variables’ selection was provided in individual studies, the original cohort of potential predictors, prior to the multivariable analysis, was never clearly specified, leading to potential risk of data-mining and, hence, methodological bias.

Similarly, although there was an indication that the addition of biomarkers to existing clinical models might improve the prediction of relevant clinical outcomes, studies varied substantially in terms of study characteristics and statistical methods used to assess prediction, thereby limiting any reliable conclusion.

On the whole, we observed considerable clinical and statistical heterogeneity in all analyses, especially with regard to clinical setting, NGAL threshold levels, time of sample collection, definition of AKI, time of AKI diagnosis, number of AKI events and assay platforms. Therefore, we have limited confidence in the validity and reliability of the observed results.