Confidence in cumulative evidence

Overview

In this chapter, we consider the methodological quality of the RELEASE study, the meta-syntheses, risk of bias and confidence in the exploratory findings. Risk of bias at primary data set level (and IPD) is detailed in Chapter 3. We used the ROBIS tool to support our reflections on (1) the methodological steps taken to reduce the risk of meta-bias and (2) whether or not meta-biases may have affected the data synthesis and findings. We describe the results of investigations into heterogeneity, inconsistency and missing data, and present the findings from the prespecified sensitivity analyses.

Methodological quality of the primary research data sets

Across the meta-syntheses, we prioritised IPD derived from RCT data sets, followed by re-analysis using the full data set inclusive of non-RCT IPD. Risk of selection bias was moderate in the RCTs informing our IPD network meta-analysis across language domains (see Appendix 10, Figure 28), with between 50% and 75% reporting adequate randomisation. Primary research study participant groups were comparable at baseline; only three data sets had participant groups that differed by age, sex or time since stroke. Risk of detection bias was low.

Three-quarters (75%) of RCT data sets included in the network meta-analyses across language domains reported outcome assessor blinding. Attrition bias was low (see Appendix 10, Figure 28). Many data sets retained participants to the end of the study, whereas others reported dropouts and non-adherence. Risk of reporting bias was highlighted, with some public domain IPD available at baseline only or partial contribution of some electronic data sets. However, these partial RCT data sets often lacked follow-up or intervention data, and thus were excluded from the network meta-analyses. By contrast, full IPD RCT data sets were also extracted from the public domain and some electronic data set contributions included IPD unreported in associated publications. Thus, the IPD RCT data sets informing the network meta-analyses were considered to be at a moderate to low risk of selection bias.

Meta-synthesis and risk of bias

Supported by the ROBIS tool, we considered the extent to which the meta-analyses and network meta-analyses were at risk of bias.105

Specification of study eligibility criteria; selection bias

The systematic review yielded a large number of records, which we screened for eligibility (see Chapters 2 and 3). Research teams were invited to contribute and collaborate. We had had no prior collaboration with 55 data set contributors. The systematic, inclusive approach to recruitment supported data set contributions that may not otherwise have been volunteered. We identified and extracted IPD from the public domain and grey literature in a range of languages (including 183 IPD from four RCTs, and 1988 IPD from 96 non-RCTs). These methodological steps were important in reducing the risk of selection bias in the meta-analyses.

The systematic development of the RELEASE study database used a priori eligibility criteria; reflected the scope of the research questions; and incorporated participant, study design and outcome data and IPD availability. Some research questions required SLT intervention data (see Chapters 6 and 7). Non-intervention data sets were also important, contributing to the exploration of patterns and predictors of language recovery (see Chapters 4 and 5). The database eligibility criteria were deliberately inclusive and were adhered to throughout.

We accepted diagnosis of participants’ aphasia in the primary research, confirming that it followed stroke. Aphasia associated with other neurological aetiologies was excluded. Formal approaches to aphasia diagnosis rely on validated language assessments, data required for inclusion in the study database and which informed our analysis. Data collection that was not quantitative or based on informal measures of language, were excluded as they were irrelevant to the research questions and analysis plan. We had no exclusions by study date, language of data collection or source, which might have affected electronic database searching or the contribution of unpublished clinical data sets. We made no other exclusions.

Data set identification and selection; availability bias

We made substantial efforts to identify all eligible IPD data sets, employing a broad search strategy (see Appendix 1) across databases and grey literature, and systematically inviting contributions from identified (potentially eligible) records and members of the Collaboration of Aphasia Trialists (see Chapter 3). The RELEASE study database also reflects data sharing with large independent aphasia databases [e.g. AphasiaBank, the Cantonese Aphasia Bank, Predicting Language Outcome and Recovery After Stroke (PLORAS)119] and clinical registries.

The RCT-optimised systematic search applied no restrictions to date, language or publications. We independently screened titles and abstracts, followed by double-checking to minimise any errors in the selection of eligible texts. We listed the data sets identified as potentially eligible for inclusion but for which the IPD were unavailable, which suggests additional RCTs (46 inclusive of a standard care or experimental SLT intervention) may exist (see Report Supplementary Material 2). We were unable to confirm that these data sets (and associated IPD) typically described as aggregate data in full text reports met our eligibility criteria or that the IPD remained available. Confirmation of the eligibility of included datasets was only possible following communication with the primary researchers and careful review of the IPD (see Chapter 3). Yet, it is probable that some eligible IPD existed but were unavailable to our analysis.

From a research-waste perspective, the possible existence of eligible but unavailable data is disappointing given that several of the planned analyses were not possible or unstable owing to lack of data availability (see Chapters 4–7). Additional data would have strengthened the network meta-analyses, potentially providing more links between network nodes (see Chapters 6 and 7) and greater insights. The number of included data sets, however, was sufficient to address the main RELEASE research questions.

The RELEASE study data set was predominantly based on data from English-speaking, high-income countries with well-developed SLT and stroke services. Non-English-language aphasia assessment tools are lacking, and limited (or absent) SLT services are important constraints on data-gathering in some countries. Thus, it is unlikely that we omitted many large, robust language data sets from non-English speaking countries.

Data collection and study appraisal (ROBIS domain 3)

We sought to minimise data extraction and synthesis errors. Data extraction was rigorously checked by a second reviewer. Data extraction of the complex SLT intervention descriptions was independently conducted by two researchers and was reviewed by the primary researchers. Collaborators informed the categorisation of narrative descriptions of SLT interventions by (i) target of impairment and (ii) a theoretical approach that supported data synthesis and analysis. We provided an opportunity to examine the data sets that informed the analyses, presenting aggregate data set descriptions (see Chapter 3) and individual data set overviews (see Report Supplementary Material 1). We detailed the approach to data synthesis, data item descriptions and group labels used in the analysis (see Chapter 3). Formal approaches to risk-of-bias analysis at primary data set level were described and undertaken by an independent reviewer, with a second reviewer rigorously checking that analysis (see Chapter 3).

Synthesis and findings

Data restrictions

We experienced restrictions in retrieving potentially eligible IPD data sets (described above) and limitations in accessing the full IPD data sets for some electronically contributed and public domain data sets. Other unavailable data included data sets that used a comprehensive aphasia assessment battery (comprising several subtests), but only summary test scores were recorded and reported. We experienced no other restrictions on data availability.

Bias

We found no evidence of publication bias based on study size (see Appendix 32), although publications with an earlier date had a higher proportion of non-significant results than findings reported in later publications (p = 0.014). We sought to reduce the risk of researcher bias by assigning data sets numerical IDs, so that collaboration-wide consensus decisions could be reached based on anonymised data sets, rather than the primary research teams.

Data synthesis

The primary data sets focused on language recovery and people with aphasia. Some included a SLT intervention; these were included in the network meta-analyses. Clinically, individual data sets were very similar in population, intervention, comparators and outcomes (see Report Supplementary Material 1), but this was not formally tested.

We used a random-effects model. Application of the Wu–Hausman108 test for fixed- versus random-effects model (described below) indicated that there was no difference based on these two models of analysis (see Appendix 33, Table 37). We considered heterogeneity by measuring relative variance. Any analysis that breached the prespecified threshold of acceptable variance (> 50%) was omitted from the interpretation of the data.

The one-stage IPD network meta-analyses did not incorporate aggregate summary data. We prioritised IPD derived from RCTs, but also considered the findings inclusive of IPD from non-RCTs, which were typically in keeping with the RCT IPD findings. Analysis on a single data set was not conducted. The prespecified minimum data analysis sample of two data sets (minimum of 20 IPD) was adhered to. However, the analysis occasionally generated clusters of small IPD or data set numbers; in such cases, these were treated with caution.

When IPD availability allowed, we conducted a prespecified subgroup analysis (see Chapter 7) to explore the possibility of heterogeneity between clinically relevant participant subpopulations. Based on the preparatory exploration of the data set, we included one post hoc subgroup. Alternative approaches to the data synthesis are unlikely to alter the findings.

Quality of the evidence

We considered the degree to which we could be confident of the study findings, based on the quality of the evidence. The patterns of language recovery findings lacked data for several planned analyses, and these were ungraded (Table 11, part A). The distribution of language scores at baseline was based on large numbers of IPD and it is unlikely that additional research will affect the finding. We have assigned a GRADE rating of ‘high’ (see Table 11, part A).

We graded the evidence informing the absolute proportion change scores, multivariable analysis and the predictors of recovery (based on RCT IPD) as moderate level of evidence (see Table 11, part A). This was downgraded from high in recognition of the number of potentially eligible data sets identified, but which were not included in the analysis. We graded reading, writing and other spoken-language findings as low because greater availability of data is likely to alter these findings (see Table 11, part B).

The network meta-analyses considering the optimum SLT intervention regimen by language domain was rated low for duration comparisons and naming outcomes, downgraded from moderate because of the lack of IPD and/or unstable network geometry (see Table 11, part C). Further research will contribute to the data, changing confidence in the findings. The overall language ability, auditory comprehension and functional communication findings relating to SLT frequency, intensity and dosage were rated as moderate because of the higher IPD numbers informing these network meta-analyses and more stable network geometry (≈ 500 IPD; see Appendices 11–14) (see Table 11, part C). The subgroup network meta-analyses were based on even smaller IPD numbers, and were thus also graded as very low (see Appendix 30, Table 34).

Heterogeneity

We noted more instances of high heterogeneity (identified as > 50% relative variance) in the subgroup network meta-analysis, due, in part, to smaller IPD sample sizes and more instances of orphan interventions. These subgroup analyses were more exploratory than confirmatory. We chose to present all the results, highlighting instances of high heterogeneity (> 50%) in the relevant tables for information (see Appendices 30 and 34).

Exploration for inconsistency

Owing to the scarcity of direct comparisons for most pairwise nodes in the network meta-analysis (see Appendices 11–14 many contained a single study and were often based on a small number of IPD; see Appendix 34), a comparison of direct and indirect evidence was not useful. Direct comparisons within small, single RCTs would be likely to have large CIs that would be difficult to show a difference in, unless that difference was excessive. Instead, we examined inconsistency between studies by calculating the relative proportion of the variability (the difference between the predicted value and the actual value) that appeared to be solely due to differences between studies, and that of the null model (fitting only an intercept or constant term). A high proportion in this case would imply that most of the differences in the data were due to differences between studies, and thus imply that the studies were heterogeneous and thus too dissimilar for reliable meta-analysis.

Missing data/missing completely at random

Contributed data sets tended to have either nearly complete or nearly incomplete data for the majority of variables, and so rarely exceeded the prespecified threshold of 20% missing data. Many variables of interest were insufficiently collected across the data sets to include in the analyses. Comparisons of missingness with the main demographic variables did not reveal any systematic relationships (i.e. not missing at random). Other variables were well represented and were sufficient across the database and by primary research data set to include in our analyses (e.g. stroke type), but there was a lack of variability in subclassifications (e.g. some stroke types had only a few participants), making estimates for that variable imprecise, and, invariably, non-significant.

Publication bias

We included both published and unpublished IPD (including three clinical registry data sets and a RCT) in the RELEASE study. Funnel plots were not possible because of the nature of the data set. Instead, we considered the distribution of studies based on the number of participants recruited and the publication of a significant finding. We found no evidence of publication bias by sample size (p = 0.77; see Appendix 32).

Other biases

The risk of potential carry-over within crossover data sets was addressed by extracting the data up to the point of crossing over, but not beyond. We extracted primary research details (see Report Supplementary Material 1) and, although some data sets did not share language recovery and rehabilitation as their main study objective, there was no indication that this had any impact on the data extracted for the RELEASE study.

Sensitivity analyses

We conducted a series of pre-planned sensitivity analyses to examine decisions made in the data synthesis and the potential impact that these decisions may have had on the findings.

Choice of language measurement

Some primary data sets had IPD on two or more assessment tools that were eligible contributors to data synthesis relating to a single language outcome. Inclusion of all relevant data was not possible as it would result in ‘double-counting’ participants in that outcome. We chose which language assessment would contribute to the data synthesis for that language domain (see Report Supplementary Material 18) using a structured, systematic approach to support decision-making:

• We reviewed the numbers of missing data (considering data at study entry and follow-up time points). The more complete data set (IPD) was prioritised.
• When there was equivocation (i.e. both assessments had equal IPD numbers, including at follow-up), we prioritised the inclusion of IPD collected on the anchor measure.

Most overlapping data situations were resolved using this approach. Only a few remained, involving a small number of IPD and data sets, and these decisions were unlikely to affect the study findings and conclusions. For example, one data set using the TOMs included data from both activity and participation subtests, available on equal numbers of IPD. We arbitrarily included the activity data, but conducted a sensitivity analysis by replacing the activity item with the participation data in the network analysis (see Appendix 9, Tables 16–18). There was no indication of a notable difference between the findings. Each case of overlapping data eligibility was recorded, along with the choices available and decisions reached (see Report Supplementary Material 18). The numbers of IPD and data sets were insufficient to support any formal sensitivity analysis or the production of any meaningful result.

Exclusion of minority measures

We plotted the distribution of all transformed measures over time and by language domain, and observed that the spread of data points gathered on all language measures as profiled by each language domain overlapped (see Appendix 6, Figure 27 and Appendix 7, Table 14). There did not appear to be a clustering of transformed language measures that were generated from any one contributing assessment. The transformed values generated by each of the language assessments were valid and within an expected range.

Random or fixed effects

We considered the impact of the choice of a random-effects model over a fixed-effects model, using both the Wu–Hausman and the alternative Wu–Hausman. We compared the difference between the estimates of the covariate and standard error of baseline score. We had inadequate data for the reading comprehension item (one RCT). For all other language domains, the W score was < 3.841. For the alternative W, most values were < –1.96 or > 1.96, except for those based on functional communication (–1.9680). We also noted a relative variability of 41.1% for the naming data set, although this dropped to 30% when other variables were included in the model (see Appendix 33, Table 37). We rejected the null hypothesis that fixed effects were more efficient.

The inclusion of historic data sets

We explored the availability of historic data sets and what bias there may be in the inclusion of data sets gathered in the 1970s, as treatment responses may reflect health-care interventions (SLT and stroke related) at that time. We found no evidence that date of recruitment contributed to the findings based on language outcomes (see Appendix 31).

Excluding non-randomised controlled trial data sets

Throughout the analysis relating to patterns of recovery, predictors of recovery and optimum approaches to SLT, we prioritised the data analysis on IPD from RCT data sets only, and then re-ran the analysis inclusive of IPD from non-RCT data sets. We observed only modest (non-significantly) higher values in the context of some analyses inclusive of non-RCT data set (often with wider CIs) (see Chapters 4–6). Inclusion of the non-RCT data sets augmented the IPD on language abilities of people with aphasia at a specified time point from onset and in analyses where RCT data were lacking or absent. The IPD derived from non-RCTs and RCTs contributed to the calculation of median aphasia severity and age at the baseline assessment for the prespecified subgroup analysis (see Chapter 7). However, given the challenges and risk of overstating subgroup analysis results, we restricted the subgroup analyses to RCT IPD only.

Registry data sets and impact on findings

The database included five registry data sets (853/5928 IPD; 14%) that provided valuable data on international, clinical populations with aphasia. Summary statistics on the registry population demographics indicated that the data sets were broadly similar to other non-RCT data sets in terms of age, sex, time since stroke, aphasia severity, handedness and years of education (see Appendix 5, Table 13). Registry data contributed to the analysis of patterns and predictors of recovery only (see Chapter 4). Reporting of therapy interventions in the registries was mostly absent or insufficient to support profiling of the intervention, and thus inclusion in that analysis.

Summary

The RELEASE study analysis was based on high- and moderate-quality data sets (see Chapters 4–7). Wherever possible, we sought to reduce the risk of selection and availability bias, and bias in the data collection, study appraisal and data synthesis processes. We found no indication of important clinical or significant statistical heterogeneity. Isolated cases of high relative variance were reported but not taken forward into the findings. We found no indication of publication or historical bias, or that any decisions taken during the meta-synthesis may have biased the findings.