Methods

Overview

We created an international, multilingual database that included IPD on demographics, language impairment, stroke and SLT interventions. The purpose of our database was to support exploration of a range of key clinical questions relating to SLT for aphasia after stroke. The REhabilitation and recovery of peopLE with Aphasia after StrokE (RELEASE) IPD meta-analysis and network meta-analysis study followed a prespecified protocol and identified suitable data sets following a systematic review of the literature (PROSPERO CRD42018110947). We did not employ a strict experimental approach and hypothesis-testing. Instead, we sought to use statistical inferencing to identify promising lines of enquiry for specific investigation in future large-scale definitive experimental investigations in which participant populations and data items could be well defined.

The RELEASE study database (Integrated Research Application System number 179505) (Figure 2) met existing data-sharing guidelines78 and the International Committee of Medical Journal Editors’ proposal,79 and the reporting observes the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and relevant extensions.80–85 We employed the Template for Intervention Description and Replication (TIDieR) to support data extraction of the complex SLT interventions,86 the Guidance for Reporting Involvement of Patients and the Public (GRIPP2)87 and the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system to support judgements of the impact of the quality of the evidence on our findings.88

FIGURE 2 Creation of the RELEASE study database. ID, identification. (Figure not reproduced here; see the original article.)

Studies included in the review

Types of studies

Data sets with IPD on people with aphasia after stroke were sought. We particularly looked for IPD data sets from RCTs, recognising their rigour and robustness for our planned analyses, by using a RCT-optimised search strategy. However, other study designs that met our eligibility criteria were also included, for example non-RCT, cohort and case series studies,89,90 as well as clinical and research registries (Box 2). Non-RCT designs (including cohort and case series) were included in covariate-adjusted analyses of recovery profiles and predictors of prognosis after stroke.89,91 We applied no restrictions to study design, data set age, country, minimum number of time points, duration of follow-up, language of data collection or publication.

Types of study designs included

The primary research studies included in the RELEASE study were categorised according to study design:

• RCTs – participants were randomly allocated to study groups that typically differed in the intervention received [or received a control (or no) intervention].
• Non-RCT studies – participants were recruited but random allocation was not applied to group selection by the primary research team.
• Cohort/case series – data sets from a single participant group that did not compare different SLT interventions.
• Registries – clinical or research databases holding information (e.g. demographic, stroke, aphasia and SLT intervention details) on people with aphasia after stroke, which was not directly gathered to address a single research question.

Interventional or non-interventional

Studies were also classified according to whether they were (1) interventional, whereby SLT was provided in the context of the study, or (2) non-interventional, whereby no SLT was provided in the context of the study.

Crossover data sets

We identified crossover data sets (crossover RCTs or non-RCTs) and carefully extracted all data up to the point of crossing over.

Types of participants

We screened all participants for inclusion in the RELEASE study database and subsequent analyses. We had no exclusion criteria relating to participants’ time since stroke, age, language impairment or access to SLT. If aphasia was not the result of a stroke, those participants’ data were excluded and the remaining IPD from that data set (when all other eligibility criteria were met) were included. We defined three participant subgroups relevant to our analysis:

1. Natural history subgroup – participants recruited within 15 days of first stroke and allocated to receive no SLT intervention for a defined period of time in the primary research study. These participants were unlikely to have received significant historical language rehabilitation or SLT.
2. Historical SLT subgroup – participants allocated to receive no SLT intervention in a primary research study, but recruited beyond 15 days of stroke onset. Given the context of aphasia research activity, we assumed that these participants had some clinical access to SLT for their aphasia prior to their study entry.
3. SLT subgroup – participants allocated to receive SLT in the context of a primary research study (research SLT) or permitted continued access to standard SLT during the study period and recruited any time after onset of aphasia.

Speech and language therapy interventions

Speech and language therapy was defined as ‘any targeted practice or rehabilitation tasks that aimed to improve language or communication abilities, activities, or participation.’41 When relevant, we sought primary research studies that included a description of a SLT intervention (Table 1) to inform our research questions. Although therapy was typically delivered by a speech and language therapist (a protected professional title in the UK), we also included other therapy providers, as this was one aspect of our planned analysis.

Speech and language therapy interventions were summarised to support clinically meaningful data synthesis and meta-analyses. Narrative descriptions of the SLT approaches, materials and procedures employed were initially extracted as direct quotations (when possible) from published reports. Descriptions were then tabulated and similar approaches were grouped by an experienced speech and language therapist. These narrative descriptions and preliminary groupings were shared among the collaborators for review and discussion. Following discussion and consensus, we identified 15 possible SLT approaches categorised from two perspectives:

1. the impairment target, for example rehabilitation of spoken language
2. the theoretical approach, for example semantic therapy.

Categories were not mutually exclusive, that is one SLT intervention could include more than one component, and thus be profiled across several categories.

Social support and stimulation of communication

Social support and stimulation interventions provided informal support and functional language stimulation to the person with aphasia, but did not include targeted therapeutic interventions that aimed to resolve participants’ speech and language impairments. Historically they have been used as an attention control in trials of SLT for aphasia or as an adjunct to SLT providing an opportunity to apply new learning in functionally relevant communicative situations.

Conventional speech and language therapy

Interventions described as ‘conventional’, ‘traditional’, ‘standard’, ‘typical’, ‘as directed by the therapist’ or ‘usual’ SLT (and for which absence of detail prevented more specific categorisation) were referred to as ‘conventional SLT’. Such interventions were often delivered in a comparative group or clinical registry. We recognise that conventional therapy in one context may not be comparable with another.

Speech and language therapy co-interventions

We did not explore the impact of pharmacological or neurostimulation (e.g. magnetic or electrical stimulation such as transcranial direct-current stimulation) co-interventions, which are not typically within the remit of therapists in the UK. Examining the contribution of these co-interventions to language recovery among people with aphasia was beyond the scope and remit of the RELEASE study. We extracted information about the presence of such co-interventions and the timing of delivery and noted this in the data analysis.

Types of outcome measures

Improved ability to use language in real-world situations is the primary goal for aphasia rehabilitation and research. People with aphasia and their family members prioritise functional language use [linked to the International Classification of Functioning, Disability and Health (ICF) components of activity and participation], alongside improvements in language ability (body function).93 Meta-analysis of functional communication measures is methodologically challenging, as they may be self-reports or proxy reports using rating scales, profiles or questionnaires. Discourse analysis, which observes the use of communication features within a sample of functional language, offers yet another approach, and debate continues about its usefulness and rigour.94,95 In trials of aphasia rehabilitation, measurement of ‘real-world’ language use and functional communication is uncommon,41 and inconsistently captured across studies.93 Measures of language impairment are considerably more common,41,96 and reflect overall language abilities (or aphasia severity) across several subtests, which typically include auditory comprehension, spoken language, reading, writing and functional language use.

Main outcome

The main outcome was change from baseline to follow-up in language performance (language recovery) on formal measures of language ability across six prespecified domains (overall language ability, auditory comprehension, spoken-language production, reading comprehension, writing and functional communication) (Box 3).

Given the multinational and multidisciplinary nature of this study, we did not predefine a list of all relevant language assessments and language versions of those measures. Instead, we established eligibility criteria that included any language assessment that (1) captured an outcome of relevance to this study, (2) was published and (3) met with the approval of the RELEASE study collaborators. Agreement was achieved on all measures included and their alignment with specific language domains. We excluded screening tools owing to their variable psychometric properties and lack of sensitivity (due to ceiling effects), and the subsequent impact that this would have on our analysis.97

Data collection time points

We extracted data on participants at recruitment or entry to the primary research study and at all available time points following this. When follow-up data were available, the time of follow-up was recorded in a continuous fashion in days since primary study baseline. When time to follow-up was given in a category (e.g. 3–6 months), the mean of this time span was taken (i.e. 4.5 months in this example) and converted to days.

Identification of data sets for inclusion

Electronic search methods

We searched the following databases from inception to September 2015 using a RCT-optimised search strategy: Cochrane Stroke Group Trials Register, Cochrane Central Register of Controlled Trials and other Cochrane Library databases (Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects and Health Technology Assessment Database), MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature, Allied and Complementary Medicine Database, Linguistic and Language Behavior Abstracts and SpeechBITE (see Appendix 1). We also reviewed all studies included in and excluded from a 2016 systematic review.41 We imposed no language restrictions.

Study records

All study records identified in the searches were screened for eligibility for inclusion. When references described potentially eligible data sets, we invited the primary research teams to confirm eligibility and (where appropriate) to contribute their IPD data sets to our RELEASE study database. We also extended an invitation to contribute through our pre-existing networks, including the Collaboration of Aphasia Trialists [www.aphasiatrials.org (accessed 5 June 2020)].

Selection of studies

One researcher (LRW or KVB) screened titles and abstracts of the records identified using the methods described above and applied the inclusion criteria. Obviously irrelevant titles or reports were excluded. Full-text copies of all the remaining data set reports were reviewed by an independent researcher (KVB, LRW or MCB). Eligibility criteria were ascertained through review of published reports for each study. When published reports were unavailable (for recently completed studies, clinical registries or similar), we clarified eligibility through discussion with the primary research team. Any disagreements about eligibility were resolved through discussion with a third reviewer.

Data collection

For all potentially eligible records identified, we used a systematic approach, starting with an e-mail to the corresponding author describing the aims and objectives of the RELEASE study, enquiring about the potential eligibility of their data set, inviting them to confirm the data set’s eligibility and to contribute the anonymised electronic data set. If there was no response, a second approach was attempted 1 month later. When e-mail addresses were invalid, or when there was no response to the second approach, we attempted contact with other members of the authorship team. If these attempts went unanswered, we recorded this as ‘no response’. All unavailable data sets and associated IPD were recorded.

We recorded all communications with primary research teams, including queries around eligibility, invitations to contribute and other correspondence. If the primary research team expressed an interest in contributing, we invited them to submit a copy of their anonymised electronic data set in an encrypted format through the Collaboration of Aphasia Trialists’ website [www.aphasiatrials.org (accessed 5 June 2020)]. We requested that all electronic data set contributions were accompanied by a data dictionary, a funder report or other reporting of that data set and findings if available. Evidence of gatekeeper (data controller) approval to share the data set with the RELEASE study collaborators and ethics approval for primary research studies were required. When the primary research team required additional permissions to share the data set for the purposes of secondary data analysis, we requested a copy. We also identified eligible IPD data sets in the public domain and extracted the relevant data for inclusion.

Data extraction

We developed and piloted a data extraction table to support systematic retrieval of data. The data items were identified with reference to our research questions and best practice in reporting of complex interventions and meta-analyses.82,84–86 (see Box 3). Information sources included published reports and communication with the primary research team to gather data that were otherwise unavailable. When possible, we confirmed data extraction with the primary research team for accuracy and completeness. Following the query resolution process, data items unreported or unavailable from other data sources were recorded as unreported. Items not relevant to that data set were marked as not applicable. For all data sets, a second researcher (MCB) double-checked the data extraction. Disagreements were resolved through discussion or the involvement of a third data extractor.

Risk of bias in included studies

We extracted information on the quality of each data set using both Cochrane quality items relating to RCTs98 and the data items from the Joanna Briggs Institute critical appraisal checklists for case–control studies, case series, cohort studies, non-RCTs (quasi-experimental studies) and analytical cross-sectional studies.99 Aspects irrelevant to our analysis were not extracted; details of the statistical analysis processes applied in the primary research were not relevant to the RELEASE study as we were working with the IPD and not summary statistics; information about exposure to the cause was irrelevant, as aphasia following stroke was a RELEASE study entry criterion; and criteria for the outcome measures used were prespecified for the RELEASE study. Similarly, items relating to data interpretation (e.g. strategies to deal with confounders) were not extracted and synthesised, as these activities reflected on the reporting of the primary research data set and not on the IPD contributed to the RELEASE study database or the planned analysis.

We considered the methodological quality of each included data set in terms of the following potential sources of bias:

• Selection bias – using reported data, we checked whether or not there was any risk of bias in the allocation of individual participants to a specific group in an included trial. In the context of RCTs, for example, we considered whether or not the randomisation sequence generated was truly random and whether or not the sequence allocation was concealed up to the time the individual was allocated to a group. In non-RCT group comparison data sets, we considered whether or not it may have been possible to randomly allocate participants across the groups.
• Performance bias – we documented any co-interventions delivered and whether or not there was any difference between RCT or non-RCT studies. Blinding participants to the delivery of SLT (or not) is almost impossible, although we did seek any examples of when it may have occurred (e.g. computer-based language stimulation vs. a non-verbal cognitive task).
• Detection bias – for all data sets with data collection at baseline and at a post-SLT intervention (or comparison group) follow-up time point, we extracted data on whether or not outcome assessors were blinded to the group, time since onset or other key information.
• Attrition bias – we examined whether or not there was evidence of a systematic difference between comparison groups in the number of dropouts (withdrawals for any reason) or number of non-adherents (those who decline to continue participation during the intervention) after study entry.
• Other biases – we considered other possible sources of bias, such as the similarity of comparison group at baseline, the primary study’s relevance to our research questions and outcome reporting bias, when outcomes collected were unavailable and did not contribute to the data synthesis.

For all possible risks of bias, we coded the studies as being at a low, unclear or high risk of bias. We also considered the impact of any potential biases on the findings using sensitivity analyses or narratively.

Data management

Data encryption and storage

All data contributed to the RELEASE study database were stored at the Nursing, Midwifery and Allied Health Professions (NMAHP) Research Unit at Glasgow Caledonian University (GCU), UK, according to the university’s information systems policy. Original primary research data sets and any additional study data were stored on a secure hard drive [a 1 terabyte IronKey™ Enterprise H300 password-protected and encrypted external hard drive (Kingston Technology Corporation, Fountain Valley, CA, USA)] with centralised management. The hard drive was kept in locked storage in a password-protected room. These data were available as data descriptors on the GCU hard drive, which restricted access to the GCU-based project management team only. The primary research data sets were preserved unmodified. Anonymised, password-protected working copies of these primary research data sets were created and stored on the NMAHP Research Unit’s designated portion of the university server. Adjustments to these working data sets did not affect the original primary research data.

Security and access

Primary research IPD data sets, data sets created for this project and analyses data sets were accessed only by the RELEASE study project management team at GCU.100 Data were not accessible to those outside the project collaboration. The project management team and collaborators governed use of the research data, and participated in data analysis plans, interpretation of the findings and preparation and review of the manuscripts based on the planned analyses. All collaborators had an opportunity to contribute to the planned secondary analyses [which were conducted in SAS^® version 2.8 (SAS Institute Inc., Cary, NC, USA)].

Data preparation for analysis

We undertook a rigorous process of preparing the IPD data sets for analysis to ensure that the data were valid, as complete as possible, consistent and uniform.

Quality assurance and verification

Each data set was cross-checked against the primary source, such as published papers, protocols and other outputs. We checked within data sets for duplicate participant data sets. We also identified duplication of participant demographics whereby participants taking part in one study went on to participate in a subsequent study led by the same research team, or were reported across dual publications (e.g. as a study group in one study but a comparison group in another). Duplicate participants were identified through (1) careful review of the supporting literature and study materials across primary research teams, (2) checking demographic data across IPD data sets contributed by the same primary research teams and (3) when possible, correspondence with the primary research team. Duplicate entries and any risk of double-counting participant data were excluded from the analysis. This was achieved by removing duplicate participant data sets from one of the data sets, and removing duplicate demographic data prior to any overview in which the participant descriptors would be double-counted. Other data were considered on a variable-by-variable and analysis-by-analysis basis.

Variable labels for data sets collected in non-English languages were translated by the contributing primary research teams. We confirmed the accuracy and completeness of translations with contributors. Following data extraction, the representative from each primary research team was sent an overview of the data items and details relating to their contributed data set. They were asked to confirm that the information was accurate and complete.

Data cleaning and mapping to unit of analysis

Data were contributed in a range of different formats such as Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA), SAS and Statistical Product and Service Solutions (SPSS) (formerly Statistical Package for the Social Sciences) [IBM Corporation, Armonk, NY, USA (IBM SPSS Statistics from version 19 onwards) or SPSS Inc., Chicago, IL, USA (version 18 and below)]. In the first instance, we cleaned all data sets to reclassify duplicated variable names, removed blank lines and transposed data into a one row per participant format (short form). We converted all data sets into SAS. Prior to analysis, data relating to prespecified variables needed for the RELEASE study analyses (see Box 3) were cleaned. These variables were decided a priori by the research and analysis team and presented to the wider RELEASE study collaboration for review and comment. We ensured that the variable entry matched an agreed uniform format to inform our planned analyses. For example, time since stroke was recorded in days, months and years across RELEASE data sets; the RELEASE collaboration agreed to record ‘time since stroke’ in the format of ‘days since stroke’. When necessary, entries as years or months after stroke were converted to days. In some cases, the record format was a date entry from which days since stroke could be calculated when date of study entry was considered. For each variable of relevance, the primary research data sets and other data sources (publications, protocols, data dictionary) were searched for that variable or indicative surrogate measures. Decisions on how to synthesise these data into a common nomenclature were based on agreement of the wider collaboration. A decision tree was used to ensure standardisation of variable names and ranges when converting data from the original data sets to the RELEASE study analysis data sets.

For some variables, the relevant data were common to all participants across a data set and not included in the contributed electronic IPD data set. For example, if the primary research entry criteria specified right-handed participants, then all participants in that data set were right-handed, whether or not this was recorded in the submitted electronic data set. Similarly, when a specific intervention was delivered to one randomised group, then (unless otherwise specified) the intervention description was common to all participants in that group. In such cases, we imputed and translated those data using one-to-many merging and in keeping with the agreed format for the RELEASE study analyses.

The collaboration then agreed on standard definitions to describe therapeutic interventions in terms of the theoretical and target approach (see Box 3). These definitions were applied to all intervention studies. In some instances, those participants who were allocated to no SLT were permitted to receive usual care SLT after the study’s active intervention period had finished. In these cases, we identified the follow-up times that corresponded to the duration of no SLT and took those observations forward for analysis in the no-SLT group. We also identified cases of usual SLT care, study SLT intervention and the corresponding follow-up time points associated with each. We identified the presence of crossover in treatment and took all measurements up to the point of crossover. These data were identified, cleaned to a standardised format and entered into the analysis data set.

According to the needs of the analysis, data-cleaning processes included examining the data set for validity of contents, such as receipt of all relevant participants’ data; matching numbers of IPD transferred with those reported in publications; matching variable names to known published assessments and ascertaining any missing domains of assessment from the original data set; examining the content of fields to ensure that they matched prescribed minimums and maximums for known assessments; ascertaining any transformations of data that had occurred prior to transfer and resolving such queries [e.g. use of a mix of t-scores and raw scores in the Aachen Aphasia Test (AAT) across studies]; examining odd values in a given range (e.g. data sets containing a zero value for assessments when this indicated that no assessment had been conducted); and converted data into a numeric format.

Data integrity and validation

Ensuring data integrity was essential to the project. We audited the data sets using statistical and database methods to detect anomalies and contradictions. For example, we checked the validity and plausibility of contributed data, specifically ensuring that data entries were within the acceptable ranges of variables and outcome measures used in the primary research. When this was not the case, we cross-checked the data with all study records or publications, and communicated with the primary research teams to confirm extreme outliers or other questionable values.98 We examined distributions and descriptive statistics to audit contributed data and identified irregularities or inconsistencies that required further clarification. These were manually checked with the source data, the data report (if available) and the SAS file.

We checked for any imbalance at baseline between individuals included in RCT or non-randomised groups. Working with IPD at the study level, we used the Wilcoxon signed-rank test to look for significant between-group differences in baseline characteristics, including age, sex, years in education, handedness, time since stroke onset (days) and aphasia severity. For comparisons of sex and handedness between groups, we used the chi-squared test.

Measures of treatment effect

We identified language outcome measures and worked with outcome data, transformed to the anchor measure, for each domain (see Anchor and minority measures and Data transformation). We defined measure of treatment effect as the mean of the absolute change from baseline on transformed measures for each of the domains of interest. Effect sizes were estimated and reported with 95% CIs and significance was set at the 5% threshold throughout.

Sample size

We prespecified that each analysis would be based on a minimum of two data sets. As our eligibility criteria dictated that each data set would have a minimum of 10 IPD, our minimum sample size was 20 people with aphasia after stroke.

Data conversion and reformatting data files

Data were converted and stored in SAS version 9.4 format for ease of management.

Data synthesis

Most SLT interventions were described at the primary research study or group level, and only rarely were aspects of that intervention (e.g. the duration or dosage) reported at IPD level in an electronic data set. SLT intervention data were therefore extracted from narrative descriptions in the primary research protocol, electronic data set, published report or through discussion with the primary research team.

A separate data set was generated for each of the following:

• study description and participant demography
• outcomes describing the domains of interest
• SLT intervention and language rehabilitation (and co-interventions when present).

Each data set contained the unique identifier (ID) for constituent participants. This meant that multiple data sets could be linked using this ID to generate a complete picture of the participant’s profile across multiple data sets, and all unique participants could be brought together for each of the planned analyses.

Anchor and minority measures

We prespecified several language domains or outcomes for analysis in the RELEASE study:

• overall language ability
• auditory comprehension
• spoken-language production (including naming)
• reading comprehension
• writing
• functional communication.

For each of these outcomes, a range of assessments (sometimes in various language adaptations) was used. To synthesise the IPD from across these multiple measures, we first identified the most commonly used measurement for each outcome. We profiled all measures used to capture data on that outcome, the number of studies using that measure and the number of IPD available, and calculated the median score, interquartile range (IQR), minimum and maximum for each measurement tool. Language or version variations were treated as separate tools.

An ‘anchor measure’ was identified for each language domain (e.g. reading) as the measurement instrument used by the most data sets in the RELEASE study database. When this approach resulted in two possible anchor measures, the assessment that had the greater number of IPD in that language domain was identified as the anchor measure. All other assessment tools measuring the same domain, but which were informed by smaller numbers of data sets, were considered ‘minority measures’. Minority measures were then transformed to match the format and range of each anchor measure (one for each language domain) through an adapted version of the method previously used in large-scale data synthesis (see Appendix 2, Box 4).

Data transformation

Our transformation process was based on the underlying assumption that the anchor and minority measures for a described outcome were similar.101 The transformations were based on an investigation of the medians, IQRs, minima and maxima of both the anchor and minority measures according to an algorithm (see Appendix 2, Box 4). A linear transformation from each minority measure to the anchor measure was applied to each quartile and repeated for each language outcome. Each threshold in the minority instrument was mapped to the corresponding threshold in the anchor measure. Values in the minority measure that fell within a quadrant were mapped to the anchor measure relative to the distance from the quadrant edges. The formula was repeated for each quadrant. In this way, the clinical relevance of changes in scores on the anchor measure was retained.

Three alternative approaches to transformation (normalising, internal normalising and direct linear) were considered but rejected. Normalising transformations (e.g. van der Waerden102) requiring pooled ranks across the measures would not be feasible owing to the differences in the width of the various scales. For both normalising and internal normalising approaches, the clinical meaning of any scores or change of scores would be lost. Direct linear transformation would retain the anchor measure values and, thus, clinical relevance. However, it would require mapping the distribution of the minority measures to the anchor measures without retaining the distribution of the anchor measure itself and, as the measures are often naturally skewed in different directions, this would affect the interpretability.

For each of our analyses, values were generated first for the anchor measure and then for the minority measures. Checks were performed to assess whether or not the ranges, values and directions of changes observed for each anchor measure were consistently reflected when transforming each minority measure into the anchor measure.

Meta-synthesis methods

Four research questions were addressed in the context of the RELEASE study, in line with the study objectives (see Box 1). The RELEASE study database supported investigation of (1) the progression of language recovery over time following aphasia after stroke, (2) the individual factors, such as demographic and aphasia profile, that potentially play a role in language recovery over time following aphasia after stroke, (3) the components of effective rehabilitation interventions that best facilitate recovery over time following aphasia after stroke and (4) whether or not certain interventions (or intervention components) are more beneficial for some participant subgroups than others.

Meta-analysis and network meta-analysis

Meta-analyses of aphasia intervention research already exist and have provided evidence-based review and data synthesis of the clinical effectiveness of SLT for aphasia after stroke (e.g. Brady et al.,41 Robey63 and Bhogal et al.64). Methodologically, these meta-analyses are usually based on the data synthesis of aggregate group-level summary data. Large group summary approaches taken by RCTs and meta-analysis of RCTs have limitations, particularly in relation to the risk of ecologic bias; aggregate data generated across a complex participant population, such as people with aphasia, risk concealing the possibility of varying individual response to a complex rehabilitation intervention.76

Traditional meta-analysis is limited to direct between-treatment (or control) comparisons and group-level descriptors of interventions and aggregated group data on the treatment effects.103 A more detailed examination of the effectiveness of a specific therapy component is not possible owing to the variability of complex SLT across data sets. Similarly, such approaches cannot accommodate or control for participant demographics or other factors relevant to recovery.41 Network meta-analysis approaches, however, allow for an estimate of the difference between two (or more) treatments that are not directly compared. For example, studies comparing A versus B or A versus C (direct comparisons) can be combined to give an estimate of B versus C (indirect comparison).104

Individual participant data network analysis approach

Network meta-analysis can be applied to aphasia intervention research using group-level summary statistics,38 but this approach does not address the limitations of ecological bias, highlighted in the previous section. Confounding is also a risk. We conducted a network meta-analysis based on a large IPD database, which allowed us to explore the subtleties of a highly heterogeneous group of people with aphasia after stroke, control for individual predictors and support the detailed exploration of the influence of participant-level covariates on SLT treatment effects across a range of language-specific measures.104 Meta-analyses based on aggregated data are typically unable to examine such interactions. In addition, we were aware of some studies that had recorded valuable information on SLT intervention at IPD level.

One-stage versus two-stage approach

There are two main approaches to an IPD network meta-analysis: a one- or two-stage approach.104 Two-stage approaches are very similar to the standard meta-analysis approach, working first with IPD to centrally generate aggregate data for each contributed data set (rather than using the primary research team’s reported summary statistics). These aggregate data could then be meta-analysed with summary statistics from other trials for which only aggregate data were available. This approach can, however, lead to bias in effects, greater heterogeneity and lower power to detect associations between language outcomes and continuous variables.104 It is also very similar in methodology to existing reviews and meta-analyses of summary data.

In contrast, a one-stage approach would allow all available IPD from across the contributed data sets and IPD retrieved from the public domain, filtered for relevance to each analysis, to be combined into a single model (which considered the clustering of IPD within a study). One benefit of this methodological approach is that it can address confounding by enabling the examination of the impact of several participant and language variables on an intervention effect at the same time.

Network meta-analysis

We performed our one-stage network meta-analysis in SAS, using PROC MIXED, with fixed demographic and treatment effects, and study as a random effect. The variance structure was allowed to be unstructured. Study homogeneity was assessed by examining the variance parameters. Network diagrams for each treatment factor were drawn up. Each comparison within each domain was illustrated by a network graph, and supplemented by a table of the number of available IPD in each RCT and treatment node. We used the graphs to explore the balance of the networks and whether or not all levels in a treatment were compared across and within studies. Each comparison in each domain was illustrated by a network graph, and supplemented by a table of the number of available IPD in each RCT and treatment level. Feasibility was contingent on the number of trials and IPD available.

The populations in each data set were expected to differ, and so a base model of age, sex, time since stroke and baseline score was created for each language domain. We also planned to examine interactions between demographic characteristics and treatments, should sample sizes be sufficient. Given the number of tests likely to be undertaken, we were aware of the high risk of false positives. However, the aim of the RELEASE study exploratory analysis was not to generate definitive answers, but to use inference to suggest priority avenues for future research; therefore, we were more concerned with estimated effect sizes than statistical significance.

Primary analysis

We used RCT data as the gold standard for each planned analysis, with only participants from RCTs included in the primary analyses. Analyses that included data from non-RCTs, cohort/case series studies, observational studies, registries and other study design types (as relevant) followed the primary analysis, to judge whether or not they supported the RCT findings (see Appendix 3, Figure 26). When they did not agree, the data were further explored to quantify this difference.

To maintain the independence of subjects, we ensured that IPD did not appear in more than one data set for intervention type comparisons, although they could appear in more than one data set when grouped by language domains (e.g. overall language ability, auditory comprehension, reading) (see Appendix 3, Figure 26).

Patterns of language recovery

We examined language recovery among people with aphasia across the prespecified language domains: overall language ability, auditory comprehension, naming, other spoken-language production, reading comprehension, writing and functional communication. Analyses were based initially on RCT data, and then expanded to include all study design types (non-RCT group comparisons, cohort, case series and registry data sets).

Descriptions of the patterns of aphasia recovery were generated using four methods across each language domain. As data availability permitted, we described the:

1. natural history of recovery among participants that had no access to SLT
2. trajectory of language recovery over time among participants by assessing their baseline and follow-up scores
3. distribution of language domain scores for participants at study entry
4. absolute proportion of change in language scores from study entry to first follow-up across all domains.

Natural history of language recovery

We defined the natural history population as a population for whom no active SLT intervention was present, and for whom there was no history of previous SLT. People who were allocated to the no-SLT group in a study were typically permitted to receive usual care SLT after the intervention period of the study had been completed. Any SLT intervention could have an impact on the post-intervention follow-up measures. Depending on the time since the onset of their aphasia, they may also have had historical SLT prior to entering the study as part of usual care. We therefore implemented the following restrictions when extracting eligible IPD to support the examination of the natural history of aphasia recovery:

1. Allocation to the no SLT group in an intervention study.
2. Enrolment to the study with baseline assessment commencing within 15 days of index stroke, to reduce the possibility of unreported initiation of SLT as part of usual care.
3. No history of prior stroke.
4. When longer-term follow-up measurements were available, we reported on the first post-intervention follow-up only, when applicable, to eliminate all instances of access to SLT as part of usual care SLT after the study intervention period, as identified in our data extraction.

We plotted baseline language domain scores along with corresponding language domain values that were available at first follow-up, to examine the trajectory of the natural history of language recovery over time in those participants who were allocated to no-SLT groups.

Language recovery in the context of speech and language therapy

We also examined the trajectory of language recovery among participants who had access to SLT; we identified those who had access to SLT either as a study participant (SLT group) or reported (or reasonable grounds to assume) historical access to SLT as part of standard health care prior to study enrolment (historical SLT group). We also identified those who were allocated to the historical SLT group (with enrolment beyond 15 days post stroke to exclude those who were part of the natural history analysis) and assessed whether or not there were sufficient data available at the first follow-up period. We sought to eliminate the impact of any SLT intervention (even usual care) in any follow-up data collection time points.

We described baseline and follow-up scores in all available groups for each language domain, when sufficient data were available. Data were presented as plots of language domain scores over time (relative to study entry) and, when applicable, stratified by SLT, no SLT or historical SLT intervention.

Language recovery: change in language scores from baseline by time since stroke

We examined the role that time since index stroke and study enrolment played on the proportion of recovery observed for each domain score. The study enrolment period in the primary data set was stratified into the following groupings:

• 0–1 month
• > 1–3 months
• > 3–6 months
• > 6 months following index stroke.

We ascertained whether or not participants were allocated to receive a SLT intervention; when possible, we described the absolute proportion of change observed for SLT versus historical SLT (excluding those who were assessed as part of the natural history analysis). We calculated the proportion of change (%) in each language domain score from baseline to first follow-up and plotted these proportion changes by study enrolment period (as categorised earlier). This was described for the RCT population, and then for all study types. We plotted each language domain separately to visualise the proportion of change observed in each domain at different study entry time points. We opted not to stratify data according to baseline severity stratum in combination with aphasia chronicity, as this would have resulted in very small strata and much wider confidence limits, thus precluding informative analyses. We also accounted for the skewness of data measuring language impairment by opting to present medians and IQRs, rather than means and standard errors of the means.

Distribution of language scores at study entry by time since stroke

All participants were selected from the total population of people with aphasia after stroke. We identified all data available at study entry for each language domain. This time point was selected as all participants were assessed prior to receiving any study intervention. We then ascertained the time of assessment relative to stroke onset. We plotted all transformed language domain observations available at study entry against time of study entry, relative to stroke onset, for each language domain. We then fitted a regression line to these observations and conducted linear regression analyses to describe whether there was a significant trend towards increased or decreased language domain scores with later time to study entry. We considered whether or not the baseline data reflected the trajectory findings based on participants’ language scores over time (see Chapter 4). We considered the distribution of language scores at clinically relevant time points (e.g. study entry between 0 and 1000 days since stroke). Finally, we considered whether or not there was any suggestion of selection bias in recruitment to aphasia studies based on participants’ language scores over time since stroke.

Predictors of language recovery

Demographic and stroke covariates identified as statistically significant from the patterns of language recovery analysis (see Appendix 3, Figure 26a) informed our primary statistical analysis of the predictors of language recovery over time. We considered evidence informing other areas of stroke rehabilitation and common assumptions in the context of aphasia rehabilitation.

Whether or not participants had received SLT in the primary research study (SLT/historical SLT or no SLT) was also included in the analyses (see Appendix 3, Figure 26b). Language recovery on a continuous scale was regressed onto the covariates of interest. Initially, each predictor was checked using univariable analysis; covariates significantly associated with the primary outcome at a univariable level (i.e. a p-value of < 0.1) were included in the multivariable regression analysis. Thereafter, covariates were introduced and retained in the final multivariable model only if found to be statistically significant. Each analysis included the study ID as a random covariate, to account for possible variability in the outcome measure between studies.

Participants’ stroke profiles, demographic characteristics and living contexts at baseline and follow-up (when available) were explored graphically and using summary statistics by showing means and standard deviations for continuous and count data (or median and IQR if data were skewed), and frequency and percentages for binary and categorical data. Standard statistical tests (e.g. chi-squared tests) were employed to check formally for statistical differences.

Factors associated with language score changes: age, sex and time since stroke

We performed regression, modelling a trend towards increasing proportion of change, and including age, sex and time since stroke (days) as covariates in this model, to ascertain whether or not there was a relationship between age, sex and time since index stroke on the degree of recovery observed.

Distribution of language scores at study entry: age, sex and living context

Building on the earlier analyses, we identified all participants’ data from across all data sets for the specified language domains at study entry in advance of any study-level intervention. Using the data on time since stroke, we plotted the language scores and explored their distribution across different age groups (≤ 55, 56–65, 66–75 and > 75 years), sex and living contexts.

Optimum approach to speech and language therapy for aphasia

The demographic and stroke covariates identified as statistically significant in earlier analyses were used to create the baseline model to consider the components of effective rehabilitation interventions for people with aphasia after stroke. Once the model was built, additional SLT variables were examined for impact on the primary outcomes (see Appendix 3, Figure 26c). Data were analysed as mixed-effects models, with data set as the random effect. Components of therapy (see Box 3) were considered on a continuum and explored for meaningful categories.

Beneficial speech and language therapy approaches to language recovery for participant subgroups

We explored whether or not some interventions (or intervention components) may be more beneficial for some participant subgroups than others. When demographic/stroke variables were significant to the model, a series of one-stage subset analyses were carried out to explore whether or not certain participant groups responded better to certain therapy interventions (see Appendix 3, Figure 26d). We also examined participant clustering within trials, which would allow us to distinguish individual participant-based interactions from any data set-based interactions.103

Risk of meta-biases

We considered not only the risk of bias in relation to the primary research data sets (see Risk of bias in included studies), but also how bias might be a risk in how we conducted our IPD meta-analyses and network meta-analyses. We used the Risk Of Bias In Systematic reviews (ROBIS) tool framework to structure our deliberations on our approach and possible risk of bias; for consideration of the potential risk of bias in the RELEASE study meta-analysis and network meta-analysis, see Chapter 8.105 Risks of selection bias, publication bias and availability bias were among those considered (detailed in the following sections) by a single researcher, and reviewed and discussed with an independent researcher. Consideration was given to any concerns relating to the specification of study eligibility (risk of selection bias), methods used to identify and select studies (risk of selection bias, publication bias), the methods used to collect data (availability bias) and appraise studies and any concerns with the synthesis and findings of our work.

Selection bias

We incorporated a systematic review component as the first stage to building our IPD database and actively invited data set contributions from primary research teams that we had no prior collaboration with. We also identified and extracted IPD reported in the public domain, including the grey literature, from primary research teams with which we had no contact. We considered the possible impact of potentially eligible but unavailable IPD data sets.106 When relevant, we tested the comparability of groups at baseline.

Publication bias

Through our rigorous approach to identification and selection of eligible data sets, we identified IPD that were conducted across a range of languages, unpublished or reported in the grey literature, thus reducing the risk of reporting bias.107 Where the limited availability of suitable data sets rendered a funnel plot unfeasible, we examined risk of publication bias by looking at the distribution between sample size and those studies that reported significant findings. It was not possible to look at publication bias in the unpublished registry data sets.

Availability bias

We considered the impact of potentially eligible data sets that were unattainable, unavailable to our study or for which the researchers were uncontactable.

Other biases

We considered the possible risk from other sources of bias, including relevance of the primary research study’s objectives. We used the ROBIS tool to consider to what extent the RELEASE study meta-analysis and network meta-analysis were at risk of bias.105

Missing data

Not all included data sets had collected (or had available) data on all variables of interest to our analyses. In such situations, the analysis was restricted to the subset of data sets with available data on the variable. However, when primary research materials indicated that a data item was collected but that it was unavailable in the contributed electronic data set, we attempted to extract that data from alternative sources, including:

• retrieving the data from the text descriptions of published manuscripts or reports
• retrieving the data from unpublished materials
• requesting the missing data items from the primary research team.

If, despite our efforts, specific variable data remained unavailable, then that data item was marked unavailable (labelled in the data set as unreported). If a variable had > 20% missing data from the primary research data set, this variable was excluded from the primary analyses.

Missing completely at random

Data were examined for patterns of loss or whether or not the data were missing completely at random by recoding the variable of interest into ‘missing’ or ‘not missing’. This variable was compared with demographic variables that may have had an influence on recording that information (e.g. participants’ age, sex, stroke), and then with the other variables available. We used the t-test for comparisons of missing data with continuous variables such as age (or the Mann–Whitney U-test, depending on the normality of the distribution). When comparisons involved categorical variables, we used the chi-squared test to indicate whether or not the missing data were substantially different from the data reported (not missing). When there was no evidence of an influence, the data were treated as if assumed to be missing completely at random. Data missing at random were excluded from our modelling and analysis.

Assessment of heterogeneity

We considered the degree to which there was any variability in the data sets gathered and synthesised in our analysis in relation to clinical, methodological or statistical heterogeneity. We considered the impact of between-study methodological differences such as study design, randomisation and blinding of outcome measurement in the context of risk of bias (see Chapter 3). Between-study differences in the choice of outcome captured or the measurement tool employed were accommodated in the data extraction and synthesis approach, which restricted data extraction to published assessment tools. We sought agreement across our international collaboration on the categorisation of outcome measurement tools and the transformation of data to an anchor measure. The objectives were to explore the impact of a wide range of SLT interventions, but, for each analysis, we collated, synthesised, analysed and presented the relevant data separately, ensuring that each meta-analysis was homogeneous in terms of participants, therapy and outcomes.98

The analysis was not a standard meta-analysis of a series of RCTs in which the treatment effect to be measured was captured using the same tool across all eligible RCTs; thus, the standard assessments of heterogeneity (e.g. the I²) were unsuitable for our meta-analyses purposes and mostly not calculable. Instead, we assessed heterogeneity by examining the relative size of the variability due to study differences to the overall variability in the data. Analyses with > 25% of variability due to data set differences underwent careful examination of the contributing data sets to ensure that no data set was having an undue influence on the findings and that the groupings (e.g. the thresholds for dosage) were reasonable and not highly unbalanced. When a contributing data set did have undue influence or groupings were unbalanced, the thresholds were adjusted. When the relative variability exceeded 50%, then the findings were highlighted as unreliable and were not taken forward or presented.

For the network IPD meta-analysis, we undertook a series of intervention comparisons within and between data sets, also known as direct and indirect comparisons. Direct comparisons would occur in the context of a single data set for which treatment A was compared with treatment B and another data set compared treatment B with treatment C. As previously described, using network meta-analysis allowed us to make comparisons between data sets; thus, we were able to directly compare treatments A and B, B and C and indirectly compare A with C. We made every attempt to ensure that, in developing the networks, we had no orphan interventions (an intervention that appears in only one study and has no direct comparison with any other intervention).

Sensitivity analyses

We conducted a series of sensitivity analyses to examine the robustness of any data synthesis decisions on the findings.

Choice of language measurement

We explored the impact of our prespecified choice of assessment tool in situations in which a data set had two or more assessment tools eligible for synthesis in a single language outcome. We considered the impact on findings had the alternative assessment been selected to go forward to data synthesis.

Exclusion of minority measures

We considered what impact the exclusion of the minority measures had on the findings. To examine the contribution that each language assessment tool made to the transformed language measures, we plotted the distribution of transformed language measures over time, stratified by the original assessment tool and presented by language domain.

Random or fixed effects

We examined the impact of including the data set identifier as the random effect or as a fixed effect using the Wu–Hausman test.108 For simplicity, the comparison was based on a simple model of language change by baseline. The model was run twice, once with data set as a fixed effect and once with data set as a random effect. The differences between the estimates of the covariate and the standard error of the baseline score were used to calculate both the Wu–Hausman and the alternative Wu–Hausman for one parameter. If the W-score was < 3.841 (≈ χ² with 1 degree of freedom), or the alternative W was < –1.96 or > 1.96 [≈ N(0,1)], then we rejected the null hypothesis that random effects were more efficient.

The inclusion of historic data sets

We explored what bias there may have been in the inclusion of historic data sets (e.g. collected prior to the establishment of specialist inpatient stroke units), when approaches to the provision of stroke care differed.109,110 Response to treatment may be influenced by overall health and improvements in stroke care, such as thrombolysis, which has been linked to better aphasia recovery.29 Similarly, a response to SLT for aphasia depends on the therapy provided, and such approaches can change over time (as suggested by Godecke et al.56,72). We conducted a sensitivity analysis (RCT data sets only) to consider whether or not the recruitment date had any impact on recovery patterns by language domain.

Excluding non-randomised controlled trial data sets

We explored the impact of excluding data sets graded as being at a high risk of bias, in particular the non-RCT data sets, from the analysis. In each analysis (except the planned subgroup analysis; see Subgroup analysis), we analysed and presented the IPD gathered in the context of RCT studies before then conducting the analysis on the wider data set inclusive of non-RCT designs.

Registry data sets and impact on findings

We explored the impact and contribution that registry data sets (clinical and other research archives contributed to the RELEASE study) made to the analysis and considered whether or not these data sets may have altered the findings.

Subgroup analysis

As time since stroke, age and stroke severity are likely to affect language recovery, we conducted subgroup analyses of these variables, in addition to including them as covariates in the modelling process.

Confidence in cumulative evidence

We reviewed the quality of the data contributing to our analyses together with the quality of our meta-synthesis and analysis approaches and considered the impact this may have had on our confidence in the cumulative results. When appropriate, one researcher independently applied the GRADE approach111 or similar tools to suggest a level of evidence.

Summary

We established an international database of IPD gathered from pre-existing primary research and clinical data sets identified through a systematic review process. Contributions were invited and additional data sets were identified in the public domain. Language measures were transformed to a single anchor measure for each language domain. We explored patterns of language recovery over time among people with no access to SLT, access to historical SLT or SLT in the context of the primary research and participants’ language performance at study enrolment. We explored the data for predictors of language recovery after aphasia. We undertook a one-stage IPD meta-analysis, incorporating network meta-analysis, to explore the impact of complex SLT interventions on language recovery for people with aphasia after stroke. We also considered whether or not specific therapy approaches were associated with the best language recovery outcomes for specific subgroups of people with aphasia.