About this chapter
Parts of this chapter have been reproduced or adapted with permission from Meiksin et al.1 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes additions and minor changes to the original text.
Parts of this chapter have also been reproduced or adapted with permission from Meiksin et al.57 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes additions and minor changes to the original text.
Parts of this chapter have also been reproduced or adapted with permission from Melendez-Torres et al.58 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes additions and minor changes to the original text.
Research design overview
The aim was to conduct a multimethod systematic review of intervention types, theories of change, processes and outcomes, and cost-effectiveness of e-health interventions employed to prevent HIV and STIs, sexual risk behaviour, alcohol and drug use or common mental illness symptoms among MSM. The review followed criteria for the good conduct and reporting of systematic reviews [e.g. the guidance from the Centre for Reviews and Dissemination (CRD)59 and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)60], and the PRISMA checklist61 is provided. The protocol was registered with PROSPERO (International Prospective Register of Systematic Reviews) (see Report Supplementary Material 1).62
Our review and synthesis of intervention descriptions and theories of change enabled us to categorise intervention types and describe theories of change (RQ1). Our review of process evaluations enabled us to identify what characteristics of interventions, providers, participants and contexts tended to facilitate or limit implementation and receipt (RQ2). Our review of outcome evaluations enabled us to estimate the effectiveness of interventions (RQ3). Our review of economic evaluations was designed to enable us to estimate the cost-effectiveness of the interventions (RQ4). Synthesis across these elements informed assessment of the value of developing an e-health intervention targeting UK MSM (RQ5).
Inclusion criteria for this review
Types of participant
The review focused on gay, bisexual and other men (including trans men) who have sex with men, including those who have been diagnosed as HIV positive, as well as those whose last HIV test was negative or who have never tested for HIV. To determine study eligibility, we operationalised this focus by requiring that MSM constituted at least half of the sample.
Types of intervention
The review focused on interactive or non-interactive e-health interventions delivered via mobile phone applications (hereafter referred to as ‘apps’), the internet or other electronic media (i.e. electronic communication technology) that aimed to provide ongoing support to populations consisting entirely or principally of MSM to prevent HIV, STIs, sexual risk behaviour, alcohol and drug use or common mental illnesses. These could include interventions that also aimed to promote HIV treatment adherence or that addressed HIV testing or pre-exposure prophylaxis, as long as these also addressed sexual risk behaviour, substance use behaviour or mental health. It excluded e-health interventions facilitating merely one-off, as opposed to ongoing, support and those addressing HIV self-testing, clinic attendance or STI partner notification only. The e-health interventions were electronically delivered; interventions delivered by human providers via electronic media were excluded. The interventions could be distributed by commercial, statutory, academic or voluntary sector agencies.
Types of control
The review focused on treatment as usual, no treatment or other active treatment control groups.
Types of outcome
The review focused on HIV or STIs, sexual risk behaviour, alcohol consumption (e.g. self-reported alcohol consumption via questionnaires or diaries), legal and illegal drug use (e.g. self-reported drug use) and anxiety or depression (clinical or self-report measures). Studies were included if they addressed any, some or all of these outcomes. Outcome measures could draw on dichotomous or continuous variables, and self-reports or reports by other raters. Behavioural outcomes could use measures of frequency (monthly, weekly or daily), the number of episodes of use or an index constructed from multiple measures. Alcohol measures could examine alcohol consumption or problem drinking. Drug outcomes could examine drugs in general or specific illicit drugs, and could include drug convictions.
Types of studies
To address RQ1, we included process and outcome evaluations providing intervention descriptions or theories of change, as well as theoretical reports. To address RQ2, we included process evaluations. To address RQ3, we included outcome evaluations. To address RQ4, we included economic evaluations. To address RQ5, we drew on all of the above. Included theoretical reports described intervention theories of change, logic models or mechanisms of effect. Included process evaluations could employ any quantitative and/or qualitative design, but were required to report empirically how delivery or receipt varied by characteristics of intervention, provider, user or context using quantitative and/or qualitative data. These studies could report exclusively on process evaluations or report process alongside outcome data. Included outcome and economic evaluations employed prospective experimental or quasi-experimental control groups.
Search strategy
Database search strategy
The search string used in a limited preliminary search in PubMed (see Appendix 1) informed the development of a more sophisticated search strategy (see Appendix 2), thereby maximising sensitivity, as recommended by the Cochrane Handbook for Systematic Reviews of Interventions.63
Search terms
A draft search strategy was compiled in the OvidSP MEDLINE database by an experienced information specialist (JF). The search strategy included strings of terms, synonyms and controlled vocabulary terms (when available) to reflect two concepts:
- concept 1: MSM
- concept 2: e-health.
These concepts were combined using the Boolean operator ‘AND’. We did not use outcome terms in our searches as these are likely to miss studies reporting non-significant effects on our outcomes, and therefore bias the review. Our searches involved different free-text and controlled vocabulary terms for each of these concepts, linked using ‘OR’. The combination of these concepts was considered specific enough to include all available studies regardless of study design. We restricted the searches by date (from 1995 onwards, as e-health interventions were unavailable prior to this), but not by language or publication type.
Key search terms were determined by using published strategies covering lesbian, gay, bisexual, transgender, queer or questioning (LGBTQ+) populations and e-health interventions, and tested using a systematic approach. Our LGBTQ+ search strategy derived from a search put together by Parker et al.64 Our e-health search strategy derived from a search published by Thabrew et al.65 Our approach to testing our search strategy was informed by the technique described by Bramer et al.66 Our search strategy was refined with the project team until the results retrieved reflected the scope of the project. The agreed OvidSP MEDLINE search was adapted for each database to incorporate database-specific syntax and controlled vocabularies. Full details of the search strings used for each database can be found in Appendix 2 and are available at the London School of Hygiene & Tropical Medicine’s Data Repository.67
Databases
The following databases were searched between 23 October 2018 and 26 November 2018 (see Appendix 2), with searches updated between 22 and 27 April 2020 (see Appendix 3). Owing to the COVID-19 lockdown in the UK and a lack of access to libraries, some databases were not included in the updated searches: OvidSP Health Management Information Consortium (HMIC), ProQuest Applied Social Sciences Index and Abstracts (ASSIA), ProQuest Sociological Abstracts and ProQuest International Bibliography of the Social Sciences (IBSS).
- ProQuest ASSIA (1987–current, as of 29 October 2018). Owing to lack of access, this database search was not updated in April 2020.
- Campbell Library (complete database as of 27 April 2020).
- EBSCO Cumulative Index to Nursing and Allied Health Literature (CINAHL) Plus (complete database as of 22 April 2020).
- Wiley Online Library The Cochrane Library (complete database as of 27 April 2020).
- CRD databases. Since 2015, the Database of Abstracts of Reviews of Effects (DARE) and the NHS Economic Evaluation Database (NHS EED) are no longer updated. The Health Technology Assessment (HTA) database was updated to 26 October 2018. These databases had not been updated, and so were not searched in 2020.
- Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) database of health promotion research (Bibliomap) (full database as of 27 April 2020).
- ProQuest Dissertations & Theses Global (1951–current, as of 27 April 2020).
- OvidSP EconLit (1886 to 16 April 2020).
- OvidSP EMBASE (1980 to 21 April 2020).
- OvidSP Global Health (1910 to 2020 week 15).
- OvidSP HMIC (1979 to July 2018). Owing to lack of access, this database search was not updated in April 2020.
- ProQuest IBSS (1951–current, as of 29 October 2018). Owing to lack of access, this database search was not updated in April 2020.
- Ovid MEDLINE ALL (1946 to 21 April 2020).
- OvidSP PsycINFO (1806 to April week 2, 2020).
- Web of Science Science Citation Index Expanded (1970–present, data last updated 21 April 2020).
- Elsevier Scopus (complete database as of 22 April 2020).
- OvidSP Social Policy & Practice (as of 2020).
- Web of Science Social Sciences Citation Index Expanded (1970–present, data last updated 21 April 2020).
- ProQuest Sociological Abstracts (1952–current, as of 29 October 2018). Owing to lack of access, this database search was not updated in April 2020.
These databases were selected to retrieve research literature from the fields of health and social sciences. We amended the list of databases that were originally intended to be searched (see Appendix 4) on the advice of, and informed by initial pilot searches by, the information scientist (JF).
Search strategy for other literature sources
The following clinical trials registers were searched for relevant ongoing and unpublished trials:
- ClinicalTrials.gov (complete database as of 27 April 2020).
- World Health Organization International Clinical Trials Registry Platform (ICTRP) (complete database as of 26 November 2018). Owing to the COVID-19 pandemic, the ICTRP search functionality was removed. Therefore, this search could not be updated.
- EPPI-Centre Trials Register of Promoting Health Interventions (TRoPHI) (full database as of 27 April 2020).
Search terms were derived from the OvidSP MEDLINE search compiled for database searching. All trial details were examined for their relevance and included if they met our inclusion criteria.
To find other grey literature, the complete OpenGrey database was searched on 1 November 2018 and again on 27 April 2020, using a version of the OvidSP MEDLINE search compiled for database searching. Google (Google Inc., Mountain View, CA, USA) was searched on 21 November 2018 in incognito mode to look for non-governmental organisation and governmental publications. Search terms were derived from the OvidSP MEDLINE search compiled for database searching. The first 100 results for each search were examined for their relevance and included if they met our inclusion criteria. The Google search was not updated in 2020.
Full details of the search strings used for these sources can be found in Appendix 2 and Appendix 3 and are available at the London School of Hygiene & Tropical Medicine’s Data Repository.67
We also carefully searched reference lists from all studies that met our inclusion criteria. We contacted subject experts during the initial search and again at the time of our updated search to identify relevant ongoing or completed research (see Appendix 5). Our protocol specified that we would hand-search journals that published included studies that were found only via reference checking, and which were not indexed on databases we had searched, but no journals met this criterion.
Information management and study selection
All citations identified by our searches conducted in 2018 were uploaded to EndNote [Clarivate Analytics (formerly Thomson Reuters), Philadelphia, PA, USA] for duplicate removal. Duplicates were identified and removed using an established technique.68 Deduplicated results were then uploaded to EPPI-Reviewer (version 4.0) (EPPI-Centre, University of London, London, UK). The updated search results retrieved in 2020 were uploaded to the same EndNote library, where citations already identified in 2018, and duplicates found within the results of the 2020 search, were removed. Any citations published before 1995 were also removed. Deduplicated results were then uploaded to EPPI-Reviewer.
To inform screening, an inclusion criteria worksheet with guidance notes was prepared and piloted by two reviewers screening batches of the same 50 references (as allocated automatically by Eppi-Reviewer). When the two reviewers disagreed, they met to discuss this and, if possible, reach a consensus. When the reviewers were unable to reach consensus regarding the inclusion of a specific article, judgement for selection was referred to a third reviewer. We planned to organise translation of reports published in languages in which no reviewers were proficient, but this issue did not arise. After this piloting process achieved an agreement rate of at least 95%, each reference was henceforth screened on the basis of title and abstract for potential inclusion by one reviewer. Full reports were obtained for those references judged as meeting our inclusion criteria or for those for which there was insufficient information from the title and abstract to judge inclusion. A second round of screening with a comparable piloting process then occurred, focused on full study reports, to determine which studies would be included in the review. We maintained a record of the selection process for all screened material.
Data extraction
Two reviewers independently extracted data from, and assessed the quality of, theory, process, outcome and economic reports meeting our inclusion criteria using existing tools.63,69,70 When the two reviewers disagreed, they met to discuss this and, if possible, reach a consensus. If the reviewers could not reach consensus regarding the particulars of data extraction for a specific study, judgement was referred to a third reviewer. Included studies were described using the EPPI-Centre classification system for health promotion and public health research,71 supplemented by additional codes developed for this review. For reports included in the theory of change synthesis (henceforth termed ‘theory reports’), we extracted data on the constructs and mechanisms described, the evidence presented in support of the theory of change and how the theory of change was developed. Intervention theories of change typically draw on existing scientific theories of behaviour (which consider factors that predict behaviours) and/or existing scientific theories of behaviour change (which propose general mechanisms of changing behaviour). We extracted data on intervention theory of change, as well as the existing scientific theories of behaviour and behaviour change that informed these. For all empirical studies, we extracted data on basic study details (target population, study location, timing and duration, research questions or hypotheses), methods (design, sampling and sample size, data collection and analysis) and intervention description (timing and duration, programme development, content and activities, providers, details of any intervention offered to the control group). For process evaluations, two reviewers used an adapted version of an existing tool72 to independently extract data reporting empirically on how processes of delivery/receipt varied with characteristics of interventions, providers, participants or contexts. For outcome evaluations, we also extracted data on allocation; sequence generation and concealment [randomised controlled trials (RCTs)]; control of confounding (quasi-experimental studies); measures, follow-up and blinding; retention; and data on outcomes/effects at follow-up(s) both overall and, when available, by sexuality and gender identity, socioeconomic status (SES) and ethnicity. For economic evaluations, we extracted data on key issues such as the perspective (direct and indirect costs), evaluation framework, source of effectiveness estimates, critical assumptions, discount rates and cost-effectiveness in the form of either incremental cost-effectiveness ratios (ICERs) or net (health) benefits. We also aimed to report on the key cost-effectiveness drivers. We aimed to involve a translator if included studies were reported in languages that could not be translated by the review team, but this issue did not arise. The data extraction tools for theory reports and for process, outcome and economic evaluation reports are provided.
Where there was a risk of missing data from published reports affecting our analysis, we contacted authors whenever possible to request additional information. If authors were not traceable or if the sought information was unavailable from the authors within 2 months of contacting them, we recorded that the study information was missing on the data extraction form, and this was included in our risk-of-bias assessment of the study.
Parts of the remainder of this chapter have been reproduced from Bonell et al.73 Contains information licensed under the Non-Commercial Government Licence v2.0.
Assessments of quality and risk of bias
Parts of this section have been reproduced or adapted with permission from Bonell et al.74
We assessed reporting bias according to Sterne et al.’s75 guidance. We aimed to reduce the effect of reporting bias by focusing the synthesis on studies rather than publications, thereby avoiding duplicated data. Following the Cho et al.76 statement on redundant publications, we attempted to identify duplicate studies and, when multiple articles reported on the same study, we extracted duplicated data only once. We prevented location bias by searching across multiple databases. We prevented language bias by not excluding any article based on language.
Assessment of theories of change
We assessed the quality of descriptions of intervention theories of change using a modified version of the criteria developed in our previous systematic reviews of school-based interventions integrating health and academic education, positive youth development and school health education interventions,77,78 modified in the light of our more recent work on realist methods.79 The assessment focused on (a) the extent to which the theory of change described the path from intervention to outcomes, (b) the clarity with which theoretical constructs were defined, (c) the clarity with which causal inter-relationships between constructs were defined, (d) the extent to which the mechanisms underlying these inter-relationships were explained and (e) the extent to which the intervention theory of change considered how mechanisms and outcomes might vary by context. The two reviewers then met to compare their assessments, resolving any differences through discussion and, when necessary, by calling on a third reviewer.
Assessment of process evaluations
We assessed the quality of the qualitative and quantitative elements of process evaluations using an EPPI-Centre tool.80 The assessment addressed the rigour of sampling, data collection and data analysis; the extent to which the study findings were grounded in the data; whether or not the study privileged the perspectives of participants; the breadth of findings (i.e. the extent to which the study explored a broad range of process issues); and the depth of findings (i.e. the extent to which the study provided in-depth insights into participant perspectives). This was then used to assign studies to two categories of ‘weight of evidence’. First, reviewers assigned a weight (low, medium or high) to rate the reliability or trustworthiness of the findings (the extent to which the methods employed were rigorous/could minimise bias and error in the findings). Second, reviewers assigned an additional weight (low, medium, high) to rate the usefulness of the findings (i.e. the extent to which these could shed light on how processes of intervention delivery/receipt varied with characteristics of interventions, providers, participants or contexts). Guidance was given to reviewers to help them reach an assessment on each criterion and on the final weight of evidence. The two reviewers then met to compare their assessments, resolving any differences through discussion and, when necessary, by calling on a third reviewer.
Assessment of outcome evaluations
For outcome evaluations, we assessed the risk of bias of each included experimental study using the tool outlined in the Cochrane Handbook for Systematic Reviews of Interventions.63 For each study, two reviewers independently judged the likelihood of bias in seven domains: sequence generation, allocation concealment, blinding of participants or personnel, blinding of outcome assessors, incomplete outcome data, selective outcome reporting and other sources of bias (e.g. recruitment bias in cluster randomised studies). Each study was subsequently identified as being at ‘high risk’, ‘low risk’ or ‘unclear risk’ of bias in each domain. In cases of disagreement, the reviewers met to seek consensus and, when necessary, referred judgement to a third reviewer. The protocol did not originally specify a quality assessment tool for non-random studies. An amendment to the protocol (see Appendix 7) stipulated that such studies would be assessed using the Risk Of Bias In Non-randomised Studies – of Interventions (ROBINS-I) tool,81 but no non-random evaluations were eligible for inclusion in the review.
Assessment of economic evaluations
We initially planned to assess the quality of economic evaluations using the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist,82 but amended the protocol (see Appendix 7) to do so using a more appropriate tool: an adapted version of the Drummond et al.83 reporting guideline. It requires the assessor to answer 24 questions regarding each study, ranging from the type of economic evaluation (e.g. cost–utility analysis) to the time horizon and rationale for the choice of modelling approach. Although the questionnaire is detailed, we expanded a number of its questions to ensure that information that was particularly relevant to this review was extracted, such as identifying uptake rates and assumptions regarding the heterogeneity of risk (see Appendix 6). Two reviewers each conducted this assessment independently, then met to compare their assessments, resolving any differences through discussion and, when necessary, by calling on a third reviewer.
Data analysis
Typology of intervention approaches
Intervention descriptions and theories of change were first analysed to develop a typology of interventions. We intended to incorporate whether or not interventions were focused solely on the prevention of alcohol or drug use; HIV, STIs and sexual risk behaviour (sometimes referred to as ‘sexual health outcomes’ in this report, for brevity); or mental illness; and whether or not they had other aims such as access to HIV testing or adherence to HIV treatment. As planned, we identified interventions’ targeted outcomes. However, we ultimately developed a typology of interventions based on authors’ narrative descriptions of intervention methods because the strongest similarities and differences between interventions emerged in relation to the approaches used, and interventions addressing similar outcomes often took different approaches to doing so.
Research question 1: synthesis of intervention theories of change
Using thematic synthesis methods,84–86 we undertook a synthesis of author narratives and depictions describing theories of how interventions were intended to generate outcomes. The aim was to develop an overarching theory of change for e-health interventions, or multiple such overarching theories if this was more appropriate. The synthesis was not restricted to studies judged to be of high quality. Instead, conclusions drawing on poorer-quality reports were given less interpretive weight. We focused on prospective theories of change rather than retrospective reflections on how interventions might have worked in the light of study findings. Author narratives describing rationales for aspects and characteristics of the intervention in which the author did not link these to theorised mechanisms of change were also excluded. Theory of change synthesis commonly uses a meta-ethnographic approach, originally developed to synthesise findings across multiple qualitative studies.87 As originally applied to qualitative research, meta-ethnographic methods draw on primary constructs (verbatim qualitative data presented in reports of primary research) and secondary constructs (author interpretations of data presented in primary research) to develop tertiary constructs (reviewer interpretations presented in syntheses). Applied to theory synthesis, such methods draw solely on primary constructs (author descriptions of theories of change).
Informed by our typology of included interventions, two reviewers undertook a pilot analysis of the theory of change extracted from each of the two highest-quality studies of the same intervention type. We initially planned to undertake line-by-line coding of theory reports in order to identify recurring narrative themes across theories of change, as we have done in previous theory syntheses.77,78 Each of two reviewers independently applied line-by-line codes, beginning with in vivo codes that closely reflected the words used in the theory reports. They then grouped and organised codes, applying axial codes reflecting higher-order themes, and met to compare and contrast the resulting coding. Because this narrative-based approach did not readily capture the well-described and complex inter-relationships between theoretical constructs present in the reports (often in the form of a diagram), we instead decided to develop a novel diagrammatic approach to theory synthesis. This methodological innovation allowed us to summarise the components of each intervention’s theory of change and the explicitly and/or implied causal relationships between them, drawing on text and diagrams present in included reports. Summarising these diagrammatically also facilitated the comparison and synthesis of these components and relationships across the included theories of change.
Like the approach we had initially planned, this novel method of theory synthesis was a form of qualitative synthesis, but one that aimed to describe theories of change primarily in terms of constructs, interconnections and interactions, rather than as narrative themes. Like conventional thematic analysis, it involved an initial stage of ‘in vivo’ coding of author descriptions to identify theories of change for each intervention (but expressed diagrammatically rather than as a set of narrative themes), followed by a stage of ‘axial’ coding to explore interconnections between in vivo coding, identifying similarities and differences across interventions to develop overarching theories of change (again expressed diagrammatically). This represents a deviation from protocol, albeit one that added, rather than detracted from, the rigour of the analysis; this protocol deviation is reported in Appendix 4.
We read and re-read textual descriptions of theories of change and (when available) diagrammatic logic models of theories of change contained in the data extraction forms relating to each study. Two reviewers then independently drew an initial diagram of the theory of change underpinning each intervention. When more than one report addressed the same intervention, reviewers used theory of change descriptions from all relevant reports to inform the diagram. The two reviewers then met to compare their diagrams for each intervention theory of change and reconciled discrepancies through discussion. Drawing on the strengths of each diagram, they developed a single diagram of each intervention’s theory of change, which included intervention components, mediators and moderators (when described by authors) and intended outcomes. When author descriptions implied, but did not explicitly state, inter-relationships between components of the theory of change, reviewers made inferences and noted when the diagrams were, in part, based on such inferences.
The two reviewers grouped these theory of change diagrams by (1) intervention type and (2) targeted health outcomes (sexual health, mental health and substance use) to explore the scope for developing overarching theories of change within and across these categories. Finding that the theory of change approaches underpinning the interventions were not patterned by intervention type or targeted outcomes, we took an inductive approach, grouping diagrams of theories of change that shared important constructs and pathways. Then, using reciprocal translation (to identify and describe similar concepts occurring across theories of change underpinning different interventions), refutational synthesis (to identify contradictory or opposing concepts occurring across theories of change) and line of argument (to synthesise distinct elements occurring across theories of change that form part of a broader whole) approaches from meta-ethnography,88 each reviewer independently analysed the diagrams within each grouping. We did this by systematically examining the constructs and the relationships between constructs presented in each intervention-specific diagram and by examining whether these recurred, appeared only once or conflicted with those depicted in other intervention-specific diagrams in the grouping. Based on their analyses, each reviewer then independently drafted one synthesised diagram for each grouping of similar intervention theories of change.
We documented each stage of this process, noting when theory of change components or relationships between these components differed between individual diagrams in the grouping, the approach used to synthesise these components (i.e. reciprocal translation, refutational synthesis, line-of-argument synthesis, or the exclusion of a theory of change component) and the resulting decision for the synthesised theory of change diagram. The two reviewers then met to compare their synthesised diagrams of change for each grouping, reconciling discrepancies and drawing on the strengths of each to develop a single synthesised theory of change diagram for each theory of change grouping. To demonstrate this process, Appendix 8 presents the theory of change diagrams for each of two individual interventions in one grouping and the resulting diagram of the synthesised theory of change for that grouping. Each synthesised theory of change was given a descriptive title inductively drawing on the central approaches of the theories of change synthesised.
In this application of meta-ethnographic methods to the synthesis of theories of change, our first-order constructs were the theory of change information described in theory reports and represented in data extraction forms; our second-order constructs (analogous to in vivo codes) were the reviewers’ interpretations of these concepts, represented in the intervention-specific theory of change diagrams; and our third-order constructs (analogous to axial codes) were the higher-order interpretations, represented by the diagrams of the synthesised theories of change developed for each inductive grouping.
Thematic synthesis of process data
We undertook a synthesis of process data using thematic synthesis methods.84–86 Syntheses of findings from qualitative and quantitative elements of process evaluations were used to understand characteristics of interventions, participants and contexts that acted as potential barriers to and facilitators of implementation and receipt, and which of these applied across or only within the domains of sexual health, substance use and mental health interventions.
Synthesis followed a meta-ethnographic approach. Second-order constructs (author narratives) were distinguished from first-order constructs (directly quoted qualitative data). In the case of findings from qualitative elements, we undertook line-by-line coding examining ‘first-order constructs’ (directly quoted qualitative data) and second-order constructs (author interpretations). In the case of findings from quantitative elements, we coded author interpretations, first checking as part of the quality assessment whether or not these aligned with the quantitative data presented (i.e. the extent to which study findings were grounded in the data). The synthesis drew these together through a thematic analysis, which developed third-order constructs by drawing connections between these data. This synthesis was not restricted to studies judged to be of high quality. Instead, conclusions drawing on poorer-quality reports were given less interpretive weight.
First, two reviewers prepared detailed results tables to describe the quality of each process evaluation report; details of the intervention examined (drawing on all included reports on the intervention, regardless of report type); study site and population; and study findings, including which aspects were explored in regard to how processes vary by intervention characteristics, providers, participants and/or context (see Appendix 9). Second, the two reviewers independently piloted coding of two high-quality process reports. They each read and re-read the study findings, applying line-by-line codes to capture the content of the data. Coding began with in vivo codes that closely reflected the words used in findings sections. The reviewers then grouped and organised codes, applying axial codes reflecting higher-order themes. They then met to compare and contrast their coding of these first two reports, developing an overall set of codes. Third, the two reviewers went on to code the remaining reports for each synthesis, drawing on the agreed set of codes, but developing new in vivo and axial codes as these arose from the analytical process. At the end of this process, the two reviewers met to compare their sets of codes. They identified commonalities, differences of emphasis and contradictions with the aim of developing an overall analysis that drew on the strengths of the two sets of codes and that resolved any contradictions or inconsistencies. We planned a priori to produce tables from our analysis demonstrating how first-, second- and third-order constructs related to one another, thereby enhancing transparency about emergent themes.
Synthesis of outcome data
We first produced a narrative account of the effectiveness of included interventions. Subsequently, we conducted a narrative synthesis by outcome, then, within this, by follow-up time and intervention type. Outcomes were categorised into sexual health outcomes, alcohol use, drug use, anxiety and depression. Two reviewers prepared detailed results tables describing intervention characteristics (drawing on all included reports on an intervention), study characteristics and study findings (see Appendix 10). We produced forest plots for each of our review outcomes, with separate plots for different outcomes and follow-up times, and pairwise comparisons between intervention types (e.g. intervention vs. no treatment control, or vs. another treatment type). Plots included point estimates and standard errors for each study, expressed as standardised mean differences (Cohen’s d) to ensure comparability across reports.
When data allowed, we calculated pooled effect sizes within each pairwise comparison (e.g. intervention type vs. control), accounting for the extent of heterogeneity among the studies (as determined both by a Cochran’s Q test and inspection of I2).89 The results of statistical tests were evaluated in accordance with the Cochrane handbook.63 If an indication of substantial heterogeneity was determined with fewer than three studies (e.g. study-level I2 value of > 50%), we did not present a pooled estimate. When we had high levels of unexplained statistical heterogeneity (e.g. study-level I2 value of > 50%) in any of our study groupings and a sufficient number of studies, we aimed to investigate this further using subgroup and sensitivity analyses.90 As is appropriate for complex interventions, we used the random-effects model in meta-analyses; we did not undertake fixed-effects analyses because these would not have been interpretable given the effect estimates and statistical procedures used.
Where we produced pooled estimates, we used a robust variance estimation meta-analysis model to synthesise effect sizes.91 This is because outcome evaluations are likely to include multiple measures of conceptually related outcomes and robust variance estimation meta-analysis improves on previous strategies for dealing with multiple relevant effect sizes per study, such as meta-analysing within studies or choosing one effect size, by including all relevant effect sizes but adjusting for interdependencies within studies.92 Unlike multivariate meta-analysis, it does not require the variance–covariance matrix of included effect sizes to be known. We estimated separate models for each outcome: HIV, STIs, defined sexual risk behaviours, alcohol use, drug use, anxiety and depression. We regarded follow-up times of < 3 months (short term), 3 months to 1 year (mid-term) and > 1 year (long term) post intervention as different outcomes, pooling first by follow-up times and, when appropriate, overall across follow-up times. We ran these models for interventions overall and, when sufficient studies were found, we ran separate models for different intervention types. Categorisation by intervention type was into ‘clinically meaningful units,’ or subgroups of interventions that are useful and meaningful from a clinical and practice perspective, and were informed by the typology derived from our prior synthesis of intervention descriptions and synthesis of theories.93 When meta-analyses were performed, we included pooled effect sizes in forest plots, with the individual study point estimates weighted by a function of their precision.
In addition, we aimed to consider all outcome evaluation evidence jointly in a network meta-analysis (NMA). A NMA integrates all pairwise comparisons for a specific outcome in the same model, to allow for the comparison of intervention types that may not have been directly compared head-to-head in empirical studies, and to combine direct, empirical head-to-head evidence with indirect estimates. However, we were unable to undertake these models with the body of evidence included because pairwise comparisons did not form an interconnected network with similar populations and outcomes. All included studies allocated individuals rather than clusters, so there was no need to account for clustering in any analyses.
We used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach, as described in the Cochrane Handbook for Systematic Reviews of Interventions, to present the quality of the evidence and the tables summarising the findings.63 GRADE is a tool for researchers to describe how close the effect estimated in a meta-analysis is to the ‘true’ effect of the intervention. The downgrading of the quality of a body of evidence for a specific outcome was based on five factors: limitations of study, indirectness of evidence, inconsistency of results, precision of results and publication bias. The GRADE approach specifies four levels of quality (high, moderate, low and very low). If we had found a sufficient number of studies, we would have drawn funnel plots to assess the presence of possible publication bias (trial effect vs. standard error). Although funnel plot asymmetry may indicate publication bias, this can be misleading with a small number of studies.
We planned to undertake a sensitivity analysis to explore whether or not the findings of the review were robust in the light of the decisions made during the review process. This would have included assessing the impact of risk of bias in the included studies via restricting analyses to studies deemed to be at low risk of selection bias, performance bias and attrition bias. However, we did not undertake these analyses because too few studies would have been included in each meta-analysis to permit meaningful interpretation. When data allowed, we planned to undertake additional exploratory meta-analyses to determine intervention effects on theorised intermediate outcomes (such as goal-setting or self-efficacy) to examine the plausibility that these might mediate or otherwise precede behavioural effects, as well as to explore whether or not intervention effects on some of our outcomes (e.g. drug use) appear to mediate effects on other outcomes (e.g. risk of HIV infection). However, the reported intermediate outcomes were diverse between studies, poorly reported so as to preclude estimation of effect sizes and unlinked to estimates of outcomes, thereby precluding a meaningful synthesis of these outcomes.
If sufficient studies had reported appropriate information, we would have examined intervention effects by participant subgroups in terms of participant sexuality and gender identity, SES and ethnicity to explore potential impacts on health inequalities. This would have drawn on existing methods involving an ‘equity lens’ to examine evidence that equity-related characteristics (individual sexuality or gender identity, SES or ethnicity) moderate intervention effects, both in terms of evidence within studies and, if enough evidence had existed, between studies of significant effect modification.
Synthesis of economic data
Measures of costs, indirect resource use and cost-effectiveness were summarised in a table. If measures of resource use were judged to be sufficiently homogeneous across studies, we planned to synthesise these using statistical meta-analysis, but this was precluded by the sparse evidence available.70 Measures of costs, indirect resource use and cost-effectiveness were adjusted for currency and inflation to the current UK context. These data were used to inform a narrative synthesis of economic evidence and applicability to the UK context (RQ4).
Interpreting overall findings
In Chapter 10, we draw together what we have learnt overall from the syntheses of theories of change, facilitators of and barriers to implementation, and outcomes of e-health interventions addressing sexual health, substance use and mental health among MSM. This discussion focuses on implications for intervention research. We assessed whether or not interventions addressing the various outcomes, or some subsets of these, appeared to have similar, or at least compatible, theories of change (e.g. similar mediating factors or mechanisms) that could pragmatically be combined to develop an overall intervention with a coherent theory of change addressing the syndemic or some sub-elements of the syndemic. We then assessed whether different or similar factors appeared to facilitate or impede implementation or receipt of interventions addressing the various outcomes in order to further develop our sense of whether these outcomes might be feasibly addressed by a single intervention or might be better addressed by separate interventions addressing some, but not all, outcomes. Finally, we draw on the findings from our outcome syntheses to judge whether or not there is scope for a single intervention addressing sexual health, substance use and mental health to have synergistic effects. We established a priori that we would judge this to be likely if there was evidence that e-health interventions addressing the various outcomes all tended to be effective, particularly when there was evidence that interventions addressing different outcomes appear to positively affect similar mediators. We determined a priori that when this was found not to be the case, for example because of evidence of a lack of, or even harmful, effect on some outcomes, or a lack of evidence for some outcomes, our recommendations would reflect this.
Patient and public involvement in data interpretation, dissemination and knowledge transfer
We assembled a patient and public involvement (PPI) stakeholder group (by contacting a mixture of organisations already known to us or found through web searches) and met with members of this group twice during the review, first in April/May 2020 and again in December 2020. Six stakeholders from Cardiff Royal Infirmary, Central and North West London NHS Foundation Trust, HIV Scotland, London Friend, MESMAC (Men Who Have Sex with Men – Action in the Community) and METRO Charity took part. Each of these organisations focuses on the health and well-being of gay, bisexual and other MSM and/or is a community organisation for such men. Meetings were held individually, rather than as group discussions as initially planned, which was a minor deviation from our protocol (see Appendix 4). This was to enable more in-depth discussion and to accommodate busy stakeholder schedules.
For the first meeting, stakeholders were sent slides with a summary of the main findings in relation to the typology of interventions, the synthesis of evidence on theories of change and factors affecting delivery and receipt (see Report Supplementary Material 2). Each of these three areas was then examined during a structured discussion. We asked stakeholders to advise on the feasibility of drawing on the interventions presented to inform the development of an overall intervention addressing the syndemic of HIV/STIs, sexual risk, alcohol and drug use and mental ill health among UK MSM, focusing on the following questions:
- What do you think would be the potential benefits of e-health approaches for the groups you work with?
- What would be the potential drawbacks?
- What types of interventions do you think would be most appropriate?
- What are your thoughts on the theoretical underpinnings of these? Do any of the three models seem to be the most appropriate? Or the most comprehensive?
- What are your feelings on the acceptability of these interventions for your client group? And for your staff?
- Do you have thoughts on the feasibility of delivering something similar? What concerns would you have?
The first round of consultations took place with six stakeholders from England, Scotland and Wales. Four were affiliated with the voluntary sector and two were affiliated with clinical settings.
For the second meeting, stakeholders reviewed slides summarising the syntheses of outcome and economic evaluations (see Report Supplementary Material 3). At this stage, we explored with stakeholders whether or not the evidence of effectiveness and cost-effectiveness overall and by subgroup suggested that it would be worth investing in the development of an e-health intervention to address multiple outcomes among UK MSM and sought input on dissemination and knowledge transfer. We asked stakeholders the following questions during this consultation:
- Having reviewed the evidence we found, what are your overall impressions of the effectiveness of e-health interventions? Which interventions, if any, do you conclude are effective or highly promising?
- In the earlier part of the review, we found more interventions targeting all areas of the syndemic. What are your thoughts about having data only for drug use, HIV, STIs and sexual behaviours?
- What concerns do you have about the limitations of the evidence that has been found?
- Are there specific areas you think are a priority for generating more evidence? Are there promising interventions that need to be evaluated?
- What are your thoughts on the (potential) cost or cost-effectiveness of these interventions relative to their impacts?
- In your opinion, would it be worth pursuing the development of an e-health intervention either for immediate scale-up or for an intervention study in the UK in the future?
- What/how should we disseminate the findings to policy and practice stakeholders?
- Do you think we should do an intervention study, and, if so, focused on what outcomes? Or do you think the evidence on sexual behaviour effects warrants immediate scale-up of an intervention focused on this outcome?
The second round of consultations took place with four stakeholders from England, Scotland and Wales. Three were affiliated with the voluntary sector and one was affiliated with a clinical setting.
Registration
The review protocol was publicly registered online.
Revisions to the protocol
The protocol was amended four times from 3 October 2018 to 5 August 2019 (see Appendix 7).
Governance and ethics review
As the principal investigator, Chris Bonell was responsible for the conduct and delivery of the work. The sponsor of the research was Professor Kara Hanson, Dean of the Faculty of Public Health and Policy, London School of Hygiene & Tropical Medicine. The co-applicants formed an investigator committee, which met monthly throughout the project, overseeing its conduct. These meetings were minuted to keep a record of tasks, deadlines and responsibilities. The research involved no human participants and drew solely on evidence already in the public realm, so research ethics approval was not required. The team followed relevant guidelines and best practice including the Social Research Association’s ethics guidelines94 and referred also to guidance recommended by the National Co-ordinating Centre for Public Engagement.95