Overall impressions of the effectiveness of e-health interventions
Stakeholders felt that it was encouraging that intervention effects were apparent on mid-term sexual risk behaviour outcomes. It was felt that mid- and long-term behaviour change were the most important in the context of e-health interventions, and that, although short-term benefits were also desirable, lasting change was critical to success. One stakeholder noted that the significant mid-term impacts may have been due to trials with longer follow-up (and their associated interventions) being of higher quality than those with shorter follow-up, perhaps biasing the results.
Significant concerns were expressed about the lack of studies reporting on mental health and alcohol use outcomes. Both of these were felt to be a very high priority for further development work, especially in the context of COVID-19, as stakeholders observed increases in client need in both areas. This need was difficult to meet as it manifested alongside new ways of working to observe social distancing rules. In addition, the data on outcomes relating to use of other drugs were felt to be insufficient to draw firm conclusions on intervention effectiveness, especially given that one study by Reback et al.115 reported some evidence of possible intervention effects on increased days of use of methamphetamines in one intervention arm. Stakeholders were encouraged by the analysis that showed a significant reduction in STI diagnosis in Mustanski et al.113
Enthusiasm for the results from the systematic review was moderate and tempered by inconsistency in outcomes and by outcomes for which there were no data. It was noted that randomised trials of these types of interventions are uncommon and that those from the voluntary sector especially would look to other types of evidence (largely observational evidence and grey literature) to inform programming. This was perhaps due to the voluntary sector prioritising interventions that are a more obvious fit from an organisational perspective and because the voluntary sector as a whole does not put the same emphasis on RCT evidence as academia and clinical services do, and are more likely to rely on their professional judgement as expert practitioners. It was therefore felt that additional observational evidence from other sources might be more pertinent, especially for the outcomes for which evidence was thin or did not exist.
Modular interventions were seen to be the most promising and likely to be accessible to a wider range of MSM. These were also described as easiest to adapt in response to local context, so that they might be useful across all parts of the UK. Video games were the least preferred intervention; one stakeholder noted that they were very expensive to develop and that issues with their performance would be a barrier to retaining participants, creating a barrier to change. In addition, they were not felt to be appealing to a wide range of MSM and would become obsolete extremely quickly. Apps were felt to be useful in the context of interventions that promote self-monitoring. Text-based interventions were felt to be appropriate if less engaging.
Concern was raised by one stakeholder about the lack of cost-effectiveness data. Based on their professional experience, e-health interventions are often used as a cost-saving measure, but end up costing far more than necessary because the sectors that produce and deliver them have a poor understanding of commissioning and developing such technologies. As a result, it was felt that the cost-effectiveness of such interventions may turn out to be suboptimal.
All stakeholders had concerns that the evidence base around the effectiveness of these interventions might not translate well to the UK, given that only one study was from Europe and the rest were from the USA and Asia. It was felt that the differences in culture, as well as health and economic systems, might affect how such interventions perform.
Views on further development and evaluation
Despite the limitations of the evidence base, all stakeholders felt that it was worthwhile to develop an e-health intervention for further evaluation. For some stakeholders, the evidence base was sufficient to warrant further exploration, whereas, for others, this judgement was based on professional opinion, as the review did not provide what they felt to be compelling evidence of effectiveness. Stakeholders judged that data from this review did not indicate the appropriateness of immediate scale-up.
All stakeholders stated that any intervention developed, and its associated evaluation, should focus on the range of outcomes included in the review, and that interventions should address these in a holistic manner. Mental health was felt to be a very high priority, alongside sexual risk behaviours and drug and alcohol use. It was noted that the context of post-COVID-19 pandemic service delivery will be especially supportive of e-health interventions as much delivery of services has moved online and people have become more comfortable and engaged with these approaches. Two stakeholders (one from clinical services and one from the voluntary sector) felt that a longer follow-up time in any future evaluation would be most useful to inform decision-making. Three stakeholders (one from clinical services and two from the voluntary sector) stressed that co-design with intended beneficiaries would be important for developing relevant, accessible interventions.
If evaluation were to be through a randomised trial, all stakeholders felt that this should compare two interventions with each other, rather than have a no-treatment control group, or consider another design where all receive the intervention. None would feel comfortable in directing individuals to a RCT with a control arm involving no treatment when an individual needed support. It was noted that this approach would be considered unethical. Given the variation in usual treatment available locally because of the UK commissioning system, stakeholders in general felt that it was better to compare interventions with one another. It was also felt that any further trial should include a large sample size and an evaluation of cost-effectiveness, which would include the set-up, as well as the running, costs of e-health interventions.
Views on dissemination
The results were felt to be of perhaps limited utility to the wider voluntary sector. The type of evidence included (RCTs) was seen to be important, but not necessarily where organisations look when programming. A key benefit of voluntary sector service provision was felt to be that organisations can be nimble in their approaches and can quickly adapt evidence-based interventions and modify them rapidly in response to feedback during implementation and service delivery. The long time frames associated with RCTs (and systematic reviews of them) meant that this evidence often was not immediately useful and that an organisation had moved on to other approaches to behaviour change and other outcomes before trials had reported. For example, voluntary sector provision had moved away from a primary focus on promoting condom use towards more multifaceted interventions focused on biomedical approaches, such as PrEP and treatment as prevention, and on promoting HIV testing.