In this chapter, we draw together the findings from the systematic review, NMA and economic evaluation. Based on the criteria for interpretation described in Chapter 2, we summarise our overall interpretation of the evidence from the NMA for the primary time point of immediately post intervention. This summary forms the basis for the implications for practice, implications for research and the conclusions reported in Chapter 10.
Systematic review
The full results of the systematic review are reported in Chapters 4–7 and are summarised briefly here. The effectiveness results are based on searches conducted in April 2018. A total of 11,990 citations were screened, and 1512 full-text articles were retrieved for screening. The review included 142 studies of > 63,500 randomised participants. Of the 142 studies, 92 were judged to be at unclear risk of bias for random sequence generation, 115 were judged to be at high or unclear risk of bias for allocation concealment and 133 were judged to be at high or unclear risk of bias for blinding of participants. We identified a protocol or trial registration for only 32 studies. This represents 23% of included studies published post 2000.
Of the 142 eligible studies, 71 contributed data to the NMA for the prevention of anxiety, 86 were included for the NMA for depression and five contributed to the narrative summary for the prevention of conduct disorder. Note that there was an overlap of studies contributing to the anxiety and depression NMA, with 47 studies reporting both an anxiety and a depression outcome.
Network meta-analyses by population and setting
This is a large and complex review, with 32 possible intervention-level NMAs conducted (condition × setting × population × time point), from which we reported 57 intervention effect estimates of the primary outcomes of self-reported symptoms of anxiety or depression. This number of analyses does not include the component NMAs or subgroup or sensitivity analyses, and does not include the additional and secondary outcomes. Below we report the findings for our primary outcomes at the primary time point of post intervention, for the intervention-level and component-level NMA. Note that, to ensure conciseness in this chapter, we concentrate on interpreting findings for which the relevant results chapter has indicated that there may be statistical evidence of an effect, as described in Chapter 2. However, for full reporting of all results and intervention effect estimates, see Chapters 4 and 5.
Anxiety: universal population, secondary setting
This was a well-populated network of 21 studies and 10,208 participants. The between-study heterogeneity was moderate across the network. The risk of bias for random sequence generation and allocation concealment was deemed to be mostly unclear. At the post-intervention time point, we found weak evidence of a small beneficial effect of CBT-based interventions (SMD –0.15, 95% CrI –0.34 to 0.04), relative to usual curriculum. There was statistical evidence that mindfulness/relaxation interventions (SMD –0.65, 95% CrI –1.14 to –0.19) were effective in preventing symptoms of anxiety. However, the CrIs for the mindfulness/relaxation effect estimate were relatively imprecise, and the effect estimate is based on two small studies, which were judged to be at unclear risk of bias for the domains of random sequence generation and allocation concealment. We also note that there was variation between the study-level effect estimates of the two mindfulness/relaxation studies. One study130 (with 30 participants) reports a large effect estimate of mindfulness/relaxation relative to a waiting list (SMD –1.01, 95% CrI –1.67 to –0.35). The second study137 compared mindfulness/relaxation with an attention control and reports a smaller effect (79 participants, SMD –0.32, 95% CrI –0.72 to 0.08). Finally, the statistical findings for both CBT and mindfulness/relaxation should be interpreted considering the possible evidence of asymmetry from the comparison-adjusted funnel plots, which suggests the presence of small-study effects or other non-reporting bias.
Anxiety: universal population, primary setting
This was a small network of 15 studies, but with a reasonable number of participants (n = 5605). There was moderate between-study heterogeneity across the network. Thirteen studies were judged to be at unclear risk of bias for random sequence generation and allocation concealment, one was judged to be at low risk of bias for both domains and one was judged to be at low risk of bias for random sequence generation only. There was weak evidence for a very small beneficial effect of CBT (SMD –0.07, 95% CrI –0.23 to 0.05), relative to usual curriculum. The comparison-adjusted funnel plot was suggestive of small-study effects or other non-reporting bias.
Anxiety: targeted population, secondary setting
This was a small network of 15 studies (2383 participants). There was mild to moderate between-study heterogeneity across the network. Three studies were judged to be at low risk of bias and seven were judged to be at unclear risk of bias for randomised sequence generation and allocation concealment. Four studies were judged to be at low risk of bias for randomised sequence generation and at unclear risk of bias for allocation concealment. There was evidence that exercise was effective in reducing self-reported anxiety in targeted secondary settings, relative to no intervention (SMD –0.47, 95% CrI –0.86 to –0.09). However, the width of the CrI indicates a relatively imprecise estimate, and exercise was evaluated in only one small study170 (with 121 participants) that was judged to be at unclear risk of bias for random sequence generation and allocation concealment. We caution against overinterpreting the finding for exercise-based interventions in this network. There was no evidence of effectiveness for the other interventions. The comparison-adjusted funnel plot did not suggest the presence of small-study effects or other non-reporting bias.
Anxiety: targeted population, primary setting
The targeted primary network was the smallest of all those considered for the prevention of anxiety with only 11 studies (1314 participants). There was substantial between-study heterogeneity (τ 0.42, 95% CrI 0.21 to 0.89) across the network. Three studies were judged to be at low risk of bias and six studies at unclear risk of bias for both the randomisation and allocation concealment domains. A further two studies were judged to be at unclear risk of bias for allocation concealment, but at low risk of bias for random sequence generation. Although there was weak evidence of a beneficial effect for CBT relative to a waiting list (SMD –0.38, 95% CrI –0.84 to 0.07), the CrI for the estimate is relatively wide. There was no evidence of small-study effects or other non-reporting biases for this network; however, the number of studies is at the lower end required for this analysis to be meaningful.266
Anxiety: tertiary/university setting
There was evidence of (statistical) inconsistency in the NMA for the prevention of anxiety in tertiary settings; therefore, we do not report effectiveness findings. The inclusion criterion that interventions needed to be implemented in the educational setting may have limited the number of eligible studies, which, in turn, may have contributed to the lack of model fit observed. The limitations are discussed further in Chapter 10.
Depression: universal population, secondary setting
At the post-intervention time point, this was the most connected network, including the greatest number of studies (34 studies; 18,094 participants), of which 25 included an intervention based on CBT. Most studies were rated as having an unclear risk of bias for at least one domain. The between-study heterogeneity was moderate across the entire network. There was weak evidence of a very small beneficial effect for CBT, compared with usual curriculum (SMD –0.04, 95% CrI –0.16 to 0.07). There was also some evidence that CBT + IPT (SMD –0.18, 95% CrI –0.46 to 0.08) was effective at reducing depressive symptoms. The comparison-adjusted funnel plots did not indicate small-study effects or other non-reporting biases.
Depression: universal population, primary setting
This was a small network of 12 studies (4116 participants) of six interventions. Only one study did not include a CBT comparator, yet the between-study heterogeneity was considered to be moderate to substantial. One study was rated as having a low risk of bias and 11 studies were rated as having an unclear risk of bias for both random sequence generation and allocation concealment. There was no evidence of an effect for any intervention relative to usual curriculum. The comparison-adjusted funnel plots did not indicate small-study effects or other non-reporting biases.
Depression: targeted population, secondary setting
Twenty-four studies contributed to this NMA. However, it was small in terms of participants (n = 3669). There was evidence of moderate to substantial between-study heterogeneity across the network. Most studies were rated as having a low risk of bias on at least one of the domains; however, allocation concealment was rated as having an unclear risk of bias in 18 studies. There was no evidence of an effect for any intervention relative to no intervention. The comparison-adjusted funnel plot did not indicate small-study effects or other non-reporting biases.
Depression: targeted population, primary setting
There were five studies (497 participants) in this NMA. Between-study heterogeneity was substantial. One study was judged to be at low risk of bias for both random sequence generation and allocation concealment, and three studies were judged to be at unclear risk of bias. One study was rated as having a low risk of bias for random sequence generation, but an unclear risk of bias for allocation concealment. There was no evidence of an effect for either CBT or occupational therapy, relative to a waiting list. A funnel plot was reported but not interpreted because of the small number of studies included.
Depression: tertiary/university setting
We observed evidence of inconsistency in the NMA for the prevention of anxiety in tertiary settings; therefore, it was not appropriate to report effectiveness findings. As a consequence of the inclusion criteria, few studies were included, which may have contributed to the lack of model fit. The limitations of the inclusion criteria are discussed in Chapter 10.
Anxiety and depression: component network meta-analysis
The taxonomy of intervention components was reported in Chapter 3. We report the anxiety and depression component NMA findings in Chapters 4 and 5. There was little robust evidence that specific combinations of components were more effective than others. The exception was a psychoeducation component: it appears that the addition of a psychoeducation component to CBT may result in a more effective preventative intervention. The mechanism for this finding is not clear. In the psychotherapeutic literature, Pompoli et al.72 found that the addition of a psychoeducation component to CBT decreased the odds of remission from adult panic disorder symptoms (i.e. made remission less likely), whereas López-López et al.82 found no evidence that adding psychoeducation to CBT changed the effectiveness for treatment of depression in adults (SMD 0.04, 95% CrI –0.66 to 0.75). Both the Pompoli et al.72 and López-López et al.82 studies were conducted in clinical adult populations, and the present study is in non-clinical child and adolescent populations, and mechanisms of psychoeducation may vary. To aid future intervention development, investigation of the potential mechanism(s) of psychoeducation should be conducted and could also explore the possibility that the mechanism of effect is likely to vary across adult and CYP populations.
Anxiety and depression: subgroup analyses
For the primary time point of post intervention, we explored whether or not effects observed in the NMA varied by mode of delivery, facilitator and focus of the intervention. On balance, there was a lack of evidence of effect modification by facilitator or mode of delivery for any population or setting combination. There was weak evidence that interventions facilitated by MHPs may be more effective. There was some evidence to suggest that interventions focused on preventing anxiety had a larger effect on reducing self-reported symptoms of anxiety than those focused on both depression and anxiety, or depression alone, and that interventions focused on preventing depression had a larger effect on reducing symptoms of depression than those focused on both depression and anxiety, or anxiety alone. However, this was not a formal statistical comparison, and, owing to the absence of participant blinding, we cannot rule out possible Hawthorne effects. Therefore, we do not consider this as providing evidence for, or against, the transdiagnostic hypothesis.
Anxiety and depression: additional primary and secondary outcomes
Additional primary and secondary outcomes were also analysed for the primary time point. There was no evidence to suggest that primary or secondary educational setting-based interventions to prevent anxiety or depression improved well-being, reduced suicidal ideation or self-harm, improved academic attainment or improved parent reports of child mental disorder symptoms. Subgroup analyses to examine whether or not intervention effects varied according to SES suggested that interventions delivered in lower SES settings may be slightly less effective than those delivered in higher/mixed SES settings. However, this is based on low SES subgroups with a maximum of four studies, and these findings may not be considered reliable. We did not test for subgroup differences. As regards intervention acceptability, most studies reported that participants were satisfied with their intervention assignment and interventions were rated by the students as reasonably enjoyable. It is of note that only two studies reported the comparative or relative enjoyment across all intervention arms. In both studies, participants preferred the control interventions.
Conduct disorder
Owing to the diversity of interventions, outcome measures and time points reported by the conduct disorder studies, results were reported narratively. Five studies were included, of which three were judged as having an unclear risk of bias for random sequence generation and allocation concealment. One study was judged as having a low risk of bias for both domains, and one was judged to be at low risk of bias for randomisation and unclear risk for allocation concealment. No study reported the primary outcome of self-reported conduct disorder symptoms at post intervention. Instead, results from parent and teacher reports and secondary measures of externalising behaviours at post intervention were summarised. There was evidence from two studies of classroom-based interventions and one study of a multisystemic intervention that externalising behaviours were reduced post intervention. Evidence of a beneficial effect was mixed from two studies of multisystemic, multicomponent and multiphase interventions. In the short term (between 1 and 3 years), there was no evidence to support intervention effectiveness. However, both studies reported evidence over the longer term (5–20 years) of a beneficial effect of the interventions for preventing self-reported conduct disorder symptoms.
Interpretation of network meta-analysis results across all networks: post intervention
With regard to comparative effectiveness at the primary post-intervention time point, we conclude that there is weak statistical evidence to support the effectiveness of school-based anxiety and depression prevention interventions, that effect sizes are modest and the evidence is not robust. CBT-based interventions were the most commonly used across the networks analysed. Despite this, across most networks, there was only weak statistical evidence to suggest that CBT-based interventions may be effective. There was evidence from the universal secondary analyses that mindfulness/relaxation interventions are effective in preventing symptoms of anxiety, and evidence from the targeted secondary anxiety analyses that exercise is effective. However, we are cautious about the overinterpretation of these results because they are based on few studies (two and one, respectively), which were judged to be at unclear risk of bias. We note that there was also weak evidence from the universal secondary depression NMA that CBT + IPT is effective at preventing depressive symptoms. However, the three studies including a CBT + IPT intervention were also rated as having mostly unclear risk of bias.
The evidence base is not robust and further weakens the statistical findings. We note that the risk of bias for random sequence generation and allocation concealment was rated as unclear across most of the networks. Meta-epidemiological evidence suggests that, for subjective outcomes (such as self-rated anxiety or depression), inadequate or unclear allocation concealment exaggerates intervention effect estimates.267 In the context of this review, the observed intervention-level effects are beneficial relative to control, but they are ‘small’.268 The potential impact of selection bias on these effect estimates should be considered in their interpretation. Future work using bias-adjusted NMA could explore the likely impact further.269
The possibility of non-reporting bias in the universal anxiety analyses, in particular, must be considered in the interpretation of the statistical findings. There was some evidence that small negative studies were absent from the anxiety analyses. Adjusting for these studies would probably further attenuate the modest effects observed.
The between-study heterogeneity was at least moderate in 9 of the 10 primary analyses, and mild to moderate in one. It is broadly accepted that statistical heterogeneity is inevitable in meta-analysis.270 However, steps should be taken to minimise potential sources of heterogeneity in advance of analysis, for example by defining coherent review inclusion and exclusion criteria. This is because the extent of between-study heterogeneity has implications for the interpretation and generalisability of results.43,103 To illustrate the difficulties heterogeneity causes for the decision-maker, we can consider a predictive interval.271,272 A 95% prediction interval estimates where the true intervention effects are expected to lie in a new study, or if the intervention were to be rolled out to similar populations (as those included in the analysis). In the presence of heterogeneity, the prediction interval fully encapsulates the uncertainty in intervention effect and will be wider than the CI or CrI. For example, in the universal secondary anxiety analysis (see Chapter 4), the ‘best-bet’ intervention for preventing anxiety is CBT [SMD –0.15 (95% CrI –0.34 to 0.04) vs. usual curriculum]. However, the corresponding 95% prediction interval is–0.47 to 0.16. We can interpret this interval as the 95% range of true SMDs to be expected if we were to implement CBT in secondary schools. That is, having considered the heterogeneity in the existing evidence for a narrowly defined set of interventions, we cannot rule out the possibility that a real-world implementation of CBT to school children might be harmful. We note that the observed heterogeneity was not explained by the subgroup analyses or metaregression, adding to the uncertainty for decision-makers seeking to implement a disorder-specific preventative intervention.
Key findings of the economic evaluation
The body of evidence described in the review of economic evidence was small (six studies) and heterogeneous. The CMD addressed, intervention type and cost-effectiveness analyses differed across the studies identified in the review. It was, therefore, not possible to quantitatively synthesise the findings. Across the studies reviewed, multiple perspectives for the analyses were taken, leading to the resources included in the analyses also varying widely. The potential long-term costs associated with missing school (such as lifetime earning capacity) were not estimated in any of the studies. The two model-based analyses reviewed suggested that hypothetical interventions could be cost-effective in the Australian context. However, the trial-based analyses suggested that the interventions were unlikely to be cost-effective in the UK or the USA. For the one intervention that was potentially cost-effective in a high-risk subgroup at an interim point of the study, the effect had disappeared with longer-term follow-up. There was very little empirical costing evidence to inform decisions on the implementation of preventative interventions.
We developed a highly stylised, ‘indicative’ CBT intervention for a microcosting study based on a universal secondary school setting. Taking the perspective of a single secondary school budget, an estimated intervention cost of £43 per student was derived. Although there are several ways in which a ‘real-world’ CBT intervention may differ, the simplistic model provides an indication of the costs that might accrue to a school budget in the first year of implementation. We also considered the costs of including a psychoeducation component within a CBT intervention, on the basis of the component NMA effectiveness results. We estimated that the potential cost of incorporating a psychoeducation component in the indicative CBT intervention might represent approximately £14 per student. Adding a psychoeducation component increases intervention costs, but this may be offset by the slightly greater improvement in anxiety scores post intervention. However, there was only weak evidence for an improvement in symptoms of depression post intervention in universal secondary settings.