Outcome evaluation results

This chapter describes the quantitative evaluation of the effectiveness of the community AHC programme (CICA).

The aims of the CICA Outcome Evaluation were to:

1. determine the effect on area-level alcohol-related hospital admissions, A&E attendances and ambulance call-outs
2. determine the effect on key crime indicators (street-level crime data) and key indicators of ASB.

Data and evaluation of models

An overview of median outcomes pre and post intervention is shown in Table 7. As indicated, any impact from the CICA intervention is likely to be relatively small at most, in particular for ambulance call-outs and reported crimes.

Pre-intervention parallel trends in the outcomes are assessed graphically for the local controls in Figures 3 and 4 and for the national controls in Figure 5. While Figures 3 and 4 indicate trends of outcomes in the selected local control LSOAs following those in the CICA intervention LSOAs, for the national controls (see Figure 5), there is some graphical evidence that trends start to converge across the pre-intervention time period. This indicates that inferences about the effect of the CICA intervention from the national comparison analyses should be interpreted with caution.

FIGURE 3 Comparisons of pre-intervention average trends of: (a) alcohol-related hospital admissions; weekend (b) and weekday (c) ambulance call-outs; and (d) A&E admissions in intervention (CICA) areas (coded 1) and local control areas (coded 0). (Figure not reproduced here; see the original article.)

FIGURE 4 Comparisons of pre-intervention average trends of: (a) alcohol-related A&E admissions; weekend (b) and weekday (c) reported crimes; and (d) reported ASB incidents in intervention (CICA) areas (coded 1) and local control areas (coded 0). (Figure not reproduced here; see the original article.)

FIGURE 5 Comparisons of pre-intervention average trends of: (a) alcohol-related admissions and weekend (b) and weekend (c) A&E admissions in intervention (CICA) areas (coded 1) and national control areas (coded 0). (Figure not reproduced here; see the original article.)

Assessment of performance of the statistical models was done based on evaluation of patterns in the residuals. Broadly speaking, patterns of model residuals confirm that the model assumptions are met, although, for some models, the patterns of residuals could be observed resulting from the low incidence rates of certain events (including zero incidents at some time points). For each of these models, we assessed whether there was evidence of zero inflation (i.e. more counts of zero than expected). There was evidence of this for ambulance call-outs and ASB (assessed using the performance R package), and for these, we reanalysed the data using zero-inflation negative binomial mixed-effects models using the glmmTMB package in R.106 Results are shown in Appendix 3 and indicate that although point estimates of the ATEs differ, confidence intervals largely overlap, and this alternative modelling would not have impacted on the inferences about effects of the CICA programme. We therefore present the original growth models rather than the zero-inflated model results as part of the main results.

Outcome evaluations

An overview of the results of all statistical analyses is shown in Table 8. A graphical overview of forest plots is shown in Figures 6–9. The primary, stepped-wedge analyses provide weak evidence of an average increase in alcohol-related hospital admissions following the implementation of CICA of about 13%, corresponding to about 20 admissions weekly, although the confidence interval was wide, −1.98 to 31.39, and did not reach statistical significance (p ~ 0.09) (see Table 8). Although this result is comparable to the effect found compared to local controls, this was not confirmed when national controls or counterfactuals were used, which indicated much smaller effect sizes and confidence intervals, including unity: 3.42 (−4.56 to 12.07; p-value 0.41) and 7.14 (−9.76 to 24.04; p-value 0.41), respectively. Nonetheless, all models consistently report an increase in hospital admissions following the introduction of the intervention. This is likely an artefact in the data from some unknown endogenous factor we could not model, as this direction of effect seems highly implausible (at least as a direct result of CICA) (see Figure 6).

FIGURE 6 Average effect (%) and approximate 95% confidence interval or credible intervals (synthetic controls) of CICA intervention for alcohol-related hospital admissions (narrow definition). (Figure not reproduced here; see the original article.)

FIGURE 7 Average effect (%) and approximate 95% confidence interval or credible intervals (synthetic controls) of CICA intervention for: ambulance call-outs during (a) weekdays and (b) weekends. (Figure not reproduced here; see the original article.)

FIGURE 8 Average effect (%) and approximate 95% confidence interval or credible intervals (synthetic controls) of CICA intervention for A&E attendances during (a) weekends and (b) weekends. (Figure not reproduced here; see the original article.)

FIGURE 9 Average effect (%) and approximate 95% confidence interval or credible intervals (synthetic controls) of CICA intervention for reported crimes during (a) weekends and (b) weekdays. (Figure not reproduced here; see the original article.)

The impact on ambulance call-outs to the areas following the CICA intervention showed an average decrease in call-outs in the weekend as well as for weekdays for both the stepped-wedge and local control analyses, but neither was statistically significant (p > 0.18) (see Table 8 and Figure 7). In contrast, the counterfactual models indicated an increase in call-outs, which was significant and was somewhat larger during weekdays (+24.3%, 13.4–35.2%) compared to weekends (+19.4%, 8.1–30.7%). Regardless, the effect size corresponds to 1–2 call-outs per LSOA per month, indicating that this is unlikely to describe any true effects from the intervention.

Results for A&E attendances similarly show some evidence of an effect of the CICA intervention, but triangulation indicates results are inconsistent (see Table 8 and Figure 8). The number of incidences on average increased following the introduction of CICA, and this is statistically significant for weekends for all methods bar the comparison to national controls (p = 0.87), but not for weekdays for the stepped-wedge analysis (p = 0.51) and national controls (p = 0.92). Effect sizes are small and correspond to < 2 additional admissions per month per LSOA. Given the inconsistency of the results, the small effect size and given that the direction of the effects is against our hypothesis, we infer that these analyses are not indicative of a true effect from the intervention.

There was similarly little evidence of an effect of the CICA intervention on the incidence of reported crimes in weekends (see Table 8 and Figure 9). Even though an average increase following the intervention was observed when compared to the counterfactual of +31.1% (p = 0.01), this was very imprecise (95% CrI 7.4% to 54.8%) and not mimicked by the other analyses which, although the stepped-wedge and local control cITS analyses also suggested an increase in the incidence, were not significant (p = 0.33 and 0.38, respectively). Note that the number of incidents of reported crimes during weekdays was so low, and often none were recorded, that the relative effects of the intervention are meaningless. Whereas the effect for weekend crimes corresponds to approximately two additional reports per LSOA per month on average, and during weekdays, these correspond to less than one.

The stepped-wedge analysis for reported ASB incidence provided weak evidence of an effect in the hypothesised direction and indicated an average reduction of 15.6% (−29.8%, 1.4%) which corresponded to less than one incidence averted per LSOA per month on average (see Table 8 and Figure 10). This was, broadly speaking, confirmed by the counterfactual analyses which indicated an average reduction of 39.3% (95% CrI −45.0% to −33.6%), but not by the comparison to local controls which did not provide any indication of an effect (p = 0.76).

FIGURE 10 Average effect (%) and approximate 95% confidence interval or credible intervals (synthetic controls) of CICA intervention for reported ASB incidents. (Figure not reproduced here; see the original article.)

Sensitivity analyses

As outlined in the Analyses Plan, we conducted additional sensitivity analyses for the stepped-wedge design to assess whether there might be a lag between the start of the implementation of the CICA intervention and any measurable effects. We assessed such lagging of effects for 6 and 12 months, and the results are presented in Table 9. Effect estimates vary considerably between the unlagged, 6-month lagged and 12-month lagged results and sometimes differ in direction of change. However, results are imprecise, and confidence intervals largely overlap. For the interpretation of Table 9, it should also be noted that the relative effects presented do not do justice to the absolute effects, which, because of the low incidence, numbers are generally small. Nonetheless, there is no indication of a pattern that would indicate that lagged effects are more likely than the null effects observed in the main analyses.

Interpretation

Overall, the analytic results do not indicate that the implementation of the CICA intervention has resulted in measurable effects on any of the health and crime outcomes (see Table 8), and there is similarly little evidence of a lagged period before any such effects are measurable (see Table 9). Although average reductions in incidence following the implementation in the stepped-wedge design were observed for reported ASB (−15.6%) and ambulance call-outs to the area (−5.5% across weekdays and weekends) in agreement with our hypothesised directions of effects, estimates were quite imprecise and mostly non-significant, while they were also not in agreement with the other design and analytic methods. Moreover, even if we would interpret any of these as causally related, for which ASB incidence seems the most likely candidate, average effect sizes are minimal and would have little practical weight.