Data cleaning and selection of control trusts
After all exclusions 38 (3 treated and 35 controls) MHTs and 133 (6 treated and 127 controls) acute NHS trusts were included. The data set for the 37 MHTs extracted from HES-APC to proxy admissions to a mental health inpatient ward included 549,343 admissions for patients aged between 18 and 75 years from December 2012 to January 2021. The data set for the 133 acute trusts extracted from HES-ED data set to proxy mental health attendances at ED contained 1,775,237 attendances for patients aged between 18 and 75 years from December 2012 to January 2021. Patient-level data were aggregated to trust level to create 24-month data series pre and post implementation at each trust.
For each of the treated and control trusts, we plotted the primary outcomes during the 24 months before and 24 months after the opening of the PDU. SGUHFT was excluded due to low data counts, indicating that not all data were submitted to HES. Selected months for the study periods for the treated trusts also had to be excluded from the analysis due to data recording and quality issues. See Table 4 for updated pre and post intervention study periods for each of the treated studies. Three control MHTs and 12 control acute trusts that had clearly irregular patterns of the outcome as a result of reporting errors, or non-submission of data to HES, were also excluded. The 20 and 10 closest peers for each of the 5 remaining treated acute trusts and the 3 remaining treated MHTs, respectively, were identified from the remaining pool of 115 control acute trusts and 32 control MHTs as described in the methods (see Appendix 5, Table 47).
Characteristics of patients
Control trusts were carefully selected based on their similarity to the treated trusts across a range of variables. These variables, which include data on capacity, workforce availability, patient characteristics, management structures, regional characteristics (including index of multiple deprivation and rurality), target similarity across the trust and catchment population as a whole, rather than across mental health services and patients presenting with mental health issues. This process thus aims to ensure that treated and control trusts are similar in their potential capacity to deal with patients presenting with mental health issues, rather than the actual capacity, but should account for a wide variety of unobserved variables that impact on the outcomes studied here. We further control for the specific characteristics of patients at risk of the outcomes studied here in the synthetic control method.
To assess whether the controls trusts were similar to the treated trusts in terms of key patient characteristics of patients with a mental health admission or ED psychiatric attendance, we compared the average distribution of key patient characteristics across the treated trusts to the average distribution across the pool of 20 selected control trusts, and across the excluded control trusts, in the 24 months prior to the opening of the PDU (see Appendix 5, Tables 48 and 49).
Psychiatric admissions
In the 24-month periods before each PDU opened, patients admitted to an MHT inpatient ward in each treated trust and their corresponding pool of control trusts were broadly similar with the following exceptions (see Appendix 5, Table 48):
- At SWLSG there were significant differences in the ethnicity and the method of admission. SWLSG had a smaller proportion of patients with white ethnicity (66.1% vs. 77.9% in the controls). SWLSG had a greater proportion of booked and planned admissions (14.7% booked and 22.1% planned vs. 6.2% and 17.5%, respectively, in the controls), admissions from mental health crisis resolution teams (29.2% vs. 15.3% in the controls) and fewer emergency admissions (< 1% vs. 4.6%).
- At LPFT, there were significant differences in the method of admission and multimorbidity. A much higher proportion of admissions came via mental health crisis resolution teams (65.2% vs. 23.7% In the controls) and there was a greater proportion of patients with two or more Elixhauser comorbidities (44.5% vs. 25.2% in the controls).
- At BSMHFT, there were significant differences in the ethnicity, source and method of admission. BSMHFT had a greater proportion of black, Asian and minority ethnicities (17.7% Asian and 16.2% black ethnicity vs. 5.6% and 4%, respectively, in the controls), fewer patients admitted from their usual place of residence (47.1% vs. 66.6%) and more admitted from penal establishments (16.1% vs. 4.3%). There were no admissions from mental health crisis resolution teams in BSMHFT (0% vs. 5% in controls) and fewer ED or emergency admission transfers from another provider (6.5% vs. 37.3%).
The majority of patients had a primary ICD-10 diagnosis code F20–29 indicating schizophrenia, schizotypal and delusional disorders (ranging from 26% to 40.2% in the treated trusts and from 23.8% to 24.5% in the aggregated control trusts).
Mental health attendances at emergency department
There were no significant differences in sex, age or ED arrival mode of patients of patients with an mental health ED attendance in the 24-month periods before each PDU opened between each treated trust and their pool of control trusts (see Appendix 5, Table 49).
There were no significant differences in ethnicity between each treated trust and their control trusts except in SWBHFT where there were greater proportions of black, Asian and mixed populations. The distribution of ED diagnosis codes and patient groups were significantly difference in all the acute treated trusts compared with their control trusts, except for ULHFT. There was no difference in the pattern of referral source between SWBHFT and their control trusts, but significantly fewer patients were referred by emergency services in Kingston Hospital NHS Foundation Trust (KHFT), ULHFT and UHBFT, and significantly more patients were referred by emergency services in STHFT, in comparison to their respective control trusts. There were significant differences in the distribution of discharge methods between each treated trust and their control trusts.
In summary, the comparisons across each treated trust and their respective pools of control trusts reveal some significant differences in the different trust catchment populations, perhaps reflecting differences in the disease burden, socioeconomic and other patient characteristics, or differences in the trust or local healthcare infrastructure, workforce or ways in which healthcare use is coded. It is not possible using these data to accurately determine the causes of these differences but since we are comparing the relative differences across the treated and control trusts, as long as these differences do not vary over time, the analysis should be unaffected. However, these differences highlight the difficulty of aggregating findings across multiple treated trusts.
Outcomes analysed
The GSC method was unable to provide adequate estimates for some of the outcomes in particular trusts due to a lack of fit in the pre-intervention period, or evidence that the result was obtained by extrapolation. These issues can arise either as a result of variability in the outcome in the treated trust that is not matched by any of the control trusts, often occurring because of data sparsity in the treated trust (e.g. KHFT had < 1000 mental health admissions in both the pre- and post-intervention periods), or evidence that the result was obtained by extrapolation.
The following outcomes related to admissions to a MHT inpatient ward failed:
- Mental health admissions and average LOS on a mental health inpatient adult ward: LPFT (lack of pre-intervention fit).
The following outcomes related to ED psychiatric attendance failed:
- ED psychiatric attendances: KHFT (lack of pre-intervention fit)
- percentage ED psychiatric attendances less than 4 hours: UHBFT (lack of pre-intervention fit)
- percentage ED psychiatric attendances less than 12 hours: all trusts except UHBFT and STHFT (lack of pre-intervention fit)
- percentage ED attendances admitted: KHFT and ULHFT (evidence of extrapolation).
Estimated trends in hospital use
Tables 16 and 17 summarise the estimated impact of the opening of the PDU on each of the outcomes during the study period after the opening of the PDU at each treated trust and in a meta-analysis at the MHTs and acute trusts, respectively. Hospital use can vary over time even without changes to the way that care is delivered and it would be misleading to attribute normal variation in hospital use to the opening of the PDU. To check this, we report the p-value, which is the probability that that an effect of at least the magnitude of that observed could have arisen by chance. If this probability is low (here we assume that a low p-value is one that is < 5%), the estimate is considered to represent a significant difference between the hospital use in the treated trust compared with the control trusts.
| MH admissions (per 10,000 trust catchment patients per month) | Length of stay (days) | Percentage of MH admissions with length of stay < 5 days | ||||||
|---|---|---|---|---|---|---|---|---|
| Trust | Diff. (RD%) | p-value | Trust | Diff. (RD%) | p-value | Trust | Diff. (RD%) | p-value |
| SWLSTG (November 2016 to October 2018) | ||||||||
| 13.8 | −0.4 (−2.5) | 0.310 | 57.3 | 2.6 (4.7) | 0.331 | 11.0 | −6.5 (−36.9) | < 0.001 |
| LPFT (January 2018 to December 2018) a | ||||||||
| 32.4 | – | – | 37.6 | – | – | 16.4 | 0.8 (5.4) | 0.578 |
| BSMHFT (November 2014 to October 2016) | ||||||||
| 17.3 | −0.5 (−2.8) | 0.749 | 73.7 | −15.5 (−17.3) | < 0.001 | 12.6 | 4.6 (58.3) | 0.044 |
| Meta-analysis (24 months after the PDU opened) | ||||||||
| - | −0.4 | 0.310 | – | −6.4 | < 0.001 | – | −0.5 | 0.688 |
| Attendance (per 10,000 trust catchment patients per month) | Wait | Admissions an acute bed at same provider | Arrival by ambulance or police | Length of wait (minutes) | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| < 4 hours | < 12 hours | ||||||||||||||||
| Trust | Diff. (RD%) | p-value | Trust | Diff. (RD%) | p-value | Trust | Diff.(RD%) | p-value | Trust | Diff. (RD%) | p-value | Trust | Diff. (RD%) | p-value | Trust | Diff. (RD%) | p-value |
| KHFT (November 2016 to October 2018) | |||||||||||||||||
| 8.6 | – | – | 99.6 | 4.0 (4.2) | 0.012 | 99.9 | – | – | 11.5 | – | – | 46.7 | −3.3 (−6.6) | 0.957 | 26.2 | −24.6 (−48.5) | 0.034 |
| ULHFT (January 2018 to December 2019) | |||||||||||||||||
| 7.0 | 0.5 (7.2) | 0.510 | 90.1 | −2.4 (−2.6) | 0.114 | 100.0 | – | – | 7.1 | – | – | 35.8 | −9.1 (−20.4) | 0.003 | 121.8 | 23.6 (24.0) | 0.328 |
| SWBHFT (November 2014 to October 2016) | |||||||||||||||||
| 9.9 | 2.0 (25.3) | 0.146 | 99.2 | 0.2 (0.2) | 0.307 | 100.0 | – | – | 13.1 | −1.1 (−8.0) | 0.877 | 58.9 | 0.4 (0.7) | 0.869 | 82.1 | −4.7 −5.4) | 0.360 |
| UHBFT (November 2012–October 2016) | |||||||||||||||||
| 3.1 | −0.1 (−2.3) | 0.904 | 99.5 | – | – | 100.0 | 0.3 (0.3) | 0.715 | 12.9 | 6.7 (106.2) | 0.159 | 57.3 | 1.8 (3.3) | 0.994 | 71.5 | −2.9 (−3.9) | 0.970 |
| STHFT (March 2019 to January 2020) a | |||||||||||||||||
| 4.7 | −1.5 (−24.9) | 0.034 | 88.2 | −1.8 (−2.0) | 0.437 | 99.9 | −0.1 (−0.1) | 0.843 | 7.9 | −3.6 (−31.3) | 0.451 | 48.8 | 5.3 (12.3) | 0.062 | 141.7 | 35.0 (32.9) | 0.550 |
| Meta-analysis (24 monthsa after the PDU opened) | |||||||||||||||||
| – | −0.2 | 0.774 | – | 0.2 | 0.907 | – | 0.1 | 0.580 | – | 0.0 | 0.989 | – | −1.0 | 0.723 | – | 1.1 | 0.902 |
As an alternative approach to presenting these data, Appendix 5, Figures 57 and 58 show the trends in each of the outcomes for the treated trusts (red lines) and the counterfactual trend estimated by the GSC method (blue line) using a pool of 10 control trusts at the MHTs and 20 at the acute trusts, respectively. The blue line is our estimate of the trend in the treated trust if the PDU had not been implemented. Note that the two lines are similar, by design, in the period prior to the opening of the PDU (i.e. to the left of the first grey dashed line), as we aimed to find a counterfactual that tracked the outcome in the treated trust over this period. The difference between the two lines in the study period after the opening of the PDU (i.e. to the right of the grey dashed line) provides an estimate of the impact of the PDU on that outcome.
Psychiatric admissions
We found no significant impact of the opening of the PDU on admissions to mental health inpatient wards at any of the treated trusts or in a meta-analysis. Although there was a consistent trend towards fewer admissions per 10,000 people per month in SWLSG (RD −2.5%, p = 0.310), and in BSMHFT (RD −2.8%, p = 0.749) for the first 24 months after the PDU opened, compared with the counterfactual, these results were not significant (see Appendix 5, Figure 57a and b).
Length of stay on mental health inpatient wards
The average LOS for patients admitted to an MHT inpatient ward was 15.5 days lower (RD −17.3%, p < 0.001) in BSMHFT (see Appendix 5, Figure 57d) compared with the counterfactual in the 24 months after the PDU opened in November 2014 (averaging 89.2 days in the counterfactual and 73.7 days in BSMHFT during this period). This trend was evident in the first 12 months after the PDU opened in November 2014 (RD −15.5%, p < 0.001), and from November 2015 to the end of the study period in October 2016 (RD −19.3%, p < 0.001). There was no significant impact of the opening of the PDU on average LOS for patients admitted to an MHT inpatient ward in SWLSG. In a meta-analysis including both trusts, there was a significant reduction of 6.4 days (p < 0.001).
Length of stay in mental health inpatient wards less than 5 days
The average proportion of patients admitted to an MHT inpatient ward for less than 5 days in SWLSG in the first 12 months after the PDU was opened was 11%. This was 6.5% lower than the counterfactual during this period (RD −36.9%, p < 0.001). Conversely, LPFT had an increase in the proportions of patients with a LOS less than 5 days of 4.6 days compared with their counterfactual (RD 58.3, p = 0.044). There was no significant impact of the opening of the PDU on LOS less than 5 days in Birmingham and Solihull. A meta-analysis across the three sites indicated no evidence of any impact (see Appendix 5, Figure 57e–g).
Emergency department mental health attendances
Between March 2019 and January 2020, during the first year after the opening of the PDU, STHFT had 1.5 fewer ED attendances per 10,000 trust population per month than the counterfactual. This represented approximately one-quarter fewer ED attendances (RD −24.9%, p = 0.034; Figure 58b, A.4). There was no significant impact of the opening of the PDU on ED attendances at any of the other treated trusts (see Appendix 5, Figure 58a–c) nor any impact after meta-analysis.
Proportion of psychiatric patients waiting at emergency departments less than 4 hours
The proportion of ED psychiatric attendances waiting to be seen, transferred or discharged in less than 4 hours stayed fairly constant after the opening of the PDU in KHFT in November 2016 compared with a sharp decline in the counterfactual from March 2017 onwards (see Appendix 5, Figure 58e). Compared with the counterfactual, the percentage was 4.0% higher in KHFT between November 2016 and October 2018 (RD 4.2%, p = 0.012). There was no significant opening of the PDU on the proportion of psychiatric ED attendances at any of the other treated trusts (see Appendix 5, Figure 58f–h) individually or in the meta-analysis.
Psychiatric waits at emergency departments less than 12 hours
All treated trusts had close to 100% of ED psychiatric waits of < 12 hours. However, no significant effects of the opening of the PDU were seen in either of the two treated trusts, UHBFT and STHFT, for which the GSC was able to produce a satisfactory counterfactual (see Appendix 5, Figure 58i and j), or in the meta-analysis.
Proportion of psychiatric admissions to an emergency department
We found no significant impact of the opening of the PDU on the proportion of ED psychiatric admissions to an acute bed at any of the treated trusts or in a meta-analysis. Although there was a consistent trend towards a greater proportion of admissions in UHBFT after the opening of the PDU (see Appendix 5, Figure 58l), and a smaller proportion in STHFT, in the months studied after the opening of the PDU (see Appendix 5, Figure 58m) compared with the counterfactual, these results were not significant.
Psychiatric arrivals at emergency departments by ambulance or police
The proportion of psychiatric patients arriving at the ED either by ambulance or police in the months studied after the opening of the PDU ranged from 35.8% in ULHFT up to 58.9% in SWBHFT. We only found a significant impact of the opening of the PDU on these proportions in ULHFT, where the rate was 9.1% lower than the rate that would have been expected if the PDU had not opened (RD −20.4%, p = 0.003; see Appendix 5, Figure 58o). Given an estimated trust catchment population size of 354,756 for patients aged between 18 and 74 years in 2018, and a rate of ED psychiatric attendances of 7.0/10,000 in this catchment population, this represents approximately 23 fewer ED psychiatric arrivals by ambulance or police each month in ULHFT. There was no significant impact of the opening of the PDU on the proportion of psychiatric arrivals by ambulance or police in the meta-analysis.
Emergency department psychiatric length of wait
Consistent with the lower proportion of patients seen within 4 hours compared with the counterfactual, the average length of ED psychiatric wait for ED psychiatric attendances in KHFT was 24.6 minutes lower. This represented an almost 50% reduction in the expected wait time if the PDU had not opened (RD −24.6%, p = 0.034; see Appendix 5, Figure 58s). At the other trusts, our estimates of impact ranged from a lower average wait time of 4.7 minutes to a higher average wait time of 35.0 minutes compared with the counterfactual, but these estimates were not significant. There was no significant impact of the opening of the PDU on ED psychiatric length of wait in the meta-analysis.
Sensitivity analysis
In sensitivity analyses – including just the nearest 10, rather than the nearest 20, control trusts in the control pool for the acute trust outcomes and vice versa for MHT outcomes – all significant results were replicated either as a significant result with a similar effect size, or with a similar effect size but not significant. As results are robust to choice of controls, we are reasonably confident that the results we are seeing are not just by chance alone.