Schwartz Center Rounds evaluation: survey

Overview

This chapter describes the survey element of the evaluation that was conducted concurrently with the case studies and represented the primary quantitative component.

Objectives

The primary objectives were to:

1. evaluate whether or not regular attendance at Rounds has an impact on health-care staff work engagement, well-being, social support for staff and behavioural change towards patients and colleagues compared with non-attenders (objective 7)
2. investigate whether or not frequency of Rounds attendance was associated with greater improvements in health-care staff work engagement, well-being, social support for staff and behavioural change towards patients and colleagues (objective 8)
3. determine factors associated with Rounds being perceived as useful, or barriers, to attending Rounds (objective 9).

Objective 1 was achieved by comparing changes over time in outcomes (0–8 months) between regular Rounds attenders and non-attenders from 10 providers. Objective 2 was achieved by examining relationships between the number of Rounds attended and level of change. Objective 3 was achieved by summarising responses to relevant survey questions (see also Chapter 6).

Methods

Design and setting

A two-wave survey of new Rounds attenders and non-attenders was conducted in 10 case study sites from 29 May 2015 to 8 October 2016. New attenders (i.e. those who had not attended a Round previously) were chosen for the intervention group so that it would be possible to detect the effect of starting to attend Rounds.

The 10 case study sites were identified, using purposive sampling, in order to maximise the variation of organisations in terms of size, location and type of provider. Four of the sites had been running Rounds for a number of years; at the other six, Rounds began during the study, meaning that we could recruit participants at their first ever Round (and most attenders would be attending their first Round). Six of the sites (including two ‘new’ Rounds providers) were also main case study sites (see Table 2). Originally we planned that the same 10 sites would fulfil both purposes, but following the pilot study it was determined that this may not yield enough new attenders, and therefore four established Rounds sites were replaced by four new Rounds sites.

Seven of the sites were acute trusts. One site ran Rounds jointly between an acute trust and a nearby mental health trust. One site was a mental health trust alone, and the final site was an independent (non-NHS) hospice. Negotiation for access to each was sensitive throughout to issues of data anonymity, confidentiality, informed consent and burden placed on staff.

The sample size for the survey was based on a power calculation involving the primary outcome, work engagement (see Appendix 13 for details).78 An increase of 0.33 in work engagement is associated with a 0.5% drop in staff absence (per staff member, this translates to more than 1 day fewer absences annually); we considered this an important difference. Based on the variance in work engagement from the national NHS staff survey, the total sample size required to detect this effect, with 80% power and allowing for clustering, was 228 (114 in each group). Allowing for a 50% response rate, a dropout between baseline and follow-up of 40–50%, and assuming that 50% of new attenders would become regular attenders (see definition below), a sample of 800 new attenders was needed. Following the pilot study, it was acknowledged that an average of 80–100 attenders in new Rounds sites, and 30–50 in established sites, could be achieved by recruiting up to four Rounds in each site, yielding a probable total sample of between 650 and 850 attenders.

For non-attenders, we assumed that the initial response rate would be lower, and the dropout rate between baseline and follow-up would be higher (as some respondents may become Rounds attenders). In addition, it was not possible to know in advance who had attended Rounds previously. Therefore, an initial random sample of 250 staff per site was chosen at eight sites. At the hospice, however, the size of the whole organisation was smaller than this, and at the joint acute/mental health site a sample of 250 staff across each organisation was used, meaning that the total initial sample size for the non-attender arm was 2680.

Recruitment and administration

Inclusion and exclusion criteria for participation were as follows. Note that categorisation into different groups could not happen until after the follow-up survey.

• Inclusion criteria:
  • (all groups) staff who have the opportunity to attend Rounds, and have not attended Rounds before the baseline survey
  • (regular attenders) staff who attended ≥ 50% of available Rounds between the two surveys
  • (irregular attenders) staff who attended at least one Round but < 50% of available Rounds between the two surveys
  • (non-attenders) staff who had not attended a Round previously, and did not attend one during the study period.
• Exclusion criteria:
  • people not employed by the provider
  • staff who had attended Rounds previously, including with a previous employer.

Recruitment and survey administration differed for attender and non-attender arms. In both cases, potential participants were first asked whether or not they had previously attended any Rounds: if they had, their participation ceased. See Appendix 13 for details of recruitment.

Measures

The 47-item questionnaire included the following measures, selected based on our literature review and initial theorising regarding how Rounds were expected to work (see Chapter 3). Unless stated, questions were asked in both the baseline and follow-up surveys in the attender and non-attender arms (see Report Supplementary Material 1 for full details of the measures and their reliability and validity):

• work engagement
• psychological well-being 12-item General Health Questionnaire (GHQ-12)
• self-reflection
• empathy
• communication with patients
• compassion
• peer support
• organisational climate for support
• absenteeism
• age, gender, occupational group, tenure with the organisation, grade, working hours
• extent of contact with patients
• (follow-up survey only) number of Rounds attended in the previous 8 months
• (attenders only) reasons for attending Rounds; whether or not they had attended in their own time; whether or not they had attended with immediate colleagues; whether or not they had ever presented at a Round; whether or not they contributed to Round discussions; and perceived usefulness of Rounds
• (non-attenders only) reasons for not attending Rounds.

Analysis strategy

The primary hypothesis was that there was a greater improvement in work engagement for regular Rounds attenders than for non-attenders. This was tested via a multilevel (to adjust for clustering within sites) analysis of covariance (ANCOVA), with work engagement at follow-up as the dependent variable, and work engagement at baseline as a covariate (this is mathematically equivalent to examining the change in work engagement with the baseline value as a covariate, but reduces non-essential covariance and other problems associated with difference scores).79 The analysis also included occupational group, grade, gender, age, length of service, working hours and extent of contact with patients as covariates.

The secondary outcomes were psychological well-being (Likert and ‘caseness’), self-reflection, empathy, communication with patients, compassion, peer support, organisational climate for support, and absenteeism. Most were examined using the same model as for the primary outcome (with checks to ensure that key assumptions of the model were not violated). The ‘caseness’ score of the GHQ-12 was analysed with a multilevel logistic regression model, and absenteeism with a multilevel negative binomial model.

Three further analyses were completed:

1. We used alternative definitions of regular attenders. It became clear during the fieldwork that using a strict 50% definition might mean that an irregular attender at one site had actually attended more Rounds than a regular attender at a different site, owing to imbalances in the frequency of Rounds at different sites. We adhered to this definition in our main analysis, but also tested alternative definitions of regular attenders in which two, three or four Rounds, respectively, were attended to meet the definition.
2. We tested the difference between anyone who had attended Rounds and those who had not, and used the number of Rounds attended as a numerical predictor of outcomes (with both linear and quadratic models): this is a direct test of whether or not the dose of Rounds has an association with changes observed.
3. To account for imperfections in the design and data collection, we also conducted three sensitivity analyses and examined potential moderators of the effects.

Pilot study

Before the survey methods were finalised, a two-wave pilot survey was conducted in two sites from 27 January 2015 to 1 January 2016 to test (1) the administration method for attenders, (2) the administration method for non-attenders and (3) the usefulness of measures (for both attenders and non-attenders). The pilot study comprised 52 questionnaires administered to attenders, 500 questionnaires sent to non-attenders, and 12 cognitive interviews.

In general, the pilot study confirmed that the methods used were appropriate, although a few small but important changes were made as a result. The full details of the methods and findings of the pilot study can be found in Appendix 14. Because most of the survey details remained unchanged, respondents to the pilot study in two organisations were also included in the main analysis.

Results

Descriptive statistics: sample

Including respondents from our pilot study, there were 1140 responses at baseline and 500 at follow-up, excluding 18 blank responses (15 in the main study and three in the pilot study) (Table 13).

There were 500 responses from participants at both time points: 207 of these were originally recruited as attenders, and 293 as non-attenders. However, 54 of the non-attender group subsequently attended at least one Round. A further six did not indicate (on the follow-up survey) whether or not they had attended any. Five of the attender group also did not indicate (on the follow-up survey) how many Rounds they had attended. Therefore, these 11 respondents could not be categorised. The total number in the final sample (293 + 207 = 500; 468 from the main study) (see Table 13) is larger than the 489 classified into the three groups (see below) because of the 11 who could not be categorised.

The numbers in each group, as defined in the protocol, were as follows:

• regular attenders – 51
• irregular attenders – 205
• non-attenders – 233.

Of the combined attender groups, 140 had attended at least two Rounds, 77 had attended at least three Rounds and 40 had attended at least four Rounds. The number of regular attenders was smaller than anticipated. This was because of a lower conversion rate to regular attendance than had been anticipated in the study design.

A breakdown of attendance, other background characteristics and survey completion details is given in Appendix 15. In many aspects, the profile of attenders and non-attenders is fairly similar. However, there is a bias towards clinical staff (particularly AHPs/scientific and technical staff) among attenders, and also towards more senior staff (particularly bands 7 and 8a). These differences are controlled for by including background variables (site of study, occupational group, grade, gender, age, length of service, working hours and extent of contact with patients) as covariates. Unless otherwise stated, all subsequent analysis includes these covariates and accounts for clustering by site using multilevel analysis.

Comparison of regular attenders and non-attenders on outcomes (primary method)

Table 14 shows the raw (unadjusted) means and standard deviations for each survey measure at baseline and follow-up in each of the groups. Of note is the high levels of poor staff well-being in our sample, as measured by the GHQ-12 caseness, suggesting that, at baseline, between one-quarter and one-third of the staff in our sample had sufficiently poor psychological health that they would benefit from professional intervention. This was a slightly higher than the 27% prevalence reported in a large previous study of NHS staff.80

There are some baseline differences between the groups, in particular between attenders and non-attenders. To examine the extent to which these reflect genuine systematic differences, as opposed to random fluctuations or differences in the background profile of the groups, we ran multilevel ANCOVAs (or equivalent multilevel generalised linear models) in which baseline values were predicted by group membership, controlling for background variables.

Most of the differences were not significant, with one important exception. Engagement was significantly lower among non-attenders than among regular attenders [adjusted difference 0.33; 95% confidence interval (CI) 0.05 to 0.62; p = 0.022]. This has implications for how subsequent results are interpreted, not least because engagement was specified as the primary outcome measure. It may be that findings could be affected by the baseline imbalance, which would justify using baseline measures as covariates in the main analysis.

There appear to be differences between the groups for self-reflection [F(2,320.8) = 4.886; p = 0.008]; however, post hoc tests of the specific group differences do not yield significant results. In particular, the difference between regular attenders and non-attenders is not quite significant (difference 0.36; 95% CI –0.02 to 0.75; p = 0.064). Nevertheless, this difference could have an impact on findings. Likewise for peer support, the overall group difference was just significant [F(2,324.2) = 3.048; p = 0.049], but the pairwise group differences were not, with the difference between regular attenders and non-attenders being 0.33 (95% CI –0.04 to 0.70; p = 0.076). For no other variables did the differences come close to significance. Correlations between survey variables are shown in Appendix 16. One aspect of the correlation between all the survey scores and the number of Rounds attended (see Appendix 16, Table 44) is that there do not appear to be any substantial correlations between psychological well-being and the expected intermediate mechanisms of self-reflection, empathy, communication with patients and compassion, suggesting that levels of psychological well-being are not related to these mechanisms in the way that had been anticipated. Table 15 shows the change from baseline to follow-up surveys in these scores for the two principal study groups, together with the adjusted difference between these groups.

It can be seen that most of the differences are small and non-significant. In particular, there is no effect found with the primary outcome (engagement).

The only significant effect is on the caseness score of GHQ-12: the binary categorisation of whether or not the respondent is sufficiently stressed as to warrant intervention. This decreased significantly more in the regular attenders than in the non-attenders group (a 13% decrease, compared with a 3% decrease in unadjusted scores). The odds ratio for this effect was 0.28 (95% CI 0.08 to 0.98); that is, the odds of being identified a case by the GHQ-12 were approximately 72% lower for regular attenders than for non-attenders.

The full results from the models reported in this section are available in Appendix 16.

Comparison of regular attenders and non-attenders on outcomes (alternative definitions)

As reported earlier, owing to inconsistency in the number of available Rounds at different sites, we also tested different thresholds for ‘regular’ attendance. We tried this with the requirement that either two, three or four Rounds should have been attended. In addition, we examined effects of any attendance.

In most cases the same conclusion was reached: there was no overall difference between regular attenders and non-attenders for most of the variables. Table 16 shows the regression coefficients for the primary outcome (engagement) and odds ratios for GHQ-12 (caseness) that had the significant result in the main test.

There was no difference between the groups in engagement regardless of the definition. For GHQ-12 caseness, measuring psychological well-being, there is a significant effect for attendance at two or more Rounds. The odds ratio of 0.40 suggests that the odds of being severely stressed are nearly two and a half times smaller for people who attend (at least two) Rounds than for people who do not attend, controlling for their psychological well-being before attendance. The effects for attending at least three or four Rounds are not statistically significant, even though the odds ratios are of a similar magnitude (for at least three Rounds) or greater (at least four Rounds). This is probably because of the smaller numbers attending at least three Rounds (n = 77) or at least four Rounds (n = 40). Even if a single Round is attended, there appears to be a significant effect: the odds of caseness are reduced by 43% compared with non-attenders.

We also examined whether or not there was any evidence that attending a greater number of Rounds was associated with a greater degree of change in outcomes. Full details are given in Appendix 16, but the analysis backed up the previous results: there was a significant effect for psychological well-being only (in both caseness and Likert form this time), with more Rounds attended associated with better well-being at follow-up.

Sensitivity analyses

As with most studies, the administration of the survey and integrity of the sample were not perfect, with three areas of concern:

1. incomplete data (< 10% on most variables, although slightly higher for some demographic variables; see Appendix 15)
2. some areas of the survey applying only to staff who had regular contact with patients
3. changes in respondents’ work between baseline and follow-up surveys.

As such, the principal analyses [those reported in Comparison of regular attenders and non-attenders on outcomes (primary method)] were repeated under different conditions.

• A full information maximum likelihood analysis was conducted (using Mplus; Muthén & Muthén, Los Angeles, CA, USA81) with all data to examine whether or not the results were affected by incomplete data.
• Analyses were repeated with only those respondents indicating that they had regular contact with patients.
• Analyses were repeated excluding any respondents who indicated in their follow-up questionnaire that their role had changed in any significant way, or (in a separate set of analyses) that they had moved organisation since the baseline survey.

In all cases, the conclusions reached were the same as those from the main analyses (i.e. that there was a significant effect on the GHQ-12 caseness score of psychological well-being, but not on the other outcomes). Therefore, we can conclude that the findings we reach (including the lack of significant results in most cases) are unlikely to be because of these areas of concern.

Moderator analysis

We examined a variety of other factors that might have had an impact on any effects of attendance at Rounds. The full results are in Appendix 16, but there were only three significant findings, which did not fit with prior expectations and are not easily explainable, and therefore are likely to have been type I errors.

Usefulness of Rounds

Respondents who had attended Rounds (whether or not regularly) were asked (in the follow-up survey) how useful they considered Rounds to be, on a scale from 1 (not at all useful) to 5 (very useful) (Table 17).

The majority of respondents in both groups found them more useful than not, but a higher proportion of regular attenders than irregular attenders described them as very useful, with a more even balance between somewhat useful and very useful. Among the irregular attenders, 5% found them not to be useful, and 16% felt neutral about their usefulness. This may give some clue as to why these respondents did not become regular attenders, although as the following section will show, there are other reasons.

We examined whether or not the perceived usefulness of Rounds (among attenders only) was associated with changes in the outcome variables. These models were based on the same types as previous models [multilevel ANCOVA for most outcomes, but multilevel logistic regression for GHQ-12 (caseness) and multilevel negative binomial regression for absence], controlling for the same covariates as the previous analysis.

Most models did not produce significant results, with no reason to suggest that the more useful an attender found Rounds, the more likely outcomes were to improve. However, the exception again was psychological well-being. For both versions of this outcome, relationships were found. In particular:

• For the caseness version of GHQ-12, there was an odds ratio of 0.56 (95% CI 0.36 to 0.86; p = 0.009); that is, for each extra degree of usefulness perceived (e.g. moving from neutral to somewhat useful), the odds of being classed as a case would diminish by about 44%.
• For the Likert version of GHQ-12, there was a regression coefficient of –0.11 (95% CI –0.16 to –0.05; p < 0.001); that is, for each extra degree of usefulness perceived (e.g. moving from neutral to somewhat useful), the GHQ-12 Likert score would move down on average by 0.11 points.

The two versions here give somewhat differing estimates of the importance of the effect; however, the conclusion from both is similar. The more useful attenders perceive Rounds to be, the higher their levels of psychological well-being after 8 months (when compared with their levels at baseline).

Other survey results

The survey included a variety of questions about perceptions of the Rounds and availability to attend (for attenders), or reasons for not attending (for non-attenders). These are summarised in the following tables.

Table 18 shows the reasons that regular and irregular attenders started attending Rounds. Being prompted to attend by managers or colleagues was an important reason for attending; however, interest in a topic was also regularly cited, and publicity had an effect.

Under the ‘other’ option, respondents were invited to write why they had begun to attend. Most responses here were variations on the above options, including combinations of them, but others included ‘I’m interested in finding out if the Trust will be improving how staff deal with their emotions regarding stress’; ‘It was offered at work and I had emotional admissions at work this last few weeks so came for support and to see if it helps’; ‘I think it has been a time of many changes and an opportunity to stop, reflect and take stock is important’; ‘I am very interested in the reflective process and encouraging staff to discuss and explore their feelings’; and ‘To see if it would add anything to what we already do in our team’. Clearly, the role of stress and emotions at work was important for some people, and also some people saw Rounds as an opportunity to take time out, reflect, and determine whether or not things could be improved.

Thirty per cent of regular attenders and 27% of irregular attenders said that they always managed to attend Rounds when they wanted to. Barriers preventing attendance are shown in Table 19 (respondents could choose as many options as they wanted).

The main reasons for not attending Rounds more regularly were practical: most of all, the pressure of time, and either not physically being able to get to Rounds (because of schedules or locations) or other things being seen as priorities. The role of seniority and autonomy in being able to attend was clear from a couple of the comments (e.g. ‘I had time, when I was on an office day. These are unrealistic for clinical floor staff like nurses who have a job to do on the shop floor’). When asked why Rounds were useful or not, one respondent said:

For most of the floor caregivers they have no idea of the existence of these Rounds let alone what happens at them. They seem to cater for middle managers, and doctors at a certain grade who can take an hour out of their day to go and attend. Mostly they are scheduled at a lunchtime. In a 40-bed ICU [intensive care unit] or A&E [accident and emergency] it seems a little unrealistic to send staff as each nurse always requires another nurse to relieve her to go anywhere.

Other questions about attending Rounds are summarised in Table 20.

Clearly, attending Rounds in one’s own time was crucial for attendance, with nearly half of all respondents having done this at least some of the time. Attendance with colleagues was another important factor, with just 10% of regular attenders not having attended with immediate colleagues. Regular attenders appeared more likely to contribute to discussions (although this may be because more Rounds attendance increases opportunities for participation in discussions).

Non-attenders were asked why they had never attended Rounds (Table 21) (respondents could select multiple options).

Some of the key reasons for non-attendance were respondents being unaware of what Rounds were, not thinking that they were invited (publicity issues) or not thinking that Rounds were relevant to their role; however, other key reasons fitted with the attenders’ barriers regarding time and other priorities.

Conclusions

The survey aimed to test whether or not regular attendance of NHS staff at Rounds was associated with higher engagement, well-being and other perceptions, attitudes and emotions than in those who did not attend. In addition, it sought to determine whether or not the extent of attendance (i.e. number of Rounds attended, or dose) was linked to the level of change.

We found no support for the primary hypothesis, that work engagement was higher as a result of attending Rounds, in terms of either regular attendance or the dose received. There are several possible explanations for this result (and for most of the secondary outcomes for which there were also no significant effects), which we discuss below.

However, an important result is for psychological well-being, for which we did find evidence of several effects. This was measured using the GHQ-12,82 which measures minor incidence of psychiatric morbidity. This tool is clinically validated, is well used in health-care and other populations, and has been shown to have links with other key outcomes across a variety of settings. Indeed, it is likely to be the best quality measure (in terms of validity and importance) of all those included in the survey. Therefore, the effects can be treated as important, particularly given the scale of the effects for the ‘caseness’ version of the measure, which is a clinically-validated classification of whether or not an individual would benefit from treatment. Among regular attenders of Rounds, the incidence dropped from 25% to 12%, compared with a drop of 37% to 34% among non-attenders.

The differences between the two groups at baseline should be acknowledged, and are an important feature of the interpretation. However, this is mitigated by three factors:

• First, the models controlled for baseline values, so the changes were the focus of the analysis.
• Second, the higher level of ‘caseness’ was found in the non-attenders group, where there was less change over time: this rules out any regression to the mean effects.
• Third, the effect with psychological well-being was not limited to this one effect but was consistent across a range of effects. In particular, there was an association with ‘caseness’ levels using different definitions of regular attendance, or comparing any attendance with non-attenders; there was an effect of the number of Rounds attended, and of the perceived usefulness of Rounds.

There were also significant effects when performing sensitivity analyses to account for incomplete data, and for several of these effects there were also associations with the other scoring method for the GHQ-12 (the Likert method). Therefore, the finding appears robust, and seems unlikely to be because of a type I error. This is despite a relatively small sample size for regular attenders (in particular, the 25% and 12% at baseline and follow-up represent just 13 and six respondents of the 51 who were defined as regular attenders according to our protocol).

Given the importance of this finding, therefore, it bears further examination of why no significant main effects were found with most of the other outcomes: not only engagement, but also self-reflection, empathy, communication with patients, compassion, peer support and organisational support. It is possible that, with the relatively small sample size, there is an effect that was not picked up, given some credence by the fact that in all five scores there was a move in the expected direction among regular attenders (the same was true for just two of the five scores in the non-attender group). It seems unlikely that it is entirely down to this, given the significant change in well-being, although it is worth noting that for empathy, communication with patients and compassion, the sample size was smaller still, as non-clinical staff generally did not respond to these questions, and therefore type II errors are more plausible for these.

It is also possible that the measures used were not sufficiently sensitive to change, or robust, to capture any differences that may have occurred. Although most had been used before in health-care settings, they are not as well-validated as the GHQ-12, and in some cases had response tendencies towards one end of the scale (particularly compassion, for which most respondents scored very highly even at baseline). It should also be considered that the intervention delivered, namely attending Rounds, was not consistent across the whole sample (even within sites); no two Rounds are the same, and in particular those sites starting Rounds (from which a substantial number of participants came) may take longer to develop a consistent approach.

Another possibility is that this is a limitation of the self-report nature of the questionnaires, combined with self-selection into the attender group. The very act of deciding to attend a Round may be an indication that the respondent is, for example, more ‘engaged’ in organisational life and has also begun to consider some of the key issues and how they might address them, meaning that scores are more positive even at baseline, giving less opportunity to improve. This is exacerbated by the fact that two-thirds of respondents did not manage to complete the questionnaire before the first Round had begun, but sometimes completed it during or after the Round. In these cases, there may have already been some effect of Rounds attendance: if not on outcomes, then on cognitive processes such as those measured. This would also explain why there might be substantial (although not always statistically significant) differences in baseline values between the groups.

The finding that psychological well-being is significant despite this actually gives further support to this hypothesis. The GHQ-12 asks about actual symptoms during the previous month; therefore, responses to this are not likely to be affected by having just attended a Round. However, the same cannot be said for many of the other measures.

Overall, the conclusion appears to be that attendance at Rounds might not have any particular impact on work engagement, or certain other attitudes and emotions, but there is good evidence to suggest that there could be a significant reduction in poor psychological well-being as a result of attending Rounds.