Intervention fidelity, reach and exposure

This chapter presents results on the STASH study intervention fidelity, reach and exposure, and progression with criterion 1 (regarding the feasibility of implementing the STASH study). Specifically, we examined if PSs carried out intervention activities as intended (fidelity); the extent to which they and non-PSs interacted with different elements of the intervention (exposure); and the extent to which the whole year group came into contact with the intervention (reach). The exploration of fidelity, exposure and reach was designed to identify barriers to and facilitators of successful implementation, and enable us to interrogate hypothesised mechanisms of action in the STASH study programme theory (see Chapter 9).

Fidelity: did the peer supporters fulfil their role as intended?

We examined the extent to which PSs carried out the role as intended in the intervention design and as instructed at the PS training sessions. The role involved setting up ‘secret’ Facebook groups; inviting friends to join these groups; posting messages from the STASH study website; having face-to-face conversations about sexual health topics; and directing friends to the STASH study website (see Chapter 2, The STASH study intervention design for the feasibility trial). For the most part, PSs fulfilled the role as intended.

Peer supporter activity in the STASH study Facebook groups

Table 9 gives an overview of PS activity. Figures reported on the number of Facebook group members exclude the PSs who created that group and the STASH study trainer. Of 104 trained PSs, 87 used Facebook to share STASH study messages. The remaining 17 PSs used only face-to-face conversation. Facebook groups comprised an average of 12 members. Group membership remained largely stable over the intervention period (four groups gained and two lost one member). Most groups (65 of 87) were mixed gender, whereas 16 were all male and seven were all female. Seventy-five groups included non-PS friends (i.e. other S4 students from the year, hereafter ‘friends’) with an average of nine friends. This suggests good potential for messages to reach beyond the PSs.

Peer supporters posted 1343 STASH study-related messages over the intervention period, averaging 15 posts per Facebook-using PS. There were 746 recorded reactions to posts – ‘likes’, comments and shares, excluding flippant or unrelated comments, and comments from the STASH study trainers – averaging nine per Facebook-using PS. The majority of PSs (77%) posted at least three messages. Relatively few (n = 15) direct messages were sent to STASH study trainers by PSs, and only two of those were sexual health specific (the majority related to technical issues).

A Facebook Analytics report (using data from the bespoke API, which allowed users to log in to the STASH study website via Facebook) indicated 146 unique visitors to the website via the API. As only 87 PSs were using Facebook to log in to the website, this suggests a number of repeat logins by PSs. PSs who logged in via Facebook made a total of 408 site visits.

Figure 9 illustrates the temporal spread of message-sharing activity by PSs who were logged into the website via the Facebook API. Activity was relatively consistent over time, with spikes particularly around follow-up sessions. The graph does not capture all posting activity, because technical problems prompted many PSs to screenshot messages and post without using the API’s copy and share function. The PS questionnaire data (n = 88) suggested that 18% had problems logging into the website (via the API) and 37% had difficulties trying to share from the website to Facebook. Although this highlights a functionality issue, and a potential barrier to full participation, it also demonstrates the capacity and willingness of PSs to adapt and get around technical issues. Most PSs (57%) disagreed that they preferred to put messages into their own words rather than use words from the STASH study website, suggesting a preference for using the provided messages, rather than creating their own.

FIGURE 9 The STASH study website message-sharing activity (by PSs logged in via Facebook API). (Figure not reproduced here; see the original article.)

Peer supporters reported that Facebook may have served as a barrier to fidelity (and exposure). Of PSs using Facebook, 13% said that some of the people they wanted to join their group were not on Facebook (ranging from zero in school 4 to 21% in school 5) and 30% of all PS participants agreed that ‘people I wanted to join my group hardly ever use Facebook’ (ranging from zero in school 4 to 83% in school 3). This was more commonly reported by young men than young women (39% vs. 24%, respectively).

Facebook activity varied by school and by individual PS (see Table 9). For example, school 3 had the fewest PSs who primarily used Facebook (42%), but the highest average (self-reported) face-to-face conversations (12 per PS, twice the average for all schools). In interviews, PSs at school 3 said that their friends were ‘hardly using’ Facebook. Some PSs at school 3 and their friends said that they had mobile app notifications switched off, and thus were not aware of new posts. Conversely, PSs at school 4 reported relatively few face-to-face conversations, but had a higher proportion of PSs using Facebook (90%), higher Facebook group membership (including more non-PS members), more posts per PS and no PS reporting that their friends did not use Facebook. The reasons underlying variation in social media use by school were difficult to determine in this small sample, but could relate to emergent subcultures within schools, in which friendship groups go with one platform or another.

Although friends interviewed knew that the STASH study Facebook groups had been set up, and some knew that there was a website, not all said that they had engaged in-depth with the topics or messages (or were not willing to say so in a group interview context). This varied slightly among interview groups, with some friends more aware of some STASH study topics and able to discuss specific messages around, for example, ‘consent’ and ‘kinds of infections’ (school 4, female friends). These participants also noted ‘links to YouTube videos and stuff’, although typically said that they had ‘never really looked’ at these or other external links, a finding echoed elsewhere by male and female participants. We had initially expected signposting to external sources to be useful (see Chapter 2), but these data suggest that the ‘ask’ of non-PS participants (following a link from a STASH study message to an external site) might be too much. The process evaluation data confirmed that we were right to replace much of the signposting and text-heavy messages with primarily visual messages (memes, infographics), as these were more positively received, with participants noting that these stood out and ‘. . . were cheesy but alright . . .’ and ‘intrigued me more’ (school 6, friends mixed group).

Peer supporter engagement in face-to-face conversations

Most of the PS questionnaire participants (86%) said that they had talked with friends about STASH study topics, with 85% of these participants reporting at least three conversations (see Table 9). There was some variation across schools, with 100% of PSs in school 3 (low level of Facebook use) reporting at least three conversations, but only 67% of PSs reporting at least three conversations in school 6. Overall, more young women (86%) than young men (83%) reported having three or more STASH study conversations with friends. This included 76% of PSs (80% of girls, 72% of boys) who had advised a friend on where to seek help on sex and relationships.

Interviews explored in more detail how PSs engaged friends in face-to-face conversations. One PS described his conversations as follows:

[I was] just saying how they can see everything on the website, and I obviously showed them [. . .] and then it was just about, like, sexual health and keeping yourself safe in all these different situations, and not just about having sex. It was about the stuff behind it like consent and all that.

Most friend interviewees also said that the conversations they had with a PS were quite general, about the fact that they were attending STASH study meetings and not necessarily much about STASH study topics per se:

Every now and then there was a ‘we done this, it was like this, and then this was said’ [. . .] But there wasn’t really that much [discussion].

Friends of PSs did seem aware of what the PSs had been asked to do, ‘Talking to people. If people are, like, worried about something’ and most had a general sense that the STASH study was about STIs, ‘diseases and stuff’ and ‘safer sex’, suggesting some diffusion of the STASH study messages.

Extent to which peer supporters were representative of the year group

Table 10 presents baseline characteristics (n = 680) by intervention group: trained PSs (regardless of whether or not role completed), nominees (peer nominated and invited to recruitment meeting, but did not attend training) and other students (neither nominated nor trained). In general, the nomination process identified students from higher socioeconomic backgrounds than other students, but the nominees who actually took up the role (the PSs) tended to be similar to other students on most of these indicators (specifically SIMD quintile, FSM and home ownership). Both nominees and PSs reported higher academic attainment. There was little difference in ethnicity, religiosity and sexual orientation between the three groups. The 50 : 50 gender split for nominees reflected gender stratification by the research team in selecting nominees; however, uptake of the role reflected the actual gender split of students in the year group (a preponderance of females).

Extent to which peer supporters reached their social network

We used SNA to evaluate the potential for diffusion of STASH study messages across the S4 year group via social network friendship ties (see Chapter 3 and Appendix 3). SNA data were generated from the ‘friendship modules’ in the baseline and follow-up questionnaires, and are distinct from friendships denoted by Facebook group membership and other PS activity. SNA friendship ties were mapped using data from all students for each of the six S4 year groups prior to the intervention.

Describing the school networks (pre intervention)

Four of the six school networks formed a single component; that is, every student in these schools had a path of ties connecting them to every other student in the network (S4 year group). Schools 4 and 6 had disconnected components, meaning that some students had no links to the rest of the year group. On average, students were separated by five or six steps (degrees of separation), with the exception of school 4, in which this average was three steps. Students averaged seven friendship ties (SD 3.8, range 1–19). This included both ingoing ties towards the node (i.e. others that had named that student as a friend) and outgoing ties from the node (friends that student had named). Network density – the proportion of actual ties from all possible ties in the network – ranged from 2% to 5%. This suggests a fairly low level of cohesion,90 although that may also reflect that students were able to name only up to six friends, and could include non-school friends in their list.

We used a stochastic actor-orientated model (see Chapter 3 and Appendix 4) to analyse change in the year group networks over time. Analysis was undertaken on students completing the friendship module at baseline and follow-up, and used data from five schools. School 4 was excluded because of a low Jaccard coefficient. This coefficient can be considered as a measure of stability: a low value means that the turnover in the network is too high to consider the data as evolving, limiting the possibility to apply RSiena (simulation investigation for empirical network analysis). For all other schools we found good values of Jaccard coefficient, ranging between 0.27 and 0.35.

Results showed that friendship ties tended to be reciprocated and two friends were likely to share the same friends. ‘Popular’ students (those most frequently named as friends) did not necessarily receive or make many more friendships over time. However, students who named more friends at baseline were more likely to add more friends at follow-up. We found that young women tended to name more friends than young men (Figure 10a) and, overall, there was a preference for same-gender friends (see Figure 10c). Level of sexual experience did not appear to be related to friendship ties (i.e. students did not tend to name friends with similar levels of sexual experience). Finally, friends were more similar at follow-up than at baseline in their assessment of STI risk with inconsistent condom use, but not in their self-assessed confidence to refuse sex without a condom.

FIGURE 10 Gender influence in the selection of named friends. Values are log-odds (95% CI). (a) The difference in number of friends named between boys and girls; (b) the gender difference in ‘popularity’; and (c) the similarity (homophily) between friends by gender. RE, random-effects model. (Figure not reproduced here; see the original article.)

Peer supporter distribution in friendship clusters

We investigated the distribution of PSs across friendship clusters. The presence of clusters represents a potential barrier to message diffusion because they suggest a level of segregation that those outside the cluster can find difficult to access. Therefore, PS presence across clusters is desirable. In Figure 11, nodes (circles) represent students and links among them indicate friendship ties. Friendship clusters (identified by using the Girvan–Newman cluster detection algorithm; see Chapter 3) are highlighted in different colours and represent groups of students with many reciprocal ties (dense connections). PSs are highlighted blue. The extent to which PSs were distributed across friendship groups varied between 33% and 80%. PSs were present in one-third of subgroups in school 4, in 45% in school 6, in half of subgroups in schools 1, 5 and 3, and in the majority of subgroups in school 2. This is consistent with findings from ASSIST40 (in which PSs were present in 50–60% of clusters) and suggests that PSs had reasonable potential to reach across the network, with some variation by school.

FIGURE 11 Peer supporter distribution in friendship subgroups by school. (a) School 1; (b) school 2; (c) school 3; (d) school 4; (e) school 5; and (f) school 6. Nodes (circles) represent students; links among them indicate friendship ties. Friendship clusters are highlighted in different colours; solid blue circles are PSs. (Figure not reproduced here; see the original article.)

The direct reach measure indicates that, overall, one-third of students (34%, n = 302) were directly connected to a PS. Schools 1 and 3 had the largest proportion of students directly connected to a PS (52%), whereas the smallest proportion was in school 6 (7%).

Peer supporter reach by Facebook groups

We also examined PS friend connections in the STASH study Facebook groups. In Figure 12, PSs are indicated in orange and friends who were at least in one Facebook group in light blue. In schools 1 and 5, 53% of students (excluding PSs) were members of a STASH study Facebook group (n = 90 and n = 71, respectively). In school 2, 37% (n = 37) of students were linked on Facebook, whereas in school 6 this was 25% (n = 31). Few students were connected through Facebook in school 3 (9%, n = 16) and 4 (2%, n = 1). As described in Peer supporter activity in the STASH study Facebook groups, PSs in school 3 tended to use face-to-face interactions rather than Facebook. School 4 encountered additional challenges in implementing the STASH study, which seems to be reflected here in the minimal reach beyond the PS.

FIGURE 12 Peer supporter distribution in Facebook memberships by school. (a) School 1; (b) school 2; (c) school 3; (d) school 4; (e) school 5; and (f) school 6. PSs are indicated in orange and friends who were at least in one Facebook group in light blue. (Figure not reproduced here; see the original article.)

Comparison of peer supporter network position with nominees who did not take up the role and other students

Comparison with those nominated and invited, who did not take up the role

We compared the network position of students who trained as PSs with those who were nominated but declined the role. To see if there were differences in centralities between the two groups, we used the permutation-based t-test procedure in UCINET version 6.6 (Analytic Technologies, Lexington, KY, USA), a type of inferential test specifically designed for network data that are not statistically independent. The number of friendship module nominations received (indegree centrality) varied only slightly between nominees and PSs [5.3 (SD 2.4) nominations vs. 5.1 (SD 2.5) nominations, respectively], and the difference in means was not significant (p = 0.492). The two groups were also similar when taking into account the network centrality of their friends [eigenvector centrality score 0.042 (SD 0.15) for nominees and 0.045 (SD 0.17) for PSs, with a high p-value of 0.889]. The proportion of the year group that PSs could potentially reach in two steps (two-step reach centrality) ranged from 9% of students in school 1 to 60% of students in school 4. Across schools, the mean number of students reached was 27 (SD 7.9) for PSs and 24 (SD 8.6) for nominees. A two-tailed test indicates a difference of –2.737 (p = 0.03). We can therefore conclude that, on average, nominees who declined the role could potentially reach fewer students in two steps than the trained PSs. These results suggest that PSs were similar in terms of friendship ties to those who were nominated and invited but did not take up the role, except that they had better two-step reach centrality. This has implications for progression criterion 1, as it suggests that, despite a lower proportional uptake, we were able to recruit a cadre who were as well positioned (or more so) as those who did not take up the role.

Comparison of network position between peer supporters and other students

Across all schools, PSs had greater potential to reach across networks than students who were not nominated (‘other students’). The average number of incoming ties (indegree) was higher for PSs: they were named as a friend on average 5.3 times (SD 4) compared with 3.5 (SD 2.2), with a p-value of 0.0001 for the two-tailed test. We can therefore conclude that there was a difference in centrality between the two groups. Similarly, PSs had potential to reach 27% of students in two steps, whereas other students had potential to reach only 20% (p = 0.0001). However, means calculated via eigenvector centrality were no different between PSs and other students [0.04 (SD 0.17) and 0.03 (SD 0.12), respectively], with a high p-value (p = 0.1677). This suggests that there was no difference in the extent to which PSs and non-PSs were connected to ‘popular’ (well-connected) others in the network.

Participant perceptions of reach and representativeness

In terms of intervention actors viewing selection of PSs as fair and coherent,60 a perception of effective reach and representativeness of PSs by participants is important. Overall, PSs were viewed as representing a broad range of students. In interviews, PSs largely suggested that the group offered reasonable representation of their year, noting ‘It was a good mix of different groups’ and ‘pretty much every group was covered’ (school 3, PS boy). Friends of PSs suggested a similar view:

Yeah, there’s one [PS] for every friend group, if you know what I mean.

School staff concurred with this view, with both school 4 teachers, for example, noting a ‘real mixed bag of pupils’. PSs also perceived that they had potential to reach across the year (perhaps more so than was suggested by the SNA):

Among friends of PSs, a perception that more students should have been ‘involved’ belied a possible lack of comprehension about the mechanism of change (PSs as conduits of messages), with possible impact on coherence of the intervention for some students:

Potential for ‘contamination’ beyond secondary 4 groups

Sharing of STASH study messages with young people (aged > 14 years) in other schools may be regarded as a positive impact of the intervention. However, it could also represent ‘contamination’ in the context of a future trial in which intervention and control schools are in close proximity. Across all six schools, 80% of friends named in the friendship module were from S4 at that school. In five schools, the proportion of named friends from outside S4 was < 10%. The exception was school 3, in which 61% of friends were from outside S4 (a relatively high proportion, which could be explained by factors such as a greater proportion of friends in other year groups). With regard to message reach beyond intervention schools, 25% of PSs across all schools said that they had shared STASH study messages with students outside their school, ranging from 8% (school 2) to 40% (school 1). Most PSs who shared in this way did so once or twice. These findings indicate some potential for both influence and contamination beyond intervention schools.

Differences between exposed and unexposed students

Using data from the follow-up questionnaire, students were categorised by exposure to the STASH study intervention as follows:

• trained PS
• exposed non-PS student who had:
  • been shown the STASH study website by a PS or accessed it themselves
  • joined, viewed posts or followed links in a STASH study Facebook group
  • talked to friends about STASH study Facebook posts
  • talked to or asked a PS about a STASH study topic
• unexposed: non-PS student who did not meet any above condition
• unknown: non-PS student who did not complete the follow-up questionnaire (meaning exposed or unexposed status could not be assigned).

Among students for whom baseline data were available, 97 (14%) of the 677 students were trained as PSs, 241 (36%) were exposed to the intervention (non-PSs) and 224 (33%) were unexposed. A total of 118 (17%) students did not complete the follow-up questionnaire and so their exposure status was unknown.

Among students completing the follow-up questionnaire (although not necessarily the baseline), 58.1% (n = 611) of the year group reported exposure to STASH study activities (including PSs). Exposure was higher for young women (62.6%, n = 342) than for young men (53.4%, n = 247).

Sample characteristics by level of exposure to the STASH study

Table 11 shows baseline characteristics by exposure. PSs and exposed students were more likely than unexposed students to be female. There were no differences across exposure group by area deprivation (SIMD) or residence type, but PSs had higher FSM eligibility than the other two groups. There was a gradient in academic attainment, with a greater proportion of PSs studying for National 5 qualifications, followed by exposed and then unexposed students. There was little difference between exposure groups in terms of ethnicity, age, religiosity and sexual orientation. There was some suggestion of a gradient in terms of sexual experience, with PSs more likely to report sexual experience, followed by exposed and then unexposed students. This suggests that the intervention may have reached the groups for whom STASH study information was most immediately relevant.

We examined student perceptions of school climate (e.g. perceived fairness, student–teacher trust, encouragement of participation) and compared across exposure groups to examine whether or not the peer nomination process selected students with a more positive perception of their school. Differences were small and seemed unlikely to be significant. In Table 11, we highlight two key measures: (1) perceived connectedness to others at school and (2) trying hard at school. Perhaps as expected, given the peer nomination process, PSs reported feeling closer to others at school than other students, but if anything they were less likely to say they tried hard at school.

Exposure to STASH study-related conversations

We examined data on students’ conversations about STASH study-related topics as a means of assessing any change between baseline and outcome that may have reflected diffusion of STASH study messages. Table 12 shows the proportion of students at baseline who reported talking about STASH study-related topics, by exposure, and the change in the proportions at follow-up. At baseline, PSs were more likely than other students to report conversations about all STASH study-related topics, and had greater increases in discussions about body and looks and STIs than other students. However, conversations about almost all of the topics increased regardless of exposure.