The process evaluation employed mixed methods to provide insight into the programme theory, logic model and evaluation design. Qualitative data presented in Chapter 5, and some quantitative data presented in Chapter 4, informed the process evaluation. Data are presented below in relation to (1) intervention-related findings on context, fidelity, exposure, reach, programme theory and logic model; and (2) evaluation design-related findings on recruitment, retention, contamination and researcher insights. A summary of how these findings address the HelpMeDoIt! progression criteria is presented in Chapter 7.
Part 1: intervention-related findings
Contextual factors
Contextual factors influencing the effect of the intervention include (1) the context in which the intervention itself takes place and (2) contextual factors that have either a negative or a positive impact on various pathways of the intervention (see Figure 4).
The HelpMeDoIt! intervention was designed to be implemented with minimal contact with the study team or health professionals, so it was delivered as an app and accompanying information website. The HelpMeDoIt! app was designed for use on both the Apple (Apple, Inc., Cupertino, CA, USA) and the Android (Google, Google Inc., Mountain View, CA, USA) operating system. Slightly more than half of participants used Android software (n = 38, 52%) on numerous different versions of smartphones (e.g. Samsung, LG, Huawei), and the remainder used Apple operating systems on iPhone devices (Apple Inc., Cupertino, CA, USA) (n = 35, 48%). The HelpMeDoIt! website was accessible from smartphones, desktop computers or tablets. Of those who accessed the website (n = 55), the majority of participants used a desktop computer (n = 35, 64.1%), followed by a smartphone (n = 17, 30.6%) and a tablet device (n = 3, 5.4%).
Contextual factors influencing the intervention have already been described in Chapter 5. In brief, these included (1) participants’ previous experience of weight loss and relapse; (2) significant life change, such as new job or moving house; (3) the influence of personality traits and mood; (4) influence and/or motivation of health issues; (5) social and group norms; and (6) multiple barriers to lifestyle change, for example environmental, participant-related, helper-related and/or social context barriers.
Fidelity of delivery of the intervention
Although the app was tested for a 3-month period using a testing group, some technical issues arose when it was initially launched via the app stores. The testing phase used a third-party platform allowing the app to be test-run before it was submitted to the app stores for release. This helped identify numerous bugs, which the software company resolved. There was no built-in testing phase after the app was available for download from the app stores. This unfortunately resulted in many technical issues being detected after the app had ‘gone live’. The majority of technical problems were associated with (1) the app crashing or working slowly because database information was downloaded to phones on each login; (2) elements of the key features not displaying correctly, which had an impact on how effectively participants could operate the app; and (3) the frequent emergence of new software bugs as a result of ‘app updates’ intended to solve previous issues. The software company had difficulty resolving software bugs for Android phones, which, unlike iPhones, are designed by numerous companies, using multiple combinations of operation systems. After several months of trying to stabilise the app, the software company carried out a ‘technical rebuild’. This did not change the content of the app but resolved the key issues and improved the overall quality of the app.
These issues had an impact on initial fidelity of the intervention being delivered as some participants were not able to access the key intervention components of the intervention via the app (i.e. goal-setting, self-monitoring and social support) and some of the app features did not work as intended. However, qualitative findings demonstrated that participants received the social support elements outside the app (e.g. meeting helpers face to face), which was in keeping with our programme theory and was not always because of technical issues but sometimes was participant and/or helper preference. Qualitative findings also indicated that some helpers did not access the website, which contained guidance on being a good helper, so they may not have delivered the social support elements as we had intended, for example using language that supported their friend’s autonomy.
Facilitators that might promote future fidelity of the intervention delivery include (1) resolving software bugs using both a ‘pre-live’ and a ‘post-live’ testing phase; (2) further streamlining the app and website based on findings from this feasibility study; (3) installation of the app on participants’ phones and a run-through with field workers at the baseline face-to-face visit (relevant for the study but not for wider roll-out); and (4) ensuring better signposting to the website.
Exposure to the intervention
As part of this feasibility study, we were interested in exploring how the intervention was used by participants and their helpers. One way of measuring this is through data use statistics; these were gathered and analysed to explore which features of the app and website were used, and how often they were used. The analysis of app and website use provided us with some meaningful insights into how the intervention was used and what elements of the intervention were potentially effective. A summary of app/website use is presented for participants in Table 34 and for helpers in Table 35. Engagement with the intervention was also explored in the qualitative work (see Chapter 5).
| Software component | Total | Range | Mean per participant (SD) | Median (IQR) |
|---|---|---|---|---|
| App use | ||||
| Number of logins | 955 | 1–408 | 17.7 (56.3) | 4.5 (2.0–11.0) |
| Number of views of main dashboard | 715 | 1–412 | 13.2 (57.1) | 1.5 (1.0–3.0) |
| Number of views of progress charts | 213 | 1–96 | 3.9 (13.7) | 1.0 (0.0–2.0) |
| Number of ‘enter your weight’ updates | 108 | 1–21 | 2.0 (4.2) | 1.0 (0.0–2.0) |
| Number of views of ‘smiles’ feature | 76 | 1–43 | 1.4 (6.0) | 0.0 (0.0–1.0) |
| Number of views of ‘rewards’ feature | 86 | 1–32 | 1.6 (5.0) | 0.0 (0.0–1.0) |
| Number of template goals created | 59 | 1–16 | 1.1 (3.0) | 0.0 (0.0–1.0) |
| Number of custom goals created | 898 | 1–143 | 16.6 (24.9) | 9.5 (4.0–18.0) |
| Number of goals deleted | 30 | 1–12 | 0.6 (2.0) | 0.0 (0.0–0.0) |
| Number of goals completed | 580 | 1–109 | 10.7 (23.1) | 3.0 (0.0–21.0) |
| Number of goals updated | 244 | 1–181 | 4.5 (24.7) | 0.0 (0.0–2.0) |
| Number of helpers invited | 45 | 1–8 | 0.8 (1.5) | 1.0 (1.0–2.0) |
| Number of uses of ‘contact helper’ feature | 27 | 1–8 | 0.5 (1.6) | 1.0 (1.0–2.0) |
| Number of smile sent to helper | 87 | 1–24 | 1.6 (6.3) | 0.0 (0.0–3.0) |
| Read summary e-mail | 150 | 1–40 | 2.8 (5.9) | 1.0 (0.0–3.0) |
| Website use | ||||
| Number of website sessions | 172 | 1–12 | 3.1 (2.6) | 2.0 (1.0–4.0) |
| Number of page views | 677 | 1–45 | 12.4 (11.1) | 8.0 (5.0–18.0) |
| Total duration of website sessions (minutes) | 372 | 1.6–43.8 | 6.8 (8.5) | 3.8 (1.2–7.5) |
| Software component | Total | Range | Mean per participant (SD) | Median (IQR) |
|---|---|---|---|---|
| App use | ||||
| Number of logins | 122 | 1–48 | 4.9 (9.5) | 2.0 (1.0–4.0) |
| Number of views of progress charts | 0 | 0 | 0.0 (0.0) | 0.0 (0.0–0.0) |
| Number of views of ‘smiles’ feature | 9 | 1–5 | 0.4 (1.0) | 0.0 (0.0–0.0) |
| Number of views of ‘rewards’ feature | 13 | 1–4 | 0.5 (1.0) | 0.0 (0.0–1.0) |
| Number of times goals viewed | 27 | 1–7 | 1.1 (1.5) | 0.0 (0.0–1.0) |
| Number of times goals liked | 21 | 1–8 | 0.8 (1.8) | 0.0 (0.0–1.0) |
| Number of uses of ‘contact friend’ feature | 9 | 1–2 | 0.4 (0.6) | 0.0 (0.0–1.0) |
| Number of smiles sent to participant | 95 | 1–46 | 3.8 (9.0) | 2.0 (1.0–3.0) |
| Website use | ||||
| Number of website sessions | 23 | 1–2 | 1.2 (0.4) | 1.0 (1.0–1.0) |
| Number of page views | 54 | 1–8 | 2.8 (2.0) | 2.0 (1.0–4.0) |
| Total duration of website sessions (minutes) | 111.7 | 1–27.3 | 5.9 (6.5) | 3.8 (2.4–5.6) |
Participant engagement with the HelpMeDoIt! app
Participant engagement with the app varied widely. Nineteen participants (26%) did not use the app at all. We assessed engagement with the app in comparison with 2017 use statistics for commercial apps, which reported that 21% of users will engage with an app only once.94 In our study, of the 54 (74%) participants who downloaded the app, six (11%) used the app only once, 48 (89%) used it twice or more, 38 (70%) used it three times or more, and six (11%) people used it ≥ 30 times. Thus, there was greater engagement with the HelpMeDoIt! app than the current standard within the app industry.
Data use statistics explored how often participants engaged with key features of the app (see Table 34). After logging in to the app, the top five activities were (1) creating custom goals, (2) viewing the main dashboard, (3) completing goals, (4) updating goals and (5) viewing progress charts. Of the 957 goals created by participants, 94% were custom goals, compared with 6% template goals, and 61% were recorded as completed.
A key feature of the intervention was the nomination by participants of one or more helpers from their social network. In total, 19 participants (36%) invited at least one helper via the app. In total, 45 helpers were invited, ranging from one to eight helpers per participant. Of the 45 invited helpers, 25 (56%) accepted the invitation and downloaded the app (helper use is reported below). Table 34 also demonstrates that participants used the social support elements of the app: the ‘send a smile’ feature; and ‘contact helper’ feature. Qualitative findings demonstrated that participants and helpers also interacted outside the app, either face to face or with a telephone call, text message or e-mail (see Chapter 5).
Participant engagement with the HelpMeDoIt! website
Of the 73 participants in the intervention group, 18 (25%) participants did not visit the website (similar to the number of participants who did not use the app). Among those who did use the website (n = 55, 75%) the number of visits ranged from 1 to 12 individual sessions, each session lasting 18 seconds to 44 minutes, with a total of 1–45 individual web pages viewed (see Table 34). Qualitative findings suggested that some participants were unclear about the purpose of the website, forgot about the website or did not find it helpful. More effective signposting of the website and the help that it can provide is an important consideration for a future trial. Although website use was limited, participants and helpers may have gathered sufficient information to proceed with the intervention from one or two visits.
Helper engagement with the HelpMeDoIt! app
App use by helpers is presented in Table 35. Forty-five individuals were invited as helpers. Twenty-eight (62%) accepted the invitation, of whom 25 (56% of nominated helpers) used the app at least once. Overall engagement was assessed against similar criteria to those used for participants: eight helpers (29% of accepted helpers) used the app only once; 17 (61%) used it twice or more; 10 (36%) used it three times or more; and only two (7%) used it 10 times or more. In total, there were 122 logins by helpers (ranging from 1 to 48). The top three features used by helpers were sending smiles to participants, viewing participant goals and liking participant goals. Qualitative findings suggested that helpers were uncertain how to help the participant using the app, with many providing support outside the app with face-to-face chats, text messages or telephone calls (see Chapter 5). The level of helper engagement with the social support element was therefore higher than indicated by the app use data.
Helper engagement with the HelpMeDoIt! website
Of the 28 helpers who accepted a participant invitation, 19 (68%) used the website at least once, and nine (32%) used the website twice. No helper visited the website more than twice. The number of pages viewed by helpers ranged from one to eight, with a session duration of 1.5 minutes to 27 minutes (see Table 35). Similar to insights from participants, qualitative findings showed that helpers were unaware either that a website was available or of how it could help them. Signposting the website more effectively to helpers should be considered for a future trial. A consideration is that helpers may not need to revisit the website often after reading the ‘helper guidance’.
Characteristics of app and website use
Table 36 shows summaries of selected measures of app use in relation to participant gender. Complete summary tables are provided in Appendix 8. Women were higher users of the app than men, but app use by helpers did not vary between helpers of male and female participants. Apart from gender, there were no other associations between participant baseline characteristics and app use. Participant age, Scottish Index of Multiple Deprivation (SIMD), BMI, physical activity and diet showed no correlation with the number of logins by participants or their helpers, or the number of goals set by the participants.
| App use | Participant gender | p-value | |
|---|---|---|---|
| Female (n = 49) | Male (n = 24) | ||
| Number of app logins (participant), median (IQR) | 4 (1, 10) | 2 (0, 3) | 0.042 |
| Number of goals created (participant), median (IQR) | 10 (0, 19) | 0.5 (0, 4) | 0.002 |
| Number of app logins (helper), median (IQR) | 2 (1, 3) | 2.5 (0, 3) | 0.947 |
Although participants were informed that this was an option, no participants enquired about the potential to be both a participant and a helper for someone else in their social network.
Potential mediating effects of intervention use
Within the intervention group, the potential mediating effects of measures of app/website use were assessed by testing the correlation between each measure of app/website use and each primary outcome measure (Table 37).
| App use | BMI (kg/m2) | Physical activity monitor | DINE questionnaire scores | |||
|---|---|---|---|---|---|---|
| Steps per day | Sedentary time | Healthy eating | Unsaturated fat | Fruit and vegetables | ||
| Number of logins (subject) | –0.381; p = 0.006 | 0.405; p = 0.050 | –0.408; p = 0.048 | 0.262; p = 0.069 | 0.273; p = 0.019 | 0.091; p = 0.551 |
| Number of goals created (subject) | –0.237; p = 0.098 | 0.332; p = 0.113 | –0.223; p = 0.295 | 0.276; p = 0.054 | 0.255; p = 0.029 | 0.266; p = 0.077 |
| Number of logins (helper) | –0.276; p = 0.182 | –0.134; p = 0.649 | 0.218; p = 0.453 | –0.056; p = 0.786 | 0.229; p = 0.242 | –0.352; p = 0.078 |
Those who made greater use of the app, as measured by the number of logins and goals created, showed greater reductions in BMI. Objectively measured physical activity (step counts and time spent in sedentary activities) was correlated with the number of logins, more than with the number of goals created. Healthy eating scores from the DINE questionnaire were correlated with increased app use, as were fruit and vegetable scores, particularly with the number of goals created. There was also a positive correlation between app use and DINE unsaturated fat scores. Helpers’ use of the app demonstrated less association with participants’ outcomes, with only fruit and vegetable scores showing some signs of a (counter-intuitive) negative association with the number of helper logins.
It is important to note that, although these identified associations could indicate mediating effects, these results could also be found as a result of reverse causality or be artefacts of another predictor of success; for example, people who are losing weight will maintain engagement with HelpMeDoIt! as it is going well, but they may have been successful anyway.
Dose–response relationship: illustrative case studies
Given the range of engagement with the app, and the possibility that for some individuals the intervention might be particularly helpful, we explored use characteristics and outcomes for the six (8%) intervention participants who used the app most frequently (range 30–408 logins). All of these participants lost weight (between 3% and 19% of their starting weight) and that weight loss typically increased as app use increased. In addition to having the largest number of logins, these six participants also had the highest engagement with the three key features of the intervention: goal-setting, self-monitoring and helper interaction. Interestingly, the participant who had greatest success with weight loss (a clinically significant weight loss of 19% body weight) was the individual who used the app most frequently, completed the greatest number of goals, had the greatest number of helpers using the app and had the greatest number of helper interactions. However, while it may be the case that using the app leads to more weight loss, it could also be that individuals may use the app more because they are being successful and losing weight.
Participant insights on Usability, Satisfaction and Ease of Use questionnaire
Overall, the response rate for the 30 individual USE questions ranged from 40% to 49% (n = 21–24 of the intervention group at follow-up). This response rate was lower than that for other measures in the questionnaire and was probably due to participants who had never, or rarely, used the app/website feeling that they could not provide informed responses. For example, it was difficult for participants to respond to ‘I can use it without written instructions’ if they had never used the app/website or had logged in only once. A ‘not applicable’ or ‘don’t know’ option should be included for a future study to improve data completeness. It is also possible that participants who did not enjoy using the app did not complete all of the USE questions, potentially skewing the data.
The USE questionnaire presented summed scores over four categories: (1) ease of use, (2) ease of learning, (3) usefulness and (4) satisfaction. Lower scores indicate more positive results. Among those participants who did respond, the mean ease of use score was 52.1 (range 14–77 from a potential range of 11–77), the mean ease of learning score was 15.2 (range 4–28 from a potential score range of 4–28), the mean usefulness score was 41.9 (range 8–56 from a potential range of 7–56) and the mean satisfaction score was 36.8 (range 7–49 from a potential range of 7–49). These findings suggest that, although the majority of participants found the app/website moderately easy to use and quite easy to learn, the majority did not find the app/website useful and were moderately unsatisfied. Qualitative findings suggest that the reasons for dissatisfaction and lack of usefulness were related to the technical issues experienced with the app. Importantly, better scores were demonstrated in participants who started using the app after the technical issues had been resolved. For example, the first five users of the app (in the presence of technical issues) reported a mean total USE score of 146 (range 113–210), compared with the final five users of the app (in the absence of technical issues), who had an improved mean total USE score of 136 (range 108–173). In addition, participants who used the app most frequently, and experienced no technical issues, provided the most positive scores. This demonstrates that when the app operated effectively it was perceived positively by participants.
Reach
Participant demographics were presented in Chapter 4. As is frequently seen in weight loss studies, most participants were female (n = 76, 69.7%).91 In terms of ethnicity, the sample was representative of people living in Scotland, with 97 (89.8%) being white British and 11 (10.2%) being of a minority ethnic or another background. The overall sample was also representative of Scotland’s relationship between obesity and socioeconomic status, with the majority of participants (n = 57, 57.6%) living in an area of high deprivation (Scottish Index of Multiple Deprivation level 1 and 2). Participants from a wide age range (25–68 years) were recruited, giving us insight into the feasibility of using an app and website intervention for different age groups. However, we had to approach (via primary care) or reach (via advertising) many people in order to recruit our sample, so it is likely to be a somewhat narrow and motivated group, which is not unexpected in a study on weight loss.
Qualitative and quantitative data were collected to measure the extent to which the intervention reached individuals other than participants, as reflected in the programme theory. Findings from participant and helper interviews demonstrated that some helpers made positive changes to their lifestyle in response to their involvement in the HelpMeDoIt! study. This included changes to physical activity and diet (see Chapter 5). Questionnaire data at follow-up also supported this finding, with 12 participants (14%) reporting their helpers making healthier food choices, 14 participants (17%) reporting their helpers increasing their physical activity, and six participants (7%) reporting that their helpers had successfully lost weight. This is an important consideration for the potential impact of this intervention in the future because if the intervention has a spillover effect, thus reaching a broader group of people, the potential impact could be positive even if only some of the individuals use the intervention as intended.
Programme theory and logic model
Chapter 2 previously described how the HelpMeDoIt! intervention was guided by a programme theory and logic model. Findings from stage 1 helped inform the second iteration of the logic model from version 1.0 to version 2.0 (see Chapter 2). In stage 2 we further explored the programme theory and the version 2.0 logic model using use statistics and qualitative data from participant and helper interviews. In addition to feedback from participants and helpers, observations by the study team and the software company helped refine a comprehensive programme theory for the HelpMeDoIt! intervention. The aim of this section is to summarise new contextual factors for consideration, how elements from the key qualitative themes influenced each other, and how they mapped onto the proposed programme theory and logic model.
Contextual factors identified within the HelpMeDoIt! programme theory
Contextual factors influenced whether or not the key mechanisms of change were successfully implemented. Various categories were identified in which contextual factors could negatively or positively influence the impact of the intervention. Categories included issues related to (1) available technology (i.e. make and model of smartphone), (2) participants themselves, (3) their helpers, (4) the environment and (5) broader contextual issues (previously described in Chapter 5). Many factors had already been considered in the early stages of development, such as aiming to develop an app/website that was easy to use and engaging, and being aware that participants would have varied access to physical activity opportunities. As the study progressed, we gained greater insight into additional contextual issues. Many of these focused on the participant–helper relationship and require consideration for a future trial. These included:
- participants feeling uncomfortable asking helpers for support or feeling that their weight goals were a burden to their helpers
- some participants feeling socially isolated and not having access to many helpers
- participants feeling embarrassed to share their weight and eating habits with their helper or to start using the app again after a long period of non-use
- invited helpers’ motivation to provide support being poorly understood (ethical constraints meant the study team could not contact helpers directly for feedback; helpers who were interviewed were contacted via the participant)
- participants’ ability to effectively and confidently ask for support, which influenced the clarity, or lack of clarity, that helpers felt towards their role.
Mapping qualitative themes to the HelpMeDoIt! programme theory and logic model
Table 38 summarises the ‘mechanisms of change’ and ‘intermediate outcomes’ from the version 2.0 logic model (see Figure 4) and identifies which of them were strongly supported by qualitative findings (i.e. discussed by the majority of participants). Theme numbers in bold strongly supported the identified mediator as a mechanism of change (e.g. increased/improved social support is strongly supported by qualitative theme number 3.3, ‘social support’). Theme numbers in italics suggest that the identified mediator is a potential mechanism of change (i.e. it was discussed by some participants but less often). However, further exploration in a future trial is needed (e.g. increased problem-solving is identified as a potential mediator by qualitative themes number 3.1 ‘goal-setting and self-monitoring’ and number 4.1 ‘change in behaviour’).
| Participant themesa | Helper themesa | |
|---|---|---|
| Logic model mechanism of change | ||
| Increased social support | 3.3 | 9 |
| Increased engagement with helpers via the app | 2, 3.3 | 6, 7, 9 |
| Increased interaction with helpers not via the app | 2, 3.3 | 9 |
| Reflect and set ongoing goals | 3.1 | 7, 9 |
| Increased action-planning | 3.1, 4.1 | 8 |
| Increased self-monitoring | 3.1 | 7, 9 |
| Increased problem-solving | 3.1, 4.1 | 8, 9 |
| Increased skills and knowledge | 3 | 9 |
| Increased motivation | 3.2, 3.4 | 8, 9 |
| Increased self-efficacy | 3 | 8 |
| Increased autonomy | 3 | 8 |
| Logic model intermediate outcomes | ||
| Improved social support | 3.3 | 7, 9 |
| Healthy habit formation | 3.1, 3.4, 4 | 8 |
| Improved self-efficacy | 3 | 8 |
| Improved self-image or self-esteem | 3, 4 | 8 |
Exploring the qualitative findings in this way demonstrated that two of the four intermediate outcomes proposed by the logic model were strongly supported by the qualitative findings: (1) improved social support and (2) healthy habit formation. Insufficient data were provided by participant/helper interviews to support the other two intermediate outcomes, namely improved self-efficacy and improved self-image or self-esteem, as integral processes in HelpMeDoIt!. Although self-esteem and self-efficacy are supported by the evidence base,22 they may not be intermediate outcomes in the HelpMeDoIt! intervention. They should, however, be included in the programme theory for further study in a future trial.
Eight out of the 11 proposed mediators of change above were found from the data to be key processes in the HelpMeDoIt! intervention. These were increased social support, increased engagement with helpers via the app, increased interaction with helpers not via the app, reflecting on and setting ongoing goals, increased action-planning, increased self-monitoring, increased skills and knowledge, and increased motivation. Four of these eight mediators emerged from the data as the strongest elements of the HelpMeDoIt! intervention: (1) increased motivation, (2) increased social support, (3) increased goal-setting and (4) self-monitoring.
Increased motivation
Participants consistently focused on the importance of motivation to their ability to successfully make lifestyle changes. A number of elements contributed to improved motivation. The template goals in the app were well liked and helped participants to set modest, achievable goals rather than more ambitious goals that had a higher risk of failure. Participants reported that small successes would increase their motivation to continue with their healthy lifestyle changes. The website was also reported to be helpful in increasing knowledge and motivation. We found that motivation, along with life contextual factors, was a precursor to engaging with and using the HelpMeDoIt! intervention. Motivation may have been generated by a participant’s life context, such as a change in their health status that made them feel that they had to get serious about improving their health and losing weight. Some participants had tried many different weight loss interventions and had maintained enough motivation to sign up for HelpMeDoIt! in the hope that this programme would work for them. Furthermore, good social support was a contributor to maintenance of motivation.
Increased social support
Qualitative interview data showed that most of those interviewed engaged with the idea of a helper even if they did not engage with the app. They said that the intervention prompted them to ask for help from people in their social networks and let these people help support them in their efforts to change their behaviour. The interviews reflected an increase and improvement in social support (emotional, instrumental and informational) for many participants.
Increased goal-setting and self-monitoring
Goal-setting and self-monitoring were promoted through the app and website. The website provided participants with guidance on setting and maintaining SMART goals related to healthy lifestyle behaviours. Some participants reported liking the template goals in the app; however, use data show that participants overwhelmingly used bespoke goals. Most participants interviewed reported that goal-setting and monitoring were important parts of changing their behaviour (especially among those participants who lost weight). In the process of monitoring goals, participants increased action-planning, increased problem-solving when goals were not working well, increased tracking of their goals and improved their healthy habit formation (e.g. by establishing daily walks or reducing sugary treats).
The revised HelpMeDoIt! programme theory and logic model
The revised version 3.0 programme theory and logic model incorporating the above findings is presented in Figure 10. Version 3.0 more clearly identifies motivation, social support, goal-setting and self-monitoring as the key mediators of change within the HelpMeDoIt! programme theory. These have been grouped together and given hierarchical priority in the logic model (illustrated by the use of solid lines and deeper colour). Improved social support and healthy habit formation were identified as the key intermediate outcomes, and have also been given hierarchical priority in the logic model (using solid lines and deeper colour). Other mediators and intermediate outcomes, which were not strongly supported by the data and require further exploration (e.g. increased self-efficacy), are illustrated by the use of dashed lines and lighter colour. Additional considerations were added to appropriate categories within the pre-conditions and contextual factors section, for example complementary use of other lifestyle apps under ‘technology’. Any future evaluation of HelpMeDoIt! will use the version 3.0 programme theory and logic model.
FIGURE 10 Version 3.0 of the HelpMeDoIt! programme theory and logic model. (Figure not reproduced here; see the original article.)
Part 2: evaluation design-related findings
Recruitment and retention
Recruitment
Overall, our multimethod strategy of recruitment was successful for recruiting a diverse sample in terms of age, gender, and socioeconomic and ethnic background. Figure 11 provides a summary of our recruitment rates by their source. The most effective methods of recruitment for HelpMeDoIt! were (1) the SPCRN, (2) newspaper advertising and (3) online Gumtree adverts. These different methods had different associated costs, which are detailed below, but all involved the research team screening people who responded. The strengths and limitations of each recruitment method are considered below.
FIGURE 11 Recruitment sources of final sample (n = 109). SPCRN, Scottish Primary Care Research Network. (Figure not reproduced here; see the original article.)
- Scottish Primary Care Research Network: the role of this network is to support the recruitment of primary care patients to research studies. One SPCRN representative liaises with GP surgeries, searches their patient database in relation to study inclusion/exclusion criteria, and posts information to eligible participants. The positive aspect of this method was the large number of eligible individuals who were able to be informed of our study (n = 1000 from five surgeries). We anticipated a 10% response rate of expressions of interest (n = 100) and achieved an 8.8% response (n = 88); of these respondents, 55% (n = 48) proceeded to sign up to the study. The only limitation of this method was that some people misinterpreted that their GP was recommending a ‘service’ to them, rather than sharing information about our study. This resulted in the study team being contacted by many people who did not own a smartphone or have internet access, but did not want to be seen as ‘letting their GP down’. Addressing these issues was time-consuming for the study team. Emphasising the purpose of the information pack is an important point for us to consider if using recruitment from primary care in a future study. This method of recruitment required minimal time input from the research team and cost £400 (approximately £8 per participant recruited).
- Newspaper advertising: a medium-sized newspaper advert (approximately 8 cm × 10 cm) was published in the Metro newspaper. The advert was printed 2 days per week for 2 weeks, on two separate occasions, and targeted mainly working-age commuters. Positive aspects of this method included a steady flow of expressions of interest (n = 54 in total), with approximately one in four proceeding to participate in the study (n = 14, 26% of expressions of interest). Challenges of this method included the time required for the trial manager to speak with and screen potential participants, many of whom were ineligible because they had a BMI of < 30 kg/m2. If this method were to be used again, we could provide an online eligibility screening form. Again, this method of recruitment required minimal time input from the research team and the total cost was £1000 (approximately £71 per participant recruited).
- Online Gumtree adverts: this platform provided opportunities to upload free, ongoing adverts for the duration of the recruitment phase. The benefit of this method included that it resulted in regular expressions of interest (n = 54), mainly from people searching for ‘weight loss’ in their local area. Limitations included a fast rate of attrition of interested individuals once they were asked for further details (n = 35 non-responders). This was time intensive for the trial manager but provided a reasonable uptake of participants to the study (n = 19, 35% of the expressions of interest). An administrator to filter the initial expressions of interest would be a more effective use of resources. This method of recruitment was free other than some time from the research team to place adverts.
Other helpful methods of recruitment included promotion of the study via a ‘whole campus’ e-mail to staff and students of the University of Glasgow, and recruitment by word of mouth. Other methods of recruitment, such as using Facebook, Twitter and online forums, were less successful but were free and not resource or time intensive to deliver. We recruited one participant via the NHS Weight Management Service, which involved presenting our study methods at a staff meeting, harnessing staff’s support and leaving promotion materials for them to distribute to suitable individuals. We also recruited one participant after we were present at a local health network event. One participant reported joining the study after seeing a leaflet or poster; these were expensive (in relation to printing costs) and time intensive (in relation to sourcing appropriate leaflet/poster locations) and would not be useful for a future study.
Our multimethod recruitment strategy resulted in 253 expressions of interest. Of these, 62 (25%) were non-responders when the study team requested further details, 49 (19%) did not meet the study eligibility criteria, 18 (7%) withdrew their interest after reading the participant information sheet, 8 (3%) noted their expression of interest after the recruitment phase ended, and 7 (3%) did not attend their scheduled baseline appointment and were noted as having withdrawn their interest. This resulted in a total sample of 109 participants.
Retention
Quantitative data on trial retention were reported in Chapter 4. In brief, a total of 84 participants were retained at 12-month follow-up (77%). Of these 84 participants, 14 (17%) provided their 12-month follow-up data in the form of a ‘minimum data set’. The split between intervention (11 of 52 participants, 21%) and control (3 of 32 participants, 9%) broadly reflected the 2 : 1 recruitment ratio. The minimum data set was used when participants were reluctant to arrange face-to-face appointments for follow-up and were at high risk of withdrawing from the study. The minimum data set calculated BMI using self-reported weight.
Attrition was lower in the control group (n = 4, 11.1%) than in the intervention group (n = 21, 28.8%). This is perhaps an effect of intervention participants being frustrated with the inconsistent quality of the app or not engaging with the intervention and thus disengaging from the study. Six participants withdrew early in the study, with two participants reporting illness/injury and four participants giving no reason. At 12 months a further 19 participants were lost to follow-up. Reasons included personal or family illness (n = 3), could not use the app (n = 1), did not lose weight (n = 1), relocated (n = 1), and non-responders (n = 8). Chapter 4 has presented the characteristics of participants lost to follow-up.
Qualitative interviews explored the feasibility of outcomes measures with participants, and whether or not the choice of measures influenced their engagement with the study. All participants interviewed reported that the measures and questionnaires used were not burdensome. Qualitative feedback from four participants at 12-month follow-up also suggested that having measurements taken at baseline and follow-up was a motivating factor that kept some participants retained in the study. Although this insight was provided by intervention participants, the impact of measurement might account for the fact that the control group lost weight in the follow-up period.
When the decision was made to arrange baseline and follow-up appointments via the trial administrator rather than the field workers, a noticeable improvement on retention as well as the timeline for recruitment was observed; this approach should be used in any future trial.
Contamination and the control group
As this was a feasibility study, it was important to explore how the HelpMeDoIt! intervention would be used in everyday life. We gathered data on any other lifestyle-related activities that participants were engaged in. Of the 109 participants, 29 (26.9%) reported using another service either at the time or within the 3 months prior to the study start date (23.6% intervention vs. 33.3% control). These included Scottish Slimmers™ (Aberdeen, UK), Weight Watchers® (Maidenhead, UK) and the NHS Weight Management Service (NHS Greater Glasgow and Clyde). Some of these activities provide similar elements to the HelpMeDoIt! intervention, such as advice on how to set achievable goals for healthy eating, and tools for self-monitoring. However, the HelpMeDoIt! intervention was designed as a complementary intervention that could enhance the activities that individuals were already engaged in, primarily by harnessing greater social support. At both baseline and follow-up, three participants had taken weight loss medication in the previous 12 months compared with five in the intervention group, and a similar proportion of control and intervention participants paid for lifestyle services (e.g. gym memberships) over the duration of the study [45.0% control (n = 11); 48.8% intervention (n = 40)]. As the numbers of people accessing these services were similar between the intervention and control groups, the risk of significant contamination bias is low. Furthermore, no participants in the control reported that they had seen or had access to the intervention content. No participant in the control group acted as a helper for a participant in the intervention group, and no nominated helper was enrolled as a participant. At the end of the study, five (14%) control participants took up the offer to use the app/website.
General insights from the study team
The study team reflected on the research processes as part of the study aim to examine the feasibility of delivering a larger study. Insights were gathered from members of the study team, support staff and field workers. Several issues were identified that need consideration for any future trial. These include the following.
Insight related to recruitment and retention
- Using a dedicated member of staff to initially screen expressions of interest and arrange baseline appointments may be a more effective strategy than communication being via the trial manager or field workers.
- Having a dedicated member of staff (our trial administrator) contacting participants for recruitment appointments, follow-up appointments and accelerometer returns. This would improve speed of recruitment and retention and minimise loss of equipment.
- The target sample size would probably have been achieved had media adverts not been published during a school holiday period, and if we had recruited via primary care several weeks earlier.
- Explore methods of engaging individuals before they provide informed consent, for example including screenshots of the app/website on the information sheet/website. This may be more engaging than simply text on an information sheet.
Insight related to outcome measures
- At baseline we collected dietary information from participants using the 24-hour multiple pass recall measure. This involved telephoning participants four times over a 10-day period and asking them to report their food intake over the previous 24 hours. This was a challenging measure to use and gathered poor-quality data. Despite obtaining a successful questionnaire completion rate of 74% (n = 81) (reflecting participant insight that questionnaires were not a burden), only 31% (n = 34) of our total sample were completed as intended, and 43% (n = 47) were completed ‘poorly’ (with the remaining 26% not completed). This measure also required significant time input (and cost) of field workers repeatedly making unsuccessful telephone calls to participants; significant time input (and cost) of research support to input poor-quality data in the dietary software system; and the disappointing quality of the dietary input software, which resulted in a lot of time spent searching the internet for comparable food macronutrients and inputting them manually. Our experience of using this measure at baseline demonstrated that it was not a feasible measure for this study. Following approval from the National Institute for Health Research (NIHR), this measure was removed from follow-up data collection. In a future study we would continue to use the other dietary measure, the DINE questionnaire, for which we received a 100% response rate.
- Although accelerometer return by participants was high (only 15 monitors were not returned over the duration of the study), valid accelerometry data were available for fewer than half the participants. Additional incentives to use and return the accelerometers could increase response rates (e.g. postcard reminder including information on the cost of replacing the activity monitor).128,129 We could also explore potential for the app to collect physical activity data.
Recommendations to improve the intervention
Finally, based on the findings of the process evaluation, a number of changes should be made to the app and website in anticipation of a future trial. These are summarised below.
The HelpMeDoIt! app
- Improve the self-monitoring aspect of the app by displaying a list of all recorded weights.
- On the helper app, display weight change instead of actual weight.
- Provide clearer guidance on how to use the app, for example to set goals.
- Devise strategies to encourage helper engagement via and/or outside the app.
- Amend the scoring so that it is easier for participants and helpers to earn reward trophies.
- Review push and e-mail notifications to ensure that they are sent with the optimal content and frequency.
- Ensure that the ‘reminder feature’ for goals updates is enabled for all participants and helpers.
- Encourage helpers’ engagement with participant progress charts by sending regular updates (e.g. via push notification, e-mail or text).
- Include a link enabling participants and helpers to manage their notification settings.
- Ensure that the app has a testing phase once it has been launched via the app stores.
- After each software update, ensure that the app is tested in-house by the software company to identify any new software bugs caused by the update.
- Explore the long-term potential of combining with other lifestyle apps.
- Overall, consider whether or not the app is needed to facilitate the unique element of social support, as, based on the feasibility study results, other methods of delivering social support are also likely to be beneficial.
The HelpMeDoIt! website
- Consider whether or not a separate website is needed and explore whether or not a future version of the intervention could provide accessible information via the app.
- Consider revising the website so that participants have the ability to add helpers, goals and weight and view summary graphs and statistics on a bigger screen.
- Encourage visits to the website by improved signposting to the website, clarifying the purpose of the website, and sending reminders.
- Include guidance on the website on how to earn rewards via the app.