Development of recruitment and retention strategy

Some material in this chapter has been reproduced from Schumacher et al.18 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text includes minor additions and formatting changes to the original text.

Methods

Aim

To develop an evidence-based recruitment and retention strategy to guide a future RCT of Wrapped.

Design

Iterative development across five stages: (1) rapid review, (2) development of provisional strategies, (3) focus groups with young people, (4) confirmation of strategies for a fRCT, (5) delivery of strategies in the fRCT and (6) review and finalisation of strategies for a future RCT.

Public and patient involvement

Six users of an existing STI self-sampling service (Preventx) aged 20–24 years (two female, four male) supported development of the strategy, inputting into Stages 2 to 6. The group had initially supported development of the Wrapped fRCT funding application and wished to continue their involvement post award.

Ethical approval

Ethical approval granted on 4 February 2020 by the Health, Science, Engineering and Technology Ethics Committee with Delegated Authority (ECDA) at the University of Hertfordshire (reference LMS/SF/UH/04061).

Procedure

Stage 1: rapid review

The rapid review aimed to identify strategies used by existing digital RCTs that achieved relatively high recruitment and/or retention rates. Pubmed, Scopus, PsycArticles, CINAHL and Cochrane databases were searched in May 2020. Keywords were combined to identify studies that assessed a digital intervention (see Report Supplementary Material 1 for the full search strategy). Articles were considered for eligibility if the study was a RCT, fRCT or a review of digital interventions, and reported recruitment and/or retention rates along with details on strategies that were used to support recruitment and retention. Protocol papers and conference proceedings were not eligible for inclusion.

Mendeley was used to eliminate duplicates and screen titles and abstracts for eligibility. Following initial screening, full-text articles were obtained and screened for eligibility. Data extraction for included articles was pre-defined as follows: study design, population, recruitment and retention rates, details of recruitment strategies, details of retention strategies and any qualitative feedback.

Data extracted from the included studies were organised into a single Excel spreadsheet, with one row per study. Columns contained information on recruitment outcome (rate or proportion of sampling pool recruited), retention outcome (percentage of participants completing the final data collection time point) and strategies used for recruitment and retention. Strategies were organised into categories (one column per category) as follows: value propositions (statements which communicate what individuals can gain from participation), engaging participants with the research (strategies used to develop and maintain participant engagement), timing and nature of survey invitations, value and implementation of financial incentives, reminders to complete activities, and actions to minimise non-response.

Stage 2: development of provisional strategies

The public and patient involvement (PPI) group first met online with the lead researcher (via Microsoft Teams) for a familiarisation session. This included discussion of the aims and design of this study and the role of the PPI group. The importance of maximising recruitment and retention in the context of trials was emphasised. A general discussion about recruitment and retention in research was initiated and members asked to begin thinking about what strategies might work, with consideration of the target population.

The intention was to follow-up this meeting with two face-to-face workshops to develop a set of provisional strategies. Due to the subsequent outbreak of COVID-19, it was however necessary to seek an alternative means of collaboration. To overcome challenges experienced in synchronising diaries and to maximise inclusion, the group agreed to work asynchronously on a shared workspace called Padlet (Padlet, San Francisco, CA, 2012), a cloud-based application which enables users to create, share and organise material together.

Two Padlets were shared in turn, the second building on the first. Five PPI members took part, with each Padlet made available for 1 week. Material was presented in columns, with members asked to work across the columns from left to right to complete each activity. The first column provided instructions on how to complete the activity. The remaining columns were themed around the categories of recruitment and retention strategies as identified in the rapid review. For the first Padlet, members were asked to provide their own suggestions for strategies against each category, before being told in a subsequent column(s) what strategies had been identified from the review. For these strategies, the group was encouraged to use the ‘upvote’ and ‘downvote’ facility to indicate those that they did and did not like and to post comments. For the value propositions category, PPI members were asked to upvote/downvote individual value propositions identified from the rapid review, make suggestions for additional value propositions, and to draft provisional adverts using a combination of these. Following completion of the first Padlet, the research team collated and refined the ideas generated across all categories and used these to create a second Padlet. This included drafting some further examples for adverts based on the feedback. This was then presented back to the group for a further round of voting. For the value propositions category, members were asked to vote on the new set of provisional adverts. For all remaining categories, members were encouraged to ‘upvote’ their preferred strategies. Those with the lowest ratings, or those that contained several negative comments, were discarded at this point and did not advance to stage 3.

Stage 3: focus groups with young people

Stage 3 consisted of data collection via focus groups. Participant groups consisted of (1) undergraduate psychology students, (2) young people in or leaving the care system and (3) members of a youth theatre organisation. Each recruiting organisation was asked to invite between six and eight young people aged 16–24 years to participate. All participants provided informed consent. Parental consent was additionally obtained for individuals under the age of 18 years.

All focus groups were conducted online via Microsoft Teams and were audio and video recorded. Focus groups lasted between 44 and 103 minutes (average 69 minutes). Two of the three groups were co-facilitated by a PPI member who was trained and supported by the team to perform this role. All recordings were transcribed verbatim, after which the original audio and visual recordings were destroyed. See Appendix 1, Table 30 for focus group schedule.

Padlet was used to support the focus group discussions. A Padlet board was shared with participants which displayed the categories retained at the end of Stage 2, with some additional ideas and prompts added by the research team where further input was required. Discussions progressed through each category with participants encouraged to voice their opinions concerning the relative merits of each strategy. The potential for unintended negative effects of the various strategies was considered. Opinions on six provisional adverts developed during Stage 2 were also sought. After each category had been discussed, participants were asked to upvote strategies/adverts that they thought would be the most successful in maximising recruitment/retention and to provide any written comments if they wished to.

Data from the transcripts and the Padlet were subjected to content analysis using a directed, deductive approach.19 Analysis was guided by the aim of determining which recruitment and retention strategies were likely to be the most and least successful. Analysis focused on each category in turn: value propositions (adverts), engaging participants with the research, timing and nature of survey invitations, value and implementation of financial incentives, reminders to complete the activities, and actions to minimise non-response. An Excel spreadsheet was prepared in advance for each category. Spreadsheets contained tabs for each potential strategy/advert. Within each tab, columns were added to organise participant views in terms of whether the strategy/advert was considered likely to have an overall positive, negative or neutral impact on recruitment and retention with relevant quotes pasted in the rows of the corresponding columns. The total number of upvotes for each provisional strategy/advert was extracted from the Padlet and noted alongside the data.

Stage 4: confirmation of strategies for the fRCT

Prior to the final PPI workshop, the votes and analysis from Stage 3 were examined by the research team to identify any strategies with mutual agreement from the PPI group and the focus groups; those with agreement were considered definitive and advanced directly to inclusion within the strategy. All other strategies were subjected to a final round of comment and voting by the PPI group via Padlet. In addition to presenting adverts to PPI members that had progressed from Stage 3, the research team decided to present the component parts of these to members, in recognition that some parts of the adverts may be liked but not others. These component parts, which typically included a call to action (e.g. ‘we need your help!’), a description of the study, and mention of the financial (or other) incentive, were also voted on. As before, the PPI group were given a week to complete the Padlet in their own time. A total of three PPI members completed this activity.

The research team met to tally the Padlet votes and took all comments into consideration. In instances where the PPI group did not reach consensus, the research team made the final decision. Adverts that were considered too similar were removed. Some adverts had to be revised to conform with the Health Research Authority (HRA) Ethics Guidance, specifically that financial incentives should not be mentioned as the first line of a recruitment message (https://s3.eu-west-2.amazonaws.com/www.hra.nhs.uk/media/documents/hra-guidance-payments-incentives-research.pdf). Two new adverts were created based on component parts receiving the highest votes. Provisional strategies that were not technically feasible were removed. Following this process, the recruitment and retention strategy was finalised.

Stage 5: delivery of strategies in the fRCT

The recruitment and retention strategy was implemented in delivery of the fRCT (see Chapter 3).

Stage 6: review and finalisation of strategies for future RCT

On conclusion of the fRCT, a final Padlet was created and shared with the PPI group. This was used to update the PPI group on changes that were made to the recruitment and retention strategy during the trial and to seek reflection on these and any further changes that could be made with a view to plan a definitive RCT. Once again, members were given 1 week to complete the Padlet in their own time. This was followed up with a final meeting held online via Microsoft Teams in which the Padlet topics were discussed to elicit further feedback. Three members completed the Padlet and attended this final meeting.

Results

Stage 1: rapid review

The database search yielded 226 articles, of which 15 duplicates were removed. The remaining 211 articles were screened by title and abstract and 134 excluded. This left 77 full-text articles which were further screened for eligibility. A total of 36 articles, describing 26 studies, met the inclusion criteria and were included in the review. See Report Supplementary Material 2 for PRISMA flow diagram and Report Supplementary Material 3 for a summary of included articles. The following sub-sections summarise the findings by strategy category. Note: While the strategies were examined within the context of each study’s reported recruitment and retention rates, no consistent pattern was observed, and therefore all strategies were considered to carry equal weight at this stage.

Value propositions

A total of seven studies referenced value propositions used to advertise the study. Value propositions used appealed to altruism, either as a petition to help others or to help with research. Value propositions also highlighted the benefits of taking part, such as financial incentives and improving personal health and well-being, and how taking part was easy, or that there was a scarcity of available space to take part. Included articles advised collaboration with experts and/or the target population to create value propositions and using professional design and/or logos to indicate that messages were coming from an expert and/or a trusted source. Table 31 in Appendix 1 provides a summary, with examples where available, of the different types of value propositions identified.

Engaging participants with the research

A total of seven studies cited use of the following as strategies to maintain participant engagement:

• Continuity across all participant materials (such as logos, colours, fonts, templates and formatting)
• Personalising materials (i.e. addressing participants by name)
• Using clear, simple and brief communications
• Inclusion of social desirability statements in communications to encourage completion of measures
• E-mails to the intervention group to remind them to use the intervention
• Directly thanking participants for joining the study and/or their ongoing participation
• Newsletters to update participants on the progress of the study

Timing and nature of survey invitations

A total of 17 studies described using multiple modes to invite participants to complete surveys including e-mail, short message service (SMS, also known as ‘text message’) text message and letters sent through the post. These were all sent at the point at which completion was required. Some studies however also used a prompt ahead of the survey delivery, alerting participants that they would soon receive the invite.

Value and implementation of incentives

A total of 15 studies referenced the use of financial incentives as a strategy to motivate and sustain participation. Financial incentives were typically distributed upon completion of an activity. On occasions, a small incentive was additionally offered upfront before the activity was completed. Three studies offered financial rewards that increased in value throughout the duration of the trial. Prize draws were utilised by seven studies. One study detailed offering higher amounts of compensation if participants completed the measures quickly, but less if a researcher needed to call and ask them for their data over the phone.

Reminders to complete research activities

A total of 18 studies detailed the use of reminders sent by e-mail, SMS text message, telephone call or a letter sent through the post. Of these studies, 15 specified the number of reminders sent (range 1–6), the remaining 3 did not specify the amount. Six studies detailed how reminders were sent at varying intervals, ranging from 3 to 7 days; the remaining 12 studies did not specify how frequently reminders were sent.

Actions to minimise non-response

Five studies described taking additional steps beyond reminders to collect as much data as possible if/when participants failed to complete surveys or other data-collection measures:

• Asking participants to complete only the primary outcome measures instead of the full survey
• Telephone calls to offer participants support to complete the measures online or to ask them to complete with the researcher over the phone
• Resending postal measures (such as STI test kits)

Stage 2: development of provisional strategies

For an image of the Padlets from Stage 2, see Report Supplementary Material 4 and 5. The following sub-sections provide a summary of the outcomes from Stage 2. For further details, including illustrative quotes from PPI members, see Appendix 1, Tables 32 and 33.

Value propositions/adverts

In workshop 1, the PPI group created their own adverts that would attract people to sign up to the study and take part for the entire 12-month trial; eight adverts were drafted. Members felt that value propositions that appealed to the values of altruism and prioritising sexual health should be used, but that those mentioning financial incentives would likely be most effective in recruiting to the trial.

In workshop 2, 12 potential adverts were presented to the PPI group: the 8 drafted by the members in workshop 1 and 4 drafted by the research team. The PPI group selected six adverts that they felt would be most successful in attracting participants to the study. See Appendix 1, Table 32 for the 12 adverts voted on in the second workshop and which ones progressed to Stage 3 for further feedback by focus group participants.

Engaging participants with the research

Public and patient involvement members felt that it was important to convey to participants that their input was valued by the research team and that this in turn would promote retention. Suggested strategies to achieve this included personalising all communications to participants and including social desirability statements. It was also felt that using continuous, uniform branding on research materials and communications, as well as using clear, concise language, would increase the likelihood that participants would attend to and act on research communications. PPI members also suggested using WhatsApp, in addition to SMS and e-mail, which they thought could be effective for keeping in touch with participants; this idea however was not progressed as the research team had concerns regarding data security. The PPI group held mixed opinions on the idea of project newsletters to provide updates on the research as a whole: some felt that this would help participants to feel valued and involved in the study, thereby increasing retention, whereas others thought that participants might find it annoying. The consensus however was that using newsletters would be acceptable, providing that they were succinct, interesting, and friendly in tone. The PPI group endorsed the idea of sending participants birthday greetings as a further strategy for conveying the value of participation but thought that Christmas greetings would likely be identified as spam.

Timing and nature of survey invitations

The PPI group recommended SMS be used for survey delivery, but also felt that combining SMS and e-mail could work well (to respond to different preferences for mode of delivery). It was considered acceptable to send a prompt 3 days before the delivery of a research activity as long as it was brief in nature.

Implementation of financial incentives

In workshop 1, the PPI group provided feedback on different strategies for implementing financial incentives. (Note: The group was informed that incentives had to be in the form of vouchers; Amazon vouchers were chosen by the research team due to their universality and ease of digital distribution.) Offering increasing amounts was considered acceptable and effective for promoting long-term retention, but offering ‘something for nothing’ was rejected: the group thought vouchers should only be distributed upon completion of an activity. Sample voucher schedules were drafted based on this feedback and presented to the PPI group in workshop 2. Members voted and agreed on a schedule offering increasing amounts. Two schedules offering increasing amounts were progressed to Stage 3 for discussion with the focus groups. A schedule in which offering something for nothing was also worked up, so as to provide an example to refer to when discussing this strategy in Stage 3.

Reminders to complete activities

Reminders were deemed acceptable and useful by the PPI group. They felt that these should be concise, friendly, and reference both the financial incentive and how each participant’s contribution makes a difference to the research. SMS was the preferred delivery method for reminders, but it was also considered acceptable to change to alternative modes of delivery (i.e. e-mail and telephone call) for non-responders. PPI members distinguished between reminders for surveys and test kits: additional reminders for test kits might be useful due to the extra effort required to complete them.

Actions to minimise non-response

Public and patient involvement members agreed that telephone calls to non-responsive participants were appropriate and potentially effective in increasing retention. While they felt that asking participants to complete the entire survey over the phone was unacceptable, asking a few brief items was considered acceptable. They advised that telephone calls should focus on checking in with the participants in terms of their well-being, given that unsolicited telephone calls which focused on non-response might make participants feel uncomfortable.

Stage 3: focus groups with young people

A total of 15 young people took part. See Table 1 for demographic characteristics of the sample.

For an image of one of the Padlets from Stage 3, see Report Supplementary Material 6. A summary of the findings is presented below. See Appendix 1, Tables 34 and 36, for further details including illustrative quotes.

Value propositions/adverts

Participants discussed the six potential adverts carried through from Stage 2, focusing on the key elements that they thought would make it successful. These included that people should be able to read and understand them quickly and easily; that any financial incentive was likely to be the most important driver of participation and should be prominent within recruitment adverts; that altruistic factors would likely also encourage participation; that adverts should convey that participation was valued; and that adverts should communicate what the expectation of participants was, striking the balance between informing people about the level of commitment required but also that participation was straightforward. For more details, see Table 34 in Appendix 1. Two focus group participants also suggested additional provisional adverts, emphasising financial incentives and altruism (see Appendix 1, Table 35). Based on the feedback from participants about which value propositions they thought should be emphasised, two further adverts were drafted by the research team to take forward into Stage 4.

Engaging participants with the research

The tone of messaging was considered as important as the information being conveyed. It was suggested that participants should be addressed by name in any communications and that the message should be signed off with the names of the researchers, rather than just providing the name of the study. All messages should have instantly recognisable branding, not look scripted, and contain playfully worded subject lines that encourage the message to be acted on. Any instructions should be clear and use plain language. Social desirability statements to complete the study were considered useful so long as they were sincerely worded and felt encouraging, not demoralising.

In terms of maintaining engagement over the 12-month trial, focus group participants felt that the study should convey to participants the impact that their involvement was having and that their participation was valued. They agreed that newsletters were one way of doing this. Although some focus group participants thought that sending a happy birthday message to participants might make them feel valued, others thought it was over-familiar.

Timing and nature of survey invitations

E-mail was the preferred method for sending survey invites, with SMS text messages being used as prompts (i.e. to notify of forthcoming research activities). Overall, it was felt that the number of separate communications should be minimised to prevent participants feeling bombarded. It was felt that invites and prompts should contain two elements: a call to action and mention of the financial incentive for that specific activity. Participants also suggested that invites include the estimated time required to complete the survey. Sending prompts by SMS text message prior to test kit arrival was thought to be useful, especially if it served to assure participants that it would arrive in discrete packaging. While two of the three focus groups had a consensus of opinion that prompts for surveys could be useful, one of the groups questioned the utility of this: the concern being that without a direct call to action it could lead to participants ignoring messages, resulting in attrition.

Value and implementation of financial incentives

One of the most important elements for maintaining engagement throughout the trial was thought to be financial incentives; offering increasing amounts over the course of the trial was preferred, although agreement on a particular voucher distribution schedule across the activities was not achieved. The two schedules offering increasing amounts were progressed to Stage 4, whereas the schedule offering something for nothing was not.

Reminders to complete activities

Focus group participants were in favour of reminders sent via SMS, emphasising the need for reminders to be friendly and encouraging, and to mention the financial incentive of completing the research activity. Some participants felt that including a completion deadline would motivate trial participants to complete the activity.

Actions to minimise non-response

Focus group participants agreed that non-responsive participants should be sent an e-mail asking if they were okay, with the aim being to further highlight the benefits of completing, such as the financial incentive, rather than making them feel guilty. Participants also felt that if there is still no response, it would be acceptable to send a letter in the post to confirm that the individual’s e-mail and/or phone number was still valid, but some did not like the idea of phone calls to participants (too intrusive).

Stage 4: confirmation of strategies for the feasibility randomised controlled trial

The following sections outline the finalised strategies for the fRCT as developed by the PPI group in collaboration with the research team. See Appendix 1, Tables 37–39 for further details including the result of voting and illustrative participant quotes. For an image of the Padlet from Stage 4, see Report Supplementary Material 7.

Value propositions/adverts

The PPI group voted on: potential adverts created by both the focus groups and by the research team based on analysis from Stage 3 (see Appendix 1, Table 37 for adverts and voting outcomes). The four adverts with the highest number of votes were finalised for use. The PPI group also voted on the individual component parts of these adverts (see Appendix 1, Table 38 for component parts and voting outcomes), the outcomes of which were then used by the research team to create two new adverts for use. The list below presents the final set of selected adverts:

1. Help us make a positive change to young people’s sexual health: make a difference by joining our study and get paid for your time.
2. Take part in a study to improve sexual health and earn up to £65 in vouchers, find out how you can make a difference today.
3. Your views are important: join our study to help improve sexual health for young people while being paid!
4. Make a difference to improve sexual health for young people – join our study and earn up to £65 in vouchers for your time.
5. Want to make a difference to young people’s sexual health? Take part in our study and get paid for your time.*
6. We need your help! Take part in a study to improve young people’s sexual health and earn up to £65 in vouchers for your time.*

*Created from value proposition segments.

Engaging participants with the research

• All communications to be sent at 5:00 p.m. as participants more likely to be responsive at this time, but not prioritise any particular day of the week.
• Use e-mail as the primary mode of communication.
• All communication to be signed off using names of the research team but should not include any photos of the team as this could make participants feel uncomfortable.
• The tone of all communications should be balanced between being relaxed and fun but professional, and reference current events (e.g. holiday period).
• Avoid sending communications over Christmas holiday period as it could result in lower response rates.
• Send newsletters by e-mail as a standalone communication (i.e. do not combine with other communications such as reminders) and include an opt-out feature if possible.
• Do not send birthday greetings – potential to be annoying and perceived as overly familiar.

The following strategies had mutual agreement from Stages 2 and 3 and as such were not presented to the PPI group for further voting:

• All communications with participants should:
  • Have an explicit purpose
  • Address participants by name
  • Be brief and clear, easy to read and with no jargon
  • Contain social desirability statements that are encouraging not shaming
• Newsletters should be sent quarterly.

Timing and nature of survey invitations

• Use e-mail for survey invites.
• Use prompts for test kits only – use this as an opportunity to check if the postal address is accurate and reassure that kit will arrive in discrete packaging.
• Note: The PPI group divided on whether prompts should be used. Research team made a decision to include for test kits only.

The following strategy had mutual agreement from Stages 2 and 3 and was not voted on:

• Text messages should be used to send prompts.

Value and implementation of financial incentives

• Provide vouchers of increasing value over the trial period as follows:
  • Month (M) 0 (joining the study) £5
  • M3 kit £10
  • M3 survey £5
  • M6 survey £10
  • M12 kit £20
  • M12 survey £15
• Provide higher amount for test kits than surveys.

Reminders to complete activities

• Send a maximum of three reminders for test kits and surveys.
• Send first test kit reminder 3 days after expected arrival of the test kit at participant’s address and all remaining reminders 10 days apart. Note: The PPI group advised all reminders sent 10 days apart. The research team decided to send the first one after 3 days to encourage timely completion.
• Emphasise the financial incentives of completion.

The following strategy had mutual agreement from Stages 2 and 3 and was not voted on:

• Text messages should be used to send reminders.

Actions to minimise non-response

The following strategies had mutual agreement from Stages 2 and 3 and were not voted on:

• Non-responders should be e-mailed; the tone of the e-mails should be friendly and supportive, should clearly outline the research activity that is outstanding and how to complete it, and mention the financial incentives.
• Non-responders should not be called: this could feel confrontational. Instead, a letter sent via post to check if their contact details were correct would be acceptable.

Stage 5: delivery of strategies in the feasibility randomised controlled trial

A small number of changes were made to the recruitment and retention strategy during the fRCT, either to enable the potential merit of alternative strategies to be assessed or in response to unexpected challenges. These are outlined in Table 2.

Stage 6: review and finalisation of strategies for future randomised controlled trial

The PPI group reflected on all above changes (see Table 2) and felt that they were acceptable. With regards to the phone calls, members felt that provided they were handled discreetly it was appropriate to call a participant who had not completed an activity. The PPI group understood why it had not been possible to tailor the content of messages or to alter their timing; the group highlighted however that participants should not receive communications at unsociable hours. Changes to voucher amounts were also considered acceptable, with members reporting that higher amounts would likely result in higher recruitment, but that if £65 was ultimately effective there was no need to increase it to £100. The PPI group recommended budgeting accordingly for a future trial and keeping in mind that changes might need to be made to accommodate any increases in amount.

The findings of the fRCT were also discussed with the PPI group. The Recommendations for future research section of the discussion reflects the consensus opinion of the PPI group and the research team.