Trial design and methods

Trial design

Young SMILES is a two-arm, pragmatic, randomised controlled feasibility trial, with a 2 : 1 allocation ratio. Randomisation was considered the most appropriate response to the original NIHR HTA programme 14/29 call that specified having two arms, a control arm and an intervention arm, and that the study design followed the Medical Research Council (MRC) Complex Intervention Guidance. The MRC Complex Intervention Guidance states that ‘Experimental designs are preferred to observational designs in most circumstances’.106 Therefore, consenting families with CYP aged 6–16 years and a parent/carer with a SMI were randomised to receive either:

• Young SMILES – a manualised, 8-week intervention for CYP with concurrent 5-week parent sessions adapted from the Family SMILES intervention, which was developed and evaluated by the NSPCC. If eligible, families were given an initial baseline assessment prior to randomisation. Then 4- and 6-month follow-up appointments were completed with an optional feedback interview. Families referred to the NSPCC were also assessed for safeguarding concerns and suitability for the programme.
• Treatment as usual (TAU) – families continue to use the same services as before, while receiving the baseline, 4- and 6-month assessments.

Protocol changes

Throughout the feasibility process, a number of protocol changes were made. Applications to the Research Ethics Committee (REC)/Health Research Authority (HRA) were made and the following changes were approved for all changes outlined:

• Primary outcome point – there was a change from a 3-month to a 4-month primary outcome point, without altering the 6- and 12-month time points. This was to assist with family availability and referral rates and to ensure that the number of ‘useful data’ was maximised.
• Data collection – a demographic questionnaire for referrers was subsequently produced to capture the variety of professional backgrounds involved in the referral process.
• Randomisation – the ratio was altered from 1 : 1 to 2 : 1 randomisation allocation procedures (two Young SMILES intervention to one control) to expedite the formation of intervention groups. This amendment took place 8 months into recruitment, which, after numerous recruitment difficulties, expedited the start of the intervention in the two sites where most referrals were received: Newcastle (32 families referred) and Warrington (16 families referred).
• Randomisation method – there was a change from using the Sealed Envelope system (www.sealedenvelope.com) to online randomisation software [www.randomisation.com (accessed 1 March 2020)], as the former system would not meet our requirements in relation to sample size and the ability to stratify by age and site without incurring costs.
• Inclusion of siblings – it was initially agreed that inclusion of siblings within the same group may have a negative effect on the group dynamics. After further consideration, this was changed to allow siblings within the same group (if in the same age band), which would also expedite the formation of groups.
• Travel – some families preferred the young person to travel to the group alone. For safeguarding, a travel consent form was developed for parents/carers to consent to the child/adolescent travelling to the intervention venue alone.
• Participant communication – following participant feedback, we simplified the participant information sheets to enable a better understanding of the aims and objectives of the trial.

Participant eligibility

Families were considered eligible if they met the following criteria:

1. Children and adolescents aged 6–16 years whose parent had serious mental illness.
2. Parents/carers with SMI and their partners, the latter of whom may or may not have had any mental health problems. SMI is defined as a severe psychiatric disorder that requires intervention, hospitalisation or ongoing treatment; parents with a diagnosis of schizophrenia or related disorder, schizoaffective disorder, bipolar disorder, severe recurrent unipolar depression, psychotic depression, severe anxiety, obsessive–compulsive disorder and post-traumatic stress disorder were sought. Parents with a current diagnosis of a severe psychiatric disorder with a minimum requirement of scheduled follow-up with secondary care services were eligible for study inclusion. The focus of our project was the children and adolescents and their outcomes, rather than the parents. Therefore, we did not intend to carry out full clinical interviews with the parents or to report diagnostic codes. We accepted the primary and secondary diagnoses reported by a key health professional, such as the GP, care co-ordinator and key worker, as most of these parents were likely to receive secondary care or be monitored in primary care. This could be gleaned during professional referral into the study or, in the case of a self-referral by the parent, we obtained the diagnosis by contacting the parent’s appropriate care co-ordinator (e.g. GP or community psychiatric nurse) after gaining the parent’s permission to do so.
3. Children and adolescents had to have at least 10 hours of contact per week with the parent/carer with SMI. (The child/adolescent did not have to live with a mentally ill parent necessarily.)
4. The parents/carers/guardians understood the purpose and remit of the intervention for themselves and their child/adolescent and consented to their attendance and completion of outcome measures and interviews.
5. Children/adolescents had to have some awareness of the parent’s mental illness, as confirmed by the parent and/or the appropriate care co-ordinator. If they had no awareness of the parent’s illness, we discussed how the parent and care co-ordinator could prepare the children/adolescents before they started group work.

Families were excluded if they exhibited the following criteria:

1. Children/adolescents of parents diagnosed with common mental health problems (e.g. mild–moderate depression) or with primary substance misuse, rather than with a primary diagnosis of SMI as defined in inclusion criterion 2, above.
2. The children/adolescents had significant cognitive impairment or a learning disability or major mental illness or behavioural problems (as verified by their GP or other health professionals involved in the family’s care) that made it impossible or unsafe for them to participate in group work.
3. The parent was extremely mentally ill at the time of eligibility assessment, which made it difficult or unsafe for them to participate in group or individual work. (It was acknowledged that these children/adolescents might be those especially in need of support and therefore this was judged on a case-by-case basis by experienced practitioners.)
4. The children/adolescents had already participated in Family SMILES (which is not applicable in the north-east of England, where Family SMILES is not available).

Trial sites

The study was conducted at the following sites:

• North East – Northumberland Tyne and Wear (NTW) Trust, conducted by the local Family Therapy Service and the Barnardo’s Young Carer’s team for Newcastle upon Tyne.
• North West – NSPCC Warrington, delivered by NSPCC practitioners at the Peace Centre, Warrington. The facilitators came mainly from a social work background and had significant experience of working with vulnerable CYP.

A third locality in NSPCC Coventry had been identified to provide Young SMILES groups; however, there were recruitment problems and so no groups were run in this area (see Appendix 1, Table 31).

Recruitment and consent

Families were identified through practitioners working in CYP services, CAMHS, Adult Mental Health services and education services. Referral sources were broadened by close co-operation with NHS practitioners and services. The NIHR’s Clinical Research Network was used to assist with recruitment and identification of suitable families via GP registers, community mental health and inpatient teams, and rehabilitation units. Posters and flyers were distributed to relevant NHS and third-sector organisations and schools. In addition, practitioners working in the community were informed of the study to assist with direct opportunistic referrals.

Two methods of identifying families were deployed:

1. Recruitment gatekeepers (e.g. community nurses, care co-ordinators and young carer’s workers) identified people from their caseload who may be eligible for the trial.
2. A member of the trial team acquired permission from the gatekeeper to screen their caseload for eligible families first, and provided a list of families to the gatekeeper.

In each case, the gatekeeper gave the potential family a study information pack (an invitation letter, information sheet and consent-to-contact form). Separate packs containing the same information were provided to CYP and parents. If a family was interested in participating, they returned the consent-to-contact form, detailing their preferred method(s) of contact. Alternatively, the family could provide verbal consent to be contacted to the gatekeeper, who would then inform the study team. Here, a verbal consent-to-contact form was completed by the gatekeeper to confirm that they had received consent.

All families that were on the caseload of the services involved in the delivery of the trial were considered to be potential participants. In addition, caseloads of services within NTW Trust were also screened for participants with the permission of the professional holding the caseload.

After gaining consent to contact, the research team contacted the family, determined eligibility and received confirmation of desire to participate, which was usually received from the mentally ill parent, but could also be from the healthy parent. If the latter, the only additional factor that had to be ascertained was that the child had at least 10 hours per week of contact with the mentally ill parent. This number was considered to be an appropriate and readily measurable amount of time to ensure that the ill parent could be fully informed about what was happening and be able to support the child.

If eligible, families were informed that they would be contacted at a later stage to arrange to complete a face-to-face baseline interview. Dates for baseline interviews were contingent on the research team receiving enough referrals from the same age group at one site (based on the minimum feasible group size of four children). Families were telephoned fortnightly during the waiting period by the research team. They were informed of progress in organising groups and given an opportunity to ask any additional questions. A one-off home visit to discuss the study was also offered.

When multiple children were recruited from the same family, for the main analysis we identified an index child and included data from only that child in the analysis. We asked the parents to nominate the index child, which was always determined by which child experienced the most difficulties in responding to parental mental illness.

At baseline, the family were given the opportunity to ask any further questions and written consent was taken. The CYP were asked to give assent to participate. Consent from their parents/carers was also obtained in support of their child’s participation. Separate consent was obtained for parents/carers. It was clarified to both that they could withdraw at any point without detriment to their care. If a parent was to withdraw, their child/children could still participate with their consent. If the parent had more than one child, all children in the family were eligible to participate but were not required to do so.

After obtaining consent, family members were asked to complete the baseline measures. All measures are standardised and designed as self-complete, with the researcher present to offer assistance. Outcome measure booklets were age-dependent. Families received a total of £50 of shopping vouchers over the data collection time points for their participation in the study (£20 at baseline and £15 for each of the 4- and 6-month follow-ups). On every visit, trial researchers identified if any adverse events had occurred and reported these as per the agreed procedures. Outcome measure visits took place at a time/location convenient for the participant, which was invariably at the family’s home.

Intervention design

Comparator: treatment as usual

The control group received TAU. This was defined as access to any services or resources to which CYP and their families would usually be referred or have access. Participation in the trial did not preclude access to these services.

At the NSPCC site, there were no other services that would be a natural alternative to Young SMILES. The two other services that were on offer at the same time as the Young SMILES study were a therapeutic service for children who have been sexually abused and an assessment and a treatment programme for children who display harmful sexual behaviour.

At the NHS site, the type of TAU offered to the children depended on how they were recruited. Some children received usual family therapy support (in the form of home visits by a systemic practitioner), some received assessment and advice relevant to their role as ‘young carers’ (by practitioners in Barnardo’s) and others did not receive any specific intervention as they were children of parents who were looked after by a care co-ordinator in a community mental health team (CMHT) (these teams focus on the parent with SMI with no routine provision for their children).

Intervention: Young SMILES

Young SMILES is an 8-week group programme (four to six CYP per group) delivered by two trained practitioners. Age groups were 6–11 years or 12–16 years. At week 4, five parallel sessions were offered to the parent/carer. CYP and parent sessions lasted 2 hours each, including time for a short break and refreshments during and after the group session. Where possible, the sessions were held in non-stigmatising venues, such as a community-based location. Hospital clinics or schools were, therefore, avoided.

In Newcastle, groups were delivered in an NHS organisation, utilising a co-delivery model where one facilitator was from the NHS (Family Therapy Service) and the other was from a third-sector organisation (Barnardo’s). Third-sector organisation facilitators had a mix of professional backgrounds, which included social work, children’s services and occupational therapy.

In Warrington, the intervention was delivered by NSPCC practitioners at the Peace Centre, which is an accessible, award-winning venue embedded within the local community. The facilitators came mainly from a social work background and had significant experience of working with vulnerable CYP.

The training manual for facilitators provided an overview of the feasibility trial and development of Young SMILES. It incorporated details about service and intervention delivery. It provided a manual to support trainers to ‘teach’ practitioners aspects of the model and how it should be delivered, as well as an ongoing guide/reference for practitioners who delivered the service.

Children and young people’s group sessions

Every session had specific aims and objectives, which for both age groups remained consistent throughout (Table 4). However, activities and communication style varied depending on the abilities of the CYP. Adapting to the learning styles of the CYP in the group was also an important role of the facilitators. The themes traversing all sessions were mental health literacy, communication and problem-solving skills. The outcome of each session was facilitated through the creation of an ‘imaginary family’ for younger children (made from cardboard cut-outs/cartoon characters created electronically and printed) and a ‘graffiti wall family’ for older children. A normal session included a ‘getting to know you’ (first session)/’welcome back’ and a ‘check-in’ on previous week, ‘ice-breaker’ activity, activities based on the session aims and snack time. A ‘weekly challenge’ at the end of each session was included to orientate and tether the CYP to the next session in order to optimise engagement.

Parent/carer group sessions

The focus of the parent/carer sessions was determined during the CYP session in the previous week (Table 5). This was to retain the child-centred basis of the intervention. CYP and parents came together in the final ‘Moving on together’ session to review progress and for the last few activities that focus on hopes and fears, achievements and moving forward. Activities for each session were standardised, but not prescriptive. As long as they met the overall objectives of the sessions and were consistent with the ethos of the Young SMILES programme the facilitator was afforded the freedom of action to change their approach to better suit the different needs and learning styles within the group.

Practitioner training

The Development and Impact Manager from NSPCC Children’s Services held a 2-day Young SMILES intervention training event, which was held in Newcastle (1 and 2 March 2017) for NTW NHS Foundation Trust and Barnardo’s Newcastle practitioners, and in Warrington (20 and 21 March 2017) for NSPCC Warrington and NSPCC Coventry practitioners. The training provided practitioners with the following: the study context, the feasibility study, intervention format and guidance, delivery location context and referral pathways, the incorporation of the imaginary family and problem-solving.

Four half-day sessions were also arranged to work through the intervention activities (11, 12 and 18 July 2017 and 5 September 2017), ensuring that practitioners felt confident supporting CYP and their parents/carers. Practitioners also identified the need to gain more of an understanding about SMI in adults. This was covered in the training in Newcastle and additionally in two sessions for Warrington and Coventry NSPCC staff (5 and 6 July 2017).

Outcome assessment

A list of the primary and secondary outcomes and tools used can be found in Table 6. A description of each measure and how scores were calculated can be located in Appendix 3. Although measures were available for all ages, some CYP experienced difficulties completing certain measures. In this instance, researchers were responsive to the CYP’s needs, supporting the CYP to ensure completion of the self-report measures. The following demographic details were also collected: age, sex, sexual identity (if aged ≥ 16 years), ethnicity, nationality, religion, education, employment and current living arrangements.

Outcome measures

Outcome measures are summarised by trial arm at baseline and follow-up. As efficacy is not within the scope of a feasibility trial, estimated effect sizes are not presented. Completeness and variability of the outcome measures will be used to inform future trial design. In cases where multiple children were recruited from the same family, parents were asked to nominate an index child for inclusion in the main analysis. Other sibling(s) were still offered the opportunity to attend Young SMILES (if their family was randomised to the intervention group). Parents were not present for any data collection from their child/children. The number of families with multiple children is presented and summaries of demographics are presented for index children and all children.

Resource utilisation

Collection of child resource utilisation data was piloted using the Child and Adolescent Service Use Schedule (CA-SUS). We aimed to develop a resource use collection tool that captures the most important aspects of resource utilisation and assess the feasibility of collecting child resource utilisation.

The CA-SUS was adapted for our study setting in consultation with Professor Sarah Byford, the designer of the CA-SUS questionnaire. This adaptation involved removing the sections on out-of-pocket expenses and employment, removing the question on education type, removing the follow-on questions asking name of hospital for the hospital service use questions, removing complementary therapist (e.g. homeopath) from the list of community services, adding NHS walk-in services and NHS Direct to the list of community services, and simplifying the questions in the criminal justice services section. A copy of the CA-SUS version used for data collection was submitted alongside this report [see project web page: www.journalslibrary.nihr.ac.uk/programmes/hta/142901/#/documentation (accessed April 2020)].

Resource utilisation data were collected retrospectively using participant recall. The adapted version of the CA-SUS records resource use in the following categories: accommodation, education, hospital services, community services, medication and criminal justice services. Resource utilisation data were collected at three time points: baseline (recalling use over the previous 6 months), 4-month follow-up (recalling use since baseline) and 6-month follow-up (recalling use since the last interview).

As the aim was to assess the feasibility of collecting resource utilisation data to inform data collection methods in a future trial, reporting of results is purely descriptive. No statistical testing was performed.

Qualitative interviews

Acceptability is a key priority in the design, implementation and evaluation phases of complex interventions.117 It is perceived as a necessary but not sufficient condition concerning the effectiveness of an intervention. Broadly speaking, acceptability is a construct that examines the degree to which an intervention is accepted by those delivering and/or receiving it.117,118 Definitions have varied considerably, depending on what theoretical perspective the researcher adopts. For example, two studies119,120 investigate acceptability in the terms of patient attitude or satisfaction, whereas other work121,122 conflates the construct with patient behaviours, such as engagement and adherence to treatment. This lack of consensus has served to undermine the credibility of acceptability as a valid assessment instrument.117

A post-intervention qualitative evaluation of Young SMILES was conducted with CYP, parents/carers and eligible referrers. This was to determine acceptability among those that received the intervention, as well as to assess how practitioners and potential referrers were able to deliver the intervention. These studies (four in total) took place after the primary outcome point. Interviews with CYP and parents/carers focused on, inter alia, barriers to and facilitators of attending, what they liked and disliked, and if they thought Young SMILES helped their family. Referrer and practitioner interviews explored which factors facilitated or hindered the implementation of Young SMILES (see Chapters 5 and 6 for a description of the methods and analysis).123

Sample size in all of the qualitative studies conducted was governed by the numbers of eligible participants referring, delivering or taking part in the intervention, rather than by traditional concepts of data saturation. In qualitative research, saturation is reported as a criterion for achieving an adequate sample size, and thus no additional information is expected to enhance or change the findings. However, this is an inappropriate quality marker where sample sizes are limited by other factors, including trial participation. In such studies, information power is a more appropriate concept,123 with the final sampling frame guided by the study aim. A less extensive sample is needed when participants taking part are targeted because of their characteristics that are aligned with the study aim, in this case trial (and intervention) participation. Recruitment ceased when all intervention facilitators and participants had been invited for interview. Data analysis in the context of this feasibility study was focused on achieving key insights that contributed to or challenged current understandings of intervention acceptability.

Sample size

Formal power calculations for between-group effects were not conducted as is appropriate for a feasibility study, which is primarily aimed at developing and piloting a new intervention and training package. The study that was piloted examined the feasibility of the Young SMILES intervention training and the acceptability of delivering this to young people in a RCT design in two different settings. The original sample size proposed at the outset was 60 randomised families (30 per group, as recommended for pilot studies124). This was deemed sufficient to facilitate the main study aims [e.g. determining feasibility of recruitment and estimating standard deviations (SDs) of the outcome measures to perform a future power calculation]. The aim was to run a minimum of three sets of CYP groups, alongside three parallel parent groups in the two recruitment sites during the recruitment period. The new target sample size was 35–40 families, following challenges during the study that significantly restricted our recruitment and follow-up periods (see Appendix 1, Table 31).

Randomisation

The randomisation ratio for the final three groups was 2 : 1. Randomisation lists were stratified by age group and site, prepared using Stata^® 15 (StataCorp LP, College Station, TX, USA) by an independent statistician with the command ralloc, using random blocks of size 3 and 6.

Blinding

Randomisation was conducted by the study co-ordinator to maintain blinding of the statistician and research team, thereby reducing detection bias. To facilitate blinding, the research team adhered to the following: (1) that families knew that they could contact the study co-ordinator should they have any queries about their allocation or Young SMILES arrangements and (2) the researcher conducting the data collection visits reminded families to refrain from revealing their allocation during follow-up visits. Should a researcher become aware of any allocations, this was recorded and reported.

Statistics

The overall focus is on summaries of the key indicators of success of the study: recruitment and participant flow. Data are reported in line with the feasibility and pilot extension of the Consolidated Standards of Reporting Trials (CONSORT) statement.125 The numbers of participants who dropped out from the intervention, withdrew their consent and did not provide follow-up outcome data are also reported.

Adverse events

During the baseline and subsequent follow-up visits, trial researchers identified if any adverse events occurred and reported, as per standard operating procedures (SOPs) [see additional files (www.journalslibrary.nihr.ac.uk/programmes/hta/142901/#/documentation; accessed April 2020)]. The Trial Steering Committee monitored participant safety during the trial and was responsible for reviewing any serious adverse events occurring as part of the trial.

Public involvement

The research team actively consulted with young people and practitioners about what the intervention would comprise. This included the structure of the group and the activities within the groups. Young people and practitioners were also consulted on the outcome measures that would be used to calculate the time taken to complete the outcome measure booklets.

Young people, parents/carers, referrers and practitioners were all interviewed to provide feedback on the trial recruitment procedures, their experience in the group and what effect the Young SMILES intervention had on experiences outside the group.