Integrating qualitative interviews in drug development and the use of qualitative evidence in product labelling and health technology assessments: a review

Abstract

Objective

Including qualitative research in clinical trial design is an innovative approach to understanding patients’ perspective and incorporate the patient’s voice in all stages of drug development and evaluation. This review aims to explore current practices, lessons learned from the literature, as well as how qualitative interviews are considered by health authorities for marketing authorization and reimbursement.

Methods

A targeted literature review of Medline and Embase databases was conducted in February 2022 to identify publications on qualitative methods embedded in clinical trial of pharmaceutical products. An additional search of guidelines and labeling claims of approved products regarding qualitative research was performed in various sources of grey literature.

Results

From the 24 publications and nine documents reviewed, we identified the research questions addressed with qualitative methods during clinical trials (e.g., change in quality of life, symptoms assessment, treatment benefit), preferred data collection methods (e.g., interviews), and data collection points (e.g., baseline and exit interviews). Moreover, the data from labels and HTAs demonstrate that qualitative data can play an important role in approval processes.

Conclusion

The use of in-trial interviews is still emerging and is not yet common practice. Although the industry, scientific community, regulatory agencies and HTAs are showing an increasing interest in the use of evidence generated via in-trial interviews, guidance from regulators and HTAs would be helpful. Developing new methods and technologies to address the common challenges for such interviews is key to progress.

Introduction

Patient-focused drug development (PFDD) aims to support systematic capture and meaningful incorporation of patients’ voices (experiences, perspectives, needs and priorities) into all stages of drug development and evaluation, and continues to be one of the innovations shaping drug development (1). Recent guidance on PFDD from the FDA suggests multiple research methods to understand what is important to patients, including qualitative, quantitative, or mixed methods research (MMR) (2). While qualitative research methods such as interviews and focus groups are used across many fields, within drug development these methods have been primarily used to support the generation of content validity evidence for patient-reported outcome (PRO) measures (3–5). The use of such methods to address other objectives within drug development, beyond the above, is relatively recent (6), and is on the increase. For example, peer-reviewed articles and guidelines around the use of in-trial interviews doubled between 2012 and 2022 (Figure 1).

Figure 1 Overview of qualitative research embedded in clinical trials (2012–2022). (Figure not reproduced here; see the original article.)

In-trial interviews provides a platform to gain insights on the drug under investigation, e.g., providing an understanding of the patient’s experience of the treatment, insights on which specific symptoms or impacts change during a trial, clarity on the meaningfulness of such changes, and supporting the interpretation of other quantitative assessments in a trial, such as PRO measures (2, 7). Such interviews can be embedded within clinical trials and included as a trial objective or may be conducted as a stand-alone study in parallel to the clinical trial, recruiting participants from the trial.

Given the novelty of such approaches, we conducted a literature and guideline review to understand (1) the current practices in the implementation of patient interviews in drug development in terms of the type of questions addressed, the methodology employed and (2) how data generated from interviews are considered by health authorities for marketing authorization and reimbursement. We share the lessons learned from the literature and recommendations on the conduct of in-trial interviews.

Methods

A targeted literature review was performed to identify clinical trials integrating interviews, and regulatory guidelines and documents, and medicinal product labels including any mention of qualitative research.

Searches of in-trial interview studies published between January 2011 and February 2022 was conducted in Medline and Embase databases (via OVID platform) using Thesaurus, MeSH, and free text terms (See Search Strategy in Table 1). The search retrieved 710 references that were reviewed and selected according to the PICOS criteria (8). Although specific therapeutic indications were initially included, the selection was extended to chronic and non-oncologic diseases if the outcome around the use and impact of in-trial interviews was discussed (Figure 2).

Figure 2 Targeted literature review PRISMA flow diagram (9). (Figure not reproduced here; see the original article.)

Searches for guidelines or documents issued by regulatory agencies as of February 2022, as well as reports from health technology agencies (HTA) or learned societies was performed. We sought to find documents which included discussion and recommendations on the use and impact of qualitative research in clinical trial settings. The searches were performed in the PROINSIGHT™ database (10) and the websites of HTA, regulatory agencies and learned societies (Table 2). This search identified 19 documents issued by regulatory agencies, 45 documents from HTAs and 57 from learned societies. The documents (including drug reports and guidelines) were reviewed and selected if they discussed or included recommendations around the use and impact of qualitative studies, in a clinical trial context. Documents with no qualitative method research information, about PROs development or not related to the context of drug development were excluded (Figure 3A).

Figure 3 (A) Grey literature review of guidelines and documents mentioning in-trail interviews PRISMA flow diagram (9). (B) Grey literature review of drug labels, drug reports and HTA reports PRISMA flow diagram. (Figure not reproduced here; see the original article.)

Finally, a search in the PROLABELS™ database was performed in February 2022, to find any examples of inclusion of evidence, data or results from interviews in product labeling. PROLABELS™ is a database on PRO endpoint strategies used to demonstrate the efficacy and safety of drugs or devices in the market approval (11). The search used free text terms related to qualitative methods or concepts and retrieved 257 labels and medical reviews (Table 2). The labels were reviewed and selected if they mentioned the use of qualitative methods (Figure 3B). HTA reports were subsequently searched to check if qualitative methods were used for reimbursement evaluation by payers.

Results

Twenty-four articles, including 22 peer-reviewed articles and two conference abstracts on in-trial interview studies were extracted; seven guidelines gave recommendations about the use of patient input and qualitative research in clinical trials; one drug label, two drug reports and four HTA reports showed how qualitative interviews have been used by health authorities (Figure 1).

Peer-reviewed articles review

The 27 studies described in the 24 articles reviewed addressed different research questions, and were based on different clinical populations, method, study design, and settings (Table 3). The research questions included: patients clinical trial experience (n = 7), treatment satisfaction and acceptability (n = 7), treatment adherence (n = 4), change in quality of life (QoL) (n = 4), assessment of signs and symptoms (n = 3), intervention success (n = 3), treatment benefit (n = 2), health outcomes (n = 1), coping strategies (n = 1), social functioning (n = 1), decision making process (n = 1) and treatment experience (n = 1). More than one research question was addressed in some studies, for example one study explored signs and symptoms and the change in QoL (28) while another explored signs and symptoms, clinical trial experience and treatment satisfaction/acceptability (29). In most studies (n = 18), interviewees included patients only; five studies included interviews with patients, parents, partners and/or a clinician, one with community members, and three studies did not mention the population interviewed. The number of interviewees ranged from 9 to 78. The therapeutic areas in which studies were conducted included oncology (n = 7), infectious diseases (n = 7), respiratory tract diseases (n = 3), digestive system diseases (n = 2), cardiovascular diseases (n = 2), nervous system diseases (n = 1), mental health disorders (n = 1) and eye disease (n = 1; Table 3). Several qualitative data collection methods were described with one-on-one interviews being the most common (n = 26). Interviews were semi-structured (n = 16), in-depth (n = 8), serial ethnographic interviews (n = 1), or key informant interviews (n = 1). Other qualitative methods included focus groups (n = 5) and direct observations of participants and processes as a complement to interviews (n = 2). Three studies mentioned the inclusion of interviews but did not specify a particular type of interviews. Interviews were conducted using different modes, including telephone (n = 9), face-to-face (n = 7), both face-to-face and by telephone/video call (n = 2); nine studies did not specify the mode used. Face-to-face interviews were conducted either at the interviewee’s home or on site (hospital). Interviewers were from qualitative (n = 13), social science (n = 2), medical (n = 1), or local trial staff (n = 1) backgrounds; ten studies did not specify the professional background of the interviewers. Among the studies retrieved, 11 followed a longitudinal design with entry/baseline and/or intermediate and/or exit interviews (12, 22–26, 29, 30, 33, 36, 37), seven were exit interviews only (13, 16, 21, 28, 31, 34, 35), two were entry interviews only (14, 26) and seven did not specify the interview time point. The interviews were embedded in phase 2 studies (n = 10), phase 2/3 (n = 1), phase 3 (n = 5), phase 4 (n = 1), in studies where the phase was not applicable (n = 7) or not specified (n = 3). Participants interviewed included the entire trial population or a subpopulation. The interviews were conducted with all study participants in three trials (14, 26, 32) and with a subset of the study participants (median of about 14%) in 20 trials (12, 13, 15–18, 21–25, 27–31, 33–36).

Review of regulatory guidelines and documents mentioning in-trial interviews

Our searches for regulatory guidelines/documents highlighted the growing interest of regulators and HTAs in qualitative research throughout product development. Since 2013, seven guidelines [two from FDA (2, 38), one from EMA (39), one from the German agency IQWIG (40), two from the Canadian Agency CADTH (41, 42) and one from the European Network for HTA (EunetHTA)-EMA (43)] and two drug reports by the CADTH (44, 45) mentioning the use of qualitative interviews were issued. In its 2013 Guidance on “the Patient’s Voice in the Evaluation of Medicines,” the EMA acknowledged the importance of “consulting and involving patients” at both pre- and post- authorization stages (39). They commented that data from real-world settings could help understand the benefit–risk ratio of a medicine. At that time, EMA drafted a list of actions supporting patient involvement in the drug development journey including the need to identify where quantitative versus qualitative input is needed, and the development and validation of new tools for eliciting values and preferences and representing benefit and risk. In its 2016 Guidance on “Patient Preference Information,” the FDA supported the value of qualitative patient preference perspective in identifying outcomes most important to patients in the context of benefit–risk assessments, and in informing clinical trial design by participating in identifying endpoints that may be of greatest importance to patients (38). The FDA provided research questions that could be addressed by several qualitative methods, such as “how do people perceive this disease/this intervention?.” Additionally, using qualitative research in the earliest stages of the drug development can help frame the questions to be pursued in subsequent studies. The FDA’s “Patient-Focused Drug Development Guidance 2” dedicated a specific appendix (number 5) on in-trial interviews (2). In this guidance, the FDA described the potential usefulness of patient input gathered using in-trial interviews, including, for example, reporting changes in symptoms or functioning, insights on participants treatment expectations, anticipated and unanticipated symptoms and side-effects, and exploring viability of proposed dosing regimen. Recommendations include: conducting interviews before (i.e., screening/baseline interviews) or after (i.e., exit interviews) patients complete the main portion of the clinical study, ensuring a trained, neutral third-party interviewer to conduct the interview even if, in some instances, interviews can be conducted with next of kin (2) as they may have unique perspectives important to medical product development programs especially with children and confused or unconscious adults (22, 25). CADTH, in its 2020 guidance for “providing patient input” indicated that patient perspectives would improve the quality of CADTH reimbursement reviews. They stated, “the information that patient groups share with CADTH helps reviewers and expert review committee members understand the impact (both good and bad) that the treatment has on those taking it and on those caring for loved ones living with the disease” (42). The evidence presented in the reimbursement evaluation is mostly drawn from clinical and qualitative studies. CADTH discussed in 2022 the need to better understand which stage of the drug development, exploring the patient’s perspective would be the most useful in providing such evidence in its “CADTH framework for patient engagement in health technology assessment” guidance (41). IQWIG, in 2020 mentioned the use of MMR in its “general methods - version 6.0” document but did not provide further details (40). EunetHTA and EMA published a joint document in 2021 to guide applicants in the design, conduct and analysis of patient preference studies suitable to inform regulatory and reimbursement decision-making throughout the treatment development life cycle (43). The document discussed the use of qualitative methods, including interviews, in the context of preference studies.

Although we found examples in the literature of studies illustrating the guideline recommendations, these guidelines were not mentioned in any of the articles retrieved from the literature review.

Case studies–telotristat ethyl, dupilumab, emicizumab and pembrolizumab

In addition to guidelines, we identified several examples demonstrating how evidence from qualitative interviews is used by health authorities. A search of EMA in the summary of product characteristics (SmPC), FDA’s labeling, and reports of major HTAs, on the use of qualitative interviews yielded four products: telotristat ethyl (Xermelo®), dupilumab (Dupixent®), emicizumab (Hemlibra®) and pembrolizumab (Keytruda®). The case of telotristat ethyl provides an understanding of what aspects of the qualitative evidence generated by the interviews have been considered by health authorities for market authorization and reimbursement. Telotristat ethyl is indicated in Malignant Carcinoid Syndrome (CS) and was approved by the FDA (February 2017) and by the EMA (September 2017). The drug developer conducted exit interviews on a subset of 35/135 patients participating in the pivotal TELESTAR study (46). In the SmPC, the EMA outlined that the patients exit interview sub-study was conducted over the telephone to assess “the relevance and clinical meaningfulness of symptom improvements […] and to further characterize the degree of change experienced during the trial” and satisfaction with telotristat ethyl. These exit interviews were not mentioned in the FDA label. The EMA public assessment report noted that a high percentage of patients (n = 21) reported high bowel movement frequency as the most important symptom of CS which impacted emotional, social and physical functioning. The interviews were also mentioned in the reports issued by the French Health Agency (HAS), All Wales Medicines Strategy Group (AWMSG) and CADTH to support telotristat ethyl reimbursement. The HAS agency expressed its inability to conclude on the results themselves due to the small sample size, which does not guarantee representativeness. The AWMSG, responsible for appraising new medicines for use in Wales considered the interviews as an added value for economic evaluation and a complement to the cost-utility measures captured in the quality-adjusted life year. Finally, the CADTH mentioned seven quotes from patients interviews to document their decision. The second case is that of dupilumab indicated for the treatment of moderate-to-severe atopic dermatitis. Dupilumab was approved by the FDA (March 2017) and by the EMA (October 2017). The drug developer conducted one-on-one interviews with study participants in Canada. The public assessment report from the CADTH summarized four key inputs from patients’ perspective. These interviews were not mentioned in the FDA label nor the EMA SmPC. The third case is emicizumab, which is indicated for routine prophylaxis of bleeding episodes in patients with hemophilia A. Emicizumab was approved by the FDA (November 2017) and the EMA (March 2018). The drug developer conducted one-on-one phone interviews with six health care professionals (HCP) involved in the phase III HAVEN program in France (three physicians, two nurses and one clinical research associate) (47). The public assessment report from the HAS mentioned only five HCP interviews and noted that the interviews were conducted because “the patient quality of life questionnaires used in the two-phase III studies did not demonstrate the impact of emicizumab on patients’ quality of life, as some investigators have suggested.” However, the data generated by the interviews were of an exploratory and not of a demonstrative nature. The interviews were mentioned in the 2018 public assessment report issued by the HAS but not in the 2019 report, nor in the FDA label and the EMA SmPC. The last case is pembrolizumab, which is indicated for the treatment of multiple cancers including the treatment of metastatic or unresectable recurrent head and neck squamous cell carcinoma. Pembrolizumab was first approved by the FDA (September 2014) and by the EMA (July 2015). The drug developer solicited patient input from a total of 13 respondents from the Life Saving Therapies Network patient group. The public assessment report from the CADTH focuses on one interviewee and two survey respondents; and they reported positive experiences of improvement of quality of life and ability to perform day-to-day tasks. Additionally, the CADTH noted that the three patients did not report any side effects associated with pembrolizumab and stated that it was effective in controlling their cancer compared to their previous treatment. Overall, the qualitative data informed the benefit–risk profile, with patients appreciating the increased efficacy of treatments, a better side effect profile, and improved quality of life. This was not mentioned in the FDA label nor in the EMA SmPC. In the four drug examples, regulators and HTA agencies focused their evaluation on the efficacy and safety of the drug and on the assessment of benefits and risks. The qualitative data generated by the interviews helped the regulators and HTA agencies to make positive decision about drugs approval and/or reimbursement.

Discussion

Use of qualitative interviews in the drug development

Combining qualitative research and clinical trials has the potential to generate additional evidence to support an investigational product. At the research or pre-clinical stage, qualitative methods, can be used as a novel, innovative approach to understand and describe the natural history of a disease and unmet medical needs. At the early stages of a drug’s clinical development, interviews are useful to inform on treatment benefit and safety profile. Indeed, patients reports on changes in the clinical manifestation of the disease may also indirectly inform the mechanism of action of the investigational drug. Additionally, when designing a phase 2 or 3 clinical trial, qualitative interviews should help determine the appropriate endpoints by providing insight into the most troublesome symptoms or aspects of functioning or quality of life that are most affected.

In-trial interviews allow for individualized, patient-centered insights of the risks and benefits of the experimental treatment and can be valuable to assess the viability of a proposed dosing regimen. Interviews can eventually support the identification, development, or adaptation of PRO measures for use in later phases or other trials in the same indication or population. Qualitative data is valuable in providing supportive evidence of the content validity of the primary or key secondary endpoints in health authorities submissions. When questions arise on the clinical meaningfulness of magnitude of change during regulatory or reimbursement review process, in-trial interviews may be critical in offering more granular data/information for the interpretation of such changes. After approval, evidence from interviews may help health authorities understand the benefit–risk ratio of a drug in a real-life context. Our literature review showed that in-trial interviews were mainly used in phase 2 or 3 trials. There was no indication, however, whether the studies were conducted for the purpose of a subsequent marketing authorization application.

Among the objectives described in the literature, eight (treatment satisfaction and acceptability, treatment adherence, change in quality of life, assessment of signs and symptoms, treatment benefit, health outcomes, social functioning, and treatment experience) could be of interest for the regulators and HTA agencies while evaluating the drug profile for market authorization or reimbursement. These objectives were explored in 21 of 27 studies (12–17, 24–36).

Regulatory and reimbursement guidance is needed to specify the information that would be most useful to regulators and HTA agencies in their assessment of the evidence that could be generated by qualitative studies embedded into clinical trials. A guideline would allow researchers to better define the objectives of their research, focus their efforts on certain assessments and provide evidence that meets the needs of agencies in their evaluation. Such guidance would complement what the FDA has initiated with its PFDD guidances (2, 48). This will enable greater certainty of decision-making by the authorities when it comes to getting a drug through the licensing and reimbursement process.

To make qualitative evidence more impactful, the objective of the qualitative interviews should be clearly defined and aligned with those of the clinical trial, and the questions asked to the participants should reflect these objectives. Crossing the data from the qualitative interviews and the clinical trial could generate new information on patients’ experience. It would be interesting to compare patients’ qualitative reports with clinical or biomarker or PRO measures. This approach could be valuable as it would involve the patients in triangulating their data, even if this design implies higher levels of data protection security for compliance with local regulations. In rare diseases (30), qualitative interviews are particularly relevant as the heterogeneity of the symptoms and their impacts on quality of life make it challenging to measure with a generic PRO questionnaire and there is often no possibility of investing in a disease-specific instrument. Whatever the objectives, therapeutic area or setting, to make in-trial interviews a successful outcome, communication and partnership between the clinical study team, the investigator sites, and the qualitative research team must be established as early as possible.

Methodological considerations

Regarding the methodology, the parameters for ensuring the quality and reproducibility of the results differ from those of quantitative research. Qualitative research aims at characterizing a phenomenon, this is why the sampling methods relies on the principle of saturation (the point where no new information seems to emerge from the data) (49, 50). Consequently, the number of interviews should be estimated before the interviews are conducted but may not be fixed. A subsample of the trial population may be interviewed (28), and it is recommended to pre-define the sampling method according to the objectives of the research (systematic, probability, purposive, convenience) to anticipate on reviewers’ comments as seen with telotristat. Interviewing all the clinical trial’s patients presents challenges which may be circumvented by achieving saturation in a smaller sample size (12, 13, 36). This aspect is often challenging to conceptualize for reviewers coming from a quantitative world and there is room for communication around this topic. Interviewing in qualitative research requires specific skills as the interview techniques have impacts on the result’s quality. The FDA generally recommends involving qualitatively trained, neutral third-party interviewers (2). Their role should be as neutral as possible, therefore choosing HCPs or study staff as interviewers is not recommended (2, 51, 52). Interviewees should be interviewed in local language to avoid cultural and language barriers. The mode of administration (in-person, telephone, online/virtual video conferences) and the setting of the interviews (individual interviews or focus groups) should be adapted to the population (29, 51).

A clinical trial can include one or more interviews at different time points. Timing considerations are directly linked to the research question (6) (Table 4). The FDA recommends conducting interviews before (screening/baseline interviews) or after (exit interviews) patients complete the main portion of the clinical study to avoid potential compromise of study integrity (2). Conducting interviews at entry mid-stage and exit of the same clinical trial allow for a longitudinal understanding of new therapies and their impact on patients’ daily life. The timing of the interviews should be discussed when qualitative research is designed. In exit interviews, especially for long-term clinical trials, recall bias is critical. Patients with chronic diseases may have difficulty describing the course of their disease before the trial and describing any changing experiences during the trial because of the complexity of their disease and the length of time since their symptoms began. In this case, interviewing patients at screening and then at several time points in the trial provides a qualitative baseline and can help manage recall bias. Longitudinal qualitative research yields a large amount of data given that the objective is to understand an experience over time. The complexity of longitudinal qualitative data makes the analysis challenging; selecting the analytical approach and developing a qualitative analysis plan are critical to ensure rigor and meet quality standards (53), and to ensure validity and reliability of the analysis. This also applies to cross-sectional analyzes conducted at trial entry or exit. It is important to note that since longitudinal qualitative research involves a larger number of interviews and analyzes, the cost is higher than that of a single interview. In-trial interviews should consider potential challenges. Regardless of what phase the trial is being conducted, the protocol, and informed consent form describing the interviews may be included in the trial protocol or may be developed separately (6). This information is not always described in the literature. The earlier in-trial interviews are envisaged in the design, the better as they can be included in the clinical protocol thereby avoiding additional amendments and ethics re-submissions. The content of the interview guides has a direct impact on the quality of the results. Hence, sufficient time should be allowed to work on interview objectives, so the resulting guide aligns with the research question and avoids being list or a survey of too many topics. Duration of the interview should consider the population interviewed. The format of the interview guide depends on the interviewer’s experience (thematic vs. narrative).

Operational considerations

Operationally, researchers must heed regulatory and data protection requirements, legal framework, ethics-related procedures, good clinical practices guidelines and pharmacovigilance. An example is the management of adverse events (AEs). Although AE reporting is not the purpose of the interviews, participants may spontaneously report them, as a result, management should anticipate how these reports will be managed as part of the operational set up of the trial.

For future considerations, an approach called Natural Language Processing (NLP) is emerging in health care research (9). NLP is a subfield of linguistics, computer science, and artificial intelligence that processes and analyzes large amounts of natural language data. NLP combined with in-trial interviews, would be a powerful strategy for analyzing and triangulating the information generated.

Limitations

Due to the nature of this literature review, various potential limitations are worth noting. The first relates to the search strategy; the search terms employed focused on specific therapeutic indications or areas, for example. The second limitation relates to the screening method used for study selection; a single researcher performed all screening. Given the purpose of this targeted, non-systematic literature review, an approach to use a single reviewer with extensive experience with literature reviews was considered appropriate. Despite these limitations, our approach ensured that we were able to address our main objective, i.e., to provide an understanding of the current use of qualitative interviews in drug development.

Conclusion

In-trial interviews is still emerging and not yet common practice. While there is increased interest in evidence generated via in-trial interviews throughout industry, scientific communities, regulatory agencies and HTAs, guidance from regulators and HTAs would be helpful and address what type of benchmark applies (e.g., good clinical practices). The scientific community could spearhead establishment of recommendations, best practice, and further development of methods/technologies to address common goals and challenges for such interviews. While acknowledging that qualitative research is not meant to replace quantitative research, and that it is a different and complementary valuable means of addressing questions during drug development, a mindset change within research and development in pharmaceutical industry is necessary to fully realize the potential of in-trial interviews as a tool for drug development.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Funding

This study was funded by Merck KGaA. A-SM, AM, RA, CD-G, and SL are consultants paid by Merck for the conduct of the literature review and analysis. The authors declare that this study received funding from Merck KGaA. The funder was involved in the study design, analysis, interpretation of data, reviewing this article, and the decision to submit it for publication.

Conflict of interest

A-SM, AM, RA, CD-G, and SL are employees of ICON. PS is an employee of LAIFE REPLY. PK is an employee of Merck KgaA. JP and ES are employees of EMD Serono (a company of Merck KgaA). ICON and LAIFE REPLY were contracted with Merck KgaA as consultants on patient-centered research activities.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. BaschE. Toward patient-centered drug development in oncology. N Engl J Med. (2013) 369:397–400. doi: 10.1056/NEJMp111464923822654 doi:10.1056/NEJMp1114649 PubMed
2. FDA. Patient-focused drug development: methods to identify what is important to patients. Maryland, US: Food and Drug Administration (2022).
3. FDA. Patient-reported outcome measures: use in medical product development to support labeling claims. Maryland, US: Food and Drug Administration (2009).
4. PatrickDLBurkeLBGwaltneyCJLeidyNKMartinMLMolsenE. Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: part 1--eliciting concepts for a new PRO instrument. Value Health. (2011) 14:967–77. doi: 10.1016/j.jval.2011.06.01422152165 doi:10.1016/j.jval.2011.06.014 PubMed
5. PatrickDLBurkeLBGwaltneyCJLeidyNKMartinMLMolsenE. Content validity--establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: part 2--assessing respondent understanding. Value Health. (2011) 14:978–88. doi: 10.1016/j.jval.2011.06.01322152166 doi:10.1016/j.jval.2011.06.013 PubMed
6. AlladinABarrioJDikariyantoVVenerusMRolstMKrolM. A literature review of methodology used in in-trial interview-studies in clinical development. presented at: ISPOR Europe, pp. 6–9. (2022).
7. MurphyABereNVamvakasSMavrisM. The added value of patient engagement in early dialogue at EMA: scientific advice as a case study. Front Med (Lausanne). (2021) 8:811855. doi: 10.3389/fmed.2021.81185535127766PMC8811124 doi:10.3389/fmed.2021.811855 PubMed
8. MethleyAMCampbellSChew-GrahamCMcNallyRCheraghi-SohiS. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 2014/11/21. (2014) 14:579. doi: 10.1186/s12913-014-0579-025413154PMC4310146 doi:10.1186/s12913-014-0579-0 PubMed
9. FangCMarkuzonNPatelNRuedaJD. Natural language processing for automated classification of qualitative data from interviews of patients with Cancer. Value Health. (2022) 25:1995–2002. doi: 10.1016/j.jval.2022.06.00435840523 doi:10.1016/j.jval.2022.06.004 PubMed
10. PROINSIGHT™. Mapi Research Trust. Available at: https://eprovide.mapi-trust.org/about/about-proinsight (Accessed February 22, 2022).
11. PROLABELS™. Mapi research trust. Available at: https://eprovide.mapi-trust.org/about/about-prolabels (Accessed February 22, 2022).
12. AgotKTaylorDCorneliALWangMAmbiaJKashubaADM. Accuracy of self-report and pill-count measures of adherence in the FEM-PrEP clinical trial: implications for future HIV-prevention trials. AIDS Behav. (2015) 19:743–51. doi: 10.1007/s10461-014-0859-z25100053PMC4415940 doi:10.1007/s10461-014-0859-z PubMed
13. AmicoKRWallaceMBekkerLGRouxSAtujunaMSebastianE. Experiences with HPTN 067/ADAPT study-provided open-label PrEP among women in Cape Town: facilitators and barriers within a mutuality framework. AIDS Behav. (2017) 21:1361–75. doi: 10.1007/s10461-016-1458-y27317411PMC5378745 doi:10.1007/s10461-016-1458-y PubMed
14. AyiekoJBrownLAnthierensSvan RieAGetahunMCharleboisED. Hurdles on the path to 90-90-90 and beyond: qualitative analysis of barriers to engagement in HIV care among individuals in rural East Africa in the context of test-and-treat. PLoS One. (2018) 13:e0202990. doi: 10.1371/journal.pone.020299030161172PMC6116983 doi:10.1371/journal.pone.0202990 PubMed
15. Van der StratenAStadlerJMontgomeryEHartmannMMagaziBMathebulaF. Women's experiences with oral and vaginal pre-exposure prophylaxis: the VOICE-C qualitative study in Johannesburg, South Africa. PLoS One. (2014) 9:e89118. doi: 10.1371/journal.pone.008911824586534PMC3931679 doi:10.1371/journal.pone.0089118 PubMed
16. StarrHLAbellJLarishALewisSDeMuroCGogateJ. Self-reported review of the value of esketamine in patients with treatment-resistant depression: understanding the patient experience in the STRIVE study. Psychiatry Res. (2020) 293:113376. doi: 10.1016/j.psychres.2020.11337632818917 doi:10.1016/j.psychres.2020.113376 PubMed
17. LinDKleinACellaDBeutlerAFangFMagestroM. Health-related quality of life in patients with recurrent pericarditis: results from a phase 2 study of rilonacept. BMC Cardiovasc Disord. (2021) 21:201. doi: 10.1186/s12872-021-02008-333882846PMC8061027 doi:10.1186/s12872-021-02008-3 PubMed
18. MoynihanCLewisRHallEJonesEBirtleAHuddartR. The patient deficit model overturned: a qualitative study of patients' perceptions of invitation to participate in a randomized controlled trial comparing selective bladder preservation against surgery in muscle invasive bladder cancer (SPARE, CRUK/07/011). Trials. (2012) 13:228. doi: 10.1186/1745-6215-13-22823190503PMC3554516 doi:10.1186/1745-6215-13-228 PubMed
19. NelsonAHarropEFenlonDMorrisJHoodK. Measuring what matters: The case for qualitative research in palliative care clinical trials. (2012).
20. NelsonAFenlonDMorrisJSampsonCHarropEMurrayN. QUALZICE: a QUALitative exploration of the experiences of the participants from the ZICE clinical trial (metastatic breast cancer) receiving intravenous or oral bisphosphonates. Trials. (2013) 14:325. doi: 10.1186/1745-6215-14-32524107437PMC3856513 doi:10.1186/1745-6215-14-325 PubMed
21. LovellNEtkindSNBajwahSMaddocksMHigginsonIJ. What influenced people with chronic or refractory breathlessness and advanced disease to take part and remain in a drug trial? A qualitative study. Trials. (2020) 21:215. doi: 10.1186/s13063-020-4129-232087745PMC7036259 doi:10.1186/s13063-020-4129-2 PubMed
22. LawrenceDSTsholoKSsaliAMupambireyiZHoddinottGNyirendaD. The lived experience of participants in an African RandomiseD trial (LEOPARD): protocol for an in-depth qualitative study within a multisite randomised controlled trial for HIV-associated cryptococcal meningitis. BMJ Open. (2021) 11:e039191. doi: 10.1136/bmjopen-2020-039191PMC803047233820784 doi:10.1136/bmjopen-2020-039191 PubMed
23. ViljoenBHofmanMSChambersSKDunnJDhillonHDavisID. Advanced prostate cancer experimental radioactive treatment-clinical trial decision making: patient experiences. BMJ Support Palliat Care. (2021):bmjspcare-2021-002994. doi: 10.1136/bmjspcare-2021-00299434373282 doi:10.1136/bmjspcare-2021-002994 PubMed
24. FauskeLPhWÆRHomplandIBrulandYS. Hope as a lifeline: Imatinib discontinuation in patients with Oligometastatic gastrointestinal stromal Tumours. Anticancer Res. (2022) 42:955. doi: 10.21873/anticanres.1555535093895 doi:10.21873/anticanres.15555 PubMed
25. SearleAHerbertGDabnerLStewardCGDaminMPielesG. Self-regulation in Barth syndrome: a qualitative perspective of adolescents, adults and parents in the U.K. Orphanet J Rare Dis. (2021) 16:404. doi: 10.1186/s13023-021-02027-534587980PMC8482682 doi:10.1186/s13023-021-02027-5 PubMed
26. CamlinCSSeeleyJ. Qualitative research on community experiences in large HIV research trials: what have we learned? J Int AIDS Soc. (2018) 21 Suppl 7:e25173. doi: 10.1002/jia2.2517330334379PMC6192898 doi:10.1002/jia2.25173 PubMed
27. GelhornHLKulkeMHO’DorisioTYangQMJacksonJJacksonS. Patient-reported symptom experiences in patients with carcinoid syndrome after participation in a study of Telotristat Etiprate: a qualitative interview approach. Clin Ther. (2016) 38:759–68. doi: 10.1016/j.clinthera.2016.03.00227041406PMC9047308 doi:10.1016/j.clinthera.2016.03.002 PubMed
28. KhanRUrenACanhamLCottrellDDrakeMJCotterillN. What are the Participants' perspectives of taking melatonin for the treatment of Nocturia in multiple sclerosis? A qualitative study embedded within a double-blind RCT. Mult Scler Int. (2018) 2018:4721505. doi: 10.1155/2018/472150530420919PMC6211202 doi:10.1155/2018/4721505 PubMed
29. ErvinCMReasnerDSHanlonJTFehnelSE. Exploring the diabetic Gastroparesis patient experience: patient exit interviews. Adv Ther. (2017) 34:2680–92. doi: 10.1007/s12325-017-0632-629079987PMC5709450 doi:10.1007/s12325-017-0632-6 PubMed
30. BharmalMGuilleminIMarrelALambertJArnouldBFatoumataF. PRM6 - using convergent mixed methods to evaluate treatment risks and benefits in rare disease: an example from a phase ii registration trial in metastatic Merkel cell carcinoma. Value Health 2017/10/01/. (2017) 20:A731. doi: 10.1016/j.jval.2017.08.1991 doi:10.1016/j.jval.2017.08.1991
31. O'HaraMJandaMMcCarthyALNicklinJWalkerGObermairA. Patient experiences of conservative treatment for early stage endometrial cancer and endometrial hyperplasia with atypia using levonorgestrel intrauterine device: a qualitative study. Gynecol Oncol Rep. (2022) 39:100914. doi: 10.1016/j.gore.2021.10091435005158PMC8718559 doi:10.1016/j.gore.2021.100914 PubMed
32. EnochJGhulakhszianACrabbDPDinahCTaylorDJ. Acceptability of intravitreal injections in geographic atrophy: protocol for a mixed-methods pilot study. BMJ Open. (2021) 11:e049495. doi: 10.1136/bmjopen-2021-049495PMC807455133895721 doi:10.1136/bmjopen-2021-049495 PubMed
33. MangeshoPEReynoldsJLemngeMVestergaardLSChandlerCI. "every drug goes to treat its own disease…" - a qualitative study of perceptions and experiences of taking anti-retrovirals concomitantly with anti-malarials among those affected by HIV and malaria in Tanzania. Malar J. (2014) 13:491. doi: 10.1186/1475-2875-13-49125495956PMC4302145 doi:10.1186/1475-2875-13-491 PubMed
34. FosterJBeasleyRBraithwaiteIHarrisonTHollidayMPavordI. Perspectives of mild asthma patients on maintenance versus as-needed preventer treatment regimens: a qualitative study. BMJ Open. (2022) 12:e048537. doi: 10.1136/bmjopen-2020-048537PMC878516535063953 doi:10.1136/bmjopen-2020-048537 PubMed
35. FosterJMBeasleyRBraithwaiteIHarrisonTHollidayMPavordI. Patient experiences of as-needed budesonide-formoterol by Turbuhaler® for treatment of mild asthma; a qualitative study. Respir Med. (2020) 175:106154. doi: 10.1016/j.rmed.2020.10615433190085 doi:10.1016/j.rmed.2020.106154 PubMed
36. RapportFClementCSeagroveACAlrubaiyLHutchingsHAWilliamsJG. Patient views about the impact of ulcerative colitis and its management with drug treatment and surgery: a nested qualitative study within the CONSTRUCT trial. BMC Gastroenterol. (2019) 19:166. doi: 10.1186/s12876-019-1085-y31615445PMC6794775 doi:10.1186/s12876-019-1085-y PubMed
37. RegnaultAWillgossTBarbicS. Towards the use of mixed methods inquiry as best practice in health outcomes research. J Patient Rep Outcomes. (2017) 2:19. doi: 10.1186/s41687-018-0043-829757311PMC5934918 doi:10.1186/s41687-018-0043-8 PubMed
38. FDA. Patient preference information–voluntary submission, review in premarket approval applications, humanitarian device exemption applications, and De novo requests, and inclusion in decision summaries and device labeling. Maryland, US: Food and Drug Administration (2016).
39. EMA. The patient’s voice in the evaluation of medicines: how patients can contribute to assessment of benefit and risk. Amsterdam, Netherlands: European Medicines Agency (2013).
40. IQWIG. General methods–version 6.0 Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen. Germany: IQWIG (2020).
41. CADTH. CADTH framework for patient engagement in health technology assessment. Ottawa: Canadian Agency for Drugs and Technologies in Health (2022).
42. CADTH. Guidance for providing patient input. Ottawa: Canadian Agency for Drugs and Technologies in Health (2020).
43. EunetHTA. CHMP & EUnetHTA parallel scientific advice: Qualification of a framework and “points to consider” for method selection along with five methods for performing patient preference studies to inform regulatory and HTAbody medical product decision-making. EunetHTA (2021).
44. CADTH. Pembrolizumab (Keytruda) HNSCC – Final Clinical Guidance Report. Ontario: Canadian Agency for Drugs and Technologies in Health (2020).
45. CADTH. CADTH Canadian drug expert committee recommendation: Dupilumab (Dupixent–Sanofi-Aventis Canada Inc.). Ontario: Canadian Agency for Drugs and Technologies in Health (2018).30920788 PubMed
46. AnthonyLErvinCLapuertaPKulkeMHKunzPBergslandE. Understanding the patient experience with carcinoid syndrome: exit interviews from a randomized, placebo-controlled study of Telotristat ethyl. Clin Ther. (2017) 39:2158–68. doi: 10.1016/j.clinthera.2017.09.01329074312 doi:10.1016/j.clinthera.2017.09.013 PubMed
47. LambertTRothschildCBaillonNGaudinHAcquadroMArnouldB. PSY223–qualitative research in HAEMOPHILIA a: EMICIZUMAB transforms lifestyle of patients with inhibitor. Value Health. (2018) 21:S474. doi: 10.1016/j.jval.2018.09.2921 doi:10.1016/j.jval.2018.09.2921
48. FDA. Patient-focused drug development: incorporating clinical outcome assessments into endpoints for regulatory decision-making. Maryland, US: Food and Drug Administration (2023).
49. SmithJAHarréRVan LangenhoveL. Rethinking methods in psychology. Thousand Oaks, CA: Sage (1995).
50. StraussACorbinJ. Basics of qualitative research techniques. Thousand Oaks, CA: Sage Publications (1998).
51. BharmalMGuilleminIMarrelAArnouldBLambertJHennessyM. How to address the challenges of evaluating treatment benefits-risks in rare diseases? A convergent mixed methods approach applied within a Merkel cell carcinoma phase 2 clinical trial. Orphanet J Rare Dis. (2018) 13:95. doi: 10.1186/s13023-018-0835-129914528PMC6006962 doi:10.1186/s13023-018-0835-1 PubMed
52. MaiersMHondrasMASalsburySABronfortGEvansR. What do patients value about spinal manipulation and home exercise for back-related leg pain? A qualitative study within a controlled clinical trial. Man Ther. (2016) 26:183–91. doi: 10.1016/j.math.2016.09.00827705840 doi:10.1016/j.math.2016.09.008 PubMed
53. TuthillELMaltbyAEDiClementeKPellowskiJA. Longitudinal qualitative methods in health behavior and nursing research: assumptions, design, analysis and lessons learned. International journal of qualitative. Methods. (2020) 19:1609406920965799. doi: 10.1177/1609406920965799PMC845982534566547 doi:10.1177/1609406920965799 PubMed