8.6: “Can We Use Social Media to Support Content Validity of Patient-Reported Outcome Instruments in Medical Product Development?” by Margaret Rothman, et al
- Page ID
- 22444
Can We Use Social Media to Support Content Validity of Patient-Reported Outcome Instruments in Medical Product Development?
- Under a Creative Commons license
- doi:10.1016/j.jval.2014.10.001
- Get rights and content
Abstract
We report a panel designed to open a dialog between pharmaceutical sponsors, regulatory reviewers, and other stakeholders regarding the use of social media to collect data to support the content validity of patient-reported outcome instruments in the context of medical product labeling. Multiple stakeholder perspectives were brought together to better understand the issues encountered in pursuing social media as a form of data collection to support content validity. Presenters represented a pharmaceutical sponsor of clinical trials, a regulatory reviewer from the Food and Drug Administration, and an online data platform provider. Each presenter shared its perspective on the advantages and disadvantages of using social media to collect this type of information. There was consensus that there is great potential for using social media for this purpose. There remain, however, unanswered questions that need to be addressed such as identifying which type of social media is most appropriate for data collection and ensuring that participants are representative of the target population while maintaining the advantages of anonymity provided by online platforms. The use of social media to collect evidence of content validity holds much promise. Clarification of issues that need to be addressed and accumulation of empirical evidence to address these questions are essential to moving forward.
Keywords
- clinical trials;
- online communities;
- patient-reported outcomes;
- social media
Introduction
Patient-reported outcomes (PROs) are increasingly recognized as important tools in adding value to the drug review and evaluation process [1] because they provide unique perspectives on medical conditions or their therapies that are known only to the patient[2]. “Content validity” describes the extent to which a PRO intended to assess such subjective outcomes actually measures the concept of interest [3] and is an ongoing process that relies on the generation of qualitative and quantitative evidence. Crucially, PRO measures must reflect patient concerns relative to the concept being assessed and, therefore, documentation of content validity relies on patient input from the target population of patients [4]. Qualitative studies are used to generate appropriate items and domains; to ensure the instrument is comprehensive relative to its intended measurement concept, population, and context of use; and to ensure patient understanding of the instrument, that is, instructions, items, and response options through cognitive debriefing [5]. Best practices usually include either individual interviews or focus groups with participants who are experiencing the target condition or have recent experience with it. These traditional methods of collecting qualitative data to support the content validity of a new or existing PRO instrument, however, are labor intensive, time consuming, and relatively expensive. Although detailed figures are not available for specific parts of the instrument, development estimates from the New England Research Institutes suggest that developing a PRO from beginning to end takes at least 24 months and costs between $1 million and $5 million [6], whereas estimates shared at the 2011 C-Path meeting suggest up to 4 years for development and costs between $725,000 and $2.1 million [7]. Furthermore, there are potential limitations in the diversity of the sample and the amount of data that may be collected to support both comprehensiveness of concepts and understanding of the PRO instrument using these methodologies [5].
Social Media
A new tool that may offer a technological boost to these efforts is “social media,” a group of Internet-based applications (such as Facebook, Twitter, forums, or blogs) that allow the creation and exchange of user-generated content. The Pew Internet & American Life Project noted in 2011 that the Internet has “changed people’s relationships with information” in many areas that affect our lives including health. For example, “online resources, including advice from peers, are a significant source of heath information in the US” [8]. In a large and nationally representative survey of US adults in 2013, Pew found that 26% of the Internet users had read or watched someone else’s experience about health or medical issues in the past year and 16% had tried to find others with the same health concerns, typically through social media [9].
The reach of such online communities is their greatest strength. For example, it is estimated that in 2013 the number of total users for the four largest social media platforms was 1.15 billion for Facebook, 500 million for Twitter, 500 million for Google+, and 238 million for LinkedIn (visual.ly/social-media-2013). Given that a proportion of the cost of PRO development lies in contacting and recruiting people with specific medical conditions and gathering data from them, social media presents an intriguing new route to accelerating research.
This new form of technology is relatively untested in terms of the adequacy of the information collected to support current definitions of best practices for data generation and analysis. This is particularly true of research to support content validity for the development of PROs for use in clinical trials to support labeling claims in the United States. To capitalize on the advantages of this new technology, stakeholders need to agree on what is appropriate methodology and focus research on resolving these issues. Currently, there is limited knowledge in the public domain on this topic.
To address this question, the authors organized a panel at a recent ISPOR meeting (2013, New Orleans) to discuss the application of social media for this type of data collection. We identified four stakeholders to share their perspective although there are certainly others who may be able to contribute knowledge, expertise, and experience to understanding the topic. Panelists who contributed to this article included representatives from a pharmaceutical sponsor of clinical trials (M.R. and A.G.), a US regulatory reviewer (E.J.P.), and a provider of a patient-powered research network for data collection through social media (P.W.). Presenters were asked to share their knowledge and experience, particularly with respect to their perception of the advantages and disadvantages of using social media to generate data to support the content validity of PRO measures in the context of drug development (Table 1). The focus of this panel was on concept elicitation.
- Table 1.Comparison of traditional methods and potential for social media in PRO development.
Traditional methods Social media Participant identification Clinicians, hospital referral, advertising Patient self-identification and membership of online communities Diagnostic validation Primarily physician or medical records, sometimes self-reported Primarily self-reported, sometimes electronic medical records of physician Data collection setting Face-to-face individual or group interviews Asynchronous message boards, instant message chat, video chat Data collection format Semi-structured interview, in-person cognitive debriefing Interactive surveys, questionnaires, rating scales, video cognitive debriefing Advantages Criterion standard widely accepted by researchers and regulators as producing high-quality outputs Rapid, cheap, participatory, large-scale, new methods iterate rapidly. Typed data do not require transcription Limitations Time and labor intensive tasks such as transcription and recruitment New, untested in the regulatory approval process
Benefits of Social Media for PRO Concept Elicitation
All the speakers indicated their belief that there are potential benefits to using social media for collecting concept elicitation data. The potential to access larger numbers of persons in the target population and thus obtain a greater amount of information in a shorter period of time is an advantage over traditional methods. Each social network has its own strengths and weaknesses (which should be tailored to each study); however, it is important to recognize that networks go through life cycles far more rapidly than do traditional social establishments [10]. Although a full review of each network’s strengths and limitations was outside the scope of the panel, a recent review by Grajales et al. [11]provides a comprehensive overview of the scientific literature describing research with blogs (e.g., WordPress), microblogs (e.g., Twitter), social networks (e.g., Facebook), professional sites (e.g., LinkedIn), wikis (e.g., Wikipedia), mashups (e.g., HealthMap), collaborative filtering sites (e.g., Reddit), media sharing sites (e.g., YouTube), and multiuser virtual environments (e.g., Second Life).
In a comparison of semi-structured interviews with blogs for purposes of PRO development, Acaster and Wild [12] reported a high degree of convergence about symptoms experienced by women with menopausal hot flashes, with no major discrepancies in themes elicited between the two methods. In a study eliciting concepts among children from ulcerative colitis, Yen et al. [13] used blogs to substantiate concepts identified through traditional interviews and found these data to be supportive. This area is still developing, however, and the most recent ISPOR PRO Good Practices Task Force guidance makes no mention of data gathered online [14]. It is worth acknowledging, however, that different forms of social media might have differing potential to offer useful data. For example, the patient-powered research network PatientsLikeMe was recently described in the “tapestry of big data” as differing from blogs and tweets in that it not only captures structured data including demographic characteristics, medication, and diagnoses but also has aspects of a social network [15].
Although the authors are unaware of examples of the use of social media to develop PRO instruments for regulatory use to support labeling claims, it is often informative to draw from examples of uses of social media in instrument development outside this specific regulatory context. For instance, among the potentially broader online population there may be more participants who would not be responsive to traditional methods such as those who are too frail to travel or are not located within reasonable travel distance of a site. In one example provided to the panel, a long-term patient with amyotrophic lateral sclerosis (ALS) (who was a psychologist in her working life) complained that the ALSFRS-R scale [16] had a floor effect—she had been a “zero” on this scale for many years. Determined not to be measured as a zero when there were many things she could still do, she collaborated with the research team and other patients with an advanced form of ALS to elicit the things that people who were nearly “locked in” to their bodies could still do, using the Internet to survey patients who might be responding using assistive technology such as eyeblink sensors or tiny switches. Within just a few weeks, a survey was fielded to over 300 patients with ALS, including those who had lost their ability to speak as would be required by traditional research, with subsequent 1-week and 3-month retests providing additional data, and three new PRO items were developed for use in those patients in the most advanced stages of ALS. These ALSFRS-Extension items have subsequently been translated into other languages [17] and are used even by the Department of Veteran’s Affairs brain bank [18], demonstrating a path from patient experience to clinical research use in a relatively short space of time. Although this example does not represent a PRO instrument that has undergone US regulatory review, it does represent the relevance and importance of broad patient input and the potential feasibility of very quickly and efficiently assessing the impact of such input on the psychometric properties of the revised instrument. This also demonstrates that the use of “social media” is not merely limited to passive monitoring of otherwise purely social channels such as Facebook or Twitter [19], which users may feel are not appropriate channels for the sharing of medical information [20].
The naturalistic discussions that patients have with one another online in forums have been used to develop PROs in other conditions too, such as multiple sclerosis (MS). The MS Treatment Adherence Questionnaire [21] and the MS Rating Scale have also been translated into other languages, evaluated against clinical measurement in the case of the MS Rating Scale [22], and are being deployed in prospective phase IV studies (e.g., EMD-Serono’s “Adherence Trial with MS Lifelines® Services,” NCT01905527). As noted in the ALS example, this PRO instrument has not undergone US regulatory scrutiny but does offer a model of some of the advantages that use of social media may bring to PRO instrument development.
More recently, the Robert Wood Johnson Foundation–funded “Open Research Exchange” (ORE) has been piloted to allow researchers from external institutions to develop, iterate upon, and deploy new PRO measures using volunteers from social media to develop and provide rapid feedback on PRO items through five distinct phases: concept elicitation, feedback, test, retest, and follow-up [23]. By selecting participants from among a wider pool, researchers can balance their sample to counteract the biases inherent in an online platform. Because the marginal resources needed to recruit 100 participants are not much higher than those needed to recruit 50 participants, the process can be more efficient than interviewer-led concept elicitation. For example, the development of an Insomnia Impact Questionnaire consisted of a concept elicitation phase with 16,331 words from 75 patients. This was used to develop items for a questionnaire that gathered item-level feedback from 54 selected participants, which yielded qualitative and quantitative data for improvement to the items. The refined questionnaire was then fielded to over 1300 patients for a psychometric test phase to establish scale performance. Each phase (concept elicitation, feedback, and test) took just 7 days. The software and the engaged population ensure that it is also relatively straightforward to automate test-retest at different intervals or the use of different response options with sample sizes adequate for statistical testing. In a second ORE example, the Treatment Burden Questionnaire, originally developed in French, underwent an English language pretest in 200 patients from around the world in less than a month, followed by a 610 patient psychometric test and a 282 patient retest, all in less than 2 months [24]. All PROs developed on the ORE platform will remain free for anybody to use or build upon, using Creative Commons licensing arrangements. It is hoped that this technology can rapidly accelerate and improve upon the range of PROs available [23].
Limitations of Social Media for PRO Concept Elicitation
Several concerns with using social media to collect concept elicitation data were noted by the panel. The primary concern expressed by the panelists related to the uncertainty of the characteristics of the respondent, extending from demographic characteristics to diagnosis. In conducting face-to-face interviews or focus groups, researchers are usually able to seek confirmation of diagnosis via physician verification through a signed waiver from the patient for access to health information, though this has the effect of biasing respondents toward those comfortable with this level of sharing. In contrast, a well-known maxim states that “On the Internet, nobody knows you’re a dog” [25]. This situation is changing though as online providers such as MediGuard undergo similar physician verification processes to their offline colleagues [26], or increasingly use electronic medical records to validate that patients are who they say they are [27]. In a concept elicitation study of 50 patients with chronic lymphocytic leukemia, 80% of the patients agreed that researchers could contact their physicians to verify their diagnosis[28]. This approach, however, might have downsides too, in that patients may feel less comfortable participating or sharing their experiences of embarrassing or stigmatized conditions, knowing that their identity will be known and verified.
Next, it was noted that recruiting the right patients is important regardless of the platform. For the development of instruments intended for use in medical product development, the respondents should adequately represent the targeted clinical trial population because content validity is context specific. In uncommon conditions such as organ transplantation, social media can have a wide reach; as many as 1% of all those receiving an organ transplant in 2009 shared their medical data on PatientsLikeMe [29], and Facebook was recently used to identify pediatric transplant patients who had been lost to follow-up by a clinical center [30]. Participants in social media currently tend to be biased, however, toward younger patients, those who are female, or those with higher levels of education relative to clinical populations [1]. With a sufficiently large sample, this issue can be addressed, however, by taking a stratified sampling approach in the recruitment phase.
Another concern related to the definition of social media, in that there are different types of digital media that may be used for collecting qualitative data. For instance, contacting patients who have volunteered to participate in a medical community associated with research (such as Genomera or 23andMe) is different from attempting to recruit people through Facebook who might not be expecting to be targeted on the basis of their profile data or medical charity pages they have “liked.” Some forms of social media such as Twitter are inherently limited to brief (140 character) communications, and it is unclear to what extent the abbreviated and asynchronous nature of online communication will hold up to the nuanced probing of a trained qualitative interviewer. Branthwaite and Pattterson[19] suggest that social media fails to capitalize on the distinguishing features of qualitative research: having a conversation (rather than an instantaneous static expression of an individual in a given moment), active listening (rather than a software textbox unable to probe the “space between words”), and an interactive “merging of minds” or “rapport” (rather than an audience of “followers”). Research will be needed to map out the advantages and disadvantages of each type of social media so that they may be more clearly understood moving forward, and if social media is going to have a place in the researchers toolkit, it is possible that these systems will adapt to overcome the limitations identified.
Conclusions
Overall, the panelists indicated that social media holds great potential as a means of recruitment of patients for qualitative research and collecting data to support the content validity of PRO measures used in product labeling as long as best practices are applied. The efforts of professional bodies such as ISPOR to embrace and validate other newer digital techniques such as e-PRO [1] provide reassurance that research required to advance our understanding can be achieved. Specific concerns that need to be addressed include ways to have more confidence in the characteristics of the participant, especially diagnostic criteria, and more research is needed to understand the specific strengths and weaknesses of different forms of social media. It may be the case that a mixture of both face-to-face and online data collection offers advantages over either methodology alone.
Source of financial support: The authors have no other financial relationships to disclose.
References
C. Acquadro, R. Berzon, D. Dubois, et al.
Incorporating the patient’s perspective into drug development and communication: an ad hoc task force report of the Patient-Reported Outcomes (PRO) Harmonization Group meeting at the Food and Drug Administration, February 16, 2001
Value Health, 6 (2003), pp. 522–531
Article | PDF (142 K) | View Record in Scopus | Full Text via CrossRef | Citing articles (214)
D.L. Patrick, L.B. Burke, J.H. Powers, et al.
Patient-reported outcomes to support medical product labeling claims: FDA perspective
Value Health, 10 (Suppl) (2007), pp. S125–S137
Article | PDF (211 K) | View Record in Scopus | Citing articles (173)
P. Fayers, D. Machin
Quality of Life: The Assessment
Analysis and Interpretation of Patient-Reported Outcomes (2007)
U.S. Department of Health and Human Services, Food and Drug Administration
Guidance for industry: patient-reported outcome measures—use in medical product development to support labeling claims
Fed Reg, 74 (2009), pp. 65132–65133
M. Rothman, L. Burke, P. Erickson, et al.
Use of existing patient-reported outcome (PRO) instruments and their modification: the ISPOR Good Research Practices for Evaluating and Documenting Content Validity for the Use of Existing Instruments and Their Modification PRO Task Force Report
Value Health, 12 (2009), pp. 1075–1083
Article | PDF (109 K) | View Record in Scopus | Full Text via CrossRef | Citing articles (64)
Rosen R. Development of patient reported outcome (PRO) measures: how feasible is a PRO for asymptomatic PCa treatment? In: FDA Public Workshop: Clinical Trial Design Issues—Drug and Device Development for Localized Prostate Cancer; San Diego, CA; May 5, 2013
Hayes R. Patient-reported outcome (PRO) instruments as drug development tools (session III). In: Consensus Science: New Tools and Tactics for Next-Gen Drug Development. Washington, DC; 2011
S. Fox
The social life of health information
Pew Research Center’s Internet & American Life ProjectPew Research Center, Washington, DC (2011)
Fox S, Duggan M. Health online 2013. Available from: http://www.pewinternet.org/2013/01/15/health-online-2013/.
C. Young
Community management that works: how to build and sustain a thriving online health community
J Med Internet Res, 15 (2013), p. e119
F.J. Grajales, S. Sheps, K. Ho, et al.
Social media: a review and tutorial of applications in medicine and health care
J Med Internet Res, 16 (2014), p. e13
S.L. Acaster, D. Wild
PMC65: a novel comparison of qualitative data sources: content analysis of semi-structured patient interviews versus weblogs (Blogs)
Value Health, 12 (2009), p. A31
Article | PDF (87 K) | View Record in Scopus | Citing articles (1)
L. Yen, E. Flood, K. Beusterien, et al.
PGI25—development of patient-reported outcomes (Pro) and observer-reported outcomes (Obsro) measures for pediatric ulcerative colitis: concept elicitation findings
Value Health, 16 (2013), p. A216
Article | PDF (82 K) | View Record in Scopus | Citing articles (1)
D.L. Patrick, L.B. Burke, C.J. Gwaltney, et al.
Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 1—eliciting concepts for a new PRO instrument
Value Health, 14 (2011), pp. 967–977
Article | PDF (1333 K) | View Record in Scopus | Citing articles (74)
G.M. Weber, K.D. Mandl, I.S. Kohane
Finding the missing link for big biomedical data
JAMA, 311 (2014), pp. 2479–2480
View Record in Scopus | Citing articles (27)
J.M. Cedarbaum, N. Stambler, E. Malta, C. Fuller
The ALSFRS-R: a revised ALS functional rating scale that incorporates assessments of respiratory function
J Neurol Sci, 169 (1999), pp. 13–21
Article | PDF (151 K) | View Record in Scopus | Citing articles (721)
S. Abdulla, S. Vielhaber, S. Körner, et al.
Validation of the German version of the extended ALS functional rating scale as a patient-reported outcome measure
J Neurol, 260 (2013), pp. 2242–2255
View Record in Scopus | Full Text via CrossRef | Citing articles (6)
C.B. Brady, K.T. Trevor, T.D. Stein, et al.
The Department of Veterans Affairs Biorepository Brain Bank: a national resource for amyotrophic lateral sclerosis research
Amyotroph Lateral Scler Frontotemporal Degener, 14 (2013), pp. 591–597
View Record in Scopus | Full Text via CrossRef | Citing articles (3)
A. Branthwaite, S. Patterson
The power of qualitative research in the era of social media
Qual Market Res, 14 (2011), pp. 430–440
View Record in Scopus | Full Text via CrossRef | Citing articles (11)
M. van der Velden, K. El Emam
“Not all my friends need to know”: a qualitative study of teenage patients, privacy, and social media
J Am Med Inform Assoc, 20 (2013), pp. 16–24
View Record in Scopus | Full Text via CrossRef | Citing articles (16)
P. Wicks, M. Massagli, A. Kulkarni, H. Dastani
Use of an online community to develop patient-reported outcome instruments: the Multiple Sclerosis Treatment Adherence Questionnaire (MS-TAQ)
J Med Internet Res, 13 (2011), p. e12
View Record in Scopus | Full Text via CrossRef | Citing articles (1)
R. Bove, E. Secor, B.C. Healy, et al.
Evaluation of an online platform for multiple sclerosis research: patient description, validation of severity scale, and exploration of BMI effects on disease course
PLoS One, 8 (2013), p. e59707
P. Wicks, B. Heywood, J. Heywood
Online platform to accelerate patient involvement in open instrument development
Qual Life Res, 22 (2013), p. 55
View Record in Scopus | Citing articles (3)
V.-T. Tran, M. Harrington, V.M. Montori, et al.
Adaptation and validation of the Treatment Burden Questionnaire (TBQ) in English using an internet platform
BMC Med, 12 (2014), p. 109
P. Steiner
On the Internet, nobody knows you’re a dog
New Yorker, 69 (1993), p. 61
View Record in Scopus | Citing articles (1)
E. Cascade, D. Ford, M. Nixon, et al.
PCV124 Combining electronic health records and patient-reported information in the UK: preliminary results from wasps (Wales Sail + Pro Study)
Value Health, 15 (2012), pp. A134–A135
Article | PDF (70 K) | View Record in Scopus | Citing articles (1)
E. Cascade, P. Marr, M. Winslow, D. Tuttle
PMS94 Integrating patient-reported outcomes (PRO) and medical record data (MR) in observational study designs: results from a direct-to-patient pilot study in gout
Value Health, 15 (2012) A51–A51
K.P. McCarrier, S. Bull, K.F. Simacek, et al.
Online social networks-based qualitative research to identify patient-relevant concepts in chronic lymphocytic leukemia
Value Health, 17 (2014), p. A196
Article | PDF (61 K) | View Record in Scopus | Citing articles (1)
P. Wicks, K.A. Sulham, A. Gnanasakthy
Quality of life in organ transplant recipients participating in an online transplant community
Patient, 7 (2014), pp. 73–84
View Record in Scopus | Full Text via CrossRef | Citing articles (3)
S. Mohammad, L. Hormaza, K. Neighbors, et al.
Health status in young adults two decades after pediatric liver transplantation
Am J Transplant, 12 (2012), pp. 1486–1495
View Record in Scopus | Full Text via CrossRef | Citing articles (17)
Copyright © 2015 International Society for Pharmacoeconomics and Outcomes Research (ISPOR). Published by Elsevier Inc.