Open Access Open Access  Restricted Access Subscription Access

Participant experiences with a new online modified-Delphi approach for engaging patients and caregivers in developing clinical guidelines

Courtney Kiera Armstrong, Sean Grant, Kathi Kinnett, Brian Denger, Ann Martin, Ian Coulter, Marika Booth, Dmitry Khodyakov


Background: Patient engagement in clinical practice guideline (CPG) development has increased significantly in recent years. However, only a few patients and caregivers join CPG development groups.

Objective: To describe participant experiences with a novel online, scalable approach for patient and caregiver engagement in CPG development.

Design: We developed and tested the RAND/PPMD Patient-Centeredness Method (RPM), a novel online modified-Delphi approach to patient engagement in CPG development that consists of an optional idea generation round and two rating rounds interspersed with an online discussion round.

Setting and Participants: Using the online ExpertLens™ system, we ran 2 concurrent panels of patients and caregivers of  individuals with Duchenne Muscular Dystrophy (DMD).

Main Outcome Measures: We surveyed all 95 panel participants about their participation and satisfaction with the process. We also conducted telephone interviews with 25 participants.

Results: Participants expressed satisfaction with various ExpertLens™ features, noting that the system fostered lively interaction among them. Panelists also appreciated participating in an educational, interactive and convenient discussion forum that allowed them to share their opinions on the importance and acceptability of different recommendations. The RPM was viewed as empowering by patients and their caregivers who felt it would be useful for CPG developers.

Discussion and Conclusion: The results of our study show the overall participant satisfaction with a novel, scalable, online approach to engaging patients and caregivers in CPG development, which allows them to share their perspectives and lived experiences using a rigorous, systematic and iterative way that is similar to how clinicians provide their input.


Duchenne Muscular Dystrophy, ExpertLens™, guideline development, lived experience of illness, Modified-Delphi method, online engagement, patient/caregiver perspectives, patient engagement, person-centered healthcare, RAND/PPMD Patient-Centeredness Method

Full Text:



Kelson, M. (2001). Patient involvement in clinical guideline development–where are we now? Journal of Clinical Governance 9 (4) 169-174.

Schünemann, H.J., Fretheim, A. & Oxman, A.D. (2006). Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement. Health Research Policy and Systems 4 (1) 22.

Shekelle, P.G., Woolf, S.H., Eccles, M. & Grimshaw, J. (1999). Developing guidelines. British Medical Journal 318 (7183) 593-596.

Shekelle, P., Woolf, S., Grimshaw, J.M., Schünemann, H.J. & Eccles, M.P.J.I.S. (2012). Developing clinical practice guidelines: reviewing, reporting, and publishing guidelines; updating guidelines; and the emerging issues of enhancing guideline implementability and accounting for comorbid conditions in guideline development. Implementation Science 7 (1) 62.

Krahn, M. & Naglie, G. (2008). The next step in guideline development: incorporating patient preferences. Journal of the American Medical Association 300 (4) 436-438.

Armstrong, M.J., Mullins, C.D., Gronseth, G.S. Gagliardi, A.R.J. (2017). Recommendations for patient engagement in guideline development panels: A qualitative focus group study of guideline-naïve patients. PLoS One 12 (3) e0174329.

Armstrong, M.J., Rueda, J.D., Gronseth, G.S. & Mullins, C.D. (2017). Framework for enhancing clinical practice guidelines through continuous patient engagement. Health Expectations 20 (1) 3-10.

Légaré, F., Boivin, A., van der Weijden, T., Pakenham, C., Burgers, J., Légaré, J., St-Jacques, S. & Gagnon, S. (2011). Patient and public involvement in clinical practice guidelines: a knowledge synthesis of existing programs. Medical Decision Making 31 (6) E45-E74.

Steinberg, E., Greenfield, S., Wolman, D.M., Mancher, M. & Graham, R. (2011). Clinical practice guidelines we can trust. Washington DC: National Academies Press.

World Health Organization. (2012). WHO Handbook for Guideline Development. Geneva: World Health Organization.

Grant, S., Hazlewood, G.S., Peay, H.L., Lucas, A., Coulter, I., Fink, A. & Khodyakov, D. (2018). Practical Considerations for Using Online Methods to Engage Patients in Guideline Development. Patient 11 (2) 155-166.

Khodyakov, D., Denger, B., Grant, S., Kinnett, K., Armstrong, C., Martin, A., Peay, H., Coulter, I. & Hazlewood, G. (2019). The RAND/PPMD Patient-Centeredness Method: a novel online approach to engaging patients and their representatives in guideline development. European Journal for Person Centered Healthcare 7 (3) 470-475.

Birnkrant, D.J., Bushby, K., Bann, C.M. et al.. (2018). Diagnosis and management of Duchenne muscular dystrophy, part 1: diagnosis, and neuromuscular, rehabilitation, endocrine, and gastrointestinal and nutritional management. The Lancet Neurology 17 (3) 251-267.

Birnkrant, D.J., Bushby, K., Bann, C.M. et al. (2018). Diagnosis and management of Duchenne muscular dystrophy, part 2: respiratory, cardiac, bone health, and orthopaedic management. The Lancet Neurology 17 (4) 347-361.

McNally, E.M., Kaltman, J.R., Benson, D.W., et al. (2015). Contemporary cardiac issues in Duchenne muscular dystrophy. Circulation 131 (18) 1590-1598.

Bushby, K., Finkel, R., Birnkrant, D.J. et al. (2010). Diagnosis and management of Duchenne muscular dystrophy, part 1: diagnosis, and pharmacological and psychosocial management. The Lancet Neurology 9 (1) 77-93.

Bushby, K., Finkel, R., Birnkrant, D.J. et al. (2010). Diagnosis and management of Duchenne muscular dystrophy, part 2: implementation of multidisciplinary care. The Lancet Neurology 9 (2) 177-189.

Fitch, K., Bernstein, S.J., Aguilar, M.D., Burnand, B. & LaCalle, J.R. (2001). The RAND/UCLA appropriateness method user's manual. Santa Monica CA: RAND CORP.

Dalal, S.R., Khodyakov, D., Srinivasan, R., Straus, S.G. & Adams, J. (2011). ExpertLens: A system for eliciting opinions from a large pool of non-collocated experts with diverse knowledge. Technological Forecasting & Social Change 78 (8) 1426-1444.

Khodyakov, D., Hempel, S., Rubenstein, L., Shekelle, P., Foy, R., Salem-Schatz, S., O’Neill, S., Danz, M. & Dalal, S. (2011). Conducting online expert panels: a feasibility and experimental replicability study. BMC Medical Research Methodology 11 (1) 174.

Khodyakov, D., Grant, S., Meeker, D., Booth, M., Pacheco-Santivanez, N. & Kim, K.K. (2016). Comparative analysis of stakeholder experiences with an online approach to prioritizing patient-centered research topics. Journal of the American Medical Informatics Association 24 (3) 537-543.

Khodyakov, D., Kinnett, K., Grant, S., Lucas, A., Martin, A., Denger, B., Peay, H., Coulter, I. & Fink, A. (2017). Engaging Patients and Caregivers Managing Rare Diseases to Improve the Methods of Clinical Guideline Development: A Research Protocol. JMIR Research Protocols 6 (4) e57.

Denger, B., Kinnet, K., Martin, A., Grant, S., Armstrong, C. & Khodyakov, D. (2019). Patient and caregiver perspectives on guideline adherence: the case of endocrine and bone health recommendations for Duchenne muscular dystrophy. Orphanet Journal of Rare Diseases 14 (1) 205.

Godau, R. (2004). Qualitative data analysis software: MAXqda and MAXdictio. Qualitative Research Journal 4 (1) 66.

Khodyakov, D., Uscher-Pines, L., Lorick, S.A., Lindley, M.C., Shier, V. & Harris, K. (2014). A qualitative analysis of the impact of healthcare personnel influenza vaccination requirements in California. Vaccine 32 (25) 3082-3087.

Nilsen, E.S., Myrhaug, H.T., Johansen, M., Oliver, S. & Oxman, A.D. (2006). Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database of Systematic Reviews (3):CD004563.

Alonso-Coello, P., Oxman, A.D., Moberg, J., Brignardello-Petersen, R., Akl, E.A., Davoli, M., Treweek, S., Mustafa, R.A., Vandvik, P.O., Meerpohl, J., Guyatt, G.H., Schünemann, H.J. & GRADE Working Group. (2016). GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: Clinical Practice Guidelines. British Medical Journal 353, i2089.

Eccles, M.P., Grimshaw, J.M., Shekelle, P., Schünemann, H.J. & Woolf, S. (2012). Developing clinical practice guidelines: target audiences, identifying topics for guidelines, guideline group composition and functioning and conflicts of interest. Implementation Science 7 (1) 60.



  • There are currently no refbacks.