Delphi study (objective 2b)

These are the notes that I took while developing my ideas for the Delphi study. The final version of each round is presented immediately below, with earlier notes right at the end. The primary source of this document in this early stage is from this document on the Delphi method, the specific details of which I haven't been able to find yet.

Round 1

Delphi study (round one) Clinicians, clinical supervisors


The UWC physiotherapy department is in the process of a curriculum review with the intention of modifying the curriculum to take into account an evolving understanding of teaching and learning practices. However, in order to make curricular changes we would first like to develop an understanding of clinicians' perceptions of the challenges involved in the clinical education of undergraduate healthcare professional students. Much research has been conducted on the requisite knowledge and skills development enabling the healthcare student to apply knowledge in a practical context i.e. “to do”. And, even though there is a good understanding of the need for additional professional attributes e.g. reflection, empathy and leadership, these are rarely explicitly covered in the undergraduate curriculum. This study is an attempt to identify attributes that will contribute to the development of a more capable graduate.

There is an increasing emphasis on South African higher education institutions developing in their graduates a set of attributes that “… build the foundations for lifelong learning, including critical, analytical, problem-solving and communication skills, as well as the ability to deal with change and diversity…” (UWC Graduate Attributes, 2010). The aim of this Delphi study is therefore to determine the attributes of a competent and capable healthcare graduate that impact on patient-centred outcomes that go beyond function. The literature differentiates between training for competence (i.e. knowledge and technical skill) and training for capability (see box below).

Competence: what individuals know or are able to do in terms of knowledge, skills, attitude Capability: extent to which individuals can adapt to change, generate new knowledge, and continue to improve their performance (Fraser & Greenhalgh, 2011)

HPCSA1 documents focus on content to be covered, SASP2 and NQF / HEQF3 documents focus on what a graduate should be able to do (i.e. competencies). However, there is little that deals with how we want our students and new graduates to be. In the dynamic and complex clinical environment, we need to ensure that our teaching practices facilitate the development of competent and capable healthcare practitioners.

This Delphi study seeks to answer the following questions: Round 1: What do we want our students / graduates “to be”, rather than what we want them “to do”? Round 2: What teaching strategies can we use in order to develop the attributes identified in Round 1? Round 3: What technologies can be used to enhance the teaching strategies identified in Round 2? Round 4 (dependent): What challenges can we foresee in terms of integrating the identified strategies and technologies in a South African clinical education context?

The significance of this study is that it will: Identify the challenges faced in clinical education in terms of developing attributes that go beyond knowledge and skill Suggest teaching strategies that may impact on addressing those challenges Highlight blended (i.e. technology-mediated) approaches to teaching and learning that may help overcome the challenges of those teaching strategies


You have been purposively selected to participate in this study because of your clinical and educational experience in the training of healthcare professionals. Your contact details were obtained by virtue of your affiliation with Your participation in this Delphi study will help us to develop a better understanding of the current challenges in clinical education, with particular reference to developing capability in addition to competence. Your input would provide a valuable perspective on the types of attributes or characteristics that have an impact on patient-centred outcomes. Specifically, I'd like to know what you think makes a “good” clinician, beyond knowledge and skills.

Procedure and time frame

The survey will be developed online using Google Docs and the link to the survey emailed to a purposively selected initial sample, using snowballing from that sample to identify and include additional participants. The participants will be asked to complete the survey within two weeks. A reminder will be sent out after two weeks, informing the panel that the survey will be closed within one week. Data from each round will be analysed for one week following closure of the questionnaire, which will inform the development of the next round of questions. Here is the timeline for the completion of the Delphi study by the end of 2011:

10 October: send out first questionnaire for Delphi 1 24 October: send out reminders to participants 28 October: close online survey for Delphi 1 28-31 October: analyse responses and use to derive questionnaire for Delphi 2 01 November: send out questionnaires for Delphi 2 14 November: send out reminder for Delphi 2 18 November: close online survey for Delphi 2 19-25: analyse responses and derive questionnaire for Delphi 3 28 November: send out questionnaires for Delphi 3 (final round) 12 December: send out reminder for Delphi 3 16 December: close online survey for Delphi 3 19-31 December: analyse results from final round and use all 3 data sets to identify: Challenges faced in clinical education Teaching strategies to address those challenges Appropriate technology that can enhance those teaching strategies If appropriate, a fourth round of questionnaires can be sent to the panel for Delphi 2, in order to identify challenges implementing the technology-mediated teaching strategies highlighted in Delphi 3

Data analysis

The survey will consist mainly of open-ended questions and will therefore be analysed qualitatively. The researchers will independently categorise responses using themes that emerge from the data, comparing their results until consensus is reached.

Ethical considerations

This study has received ethical clearance from the University of the Western Cape Ethics Committee (project registration number: 09/8/16). You are not required to participate in this research project and non-participation will have no negative effects on you. All of your responses will be anonymous. You may decide at any stage to withdraw from the project, and have your responses removed from the database. If you choose to participate in the first round, you are under no obligation to participate in subsequent rounds.

With regards to these considerations, please ensure that you complete the items in the box below before submitting your responses.

I have no further questions about this study. The information has been explained to me to my satisfaction I consent to participate in this study I do NOT want to be included in subsequent rounds of this study

If you have any questions before continuing, please contact the researcher at

References Fraser, S. W., & Greenhalgh, T. (2001). Coping with complexity: educating for capability. BMJ, 323(October), 799-803 University of the Western Cape (2010). The Development of Graduate Attributes at UWC

Questions 1. How many years of clinical experience do you have? 2. Do you supervise healthcare students in clinical settings? 3. If you answered Yes to Question 2, how many years of experience do you have in supervising healthcare students? 4. What do you believe makes a competent healthcare professional i.e. what knowledge and technical skills are important? 5. What do you believe makes a capable healthcare professional i.e. what values and qualities, other than knowledge and technical skills, are important? 6. What are the professional attributes that you believe have the biggest impact on patient care? 7. What are the personal values that you think impact on professional practice and patient care? 8. What do you think is the role of professional ethics and ethical reasoning in patient care? 9. What does it take “to be” a healthcare professional i.e. what sort of character or disposition makes a good healthcare professional? 10. How can healthcare students be better prepared to deal with the complexity of clinical environments experienced during their training? 11. What do you believe are the main challenges faced by: 1. Second year students who have very limited patient contact? 2. Third year students who are only beginning work with patients regularly? 3. Fourth year students who are moving towards the end of their studies? 4. New graduates who are expected to practice independently? 12. What are the main challenges you have experienced when working with / supervising undergraduate healthcare students? 13. What would you say is the main area that healthcare students and new graduates need additional opportunities for development? 14. Can you think of any potential solutions to the problems you have highlighted in Question 10, 11 and 12? 15. Do you believe that new healthcare graduates should be allowed to practice independently after graduation? 16. What is the reason for your response in Question 14? 17. Do you know of anyone else who you think would be useful to include in this survey? Please add their name and email address in the text field below. 18. What is your profession? 19. What is the highest degree you have obtained? 20. What is your current position? 21. Are you Male Female? 22. How old are you? 23. Where do you live (City and Country)

Thank you for participating in this survey.


The aim of this study is to identify a set of characteristics that we value in professionals, and then how to develop those characteristics within a blended learning module.

While doing this component, keep in mind the results of the survey and systematic review, as well as the document analysis, and PHT203.

What do I want to learn from the panels? What are the inclusion criteria for being on a panel? They're actually just people I know about. It's not a database I've queried. The sample was purposively selected.

Do I mention that I decided not to include patients?

Does this leave out practical modules? I don't think so. Practical modules can still include an online component, especially if we consider that an improvement in communication should be the emphasis.

What is blended learning? Does everyone on the panel need to know what this is?

Who will I ask to be on the panels? Clinical educators, educational technologists, clinicians(?), supervisors(?)

Do all questions need to be relevant to everyone on the panel? If so, then the questions will need to be more generic, which may be a problem.

Look at CanMEDS framework for the qualities of a health professional. Each round should include a short demographic section for participants to establish credibility. Each participant should also be asked to sign a consent form

Round 1 (clinicians, clinical educators) Focus: What makes a “good” physiotherapist, outside the domain of competence (i.e. technical skill, knowledge)? What is lacking in current students / new graduates? What characteristics that impact on patient outcomes? What are the greatest challenges faced by students and new graduates?

What are the characteristics of a good physiotherapist? Should these be derived from the literature (e.g. CanMEDS roles)? If everyone chooses their own characteristics, we'll just end up with a lot of different ones (this is good…there doesn't need to be consensus). What am I looking for? Technical skill Ability to reason through problems Empathy Compassion Respect

CanMEDS roles (each of these roles has a set of key competencies that are enabled in certain ways): Medical experts apply medical knowledge, clinical skills and professional attitudes in their provision of patient-centred care Communicators facilitate the relationship with patients before, during and after the encounter Collaborators work within a team to achieve optimal patient care Managers are part of organisations that work to create sustainable practices, allocate resources, contribute to effective systems Health advocates use their expertise and influence to advance the health of patients, communities and populations Scholars demonstrate a commitment to reflective learning, and the creation, dissemination, application and translation of medical knowledge Professionals are committed to the health and well-being of individuals and society through ethical practice, profession-led regulation, and high standards of behaviour

Round 2 (clinical educators. Clinical supervisors, lecturers) Focus: How can you teach to develop the above characteristics? What teaching strategies do you use? What learning activities can you suggest (not the format)? Can you add to the characteristics already identified?

The questions in the following rounds are (mainly) defined by responses from the first round. It's also important to emphasise the role of assessment as well as teaching. This is related to constructive alignment of the module (see the document analysis). Do you agree that the characteristics described are important for physiotherapy graduates? Can you think of any other characteristics that are not included here? What teaching methods / formats can you use to develop the above characteristics? How would you assess whether or not these characteristics have been developed? What teaching methods would you apply to develop the following domains: Intellectual ability / knowledge (recall, comprehension, application, analysis, evaluation, synthesis) How do level descriptors fit into this: Level 5: understands key terms, concepts, facts, principles, rules and theories Level 6: able to apply key terms, etc. of that field, as well as how knowledge relates to other fields Level 7: able to apply and evaluate the key terms, etc. and how that relates to other fields Attitudes, approaches and values (behaviour, teamwork, personal qualities, professionalism, motivation) Skills (technical skill, communication, time management, problem-solving, self-learning) How would you write these characteristics into your module learning outcomes?

Round 3 (educational technologists, clinicians who use technology, clinicians who teach with technology) Focus: What tools can be used to develop the above characteristics? Is there anything you can add to these characteristics? What educational technologies are available that might be used to facilitate the teaching methods mentioned above? What are the features that are most important? Cost, intuitive user interface, devices used to access? How would you facilitate the development of the roles / characteristics mentioned above?

Round 4 Take what is learned from the previous 3 rounds and go back to the second panel with a summary: These are the characteristics of new graduates that are important We can develop these characteristics using the following methods And the following tools What are the challenges you think might arise from the use of these methods and tools? What do you think of using these tools to teach in this way in order to develop these characteristics?

The Delphi study will use as its basis the modules that were identified during the document analysis. These modules will be presented to the panel who will seek to determine which of them is most suitable to be adapted to a blended learning approach.


The Delphi method is used for the “…systematic solicitation and collation of judgments on a particular topic through a set of carefully designed sequential questionnaires interspersed with summarized information and feedback of opinions derived from earlier responses” (Delbecq, Van de Ven & Gustafson, 1975, p. 10). The fact that the technique uses questionnaires also means that participants don't have to share geographical space.

The technique is useful to achieve the following types of objectives (Delbecq et al., 1975):

  • To develop a range of possible programme alternatives
  • To explore underlying assumptions or information leading to different judgments
  • To seek out information to try and generate a consensus
  • To correlate informed judgments on a topic spanning a range of disciplines
  • To educate the response group around the diverse, interrelated aspects of a topic


  1. Develop a set of questionnaires (3 or 4 in total, each designed around the responses of the previous one)
  2. Identify, select and contact respondents (a nomination process of individuals with expertise)
  3. Select a sample size (30 can be considered an upper limit, usually 10-20 is appropriate)
  4. Develop and send the first questionnaire
  5. Analyse the results
  6. Develop and send the second questionnaire (this questionnaire is based in large part on the responses from the first questionnaire)
  7. Analyse the results
  8. Develop and send the third questionnaire (based largely on the responses to the second questionnaire, it also asks participants to confirm themes / categories / ideas from the previous questionnaires)
  9. Analyse the results to determine if there is a consensus
  10. Complete a report of the final results


  • Delbecq, Van de Ven & Gustafson (1975). Group techniques for program planning: A guide to nominal group and Delphi processes, Chapter 3, Methodology - Delphi study. ISBN 0673075915
phd_delphi.txt · Last modified: 2011/10/10 10:23 by Michael Rowe