A REFLECTION ON THE USE OF THE DELPHI TECHNIQUE IN NURSING

ABSTRACT

Objective:

This article aims to reflect on the Delphi technique, a systematic process that uses the collective opinion of a group of experts to reach consensus on specific issues. Focusing on the field of nursing, we examine in a reflective manner the different modalities of this technique and its stages.

Method:

This is a theoretical reflection on the use of the Delphi technique, listing its different forms and conducting a critical evaluation of the method throughout its different stages.

Results:

Different variants of the technique are observed in the available literature, namely Traditional Delphi, Modified Delphi, e-Delphi, and the RAND/UCLA method, accompanied by epistemological and methodological changes in the traditional understanding of the Delphi technique. Despite the variability of formats, the technique should include three principles: the panel of experts, the anonymity of the responses, and the interactive process of controlled feedback. Throughout the article, a reflection is made on its stages, justification of the study area, expert panel, characteristics of the rounds, and termination criteria.

Conclusion:

We suggest our evaluation criteria so that authors, researchers, and reviewers can analyze the use of the Delphi technique in nursing research. It is also recommended that clear guidelines be created for presenting studies using the Delphi technique in the health area, as there are other established guidelines for other types of methods.

DESCRIPTORS:
Delphi Technique; Consensus; Qualitative Research; Research; Nursing

RESUMO

Objetivo:

refletir sobre a técnica de Delphi, um processo sistemático que utiliza a opinião coletiva de um grupo de especialistas para alcançar consenso sobre determinados assuntos. Focalizando especificamente na área da enfermagem, nossa intenção é examinar, de forma reflexiva, as diferentes modalidades dessa técnica e suas etapas.

Método:

reflexão teórica sobre a utilização da técnica Delphi, elencando suas diferentes formas e realizando uma avaliação crítica do método ao longo das suas diferentes etapas.

Resultados:

observa-se na literatura disponível diferentes variantes da técnica, designadamente o Delphi Tradicional, o Delphi modificado, o e-Delphi e o método RAND/UCLA, acompanhados por mudanças epistemológicas e metodológicas no entendimento tradicional da técnica de Delphi. Apesar da variabilidade de formatos, a técnica deve incluir três princípios: o painel de peritos, o anonimato das respostas e o processo interativo de retroalimentação controlada. Ao longo do artigo, é realizada uma reflexão sobre as suas etapas, justificação da área em estudo, painel de peritos, características das rondas e critérios de término.

Conclusão:

Sugerimos nossos critérios de avaliação para que autores, pesquisadores e revisores possam analisar a utilização da técnica de Delphi na pesquisa em enfermagem. Recomendada-se ainda a criação de diretrizes claras para a apresentação de estudos com a utilização do Delphi na área da saúde, assim como existem outras orientações estabelecidas para outro tipo de métodos.

DESCRITORES:
Técnica Delphi; Consenso; Pesquisa Qualitativa; Pesquisa; Enfermagem.

RESUMEN

Objetivo:

reflexionar sobre la técnica de Delphi, un proceso sistemático que utiliza la opinión colectiva de un grupo de especialistas para alcanzar consenso sobre determinados asuntos. Focalizando específicamente en el área de la enfermería, nuestra intención es examinar, de forma reflexiva, las diferentes modalidades de esa técnica y sus etapas.

Método:

reflexión teórica sobre la utilización de la técnica Delphi, enumerando sus diferentes formas y realizando una evaluación crítica del método a lo largo de sus diferentes etapas.

Resultados:

en la literatura disponible se observan diferentes variantes de la técnica, a saber: el Delphi Tradicional, el Delphi modificado, el e-Delphi y el método RAND/UCLA, acompañados por mudanzas epistemológicas y metodológicas en el entendimiento tradicional de la técnica de Delphi. A pesar de la variabilidad de formatos, la técnica debe incluir tres principios: el panel de peritos, el anonimato de las respuestas y el proceso interactivo de retroalimentación controlada. A lo largo del artículo, es realizada una reflexión sobre sus etapas, justificación del área en estudio, panel de peritos, características de las rondas y criterios de término.

Conclusión:

Sugerimos nuestros criterios de evaluación para que autores, pesquisadores y revisores puedan analizar la utilización de la técnica de Delphi en la pesquisa en enfermería. Se recomienda la creación de directrices claras para la presentación de estudios con la utilización del Delphi en el área de la salud, así como existen otras orientaciones establecidas para otro tipo de métodos.

DESCRIPTORES:
Técnica Delphi; Consenso; Pesquisa Cualitativa; Pesquisa; Enfermería.

INTRODUCTION

Consensus-generating techniques are designed to answer a research question. In this methodology, knowledge is co-created by incorporating multiple perspectives and types of knowledge.1 In 1972, these authors emphasized that the consensus technique provided a way to process a large number of ideas, offering a way to bypass organizational restrictions on idea creation, encouraging maximum participation from everyone to solve problems, and providing a way to select good ideas.2 The consensus technique is referenced in 1970 in an activity of the US Army in generating the best ideas on how the army should be organized and equipped. Using this strategy, the institute obtained more than 250 ideas.2 In this particular case, the request for expert opinion has been recognized as a reliable data collection method.1 Thus, consensus techniques have been widely used in health and nursing research to achieve a common understanding of important issues and to inform evidence-based clinical decisions. 3^-6

Currently, consensus-generating methodologies through expert opinion are essentially developed through Delphi techniques or their variants.1 The Delphi technique is a systematic process that uses the collective opinion of the members of a panel.7 The technique implies a group decision-making process characterized by each group member presenting their ideas anonymously but never face-to-face and over several rounds.

The Delphi technique was first developed in the 1950s by Norman Dalkey and Olaf Helmer at the RAND Corporation in an attempt to gain consensus from experts.8 The term "Delphi" refers to Greek mythology, alluding to the oracle of Delphi in Ancient Greece, where priests gave future predictions based on divine visions.9 The prophetic meaning, then understood as synonymous with good judgment on a certain issue, inspired the choice of the name at the beginning of the 1950s.3

The technique spread to various areas of research, notably health and nursing. Delphi, in its different variants, has played a fundamental role in the last decades in developing the orientation of best practices using collective intelligence where research is limited, ethically, and logistically difficult, or the evidence is contradictory.7

The relevance and objectives of the Delphi techniques differ among the various disciplines. Although Delphi techniques are primarily used in the context of technical and natural sciences to analyze future developments, they are often used in the health sciences to find consensus.6 This is because, after a slow initial acceptance in the health field, it is now a widely used technique to assess current knowledge, resolve controversies in management, formulate theoretical or methodological guidelines, develop evaluation tools indicators, formulate recommendations, or other topics.7

In the health sciences, researchers mainly use the Delphi technique when the available knowledge is incomplete or subject to uncertainties, as well as in situations where higher levels of evidence are not available or cannot be used. Throughout this article, we intend to reflect on the different modalities of this technique and its stages, focusing specifically on the field of nursing.

VARIANTS OF THE TECHNIQUE

Currently, there are several variants of the technique, such as Traditional Delphi, Modified Delphi, E-Delphi, and RAND/UCLA. The development of new variants was also accompanied by epistemological and methodological changes in the traditional understanding of Delphi.6 Despite this variability, the different manifestations are recognized as the Delphi technique as long as they include three fundamental characteristics: careful selection of a group of experts, anonymity in responses, and a controlled interactive feedback process.

Tradicional Delphi

The way the first round is elaborated determines the Delphi technique used. In this more qualitative approach, the questionnaire for the first round consists of a set of open-ended questions to guide the generation of ideas, present opinions, and obtain consensus. There is a uniquely qualitative context, ensuring that the panel members expose the complexities of a problem based on their understanding of the topic. However, given the complexity of the content that can be submitted and the amount of information that can be generated, this can lead to disinterest among participants and compromise the reliability and validity of the results3. In a review study conducted on the subject, of the 764 studies with Delphi variants included in this analysis, 329 (43.06%) were identified as having used Traditional Delphi1.

Modified Delphi

The Modified Delphi does not follow a uniform criterion, but in principle, a coordinating group acts as a facilitator of the process within the group, making communication more effective.7 The research coordinating group initially identifies the relevant issues for the study object. This determines the need to develop a prior instrument, which can be elaborated from a literature review, interviews, focus groups, or other forms of consultation.3 The first round may consist of direct analysis of the instrument items. However, the term “modified” in Delphi studies is discrepant and without any universally accepted criterion. The only common thing in the Modified Delphi technique is the active effort of the coordination group in generating consensus. The coordination group conducts prior research in the area of the problem to be investigated and subsequently focuses on achieving consensus among the panel members. However, this active participation of the coordination group may cause bias in the opinion of the expert members.7 In a review study conducted on the subject, of the 764 studies with Delphi variants included in this analysis: 426 (55.76%) were identified as using this technique1.

E-Delphi

The E-Delphi is a variation of the Delphi technique that uses information and communication technologies (ICT) to consult with the expert group. The E-Delphi allows the process to be carried out more quickly, accessibly, and efficiently, in addition to expanding the possibility of participation of experts from different geographical regions. The electronic Delphi survey (also called e-Delphi) helps in the global representation of panel members, saves time, and fixes the rounds of inquiry using technology without physical voting. This process involves selecting experts after researching for eligibility on the Web, with more invitations being sent by email to participate in the project.7 E-Delphi surveys can increase sample size and diversity by extrapolating forms to international borders. They can also lead to reduced administrative costs and time investments and a reduction in management burden through digital data collection, management of anonymous individual responses, and innovative communication of participants through automated email reminders by the system indicating to participants that they still need to complete the survey.10 However, the acceptance rate among experts may be low. Researchers consider this attrition rate of acceptance among experts higher during the invitation process.7 On the other hand, the lack of face-to-face interaction can be a deterrent for the recruitment and retention of participants. In some articles, this technique is categorized as Modified Delphi.13

RAND/UCLA

The RAND-UCLA appropriateness technique was developed in 1980 in the United States by the Research and Development Corporation (RAND) and the University of California - Los Angeles (UCLA).9^,11 In this technique, there are 2 independent groups: the core panel and the expert panel. The first group guides the second by preparing synthesized information obtained through a systematic literature review. In summary, the steps include a systematic review, selection of experts, generation of an assessment form, an assessment survey in the first phase, a face-to-face meeting where the panel members discuss areas of disagreement, final classifications and analysis of these classifications, and the drafting of a written summary of the areas where they agree.11

This technique has a specific purpose, using a systematic literature review as a basis, with a great impact on health recommendation guides.1 The RAND-UCLA has been used to develop clinical practice guidelines, disease classification systems, research agendas, and quality improvement interventions.11

In a review study conducted on the subject, of the 764 studies with Delphi variants included in this analysis, only 9 (1.18%) are described using this technique1. However, we believe that these results may be due to poor identification of the method in the reports, which indiscriminately portray the Delphi technique for any of its variants.

HOW TO USE THE DELPHI TECHNIQUE?

There are no defined standards for reporting Delphi studies in health, namely in nursing, unlike other research tools.7 There are also no validated quality parameters to evaluate Delphi studies. As these authors mention in these review studies, there is a need to improve the reporting of Delphi studies just as there are other guidelines like CONSORT®, STROBE®, and PRISMA®, including a standard set of quality indicators.4^,12^,13

Attempts have been made to identify quality parameters for conducting and evaluating Delphi studies, specifically in palliative care.4 The guidance for Delphi studies (CREDES- Conducting and REporting of DElphi Studies) is a popular guide developed for the use of the Delphi technique in palliative care. The authors recognized significant variations in the analyzed studies and proposed CREDES standards.4 This methodology includes various steps, namely: justification; planning and process; definition of consensus; input of information; bias prevention; interpretation and handling of results; external validation; objective and justification; expert panel; description of methods; procedure; definition and achievement of consensus; results; discussion of limitations; appropriateness of conclusions; and finally, publication and dissemination.4 This tool has not been validated for other areas of health nor universally accepted for conducting studies with the Delphi technique.

STAGES OF THE DELPHI TECHNIQUE

Nasa et al. propose that the technique should presuppose at least four stages (Figure 1 ).7 Therefore, we will describe the method based on these 4 stages.

Figure 1-
Quality Assessment of Delphi Studies. (Adapted from Nasa et al.7).

Study Area

A justification for choosing the Delphi technique as the most suitable method for the problem under study must be provided. When selecting this technique to answer a specific research question, it is important to keep in mind its constructivist nature.4 The use of Delphi is a resource for problematic areas where evidence is not available, knowledge is uncertain and incomplete, and expert judgment is better than individual opinion. The criteria used to identify the problematic area and the process followed should be documented.7

Expert Panel

The selection of panel members is undoubtedly the most important aspect of Delphi. There are no standard criteria used to define panel members. In this step, defined a priori, the composition and characteristics of the panel, the number of participants (participant flow diagram), techniques used to invite participants, and the geographical scope of Delphi should be specified.5

Is it possible to assume a priori that all group elements are “wise”?6 This is a common question at this stage because classifying panel members as 'experts' is very controversial.7 An expert can be defined as someone with knowledge and experience on a certain subject.8 However, it is difficult to measure experience quantitatively, so panel selection should follow predefined criteria.7 The criteria for expert selection and transparent information about the recruitment of the expert panel, including sociodemographic data and other details and information about expertise regarding the subject matter, should be communicated.4

Another issue is whether to use homogeneity or heterogeneity when choosing experts.

A homogeneous group may be more reliable for a specific study objective, being suitable for resolving unresolved issues of a specific problem.7 However, some authors suggest that heterogeneity in a decision group may lead to better performance than homogeneity, where a diverse panel helps to achieve a broader perspective and obtain consensus generalization.5^,6 Cognitive diversity in a group of experts can support innovative and creative discussion processes.6^,7

Regarding the appropriate size of the panel, it is described with enormous variability. According to these authors, in a mapping conducted on the topic, they mention that the panel size varied between 3 and 731 experts in the studies analyzed.6

The number of individuals needed to apply the technique is another source of disagreement among researchers, mentioning that this issue may be related to the topic investigated, the complexity of the problem, the selected approach, the available resources, and the variety of knowledge needed to achieve consensus.3

There is no standard size for panel members, and it can vary from 10 to 1000. (Usually between 10 and 100).7 A panel with a three-digit sample size is uncommon, so a two-digit number close to 30-50 is considered optimal in the final rounds for a homogeneous Delphi.6^,7

The appropriate size depends on the problem's complexity, the panel's homogeneity or heterogeneity, and the availability of resources. Some authors suggest that a minimum number of 10 members may be sufficient, mentioning that a larger quantity does not produce significant gains to reach consensus.7^,3^,9 However, resorting to as many participants as possible is advisable, reducing the risk of bias.3

Rounds

The strength of the process in the Delphi technique is the anonymity of the experts, the controlled feedback, and the different interactive rounds.7 The anonymous process avoids the social pressure of the group to act in accordance with a dominant view.4

The anonymity of members in the Delphi technique removes inherent bias, such as dominance and group conformity (defined as groupthink), observed in face-to-face group meetings where participants feel more comfortable providing opinions.7

"Controlled feedback" is another classic characteristic of the Delphi technique. It is termed "controlled" because the moderator decides on the feedback provided based on responses to items and open comments.7 After each round of research, the data obtained are analyzed and presented in an easily interpretable format for all. It has been recommended that feedback be included after each round, including qualitative comments and statistical measures.5^,6 Each participant should receive the panel's results, the participant's response, and a summary of all comments received. These data inform each participant of their position relative to the rest of the group, thereby assisting in decisions about responses during future Delphi rounds.5

Termination Criteria

The most common number of rounds in the Delphi technique is two or three12, although the total number of rounds can vary dramatically between studies.1^’7Some studies define a priori the number of rounds that will be executed.12 This aspect can be considered a bias since a fixed number of rounds without evaluating the stability of the results compromises statistical robustness.7 Therefore, as many rounds as necessary should be executed to reach the previously defined level of consensus. On the other hand, to satisfy the premise of feedback characteristic of the Delphi technique, a minimum of at least two rounds is necessary, even if the degree of agreement is reached at the first instance.

The termination criteria should be defined a priori, integrating not only the consensus criterion but also the stability criterion. The most common definition of consensus is percentage agreement, with 75% being the average threshold to define consensus.12 However, it is observed that percentage agreement varies widely, with results from 50% to 97% being arbitrarily selected.7

Studies show that beyond consensus percentages, the stability of responses should be a determinant for the Delphi's objective.7^,9^,12 Stability is defined as the consistency of responses between successive rounds.7 It is defined by the absence of new contributions and few changes in panel responses between rounds, represented by a low dispersion of responses.9 In other words, consensus can be present in unstable responses, and stability can be present without consensus; therefore, achieving response stability should be an appropriate termination criterion.7

The application of the Delphi technique in nursing lacks established standards and quality parameters. It requires a description of the technique variant used and an exhaustive description of how each stage was conducted.

CONCLUSION

After reflecting on the Delphi technique and its different variants, it is possible to conclude that the quality of the technique is based on its correct planning, the adequacy of the different stages, and the process by which consensus is identified. Despite the variability of formats, the method must include three principles to be considered Delphi, namely: expert panel, anonymity of responses, and the controlled interactive feedback process. When executed correctly and with methodological rigor, this technique can significantly contribute to expanding knowledge on a particular topic. Authors should strive to provide enough details about the technique they use, justifying each of the stages.

On the other hand, researchers should be aware of the technique's limitations, notably its difficulties in a process that can be time-consuming, with poor adherence from participants, and with possible loss of participants over the rounds. It suggests the need to create and validate standard quality parameters to evaluate the use of the Delphi technique in Nursing.

REFERENCES

• 1. Jandhyala R. Delphi, non-RAND modified Delphi, RAND/UCLA Appropriateness Method and A Novel Group Awareness and Consensus Methodology for Consensus Measurement: A Systematic Literature Review. Curr Med Res Opin [Internet]. 2020 [cited 2023 Jul 16];36(11):1873-87. Available from: https://doi.org/10.1080/03007995.2020
  » https://doi.org/10.1080/03007995.2020
• 2. Taylor CW. Panel Consensus Technique: A New Approach to Decisionmaking. J Creative Behavior [Internet]. 1972 [cited 2023 Jul 16];6(3):187. Available from: https://doi.org/10.1002/j.2162-6057.1972.tb00928.x
  » https://doi.org/10.1002/j.2162-6057.1972.tb00928.x
• 3. Rocha- FilhoC, Cardoso T, Dewulf N. Método e-delphi modificado: um guia para validação de instrumentos avaliativos na área da saúde [Internet]. Curitiba, (PR)BR: Brazil Publishing; 2019 [cited 2023 Jul 16]. Available from: https://doi.org/10.31012/978-65-5016-268-9
  » https://doi.org/10.31012/978-65-5016-268-9
• 4. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DELphi Studies (CREDES) in Palliative Care: Recommendations Based on a Methodological Systematic Review. Palliative Med [Internet]. 2017 [cited 2023 Jul 16];31(8):684-706. Available from: https://doi.org/10.1177/0269216317690685
  » https://doi.org/10.1177/0269216317690685
• 5. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. PLoS One [Internet]. 2011 [cited 2023 Jul 16];6(6):e20476. Available from: https://doi.org/10.1371/journal.pone.0020476
  » https://doi.org/10.1371/journal.pone.0020476
• 6. Niederberger M, Spranger J. Delphi Technique in Health Sciences: A Map. Front Public Health [Internet]. 2020 [cited 2023 Jul 16];8:457. Available from: https://doi.org/10.3389/fpubh.2020.00457
  » https://doi.org/10.3389/fpubh.2020.00457
• 7. Nasa P, Jain R, Juneja D. Delphi Methodology in Healthcare Research: How to Decide Its Appropriateness. World J Methodol [Internet]. 2021 [cited 2023 Jul 16];11(4):116-29. Available from: https://doi.org/10.5662/wjm.v11.i4.116
  » https://doi.org/10.5662/wjm.v11.i4.116
• 8. Barrett D, Heale R. What are Delphi Studies? Evidence-Based Nursing [Internet]. 2020 [cited 2023 Jul 16];23(3):68-9. https://doi.org/10.1136/ebnurs-2020-103303
  » https://doi.org/10.1136/ebnurs-2020-103303
• 9. Oliveira C, Tavares J, Leão R, Filho V, Andrade F. Delphi, técnica de validação para pesquisas na pandemia. In: Costa ACMSF, Souza RJC, Barros MLCMGR, editors. Internacional Saúde Única (Interface Mundial) [Internet]. 2nd ed. Recife, (PE)BR: Even3 Publicações; 2020 [cited 2023 Jul 16]. Available from: https://doi.org/10.29327/icidsuim2020.305828
  » https://doi.org/10.29327/icidsuim2020.305828
• 10. Hall DA, Smith H, Heffernan E, Fackrell K; Core Outcome Measures in Tinnitus International Delphi (COMiT’ID) Research Steering Group. Recruiting and Retaining Participants in e-Delphi Surveys for Core Outcome Set Development: Evaluating the COMiT'ID Study. PLoS One [Internet]. 2018 [cited 2023 Jul 16];13(7):e0201378. Available from: https://doi.org/10.1371/journal.pone.0201378
  » https://doi.org/10.1371/journal.pone.0201378
• 11. Broder M, Gibbs S, Yermilov I. MSR10 An Adaptation of the RAND/UCLA Modified Delphi Panel Method in the Time of COVID-19. Value Health [Internet]. 2022 [cited 2023 Jul 16];25(7):S519. Available from: https://doi.org/10.1016/j.jval.2022.04.1217
  » https://doi.org/10.1016/j.jval.2022.04.1217
• 12. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining Consensus: A Systematic Review Recommends Methodologic Criteria for Reporting of Delphi Studies. J Clin Epidemiol [Internet]. 2014 [cited 2023 Jul 16];67(4):401-9. Available from: https://doi.org/10.1016/j.jclinepi.2013.12.002
  » https://doi.org/10.1016/j.jclinepi.2013.12.002
• 13. Banno M, Tsujimoto Y, Kataoka Y. The Majority of Reporting Guidelines are not Developed with the Delphi Method: A Systematic Review of Reporting Guidelines. J Clin Epidemiol [Internet]. 2020 [cited 2023 Jul 16];124:50-7. Available from: https://doi.org/10.1016/j.jclinepi.2020.04.010
  » https://doi.org/10.1016/j.jclinepi.2020.04.010

NOTES

• TRANSLATED BY

  Ingrid de Ruiter

Edited by

EDITORS

Associated Editors: José Luís Guedes dos Santos, Maria Lígia Bellaguarda. Editor-in-chief: Elisiane Lorenzini.

Publication Dates

• Publication in this collection
  05 Aug 2024
• Date of issue
  2024

History

• Received
  30 Aug 2023
• Accepted
  13 Nov 2023