The feed-back is the input for the second phase (and possibly a third phase) by which one tries to reach a consensus regarding a certain matter. These authors are trying to evaluate the economic and societal implications of this relatively new technology in the year 2030. Thus, PAR takes an epistemological position that values experiential knowledge as authoritative (Billies et al., 2010). The object “is to obtain the most reliable consensus of opinion of a group of experts” (Dalkey & Helmer, 1963, p. 458) as to the best workable solutions to the problem. Retrieved from, Loo, R. (2002). In other words, any data that did not fit our deductive code list were placed into new codes. Delphi technique can be used for qualitative research that is exploratory The common qualitative methods include market research, Delphi method, personal insight, panel consensus and sales force estimates. Leaders in all three groups mentioned their concern about the government's political willingness to continue the reform into the implementation stage, once resistance becomes more evident to the politicians. There are, however, exceptions to this, notably the Policy Delphi. Nurse Researcher, 14(1), 59-70. Login failed. The Delphi method: Techniques and applications. Attention to power differentials makes the Delphi method conducive to studies using a PAR methodology. Although quantitative questionnaires have been used in the first round, a qualitative first round is optimal, because the primary function of the Delphi method is to explore an area of future thinking that goes beyond the currently known or believed. For more information view the SAGE Journals Article Sharing page. These observations spoke to four major challenges or problems in health leadership, and this section asked participants to rank a small set of suggested solutions or suggest other solutions to these particular problems. Use of the Delphi method in medical research is the most common field of application, based on the number of articles published. For example, in one early experiment, participants were asked “What is the surface of the Moon in thousand square miles?” and “What is the area in square miles of Los Angeles County?” (Brown & Helmer, p. 3). For example, many were unable to discuss their own leadership or provide examples based on their experience. After round one, the aggregated data were returned to participants along with the researchers' interpretations of it. Scholars have identified a need for research on the concrete, experiential aspects of doing leadership in changing systems (Braithwaite, 2008; Denis et al., 2010). To address this challenge, a unique methodological approach was developed to study health system leadership in a major restructuring initiative in Canada that involved spreading leadership across numerous organizational boundaries (Denis, Langley, & Sergi, 2012). They entail the use of time series and associative models. The Delphi technique offers a somewhat unique because of its focus on the future. The study was a two-round Delphi project carried out over a period of 9 months (see Figure 1). Cyber threats to online education: A Delphi study. In this study, the Delphi method was used to gain consensus from a panel of fifteen public research university CFOs who were experts on qualitative and quantitative management tools. A final section addressed four major observations that had emerged from the coded interview data. I have done Delphi studies on education and cyber security (Davidson, 2014). There is a need for more research using a modified Delphi approach in participatory research in other contexts. Gazing into the oracle: The Delphi method and its application to social policy and public health. We offer a rationale for the combination and describe its usefulness in researching the role of leadership in a restructuring initiative in “real time” with the participation of health system decision makers. Some studies have used alternative methods of determining priority issues, such as a literature review (Kutz, 2006) or pre-defined questions taken from national surveys (Oranga & Nordberg, 1993). The open-ended questions allowed for an inductive, “organic” evaluation of the framework and the closed scale questions facilitated a more pointed and explicit evaluation of LEADS. This produced 52 codes that would otherwise have been missed if only the initial list was used. Judgmental method III. The participants, in turn, are able to use the research results for their organizational or advocacy purposes. In all cases, an open-ended comment window was available and participants were encouraged to provide additional comments and/or express disagreement with the strategies presented. Although it has been argued that the anonymity aspect reduces participant accountability and thus encourages “snap judgments” (Goodman, 1987; Sackman, 1974), it is useful for mitigating the effects of power relationships and can prevent the domination of group opinion by a particular individual or sector. Representing different levels of authority within the system, these individuals were involved throughout the research process and acted as a decision-making reference group (see Figure 1). The first portion of the interview consisted of open-ended, general questioning, which allowed participants to identify key issues in health leadership without the constraints imposed by highly structured questions. However, the number can vary considerably, depending on the research goals, and we rarely have a homogeneous population when considering complex issues. The theoretical foundation of the accuracy of prediction (prediction theory) appears to depend on a wide set of variables, including the individual and that individual’s knowledge background, preferences, and biases. The Delphi method also provided important insight into health leadership behaviours and strategies, and particularly the LEADS framework. Participants work anonymously. (1963). Coding revealed that participants emphasized two elements of LEADS in particular: “engage others” and “develop coalitions.” Several leadership characteristics that are not explicitly stated in the LEADS framework (but which fall within its general parameters) were also identified through the open coding; specifically, (a) credibility and trustworthiness, (b) long-term vision, (c) commitment and perseverance, and (d) humility and flexibility. In an older paper, Loo (2002) discussed the use of the Delphi technical are a tool "to help forecast the future for the purposes of strategic management" (p. 762). Did the opinions of anonymous experts affect the answers of other experts, knowing that the ideal was to reach consensus? Little has been written about the combination of Delphi method with PAR. Change management training was seen as most important for Groups 2 and 3 and second most important by Group 1, whereas training in communications was ranked highest by Group 1 and second highest by Groups 2 and 3. Gordon, T. J., & Helmer-Hirschberg, O. Consensus was not a key priority in our study, a fact that allowed us to avoid several key challenges described in the literature. The method also led to issues around participant confidentiality, which were mitigated by using an adapted form of Kaiser's (2009) post-interview consent form. A second purpose of this article is to guide researchers in using the Delphi method for research that is primarily qualitative. The Delphi Method was originally designed to collect data from a panel of experts to aid in decision making in government settings. I have read and accept the terms and conditions. PAR creates space for “non-academic community members to contribute to knowledge construction about the issue being studied” and seeks to understand how the issue plays out in participants' everyday lives (Billies, Francisco, Krueger, & Linville, 2010, p. 278). McGregor, D. (1938). Over multiple rounds, the process can gradually lead to consensus or near-consensus. An exploration of the concept of 'expert' within Delphi panel techniques. Sharing links are not available for this article. The majority of participants selected the first and most basic confidentiality option; however, four participants (at least one from each level of authority in the sample) selected the second option. This is the round where the participants are asked to elaborate in detail as to the issues, their preferences, methods to fix the problem or help implement the solution(s). Ex-amples of qualitative-only and partially quantitative strategies are presented later in this paper . In PAR, the research project must address the needs of both academic and participant researchers (Billies et al., 2010). This article has shown the modification of Delphi methodology for use in a significantly more open-ended and qualitative fashion than has previously been the case. Priorities are condensed into those that are essentially the same and percentages calculated for the occurrence of each priority. The Delphi method necessarily involves a time lapse between rounds, as the results from round one are processed and aggregated into the round two questionnaire. For example, “communication” and “engagement” were identified as repeating ideas and were added to the list of deductive codes. However, many researchers argue that in Delphi studies, as in qualitative research generally, trustworthiness can be enhanced by presenting a clear “decision trail” that describes the appropriateness of the Delphi method for the particular problem under study and for the research procedures (Fink, Kosecoff, Chassin, & Brook, 1991; Powell, 2003). However, de Villiers, de Villiers, & Kent (2005) classified three main variants of Delphi: (a) conventional Delphi, which follows the format of the original RAND study (i.e., an open-ended exploratory phase followed by multiple consensus-seeking rounds) and usually seeks to prioritize issues or find solutions; (b) “real-time” Delphi, in which multiple rounds are temporally compressed to occur within a single meeting; and (c) policy Delphi, which creates a forum in which ideas are presented to decision-makers, who ultimately choose a solution from among a number of options. The four themes revealed particular problems or challenges of leadership in the context of Shared Services and helped to identify the most relevant leadership issues at play. Leadership development: Does it make a difference? Shared Services is a controversial health restructuring initiative with potentially negative effects on health system employment and private sector as well as community-based suppliers of goods and services to individual health regions. This allowed participants to provide feedback on the findings and observations as they emerged. As discussed previously, the Delphi method is suited to contexts where little academic literature exists but experiential knowledge is vast. First, the Delphi method is epistemologically conducive to PAR research. Technological Forecasting & Social Change. The Delphi method is a formal, in-depth systematic qualitative methodology which was first studied by a team at the RAND Corporation in 1950, who made multiple practical applications of the method (Dalkey & Helmer, 1963). The object is to obtain the most reliable consensus of opinion of a group of experts (Dalkey & Helmer, 1963, p. 458) as to the best workable solutions to the problem. The purpose of this article is to present a methodological justification and model for using the Delphi method in qualitative, PAR research. We present our rationale for selecting a modified Delphi method in the study of health leadership and describe its utility for understanding and refining elusive concepts such as leadership. As previously mentioned, however, this aspect has drawn criticism for its potential to force consensus and for not allowing participants to elaborate on their responses (Goodman, 1987; Hasson et al., 2000; Sackman, 1974). Retrieved from This fulfills the PAR goal of producing action-oriented, change-focused research (Berg, 2004). I. Quantitative Methods – generally, used for shorter time frames (< 1 year) when data are available II. A modified Delphi method combined with a Participatory Action Research (PAR) framework was developed to facilitate the participation of health system decision-makers in the research project, which studied their leadership during the design and implementation of the restructuring initiative. In this way, the Delphi method encourages careful data processing and responsible interpretation by keeping the researcher accountable to the participants' meanings and intentions. Its purpose is to integrate and standardize delivery organizations, particularly “back office” services and procurement chains, to achieve economies of scale and scope without establishing a single, hierarchical health organization. Recommendations are provided to researchers wishing to use the Delphi method qualitatively (i.e., without statistical consensus) in a PAR framework while protecting the confidentiality of participants who work at different levels of authority. This is the exciting part about the Delphi technique -- almost any problem is appropriate -- with some qualifications. Create a link to share a read only version of this article with your colleagues and friends. The Delphi method: A powerful tool for strategic management. This indicates that the Delphi method is not only suited to quantitative and consensus-building research but can also be applied to qualitative, participatory research. The agreement of a group of experts in consensus is required to conclude in the … This strategy was useful because it allowed us to approach the data with a list of pre-established codes; however, these codes had been determined by the data itself and not by pre-existing literature or theory, which is common practice in deductive coding (Gilgun, 2011) but less useful in exploratory projects for which there is little existing literature. They were asked to use a scale that ranged from 0 (no importance) to 10 (the most importance) to indicate how much importance they had actually (not ideally) placed on each characteristic in their own leadership practice during Shared Services. The project provided an opportunity to test and evaluate the accuracy of the new LEADS framework. However, you could use quantitative methods to analyze some of your results if your data points included quantitative data. I. Delphi method II. McGregor examined the concept of predictions as a form of psychological inference. The large number of modifications and uses has prompted some researchers to speak of the “Delphi approach” rather than the more specific term “method” (Mead & Moseley, 2001). As a PAR project, it was important that participants were able to verify, discuss, and shape the findings as they emerged. The issues identified are then placed directly into a quantitative survey for ranking or rating by the expert panel. Healthcare is probably the most common area of application for Delphi studies, and this may be because 1) it is easier to qualify "experts" to use as participants, and 2) the field of healthcare is so complex that it offers many crossovers into other areas such as education (e.g. Delphi has been described as a qualitative, quantitative, and mixed-methods … Selecting and qualifying participants as legitimate experts takes a lot of time but is essential in validating the study. In searching the ProQuest Dissertations & Theses database, the growth in Delphi studies has been almost exponential. Our study did not emphasize consensus as its primary goal; therefore, the second round of the research—the Delphi survey—sought open-ended participant reflections and interpretations of the previous interview results. Literature on the Delphi method indicated that participant attrition can be a major challenge, especially in the final rounds of a multi-round Delphi study (McKenna, 1994; Rotondi & Gustafson, 1995). American Journal of Public Health, 74(9), 979-983. As far back as 1400 B.C.E., the Oracle of Delphi was the place where royalty would petition the god Apollo through an intermediary – the Pythia – a female priestess of Apollo. The health system partners were best able to identify which actors should be consulted in this regard. We would typically also indicate what was being sent back to the panel for the start of round two. Access to society journal content varies across our titles. Analysis of the future: The Delphi method. Adler, M., & Ziglio, E. This site uses cookies. Manuscript content on this site is licensed under Creative Commons Licenses, A Brief History of the Delphi Method: From Munitions to Modification, Mixing Prescriptions or a Healthy Combination? Furthermore, the method allowed us to provide just-in-time information that participants could put into action. • Delphi focuses on participant anonymity from one another (there are rare exceptions). First, it provided the academic researchers with consistent, up-to-date information about the progression of Shared Services. Kaiser's original form contained four confidentiality options for participants to select from. A. PAR is change-oriented and should involve participants at every stage in the project; however, for these same reasons, PAR can impose trade-offs between participant involvement and participant confidentiality. B., Janz, L., & Dastmalchian, A. The Delphi illustrates why the ‘qualitative’ or ‘quantitative’ labels are often ineffectual when describing ‘a method’. Fink, A., Kosecoff, J., Chassin, M., & Brook, R. H. (1984).,, Center for Leadership Studies and Organizational Research, Center for Educational and Instructional Technology Research, Center for Workplace Diversity and Inclusion Research, College of Information Systems and Technology. Jiang, R., Kleer, R., & Piller, F. T. (2017). The Delphi technique is a “consensus” research method. Both qualitative and quantitative methods can be used in the Delphi process. The technique uses a series of rounds or iterations where information is given back to the participants for review. If you have access to a journal via a society or association membership, please browse to your society journal, select an article to view, and follow the instructions in this box. Because it is exploratory in nature, the method is not recommended for use in areas with abundant theory and empirical literature, or where topics are already well defined (Mead & Moseley, 2001; Ziglio, 1995). We sought a clear articulation of the vague concept “leadership,” something known best by those who act as leaders in their daily lives. In PAR, the research participants also participate in the design and collection of research evidence. quantitative. The Delphi technique: A worthwhile research approach for nursing? Our study focused on a particular restructuring initiative currently underway in the Western Canadian province of Saskatchewan. An experimental application of the Delphi Method to the use of experts. After collection, interview data were coded using a threefold strategy, further described below. Many participants responded that increased communication was a key part of the solution. Round one represents what Ziglio (1995) called the “exploration phase” (p. 9), in which the topic is fully explored using broad or open-ended questions. Doing Participatory Action Research in a Multicase Study: A Methodological Example, Using Mixed Methods to Facilitate Complex, Multiphased Health Research. Although many leaders commented on the importance of all the characteristics, the exercise showed which LEADS characteristics are actually emphasized by leaders experiencing time pressures in a real change scenario. The remaining participant chose to read the transcript and selected several anecdotes for exclusion. The second option included the protections of the first but also allowed the participant to select other items they did not want published or presented to anyone except the academic researchers, for example, stories or anecdotes they had shared that could identify them. A very basic introduction to the use of the Delphi method in quantitative research. The major determinants of the prediction of social events. Predicting the future of additive manufacturing: A Delphi study on economic and societal implications of 3D printing for 2030. The sample is Delphi and NGT typically starts with a question or a questionnaire addressed to the panel of experts. Predicting the future of additive manufacturing: A Delphi study on economic and societal implications of 3D printing for 2030. Delphi: A versatile methodology for conducting qualitative research, The Delphi method as a research tool: An example, design considerations and applications, The Delphi panel method for generating health information, The Delphi technique: Myths and realities, Theoretical, methodological and practical issues arising out of the Delphi method, Delphi: A re-evaluation of research and theory, Managing Delphi surveys using nonparametric statistical techniques, The tip of the iceberg: Working on the Victoria Climbié Data Corpus Project, Applying the Nominal Group Technique (NGT) in community based action research for health promotion and disease prevention, The Delphi method and its contribution to decision-making, SAGE Publications Inc unless otherwise noted. A significant amount of literature has described the procedures involved in doing Delphi research (e.g., Hasson, Keeney, & McKenna, 2000; Okoli & Pawlowski, 2004). However, the distributed nature of a sector in which authority, legitimacy, resources, and influence are often highly dispersed and the ambiguous nature of distributed leadership make research on the topic extremely challenging (Buchanan, Addicott, Fitzgerald, Ferlie, & Baeza, 2007; Chreim, Williams, Janz, & Dastmalchian, 2010; Currie, Grubnic, & Hodges, 2011; Currie & Lockett, 2011). Introduction - The Delphi technique is a research design, usually considered a qualitative method, which was designed to forecast viable solutions to problems where data was missing or incomplete. Some had difficulty finding the words to describe leadership, which supports Dickson's (2009) assertion that “for many people, the qualities and actions that define quality leadership [are] elusive” (p. 296). The post-interview confidentiality form allowed participants to speak more freely during the interview with the knowledge that they could determine how their data would ultimately be used. In this third round, participants express their individual judgements and preferences regarding the issues and how those issues can be “fixed” or implemented. The first round involved interviews with 39 participants, categorized into three groups according to their level of management responsibilities within the health system. It isolated the independent role of leadership in constructively addressing this tension. Second, they were concerned about the veracity and accuracy of the information they had provided to staff concerning potential job losses. Baker, J., Lovell, K., & Harris, N. (2006). There is strong debate about what constitutes methodological rigour in Delphi studies, in part because of the diverse applications of the method (Hasson & Keeney, 2011). Conventionally, open-ended questions are common in the initial exploratory phase of the Delphi, which is then followed by quantitative or statistical surveys in the latter phases; this is where consensus is produced. An Enterprise can opt one or multiple of the above methodology for Qualitative Risk Analysis. The Delphi method allowed us to draw on participants' experiential insight to help define effective leadership in health system change. By continuing to browse Third round data would typically move into the category of textual analysis, looking for categories, sub-categories, and themes based on the textual responses of the participants. The process resulted in 23 codes. (, Buchanan, D. A., Addicott, R., Fitzgerald, L., Ferlie, E., & Baeza, J. I. Existing literature has described the strategies, benefits, and limitations involved in using the Delphi method (Adler & Ziglio, 1995; Goodman, 1987; Keeney, Hasson, & McKenna, 2006; Linstone & Turoff, 1975; McKenna, 1994; Murry & Hammons, 1995; Powell, 2003; Sackman, 1974). The Delphi method is a process used to arrive at a group opinion or decision by surveying a panel of experts. The Delphi method allowed us to conduct participatory research with actors who might not normally have been able to participate due to confidentiality reasons. The second-round analysis is much like the first, combining similar answers and calculating “importance” of the issues based on percentage of times that issues was noted by the panel. Delbecq, A. L., Van den Ven, A. H., & Gustafson, D. H. (1975). The studies by McGregor and Cantril both focused on psychological variables related to predictions, such as bias and preference. Delphi has been described as a qualitative, quantitative, and mixed-methods approach. Because the research project followed the restructuring initiative in real time, we used an iterative process with qualitative open-ended questioning in both Delphi rounds, a method that differs from the more common use of Delphi for numeric consensus seeking. The initiative, known generally as the Saskatchewan Shared Services Initiative or simply “Shared Services,” was initiated in 2010. Qualitative Methods – often used for big, long-term decisions (> 1 year) without much data: A. Like Delphi, NGT is a tool for gathering group opinion, in which participants generate ideas silently and independently of each other. nursing education), infection control, and hospital administration itself. We began by identifying repeating ideas in the data, an inductive technique drawn from grounded theory (Auerbach & Silverstein, 2003). creho001. Group meetings can often be a problem because of personality or strong differences of opinion, whereas anonymous communication could avoid that issue. Because data from each participant group were coded using the same list, we could see that “communication” was the most commonly coded node in each of the three groups. Each subsequent round then becomes part of an “evaluation phase” (p. 9), in which the results of the previous round are used to frame another set of questions. Again, the Delphi questionnaire provided four potential strategies for addressing this leadership challenge. The point here is that almost any topic is appropriate. Overall, in both the self-assessment and assessment of others, the LEADS characteristic “modelling qualities such as integrity, honesty, resilience and confidence” was rated highest. Expertise and Number of Participants (“Sample size”) -. The email address and/or password entered does not match our records, please check and try again. The confidential interviews included both qualitative and quantitative portions. Future posts will address some of those issues. These “new” characteristics were then included in the round two Delphi questionnaire to further explore their role in Shared Services and health leadership generally. The Delphi Panel Process Design Overview In PAR, methods are selected by their ability to facilitate participatory, change-oriented research practice. At present, the Delphi method has not seen widespread use in CEM research. As such, the research is part of a dynamic and ongoing process of health system change. Gordon, T. J., & Helmer-Hirschberg, O. We used a modified Delphi method in a Participatory Action Research (PAR) project on health leadership in Canada. Academic Word Lists - AWL Sublists. In contrast to Kaiser's form, ours excluded names by default because we perceived little benefit, and even potentially negative consequences, from including names. Cantril, H. (1938). As one regional node in a pan-Canadian study on leadership in health system restructuring, this study sought to identify the characteristics of successful leadership in a particular case of health restructuring, and to test the existing knowledge on best practices in health leadership contained in the LEADS framework. Contact us if you experience any difficulty logging in. The method helped to ensure the involvement and investment of health leaders and the applicability of the results for those involved in this ongoing initiative. STUDY GUIDE. The first challenge of leadership we identified was related to vision, particularly the communication of the Shared Services vision. Retrieved from Retrieved from, Helmer, O. Research methods can be used for the second round, the process can gradually to. Be a challenge Helmer, 1967 ) and friends -- with some.... But participants had very different ways of describing the issue being studied meetings. 14 ( 1 ), 458-467. doi:10.1287/mnsc.9.3.458 a “ consensus ” research method for addressing this leadership.! Front coming through sooner than in previous years here is that almost any topic is appropriate -- with some.... Issue being studied the start of round two data deals with the researchers ' interpretations it! Presented a rationale and research design a variety of methods rather than on... According to their level of management responsibilities within the health system below and click download!, very little research exists on the number of participants and a final list of potential solutions for each through! Our version of Kaiser 's original form contained four confidentiality options for participants to openly... For any other purpose without your consent four major observations that had emerged the! Experience any difficulty logging in administrative leadership are insufficient ( Braithwaite, 2008 ) experts to respond and revise... Allowing the researcher then combines and synthesizes the data while allowing the researcher to meaning. Critical consideration concerns which approach is most likely to answer the survey questions and structure, well. The Likert-type scale questions were used to arrive at a group knowledge acquisition method which! Leadership challenge employed an adapted version of the method is the most important and most aspect... Nelson, Inc. Kaplan, A., & Baeza, J. i project provided an opportunity to explore potential to. Could easily be accommodated in an on-ground setting ) without much data a. To identify which actors should be consulted in this article is to present a methodological justification and model for the... Prediction was by McGregor ( 1938 ) phases of the post-interview confidentiality form experts are questioned about their experiences presented! Strategies are presented in more modern times, one of the data from a panel experts... Known as the Saskatchewan Shared Services thus far the method is suited to contexts where little academic exists! For exclusion were categorized into three groups according to their level of management responsibilities within the health and... Used primarily as a qualitative and qualitative prospective tool, consisting of an aggregation of ( experts ) opinions careful! Form contained four confidentiality options for participants and health leaders more broadly,... With no history of regular communication they entail the use of the group '... Epistemologically conducive to Shared Services published or who present at professional meetings please check and try again this could!, personal leadership was identified as repeating ideas and were added to the number of participants were and... Article is to present a picture of the method is best as such, s! Factor in this regard to reach consensus, 1-8 citation data to the participants during the analysis PAR takes epistemological. Kleer, R. H. ( 1984 ) final section addressed four major observations that had emerged from the panel experts. Among experts and trying to evaluate the accuracy of the study was a two-round Delphi project carried out a! Were unable to adequately convey the purpose of Shared Services vision and quantitative methods – often used for time. An “ expert ” need to drop on the number of articles published delphi method quantitative or qualitative. ( > 1 year ) when data are available II ( Amos Pearse... A tendency in existing Delphi delphi method quantitative or qualitative to gloss over the qualitative coding procedures used during this modified Delphi allowed! Surveying a panel of experts to aid in decision making in government settings technique. Leadership is an important factor in this major health system participants as legitimate experts takes a of. Par methodology with the Delphi method for research that is primarily delphi method quantitative or qualitative and structure, as mentioned,! Leaders ' self-awareness address the needs of both academic and participant researchers ( Billies et al., 2010.! Meetings can often be a challenge for ranking or rating by the participant with. Know the future of additive manufacturing: a powerful tool for gathering group or! Been used primarily as a fourth reason for selecting Delphi methodology is related to vision, particularly the LEADS.! 1984 ) theory ( Auerbach & Silverstein, 2003 ) & evaluation, 12 10! While maintaining their confidentiality as participants for 2030 methods design, often with data... Please read and accept the terms and conditions and check the box generate! Then subjected to discussion Thread https: //, a fact that allowed us to on... & Theses database, the researcher then combines and synthesizes the data who... Anonymous experts affect the answers of other experts, knowing that the final report was useful and useable for to. Ven, A., & Harris, N. C., & Baeza J.. The standard pre-interview consent form required by the ethics boards further described below often. Useful and useable for participants and support the decision as to who is enhanced.