Development and Content Validity of IFERA-II for Adults^*

Desenvolvimento e Validade de Conteúdo do IFERA-II para Adultos

Abstract

Considering the importance of the functional assessment, a scale was developed to evaluate Executive Functions (EFs), State Regulation (SR) and Delay Aversion (DA) in adults, based on a children’s scale, and evidence of content validity was investigated. In Phase 1, items from the children’s version were adapted to generate the Inventory of Difficulties in Executive Functions, Regulation and Delay Aversion for Adults (IFERA-II), with 28 items of EFs (inhibition, working memory and flexibility), RS and DA. Analysis by judges verified the representativeness of the contents of the items. After adaptations, 18 adult participants responded to the IFERA-II and were interviewed regarding comprehension, need for examples and clarity of vocabulary. A new revision led to the preliminary version of the IFERA-II, the psychometric properties of which should be investigated.

Keywords:
psychometrics; evaluation; cognition; neuropsychology; self-report

Resumo

Diante da importância da avaliação funcional, foi elaborada uma escala para avaliar Funções Executivas (FE), Regulação do Estado (RE) e Aversão ao Adiamento (AA) em adultos, baseada em uma escala infantil, e foram investigadas evidências de validade de conteúdo. Na Fase 1, itens da versão infantil foram adaptados gerando o Inventário de Dificuldades em Funções Executivas, Regulação e Aversão ao Adiamento para Adultos (IFERA-II), com 28 itens de FE (inibição, memória de trabalho, flexibilidade), RE e AA. Análise pelos juízes verificou representatividade de conteúdo dos itens. Após adequações, 18 participantes adultos responderam ao IFERA-II e à entrevista sobre compreensão, necessidade de exemplos e clareza de vocabulário. Nova revisão resultou na versão preliminar do IFERA-II, cujas propriedades psicométricas devem ser investigadas futuramente.

Palavras-chave:
psicometria; avaliação; cognição; neuropsicologia; autorrelato

Executive Functions (EFs) refer to high-level skills that, through control over other processes, allow the regulation of thoughts, emotions and actions and the direction of behavior towards goals. A widely accepted model and one of the most referenced in the area (Baggetta & Alexander, 2016) considers three core skills: a) inhibition, the ability to inhibit inappropriate behavior or impulse, including the ability to inhibit attention to distractors; b) working memory, the ability to sustain, update and manipulate information mentally; and c) cognitive flexibility, the ability to change the attentional focus or perspective and adapt to different demands. These skills participate in the performance in more complex situations (e.g. planning or decision making) (Diamond, 2013).

In the context of neuropsychology, considering particularly the studies on Attention Deficit Hyperactivity Disorder (ADHD), in addition to EFs, other skills have been the subject of investigations. Among them, state regulation (SR) can be mentioned and ispart of the so-called Cognitive-Energetic Model. This model proposes that the overall efficiency of information processing is determined by the interaction between attention mechanisms, state factors and EFs. More specifically, SR refers to the mobilization of energy required to drive the individual’s behavior towards a goal, partly overlapping with concepts such as mental effort and motivation (Sergeant, 2000; 2005).

Another important construct in this area is delay aversion (DA), characterized by the tendency to choose an immediate reward, albeit a smaller one, in preference to another, of greater value, but which requires a delay or waiting. For example, in the context of ADHD studies, it is suggested that these individuals have a motivational pattern that leads them to perceive delays as extremely aversive (Sonuga-Barke, 2005). Both SR and DA can be related to EFs and overlap with the concept of hot EFs, i.e., aspects of EFs needed in situations with significant emotional or motivational loads, such as in decision making and regulation of emotions. These aspects of EFs are typically assessed in tasks that involve delaying the reward (Zelazo, 2015). According to Zelazo, the main distinction between hot and cold EFs is how much motivational and emotional management is required.

The role of EFs (cold or hot) for different outcomes at different stages throughout life, including learning and school performance, academic procrastination, coping strategies, physical and mental health, propensity to risk behaviors (such as substance abuse) and involvement in crime, is well documented in literature (e.g. Diamond, 2013; Moffitt et al., 2011; Rabin et al., 2011; Reynolds et al., 2019; Seabra et al., 2014; Villegas & Cruz, 2015), adding importance to the evaluation of EFs. In addition, deficits in EFs, including SR and DA, have been identified and associated with the characteristic signs of neurodevelopmental disorders, such as ADHD (Coghill et al., 2018; Silverstein et al., 2018; Wagner et al., 2016), as well as Autistic Spectrum Disorder and Learning Disorders, among others (American Psychological Association [APA], 2014; Berenguer et al., 2018; Taghizadeh et al., 2017).

Despite the relevance of this skill set, some methodological difficulties remain for its measurement. In fact, an important discussion in neuropsychology refers to the ecological validity of its assessment instruments. The evaluation of EFs through performance tests have been criticized, as these tests have been considered limited in terms of their ecological validity and prediction of the behavior in the ‘real world’ environment (Garcia-Barrera et al., 2014; Isquith et al., 2013; Roth et al., 2014; Viklund et al., 2019; Zimmermann et al., 2014).

Some authors argue that, given the complexity of the construct, a multi-method assessment should be considered, with an important role for the functional measures in measuring EFs (Garcia-Barrera et al., 2014; Isquith et al., 2013). Functional measures are those that aim to assess the individual’s performance in everyday (or ‘real world’) tasks, which are usually carried out using evaluation scales (Zimmermann et al., 2014). A proposal for a multi-method assessment, however, may encounter lack of functional measures to assess these skills. According to Naglieri and Goldstein (2014), there are few scales for evaluating EFs with published evidence of validity and reliability. In the national context, this gap may be even greater.

Considering this discussion regarding the limitations of EF measures, since the 1990s there has been a growing interest in the development of scales for the evaluation of these functions. The Behavior Rating Inventory of Executive Function (BRIEF) was one of the first instruments developed for this purpose, and, currently, there are versions to evaluate pre-school children to adults (Roth et al., 2014). This interest was followed with the development of other scales, such as the Frontal Systems Behavior Scale (FrSBe), the Behavioral Assessment of the Dysexecutive Syndrome - Dysexecutive Questionnaire (BADS-DEX) (Garcia-Barrera et al., 2014), the Barkley Deficits in Executive Functioning Scale (BDEFS) (Barkley, 2014; Barkley & Murphy, 2010) and, more recently, the Adult Executive Functioning Inventory (ADEXI; Holst & Thorell, 2018) and The Executive Checklist (EC-10; Viklund et al., 2019), among others.

Taking the BDEFS as an example, the scale was developed with the aim of evaluating EF deficits in activities of the daily living of adults with ADHD (Barkley, 2014). A study with the instrument, which aimed to determine the contribution of the EFs in the impairment in occupational functioning of adults with the disorder, concluded that the scale had a greater contribution in predicting impairment in activities of daily living, in particular occupational adjustment, in comparison to performance tests. The authors concluded that the different methods of evaluation of EFs (performance tests x functional scales) measure different aspects of EFs, so that they should not be taken as indices in isolation (Barkley & Murphy, 2010).

Toplak et al. (2013) listed 12 scales used to assess EFs, including 10 with versions suitable for the assessment of adults. In the same study, the authors investigated the relationship between performance tests and EFs scales from 20 studies, with samples from children to adults. Of these 20 investigations, 13 used the BRIEF, five the BADS-DEX (1 study used both) and three an impulsivity scale not identified by the authors. Therefore, the BRIEF and DEX seem to be among the functional instruments most used in international studies. Regarding their findings, in agreement with Barkley and Murphy (2010), Toplak et al. pointed out that performance tests and scales seem to evaluate different aspects of cognitive functioning and that both types of assessment provide different information that can contribute to the understanding of the patient’s difficulties in a clinical context.

This conclusion has been corroborated by other authors (Barkley, 2014; Holst & Thorell, 2018; Thorell & Catale, 2014). Thorell and Catale, for example, mention that scales make it possible to capture more global aspects of executive behavior compared to performance tests. In a theoretical article, Isquith et al. (2013) summarized some evidence related to the utility of scales in the measurement of EFs, mentioning their relationship with neurological substrates associated with EFs and functioning in daily activities, including academic performance, for example. Although their discussion is directed to the evaluation of EFs in children, their conclusions can also be applied to the evaluation of adults. The authors also emphasized that performance tests and functional measurement scales measure different aspects of EFs and both contribute to the comprehension of the difficulties of a particular patient, encouraging a multilevel or multi-method assessment using different instruments.

Although scales seem to have greater ecological validity compared to performance tests, are easy to administer and provide important information, for example for screening, they also present some limitations. Among them, there is the fact that evaluators have less control over environmental factors that can affect the evaluation (e.g. the different environmental demands of the patients’ place of work or study) (Roth et al., 2014). Another criticism concerns the fact that some of these instruments are relatively long (e.g. BRIEF: 63 to 86 items; BDEFS-CA: 70 items) (Thorell & Catale, 2014).

In Brazil, some scales have already been translated. For example, Carim et al. (2012) translated and adapted the BRIEF. Accuracy values were considered adequate and principal component analysis identified two dimensions similar to those of the original version. Macuglia et al. (2016) adapted and investigated evidence of validity of the BADS which, in addition to the performance evaluation subtests, includes the DEX, a functional measure. Analysis was performed by judges and a focus group was conducted. The study concluded by attesting to the quality of the BADS and DEX items. In addition to these studies, there are ongoing and unpublished initiatives and investigations by some research groups on functional EFs measures for assessing the adult population (e.g. BRIEF - version for adults by the group of Prof. Elizeu Macedo, Universidade Presbiteriana Mackenzie). BDEFS (Barkley, 2018) was recently published in Brazil. However, a limitation remains in terms of instruments available at national level.

In the context of childhood and adolescence, in order to reduce the limitation of instruments available in the area, the Difficulties in Executive Functions, Regulation and Delay Aversion Inventory - Version for children and adolescents (Inventário de Dificuldades em Funções Executivas, Regulação e Aversão ao Adiamento - Versão para crianças e adolescentes - IFERA-I) (Trevisan & Seabra, 2014) was developed. The development of IFERA-I sought to integrate, in addition to the EFs construct (based on the model of Diamond, 2013), that of SR (Sergeant, 2000; 2005) and DA (Sonuga-Barke, 2005). All of these constructs were considered in the development of the IFERA-I items, as the scale was initially designed to assess children and adolescents with ADHD, through the reports of both parents and teachers. Accordingly, IFERA-I provides the possibility of measuring all these skills, having the potential for clinical and research use and considering reports from multiple informants (Trevisan, Berberian et al., 2020).

IFERA-I proved to be sensitive in identifying individuals with and without ADHD, with the latter having greater difficulties in all the skills assessed (Trevisan, Dias et al., 2020). However, the instrument has also been shown to be suitable for assessing these skills in non-clinical samples, being useful in delimiting difficulties and facilities in specific skills. For example, in healthy children and adolescents, the indices assessed by IFERA-I are moderately to highly related to indicators of inattention and hyperactivity (Trevisan, Berberian et al., 2020). In a sample of preschool children (non-clinical sample), a recent study showed that the EFs indices of IFERA-I (especially flexibility and inhibition) are important predictors of behavioral indices (including emotional symptoms and relationship problems, for example) (Dias et al., 2017). Other evidence (Dias et al., 2020) suggests the usefulness of IFERA-I in predicting academic performance, even after 2 years. Working memory and DA skills, measured in preschool children, predicted children’s performances in reading and mathematics, respectively, at the end of the 2^nd year.

These studies illustrate the adequacy and usefulness of IFERA-I in the evaluation of EFs, SR and DA in children and adolescents, even in non-clinical samples. However, the difficulty in assessing these skills in adults remains. Considering this demand, IFERA-I was used to develop an instrument more coherent to the demands of adult life. This process gave rise to the Difficulties in Executive Functions, Regulation and Delay Aversion Inventory - Adult version, (IFERA-II), presented in this article.

The development of items, based on theoretical models of the area (Naglieri & Goldstein, 2014), was then followed by the important stage of obtaining evidence of the instrument’s content validity. This evidence refers to the representativeness and scope of the items to assess the domain that is intended (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014). This procedure aims to guarantee the quality of the items, which will provide greater security in the subsequent analyses regarding the psychometric properties of the test. In this process, in addition to the elaboration, choice or adaptation of items based on consistent theoretical models, a strategy used is the analysis of expert judges, who judge the clarity and representativeness of the items. Therefore, determining the evidence of content validity of an instrument is a process that permeates its entire construction/adaptation (Pasquali, 2010; Peixoto & Ferreira-Rodrigues, 2019).

Considering the need to adhere to methodological rigor in the construction/adaptation of instruments and verification of their psychometric properties and in view of the scarcity of functional measurement scales for the evaluation of EFs, SR and DA for the adult population in the national and even international contexts, the aim of this study was to develop a new scale for the evaluation of EFs, SR and DA in adults, based on the adaptation of the items of IFERA-I, and to investigate the quality of the items and evidence of content validity of the new instrument.

Method

IFERA-II construction process took place in 3 stages: 1) Adaptation of items from IFERA-I version (Trevisan & Seabra, 2014; Trevisan, Berberian et al., 2020); 2) Investigation of evidence of content validity, through the analysis of expert judges; and 3) Pilot Study, applying the instrument with a group of volunteers.

Step 1 - Adaptation of the scale items

The adaptation of the items of IFERA-II, from the pre-existing version, IFERA-I, already validated in the national context (Trevisan, Berberian et al., 2020), sought to maintain the same constructs present in each item of the instrument, based on the models that supported its development: EFs (Diamond, 2013), SR (Sergeant, 2000, 2005) and DA (Sonuga-Barke, 2005). The adaptation carried out was specific to the writing of the items, attempting to specify the difficulties/situations to the context of an adult’s life in a more appropriate way, without changing the ability assessed.

This process led to version A of IFERA-II (Trevisan et al., 2016), composed of 28 items, representing the five dimensions of Working Memory (WM), Inhibitory Control (IC), Cognitive Flexibility (CF), Delay Aversion (DA) and State Regulation (SR), with a Likert-type score with five options (Never, Rarely, Sometimes, Often, Always). The items depict examples of day-to-day behaviors/situations that require the different skills contemplated by the IFERA-II. Some examples are: WM - “When you are in the middle of an activity, you often get lost or forget what you were doing”; IC - “You begin to answer a question, even before it is finished”; CF - “You take a long time or have difficulty finding a new way or different alternatives to solve a problem when you are stuck”; DA - “When you want something, you expect to get it immediately”; and SR - “You have difficulty starting an uninteresting task, needing help or more time for it”.

Step 2 - Investigation of evidence of content validity

Participants

Initially, six expert judges analyzed the items. All of them were psychologists, two with Master’s degrees and four with doctorates, all having clinical and research experience in neuropsychology. Subsequently, in the final stage of the evaluation of the modified instrument based on the observations of the six judges, a seventh judge, with a doctorate and also with experience in neuropsychology, participated.

Instruments and procedures

In addition to the initial version of IFERA-II (version A), described in the previous step, the six judges received an evaluation protocol. On the instrument, the judges had to evaluate the 28 items based on the following criteria:

• 1) content of each item of the scale (choosing among the alternatives Working Memory, Inhibitory Control, Flexibility, Delay Aversion and State Regulation);

• 2) clarity of the task instructions (Are the instructions for the task clear? Being: 0 - no/ 1 - yes, with reservations/ 2 - yes);

• 3) clarity of the item wording (Is the item wording clear? Being: 0 - no/ 1 - yes, with reservations/ 2 - yes);

• 4) need for inclusion of an example in the item (Is there a need for an example for the item? Being: 0 - no/ 1 - yes).

In addition to the 28 items, for the evaluation instrument, examples were created for each item on the scale, and the judges were asked to evaluate the examples, as follows: 1) relevance of the example to the content of the item (Is the wording of the example relevant to the item? Being: 0 - no/ 1 - yes, with reservations/ 2 - yes); and 2) clarity of the example wording (Is the example wording clear? Being: 0 - no/ 1 - yes, with reservations/ 2 - yes). The analysis of the examples was carried out because, in the event of the inclusion of examples in certain items, they would already have been written and evaluated by the judges (one judge did not respond to that part of the instrument, therefore the analysis of the examples consisted of responses from five judges).

In the analysis of both, the items and the examples, there were open questions in which the judges could include suggestions for changing the items/examples. For all criteria, the percentage of concordance between judges was calculated. An acceptable concordance rate of 80% among the expert judges was considered. Items/examples with less agreement in any of the criteria were reviewed.

This initial analysis by the six judges led to some adjustments to the scale made by the authors (described in the Results section). The resulting version, version B, was then submitted to a seventh judge. This judge only performed the analysis of the content of each item of the scale. In other words, after reformulating the items, this new analysis aimed to guarantee the representation of the constructs in the revised items. For this analysis, the judge received the evaluation protocol, but was asked to answer only criterion 1) content of each item on the scale (having to choose among the alternatives Working Memory, Inhibitory Control, Flexibility, Delay Aversion and State Regulation).

Step 3 - Pilot Study

Participants

A total of 18 individuals participated in this stage, with a mean age of 36 years (SD = 14.06; minimum age = 18 years and maximum age = 55 years), 10 of whom were male (55.6%). Of the participants, 1 had only complete Elementary Education I, 2 had incomplete High School Education; 4, complete High School Education; 3 were enrolled in Higher Education; 3 had graduated and 6 were taking graduate courses.

Instruments and procedure

This stage of the study was submitted as part of IFERA-II validity evidence investigation project and was approved by the CEP (CAAE: 67459517.7.0000.5435). All participants signed a consent form. In this stage, version ‘B’ of IFERA-II was individually applied to the 18 participants. The application was in a single session of approximately 45 minutes, conducted by an Educational Psychology Master’s student. IFERA-II instrument - version ‘B’ was delivered to the participants, who had to read it silently and respond to each item. After responding to each item, marking their answer on the IFERA-II instrument, the participants answered, orally, a set of questions about the item. The evaluator recorded the responses on a pilot study protocol. Afterwards, the same procedure was repeated for the next item and successively until completion of the 28 items on the scale.

The aim of this procedure was to verify the clarity and applicability of the items and of the scale in general. The pilot study protocol was used to systematize this process and contained three questions, which were asked for each item. In the first question, the evaluator questioned the participant’s understanding of the item (‘What did you understand about this item?’, Based on the answer given, the score could be 1 - understood or 0 - did not understand); the second investigated the need for an example in the item (‘Do you think the item needs an example?’, being 0 - No or 1 - Yes); and, the third question verified the occurrence of non-accessible vocabulary (‘Is there a word that you did not understand or that makes it difficult to understand?’, being 0 - No and 1 - Yes, with them asked to indicate the word). Again, an acceptable concordance rate of 80% was stipulated. Accordingly, items were considered adequate if 80% or more of the respondents demonstrated an adequate comprehension of their content; examples were included if at least 20% of the respondents suggested inclusion; and vocabulary revision (considering the suggestions given by the participants) was performed if at least 20% of the respondents suggested the need for adjustment. The results of this pilot application were tabulated, and new changes were made to improve the items, including examples and clarifications in the wording. This process gave rise to the final version of IFERA-II.

Results and Discussion

Step 1

Initially, the items of IFERA-II were elaborated based on the adaptation of the items of the original version of IFERA-I. In general, few adaptations/changes were necessary, since the items describe behaviors in a relatively generic way (for example: ‘When you are in the middle of an activity, you often get lost or forget what you were doing’, in which there is no specification of the type of ‘activity’, so that the item could be used in the evaluation of children, adolescents or adults). Major changes were necessary in the elaboration and/or adaptation of the examples, which were adapted to typical situations experienced by an adult [In the case of the item illustrated above, the proposed example was: When you are doing something (a project, organizing a closet, going to look for something in another room), You have difficulty maintaining your focus or end up forgetting what you were doing]. An illustration of the type of adaptation performed is shown in Table 1, which includes an item and example from the original version of IFERA-I and its modified version for IFERA-II. The adaptation of the items and examples gave rise to version ‘A’ of IFERA-II, submitted to the analysis of 6 judges.

Table 1
Example of Adaptations made to the IFERA-I Items for the Elaboration of IFERA-II

IFERA-I item	IFERA-II (adapted) item
She/he has difficulty maintaining and manipulating information mentally. For example, counting backwards.	You have difficulty maintaining and manipulating information mentally. Example: Performing a mental calculation or being able to think of several things at the same time that are important for solving a problem or for a task.
She/he has difficulty with tasks or activities that include several stages. For example, for young children, getting dressed completely without reminders; for older children, doing all the homework independently.	You have difficulty with tasks or activities that include several stages and end up getting lost. Example: When performing a long job (such as a cookery recipe or a job report), you have trouble checking whether something is missing and doing the parts in the right order, without skipping any.

Step 2

The results obtained through the analysis of the expert judges are summarized in Tables 2 and 3, which illustrate the concordance of the judges regarding the representativeness of content, clarity and the need for an example of each item. Regarding the general instructions of the instrument, there was 80% concordance regarding their clarity.

Table 2
Judges’ Concordance regarding the Content Representativeness of each IFERA-II Item (prior to the reviews carried out)

Items	Construct	Item*	WM (%)	IC (%)	FL (%)	DA (%)	SR (%)
1	IC	When you are talking to someone or when you are asked a question, you respond in a hurry, without considering all options or answer possibilities.		100
2	WM	When you are in the middle of an activity, you often get lost or forget what you were doing.	66.6		16.6		16.6
3	CF	In a new environment, you feel uncomfortable or have trouble getting used to it.			50.0		50.0
4	WM	You have difficulty maintaining and manipulating information mentally.	100
5	WM	You have difficulty with tasks or activities that include several stages and end up getting lost.	50.0		50.0
6	CF	You take a long time or have difficulty finding a new way or different alternatives to solve a problem when you are stuck.	16.6	16.6	50.0		16.6
7	DA	When you want something, you expect to get it immediately.				100
8	DA	When you know something you really want is going to happen , you have a hard time waiting and do not stop thinking or talking about it.				66.6	33.3
9	SR	You have difficulty starting a task that you find boring or uninteresting, needing help or more time for it.				16.6	83.3
10	SR	You have difficulty or feel uncomfortable doing things that take longer or that require a lot of mental effort.	33.3	16.6			50.0
11	SR	You alternate at different moments: sometimes you are very fast and at other times you seem too slow to do the activities.		16.6	16.6		66.6
12	DA	You have difficulty concentrating on day-to-day activities when you know that something important is going to happen.		33.3		50.0	16.6
13	CF	When you get used to doing things one way, you feel uncomfortable or have difficulty doing them another way.			66.6		33.3
14	IC	You make decisions quickly without considering possible consequences.		66.6		16.6	16.6
15	DA	When something important is going to happen (such as a job evaluation or promotion), you get upset or have a hard time waiting and want it to happen soon.				66.6	33.3
16	DA	You prefer to gain something simpler immediately than to wait for something more interesting later.				100
17	SR	You have difficulty doing things that you find boring, but you have an easy time doing things you like, even if they are difficult.					100
18	SR	You are very interested in new objects or activities, but lose interest when the novelty wears off.			16.6		83.3
19	SR	You have difficulty controlling your speed in tasks, sometimes you go very fast and other times you take a long time to do the same activity.	33.3				66.6
20	CF	You have difficulty or feel uncomfortable changing the way you are used to carrying out day-to-day activities.			83.3		16.6
21	CF	If you are interrupted in an activity, having to pay attention to something else, when you return you have difficulty paying attention again to what you were doing before.	50.0		33.3		16.6
22	IC	You begin to answer a question even before it is finished.		83.3			16.6
23	IC	You get distracted by things that are not important at the time.		83.3			16.6
24	WM	You have difficulty remembering long instructions.	100
25	WM	When you read a text or after a long conversation, you have trouble remembering all the main ideas.	100
26	IC	You are very restless or agitated.		33.3			66.6
27	IC	You have difficulty stopping an activity immediately when you need to or when you are asked to do so.		66.6	33.3
28	WM	You have difficulty remembering the various stages of an activity.	83.3		16.6

• Note. * The ‘Item’ column contains only the main description of the item, which is fundamental and sufficient to understand the results. Examples were not included due to space limitations.

Table 3
Judges’ Concordance regarding Clarity and the Need for an Example of each IFERA-II Item

Item	Construct	Clarity* (%)			Need for Example** (%)		Item	Construct	Clarity* (%)			Need for Example** (%)
		0	1	2	0	1			0	1	2	0	1
1	IC			100	80	20	15	DA			100	100
2	WM		20	80	100		16	DA			100	80	20
3	FL			100	60	20	17	SR			100	100
4	WM		60	40	40	60	18	SR			100	100
5	WM			100	100		19	SR			100	100
6	FL			100	100		20	FL	20		80	60	40
7	DA		20	80	80	20	21	FL			100	100
8	DA			100	100		22	IC			100	100
9	SR			100	100		23	IC			100	100
10	SR			100	100		24	WM			100	100
11	SR		20	80	80	20	25	WM			100	100
12	DA			100	100		26	IC			100	100
13	FL			100	80	20	27	IC			100	100
14	IC			100	100		28	WM			100	100

• Note. *0 - no/ 1 - yes, with reservations/ 2 - yes.
• **0 - no/ 1 - yes.

According to the analysis of the judges, items 2, 3, 5, 6, 8, 10, 11, 12, 13, 14, 15, 19, 26 and 27 (14 out of the 28 items) obtained concordance below 80% in relation to the representativeness of content. The judges indicated lack of clarity in one item (4) and the need for the inclusion of an example in three items (3, 4 and 20). It was observed that the lack of concordance between the judges regarding representativeness of content was balanced among the types of items (2 items of WM; 3 of FL; 3 of CI; 3 of SR; and 2 of DA). This fact suggests that the lack of agreement was not due to the incongruity in a specific construct; but that it was probably more related to specific item writing issues.

After the judges’ evaluation, items that did not present at least 80% concordance among the evaluators in each criterion were subjected to more detailed analysis by the authors. To make the construct more explicit within the items, as well as to make the items clearer to the respondents (especially in the case of item 4), some adjustments were made in the wording. Furthermore, although the judges indicated the need for an example in only three items (3, 4 and 20), in order to clarify and specify their content requirements, it was decided examples should be included in eight items (2, 3, 4, 5, 6, 10, 12 and 21). This process also took into account some comments made by the judges in the evaluation protocol. Only in one item (20), despite the suggestion of the judges in favor of including an example, after analysis by the authors, they decided not to do so, as it was understood that adjustments in the wording were sufficient to make the evaluated content clear and specific (there was greater than 80% concordance among the judges regarding the requirement of this item).

The judges also evaluated the examples generated for each item in relation to the suitability of the example to the content of the item and the clarity of the wording of the example. One of the judges did not respond to this part. Therefore, the analysis considered the responses of five judges. In general, for 27 items there was 80 to 100% concordance regarding the relevance of the example and the clarity of the wording of the example. Only one item (17, SR) did not achieve the criterion of 80% concordance among the judges in relation to the relevance of the example. Following suggestions from the judges, this example was changed. Since some judges made suggestions for the improvement of other item examples, these were considered by the authors and accepted when deemed relevant, despite the good evaluation of the quality of the examples. This procedure allowed the elaboration of a bank of examples.

This first stage of the analysis of the judges and the subsequent review led to version ‘B’ of IFERA-II. This new version was sent to a seventh judge, who proceeded to (re)analyze the content of the reformulated items, in order to certify that the changes made would not have altered the content requirements of the items. There was total concordance in the analysis of this judge in relation to the intended constructs of each item. No further changes were made at this stage.

Step 3

Version ‘B’ of IFERA-II, obtained from the previous step, was then used in a pilot study. Table 4 shows the percentage of individuals that reported an adequate understanding of each item, in addition to the percentage of participants that responded affirmatively to the need (inclusion or maintenance) for an example and difficulty in vocabulary of the item (or example).

Table 4
Percentage of Participants that Reported Adequate Comprehension of the Items, the Need for an Example and Vocabulary Difficulties

Item	Adequate comprehension* (%)	Need for example (%)	Vocabulary difficulties*** (%)	Item	Adequate comprehension* (%)	Need for example (%)	Vocabulary difficulties*** (%)
1	72	17	33	15	100	11	17
2	100	33	0	16	100	11	6
3	83	61	28	17	67	28	44
4	78	67	61	18	72	39	22
5	89	61	39	19	61	44	50
6	94	28	11	20	94	6	17
7	100	6	6	21	100	33	33
8	100	6	6	22	100	6	6
9	89	17	22	23	94	22	6
10	100	44	11	24	89	17	11
11	61	22	67	25	94	11	6
12	100	56	22	26	94	0	22
13	100	0	0	27	89	33	11
14	89	17	11	28	94	28	28

• Note.* Adequate comprehension: 1-understood / 0-did not understand
• ** Need for an example of the item: 0-No / 1-Yes
• *** Vocabulary difficulties: 0-No / 1-Yes

The analysis of the pilot application showed that some items were more complex, with less than 80% of the participants demonstrating full understanding of their content (six items: 1, 4, 11, 17, 18 and 19). For questions about example inclusion and vocabulary difficulties, the inverted criterion was used, that is, if 20% or more of the participants indicated a change, this would be considered by the authors (at least 80% of the respondents needed to agree with the current form of presentation of the item for it to be maintained, without revision). Therefore, in 15 items (2, 3, 4, 5, 6, 10, 11, 12, 17, 18, 19, 21, 23, 27 and 28) the criterion indicated inclusion/maintenance of an example and in 14 items (1, 3, 4, 5, 9, 11, 12, 17, 18, 19, 21, 23, 27 and 28) the criterion indicated problems with vocabulary. There was a reasonable overlap, with practically the same items indicated for inclusion of an example and adaptation of vocabulary. This may also be linked to the understanding of the item itself, as among the six items with the lowest (<80%) comprehension rate, all were indicated for vocabulary adaptation and five (except for item 1) for inclusion of an example.

Based on the results of this assessment, examples were maintained (in 8 items) or inserted (in 7 items) (previously evaluated by the judges) for the 15 items identified above (those with 20% or more concordance regarding the need for an example). The levels of concordance for comprehension of the item and for vocabulary difficulties, as well as qualitative observations were considered for the other modifications of the items. All of these items were carefully reviewed with respect to their wording and clarity, looking for synonyms to replace the words identified as problematic in the assessment described above. For this, the observations and suggestions provided by the participants during the application were considered (the qualitative observations were even considered for the items in which there was adequate understanding and no indication of a need for an example or difficulty with vocabulary, when relevant). Table 5 summarizes some of the main adjustments made to each item.

Table 5
Main Adjustments made by Item after the Pilot Application

Item	Modifications made after the pilot study
1	Exchange of the word ‘precipitada’ for ‘apressada’
2, 3	Example maintained
4, 5, 6 and 10	Example maintained; wording adjustment performed in the example
11	Example included; in the item: removed ‘(demonstra)’ and changed from ‘agitado/acelerado’ to ‘acelerado’
12	Example maintained; in the item: changing from ‘rotina’ to ‘dia a dia’; wording adjustment performed in the example
17	Included example; in the item: exchange of ‘porém tem facilidade para outras coisas que são difíceis, mas que gosta’ for ‘mas tem facilidade para coisas que gosta mesmo que sejam difíceis’
18	Example included; in the item: changed from ‘a atividade se torna conhecida ou rotineira’ to ‘quando acostuma-se com a novidade’.
19	Example included; in the item: withdrawn ‘para uma atividade’; switching from ‘se organizar no tempo' to ‘em controlar sua velocidade nas tarefas’
20	In item: removed ‘uma ordem’
21	Example maintained; in the item: changing ‘para se engajar e prestar atenção’ to ‘em prestar atenção’; wording adjustment performed in the example
23	Example included
26	In item: removed ‘na maioria das situações’
27	Example included
28	Example included; in the item: changed from ‘tarefa’ to ‘atividade’

Closing of IFERA-II final version

The procedures described led to the final version of IFERA-II, with the same number of items as the initial version (28), the same division into subscales (EFs: WM with 6 items; IC with 6 items; CF with 5 items; DA with 5 items; and SR with 6 items), containing examples that illustrate the individual’s day-to-day situations and accessible vocabulary, making the instrument very suitable for self-application.

IFERA-II was developed in order to contemplate, in a single functional measurement instrument, EFs skills, specifically working memory, inhibitory control and flexibility (Diamond, 2013), DA (Sonuga-Barke, 2005) and SR (Sergeant, 2000, 2005). This option, as previously described, was made in the development of IFERA-I in order to try to encompass the complexity of ADHD neuropsychology (Coghill et al., 2018; Wagner et al., 2016). Despite the specificity in their conception, both IFERA instruments also allow the evaluation of these constructs in non-clinical samples and those with other diagnostic conditions, and may, in the future, help to expand investigations in the area and providing contributions to the clinical evaluation and direction of intervention.

Considering the recent emphasis on the use of functional measures in the evaluation process (Barkley, 2014; Garcia-Barrera et al., 2014; Holst & Thorell, 2018; Isquith et al., 2013; Thorell & Catale, 2014; Toplak et al., 2013; Viklund et al., 2019) and the still limited availability of functional measurement instruments (Naglieri & Goldstein, 2014), especially in the Brazilian context (e.g. SATEPSI search, updated in April/2020, not resulting in the location of instruments), the future availability of IFERA-II may constitute an important contribution. Therefore, a fundamental first step is the assessment of the relevance and content of the items of the new instrument/version (AERA et al., 2014; Naglieri & Goldstein, 2014; Pasquali, 2010; Peixoto & Ferreira-Rodrigues, 2019). Strategies to achieve the content validity of an instrument include the consideration of a consistent theoretical foundation in the construction of the items and the analysis of expert judges. In this study, the development of the items of IFERA-II took place from the adaptation of the items of the original version of IFERA-I (Trevisan & Seabra, 2014), based on consistent theoretical models of the area (Diamond, 2013; Sergeant, 2000, 2005; Sonuga-Barke, 2005; Wagner et al., 2016). It should be highlighted that the original IFERA-I version presented satisfactory psychometric properties, which include satisfactory reliability indices and evidence of content validity and in relation to other variables, including convergence patterns with instruments that assess related constructs, evidence of construct validity (internal structure) (Trevisan, Berberian et al., 2020) and evidence of validity of competing criteria with a clinical sample (Trevisan, Dias et al., 2020). After its construction, the first version of the new instrument was submitted to judges. Items that showed 80% or more concordance between evaluators were not modified. The others, which were distributed relatively evenly among the five constructs evaluated, were carefully reviewed. This procedure is essential, since, the judges, who are experts in the area, are in a better position to provide an analysis about the representativeness of the content that the items intend to evaluate.

After the first revision of the instrument based on the analysis of the judges, the pilot study also made it possible to identify some areas of difficulty and make new adjustments. Accordingly, this first study with IFERA-II resulted in a relatively broad instrument in terms of the measured constructs, based on theoretical models that allow a relatively comprehensive understanding of the functioning of EFs and related abilities. Other functional instruments for assessing the adult population are being studied in the national context and, despite the recent publication of BDEFS (Barkley, 2018), tools such as IFERA-II are still scarce. Furthermore, among the existing instruments, EFs are the dimension primarily investigated, to the detriment of DA and RS, which have also been shown to be relevant, for example, to the understanding of the neuropsychology of ADHD (Wagner et al., 2016). Considering the criticism regarding the length of some of these scales (Thorell & Catale, 2014), IFERA-I and IFERA-II can be considered relatively brief instruments, enabling their use in screening processes.

By developing and presenting a new functional measure, the present study aimed to contribute to the discussion on the ecological validity of EFs measures (Garcia-Barrera et al., 2014; Isquith et al., 2013; Roth et al., 2014; Viklund et al., 2019; Zimmermann et al., 2014) and to minimize the current limitations regarding the availability of these measures, which creates some obstacles for the multi-method evaluation of EFs, as suggested by several authors (Garcia-Barrera et al., 2014; Isquith et al., 2013). From this study, the final version of IFERA-II was created, and new investigations are underway to verify its psychometric properties, including information about its reliability and evidence of validity (convergence with other variables and external criteria). The findings of this study indicate evidence of content validity and relevance of the items of IFERA-II, allowing the continuation of investigations for the future availability of this tool.

Final Considerations

The study described the construction process of IFERA-II, which included adaptation of items from IFERA-I version, investigation of evidence of content validity through analysis of expert judges and a pilot study, through the application of the instrument with a group of volunteers. The result of this process was an instrument with evidence of content validity and satisfactory quality of items and examples. From this stage, based on the solid theoretical foundation and methodological rigor that permeated the construction of the instrument, future research should advance in the study of its psychometric properties.

References

• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
• American Psychiatric Association. (2014). Manual diagnóstico e estatístico de transtornos mentais (5ª ed). Artmed.
• Baggetta, P., & Alexander, P. A. (2016). Conceptualization and operationalization of executive e function. Mind, Brain, and Education, 10(1), 10-33. https://doi.org/10.1111/mbe.12100
  » https://doi.org/10.1111/mbe.12100
• Barkley, R. A. (2014). The assessment of Executive Functioning using the Barkley Deficits in Executive Functioning Scales. In S. Goldstein & J. A. Naglieri (Eds.), Handbook of Executive Functioning (pp. 245-264). Springer Science and Business Media.
• Barkley, R. A. (2018). BDEFS - Escala de Avaliação de Disfunções Executivas de Barkley [BDEFS - Barkeley Deficits in Executive Funcioning Scale]. Hogrefe.
• Barkley, R. A., & Murphy, K. R. (2010). Impairment in Occupational Functioning and Adult ADHD: The predictive utility of Executive Function (EF) Ratings versus EF Tests. Archives of Clinical Neuropsychology, 25(3), 157-173. https://doi.org/10.1093/arclin/acq014
  » https://doi.org/10.1093/arclin/acq014
• Berenguer, C., Rosello, B., & Leader, G. (2018). A review of executive functions in autism spectrum disorder and attention deficit hyperactivity disorder.Journal of Educational and Developmental Psychology ,8(2), 107-119. http://doi.org/10.5539/jedp.v8n2p107
  » http://doi.org/10.5539/jedp.v8n2p107
• Carim, D. B., Miranda, M. C., & Bueno, O. F. A. (2012). Tradução e adaptação para o português do Behavior Rating Inventory of Executive Function - BRIEF. Psicologia: Reflexão e Crítica , 25(4), 653-661. https://doi.org/10.1590/S0102-79722012000400004
  » https://doi.org/10.1590/S0102-79722012000400004
• Coghill, D., Toplak, M., Rhodes, S., & Adamo, N. (2018). Cognitive functioning in ADHD. In T. Banaschewski, D. Coghill & A. Zuddas (Eds.), Oxford Textbook of Attention Deficit Hyperactivity Disorder (pp. 94-102). Oxford University Press.
• Diamond, A. (2013). Executive functions. Annual Reviews of Psychology, 64, 135-168. https://doi.org/10.1146/annurev-psych-113011-143750
  » https://doi.org/10.1146/annurev-psych-113011-143750
• Dias, N. M., Trevisan, B. T., Leon, C. B. R., Prust, A. P., & Seabra, A. G. (2017). Can executive functions predict behavior in preschool children? Psychology & Neuroscience, 10(4), 383-393. https://doi.org/10.1037/pne0000104
  » https://doi.org/10.1037/pne0000104
• Dias, N. M., Prust, A. P., & Seabra, A. G. (2020). Investigação longitudinal da contribuição das funções executivas na predição de leitura e aritmética no início do Ensino Fundamental [Manuscrito submetido para publicação]. Programa de Mestrado e Doutorado em Distúrbio do Desenvolvimento, Universidade Presbiteriana Mackenzie.
• Garcia-Barrera, M. A., Duggan, E. C., Karr, J. E., & Reynolds, C. R. (2014). Examining executive functioning using the Behavior Assessment System for Children (BASC). In S. Goldstein & J. A. Naglieri (Eds.), Handbook of Executive Functioning (pp. 283-300). Springer Science and Business Media.
• Holst, Y., & Thorell, L. B. (2018). Adult Executive Functioning Inventory (ADEXI): Validity, reliability, and relations to ADHD.International journal of methods in psychiatric research , 27(1), e1567. https://doi.org/10.1002/mpr.1567
  » https://doi.org/10.1002/mpr.1567
• Isquith, P. K., Roth, R. M., & Gioia, G. (2013). Contribution of rating scales to the assessment of executive functions. Applied Neuropsychology: Child, 2(2), 125-132. https://doi.org/10.1080/21622965.2013.748389
  » https://doi.org/10.1080/21622965.2013.748389
• Macuglia, G. R., Almeida, R. M. M. de, Santos, F. C., & Giacomoni, C. H. (2016). Behavioural Assessment of the Dysexecutive Syndrome (BADS): Adaptação e evidências de validade [Behavioral Assessment of the Dysexecutive Syndrome (BADS): Adaptation and Validity Evidence]. Psico-USF, 21(2), 219-232. https://doi.org/10.1590/1413-82712016210201
  » https://doi.org/10.1590/1413-82712016210201
• Moffitt, T. E., Arseneault, L., Belsky, D., Dickson, N., Hancox, R. J., Harrington, H., Houts, R., Poulton, R., Roberts, B. W., Ross, S., Sears, M. R., Thomson, W. M., & Caspi, A. (2011). A gradient of childhood self-control predicts health, wealth, and public safety. Proceedings of the National Academy of Sciences, 108(7), 2693-2698. https://doi.org/10.1073/pnas.1010076108
  » https://doi.org/10.1073/pnas.1010076108
• Naglieri, J. A., & Goldstein, S. (2014). Assessment of executive function using rating scales: Psychometric considerations. In S. Goldstein & J. A. Naglieri (Eds.), Handbook of Executive Functioning (pp. 159-170). Springer Science and Business Media.
• Pasquali, L. (2010). Instrumentação psicológica: Fundamentos e práticas Artmed.
• Peixoto, E. M., & Ferreira-Rodrigues, C. F. (2019). Propriedades psicométricas dos testes psicológicos. In M. Baptista, M. Muniz, C. Reppold, C. H. Nunes, L. Carvalho, R. Primi, A. P. Noronha, A. Seabra, S. Wechsler, C. Hutz, & L. Pasquali (Eds.), Compêndio de Avaliação Psicológica (pp. 29-39). Vozes.
• Rabin, L. A., Fogel, J., & Nutter-Upham, K. E. (2011). Academic procrastination in college students: The role of self-reported executive function. Journal of Clinical and Experimental Neuropsychology, 33(3), 344-357. https://doi.org/10.1080/13803395.2010.518597
  » https://doi.org/10.1080/13803395.2010.518597
• Reynolds, B. W., Basso, M. R., Miller, A. K., Whiteside, D. M., & Combs, D. (2019). Executive function, impulsivity, and risky behaviors in young adults.Neuropsychology, 33(2), 212-221.https://doi.org/10.1037/neu0000510
  » https://doi.org/10.1037/neu0000510
• Roth, R. M., Isquith, P. K., & Gioia, G. A. (2014). Assessment of executive functioning using the Behavior Rating Inventory of Executive Function (BRIEF). In S. Goldstein & J. A. Naglieri (Eds.), Handbook of Executive Functioning (pp. 301-332). Springer Science and Business Media.
• Seabra, A. G., Muniz, M., Reppold, C. T., Dias, N. M., Siquara, G., Tourinho, A. M. O., Gurgel, L., & Teixeira, L. (2014). Funções executivas e desempenho escolar. In A. G. Seabra, J. A. Laros, E. C. Macedo, & J. N. Abreu (Eds.), Inteligência e funções executivas: avanços e desafios para a avaliação neuropsicológica (pp. 211-225). Memnon.
• Sergeant, J. (2000). The cognitive-energetic model: An empirical approach to attention-deficit hyperactivity disorder. Neuroscience and Biobehavioral Reviews, 24(1), 7-12.
• Sergeant, J. A. (2005). Modeling attention-deficit/hyperactivity disorder: A critical appraisal of the cognitive-energetic model. Biological Psychiatry, 57(11), 1248-55. https://doi.org/10.1016/j.biopsych.2004.09.010
• Silverstein, M. J., Faraone, S. V., Leon, T. L., Biederman, J., Spencer, T. J., & Adler, L. A. (2018). The relationship between executive function deficits and DSM-5-Defined ADHD Symptoms.Journal of attention disorders, 24(1), 41-51. https://doi.org/10.1177/1087054718804347
  » https://doi.org/10.1177/1087054718804347
• Sonuga-Barke, E. J. S. (2005). Causal models of attention-deficit/hyperactivity disorder: From common simple deficits to multiple developmental pathways. Biological Psychiatry, 57(11), 1231-1238. https://doi.org/10.1016/j.biopsych.2004.09.008
  » https://doi.org/10.1016/j.biopsych.2004.09.008
• Taghizadeh, H., Soltani, A., Manzari, T. H., & Zeinaddiny, M. Z. (2017). The structural model of the role of executive functions in learning performance of students with specific learning disabilities. Journal of child mental, 4(2), 25-36.
• Thorell, L. B., & Catale, C. (2014). The assessment of Executive Functioning using the Childhood Executive Functioning Inventory (CHEXI). In S. Goldstein & J. A. Naglieri (Eds.), Handbook of Executive Functioning (pp. 359-366). Springer Science and Business Media.
• Toplak, M. E., West, R. F., & Stanovich, K. E. (2013). Practitioner review: Do performance-based measures and ratings of executive function assess the same construct? Journal of Child Psychology and Psychiatry, 54(2), 131-143. https://doi.org/10.1111/jcpp.12001
  » https://doi.org/10.1111/jcpp.12001
• Trevisan, B. T., Berberian, A. A., Dias, N. M., & Seabra, A. G. (2020). Development and psychometric properties of the IFERA-I [Manuscript submitted]. Master and Doctoral Program of Developmental Disorders, Mackenzie Presbyterian University.
• Trevisan, B. T., Dias, N. M., Berberian, A. A., & Seabra, A. G. (2020). Psychometric properties of the IFERA-I in a sample of children with ADHD [Manuscript submitted]. Master and Doctoral Program of Developmental Disorders, Mackenzie Presbyterian University.
• Trevisan, B. T., Dias, N. M., & Seabra, A. G. (2016). Inventário de Dificuldades em Funções Executivas, Regulação e Aversão ao Adiamento para Adultos (IFERA-II)[ Difficulties in Executive Functions, Regulation and Delay Aversion Inventory - Version for adults ] [not published inventory].. Universidade Presbiteriana Mackenzie.
• Trevisan, B. T., & Seabra, A. G. (2014). Inventário de Dificuldades em Funções Executivas, Regulação e Aversão ao Adiamento - Versão para crianças e adolescentes (IFERA-I) [Difficulties in Executive Functions, Regulation and Delay Aversion Inventory - Version for children and adolescents IFERA I] [not published inventory]. Universidade Presbiteriana Mackenzie.
• Viklund, H., Ausén, B., Hagman, G., & Thorell, L. B. (2019). The Executive Checklist (EC-10)- a new rating instrument for clinicians assessing dysexecutive behavior.Applied Neuropsychology: Adult, 1-10. https://doi.org/10.1080/23279095.2019.1660881
  » https://doi.org/10.1080/23279095.2019.1660881
• Villegas, A. L. R., & Cruz, J. S. (2015). Executive functioning and adaptive coping in healthy adults. Applied Neuropsychology: Adult, 22(2), 124-131. https://doi.org/10.1080/23279095.2013.864972
  » https://doi.org/10.1080/23279095.2013.864972
• Wagner, F., Rohde, L. A. D., & Trentini, C. M. (2016). Neuropsicologia do transtorno de déficit de atenção/hiperatividade: Modelos neuropsicológicos e resultados de estudos empíricos. [Neuropsychology of Attention-Deficit/Hyperactivity Disorder: Neuropsychological Models and Results of Empirical Studies]. Psico-USF , 21(3), 573-582. https://doi.org/10.1590/1413-82712016210311
  » https://doi.org/10.1590/1413-82712016210311
• Zelazo, P. D. (2015). Executive function: Reflection, iterative reprocessing, complexity, and the developing brain.Developmental Review, 38, 55-68. https://doi.org/10.1016/j.dr.2015.07.001
  » https://doi.org/10.1016/j.dr.2015.07.001
• Zimmermann, N., Cardoso, C. de O., Kochhann, R., Jacobsen, G., & Fonseca, R. P. (2014). Contributions of the ecological approach to the neuropsychology of executive functions. Temas em Psicologia, 22(3), 639-654. https://doi.org/10.9788/TP2014.3-09
  » https://doi.org/10.9788/TP2014.3-09

• *
  Apoio: CNPq Research Productivity Scholarship (A. G. Seabra, L. R. R. Carreiro e N. M. Dias).

Publication Dates

• Publication in this collection
  08 Feb 2021
• Date of issue
  2021

History

• Received
  16 Jan 2019
• Reviewed
  13 Mar 2020
• Accepted
  29 Apr 2020