Development and validation of an instrument (CSII – Brazil) to assess users’ conceptual and procedural knowledge of continuous subcutaneous infusion systemsABSTRACT
Objective: To develop, adapt and validate an instrument named “CSII – Brazil” to assess users’ conceptual and procedural knowledge of continuous subcutaneous insulin infusion systems.
Materials and methods: Methodological and exploratory study developed in three stages: a) instrument development; b) content validation and cultural adaptation (evaluation by a committee of experts and pre-test with CSII users); c) psychometric validation through instrument application in a sample of 60 patients by means of the web tool e-Surv. Internal consistency and reproducibility analyses were performed within IBM SPSS Statistics 20 programming environment.
Results: The 16 multiple-choice question instrument successfully attained a content validity index of 0.97, showing satisfactory internal consistency, with 0.61 Cronbach's alpha [95% CI 0.462-0.746] and an intraclass correlation coefficient of 0.869 [95% CI: 0.789-0.919] between the test and retest scores.
Conclusion: The CSII – Brazil instrument is considered adequate and validated to assess continuous subcutaneous infusion system users’ conceptual and procedural knowledge.
Keywords Diabetes mellitus; insulin infusion system; knowledge; cultural adaptation; validation studies
INTRODUCTION
Treatment for type 1 diabetes mellitus (T1D) has made significant progress over the past 50 years, mainly with the advent of the continuous subcutaneous insulin infusion (CSII) in the 90's, which allows for the patient to obtain more accurate doses of insulin, comfort and safety (1).
A number of patients with T1D can be candidates for CSII use, but for successful treatment, users need to be the target of continuous training and monitoring regarding proper use of the system (2).
CSII users must be instructed about basic concepts related to terms appearing in the device menu and messages prompted by patient's use (conceptual knowledge) as well as essential actions to be performed to use the device (procedural knowledge), such as to how to insert and connect the infusion system and how to set up basic parameters such as time, date, basal insulin doses, food and glycemic correction boluses. Users are also required to learn how to program advanced settings (such as temporary basal rates, extended/dual/square wave), be capable to identify alarms and solve the most common problems involving CSII use, such as cannula or catheter occlusion and non-administration of insulin. Furthermore, users must also be instructed on how to properly dispose of waste yielded by supplies in proper packaging (3).
CSII is a high-cost and delicate management treatment, but it offers numerous resources for users who manage to learn how to make use of it. Training and education by healthcare providers is thus a constant need in order to avoid the risk of poor glycemic control, diabetic ketoacidosis, lipodystrophy and non-adherence to treatment (4).
An instrument to assess patients’ knowledge and use of CSII is an excellent tool in diabetes education, as it allows a more objective way to evaluate users’ understanding of how the system works and how to use it. In order to ensure instrument quality and reliability, validity must be measured (5). An instrument for this purpose has not been found in the literature.
The aim of this study was to develop, culturally adapt, and validate an instrument to assess users’ conceptual and procedural knowledge of CSII (CSII – Brazil).
MATERIALS AND METHODS
This is a methodological and exploratory study carried out from July 2017 to August 2019 in the city of Belo Horizonte, State of Minas Gerais, Brazil. The project was approved by the Ethics and Research Committee Involving Human Beings (CAAE number 65656117.6.1001.5138) at Santa Casa of Belo Horizonte Hospital. Agreement to participate in the study was obtained by using a Free Informed Consent Form signed by participants upon accessing an electronic questionnaire within the e-Surv webtool.
Stage 1 – Instrument development
Based on national and international T1D guidelines, in addition to user manuals and instructions provided by different CSII models and brands, a panel of researchers with expertise in diabetes, comprising a dietitian, a nurse and two endocrinologists, developed the instrument and assessed all stages until the final version (6–10).
Specific topics on CSII use were identified for testing in the questionnaire and a first version (V1) was drafted with 17 questions. The instrument was designed to be brief and a self-administered questionnaire (Figure 1).
In establishing a general conceptual structure, the instrument was developed in two parts: the first one made up of yes or no questions aimed to assess self-reported basic conceptual knowledge about T1D concepts which are referred to in the device menu and messages prompted by patient's use. These comprised: sensitivity factor, insulin-carbohydrate ratio, glycemic objective, active insulin time and others. The reason for having this first part was the need to assess whether users report being able to understand basic concepts of their treatment related to the terms used to name them in the device.
The second part comprised questions on user procedural knowledge about CSII therapy, ranging from dealing with basic systems (technique and regularity of the exchange of supplies, adjustment of date and time, administration of basal doses and bolus of insulin, correct disposal of supplies and special bolus programming, temporary basal rates and other advanced CSII settings).
Stage 2 – Assessment by expert committee
For content validation, a link to access the instrument on the web platform e-surv was sent by e-mail to 27 experts (labelled as Judges Committee), among them endocrinologists (n = 10), nurses (n = 3) and dietitians (n = 7) with experience in CSII therapy, as well as applied linguists (n = 7) from different regions of Brazil, who assessed each item in V1 and rated them with the following options: One star standing for need for full reformulation; Two stars, partial reformulation (substantial revision needed); Three stars, need for partial reformulation, with minor editing to enhance text style; and Four stars – no need for reformulation. A comment box was also provided for the experts’ considerations and suggestions.
Once assessment was concluded, the Content Validity Index (CVI – the level of agreement of experts on adequacy of the items) was computed: the number of scores 3 and 4, divided by the total number of scores by all members of the committee. Results greater than or equal to 0.78 being considered acceptable (11).
After reviewing V1, following experts’ suggestions, 2 questions were merged into single one, since they dealt with the same topic (general alarm), and version 2 (V2) was obtained, with 16 questions.
Stage 3 – Cultural adaptation
Cultural adequacy of V2 was verified through a pre-test (face-to-face) with 10 CSII users of a public diabetes center. Each participant was requested to read the whole instrument; subsequently, the participant was asked to point out whether the items were clear, accurate, and relevant. The participant's feedback was discussed by the panel of researchers responsible for developing the questions, who deemed all comments relevant and reformulated those items that reached less than 80% of agreement (12). The reformulated items were subsequently tested in the same group of participants.
When there was no need for further reformulation, either in the number of questions or in their content, and thus no need for a new assessment by the researchers, version (V3) was obtained, which was considered culturally adequate to be submitted to psychometric validation.
Stage 4 – Adaptation for web access
At this stage, an electronic version was prepared on the web platform e-Surv. Five CSII users recruited at a public diabetes center received a link to access and answer a version of the instrument on the Web, to test its use and describe their experience. No technical difficulty in accessing and answering the questionnaire via the e-Surv platform was reported.
Stage 5 – Validation
A link to access an electronic version of the instrument on the e-Surv platform was sent via e-mail to 103 patients referred by endocrinologists from public and private diabetes centers from Belo Horizonte, Minas Gerais, Brazil. Exclusion criteria were patients who were under 18 years old and patients who did not answer or answered incorrectly the questions in the first part of the instrument. A retest was performed in the final sample of 60 patients (after exclusions), with a minimum interval of 7 days and a maximum of 21 days between the tests (13).
To assess the number of correct answers, a score of one to three was assigned to each alternative in each question. The coding of participant responses was based on an increasing order, with the lowest score being assigned to the wrong answer and the highest one to the correct answer. The instrument has twelve three-alternative multiple choice questions and four two-alternative multiple choice questions. The maximum expected score if respondent answers all questions correctly is 44 whereas incorrect answers in all questions yields a minimum 16.
Statistical analysis
Absolute and relative frequencies were used to describe the sample characteristics and the proportion of correct answers. Internal consistency and reproducibility were verified to analyze the reliability of the construct. Cronbach's alpha (CA) was used to assess the internal consistency of the instrument.
The instrument reproducibility was evaluated through test-retest (temporal stability), computing the intraclass correlation coefficient (ICC) and Kappa index were added to the percentage of concordant responses in the test and retest, defined as the ratio between the number of individuals who selected the same answer (regardless of being correct or incorrect) at both test and retest and the total number of individuals (11,12). Floor and ceiling effects were measured by the number of respondents receiving the minimum and maximum scores, respectively.
The significance level adopted for the statistical tests was 5%. For data analysis, SPSS version 20.0 was used.
RESULTS
Instrument development
This stage spanned 6 months and consisted in bibliographic search and review and meetings by a panel of researchers. A first version of the instrument made up of 25 questions was drafted, which were eventually reduced, as 7 questions were discarded and 2 of them merged into a single one, for the sake of clarity and convenience. V1 was finalized with 17 questions, which became 16 in V2 and V3, with a number of alternatives per question ranging from 2 to 3 (totaling 44 items – each alternative being considered an item, for the purposes of statistical analysis).
All steps described in the literature were followed in the validation process.
Assessment by expert committee
Of the 28 members in the expert committee, 35.7% were physicians, 25% dietitians, 25% applied linguists, and 10.7% nurses. Among healthcare professionals, most (71.4%) had treated patients who had used or were using CSII (Table 1).
The instrument achieved good scores by the committee regarding clarity and relevance, with a total CVI of 0.97. For item relevance, the average CVI was 0.99. For clarity, the index was 0.95 (Table 2).
Validation
Among the 60 patients who answered the final version of the instrument (test and retest), age ranged from 18 to 82 years old (36.2 ± 12.43 years) and gender was predominantly female. All patients in the sample had attended elementary school and most of them reported a monthly income of over five times the minimum wage in Brazil. 21,6% of them had purchased their CSII themselves (Table 3). Due to its homogeneity, the instrument was considered as unidimensional. In this context, the total CA alpha value was 0.61 (95% CI: 0.462-0.746). Floor effects (percent with minimum score) were 0%, and ceiling effects (percent with maximum score) 7%.
An ICC value of 0.869 (95% CI: 0.789-0.919) was obtained. The Kappa coefficient, which assesses the degree of agreement, varied between 0.5-1.0 (mean: 0.80). When the alpha absence index was calculated, there was a slight impact on reducing AC and no questions were removed (Table 4). The instrument is available in Supplementary Material.
Correlation between test and retest answers agreement percentage and Cronbach's alpha coefficient for CSII – Brazil
The average final score during test was 39.4 (31-44 marks). 34 participants (56.6%) scored over 40 (90% correct answers).
Among the 16 questions in the instrument, four of them had a success rate greater than 90%. They deal with the topics: adjusting time and date of the pump whenever necessary, observing information prompted by alarms, eliminating air bubbles in the infusion set when changing disposables, and rotating infusion sites.
Four questions showed less than 70% correct answers. The topics dealt with in them were: measures in case of occlusion alarm (63.3% answered that they performed the change of the entire infusion set); changing the infusion set (63.3% answered that they changed the set every 3 days); special bolus programming (58.3% of patients showed knowing how to program) and disposal location for the infusion set, syringes and needles (only 33.3% of patients performed correct disposal in a rigid plastic container or in a suitable packaging for discard).
DISCUSSION
CSII is an expensive therapy, not very accessible in Brazil, but a promising one with numerous resources for T1D management, which merits research towards its dissemination. It represents a treatment option that must be carefully prescribed in order to guarantee successful results.
Implementing an instrument with psychometric qualities tested for the clinical practice of diabetes educators will enable a more efficient direction in patient follow-up. In addition to contributing to education of CSII users in the Brazilian cultural context, the design and development of such an instrument can be of fundamental importance in the identification of flaws in relation to the practices oriented to the use of CSII. This evaluation, in turn, will make it possible to readjust the teaching-learning process, so that the use of CSII can be optimized, guaranteeing the benefits that this system can bring to the user.
The collaborative work by an interdisciplinary team (a panel of researchers with expertise in diabetes and CSII) to develop the instrument made it possible to clarify and solve problems encountered during the process of drafting the items, in addition to allowing the adaptation of concepts and terms to the language used by the target subjects.
Assessment by an expert committee through the web tool e-Surv proved a reliable and efficient methodology (14). This furthered collaboration between healthcare and applied linguistics experts towards adaptation of the instrument, in addition to offering experts a channel to provide suggestions.
The instrument scored successfully in terms of agreement within the expert committee regarding clarity and relevance, with an excellent CVI (0.97). It should be noted that the maximum value for CVI is equal to 1, so the results obtained are close to the maximum evaluation limit (11).
A face-to-face test with a small group of people (15) proved successful. In carefully developed instruments, two or three face-to-face tests are usually sufficient, which was confirmed in our sample, which required two rounds of tests (16).
As regards internal consistency, a CA index of 0.61 was obtained. A CA index above 0.5 is acceptable when the sample is between 25 to 50 participants; in a basic investigation or exploratory research, a value of around 0.60 would suffice to meet acceptability (17–19). Several studies report interferences to which the CA index is subject, which demands caution in its interpretation. It is worth noticing that instruments with a large number of items or samples have higher CA values (20–22).
The authors believe that an adequate CA index was achieved, considering the context in which the study was performed, with a restricted sample due to the fact that the therapeutics is still poorly available in our country (12).
The CA absence index was carried out, but no questions were excluded, due to the small difference that would result in the final CA. As ours is a brief instrument, with a reduced number of questions pertaining to a single domain, removing a question would have a negative impact; thus, all of them were considered essential by the panel of experts.
The results showed good stability of the instrument, with a high level of agreement between test and retest (all questions showed more than 80% agreement). The ICC (0.869) and the Kappa index (mean 0.80) proved that the instrument showed stability, reproducibility and confidence (11).
No studies were found describing the development and validation of instruments to assess the knowledge of CSII users, which did not allow for our results to be compared.
The first part of our instrument seeks to elicit users’ self-reported conceptual knowledge deemed necessary for a successful treatment with insulin. In the case of CSII, it is important that users be familiar with terms that the system displays as messages. Assessing clear understanding of these terms is then essential to adequate use of the system.
For the validation stage, patients who scored poorly in part 1 were excluded for retest. Daily clinical practice recommends that, whenever a patient is in doubt or does not know how to answer a question, administration of instruments must be interrupted so that the patient's doubt is clarified, and the instrument can be newly applied after clarification. This is seen as a moment to improve or revise concepts that may have not be consolidated.
The second part of the instrument was meant to assess CSII users’ procedural knowledge regarding decisions they are required to make when using a CSII.
The instrument is intended to be self-administered and in patient familiar settings, such as a physician's waiting room. Feedback provided by the instrument allows endocrinologists to identify the difficulties faced by a patient and, even during clinical consultation.
When we analyzed the performance of the patients in the test, the question that obtained the lowest rate of correct answers did not surprise us; this was related to proper handling or correct disposal of sharps. A solution to incorrect behavior by T1D patients is to instruct and train them (23,24).
It was also expected that the question about the user's ability to deal with special boluses (double/square, wave/prolonged) would have the lowest hit rates, as this is not frequently addressed in patient education or training.
Currently in Brazil we are experiencing constant delays in the delivery of supplies, and while developing this instrument we have noticed how patients make wrong decisions due to this, especially regarding inappropriate use of catheters and cannulas for longer than recommended. The risk of complications such as infections and interruption of insulin administration increases with this poor practice.
As regards other items in the instrument, participants’ performance was homogeneous and with good success rates; however, there are some gaps in the skills of these patients, which points to the need to intensify care and education.
In conclusion, the instrument proved useful, reliable and stable for application to users with CSII to assess conceptual and procedural knowledge with a view to enhancing health care practices.
-
Ethics approval and consent to participate: The study was approved by the Research Ethics Committee of Santa Casa De Belo Horizonte Group (CAAE number 65656117.6.1001.5138). Agreement to participate in the study was obtained by using a Free Informed Consent Form signed by participants upon accessing an electronic questionnaire within the e-Surv webtool.
-
Funding: not applicable.
Acknowledgments:
the authors would like to thank all the expert committee participants, who have provided valid suggestions for making the instrument easier to patient understanding and useful for a longer time. We also thank colleagues who referred potential participants, and patients who answered the questionnaire and shared their stories with us.
REFERENCES
- 1 Janež A, Guja C, Mitrakou A, Lalic N, Tankova T, Czupryniak L, et al. Insulin Therapy in Adults with Type 1 Diabetes Mellitus: a Narrative Review. Diabetes Ther. 2020;11(2):387-409.
- 2 Bode BW, Davidson PC, Fredrickson LP, Gross TM, Sabbah HT. Diabetes management in the new millennium using insulin pump therapy. Diabetes Metab Res Rev. 2002;18 (Suppl 1):S14-20.
- 3 Selvan C, Ghosh S, Mukhopadhyay S. Insulin pump – A review. J Indian Med Assoc. 2013;111(11):746-50.
- 4 Jackisch BI, Wagner VM, Heidtmann B, Lepler R, Holterhus PM, Kapellen TM, et al. Comparison of continuous subcutaneous insulin infusion (CSII) and multiple daily injection (MDI) in pediatric type 1 diabetes: a multicentre matched-pair cohort analysis over 3 years. Diabet Med. 2008;25(1):80-5.
- 5 Rattray J, Jones MC. Essential elements of questionnaire design and development. J Clin Nurs. 2007;16(2):234-43.
- 6 Chiang JL, Kirkman MS, Laffel LMB, Peters AL. Type 1 diabetes through the life span: A position statement of the American Diabetes Association. Diabetes Care. 2014;37(7):2034-54.
-
7 Diretrizes da Sociedade Brasileira de Diabetes 2019-2020. Disponível em: https://www.diabetes.org.br/profissionais/images/DIRETRIZES-COMPLETA-2019-2020.pdf
» https://www.diabetes.org.br/profissionais/images/DIRETRIZES-COMPLETA-2019-2020.pdf - 8 Silveira D, Angelo T, Pinheiro RM, Gratieri T. Infusão subcutânea contínua de insulina: cenário nacional e internacional. Cad Saúde Coletiva. 2016;24(4):496-501.
- 9 Umpierrez GE, Klonoff DC. Diabetes technology update: Use of insulin pumps and continuous glucose monitoring in the hospital. Diabetes Care. 2018;41(8):1579-89.
- 10 Reece SW, Hamby Williams CL. Insulin Pump Class: Back to the Basics of Pump Therapy. Diabetes Spectr. 2014; 27(2):135-40.
- 11 Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciênc Saúde Coletiva. 2011;16(7):3061-8.
- 12 Coluci MZO, Alexandre NMC, Milani D. Construção de instrumentos de medida na área da saúde. Ciênc Saúde Coletiva. 2015;20(3):925-36.
- 13 Souza AC, Alexandre NMC, Guirardello EB. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saúde. 2017;26(3):649-59.
- 14 Torres HC, Chaves FF, Silva DDR, Bosco AA, Gabriel BD, Reis IA, et al. Tradução, adaptação e validação de conteúdo do Diabetes Medical Management Plan para o contexto brasileiro. Rev Latino-Am Enfermagem. 2016;24:e2740.
- 15 Pasquali L. Princípios de elaboração de escalas psicológicas. Rev Psiquiatr Clín (São Paulo). 1998;25(5):206-2013.
- 16 Martins GA. Sobre confiabilidade e validade. Rev Bras Gest Negocios. 2006;8(20):1-12.
- 17 Davis FB. Educational measurements and their interpretation. Belmont, California: Wadsworth Publishing Co.; 1964.
- 18 DeVellis RF. Scale development: Theory and applications. Newbury Park, CA: SAGE Publications; 1991.
- 19 Maroco J. Qual a fiabilidade do alfa de Cronbach? Questões antigas e soluções modernas? Laboratório Psicol. 2006;4(1):65-90.
-
20 Streiner DL. Statistical Developments and Applications. Being Inconsistent About Consistency: When Coefficient Alpha Does and Doesn't Matter. Streiner Scales and Indexes. 2003;80(3):217-22. Available from: https://www.rotman-baycrest.on.ca/files/publicationmodule/@random45f5724eba2f8/JPersAssess03_80_217_222.pdf
» https://www.rotman-baycrest.on.ca/files/publicationmodule/@random45f5724eba2f8/JPersAssess03_80_217_222.pdf - 21 Brown JD. The Cronbach alpha reliability estimate. Shiken JALT Test Eval SIG Newsl. 2002;6(1):17-8.
- 22 Fan X, Thompson B. Confidence Intervals for Effect Sizes. Educ Psychol Measurement. 2001;61(4):517-31.
- 23 Cunha GH, Barbosa RVA, Fontenele MSM, Lima MAC, Franco KB, Fechine FV. Insulin therapy waste produced in the households of people with diabetes monitored in Primary Care. Rev Bras Enferm. 2017;70(3):618-25.
- 24 Tapia CEV. Diabetes mellitus e o descarte de seringas e agulhas. Rev Gaúcha Enferm 2009;30(2):228-34.
Supplementary Material
INSTRUMENTO SICI – BRASIL
Qual a marca e o modelo da bomba de Insulina que você utiliza? Marca______________________ Modelo______________________
Publication Dates
-
Publication in this collection
21 Dec 2020 -
Date of issue
Feb 2021
History
-
Received
27 Apr 2020 -
Accepted
26 Sept 2020
