Open-access Effects of educational technologies on the prevention and treatment of diabetic ulcers: A systematic review and meta-analysis*

Objective:  to analyze the effects of educational technologies in the prevention and treatment of diabetic ulcers.

Method:  a systematic review conducted in seven databases, a bibliographic index, an electronic library and the Gray Literature. The sample consisted of 11 randomized controlled clinical trials. The synthesis of the results was descriptive and through meta-analysis.

Results:  the predominant educational technologies were training sessions and verbal guidelines, with soft-hard technologies standing out. When compared to usual care, the educational technologies presented a protective factor to prevent the incidence of diabetic ulcers (RR=0.40; 95% CI=0.18-0.90; p=0.03) and the certainty of the evidence assessment was low. The educational technologies also had a protective factor to prevent the incidence of lower limb amputations (RR=0.53; 95% CI=0.31-0.90; p=0.02) and certainty of the evidence was very low.

Conclusion:  soft-hard educational technologies such as structured verbal guidelines, educational games, lectures, theoretical-practical training sessions, educational videos, folders, serial albums and playful drawings, and hard technologies such as therapeutic footwear, insoles, infrared digital thermometer, foot care kits, Telemedicine app and mobile phone use, were effective for the prevention and treatment of diabetic ulcers, although more robust studies are required.

Descriptors: Diabetes Mellitus; Diabetic Foot; Diabetes Complications; Educational Technology; Systematic Review; Meta-Analysis


Objetivo:  analizar los efectos de las tecnologías educativas en la prevención y tratamiento de la úlcera diabética.

Método:  revisión sistemática realizada en siete bases de datos, un índice bibliográfico, una biblioteca electrónica y literatura gris. La muestra estuvo compuesta por 11 ensayos clínicos controlados aleatorizados. La síntesis de los resultados fue descriptiva y mediante metaanálisis.

Resultados:  las tecnologías educativas predominantes fueron la capacitación y la orientación verbal, se destacaron las tecnologías blandas-duras. En comparación con la atención habitual, las tecnologías educativas demostraron ser un factor protector para prevenir la incidencia de úlcera diabética (RR=0,40; IC 95%=0,18-0,90; p=0,03) y la evaluación de certeza de evidencia fue baja. Las tecnologías educativas también demostraron ser un factor protector para prevenir la incidencia de amputación en miembros inferiores (RR=0,53; IC 95%=0,31-0,90; p=0,02) y la certeza de evidencia fue muy baja.

Conclusión:  tecnologías educativas blandas-duras, como orientación verbal estructurada, juegos educativos, clase expositiva, capacitación teórico-práctica, video educativo, folder, rotafolio educativo y dibujos lúdicos, y tecnologías duras, como calzado terapéutico, plantillas, termómetro infrarrojo digital, kits para el cuidado de los pies, aplicación de telemedicina y teléfono móvil, resultaron efectivas para la prevención y el tratamiento de la úlcera diabética, sin embargo, es necesario que se realicen estudios más robustos.

Descriptores: Diabetes Mellitus; Pie Diabético; Complicaciones de la Diabetes; Tecnología Educacional; Revisión Sistemática; Metaanálisis


Objetivo:  analisar os efeitos das tecnologias educativas na prevenção e tratamento da úlcera diabética.

Método:  revisão sistemática conduzida em sete bases de dados, um índice bibliográfico, uma biblioteca eletrônica e na literatura cinzenta. A amostra foi constituída de 11 ensaios clínicos controlados randomizados. A síntese dos resultados foi descritiva e por meio de metanálise.

Resultados:  as tecnologias educativas predominantes foram os treinamentos e as orientações verbais, destacando-se as tecnologias leve-duras. Na comparação com o cuidado usual, as tecnologias educativas apresentaram fator de proteção para prevenção da incidência de úlcera diabética (RR=0,40; IC 95%=0,18-0,90; p=0,03) e a avaliação de certeza da evidência foi baixa. As tecnologias educativas também tiveram fator de proteção para prevenção da incidência de amputação em membros inferiores (RR=0,53; IC 95%=0,31-0,90; p=0,02) e a certeza da evidência foi muito baixa.

Conclusão:  as tecnologias educativas leve-duras, como orientações verbais estruturadas, jogos educativos, aula expositiva, treinamentos teórico-práticos, vídeo educativo, folder, álbum seriado e desenhos lúdicos, e as tecnologias duras, a exemplo do calçado terapêutico, palmilhas, termômetro digital de infravermelho, kits de cuidados com os pés, aplicativo de telemedicina e telefone móvel, foram efetivas para prevenção e tratamento da úlcera diabética, porém, estudos mais robustos são necessários.

Descritores: Diabetes Mellitus; Pé Diabético; Complicações do Diabetes; Tecnologia Educacional; Revisão Sistemática; Metanálise


Highlights:

(1) Educational technologies improved foot self-care.

(2) Educational technologies contributed to diabetic ulcer healing.

(3) Educational technologies were effective in preventing diabetic ulcers.

(4) Educational technologies presented a protective factor for amputation.

(5) It is recommended to use educational technologies in the prevention and treatment of diabetic ulcers.

Introduction

Diabetic ulcers are a health problem resulting from chronic complications of diabetes mellitus, such as peripheral neuropathy and peripheral arterial disease. Peripheral neuropathy causes protective sensitivity loss, foot deformity, joint mobility limitation and abnormal biomechanical load on the feet, leading to the formation of calluses, subcutaneous hemorrhage and ulceration. Usually caused by atherosclerosis, peripheral artery disease is a risk factor for poor healing of diabetic ulcers and for lower limb amputation. Thus, diabetic ulcers are classified as neuropathic, ischemic or neuroischemic(1).

Diabetic ulcers generate significant suffering and financial costs for the patients, in addition to overloading family members and health professionals and services, emphasizing the need for strategies that include elements of prevention, patient and team education, multidisciplinary treatment and rigorous monitoring(2). Diabetic ulcer treatment should include relief of plantar pressure, removal of calluses, protection and drainage of blisters, treatment of fungal infections, intervention to accelerate healing, foot self-care guidelines and management of peripheral artery disease, in order to reduce ulceration complications such as delays in the healing process, presence of infections and lower limb amputations(1).

In Spain, 44.1% of the patients with diabetes mellitus had neuroischemic ulcers, of which 20.3% were neuropathic and 20.3% were ischemic, with presence of infection as an aggravating factor in 41.4% of the cases(3). The cumulative incidence of diabetic ulcers in Japan was 0.2% at 12 months, 2.4% at 60 months and 5.8% at 120 months, and most of these patients did not return for reevaluations(4), highlighting the importance of care continuity and of implementing health education strategies to improve adherence to the therapy and prevent foot complications. A Brazilian study found that 1.9% of the patients with diabetes mellitus had diabetic ulcers, 59% had diabetic neuropathy, 69.6% were at risk of developing diabetic foot, and 86.3% of the patients reported never having undergone any clinical foot examination(5).

Diabetic ulcers can be caused by trauma, inappropriate shoes, mycotic infections, nail problems, calluses, dry skin and cracks(2,6). In addition to that, a study verified that patients with moderate knowledge about self-care practices were more likely to perform foot self-care, dry the interdigital spaces, moisturize the feet with creams and observe the presence of mycosis and ingrown toenails, when compared to those with insufficient knowledge(7).

Diabetic foot is a complication that requires thorough monitoring and behavioral changes. Thus, educational technologies can be effective in controlling diabetes mellitus, stimulating the promotion of foot care and, in the long term, they can enable a reduction in costs, foot complications and amputations(8). In this perspective, an educational intervention, with a practical skills session and foot care kit, reduced the risk factors for ulceration and improved the preventive behavior of foot self-care(6).

Training sessions, verbal guidelines, leaflets, apps, videos and didactic games are educational technologies that can be used for the health education of professionals and patients with diabetes mellitus. Thus, structured education, callus removal, use of therapeutic footwear and physical exercises related to the feet and mobility are beneficial to improve modifiable risk factors for foot ulcerations(9). In addition, the digital educational technology developed for nurses allows greater reach due to ease of access and to time, spatial and schedule flexibility, in addition to offering reduced costs. This educational strategy enables professional updating, qualification and training, contributing to the implementation of preventive interventions to reduce foot complications in patients with diabetes mellitus(8).

Educational technologies ease care management and, according to Merhy, they can be classified into soft, soft-hard and hard. Soft technologies consist of relationships such as welcoming, bonding and patient autonomy, through open dialog, qualified listening and group dynamics. Soft-hard technologies correspond to structured knowledge, such as serial albums, educational videos, pamphlets and posters. Hard technologies comprise material resources such as technological devices and registration forms(10).

The diverse evidence about the effects of educational technologies to prevent the incidence of diabetic ulcers and foot complications is indispensable to guide the clinical practice and incorporate these technologies in the care of patients with diabetes mellitus, in order to improve care quality, comprehensive assistance, foot self-care and the patients’ quality of life and satisfaction levels, in addition to reducing costs, hospital admissions and non-traumatic lower limb amputations.

In view of the above, this systematic review and meta-analysis aimed at analyzing the effects of educational technologies in the prevention and treatment of diabetic ulcers.

Method

Type of study

This is a systematic review and meta-analysis prepared according to the recommendations set forth in the Cochrane collaboration, based on the following stages: 1) Elaboration and registration of the systematic review protocol; 2) Delimitation of the guiding question; 3) Definition of the eligibility criteria; 4) Search and selection of studies; 5) Data collection; and 6) Synthesis and presentation of the systematic review results(11). The Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA) guidelines were adopted to draft the systematic review and meta-analysis report(12).

The review protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO), under number CRD42021287241(13).

Locus

The systematic review and meta-analysis was conducted in Teresina, capital city of Piauí, Brazil.

Period

The systematic review and meta-analysis took place from January to October 2022.

Research question

Formulation of the research question was delimited based on the PICOS acronym (P: Population or Patients; I: Intervention; C: Comparison; O: Outcomes; S: Study design), where P=Population (patients with diabetes mellitus), I=Intervention (educational technologies); C=Comparison (Control Group without receiving the intervention through educational technologies or receiving usual care), O=Outcomes (reduction in the incidence of ulcerations and diabetic ulcer complications) and S=Study design (randomized controlled clinical trials)(14). In the comparison group, usual care consisted of the routine assistance offered by the service, such as consultations, verbal guidelines, clinical foot examination and use of therapeutic shoes. Delays in the healing process, presence of infections and lower limb amputations were considered as diabetic ulcer complications. The expected outcomes were reduction of ulceration and amputation in the lower limbs, in addition to improvement in diabetic ulcer healing. Thus, the following guiding question was formulated: Which are the effects of educational technologies on the prevention and treatment of diabetic ulcers in patients with diabetes mellitus? In this perspective, the care measures in the healing of ulcerations and the assistance provided to complications related to diabetic ulcers were considered as treatment.

Eligibility criteria

The materials included were randomized controlled clinical trials that evaluated the effects of using educational technologies in the prevention and treatment of diabetic ulcers in patients with diabetes mellitus, without any time or language restrictions. The exclusion criteria were as follows: course completion papers, monographs, book chapters and materials that did not answer the guiding question. It is emphasized that randomized controlled clinical trials do not usually include the Gray Literature, that is, the one consisting of course conclusion papers, monographs and book chapters, representing an exclusion criterion in this study.

Bibliographic survey and search strategy

For the bibliographic survey, databases, a bibliographic index and an electronic library were consulted, namely: Medical Literature Analysis and Retrieval System on-line (MEDLINE via PubMed®); Cumulative Index to Nursing and Allied Health Literature (CINAHL-EBSCO); Web of ScienceTM; Scopus; Embase; Cochrane Central Register of Controlled Trials (CENTRAL Cochrane); Base de Dados em Enfermagem (BDENF); the Literatura Latino-Americana e do Caribe em Ciências da Saúde (LILACS) bibliographic index, via Biblioteca Virtual em Saúde (BVS); and the Scientific Electronic Library Online (SciELO) library. The searches were carried out on the Journals Portal of Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), through access to the Comunidade Acadêmica Federada (CAFe) of the Federal University of Piauí.

The search strategies were developed by combining controlled descriptors and keywords, using the “OR” and “AND” Boolean operators according to the particularities of each database, index or library. In this sense, the Medical Subject Headings (MeSH) controlled vocabulary was consulted to select the search terms in the MEDLINE via PubMed®, Web of ScienceTM, Scopus and CENTRAL Cochrane databases, based on the following search strategy: ((((“diabetes mellitus”[MeSH Terms]) OR (“diabetes”[All Fields])) AND ((((((((((“educational technology”[MeSH Terms]) OR (“instructional technology”[All Fields])) OR (“multimedia”[MeSH Terms])) OR (“health education”[MeSH Terms])) OR (“educational intervention”[All Fields])) OR (“education, distance”[MeSH Terms])) OR (“communications media”[MeSH Terms])) OR (“instructional film and video”[All Fields])) OR (“audiovisual aids”[MeSH Terms])) OR (“teaching materials”[MeSH Terms]))) AND (((((“foot ulcer”[MeSH Terms]) OR (“plantar ulcer”[All Fields])) OR (“diabetic foot”[MeSH Terms])) OR (“foot ulceration”[All Fields])) OR (“foot ulcer diabetic”[All Fields]))) AND ((((((“clinical trial”[Publication Type]) OR (“clinical trial”[All Fields])) OR (“controlled clinical trial”[Publication Type])) OR (“controlled clinical trial”[All Fields])) OR (“randomized controlled trial”[Publication Type])) OR (“randomized controlled trial”[All Fields])). In the other databases, bibliographic index and electronic library, the search strategies used were similar, and the CINAHL Headings controlled vocabulary was used in CINAHL-EBSCO, Emtree in Embase and the Descriptors in Health Sciences (Descritores em Ciências da Saúde, DeCS) in BDENF, LILACS and SciELO. The keywords were selected from the suggestions of the controlled vocabularies and thorough prior in-depth readings on the theme.

In order to contemplate the Gray Literature, secondary searches were carried out in the following sources: clinical trial registry websites, such as ClinicalTrials.gov (National Institutes of Health, NIH, USA) and The Brazilian Clinical Trials Registry (via the ReBEC Platform), the CAPES theses and dissertations catalog, the University of São Paulo (USP) digital theses and dissertations library portal and the DART-Europe E-Theses Portal. In addition to that, the lists of final references of the randomized controlled trials included were manually analyzed in order to find important studies to be added.

Selection of the studies was initially developed by two reviewers, independently and blindly, following the stages indicated in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 statement, namely: identification, screening and inclusion(12). The first step was to read the titles and abstracts. After applying the inclusion and exclusion criteria, the studies were eligible for the next stage, which consisted in reading the full-texts. The inclusion and exclusion criteria were applied again to reach the review sample. It is noted that, in the selection stage, there was disagreement between both reviewers regarding the inclusion of 12 studies; therefore, a third reviewer was called upon.

Subsequently, a manual search was performed in the references of the studies included. The Rayyan app was used to store, organize and remove duplicates and to blindly select the studies(15). It is noted that the Rayyan app version used was the free one. In addition to that, the team of reviewers underwent prior training to learn how to use this tool in selection of the studies. Search and selection of the studies were carried out from January to May 2022.

Data collection

Data extraction was by means of a form prepared by the authors of this review, containing the following items: authors; title of the study; year of publication; study locus; population and sample; information about the method; randomization; blinding; statistical analysis; follow-up time; type and classification of the educational technology; intervention group; control group; main results; and conclusion. Data collection was carried out independently by two reviewers, from June to August 2022. In relation to the items and/or divergent information, meetings were scheduled between the reviewers to discuss and resolve the discordant aspects until reaching consensus.

Data treatment and analysis

To assess the risk of bias in the randomized controlled clinical trials, we used the Revised Cochrane risk-of-bias tool for randomized trials (RoB 2), proposed by the Cochrane collaboration, which has five domains: bias arising from the randomization process; bias due to deviations from intended intervention; bias due to missing outcome data; bias in measurement of the outcome; and bias in selection of the reported result(16). This evaluation was performed by two independent reviewers. The doubts were discussed at the meetings, seeking consensus.

The synthesis of the results was performed descriptively and through meta-analysis. Thus, when performing the meta-analyses, the randomized controlled clinical trials were grouped into incidence of diabetic ulcers and lower limb amputations. The meta-analysis analysis model used was the random effect, performed using the Review Manager (RevMan) software, version 5.3, from the Cochrane collaboration.

The quality of the evidence assessment was elaborated according to the Grading of Recommendations Assessment, Development and Evaluation Working Group (GRADE)(17). The evaluation was performed for each outcome analyzed. In the meta-analysis, the outcomes evaluated were the incidence values of diabetic ulcers and lower limb amputations regarding use of educational technologies. Certainty of the evidence can be assessed as high (strong confidence that the true effect is close to the estimated one), moderate (moderate confidence in the estimated effect), low (limited confidence in the estimate of the effect) and very low (very limited confidence in the estimate of the effect). The certainty of the evidence assessment was performed using the GRADEpro software(18).

Results

The bibliographic survey identified 2,984 studies: 298 in the databases, bibliographic index and electronic library and 2,686 in the Gray Literature. After removing the duplicates and applying the eligibility criteria, the sample resulted in 11 randomized controlled clinical trials(19-20). Figure 1 presents the detailed flowchart corresponding to the selection process of the studies included in the systematic literature review.

Figure 1 -
Flowchart corresponding to the selection process of the randomized controlled clinical trials included in the systematic review, adapted from the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA). Teresina, PI, Brazil, 2022

The randomized controlled clinical trials included a total of 3,115 participants(19-20). In relation to the loci of the studies, there was prevalence of Brazil(19,21), Norway(22,23) and the United States(24-25), with two studies each. The year of publication varied from 2000 to 2020 and the follow-up time, from one to 24 months. 23 soft-hard technologies, 16 hard technologies and one soft technology were identified in the intervention groups. The predominant educational technologies were training sessions in six studies(22,25-26,27,23-20) and verbal guidelines, in five(19-28,24,29-27). It was evidenced that, in five studies, the control groups did not receive any intervention through any educational technology(19,22,21-20). The descriptive synthesis of the randomized controlled clinical trials included is presented in Figure 2.

Figure 2 -
Synthesis of the randomized controlled clinical trials included (n=11). Teresina, PI, Brazil, 2022

Figure 3 describes the risk of bias assessment using the RoB 2 tool, performed by domains for the 11 randomized controlled clinical trials included in the systematic review.

Figure 3 -
Risk of bias assessment of the randomized controlled clinical trials in each domain of the Revised Cochrane risk-of-bias tool for randomized trials (RoB 2). Teresina, PI, Brazil, 2022

Of the 11 randomized controlled clinical trials, 27.3% (n=3) presented low risk of bias, 9.1% (n=1) had uncertain risk of bias, and 63.6% (n=7) were categorized as with high risk of bias. Seven studies(22-26,27,23-20) were evaluated as with high risk in the bias domain in measurement of the results, as there was no blinding of the outcome evaluators. Four studies(22,26,27,23) had a high risk in the bias domain due to deviations from the designated interventions, as a result of lack of blinding of the participants and the professionals who applied the intervention. One study(29) was classified as having uncertain risk of bias, as it does not specify whether there was blinding of the evaluators. One study(26) has some concern in the domain bias resulting from the randomization process, as randomization was performed but there are no details of the process in the method. One study(20) presented some concern in the bias domain due to deviations from the designated interventions, as it did not clearly specify whether there was blinding of the professionals who applied the intervention.

In the meta-analysis, only randomized controlled clinical trials with similar characteristics were included, with regard to the interventions employed, in which the effects of the educational technologies were evaluated by the development of diabetic ulcers and lower limb amputations in the intervention and control groups. The Relative Risk (RR) was described in the last column of the forest plot, as shown in Figure 4.

(A.1)
Educational technologies versus usual care for the prevention of diabetic ulcers
(A.2)
Educational technologies versus usual care for the prevention of lower limb amputations

In Figure 4 A.1, the meta-analysis, with six studies included(19,24-27), which compared the educational technologies with usual care, evidenced that the use of educational technologies presented a protective factor for preventing the incidence of diabetic ulcers (RR=0.40; 95% CI=0.18-0.90; p=0.03). In Figure 4 A.2, the meta-analysis, also with six studies(22-24,26-27,23), indicates that the educational technologies have a protective factor to prevent the incidence of lower limb amputations, when compared to usual care (RR=0.53; 95% CI=0.31-0.90; p=0.02).

In Figure 4 A.1, the Higgins inconsistency statistical test (I2) classified heterogeneity across the studies as substantial (I2=70%). In contrast, in Figure 4 A.2, heterogeneity was indicated as not important (I2=0%).

Table 1 presents the certainty assessment of the meta-analyses evidence according to the GRADE criteria. The estimated effect of the educational technologies for preventing the incidence of diabetic ulcers was RR=0.40, when compared to usual care, with low certainty of the evidence. The estimated effect of the educational technologies to prevent the incidence of lower limb amputations was RR=0.53, when compared to usual care, presenting very low certainty of the evidence.

Table 1 -
Synthesis of the certainty of the evidence assessment, according to the Grading of Recommendations Assessment, Development and Evaluation (GRADE). Teresina, PI, Brazil, 2022

Discussion

This study analyzed the effects of educational technologies on the prevention and treatment of diabetic ulcers, and the results evidenced that soft-hard educational technologies such as structured verbal guidelines, educational games, lectures, training sessions through workshops and interactive practice, educational videos, illustrative and didactic folders, serial albums and playful drawings, and hard technologies such as therapeutic footwear, insoles, digital infrared thermometer and foot care kits, contributed to reducing both the incidence of diabetic ulcers(19,24-26,27) and the risk of foot complications(21). In addition to that, the theoretical-practical training sessions, which are soft-hard technologies, and the Telemedicine apps and mobile phone use, which are classified as hard technologies, improved diabetic ulcer healing and reduced the incidence of lower limb amputations in the intervention groups(22,23).

Educational technologies consist of knowledge enriched by human action, and are not merely about the construction and use of devices; they involve a systematic set of diverse scientific knowledge that enables planning, execution, control and monitoring of the educational process(30). From this perspective, the particularities of the educational technologies explain the prevalence of soft-hard and hard technologies in the randomized controlled clinical trials included in this systematic review.

It was evidenced that eight randomized controlled clinical trials(19-28,24-21) used soft-hard and/or hard educational technologies for the prevention of diabetic ulcers, which were effective in reducing the incidence of ulcerations in five studies(19,24-26,27). On the other hand, three randomized controlled clinical trials(22,23-20) used soft-hard and/or hard educational technologies in the treatment of diabetic ulcers, of which two(22,23) found a considerable effect and recorded a higher percentage of total diabetic ulcer healing.

The soft-hard educational technologies were the most used in the prevention and treatment of diabetic ulcers in the intervention groups. Thus, a randomized controlled clinical trial carried out in Brazil, which used soft-hard technology, found that the implementation of educational technologies through a focus group and educational games addressing foot complications reduced the incidence of ulcerations and the recurrence of diabetic ulcers in the Intervention Group(19). In Italy, a randomized controlled clinical trial, which in its Intervention Group used verbal guidelines on the risk factors for foot ulcerations and interactive practice, classified as soft-hard technologies, presented a significant effect in reducing the incidence of diabetic ulcers, as the Intervention Group had an incidence of 0% and the Control Group, 10%. In addition, they contributed to lowering the Body Mass Index and glycated hemoglobin, reinforcing that brief and low-cost educational technologies can reduce the incidence of foot ulcerations in patients with diabetes mellitus, in addition to being more likely to be applied in the routine clinical practice(27).

Through a theoretical-practical approach, workshops and interactive practice, classified as soft-hard technologies, the training sessions proved to be effective in the prevention and treatment of diabetic ulcers and were the most prevalent educational technologies in six studies(22,25-26,27,23-20). In Norway, through theoretical-practical training and Telemedicine, respectively classified as soft-hard and hard technologies, the educational technologies improved diabetic ulcer healing and reduced the number of amputations, as 82.1% of the patients in the Intervention Group presented ulcer healing in 12 months, with 5.1% incidence of amputations in the Intervention Group and 14.1% in the Control Group. In addition to that, this intervention increased the confidence of Primary Health Care nurses, who improved their skills in treating wounds, enabling a more comprehensive care for diabetic ulcers(22).

The incidence of diabetic ulcers was estimated in six randomized controlled clinical trials(19,24-27). Based on the meta-analysis, it was evidenced that the educational technologies presented a protective factor for preventing the incidence of diabetic ulcers, emphasizing the importance of using these resources in the assistance provided to patients with diabetes mellitus. In a prospective cohort study, the cumulative incidence of diabetic ulcers was 5.6% in two years, with the following risk factors for ulcerations: previous history of ulcerations or amputations, insulin consumption, distal neuropathy and foot deformity(31). This emphasizes the need for care continuity to control the risk factors and for educational technologies aimed at preventing complications in patients with diabetes mellitus.

In this meta-analysis, the home-based educational session conducted in the United Kingdom with illustrations of injuries on the feet and a handout, classified as soft-hard technologies, did not present any statistically significant difference between the intervention and control groups regarding the prevention of diabetic ulcer incidence; however, there was an improvement in foot care behaviors in the Intervention Group in relation to checking the shoes before wearing them, daily foot washing and use of moisturizing creams(29). On the other hand, a randomized controlled clinical trial carried out in the United States, which used enhanced therapy through educational video (soft-hard technology) associated with the use of therapeutic insoles and shoes (hard technologies), foot reevaluation (soft-hard technology) and use of a portable infrared thermometer to measure foot temperature (hard technology), identified a protective effect of this intervention for the prevention of diabetic ulcers, as there was a four-fold decrease in the risk of developing foot ulcers, with 29.3% incidence of ulcerations in the usual care group and 8.5% in the enhanced therapy group(25).

In China, through lectures, practical workshops and skills exercises, which are soft-hard technologies, and the distribution of foot care kits, which included nail clippers, foot cream, mono-filament 10 g, thermometer to measure temperature of the water to wash the feet, pieces of cotton with alcohol and a mirror, which correspond to the hard technologies, the educational technologies had a significant effect in preventing the incidence of diabetic ulcers and amputations, in addition to the participants increasing their knowledge and foot care. Furthermore, the patients in the Control Group were nearly 24 times more likely to develop foot ulcers. This educational program asked the patients to perform daily foot care with the help of a mirror for foot inspection and invited at least one family member to participate in the classes and help the patients, which ensured more effective home-based foot care(26).

Foot complications increase the likelihood of ulcerations, infections and amputations in people with diabetes mellitus. In this sense, a randomized controlled clinical trial, whose intervention consisted of verbal guidelines, which are soft-hard technologies, use of therapeutic shoes and an infrared thermometer, which are hard technologies, found that the patients in the Control Group had 10.3% more risks of developing some foot complication, with no statistical difference in terms of quality of life between the groups. In addition to that, the Control Group had seven ulcers and two Charcot fractures, with two patients developing infection and requiring amputation, whereas the Intervention Group had one ulcer and no amputations, highlighting that home-based self-monitoring of daily foot temperature, associated with health education and use of appropriate footwear, is an adjuvant tool for the prevention of diabetic ulcers and foot amputations(24).

In a systematic review with meta-analysis, thermometry had a protective effect when compared to standard toe care to prevent the incidence of diabetic ulcers (RR=0.53; 95% CI=0.29-0.96; p=0.03), and the authors encourage managers, public health services, professionals, patients, family members and caregivers to implement this preventive technique by monitoring plantar temperature using infrared thermometers, both in the clinical and home contexts(32). In this systematic review and meta-analysis, two randomized controlled clinical trials used thermometry associated with educational interventions(24-25), which may have enhanced the effect of the educational technologies for the prevention of diabetic ulcers.

The incidence of lower limb amputations was estimated in six randomized controlled clinical trials and, in the meta-analysis, the educational technologies presented a protection factor to prevent amputations(22-24,26-27,23). In a randomized controlled clinical trial, which employed Telemedicine in the community, classified as a hard technology, the incidence of amputations was 6.4% in the Intervention Group and 14.8% in the Control Group(23). However, a study that used soft-hard technologies through a 30-minute face-to-face class and a 90-minute interactive practice on risk behaviors did not record the incidence of amputations between the control and intervention groups, which can be justified due to the brief 6-month follow-up period(27).

Non-traumatic lower limb amputations are recurrent complications in patients with diabetes mellitus, generate increased costs for health services, extend the hospitalization times, reduce quality of life, exert impacts on mental health and affect the patients’ productive lives. Thus, foot care management, which includes health education, should value holistic care, accessibility, loyalty and care longitudinality. In this assumption, an education and continuous foot care treatment program in Spain detected that, of the total of 33 diabetic ulcers, 17 evolved to amputation and 16 were in patients who did not adhere to the program(33).

In this perspective, diabetic foot complications are a public health problem due to the increase in the number of patients with diabetes mellitus, the increased life expectancy of the population and the growth of associated comorbidities. However, the expansion of the assistance provided, which includes both early intervention in patients with diabetic ulcers to avoid gangrene and appropriate treatments such as performing the necessary vascular procedures and mandatory education on foot care, can lead to a reduction in the number of lower limb amputations(34).

Three randomized controlled clinical trials addressed the effect of educational technologies on diabetic ulcer healing(22,23-20). Although the educational technologies employed, which were soft-hard and hard, did not exert any statistically significant effect on reducing the ulcer healing times(22,23), there was a reduction in the size of the ulcers(20), with 82.1% of the patients presenting ulcer healing in the Intervention Group and 76.9% in the Control Group at 12 months. This reinforces that educational technologies should also be used in the diabetic ulcer treatment stage(22).

The effect of educational technologies on foot self-care was verified in four randomized controlled clinical trials(28,26-29,21). Even without significant differences in behavioral changes (p=0.26), the attitudes regarding foot self-care increased in both groups(28). In addition to that, an educational intervention for foot self-care, through an operative group which used soft-hard and hard technologies, had a significant effect in the treatment group after seven days (p<0.001) and 15 days (p<0.001), when compared to the Control Group, in relation to the reduction of the risks for foot complications, such as an improvement in the preservation of the skin and annexes, tissue perfusion, pulses, edema and plantar pressure distribution. This evidences that systematized educational interventions with brief follow-up periods are also effective(21).

Thus, to enhance the effect, health education should reduce language barriers and involve the patients in their own care plan to raise awareness about the disease and prevent complications, as most patients are unaware of the severity of these complications and follow negligent practices in the long term, due to low education and risky cultural practices. Despite the challenges, health education is a responsibility of professionals, who must use every opportunity to provide specific education, even combining the types of educational technologies available, with the objective of improving the skills of patients with diabetes mellitus in foot self-care(35).

An integrated care project, which included timely referral, weekly virtual clinic, healthy lifestyle support, community nurse training, app delivery and personalized educational support, increased engagement in education from 5% to 71% of those newly diagnosed with diabetes mellitus, in addition to reducing the incidence of major amputations from 13 to three procedures per 10,000 patients a year and of minor amputations from 26 to 18 procedures per 10,000 patients a year. This care model also significantly reduced the daily occupation of beds by people with diabetes mellitus in a district general hospital(36). In line with this systematic review, when associated with better structuring of the care network and professional training, educational technologies are more effective in reducing foot amputations and hospitalization due to complications arising from diabetes mellitus.

Regarding the satisfaction levels provided by the educational technologies, both randomized controlled clinical trials that evaluated this outcome concluded that there was no statistically significant difference between the intervention and control groups(22,23). However, the concern with satisfaction in the development of educational technologies is essential, as it influences the participants’ adherence to the intervention proposed.

The randomized controlled clinical trials included did not measure the costs of the educational technologies for the prevention and treatment of diabetic ulcers. Thus, the studies pointed out the need to carry out surveys comparing the costs of the educational and monitoring programs implemented with usual care, as it is expected that, in the long term, these interventions will present better cost-effectiveness, cost-benefit and cost-efficacy ratios in preventing foot complications and, consequently, reduce expenditure in health services and improve the quality of life of patients with diabetes mellitus(24-25,27).

In relation to the limitations of this systematic review, the reduced number of randomized controlled clinical trials on the effects of educational technologies on the treatment and incidence of diabetic ulcers stands out, in addition to the number of studies with a high risk of bias, which contributed to the lower certainty of the evidence.

The results of this systematic review may contribute to expanding the use of educational technologies in the care of patients with diabetes mellitus. In addition to that, this scientific evidence will assist health professionals in choosing the most assertive type of educational technology for the prevention and treatment of diabetic ulcers in the clinical practice.

Conclusion

Soft-hard educational technologies such as structured verbal guidelines, educational games, lectures, training through workshops and interactive practice, educational video, illustrative and didactic folders, serial albums and playful drawings, and hard technologies such as therapeutic footwear, insoles, digital infrared thermometer and foot care kits, exerted a positive effect on the prevention of diabetic ulcers and helped reduce the incidence of ulcerations and the risk of foot complications, in addition to enabling improvements in foot care. In relation to the treatment, both the soft-hard technologies through theoretical-practical training sessions, and the soft technologies such as Telemedicine apps and use of mobile phones, contributed to the evolution of diabetic ulcer healing, standing out as useful strategies in foot care management in patients with diabetes mellitus.

The meta-analysis results indicated that the educational technologies presented a protective factor for preventing the incidence of diabetic ulcers, with substantial heterogeneity across the studies and a low certainty of the evidence assessment, highlighting that, in further research studies, there may be a change in the estimate of the effect. In addition to that, the educational technologies had a protective factor to prevent the incidence of lower limb amputations, when compared to usual care. Heterogeneity was indicated as not important, and certainty of the evidence was assessed as very low.

In view of this, the use of educational technologies is recommended, especially soft-hard and hard, in the prevention and treatment of diabetic ulcers to reduce complications such as non-traumatic lower limb amputations, in addition to conducting more robust and well-designed randomized controlled clinical trials at different care levels for patients with diabetes mellitus, which would later allow developing systematic reviews in different care contexts, with a view to reducing the risk of bias and inconsistencies, as well as improving homogeneity of the studies and certainty of the evidence, in order to incorporate those educational technologies that proved to be effective in foot care.

References

  • 1. Schaper NC, Netten JJ, Apelqvist J, Bus SA, Hinchliffe RJ, Lipsky BA. Diretrizes práticas do IWGDF sobre a prevenção e o tratamento do pé diabético [Internet]. The International Working Group on the Diabetic Foot; 2019 [cited 2022 Oct 10]. Available from: https://iwgdfguidelines.org/wp-content/uploads/2020/12/Brazilian-Portuguese-translation-IWGDF-Guidelines-2019.pdf
    » https://iwgdfguidelines.org/wp-content/uploads/2020/12/Brazilian-Portuguese-translation-IWGDF-Guidelines-2019.pdf
  • 2. Schaper NC, Netten JJ, Apelqvist J, Bus SA, Hinchliffe RJ, Lipsky BA. Practical Guidelines on the prevention and management of diabetic foot disease (IWGDF 2019 update). Diabetes Metab Res Rev. 2020;36(S1):e3266. https://doi.org/10.1002/dmrr.3266
    » https://doi.org/10.1002/dmrr.3266
  • 3. Bundó M, Llussà J, Serra M, Iglesia PP, Gimbert RM, Real J, et al. Incidence and characteristics of diabetic foot ulcers in subjects with type 2 diabetes in Catalonian primary care centres: An observational multicentre study. Prim Care Diabetes. 2021;15(6):1033-9. https://doi.org/10.1016/j.pcd.2021.08.002
    » https://doi.org/10.1016/j.pcd.2021.08.002
  • 4. Oe M, Fukuda M, Ohashi Y, Shimojima Y, Tsuruoka K, Qin Q, et al. Evaluation of foot ulcer incidence in diabetic patients at a diabetic foot ulcer prevention clinic over a 10-year period. Wound Rep Reg. 2022;30(5):546-52. https://doi.org/10.1111/wrr.13039
    » https://doi.org/10.1111/wrr.13039
  • 5. Lira JAC, Nogueira LT, Oliveira BMA, Soares DR, Santos AMR, Araújo TME. Factors associated with the risk of diabetic foot in patients with diabetes mellitus in Primary Care. Rev Esc Enferm USP. 2021;55:e03757. https://doi.org/10.1590/S1980-220X2020019503757
    » https://doi.org/10.1590/S1980-220X2020019503757
  • 6. Nguyen TPL, Edwards H, Do TND, Finlayson K. Effectiveness of a theory-based foot care education program (3STEPFUN) in improving foot self-care behaviours and foot risk factors for ulceration in people with type 2 diabetes. Diabetes Res Clin Pract. 2019;152(2019):29-38. https://doi.org/10.1016/j.diabres.2019.05.003
    » https://doi.org/10.1016/j.diabres.2019.05.003
  • 7. Batista IB, Pascoal LM, Gontijo PVC, Brito PS, Sousa MA, Santos M Neto, et al. Association between knowledge and adherence to foot self-care practices performed by diabetics. Rev Bras Enferm. 2020;73(5):e20190430. https://doi.org/10.1590/0034-7167-2019-0430
    » https://doi.org/10.1590/0034-7167-2019-0430
  • 8. Marques ADB, Moreira TMM, Carvalho REFL, Chaves EMC, Oliveira SKP, Felipe GF, et al. PEDCARE: validation of a mobile application on diabetic foot self-care. Rev Bras Enferm. 2021;74(Suppl 5):e20200856. https://doi.org/10.1590/0034-7167-2020-0856
    » https://doi.org/10.1590/0034-7167-2020-0856
  • 9. Netten JJ, Raspovic A, Lavery LA, Monteiro-Soares M, Rasmussen A, Sacco ICN. Prevention of foot ulcers in the at-risk patient with diabetes: a systematic review. Diabetes Metab Res Rev. 2020;36(S1):e3270. https://doi.org/10.1002/dmrr.2701
    » https://doi.org/10.1002/dmrr.2701
  • 10. Merhy EE. Saúde: a cartografia do trabalho vivo. 3. ed. São Paulo: Hucitec; 2002.
  • 11. Higgins JPT, Thomas J. Cochrane handbook for systematic reviews of interventions. 2. ed. Hoboken, NJ: Wiley Blackwell; 2019.
  • 12. Page MJ, Mckenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372(71):1-9. https://doi.org/10.1136/bmj.n71
    » https://doi.org/10.1136/bmj.n71
  • 13. National Institute for Health Research. International prospective register of systematic reviews [Internet]. 2021 [cited 2022 Oct 10]. Available from: https://www.crd.york.ac.uk/prospero/
    » https://www.crd.york.ac.uk/prospero/
  • 14. Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER. A comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14(579):1-10. https://doi.org/10.1186/s12913-014-0579-0
    » https://doi.org/10.1186/s12913-014-0579-0
  • 15. Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan - a web and mobile app for systematic reviews. Syst Rev. 2016;5(2010):1-10. https://doi.org/10.1186/s13643-016-0384-4
    » https://doi.org/10.1186/s13643-016-0384-4
  • 16. Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022) [Internet]. London: Cochrane; 2022 [cited 2022 Oct 12]. Available from: http://www.training.cochrane.org/handbook
    » http://www.training.cochrane.org/handbook
  • 17. Schünemann H, Brozek J, Guyatt G, Oxman A. GRADE Handbook. Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach [Internet]. 2013 [cited 2021 July 6]. Available from: https://gdt.gradepro.org/app/handbook/handbook.html#h.svwngs6pm0f2
    » https://gdt.gradepro.org/app/handbook/handbook.html#h.svwngs6pm0f2
  • 18. Grade Working Group. GRADEPro [Software]. Hamilton: McMaster University/Evidence Prime Inc.; 2022 [cited 2022 July 6]. Available from: https://gradepro.org/
    » https://gradepro.org/
  • 19. Cisneros LL. Avaliação de um programa para prevenção de úlceras neuropáticas em portadores de diabetes. Rev Bras Fisioter. 2010;14(1):31-7. https://doi.org/10.1590/S1413-35552010000100006
    » https://doi.org/10.1590/S1413-35552010000100006
  • 20. Donohoe ME, Fletton JA, Hook A, Powell R, Robinson I, Stead JW, et al. Improving foot care for people with diabetes mellitus - a randomized controlled trial of an integrated care approach. Diabet Med. 2000;17(8):581-7. https://doi.org/10.1046/j.1464-5491.2000.00336.x
    » https://doi.org/10.1046/j.1464-5491.2000.00336.x
  • 21. Iversen MM, Igland J, Smith-Strom H, Ostbye T, Tell GS, Skeie S, et al. Effect of a telemedicine intervention for diabetes-related foot ulcers on health, well-being and quality of life: secondary outcomes from a cluster randomized controlled trial (DiaFOTo). BMC Endocr Disord. 2020;20(157): 1-8. https://doi.org/10.1186/s12902-020-00637-x
    » https://doi.org/10.1186/s12902-020-00637-x
  • 22. Lavery LA, Higgins KR, Lanctot DR, Constantinides GP, Zamorano RG, Armstrong DG, et al. Home monitoring of foot skin temperatures to prevent ulceration. Diabetes Care. 2004;27(11):2642-7. https://doi.org/10.2337/diacare.27.11.2642
    » https://doi.org/10.2337/diacare.27.11.2642
  • 23. Lavery LA, Higgins KR, Lanctot DR, Constantinides GP, Zamorano RG, Athanasiou KA, et al. Preventing diabetic foot ulcer recurrence in high-risk patients: use of temperature monitoring as a self-assessment tool. Diabetes Care. 2007;30(1):14-20. https://doi.org/10.2337/dc06-1600
    » https://doi.org/10.2337/dc06-1600
  • 24. Liang R, Dai X, Zuojie L, Zhow A, Meijuan C. Two-Year Foot Care Program for Minority Patients with Type 2 Diabetes Mellitus of Zhuang Tribe in Guangxi, China. Can J Diabetes. 2012;36(1):15-8. https://doi.org/10.1016/j.jcjd.2011.08.002
    » https://doi.org/10.1016/j.jcjd.2011.08.002
  • 25. Lincoln NB, Radford KA, Game FL, Jeffcoate WJ. Education for secondary prevention of foot ulcers in people with diabetes: a randomised controled trial. Diabetologia. 2008;51(11):1954-61. https://doi.org/10.1007/s00125-008-1110-0
    » https://doi.org/10.1007/s00125-008-1110-0
  • 26. Monami M, Zannoni S, Gaias M, Nreu B, Marchionni N, Mannucci E. Effects of a Short Educational Program for the Prevention of Foot Ulcers in High-Risk Patients: A Randomized Controlled Trial. Int J Endocrinol. 2015;2015(615680):1-5. https://doi.org/10.1155/2015/615680
    » https://doi.org/10.1155/2015/615680
  • 27. Moreira JB, Muro ES, Monteiro LA, Iunes DH, Assis BB, Chaves ECL. The effect of operative groups on diabetic foot self-care education: a randomized clinical trial. Rev Esc Enferm USP. 2020;54:e03624. https://doi.org/10.1590/S1980-220X2019005403624
    » https://doi.org/10.1590/S1980-220X2019005403624
  • 28. Smith-Strom H, Igland J, Ostbye T, Tell GS, Hausken MF, Graue M, et al. The Effect of Telemedicine Follow-up Care on Diabetes-Related Foot Ulcers: A Cluster-Randomized Controlled Noninferiority Trial. Diabetes Care. 2018;41(1):96-103. https://doi.org/10.2337/dc17-1025
    » https://doi.org/10.2337/dc17-1025
  • 29. Subrata SA, Phuphaibul R, Grey M, Siripitayakunkit A, Piaseu N. Improving clinical outcomes of diabetic foot ulcers by the 3-month self- and family management support programs in Indonesia: A randomized controlled trial study. Diabetes Metab Syndr. 2020;14(5):857-63. https://doi.org/10.1016/j.dsx.2020.05.028
    » https://doi.org/10.1016/j.dsx.2020.05.028.
  • 30. Nietsche EA, Backes VMS, Colomé CLM, Ceratti RN, Ferraz F. Education, care and management technologies: a reflection based on nursing teachers’ conception. Rev. Latino-Am. Enfermagem. 2005;13(3):344-53. https://doi.org/10.1590/S0104-11692005000300009
    » https://doi.org/10.1590/S0104-11692005000300009
  • 31. Yazdanpanah L, Shahbazian H, Nazari I, Arti HR, Ahmadi F, Mohammadianinejad SE, et al. Incidence and Risk Factors of Diabetic Foot Ulcer: A Population-Based Diabetic Foot Cohort (ADFC Study) -Two-Year Follow-Up Study. Int J Endocrinol, 2018;2018(7631659):1-9. https://doi.org/10.1155/2018/7631659
    » https://doi.org/10.1155/2018/7631659
  • 32. Araújo AL, Negreiros FDS, Florêncio RS, Oliveira SKP, Silva ARV, Moreira TMM. Effect of thermometry on the prevention of diabetic foot ulcers: a systematic review with meta-analysis. Rev. Latino-Am. Enfermagem. 2022;30:e3525. https://doi.org/10.1590/1518-8345.5663.3525
    » https://doi.org/10.1590/1518-8345.5663.3525
  • 33. Calle-Pascual AL, Duran A, Benedí A, Calvo MI, Charro A, Diaz JA, et al. A preventative foot care programme for people with diabetes with different stages of neuropathy. Diabetes Res Clin Pract. 2002;57(2002):111-7. https://doi.org/10.1016/s0168-8227(02)00024-4
    » https://doi.org/10.1016/s0168-8227(02)00024-4
  • 34. Lin CW, Armstrong DG, Lin CH, Liu PH, Hung SY, Lee SR, et al. Nationwide trends in the epidemiology of diabetic foot complications and lower-extremity amputation over an 8-year period. BMJ Open Diab Res Care. 2019;7:e000795. https://doi.org/10.1136/bmjdrc-2019-000795
    » https://doi.org/10.1136/bmjdrc-2019-000795
  • 35. Makiling M, Smarth H. Patient-Centered Health Education Intervention to Empower Preventive Diabetic Foot Self-Care. Adv Skin Wound. 2020;33(7):360-5. https://doi.org/10.1097/01.ASW.0000666896.46860.d7
    » https://doi.org/10.1097/01.ASW.0000666896.46860.d7
  • 36. Watt A, Beacham A, Palmer-Mann L, Williams A, White J, Brown R, et al. Service user and community clinician design of a partially virtual diabetic service improves access to care and education and reduces amputation incidence. BMJ Open Diab Res Care. 2021;9:e001657. https://doi.org/10.1136/bmjdrc-2020-001657
    » https://doi.org/10.1136/bmjdrc-2020-001657
  • *
    Paper extracted from doctoral dissertation “Construção e validação de intervenção educativa em ambiente virtual de aprendizagem sobre prevenção e manejo do pé diabético para enfermeiros da Atenção Primária à Saúde”, presented to Universidade Federal do Piauí, Teresina, PI, Brazil.
  • How to cite this article
    Lira JAC, Rocha ASC, Bezerra SMG, Nogueira PC, Santos AMR, Nogueira LT. Effects of educational technologies on the prevention and treatment of diabetic ulcers: A systematic review and meta-analysis. Rev. Latino-Am. Enfermagem. 2023;31:e3945 [cited year mon day]. Available from: URL . https://doi.org/10.1590/1518-8345.6628.3945
  • All authors approved the final version of the text.

Edited by

  • Associate Editor:
    Maria Lúcia Zanetti

Data availability

Data citations

National Institute for Health Research. International prospective register of systematic reviews [Internet]. 2021 [cited 2022 Oct 10]. Available from: https://www.crd.york.ac.uk/prospero/

Schünemann H, Brozek J, Guyatt G, Oxman A. GRADE Handbook. Handbook for grading the quality of evidence and the strength of recommendations using the GRADE approach [Internet]. 2013 [cited 2021 July 6]. Available from: https://gdt.gradepro.org/app/handbook/handbook.html#h.svwngs6pm0f2

Publication Dates

  • Publication in this collection
    19 June 2023
  • Date of issue
    2023

History

  • Received
    07 Dec 2022
  • Accepted
    08 Apr 2023
location_on
Escola de Enfermagem de Ribeirão Preto / Universidade de São Paulo Av. Bandeirantes, 3900, 14040-902 Ribeirão Preto SP Brazil, Tel.: +55 (16) 3315-3451 / 3315-4407 - Ribeirão Preto - SP - Brazil
E-mail: rlae@eerp.usp.br
rss_feed Acompanhe os números deste periódico no seu leitor de RSS
Acessibilidade / Reportar erro