Construction and validation of an educational technology on human papillomavirus vaccine for adolescents

Construcción y validación de una tecnología educativa sobre la vacuna virus del papiloma humano para adolescentes

ABSTRACT

Objective:  To construct and validate a comic-type educational technology on the human papillomavirus vaccine.

Methods:  A methodological study developed based on the manuals, informative notes of the Ministry of Health and with the suggestions of the target audience. Content validation was performed with 11 expert judges and the validation of appearance, with 32 adolescents. For the judges, the Content Validity Index was used and for the target audience the proportion of agreement was considered.

Results:  The comic book obtained a content validation index total of 0.95, a validation index considered adequate. It was identified a concordance greater than 0.80 in the validation of appearance with the target public, indicating an excellent degree of agreement.

Conclusion:  The technology was considered valid by the judges and the target audience, and can be used as an educational tool to guide the practice in the acceptability scenario of the vaccine.

Implications for practice:  The technology may contribute to the stimulation of adhesion to the human papillomavirus vaccine, since, in Brazil, the rates of adhesion to this vaccine are still low.

Keywords: Papillomavirus Vaccines; Comic Books; Validation Studies

RESUMEN

Objetivo:  Construir y validar una tecnología educativa del tipo historietas sobre la vacuna contra el virus del papiloma humano.

Métodos:  Estudio metodológico desarrollado con base en los manuales, notas informativas del Ministerio de la Salud y con las sugerencias del público objetivo. La validación de contenido fue realizada con 11 jueces especialistas y la validación de la apariencia, con 32 adolescentes. Para los jueces, se utilizó el Índice de Validez de Contenido y para el público objetivo se consideró la proporción de concordancia.

Resultados:  Las historietas obtuvieran un índice de validación de contenido total de 0,95, índice de validación considerado adecuado. Se identificó concordancia mayor que 0,80 en la validación de la apariencia con el público objetivo, indicando óptimo grado de concordancia.

Conclusión:  La tecnología fue considerada válida por los jueces y el público objetivo, pudiendo ser utilizada como instrumento educativo para orientar la práctica en el escenario de aceptabilidad de la vacuna.

Implicaciones para la práctica:  La tecnología puede contribuir con el estímulo a la adhesión a la vacuna contra el virus del papiloma humano, ya que en Brasil los índices de adhesión a esta vacuna todavía son bajos.

Palabras chave: Vacunas contra Virus del Papiloma; Historietas; Estudios de Validación

RESUMO

Objetivo:  Construir e validar uma tecnologia educativa do tipo história em quadrinhos sobre a vacina contra o papilomavírus humano.

Métodos:  Estudo metodológico desenvolvido com base nos manuais, notas informativas do Ministério da Saúde e com as sugestões da população alvo. A validação de conteúdo foi realizada com 11 juízes especialistas e a validação de aparência com 32 adolescentes. Para os juízes foi utilizado o Índice de Validade de Conteúdo e para o público alvo considerou-se a proporção de concordância.

Resultados:  A história em quadrinhos obteve um índice de validação de conteúdo total de 0,95, índice de validação considerado adequado. Identificou-se concordância maior que 0,80 na validação de aparência com o público alvo, indicando ótimo grau de concordância.

Conclusão:  A tecnologia foi considerada válida pelos juízes e pela população alvo, podendo ser utilizada como instrumento educativo para orientar a prática no cenário de aceitabilidade da vacina.

Implicações para a prática:  A tecnologia poderá estimular a adesão à vacina contra o papilomavírus humano, uma vez que, no Brasil, os índices de adesão a essa vacina ainda são baixos.

Palavras-chave: Vacinas contra Papillomavirus; Histórias em Quadrinhos; Estudos de Validação

INTRODUCTION

The Human Papillomavirus (HPV) is a virus that is related to various types of cancers. In men, it is related to penis, anus and oropharynx cancer; and, in women, it is related to cervical cancer. In the female population, cervical cancer is the fourth leading cause of cancer death in Brazil, being the third most frequent type of cancer, followed by breast and colorectal cancer¹.

The virus is responsible for up to 70.0% of cases of cervical cancer in the world, as well as 90.0% of cases of anus cancer, up to 60.0% of cases of vaginal cancer and up to 50% of cases of vulvar cancer².

In this context, the human papillomavirus vaccine or quadrivalent, which involves protection against four types of viruses (6, 11, 16, 18), emerges as a public health strategy in the search to prevent malignant neoplasms related to the virus, collaborating in reducing the incidence, prevalence and mortality of the different types of cancers².

In Brazil, since 2014 the vaccine is made available in the Unified Health System, initially for girls and more recently for boys. The goal of the Ministry of Health is to vaccinate 80.0% of the target population. In 2017, girls in the age group of 09 to 14 years and boys of 11 to 14 years corresponded to the target audience^2,3.

However, adherence to the vaccine is still considered unsatisfactory, mainly in relation to the second dose, being below the goal recommended by the Ministry of Health. In 2015, in Brazil, the vaccination coverage in the female sex, in the age groups of 09 to 12 years, was 53.7% for the first dose and 33.0% for the second dose of the vaccine⁴.

In view of this, it is necessary to encourage adherence to the complete vaccination scheme, in order to guarantee the protection of adolescents against human papillomavirus and possible associated diseases. Some factors may positively influence adherence to the complete vaccination scheme: health team recommendation, prescription of the vaccine by the doctor, objective and clear information on safety, efficacy, and easy access to vaccine⁵.

From this perspective, the school environment emerges as a potentiating scenario for the production of citizenship and incorporating knowledge, attitudes and practices of health promotion. The adoption of technologies in the school, such as booklets, comics and videos, can collaborate with learning skills, knowledge and the perception of risk/vulnerability towards vaccination, seeking to transform the empirical reality into scientific practice^6,7.

An integrative review study on the use of existing and/or used educational technologies that can promote the adhesion of vaccination against the human papillomavirus⁸ has shown that comic-type technology has not been cited, which brings innovation in the scope of strategies and/or technologies for actions to promote vaccination against human papillomavirus.

In this context, it is believed that the distinctive style of comics is one of the reasons that captivates adolescents, and consequently of great success with young audiences⁸. Therefore, comic-type educational technology can potentiate new health education practices by providing clear, objective, and safe information about the human papillomavirus vaccine through a dynamic and enjoyable reading.

In view of the above and given the importance of the topic for the development of more efficient and effective health actions regarding the adhesion to vaccination, the study aimed to build and validate a comic-type educational technology on the vaccine against human papillomavirus.

METHODS

Methodological study carried out in three municipal schools of Horizonte, in the State of Ceará, Brazil, from July 2017 to July 2018. These schools were chosen for being part of the "Programa Saúde na Escola" ("Health Program at the School" in free translation) in the municipality.

The development of technology in form of comics involved the following steps in the process of building didactic material for health care: sending the project to the ethics committee in research, elaboration of the material and evaluation by professionals from different areas^9,10. In addition, in this study, the target population participated in the technology development and validation phases (Figure 1).

Figure 1
Stages of the development process of the comic book. Adapted from Echer⁹.

The first stage refers to the approval of the research project by the Research Ethics Committee of the State University of Ceará. The study respected the guidelines and norms regulating research involving human beings, supported by the Resolution 466/12, and it was approved on January 14th, 2017 under CAAE No. 63175416.7.0000.5534 and Opinion no. 1,891,601.

The theoretical basis for the elaboration of the comics was made through a review of the literature and informative notes of the Ministry of Health^2,3,11,12. In this stage, in addition to the literature review, two meetings were held with 10 adolescents from the 7^th grade in one of the schools linked to the "Programa Saúde na Escola" in the municipality. At that time, the youngsters reported the doubts about the vaccine and what they would like to see in an educational material about the HPV vaccine.

As for the suggestions, the teens reported: to present the number of doses that need to be given and the reasons for the two doses, how the virus is transmitted and what are the benefits and the age to get vaccinated. In addition, the young people suggested that the story featured drawings of teenagers talking about the vaccine; drawings with sick and healthy young people, highlighting the future of a young man who has not been given the vaccine, all in a fun story with romance and teachings. Such points were included in the developed material, adding scientific knowledge in a language accessible to the target audience.

Thus, after reviewing the literature and the suggestions of adolescents, the content to compose the story script was defined as: what is the vaccine against human papillomavirus; forms of virus contagion; vaccination scheme for the adolescent; age range of the vaccination; relationship between papillomavirus and cancer. The drawings and layout were developed by a design in the program Corel Draw.

After the elaboration of the technology, the comic book validation was carried out with eleven judges, eight health professionals, specialists in sexual health and adolescence, who evaluated the content, and three technical judges (designers) who evaluated only the appearance. The number of judges was defined based on the literature, which indicates a variation between five and twenty, considering the necessary experience, qualification, training and availability of the professionals¹³.

For the participation in the research, the judges were initially selected by convenience and as a judge was identified and fit the eligibility criteria, it was requested the indication of other names that fit the criteria of inclusion of the research¹⁴.

Thus, the Lattes Platform was searched to evaluate whether the indicated researchers met the inclusion criteria, such as having scientific production, and/or training, and/or professional practice in the area of ​​interest, for expert judges, and to have training and/or experience in the elaboration of comics, scientific production in the area of ​​social communication, for technical judges. For each category, a minimum score of four and three points was considered, respectively.

After this process, the judges were invited to participate in the study by receiving an invitation letter via e-mail or in person. With the acceptance of the research participation, the Free and Informed Consent Form was sent, the evaluation instrument with questions about content and appearance and a copy of the comic book which could be sent by e-mail or printed, depending on the option of the participant.

After validation with the judges, the comics were presented to the adolescents during the health education activities of the "Programa Saúde na Escola", developed by nurses of the Family Health Strategy. The criteria used for inclusion in the sample were: adolescents aged 11 to 14 years, target age range of the vaccine at the time of data collection, students regularly enrolled in schools attended by the "Programa Saúde na Escola", and that were developing their school activities during this research phase period. As exclusion criteria, was considered being absent from school due to not showing up, medical leaves or suspensions in this period.

The adolescents were approached during three activities of the nurse of the Family Health Strategy in schools linked to the "Programa Saúde na Escola". The invitation was made in advance to sixty adolescents. However, only the adolescents who brought the free and informed consent signed by the parents, besides the free and informed consent signed by the student themselves, participated in the study.

Thus, for the validation of appearance with the adolescents, participated 32 students who presented the signed terms. The activity consisted in the delivery of the instrument of data collecting and printed copies of the comic strip for group reading. After the reading was completed, the instrument was filled out, which contained seven objective and two open questions. The collection process was finished on the third meeting when the suggestions of the open questions were repeated and did not add new recommendations for changes¹⁵.

Content Validation Index was used by the judges to validate the content and the proportion of agreement was considered for the target audience. It is worth noting that the data collection instrument contemplated the Likert scale with a score of 1 to 4, in which the items define: 1 - not relevant or not representative, 2 - item needs major revision to be representative, 3 - item needs little revision to be representative, 4 - relevant or representative item.

The content validity index and the proportion of agreement are calculated by the sum of the items that were marked with the scores of 3 or 4 divided by the total number of responses. Items that received scores 1 or 2 were reviewed or eliminated¹³.

For the evaluation of the comic as a whole, the formula of the average of the indices of each item, calculated separately, was used, that is, we add up all indexes of content validity calculated separately and divide it by the total number of items considered in the evaluation¹³. In order for the scale to be judged as having validity of content and acceptable appearance, it must reach content validity index between items of 0.7 or higher and average of validity index of the scale of 0.9 or higher¹⁴.

RESULTS

The comic book was published with two-sided printing, colored ink, on A4 sized paper, with dimensions of 14x20mm, in a total of 12 pages. The definitive title was "KICK HPV OUT". On the back cover, there is the catalog sheet, where the source of the material, the name of the author, the instructor and the teaching institution to which they belong are explained, as well as technical data (illustration and diagramming) (Figure 2).

Figure 2
Sequence of pages from left to right: Cover with the title. Page 1: presentation of the characters and content. Page 7: Information on the number of doses and disease that the vaccine protects against. Pages 8 and 9: Information about the vaccine. Page 12: final reflection on the vaccine.

Cover with the title: KICKING HPV OUT

Page 1:

- Hello, my name is Mateus! I would like your attention to talk about the vaccine against HPV. HPV is the virus that can cause cancer. In this comic, you will get to know clear and safe information about the vaccine. I would like to present my friends that also are part of the story.

- Marina

- Letícia

- Júlia

- Tiago

- Pedro

Page 7:

- I've already taken a dose. Do you think it is necessary to take another dose? C'mon! Getting another prick for nothing.

- Well with me it is kiss on the shoulder and vaccine's prick on the arm for HPV go away.

- The vaccine only protects if you take both doses. What is a small sting compared to the health dose that the vaccine can offer us!?

- Who doesn't take the vaccine can contract warts in the genital region or a serious disease, like cervical cancer, anal cancer, penile cancer, throat cancer...

Page 8:

- Did you know that cervical cancer is the fourth leading cause of death of women by cancer in Brazil!?

- Wow!!! Julia! And how does the vaccine avoid that people get sick?

- Hum... Tiago, I think the nurse told us that, another day, here at school. - Vaccines produce antibodies against diseases, in this case, against HPV.

- Antibodies are like soldiers that combat these viruses.

- Hey, stop! Out of here!!!

- Out of here!!! Thought that I was alone, right? I have backup!

Page 9:

- These antibodies continue to be produced in our organism. - But, in order for this to happen, you need to take the second dosage of the vaccine 6 months from now, when will be produced more antibodies able of protecting us.

- Hum ... So to grow strong and healthy tomorrow, it is necessary to prevent today, right?

- Guys, I'm going to take my vaccine. I want to be one of the first!

- Bye! Don't forget the group assignment, tomorrow.

- I am taking the second dose today.

- I'm gonna take the first.

Banner: VACCINATION

Page 12:

- Julia, we have to go...

- Yeah, class is about to get started.

- But, before we go, I need to leave my message for all of you who are reading: it's no use taking the first step towards your goals if you're not going to keep pursuing them. To achieve them, the kick off must be given, but it is also necessary to insist and not give up in face of the first difficulties. - Just as it is with our dreams, the vaccine against HPV also needs to be taken seriously. It's no use to take the first dose of the vaccine and not take the second dose. - It's necessary to guarantee the complete vaccine scheme to be protected. - Each teenager is unique, but everyone needs protection! What about you, are you just gonna keep waiting? Go for it, let's all vaccinate!

Content validation was performed by eight judges, of whom seven (87.5%) were nurses and one a social assistant (12.5%). One of the nurses, besides the training in nursing also has a degree in biology, with teaching experience of this discipline to middle school adolescents.

All judges were female, five (62.5%) were between 30 and 35 years of age, two (25.0%) were between 40 and 45 years of age and one (12.5%) was 60 years old. With regard to academic degrees, six (75.0%) had masters and two (25.0%) specialization. As for working time in the area, one of the specialists had 36 years of career (12.5%); four specialists had 10 to 20 years (50.0%); three specialists had three to ten years (37.5%).

With respect to the area of ​​professional performance, three nurses were university professors (37.5%); two nurses from the technical area of ​​the Regional Health Coordination of the State of Ceará (25.0%), one from the area of ​​Women's and Children's Health and the other from the National Immunization Program; a Family Health Strategy nurse (12.5%), having been a coordinator of the "Programa Saúde na Escola" for two years and a social worker (12.5%) from the Family Health Support Unit, which currently coordinates the "Programa Saúde na Escola".

The judges attributed 1.0 to the content validity index for seven of the ten items evaluated, including: up-to-date information, clarity and objectivity of the information, appropriate to the target audience, and promoting knowledge about the human papillomavirus vaccine.

As for the validation of appearance, this was done by the judges specialized in the health area and also by three technical judges (designers), totaling eleven judges. All designers were male, aged between 34 and 46 years.

With regard to the area of ​​formation, one has a degree in Letters, being a lecturer of the English course and drawing course; two finished high school. All are self-employed professionals. As for the working time in the area, the designers had an average of 15 years in the profession.

The judges attributed 1.0 to the index of validity for appearance, for nine of the eleven items evaluated, among which: drawings and their quantity, character presentation, subtitles, attractiveness, visual composition, adequate size and font, contrast with colors and number of pages.

Thus, the index of validity for the content and appearance of the comic was calculated separately. The validity index for the content was 0.94 and for the appearance was of 0.98. The overall content validity index of the comic strip was 0.95, demonstrating that both content and appearance were considered adequate by the panel of judges.

After validation by the judges, the appearance validation was performed with 32 adolescents. Of these, thirteen (40.6%) were male and nineteen (59.3%) were female. In terms of age, six (18.7%) were 11 years old, two (6.2%) were 12 years old, eleven (34.3%) were 13 and thirteen (40.6%) were 14 years old. Regarding the school year, eight (25.0%) were in the 7^th grade, eleven (34.3%) in the 8^th grade and thirteen (40.6%) were in the 9^th grade.

All the items evaluated by the adolescents showed a proportion of agreement above the minimum recommended by the literature, being the index 0.80 the lowest obtained. The evaluated items were: importance of information; increased knowledge about the vaccine; easy understanding of the story; interest of the drawings; attractiveness of the colors; adequate numbering of pages and motivation of the reading until the end.

At the end of the instrument, the adolescents' opinion about the story was asked. The adolescents emphasized the teaching that the story offered, praised the colors and the attractiveness of the drawings. Here are some of the teen's considerations about technology:

"Alert about the disease and encourages you to take the vaccine"

"I liked the part of football, of the vaccine and of the teaching that story offers"

"The fact of alerting young people about the importance of the HPV vaccine"

Regarding the suggestions, nine (28.1%) adolescents suggested some alteration in the technology. The most cited suggestion was the continuation of the story, requesting that the authors increase the number of pages. However, the majority of adolescents 23 (72%) considered that the number of pages was adequate. Thus, the initial number of was maintained.

After validation with the target audience it was necessary to return to the designer to make the changes suggested by the adolescents, which gave rise to the final version of the comic book.

Thus, the final version was composed of six characters with similar age range to the target audience of the vaccine. The story has passed in the school environment, addressing romance, dreams, football, classroom, future of adolescents, but having as its main focus the vaccine against human papillomavirus. At the end, there was a reflection on the need for the vaccine for adolescent health, not being enough taking a single dose, since the protection against the virus is only guaranteed with the complete vaccination scheme.

DISCUSSION

The evaluation of judges and adolescents has shown that educational technology is a relevant and valid material regarding the content that one wishes to present, guidelines on the human papillomavirus vaccine.

Frequently, the vaccination scheme against human papillomavirus is not completed and it is often necessary to reinforce objective, clear and practical information on the safety and efficacy of the vaccine⁵. Thus, the development and implementation of technologies that seek to promote adherence to vaccination can contribute to the increase of vaccination coverage and, consequently, reduction in the incidence, prevalence and mortality of cervical, vulva, penis and anus cancer. The process of constructing and validating the comic strip of this study covered professionals from different areas, which allowed diverse approaches on the same theme, expanding and perfecting the constructed material⁹.

The comic book developed in this study was evaluated with an excellent content and appearance validation index by the technical judges and target population. Other methodological studies on the development of educational technologies also validated their materials with excellent content validation indexes: construction and validation of educational material for prevention of metabolic syndrome in adolescents with a global validity index of 0.98¹⁶, construction and validation of an educational manual for promotion of breastfeeding with an overall validity index of 0.97¹⁷.

As for the appearance validity, the comic book also obtained an excellent index of validity when evaluated by both the judges and the target population. A study on the construction of a booklet to prevent vertical transmission of HIV also obtained an optimal evaluation, a level of agreement varying from 91.1% to 100.0% among the judges and above 75.0% among puerperal and pregnant women¹⁸.

Thus, both in this study and in other studies on the construction and validation of educational technology, there was a concern to include the target audience in the appearance validity, which contributed to the quality and relevance of the material.^15,16,17,18

In this context, the literature points out that adolescents consider comics interesting, stimulating and somewhat different from what they are accustomed to reading⁸, being pertinent the use of this didactic material in the classroom, since the learning can be achieved in a playful way.

These educational technologies are well accepted by adolescents, since they present in a ludic way themes that are considered controversial and embarrassing, which provides a dialogic, participatory and reflexive relationship of various subjects involving myths and taboos¹⁹.

Considering the adolescents' opinions about the technology developed in this study, it was noticed that the way the HPV vaccine was approached, through the experiences and doubts of the adolescent's daily life, contributed to the construction of a quality comic book in the scenario of educational activities in health.

Thus, it is fundamental to include didactic methodologies that provide the adolescents with skills and competences to understand the content, in order to restructure previous knowledge in a critical and reflexive way⁸. Therefore, comics can be used as a device for meaningful learning, because it involves the student in a differentiated way, through the integration between reality and fantasy, providing an immersion in the context of the story, which allows familiarity, appropriation of the situation and of the knowledge provided by the material⁸.

In this way, the educational technology in question, comics, can be considered an important vehicle of communication on health-related topics, especially for adolescents, due to the practicality and captivating reading. Although the results show that the content of the educational technology is validated by health professionals and easily understood by the target population, it is considered necessary to develop other studies that prove the clinical validity of this material in different regions, since the intention of the comics was to improve adolescent adherence to the vaccine.

CONCLUSION

Educational technology on the human papillomavirus vaccine was considered valid by judges and the target population and could be used as an educational tool to guide practice and improve vaccine adherence.

It is believed that this educational material can be used both by health professionals, especially nurses, in the implementation of health education for adolescents, as well as by education professionals in classrooms, through workshops as an option to the traditional educational practices. To print the comics, the interested professional can contact the authors of this research to request the version for printing.

However, the importance of updating educational technology is emphasized, since scientific knowledge about the papillomavirus vaccine is constantly being renewed.

REFERENCES

• ¹ Instituto Nacional do Câncer (INCA). [Internet]. 2016; [cited 2017 nov 10]. Available from: http://www2.inca.gov.br/wps/wcm/connect/tiposdecancer/site/home/colo_utero/definicao
  » http://www2.inca.gov.br/wps/wcm/connect/tiposdecancer/site/home/colo_utero/definicao
• ² Ministério da Saúde (BR). Secretaria de Vigilância em Saúde. Departamento de Vigilância de Doenças Transmissíveis. Coordenação Geral do Programa Nacional de Imunizações. Guia Prático sobre o HPV. Guia de perguntas e respostas para profissional de saúde. Brasília: Ministério da Saúde; 2014.
• ³ Ministério da Saúde (BR). Secretaria de Vigilância em Saúde. Departamento de Vigilância das doenças transmissíveis. Nota informativa nº 154, de 2017/GGPNI/ DEVIT/SVS/MS. Ampliação da faixa etária de 11 a 14 anos para meninos com a vacina HPV quadrivalente. Brasília: Ministério da Saúde; 2017.
• ⁴ Ministério da Saúde (BR). Datasus. Sistema de Informação do Programa Nacional de Imunizações (SI-PNI). Estratégia de Vacinação contra HPV - 2015. Coberturas vacinais - HPV Quadrivalente - Sexo feminino de 09 a 12 anos por idade e dose Total Brasil. Brasília: Ministério da Saúde; 2015.
• ⁵ Monteiro DLM, Ballalai I, Almeida JAM. Transformando vacina em vacinação: a importância da recomendação médica [Editorial]. Femina. 2015 set/out;43(5). Available from: http://files.bvs.br/upload/S/0100-7254/2015/v43n5/a5314.pdf
  » http://files.bvs.br/upload/S/0100-7254/2015/v43n5/a5314.pdf
• ⁶ Silva KL, Sena RR, Gandra EC, Matos JAV, Coura KRA. Promoção da saúde no programa saúde na escola e a inserção da enfermagem. Rev Min Enferm [Internet]. 2014;18(3):614-2. Available from: http://www.dx.doi.org/10.5935/1415-2762.20140045
  » http://www.dx.doi.org/10.5935/1415-2762.20140045
• ⁷ Interaminense INCS, Oliveira SC, Leal LP, Linhares FMP, Pontes CM. Tecnologias educativas para promoção da vacinação contra o papiloma vírus humano: revisão integrativa da literatura. Texto Contexto Enferm [Internet]. 2016 jun 27;25(2). Available from: http://dx.doi.org/10.1590/0104-07072016002300015
  » http://dx.doi.org/10.1590/0104-07072016002300015
• ⁸ Kawamoto EM, Campos LML. Histórias em quadrinhos como recurso didático para o ensino do corpo humano em anos iniciais do ensino fundamental. Ciência & Educação [Internet]. 2014 jan/mar;20(1):147-158. Available from: http://dx.doi.org/10.1590/1516-731320140010009
  » http://dx.doi.org/10.1590/1516-731320140010009
• ⁹ Echer IC. Elaboração de manuais de orientação para o cuidado em saúde. Rev Latino-Am Enfermagem [Internet]. 2005 set/out;13(5):754-7. Available from: http://dx.doi.org/10.1590/S0104-11692005000500022
  » http://dx.doi.org/10.1590/S0104-11692005000500022
• ¹⁰ Moreira MF, Nóbrega MML, Silva MIT. Comunicação escrita: contribuição para a elaboração de material educativo em saúde. Rev Bras Enferm [Internet]. 2003 abr;56(2):184-8. Available from: http://dx.doi.org/10.1590/S0034-71672003000200015
  » http://dx.doi.org/10.1590/S0034-71672003000200015
• ¹¹ Ministério da Saúde (BR). Secretaria de Vigilância em Saúde. Departamento de vigilância das doenças transmissíveis. Coordenação geral do Programa Nacional de Imunizações. Informe técnico da vacina papiloma vírus humano 6, 11, 16, 18 (recombinante). Brasília: Ministério da Saúde; 2015.
• ¹² Ministério da Saúde (BR). Secretaria de Vigilância em Saúde. Departamento de Vigilância das doenças transmissíveis. Nota informativa nº 149, de 2015/GGPNI/DEVIT/SVS/MS. Brasília: Ministério da Saúde; 2015.
• ¹³ Coluci MZO, Alexandre NMC, Milani D. Construção de instrumentos de medida na área da saúde. Ciênc. Saúde Coletiva [Internet]. 2015 mar;20(3):925-36. Available from: http://dx.doi.org/10.1590/1413-81232015203.04332013
  » http://dx.doi.org/10.1590/1413-81232015203.04332013
• ¹⁴ Polit DF, Beck CT. Fundamentos de pesquisa em enfermagem: avaliação de evidências para a prática da enfermagem. 7ª ed. Porto Alegre: Artmed; 2011.
• ¹⁵ Oliveira SC, Lopes MVO, Fernandes AFC. Development and validation of an educational booklet for healthy eating during pregnancy. Rev Latino-Am Enfermagem [Internet]. 2014 aug;22(4):611-20. Available from: http://dx.doi.org/10.1590/0104-1169.3313.2459
  » http://dx.doi.org/10.1590/0104-1169.3313.2459
• ¹⁶ Moura IH, Silva AFR, Rocha AESH, Lima LHO, Moreira TMM, Silva ARV. Construction and validation of educational materials for the prevention of metabolic syndrome in adolescents. Rev Latino-Am Enfermagem [Internet]. 2017 oct;25:e2934. Available from: http://dx.doi.org/10.1590/1518-8345.2024.2934
  » http://dx.doi.org/10.1590/1518-8345.2024.2934
• ¹⁸ Lima ACMACC, Bezerra KC, Sousa DMN, Rocha JF, Oriá MOB. Development and validation of a booklet for prevention of vertical HIV transmission. Acta Paul Enferm [Internet]. 2017 abr;30(2):181-9. Available from: http://dx.doi.org/10.1590/1982-0194201700028
  » http://dx.doi.org/10.1590/1982-0194201700028
• ¹⁷ Costa PB, Chagas ACMA, Joventino ES, Dodt RCM, Oriá MOB, Ximenes LB. Construção e validação de manual educativo para a promoção do aleitamento materno. Rev Rene [Internet]. 2013; [cited 2018 jul 13]; 14(6):1160-7. Available from: http://www.periodicos.ufc.br/rene/article/view/3732/2952
  » http://www.periodicos.ufc.br/rene/article/view/3732/2952
• ¹⁹ Mariano MR, Pinheiro AKB, Aquino OS, Ximenes LB, Pagiluca LMF. Jogo Educativo na promoção da saúde de adolescentes: revisão integrativa. Rev Eletr Enf [Internet]. 2013;15(1):265-73. Available from: http://dx.doi.org/10.5216/ree.v15i1.17814
  » http://dx.doi.org/10.5216/ree.v15i1.17814

Publication Dates

• Publication in this collection
  10 July 2019
• Date of issue
  2019

History

• Received
  26 Feb 2019
• Accepted
  15 Apr 2019