Open-access Psychometric assessment of oral health-related quality of life questionnaires cross-culturally adapted for use in Brazilian adults - a systematic review

Avaliação psicométrica de questionários de qualidade de vida relacionada à saúde bucal adaptados culturalmente para uso em adultos brasileiros - uma revisão sistemática

ABSTRACT

Objective:  This study aimed to review the psychometric properties of oral health-related quality of life (OHRQoL) questionnaires for the Brazilian adult population.

Methods:  A systematic review was performed based on the COSMIN guidelines (PROSPERO CRD42022300018). The studies were obtained through electronic searches in the PubMed/MEDLINE, Web of Science, Lilacs, VHL (BIREME), SciELO, and Embase databases.

Results:  The search was performed in December 2022. Articles on OHRQoL that reported the cross-cultural adaptation of instruments into Portuguese (Brazil) and evaluated the psychometric properties of measuring instruments in adult patients were included. Those about the development of a novel instrument and participants under 18 years of age were excluded. Information was collected on the country, type of instrument validated, psychometric tests, and adaptation process. The certainty of the evidence was assessed using GRADEpro program. The search returned 6,556 articles, and 14 were considered for this review. However, two studies did not report the cross-cultural adaptation process. Content validity, internal consistency, criterion validity, construct validity, reliability, general discriminant validity, Cronbach’s alpha value, and general intraclass correlation coefficient value were confirmed in 12 studies. Cronbach’s alpha ranged from 0.69 to 0.96. The certainty of the evidence was considered moderate and low. This study has some limitations, such as the lack of information in some reviewed studies, the unavailability of Brazilian instruments, and absence of longitudinal validation of some instruments.

Conclusions:  In conclusion, there are 14 OHRQoL instruments adapted for Brazilian adults that can be used with caution by researchers and clinicians, since they presented moderate to low certainty of the evidence.

Keywords: Quality of life; Questionnaires; Validation study; Psychometrics; Systematic review

RESUMO

Objetivo:  Este estudo teve como objetivo revisar as propriedades psicométricas dos questionários de qualidade de vida relacionada à saúde bucal (OHRQoL) para a população adulta brasileira.

Métodos:  Foi realizada uma revisão sistemática com base nas diretrizes Consensus-based standards for the selection of health measurement instruments - COSMIN (International Prospective Register of Systematic Reviews - PROSPERO CRD42022300018). Os estudos foram obtidos por meio de buscas eletrônicas nas bases de dados United States National Library of Medicine (PubMed)/ Medical Literature Analysis and Retrieval System Online (MEDLINE), Web of Science, Literatura Latino-Americana e do Caribe em Ciências da Saúde (Lilacs), Biblioteca Virtual em Saúde - BVS (Centro Latino-Americano e do Caribe de Informação em Ciências da Saúde - BIREME), Scientific Electronic Library Online (SciELO) e Embase. A busca foi realizada em dezembro de 2022. Foram incluídos artigos que relatavam a adaptação transcultural de instrumentos (QVRSB) para o português (Brasil) e que avaliavam as propriedades psicométricas de mensuração de instrumentos (QVRSB) em pacientes adultos. Foram excluídos aqueles sobre o desenvolvimento de um novo instrumento e com participantes menores de 18 anos. Foram coletadas informações sobre país, tipo de instrumento validado, testes psicométricos e processo de adaptação. A certeza da evidência foi avaliada usando GRADE.

Resultados:  A pesquisa retornou 6556 artigos, e 14 foram incluídos nesta revisão. Dois estudos não relataram o processo de adaptação transcultural. A validade de conteúdo, consistência interna, validade de critério, validade de constructo, confiabilidade, validade discriminante geral, valor alfa de Cronbach e valor geral do coeficiente de correlação intraclasse foram confirmadas em 12 estudos. O alfa de Cronbach variou de 0,69 a 0,96. A certeza da evidência foi considerada moderada e baixa. Esta pesquisa apresenta algumas limitações, como falta de informação em alguns estudos revisados; indisponibilidade de instrumentos brasileiros; ausência de validação longitudinal de alguns instrumentos.

Conclusões:  Em conclusão, existem 14 instrumentos de QVRSB adaptados para adultos brasileiros que podem ser utilizados com cautela por pesquisadores e clínicos, uma vez que apresentam moderada a baixa certeza de evidência.

Palavras-chave: Qualidade de vida; Questionários; Estudo de validação; Psicometria; Revisão sistemática

INTRODUCTION

Quality of life is defined as the individuals’ perception of their position in life in the context of the culture and value systems in which they are inserted, as well as their goals, expectations, standards, and concerns1,2. The part concerning quality of life affected by oral health and orofacial conditions is named Oral Health-Related Quality of Life (OHRQoL). It investigates how oral health can affect function, psychological state, social factors, and pain or discomfort of individuals3.

To understand the aspects covered by the OHRQoL, a multidimensional evaluation based on specific or generic structured questionnaires is used. This can offer the researchers a wide selection of options to use different instruments according to the objectives of their research, in addition to allowing the evaluation of the impacts of the orofacial health condition in an individual’s life through the psychological, physical, and social dimensions4. However, these questionnaires have limitations in their applicability when they are developed in English-speaking countries and with sociocultural realities different from Brazil. In these cases, the questionnaire must undergo a process of cross-cultural adaptation and psychometric validation before being used in Brazil5.

The equivalence and preservation of these instruments are outlined by standardized validation and cross-cultural adaptation guidelines, consisting of six stages:

  1. Translation;

  2. Synthesis;

  3. Reverse translation;

  4. Review by the expert committee;

  5. Test of the pre-final version, and

  6. Submission and evaluation of all reports written by the committee6.

In addition, these instruments must ensure the reliability of their results through psychometric properties that use quality criteria for their measurements, such as content validity, internal consistency, construct validity, reproducibility, responsiveness, convergent validity, discriminant validity, and interpretation5,7.

Some OHRQoL questionnaires have been translated into Brazilian Portuguese and are available for use8,9. However, there is a gap between the reliability and the cross-cultural adaptation method employed, requiring a critical evaluation of these translated versions to verify the adapted measure and the preservation of the original instrument’s psychometric properties.

This systematic review aimed to reassess the psychometric properties of OHRQoL questionnaires adapted for the Brazilian adult population and identify their suitability for research and clinical practice in Brazil.

METHODS

The present systematic review is registered in the International Prospective Register of Systematic Reviews (PROSPERO) under CRD42022300018, and was performed based on the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN) guidelines for systematic reviews of patient-reported outcome measures (PROMs)10. The outcomes of interest were psychometrics and cross-cultural adaptation.

PICO question

Are oral health-related quality of life instruments cross-culturally adapted for application in Brazilian adults reliable?

Eligibility criteria

For this systematic review, studies that met the following inclusion criteria were selected:

  1. Validation and cross-cultural adaptation studies of OHRQoL instruments into Brazilian Portuguese;

  2. Studies that evaluated the psychometric properties of measurement of OHRQoL instruments in adult patients; and

  3. Studies that reported at least one of the measurement properties: reliability, internal consistency, measurement error, content validity, construct validity, criterion validity, discriminant validity, and/or convergent validity.

Systematic reviews of OHRQoL measures, studies reporting OHRQoL assessment through instruments, construction (development) and validation of a novel instrument, questionnaires that had a single item, and translation into Portuguese from Portugal were excluded.

Search strategy

The studies were obtained through electronic searches in the United States National Library of Medicine (PubMed)/Medical Literature Analysis and Retrieval System Online (MEDLINE), Web of Science, Latin American and Caribbean Health Sciences Literature (Lilacs), Virtual Health Library (VHL)/(Biblioteca Regional de Medicina - BIREME), Scientific Electronic Library Online (SciELO), and Embase databases. The keywords used were searched in Health Sciences Descriptors (DeCS), Medical Subject Headings (MeSH), and published manuscripts on OHRQoL.

The Boolean operators AND and OR were used in combination with the following terms: quality of life, oral health quality of life, instrument, scale, questionnaire, measurement, measurement tool, psychometrics, reliability, validity, instrument validation, cross-cultural adaptation, instrument translation, Brazilian version, Brazil, Portuguese, and Brazilian Portuguese. A general search strategy was adapted to the characteristics of each database (Table 1) to identify studies of interest for this review. Databases were explored for articles and abstracts with no language restriction. In addition, a complementary scan on the gray literature through Google Scholar was performed. References in all included studies were checked for additional studies. The investigation in the respective databases were performed until December 2022.

Table 1.
Search strategy utilized for each database.

Study selection

The Rayyan tool (https://rayyan.qcri.org/welcome) was used in the selection of studies, management, and citation of references during the development of this review11. The selection process was performed by three reviewers (DWDO, FSL, and YGG) in two stages. In the first phase, the they independently identified all relevant studies through electronic search methods based on the eligibility criteria applied to titles and abstracts. For studies that seemed to meet the inclusion criteria or for which sufficient data were not found in the title and abstract to make a clear decision, the complete text was pre-selected. In the second phase, the pre-selected studies were read in full by the same researchers to decide whether or not the it met the inclusion criteria. When necessary, the authors of the papers were contacted by email to clarify questions related to the research. All the studies excluded, at this point or later, were recorded along with the reasons for rejection. Observational studies that met the eligibility criteria were included in the final analysis and submitted to data synthesis. Articles found twice or more were considered only once. Disagreements were resolved by consensus among the three reviewers; this procedure was applied at all stages. The reviewers were trained for each database before the research.

Data extraction

Data were recorded qualitatively to allow comparisons among the selected studies, and each researcher qualitatively assessed them through an evaluation form. Data were collected on the following items: author, year of publication, country, study design, characteristics of the participants (sex and mean age), original language of the instrument, cross-cultural adaptation process, target population, main reported results, conclusion, name of the instrument, acronym, generality or specificity of the instrument, method of completion, domains, number of items, scoring, assessment period, time of completion, availability of the questionnaire in Brazilian Portuguese, internal consistency, criterion validity, construct validity, reliability, discriminant validity, translation, back-translation, synthesis, committee approach, pre-test, and psychometric evaluation.

Assessment quality

The included studies had their quality assessed by the psychometric validation and adaptation process10. The identified psychometric properties were then evaluated according to nine assessment criteria: content validity, internal consistency, criterion validity, construct validity, reproducibility, responsiveness, floor and ceiling effects, and interpretability. It was assigned a positive (+), undetermined (?), or negative (-) rating for each of these measures, or zero (0) if no information was available. It was recommended to present the evaluation results in a table but not using an overall score, as this gives equal importance to each psychometric property12.

The cross-cultural adaptation was evaluated according to established guidelines13, namely:

  1. Translation;

  2. Back-translation;

  3. Committee review;

  4. Pre-test; and

  5. Re-examination of score weighting.

In the first step, at least two qualified translators translated the scale from the original language into the target language. In the second step, two independent translators translated the translated version back into the original language, in order to ensure that it reflected the content of the original version. The third step ideally involved a committee review to develop the next-to-last version for pre-testing, and the fourth step consisted of applying this version to 30-40 individuals from the target population. The final step aimed to re-examine the score weighting, considering the cultural context. It was assigned present, absent, or unclear for each item.

Certainty assessment

The certainty of the evidence was assessed according to COSMIN guidelines10, using the GRADEpro program, depending on each factor analyzed (risk of bias, indirect evidence, inconsistency, and imprecision), being classified as high, moderate, low, or very low. It started with high quality and was reduced by one or two levels when serious or very serious risk of bias, indirect evidence, inconsistency, or imprecision were identified. The outcomes assessed were psychometric analysis and cross-cultural adaptation.

RESULTS

Search and selection

The electronic search retrieved 6,556 articles, of which 1,635 duplicates were removed. The manual search identified three additional studies. In the first phase, 4,879 publications were excluded. In the second, 30 studies were excluded. Therefore, 14 articles3,14,15,16,17,18,19,20,21,22,23,24,25,26 were considered in this review (Figure 1)27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56.

Figure 1.
Flowchart of the included studies.

Qualitative assessment

All revised studies3,14,15,16,17,18,19,20,21,22,23,24,25,26 presented a cross-sectional design and were carried out in Brazil. The number of participants ranged from 1214,17 to 50424. The age of the participants ranged from 2421,24,25 to 69 years23. Two studies18,23 did not undergo the cross-cultural adaptation process (supplementary material 1).

Supplementary material 2 shows the health conditions evaluated by the instruments. The self-completion method and the interview were used to fill out the questionnaires3,14,15,16,17,18,19,20,21,22,23,24,25,26. Four studies did not report the scoring16,17,21,23. The adapted instrument was available in six publications3,14,15,17,23,24.

Two studies14,17 did not report psychometric validation (Table 2). All adaptation steps were disclosed in 11 studies3,14,15,16,17,19,20,21,24,25,26 (Table 3).

Table 2.
Psychometric assessment of oral health-related quality of life instruments.

Table 3.
Cross-cultural adaptation assessment of oral health-related quality of life instruments.

Certainty assessment

The certainty of the evidence was downgraded by the risk of bias and indirectness, being considered low for psychometric analysis outcome and moderate for cross-cultural adaptation outcome (Table 4).

Table 4.
Level assessment of systematic reviews.

DISCUSSION

The assessment of individuals’ oral health based only on clinical criteria makes it difficult to identify and recognize its impact on general well-being and people’s lives4. Due to this multidimensionality, the use of instruments is necessary to properly assess the patients, recording their subjectivity in a standardized and reproducible way3,26. In the present review, 14 OHRQoL instruments were adapted to Brazil and psychometrics properties were evaluated. All of them proved to be valid and ready for use.

The included instruments had the original language in English. To understand the OHRQoL in Portuguese-speaking population, the scales had to be properly translated and culturally adapted for use in this population57. All the studies were carried out in Brazil, which was expected, as the they adapted instruments to be used in the Brazilian culture. The studies were designed in accordance with Beaton et al.6, who recommended a cross-sectional study, indicating that data were collected at a specific time without additional follow-up. Researchers who carry out cross-sectional analyses generally have greater difficulty creating a consistent report on interventions, treatments, and other variables in quality of life5.

All instruments used to assess quality of life must have the basic properties of reproducibility, validity, and sensitivity to changes58. In this review, both specific3,14,15,17,18,19,20,21,22,23,25 and generic16,24,26 instruments were found, indicating that Brazilian researchers can use different instruments according to the research intention and/or oral condition. When specific instruments are not available for a particular condition, generic instruments are used, developed to reflect the impact of general oral health on the individuals` life. One of the main limitations of their use is the inability to detect small differences after intervention or in a specific condition of low reproducibility59. On the other hand, specific instruments individually assess the impact of a given condition and have greater sensitivity and detection capacity to some type of change in the study after a determined intervention58,60.

The notoriety of cross-cultural adaptation lies in producing instruments that are equivalent in different cultures, maintaining their content and validity in a different cultural context61. Through these instruments, it is possible to enable a better form of expression, language understanding and evaluation, allowing the best results of an investigation, and consequently, the promotion of care humanization62). Most of the reviewed studies followed the adaptation guidelines satisfactorily. It is estimated that the Brazilian culture experiences and the country’s context were inserted in the validated instruments that were prepared to be used in Brazil.

However, some studies have shown to be deficient in the cross-cultural adaptation process18,22,23. A flawed translation and adaptation process will affect the instrument’s reliability, creating an inconsistency between the original and the translated version, which may compromise the validity and reliability of an item and/or the domain of the instrument6,57. These psychometric properties express the information about the instruments validity, helping the researcher choose the potential instrument.

Two studies in this review14,17 did not assess the psychometric criteria suggested by Terwee et al.12. Through psychometric tests, it is possible to verify the instrument’s reliability and whether it measures what it is intended. After these psychometric tests are applied, a report is sent to the reader regarding the instrument’s reliability and validity. Consequently, when a researcher intends to investigate a specific or generic oral condition, these instruments become effective since they had undergone the whole psychometrics validation. The advantage of using adapted and validated instruments is time and effort savings, in addition to avoiding erroneous comparisons between different translated versions6.

A traditional method to estimate the reliability of the internal consistency of a questionnaire is Cronbach’s alpha63. It measures the correlation between answers by analyzing the profile of the responses given by the participants64,65. The minimum acceptable value for alpha is 0.70; on the other hand, very high values (greater than 0.90) may be related to redundancy or duplication of items, which may mean that several items measure the same construct66. In this case, duplication or redundancy must be eliminated. The studies used in this review had an alpha ranging from 0.6916 to 0.9617, demonstrating that the instruments adapted for Brazil have an ideal coefficient, reinforcing their reliability.

The included studies3,14,15,16,17,18,19,20,21,22,23,24,25,26 presented two methods for recording the individuals’ reports: the self-completion method (self-report scale), in which the instrument is filled out by the participant, requiring greater individual cooperation67; and the interview in which the instrument is completed by the observer, which may present problems due to the interference of the interviewer’s experience64,68. Researchers need to pay attention to this fact when applying the instrument to their research or clinical activity in order to prevent information bias69.

The time to fill the instrument was not reported in most studies. This is significant data to be gathered, as the researcher should be aware of the time that will be allocated to the data collection when using the instrument.

The indication of the period to be considered in the participant’s response was not reported in most of the studies as well16,18,19,20,21,22,23,24,25,26. This information is important because the Brazilian version will be applied and reapplied frequently and, as the author did not include the evaluation period, there may be a response70 and/or methodological71 bias, compromising the investigation.

The ultimate objective of cross-cultural adaptation is to produce an instrument to be applied to a population with culture and/or language different from the original instrument6,28. However, some reviewed studies did not provide a Brazilian version of the adapted instruments16,18,19,20,21,22,25,26. This can limit their use and citation, or even stimulate other authors to develop a similar instrument.

The GRADEpro tool is widely used to verify the quality and certainty of the evidence in systematic reviews72. The overall certainty of the evidence in this review can be described as moderate to low, indicating significant constraints on the imprecision of results or lack of data to support a strong conclusion. As a result, recommendations based on this evidence are less secure and subject to change as new information emerges73. It is important to note that a moderate to low certainty rating does not necessarily imply that the instruments in question are ineffective or harmful. This simply indicates that the available evidence is not robust enough to provide a definitive answer74, indicating the need for caution in clinical decision-making.

The assessment of the quality of life includes subjective experiences that contribute to an individual’s evaluation parameter75. The PROMs are questionnaires that collect health outcomes directly from the people who experience them76. In addition, these instruments allow the investigation of symptoms, quality of life, functional and emotional status, and dysfunctions, as well as contribute to the decision, planning, and evaluation of certain types of treatment77. PROMs can be seen as important strategies to support clinical decisions and most indicated treatments, compare the results among health professionals, encourage quality improvement, and evaluate public health practices and policies78. Thus, measuring the disease’s impact on the patient’s quality of life becomes an increasingly essential tool, especially when its properties have been reaffirmed as valid and reproducible parameters. This review showed that several instruments, specific or generic, are available with the purpose of assessing the OHRQoL of Brazilian adult patients.

The present review found some limitations in the included studies, such as the lack of information in some reviewed studies, non-availability of Brazilian instruments, and absence of longitudinal validation of the reviewed instruments. It is suggested that cross-sectional studies be carried out to longitudinally validate the OHRQoL instruments adapted for Brazil. Researchers are encouraged to publish their validated OHRQoL instruments.

In conclusion, there are 14 OHQoL instruments adapted for Brazilian adults. The major studies provided information regarding the exact processes of validation, translation, and cultural adaptation. Additionally, three papers failed to prove the validity of quality of life among oral potentially malignant disorder (OPMD QoL), orthognathic quality of life questionnaire (OQLQ), and oral health impact profile-edent (OHIPEDENT) instruments, making it difficult for researchers to choose based on psychometric properties. In general, the generic and specific OHRQoL instruments adapted for the Brazilian adult population can be used with caution by researchers and clinical dentists in Brazil, since they presented moderate to low certainty of the evidence. Specifically, the OPMD QoL, OQLQ, and OHIPEDENT scales require further validation.

ACKNOWLEDGEMENTS:

We would like to thank the Postgraduate Program in Dentistry at the Universidade Federal dos Vales do Jequitinhonha e Mucuri, which provided technical support for the development and implementation of this study, and the Research Foundation of the State of Minas Gerais (FAPEMIG) and Coordination for the Improvement of Higher Education Personnel for the academic grant.

REFERENCES

  • 1 Rasafiani M, Sahaf R, Shams A, Vameghi R, Zareian H, Akrami R. Validity and reliability of the persian version of the world health organization quality of life questionnaire - the older adults edition. Iranian Journal of Ageing 2020; 15(1): 28-41. http://dx.doi.org/10.32598/sija.2020.3.110
    » https://doi.org/http://dx.doi.org/10.32598/sija.2020.3.110
  • 2 Gobierno de México. Calidad de vida para un envejecimiento saludable [Internet]. 2022 [cited on April 17, 2023]. Available at: Available at: https://www.gob.mx/inapam/es/articulos/calidad-de-vida-para-un-envejecimiento-saludable?idiom=es
    » https://www.gob.mx/inapam/es/articulos/calidad-de-vida-para-un-envejecimiento-saludable?idiom=es
  • 3 Douglas-De-Oliveira DW, Lages FS, Paiva SM, Cromley JG, Robinson PG, Cota LOM. Cross-cultural adaptation of the Brazilian version of the Dentine Hypersensitivity Experience Questionnaire (DHEQ-15). Braz Oral Res 2018; 32: e37. https://doi.org/10.1590/1807-3107bor-2018.vol32.0037
    » https://doi.org/https://doi.org/10.1590/1807-3107bor-2018.vol32.0037
  • 4 Chantre M, Mendes S, Bernardo M. Oral health-related quality of life in Portuguese undergraduate students. J Clin Exp Dent 2021; 13(12): e1202-8. https://doi.org/10.4317/jced.58810
    » https://doi.org/https://doi.org/10.4317/jced.58810
  • 5 Al Maqbali M, Gracey J, Rankin J, Dunwoody L, Hacker E, Hughes C. Cross-cultural adaptation and psychometric properties of quality of life scales for arabic-speaking adults: a systematic review. Sultan Qaboos Univ Med J 2020; 20(2): e125-37. https://doi.org/10.18295/squmj.2020.20.02.002
    » https://doi.org/https://doi.org/10.18295/squmj.2020.20.02.002
  • 6 Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976) 2000; 25(24): 3186-91. https://doi.org/10.1097/00007632-200012150-00014
    » https://doi.org/https://doi.org/10.1097/00007632-200012150-00014
  • 7 Souza AC, Alexandre NMC, Guirardello EB. Psychometric properties in instruments evaluation of reliability and validity. Epidemiol Serv Saude 2017; 26(3): 649-59. https://doi.org/10.5123/S1679-49742017000300022
    » https://doi.org/https://doi.org/10.5123/S1679-49742017000300022
  • 8 Pires CPDAB, Ferraz MB, Abreu MHNG. Translation into Brazilian Portuguese, cultural adaptation and validation of the oral health impact profile (OHIP-49). Braz Oral Res 2006; 20(3): 263-8. https://doi.org/10.1590/s1806-83242006000300015
    » https://doi.org/https://doi.org/10.1590/s1806-83242006000300015
  • 9 Paraguassu ÉC, Cardenas AMC. Systematic review of current medical literature on the impact of oral health on quality of life. Int J Adv Res Sci Eng Technol 2019; 6(3): 115-23. https://dx.doi.org/10.22161/ijaers.6.3.14
    » https://doi.org/https://dx.doi.org/10.22161/ijaers.6.3.14
  • 10 Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res 2018; 27(5): 1159-70. https://doi.org/10.1007/s11136-018-1829-0
    » https://doi.org/https://doi.org/10.1007/s11136-018-1829-0
  • 11 Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan- a web and mobile app for systematic reviews. Syst Rev 2016; 5(1): 210. https://doi.org/10.1186/s13643-016-0384-4
    » https://doi.org/https://doi.org/10.1186/s13643-016-0384-4
  • 12 Terwee CB, Bot SDM, de Boer MR, van der Windt DAWM, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol 2007; 60(1): 34-42. https://doi.org/10.1016/j.jclinepi.2006.03.012
    » https://doi.org/https://doi.org/10.1016/j.jclinepi.2006.03.012
  • 13 Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol 1993; 46(12): 1417-32. https://doi.org/10.1016/0895-4356(93)90142-n
    » https://doi.org/https://doi.org/10.1016/0895-4356(93)90142-n
  • 14 Almeida IF, Freitas KS., Almeida DB, Bastos AOS, Farias MT, Oliveira MC. Cross-cultural adaptation of a quality of life questionnaire for individuals with oral potentially malignant disorders in the Brazilian context. Acta Odontol Latinoam 2021; 34(1): 71-80. https://doi.org/10.54589/aol.34/1/071
    » https://doi.org/https://doi.org/10.54589/aol.34/1/071
  • 15 Silveira JO, Cota LOM, Bendo CB, Faria SFS, Costa FO. Validation of the Brazilian version of the Halitosis Associated Life-Quality Test (HALT). Braz Oral Res 2020; 34: e098. https://doi.org/10.1590/1807-3107bor-2020.vol34.0098
    » https://doi.org/https://doi.org/10.1590/1807-3107bor-2020.vol34.0098
  • 16 Abegg C, Fontanive VN, Tsakos G, Davoglio RS, Oliveira MMC. Adapting and testing the oral impacts on daily performances among adults and elderly in Brazil. Gerodontology 2015; 32(1): 46-52. https://doi.org/10.1111/ger.12051
    » https://doi.org/https://doi.org/10.1111/ger.12051
  • 17 Araújo AM, Miguel JAM, Gava ECB, Oliveira BH. Translation and cross-cultural adaptation of an instrument designed for the assessment of quality of life in orthognatic patients. Dental Press J Orthod 2013; 18(5): 99-106. https://doi.org/10.1590/s2176-94512013000500017
    » https://doi.org/https://doi.org/10.1590/s2176-94512013000500017
  • 18 Gava ECB, Miguel JAM, Araújo AM, Oliveira BH. Psychometric properties of the Brazilian version of the Orthognathic Quality of Life Questionnaire. J Oral Maxillofac Surg 2013; 71(10): 1762.e1-8. https://doi.org/10.1016/j.joms.2013.05.020
    » https://doi.org/https://doi.org/10.1016/j.joms.2013.05.020
  • 19 Kallás MS, Crosato EM, Biazevic MGH, Mori M, Aggarwal VR. Translation and cross-cultural adaptation of the manchester orofacial pain disability scale. J Oral Maxillofac Res 2013; 3(4): e3. https://doi.org/10.5037/jomr.2012.3403
    » https://doi.org/https://doi.org/10.5037/jomr.2012.3403
  • 20 Campos JADB, Carrascosa AC, Maroco J. Validity and reliability of the Portuguese version of Mandibular Function Impairment Questionnaire. J Oral Rehabil 2012; 39(5): 377-83. https://doi.org/10.1111/j.1365-2842.2011.02276.x
    » https://doi.org/https://doi.org/10.1111/j.1365-2842.2011.02276.x
  • 21 Sardenberg F, Oliveira AC, Paiva SM, Auad SM, Vale MP. Validity and reliability of the Brazilian version of the psychosocial impact of dental aesthetics questionnaire. Eur J Orthod 2011; 33(3): 270-5. https://doi.org/10.1093/ejo/cjq066
    » https://doi.org/https://doi.org/10.1093/ejo/cjq066
  • 22 Bortoluzzi MC, Manfro R, Soares IC, Presta AA. Cross-cultural adaptation of the orthognathic quality of life questionnaire (OQLQ) in a Brazilian sample of patients with dentofacial deformities. Med Oral Patol Oral Cir Bucal 2011; 16(5): e694-9. https://doi.org/10.4317/medoral.16938
    » https://doi.org/https://doi.org/10.4317/medoral.16938
  • 23 Souza RF, Patrocínio L, Pero AC, Marra J, Compagnoni MA. Reliability and validation of a Brazilian version of the Oral Health Impact Profile for assessing edentulous subjects. J Oral Rehabil 2007; 34(11): 821-6. https://doi.org/10.1111/j.1365-2842.2007.01749.x
    » https://doi.org/https://doi.org/10.1111/j.1365-2842.2007.01749.x
  • 24 Oliveira BH, Nadanovsky P. Psychometric properties of the Brazilian version of the Oral Health Impact Profile - short form. Community Dent Oral Epidemiol 2005; 33(4): 307-14. https://doi.org/10.1111/j.1600-0528.2005.00225.x
    » https://doi.org/https://doi.org/10.1111/j.1600-0528.2005.00225.x
  • 25 Hanan SA, Cohen-Carneiro F, Herkrath FJ, Oliveira BH, Loguércio AD, Martins LM, et al. Validation of the brazilian version of the oral health impact profile-aesthetic questionnaire. Braz Dent J 2022; 33(1): 77-86. https://doi.org/10.1590/0103-6440202204237
    » https://doi.org/https://doi.org/10.1590/0103-6440202204237
  • 26 Perazzo MF, Ortiz FR, Pérez-Díaz PA, Tsakos G, Zini A, Büssing A, et al. Brazilian version of Positive Oral Health and Well-Being: cross-cultural adaptation and psychometric analysis. Braz Oral Res 2022; 36: e051. https://doi.org/10.1590/1807-3107bor-2022.vol36.0051
    » https://doi.org/https://doi.org/10.1590/1807-3107bor-2022.vol36.0051
  • 27 Martins-Júnior PA, Ramos-Jorge J, Paiva SM, Marques LS, Ramos-Jorge ML. Validations of the Brazilian version of the Early Childhood Oral Health Impact Scale (ECOHIS). Cad Saude Publica 2012; 28(2): 367-74. https://doi.org/10.1590/S0102-311X2012000200015
    » https://doi.org/https://doi.org/10.1590/S0102-311X2012000200015
  • 28 Paiva BSR, Barroso EM, Cadamuro SA, Paula LAB, Pirola WE, Serrano CVMP, et al. The Children’s International Mucositis Evaluation Scale is valid and reliable for the assessment of mucositis among Brazilian children with cancer. J Pain Symptom Manage 2018; 56(5): 774-80.e2. https://doi.org/10.1016/j.jpainsymman.2018.07.015
    » https://doi.org/https://doi.org/10.1016/j.jpainsymman.2018.07.015
  • 29 Martins MT, Ferreira FM, Oliveira AC, Paiva SM, Vale MP, Allison PJ, et al. Preliminary validation of the Brazilian version of the Child Perceptions Questionnaire 8-10. Eur J Paediatr Dent 2009; 10(3): 135-40.PMID: 19761288
  • 30 Tesch FC, Oliveira BH, Leão A. Semantic equivalence of the Brazilian version of the Early Childhood Oral Health Impact Scale. Cad Saude Publica 2008; 24(8): 1897-909. https://doi.org/10.1590/S0102-311X2008000800018
    » https://doi.org/https://doi.org/10.1590/S0102-311X2008000800018
  • 31 Daher A, Versloot J, Costa LR. The cross-cultural process of adapting observational tools for pediatric pain assessment: the case of the Dental Discomfort Questionnaire. BMC Res Notes 2014; 7(1): 897. https://doi.org/10.1186/1756-0500-7-897
    » https://doi.org/https://doi.org/10.1186/1756-0500-7-897
  • 32 Pimenta WV, Traebert J. Adaptation of the Oral Aesthetic Subjective Impact Score (OASIS) questionnaire for perception of oral aesthetics in Brazil. Oral Health Prev Dent 2010; 8(2): 133-7. PMID: 20589246
  • 33 Rebouças AP, Bendo CB, Abreu LG, Lages EMB, Flores-Mir C, Paiva SM. Cross-cultural adaptation and validation of the Impact of Fixed Appliances Measure questionnaire in Brazil. Braz Oral Res 2018; 32: e14. https://doi.org/10.1590/1807-3107bor-2018.vol32.0014
    » https://doi.org/https://doi.org/10.1590/1807-3107bor-2018.vol32.0014
  • 34 Bendo CB, Paiva SM, Viegas CM, Vale MP, Varni JW. The PedsQLTM Oral Health Scale: feasibility, reliability and validity of the Brazilian Portuguese version. Health Qual Life Outcomes 2012; 10: 42. https://doi.org/10.1186/1477-7525-10-42
    » https://doi.org/https://doi.org/10.1186/1477-7525-10-42
  • 35 Castro RAL, Cortes MIS, Leão AT, Portela MC, Souza IPR, Tsakos G, et al. Child-OIDP index in Brazil: cross-cultural adaptation and validation. Health Qual Life Outcomes 2008; 6: 68. https://doi.org/10.1186/1477-7525-6-68
    » https://doi.org/https://doi.org/10.1186/1477-7525-6-68
  • 36 Abanto J, Tsakos G, Paiva SM, Goursand D, Raggio DP, Bönecker M. Cross-cultural adaptation and psychometric properties of the Brazilian version of the scale of oral health outcomes for 5-year-old children (SOHO-5). Health Qual Life Outcomes 2013; 11: 16. https://doi.org/10.1186/1477-7525-11-16
    » https://doi.org/https://doi.org/10.1186/1477-7525-11-16
  • 37 Goursand D, Paiva SM, Zarzar PM, Pordeus IA, Grochowski R, Allison PJ. Measuring parental-caregiver perceptions of child oral health-related quality of life: psychometric properties of the Brazilian version of the P-CPQ. Braz Dent J 2009; 20(2): 169-74. https://doi.org/10.1590/s0103-64402009000200014
    » https://doi.org/https://doi.org/10.1590/s0103-64402009000200014
  • 38 Santos PM, Gonçalves AR, Marega T. Validity of the Psychosocial Impact of Dental Aesthetics Questionnaire for use on Brazilian adolescents. Dental Press J Orthod 2016; 21(3): 67-72. https://doi.org/10.1590/2177-6709.21.3.067-072.oar
    » https://doi.org/https://doi.org/10.1590/2177-6709.21.3.067-072.oar
  • 39 Daher A, Versloot J, Leles CR, Costa LR. Screening preschool children with toothache: validation of the Brazilian version of the Dental Discomfort Questionnaire. Health Qual Life Outcomes 2014; 12: 30. https://doi.org/10.1186/1477-7525-12-30
    » https://doi.org/https://doi.org/10.1186/1477-7525-12-30
  • 40 Barbosa TS, Gavião MBD. Qualidade de vida e saúde bucal em crianças - parte II: versão brasileira do Child Perceptions Questionnaire. Ciênc Saúde Coletiva 2011; 16(7): 3267-76. https://doi.org/10.1590/S1413-81232011000800026
    » https://doi.org/https://doi.org/10.1590/S1413-81232011000800026
  • 41 Goursand D, Paiva SM, Zarzar PM, Ramos-Jorge ML, Cornacchia GM, Pordeus IA, et al. Cross-cultural adaptation of the Child Perceptions Questionnaire 11-14 (CPQ11-14) for the Brazilian Portuguese language. Health Qual Life Outcomes 2008; 6: 2. https://doi.org/10.1186/1477-7525-6-2
    » https://doi.org/https://doi.org/10.1186/1477-7525-6-2
  • 42 Barbosa TS, Steiner-Oliveira C, Gavião MBD. Tradução e adaptação brasileira do Parental-Caregiver Perceptions Questionnaire (P-CPQ). Saúde Soc 2010; 19: 698-708. https://doi.org/10.1590/S0104-12902010000300020
    » https://doi.org/https://doi.org/10.1590/S0104-12902010000300020
  • 43 Torres CS, Paiva SM, Vale MP, Pordeus IA, Ramos-Jorge ML, Oliveira AC, et al. Psychometric properties of the Brazilian version of the Child Perceptions Questionnaire (CPQ11-14) - short forms. Health Qual Life Outcomes 2009; 7: 43. https://doi.org/10.1186/1477-7525-7-43
    » https://doi.org/https://doi.org/10.1186/1477-7525-7-43
  • 44 Martins LGT, Cremona-Parma GO, Cristiano D, Possamai CF, Sônego F, Traebert J. Cross-cultural adaptation and validation of the COHIP-SF19 to be used in Brazil. Pesqui Bras Odontopediatria Clín Integr 2018; 18(1): e4046. http://dx.doi.org/10.4034/PBOCI.2018.181.105
    » https://doi.org/http://dx.doi.org/10.4034/PBOCI.2018.181.105
  • 45 Barbosa TS, Vicentin MDS, Gavião MBD. Qualidade de vida e saúde bucal em crianças - parte I: versão brasileira do Child Perceptions Questionnaire 8-10. Ciênc Saúde Coletiva 2011; 16(10): 4077-85. https://doi.org/10.1590/S1413-81232011001100013
    » https://doi.org/https://doi.org/10.1590/S1413-81232011001100013
  • 46 Soares RC, Moysés ST, Rocha JS, Baldani MH, Werneck RI, Moysés SJ. Cross-cultural adaptation of the Dental Neglect Scale for five-year-old children in Brazil. Braz Oral Res 2021; 35: e115. https://doi.org/10.1590/1807-3107bor-2021.vol35.0115
    » https://doi.org/https://doi.org/10.1590/1807-3107bor-2021.vol35.0115
  • 47 Silva SB, Cabral TM, Pinto TMP, Durand LB. Instrument of self-perception and knowledge of dental erosion: cross-cultural adaptation to the Brazilian population. Braz Oral Res 2015; 29: 53. https://doi.org/10.1590/1807-3107BOR-2015.vol29.0053
    » https://doi.org/https://doi.org/10.1590/1807-3107BOR-2015.vol29.0053
  • 48 Hilasaca-Mamani M, Barbosa TS, Fegadolli C, Castelo PM. Validity and reliability of the quality of masticatory function questionnaire applied in Brazilian adolescents. Codas 2016; 28(2): 149-54. https://doi.org/10.1590/2317-1782/20162015070
    » https://doi.org/https://doi.org/10.1590/2317-1782/20162015070
  • 49 Campos JADB, Carrascosa AC, Zucoloto ML, Maroco J. Validation of a measuring instrument for the perception of oral health in women. Braz Oral Res 2014; 28:S1806-83242014000100244. https://doi.org/10.1590/1807-3107bor-2014.vol28.0033
    » https://doi.org/https://doi.org/10.1590/1807-3107bor-2014.vol28.0033
  • 50 Mello dos Santos C, Hugo FN, Leal AF, Hilgert JB. Comparison of two assessment instruments of quality of life in older adults. Rev Bras Epidemiol 2013; 16(2): 328-37. http://dx.doi.org/10.1590/S1415-790X2013000200009
    » https://doi.org/http://dx.doi.org/10.1590/S1415-790X2013000200009
  • 51 Possebon APR, Faot F, Machado RMM, Nascimento GG, Leite FRM. Exploratory and confirmatory factorial analysis of the OHIP-Edent instrument. Braz Oral Res 2018; 32: e111. https://doi.org/10.1590/1807-3107bor-2018.vol32.0111
    » https://doi.org/https://doi.org/10.1590/1807-3107bor-2018.vol32.0111
  • 52 Abanto J, Tsakos G, Ardenghi TM, Paiva SM, Raggio DP, Sheiham A, et al. Responsiveness to change for the Brazilian Scale of Oral Health Outcomes for 5-year-old children (SOHO-5). Health Qual Life Outcomes 2013; 11: 137. https://doi.org/10.1186/1477-7525-11-137
    » https://doi.org/https://doi.org/10.1186/1477-7525-11-137
  • 53 Abanto J, Paiva SM, Sheiham A, Tsakos G, Mendes FM, Cordeschi T, et al. Changes in preschool children’s OHRQoL after treatment of dental caries: responsiveness of the B-ECOHIS. Int J Paediatr Dent 2016; 26(4): 259-65. https://doi.org/10.1111/ipd.12192
    » https://doi.org/https://doi.org/10.1111/ipd.12192
  • 54 Godinho GF, Cavalheiro A, Luís HS, Mexia R. Validation of the Oral Impacts on Daily Performance index among the Portuguese population. Cienc Saude Colet 2018; 23(12): 4351-61.
  • 55 Amaral J, Sanches C, Marques D, Patto JV, Barcelos F, Mata A. Validation of oral health impact profile-14 and its association with hypossialia in a Sjögren’s Syndrome Portuguese population. Acta Reumatol Port 2018; 43(2): 137-45. PMID: 30091957
  • 56 Da Mata ADSP, da Silva Marques DN, Freitas FMF, de Almeida Rato JP, Trindade RTVMR, Barcelos FAD, et al. Translation, validation, and construct reliability of a Portuguese version of the Xerostomia Inventory. Oral Dis 2012; 18(3): 293-8. https://doi.org/10.1111/j.1601-0825.2011.01879.x
    » https://doi.org/https://doi.org/10.1111/j.1601-0825.2011.01879.x
  • 57 Praveen S, Parmar J, Chandio N, Arora A. A systematic review of cross-cultural adaptation and psychometric properties of oral health literacy tools. Int J Environ Res Public Health 2021; 18(19): 10422. https://doi.org/10.3390/ijerph181910422
    » https://doi.org/https://doi.org/10.3390/ijerph181910422
  • 58 Rodrigues Neto JF, Ferreira CG. Qualidade de vida como medida de desfecho em saúde. Rev Méd Minas Gerais 2003; 13(1): 42-6.
  • 59 Krahn M, Bremner KE, Tomlinson G, Ritvo P, Irvine J, Naglie G. Responsiveness of disease-specific and generic utility instruments in prostate cancer patients. Qual Life Res 2007; 16(3): 509-22. https://doi.org/10.1007/s11136-006-9132-x
    » https://doi.org/https://doi.org/10.1007/s11136-006-9132-x
  • 60 Roncada C, Mattiello R, Pitrez PM, Sarria EE. Specific instruments to assess quality of life in children and adolescents with asthma. J Pediatr (Rio J) 2013; 89(3): 217-25. https://doi.org/10.1016/j.jped.2012.11.010
    » https://doi.org/https://doi.org/10.1016/j.jped.2012.11.010
  • 61 Borsa JC, Damásio BF, Bandeira DR. Adaptação e validação de instrumentos psicológicos entre culturas: algumas considerações. Paidéia (Ribeirão Preto) 2012; 22(53): 423-32. https://doi.org/10.1590/S0103-863X2012000300014
    » https://doi.org/https://doi.org/10.1590/S0103-863X2012000300014
  • 62 Rodríguez AM, Concha PJ, Pereira DI, Machuca LL. Adaptação transcultural e validação de um questionário de cuidado humanizado em enfermagem para uma amostra da população Chilena. Rev Cuid 2018; 9(2): 2245-56. https://doi.org/10.15649/cuidarte.v9i2.531
    » https://doi.org/https://doi.org/10.15649/cuidarte.v9i2.531
  • 63 Espinoza SC, Novoa-Muñoz F. Advantages of ordinal alpha versus Cronbach’s alpha, illustrated using the WHO AUDIT test. Rev Panam Salud Publica 2018; 42: e65. https://doi.org/10.26633/RPSP.2018.65
    » https://doi.org/https://doi.org/10.26633/RPSP.2018.65
  • 64 Douglas-de-Oliveira DW, Vitor GP, Silveira JO, Martins CC, Costa FO, Cota LOM. Effect of dentin hypersensitivity treatment on oral health related quality of life - a systematic review and meta-analysis. J Dent 2018; 71: 1-8. https://doi.org/10.1016/j.jdent.2017.12.007
    » https://doi.org/https://doi.org/10.1016/j.jdent.2017.12.007
  • 65 Hora HRM, Monteiro GTR, Arica J. Confiabilidade em questionários para qualidade: um estudo com o Coeficiente Alfa de Cronbach. Produto & Produção 2010; 11(2): 85-103. https://doi.org/doi:10.22456/1983-8026.9321
    » https://doi.org/https://doi.org/doi:10.22456/1983-8026.9321
  • 66 Streiner DL. Being inconsistent about consistency: when coefficient alpha does and doesn’t matter. J Pers Assess 2003; 80(3): 217-22. https://doi.org/10.1207/S15327752JPA8003_01
    » https://doi.org/https://doi.org/10.1207/S15327752JPA8003_01
  • 67 Jeffrey J, Klomhaus A, Enenbach M, Lester P, Krishna R. Self-report rating scales to guide measurement-based care in child and adolescent psychiatry. Child Adolesc Psychiatr Clin N Am 2020; 29(4): 601-29. https://doi.org/10.1016/j.chc.2020.06.002
    » https://doi.org/https://doi.org/10.1016/j.chc.2020.06.002
  • 68 Callias M. Handbook of research methods in clinical psychology. J Child Psychol Psychiatry 2005; 32: 559. https://doi.org/10.1111/j.1469-7610.2004.01473_1.x
    » https://doi.org/https://doi.org/10.1111/j.1469-7610.2004.01473_1.x
  • 69 Wang X, Cheng Z. Cross-sectional studies: strengths, weaknesses, and recommendations. Chest 2020; 158(1S): S65-S71. https://doi.org/10.1016/j.chest.2020.03.012
    » https://doi.org/https://doi.org/10.1016/j.chest.2020.03.012
  • 70 Burchett D, Ben-Porath YS. Methodological considerations for developing and evaluating response bias indicators. Psychol Assess 2019; 31(12): 1497-511. https://doi.org/10.1037/pas0000680
    » https://doi.org/https://doi.org/10.1037/pas0000680
  • 71 Bykov K, Patorno E, D’Andrea E, He M, Lee H, Graff JS, et al. Prevalence of avoidable and bias-inflicting methodological pitfalls in real-world studies of medication safety and effectiveness. Clin Pharmacol Ther 2022; 111(1): 209-17. https://doi.org/10.1002/cpt.2364
    » https://doi.org/https://doi.org/10.1002/cpt.2364
  • 72 Baddini-Martinez J, Ferreira J, Tanni S, Alves LR, Cabral Junior BF, Carvalho CRR, et al. Diretrizes brasileiras para o tratamento farmacológico da fibrose pulmonar idiopática. Documento oficial da Sociedade Brasileira de Pneumologia e Tisiologia baseado na metodologia GRADE. J Bras Pneumol 2020; 46(2): e20190423. https://doi.org/10.36416/1806-3756/e20190423
    » https://doi.org/https://doi.org/10.36416/1806-3756/e20190423
  • 73 Slack MK, Draugalis JR. Establishing the internal and external validity of experimental studies. Am J Health Syst Pharm 2001; 58(22): 2173-81; quiz 2182-3. https://doi.org/10.1093/ajhp/58.22.2173
    » https://doi.org/https://doi.org/10.1093/ajhp/58.22.2173
  • 74 Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336(7650): 924-6. https://doi.org/10.1136/bmj.39489.470347.AD
    » https://doi.org/https://doi.org/10.1136/bmj.39489.470347.AD
  • 75 Fleck MPA. O instrumento de avaliação de qualidade de vida da Organização Mundial da Saúde (WHOQOL-100): características e perspectivas. Ciênc Saúde Coletiva 2000; 5: 33-8. https://doi.org/10.1590/S1413-81232000000100004
    » https://doi.org/https://doi.org/10.1590/S1413-81232000000100004
  • 76 Ben Bouazza Y, Chiairi I, El Kharbouchi O, De Backer L, Vanhoutte G, Janssens A, et al. Patient-reported outcome measures (PROMs) in the management of lung cancer: a systematic review. Lung Cancer 2017; 113: 140-51. https://doi.org/10.1016/j.lungcan.2017.09.011
    » https://doi.org/https://doi.org/10.1016/j.lungcan.2017.09.011
  • 77 Loures MC, Porto CC. A avaliação da qualidade de vida: guia para profissionais de saúde. Ciênc Saúde Coletiva 2011; 16(7): 3349-50. https://doi.org/10.1590/S1413-81232009000600040
    » https://doi.org/https://doi.org/10.1590/S1413-81232009000600040
  • 78 Churruca K, Pomare C, Ellis LA, Long JC, Henderson SB, Murphy LE, et al. Patient-reported outcome measures (PROMs): a review of generic and condition-specific measures and a discussion of trends and issues. Health Expect 2021; 24(4): 1015-24. https://doi.org/10.1111/hex.13254
    » https://doi.org/https://doi.org/10.1111/hex.13254
  • FUNDING:
    none.

Publication Dates

  • Publication in this collection
    27 Oct 2023
  • Date of issue
    2023

History

  • Received
    22 Mar 2023
  • Reviewed
    10 July 2023
  • Accepted
    01 Aug 2023
location_on
Associação Brasileira de Saúde Coletiva Av. Dr. Arnaldo, 715 - 2º andar - sl. 3 - Cerqueira César, 01246-904 São Paulo SP Brasil , Tel./FAX: +55 11 3085-5411 - São Paulo - SP - Brazil
E-mail: revbrepi@usp.br
rss_feed Acompanhe os números deste periódico no seu leitor de RSS
Acessibilidade / Reportar erro