Open-access Reliability and construct validity of the Long Version of Disability Assessment for Dementia (Brazilian Version)

Confiabilidade e validade de construto da Escala de Avaliação de Incapacidade na Demência - Versão Longa (DADL-BR)

Abstract

Objectives  Evaluate the reliability test-retest (intra and inter-examiner) and construct validity of the Long Version of Disability Assessment for Dementia (Brazilian Version - DADL-BR).

Methods  The DADL-BR was applied to 58 caregivers/family of older adults with dementia. The inter-examiner (n = 30) and intra-examiner (n = 28) reliability was assessed using the kappa test and Pearson’s correlation coefficient, and the internal consistency was assessed using Cronbach’s alpha. The construct validity (n = 48) was performed by comparing DADL-BR with MMSE.

Results  The results of the intra and inter-examiner demonstrated good reliability (0.72 and 0.74), as well as nearly perfect correlations (0.99) for both inter- and intra-examiner reliability. The internal consistency was excellent (0.87) and the results demonstrated a good correlation (0.74) between DADL-BR and MMSE in construct validity analysis.

Conclusions  DADL-BR can be considered reliable instrument and with good construct validity, considering internal consistency and test-retest reliability and may be constitute a useful instrument for assessing the occupational performance profile of older adults with dementia.

Keywords:  Activities of Daily Living; Aged; Dementia; Reproducibility of Results; Validity of Tests

Resumo

Objetivos  Avaliar a confiabilidade teste-reteste (intra e inter-examinadores) e a validade de construto da Versão Longa da Escala de Avaliação de Incapacidade na Demência - Versão Longa (DADL-BR).

Métodos  O DADL-Br foi aplicado em 58 cuidadores/familiares de idosos com demência. A confiabilidade inter-examinadores (n = 30) e intrae-xaminadores (n = 28) foi avaliada pelo teste kappa e coeficiente de correlação de Pearson, e a consistência interna foi avaliada pelo Alfa de Cronbach. A validade de construto (n = 48) foi realizada comparando-se o DADL-BR com o Mini Exame do Estado Mental (MEEM).

Resultados  Os resultados intra e inter-examinadores demonstraram boa confiabilidade (0,72 e 0,74), bem como correlações quase perfeitas (0,99) tanto para confiabilidade inter quanto intra-examinadores. A consistência interna foi excelente (0,87) e os resultados demonstraram boa correlação (0,74) entre o DADL-BR e o MEEM na análise de validade de constructo.

Conclusões  O DADL-BR pode ser considerado um instrumento confiável e com boa validade de constructo, considerando-se a consistência interna e a confiabilidade teste-reteste, e útil para avaliar o perfil de desempenho ocupacional de idosos com demência.

Palavras-chave:  Atividades de Vida Diária; Idoso; Demência; Reprodutibilidade dos Resultados; Validade dos Testes

Introduction

According to the Azheimer’s Disease International there are currently 55 million people living with dementia in the world, and this number will set to rise to 139 million by 2050, with the greatest increases in low middle income countries. Furthermore, dementia is considered as a major cause of disability and mortality among the older adults population; therefore, monitoring its prevalence is considered a priority in the healthcare area (Alzheimer’s Disease International, 2023).

Dementia mostly results from chronic and progressive neurodegenerative diseases and is associated with loss of physical function and cognition, significantly interfering with the lives of the patients, as well as their families. Dementia is a highly disabling disorder that results in intense and continuous care that can cause anxiety and depression in caregivers (Cieto et al., 2014).

To promote improved quality of life during aging and to prevent functional losses that are commonly seen in this population, health workers must be equipped with technologies that allow them to perform correct diagnoses, contributing to more secure and cohesive intervention (Fonseca & Rizzotto, 2008).

In this context, functional assessments are considered of extreme importance because they identify functional deficits by verifying the level of that disease interference with occupational performance and assist in the diagnosis of dementia (Duarte et al., 2007).

According to the American Occupational Therapy Association (AOTA), the occupational performance is characterized by the complex and dynamic interaction of client factors (values, beliefs and spirituality, functions, and body structures); performance skills (motor, process, and social interaction skills); performance patterns (habits, routines, rituals, and roles); and contexts and environments (cultural, personal, physical, social, temporal, and virtual) (American Occupational Therapy Association, 2008, 2014).

This complex and dynamic interaction is realized in the Occupations, that are daily activities, in which the subjects are engaged, specifically basic activities of daily living (BADL), rest and sleep, instrumental activities of daily living (IADL), education, work, leisure, and social participation. These activities have purpose, meaning, and produce a sense of usefulness to the individual (American Occupational Therapy Association, 2008, 2014).

The Disability Assessment for Dementia (DAD) evaluates functional abilities in activities of daily living, through the caregiver’s perceptions of the older adults with dementia and consists of 40 items (17 related to BADL, 22 related to the IADL, and 1 related to leisure) (Gélinas et al., 1999).

The version adapted for use in Brazil (DAD-BR; Carthery-Goulart et al., 2007), despite being considered the most complete among the functional assessment tools for dementia used in Brazil, is missing many occupations and activities that are present in the daily lives of the older adults. Due to these omissions, the DAD-BR may underestimate the occupational performance of the older adults (Canon & Novelli, 2013).

The Long Version of Disability Assessment for Dementia (DADL-BR; Canon et al., 2016) includes items from the AOTA, related to occupations and based on Brazilian cultural context. The instrument is composed of 20 domains (hygiene, dressing, continence, eating, functional mobility, personal device care, sexual activity, rest and sleep, meal preparation, communication management, going on outings, finance and correspondence, health maintenance, care of others, care of pets, leisure and housework, religious activity, education, work, and social participation) and 109 items.

The DADL-BR is an instrument with a wide repertoire of activities, that allows a reliable evaluation of the daily life of the older adults with dementia, providing a mapping of occupational performance, since, like the DAD-BR, it evaluates the impact of cognition on functional performance, based on executive functioning (initiative, organization/planning and effective realization of the activities) (Canon et al., 2016).

Another relevant aspect of DADL-BR, is that it allows the choice of domains to be evaluated, based on the group of activities, since the score is independent, being done by domain (Canon et al., 2016).

This instrument contributes to the establishment of an adequate intervention plan for the skills and limitations of the older adults with dementia, as well as to monitor the evolution of the disease, about occupational performance (Canon et al., 2016).

The Scientific Advisory Committee of the Medical Outcomes Trust states that, in addition to cross-cultural adaptation, it is important to evaluate reliability and validity, which are are fundamental to the choice of an assessment tool, once its effectiveness is proved (Pasquali, 1996; Alexandre & Coluci, 2011; Lohr, 2002)

The first is the ability to reproduce the results consistently over time and space (intra-examiner reproducibility) or across different observers (inter-examiner reproducibility). The latter is the degree to which the instrument measures what it purports to measure, thus verifying the consistency of content,

The analysis of internal consistency, which is a measure of the accuracy of the assessment tool, that will quantify the homogeneity of the instrument, is another aspect that should be considered (Alexandre & Coluci, 2011).

The objective of this study was to evaluate the reliability of DADL-BR, as well as the construct validity, by comparing it with the Mini-Mental State Examination (MMSE) (Folstein et al., 1975; Brucki et al., 2003).

Materials and Methods

This is a descriptive, observational, and cross-sectional study. The study was approved by the Research Ethics Committee of the Federal University of São Paulo (approval number 311.416), and all participants signed the consent form.

The study was carried out in three clinics at the Federal University of São Paulo with a convenience sample of 58 caregivers that responded to DADL-BR and 48 older adults with dementia who responded to MMSE.

The inclusion criteria for the older adults participants were to be 60 years or older, to have a diagnosis of dementia according to the criteria of International Statistical Classification of Diseases and Related Health Problems (ICD-10; Wells et al., 2011) and Diagnostic and Statistical Manual of Mental Disorders (DSM-V; American Psychiatric Association, 2013), duly documented in the medical records, and without language disorder. The caregiver participant inclusion criteria were to be 18 years or older, in contact with the older adults participant for at least 24 h/week or 3 hours per day and to not have a diagnosis of psychiatric illness (e.g., schizophrenia, bipolar disorder, or severe depression).

The analysis of intra-examiner reliability of DADL-BR included data from 28 caregivers, with a range of 15-20 days between the two applications, to verify that the results produced are consistent across time. The second application occurred by telephone, because not all subjects of the sample could return to the clinics within the period specified in the study. Many participants were low-income and additional in-person consultations could burden the family budget, as well as disrupt their work routine.

The analysis of inter-examiner reliability included data from 30 caregivers. In this evaluation both examiners were present at the time of application, one of the two conducted the assessment and both scored, from the caregivers responses, their evaluation sheets independently.

The analysis of internal consistency included data from 58 caregivers and aimed to correlate each domain of DADL-BR with the total score of the DADL-BR to assess the homogeneity of the instrument.

For the construct validity, DADL-BR was correlated with MMSE. The MMSE is a cognitive screening test that assesses the following aspects: temporal and spatial orientation, short-term memory, attention, calculation, language, and visuospatial ability and the scores range from 0 to 30 points. The sample used for this analysis consisted of 48 caregivers who responded to DADL-BR and 48 older adults participants with dementia who responded the MMSE.

We hypothesized that older adults participants with less cognitive impairment as measured on MMSE would have higher occupational performance as measured by DADL-BR.

For reliability analysis were used Pearson’s correlation coefficients to analyze the correlations between the evaluations and Kappa test to confirm the correlations in the ratings. The analysis of internal consistency used the Cronbach’s alpha, with acceptable values of 0.7-0.95 (Tavakol & Dennick, 2011).

For the construct validity were used the Pearson’s and Spearman’s correlation coefficients, associating the total score of DADL-BR and their domains and the total score of DADL-BR with MMSE.

The level of significance was set for values of p < 0.05.

Results

Table 1 presents the demographics data (age, sex and education) of the caregivers and the older adults participants with dementia.

Table 1
Demographics data of older adults with dementia and their caregivers.

Regarding to the relationship between the caregivers with the older adults with dementia: 43.1% were children, 36.2% were spouses, 13.8% were formal caregivers, and 6.9% were another degree of kinship (niece and sister).

Table 2 presents the Pearson’s correlation coefficient and the values of Kappa obtained in the intra- and inter-examiner assessment.

Table 2
Pearson’s correlation coefficient and Values of Kappa for intra- and inter-examiner assessment of the DADL-BR.

The Cronbach’s alpha, was α = 0.87, suggesting a good and acceptable value (Tavakol & Dennick, 2011).

Table 3 presents the internal consistency of DADL-BR, as measured by the correlation of domains with the total score through the Spearman’s coefficient.

Table 3
Spearman’s correlation coefficient between the domains and the total score of DADL-BR (n = 58).

The internal consistency of DADL-BR was high and most of the domains were statistically significant with the total score of DADL-BR. Only three domains did not present significant correlation with the total score (Sexual Activity, Care of Pets and Work).

Table 4 presents the values obtained in the Spearman’s correlation coefficient between the domains of DADL-BR and the total score of MMSE and the values obtained in the Pearson’s correlation coefficient among the total score of DADL-BR and the total score of MMSE.

Table 4
Correlation coefficient (Pearson and Spearman) between the domains and the total score of DADL-BR with the total score of Mini-Mental State Examination (n = 48).

The results presented in Table 4 demonstrated high correlation coefficients and were statistically significant for most domains.

Discussion

Table 5 presents the results obtained in the original study (DAD), the Brazilian version (DAD-BR) and the Brazilian Long Version (DADL-BR) in relation to reliability (correlation coefficients and internal consistency).

Table 5
Comparison of the results of reliability (correlation coefficient and internal consistency) among the original version of DAD, the DAD-BR, and DADL-BR.

The results were very similar across the measures, adding items did not affect the psychometric properties even improved the internal consistency of the instrument of the Brazilian version.

Landis & Koch (1977) proposed criteria that are widely used in reliability studies, to stratify the results of kappa test: 0.8-1.0 is almost perfect, 0.6-0.8 good (substantial), 0.6-0.4 moderate, 0.4-0.2 regular, 0.0-0.2 discreet, and 0.0 to 1.0 poor.

Based on the results of this study, the intra and inter-examiner reliability of DADL-BR was considered good, and the Pearson’s correlation coefficient was almost perfect.

High values of internal consistency show that all items that compose the assessment tool evaluate the same construct or phenomenon. Tavakol & Dennick (2011) suggest that the acceptable range of values are between 0.7 and 0.95. However, some authors report that values above 0.90 may indicate that some items are redundant and test the same question in different ways. This was not observed in DADL-BR, since the maximum value obtained was 0.87.

Similar to the DAD-BR, DADL-BR was also significantly correlated with MMSE (0.74, p<0.01), thus demonstrating the construct validity of the instrument. Such findings corroborate with validation studies performed with the DAD in Italy (De Vreese et al., 2008) and China (Mok et al., 2005). As the MMSE and the DADL-Br assess different constructs were not expected to find perfect correlations between instruments, but an impact of the cognition on functionality.

Only three domains showed no statistically significant correlation: Sexual Activity, Care of Pets, and Work, which can be justified, by the high prevalence of “Not Applicable—N/A” responses to the corresponding items, thus demonstrating that the older adults of this sample never performed such activities and/or ceased to perform them before the diagnosis of dementia.

The functional performance is an important criterion to be evaluated for the correct diagnosis of Neurocognitive Disorders (NCD), which according to DSM-5 are differentiated in Mild and Major, the first one related to Mild Cognitive Impairment (MCI) and the second to dementia (Sachdev et al., 2014).

The diagnostic criteria for dementia are based on impairments in areas such as: attention, executive functioning, learning and memory, language, motor perception and social cognition, which significantly interfere in the social and occupational functioning of the older adults (Sachdev et al., 2014).

Individuals diagnosed with mild NCD or MCI tend to remain autonomous and independent in most daily activities, as only subtle changes in functional performance are reported, mainly in the advanced activities of daily living, that demand more of cognitive functioning and are not essential in the lives of individuals, such as driving, using technology, and traveling (Rodakowski et al., 2014; Jekel et al., 2015). The inclusion of these items related to advanced activities could provide more concrete data about the functional performance of these individuals.

Studies (Rodakowski et al., 2014; Jekel et al., 2015) indicate that activities such as shopping, medication intake, telephone use, use of technological equipment and management of appointments already present problems in their execution by MCI older adults.

The correct diagnosis for MCI is fundamental, since this may be a pre-clinical stage of dementia, and the sooner the diagnosis is made, the better, since early intervention can prevent or delay this conversion (De Vriendt et al., 2013).

The current scenario, in wich, approximately 46.8 million people in the world have dementia (World Health Organization, 2016), a highly disabling illness, which generates a high level of stress and burden on the caregivers, as well as substantial number of financial resources, functional assessment tools that contribute for a correct diagnosis, they become fundamental.

Another aspect that is important to mention in the use and application of this expanded version is the possibility to refine the perception about the performance in daily function allows us to understand better this performance and which activities, within each group, can be the best predictors of dementia (Giebel et al., 2015; Di Carlo et al., 2016), as well as to identify the initial limitations in the activities, providing interventions that stimulate independence for as long as possible.

An important characteristic of this scale is that the total score is based on the groups of activities that make up the daily lives of older adults, being expressed as a percentage of functional performance capacity. And it is also possible to analyze the functional performance capacity in the subgroups of activities using the same reasoning to produce the total score, which is calculated with the following formula: total “YES” responses divided by 109 (referring to the total number of items), minus the number of “NA” responses, times 100 (“yes” responses/109 - “NA” responses” x 100). The maximum score is 100%, and the higher the score, the better the person's occupational performance.

Few limitations of the study were identified: difficulty in performing the two assessments intra-examiner face to face, due to the economic condition of the population and the availability to participate of the study; the shortage in literature of studies that could discuss the data from occupational performance of the older adults with dementia, specifically regarding the activities individually and not only in relation to the groups of activities (occupations), to corroborate the findings of this research, and the studies with a larger sample could assist in the analysis of the stability of the psychometric properties of DADL-BR.

In conclusion, the analysis of the psychometric properties demonstrated excellent internal consistency of DADL-BR, as well as a good inter- and intra-examiner reliability. Construct validity was confirmed by good correlation coefficients with MMSE, confirming the hypothesis that less cognitive impairment is related to better occupational performance of older adults with dementia.

Acknowledgements

This study was financially supported by the Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP).

  • How to cite: Canon, M. B. F., Silva, F. P., Saviotti, K. R. S. S., Gélinas, I., Lemos, N. F. D., & Novelli, M. M. P. C. 2024). Reliability and construct validity of the Long Version of Disability Assessment for Dementia (Brazilian Version). Cadernos Brasileiros de Terapia Ocupacional, 32, e3657. https://doi.org/10.1590/2526-8910.ctoAO281936572
  • Funding Source
    The São Paulo Research Foundation - Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP - 2013/10943-0).

References

  • Alexandre, N. M., & Coluci, M. Z. (2011). Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciencia & Saude Coletiva, 16(7), 3061-3068.
  • Alzheimer’s Disease International - ADI. (2023). Dementia facts & figures. Retrieved in 2023, September 23, from https://www.alzint.org/about/dementia-facts-figures
    » https://www.alzint.org/about/dementia-facts-figures
  • American Occupational Therapy Association - AOTA. (2008). Occupational therapy practice framework: domain and process (2nd edition). The American Journal of Occupational Therapy, 62(6), 625-683.
  • American Occupational Therapy Association - AOTA. (2014). Occupational therapy practice framework: domain and process (3rd edition). The American Journal of Occupational Therapy, 68(Suppl. 1), S1-S48.
  • American Psychiatric Association - APA. (2013). Diagnostic and statistical manual of mental disorders. Arlington: American Psychiatric Publishing.
  • Brucki, S. M. D., Nitrini, R., Caramelli, P., Bertolucci, P. H. F., & Okamoto, I. H. (2003). Sugestões para o uso do mini-exame do estado mental no Brasil. Arquivos de Neuro-Psiquiatria, 61(3B), 777-781.
  • Canon, M. B. F., & Novelli, M. (2013). Estudo dos instrumentos de avaliação funcional em demência comumente utilizados no Brasil. Revista de Terapia Ocupacional da Universidade de São Paulo, 23(3), 253-262.
  • Canon, M. B. F., Almeida, M. H. M., & Novelli, M. M. P. C. (2016). Escala de avaliação de incapacidade na demência versão longa (DADL-BR). Cadernos de Terapia Ocupacional da UFSCar, 24(2), 323-334.
  • Carthery-Goulart, M. T., Areza-Fegyveres, R., Schultz, R. R., Okamoto, I., Caramelli, P., Bertolucci, P. H. F., & Nitrini, R. (2007). Adaptação transcultural da escala de avaliação de incapacidade em demência (Disability Assessment For Dementia - DAD). Arquivos de Neuro-Psiquiatria, 65(3b), 916-919.
  • Cieto, B. B., Valera, G. G., Soares, G. B., Cintra, R. H. S., & Vale, F. A. C. (2014). Dementia care in public health in Brazil and the world: a systematic review. Dementia & Neuropsychologia, 8(1), 40-46.
  • De Vreese, L. P., Caffarra, P., Savarè, R., Cerutti, R., Franceschi, M., & Grossi, E. (2008). Functional disability in early Alzheimer’s disease: a validation study of the Italian version of the disability assessment for dementia scale. Dementia and Geriatric Cognitive Disorders, 25(2), 186-194.
  • De Vriendt, P., Gorus, E., Cornelis, E., Bautmans, I., Petrovic, M., & Mets, T. (2013). The advanced activities of daily living: a tool allowing the evaluation of subtle functional decline in mild cognitive impairment. The Journal of Nutrition, Health & Aging, 17(1), 64-71.
  • Di Carlo, A., Baldereschi, M., Lamassa, M., Bovis, F., Inzitari, M., Solfrizzi, V., Panza, F., Galluzzo, L., Scafato, E., & Inzitari, D. (2016). Daily function as predictor of dementia in Cognitive Impairment, No Dementia (CIND) and Mild Cognitive Impairment (MCI): an 8-year follow-up in the ILSA study. Journal of Alzheimer’s Disease, 53(2), 505-515.
  • Duarte, Y. A. O., Andrade, C. L., & Lebrão, M. L. (2007). O Índex de Katz na avaliação da funcionalidade dos idosos. Revista da Escola de Enfermagem da U S P., 41(2), 317-325.
  • Folstein, M. F., Folstein, S. E., & McHugh, P. R. (1975). “Mini-mental state”: a practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research, 12(3), 189-198.
  • Fonseca, F. B., & Rizzotto, M. L. F. (2008). Instrument construction for socio-functional evaluation in elderly people. Texto & Contexto Enfermagem, 17(2), 365-373.
  • Gélinas, I., Gauthier, L., McIntyre, M., & Gauthier, S. (1999). Development of a functional measure for persons with Alzheimer’s disease: the disability assessment for dementia. The American Journal of Occupational Therapy, 53(5), 471-481.
  • Giebel, C. M., Challis, D., & Montaldi, D. (2015). Understanding the cognitive underpinnings of functional impairments in early dementia: a review. Aging & Mental Health, 19(10), 859-875.
  • Jekel, K., Damian, M., Wattmo, C., Hausner, L., Bullock, R., Connelly, P. J., Dubois, B., Eriksdotter, M., Ewers, M., Graessel, E., Kramberger, M. G., Law, E., Mecocci, P., Molinuevo, J. L., Nygård, L., Olde-Rikkert, M. G., Orgogozo, J. M., Pasquier, F., Peres, K., Salmon, E., Sikkes, S. A. M., Sobow, T., Spiegel, R., Tsolaki, M., Winblad, B., & Frölich, L. (2015). Mild cognitive impairment and deficits in instrumental activities of daily living: a systematic review. Alzheimer’s Research & Therapy, 7(1), 17.
  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174.
  • Lohr, K. N. (2002). Assessing health status and quality-of-life instruments: attributes and review criteria. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 11(3), 193-205.
  • Mok, C. C., Siu, A. M., Chan, W. C., Yeung, K. M., Pan, P. C., & Li, S. W. (2005). Functional disabilities profile of chinese elderly people with Alzheimer’s disease - a validation study on the chinese version of the disability assessment for dementia. Dementia and Geriatric Cognitive Disorders, 20(2-3), 112-119.
  • Pasquali, L. (1996). Theory and methods of measurement in behavioral sciences. Brasilia: Research Laboratory Evaluation and Measurement, Institute of Psychology.
  • Rodakowski, J., Skidmore, E. R., Reynolds III, C. F., Dew, M. A., Butters, M. A., Holm, M. B., Lopez, O. L., & Rogers, J. C. (2014). Can performance on daily activities discriminate between older adults with normal cognitive function and those with mild cognitive impairment? Journal of the American Geriatrics Society, 62(7), 1347-1352.
  • Sachdev, P. S., Blacker, D., Blazer, D. G., Ganguli, M., Jeste, D. V., Paulsen, J. S., & Petersen, R. C. (2014). Classifying neurocognitive disorders: the DSM-5 approach. Nature Reviews. Neurology, 10(11), 634-642.
  • Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2(2), 53-55.
  • Wells, R. H. C., Bay-Nielsen, H., Braun, R., Israel, R. A., Laurenti, R., Maguin, P., & Taylor, E. (2011). CID-10: classificação estatística internacional de doenças e problemas relacionados à saúde. São Paulo: EDUSP.
  • World Health Organization - WHO. (2016). Mental health of older adults. Geneva. Retrieved in 2023, September 23, from http://www.who.int/mediacentre/factsheets/fs381/en/
    » http://www.who.int/mediacentre/factsheets/fs381/en/

Edited by

  • Section editor
    Profa. Dra. Mariana Midori Sime

Publication Dates

  • Publication in this collection
    15 Apr 2024
  • Date of issue
    2024

History

  • Received
    16 Oct 2023
  • Reviewed
    26 Oct 2023
  • Reviewed
    10 Dec 2023
  • Reviewed
    13 Jan 2024
  • Accepted
    21 Feb 2024
location_on
Universidade Federal de São Carlos, Departamento de Terapia Ocupacional Rodovia Washington Luis, Km 235, Caixa Postal 676, CEP: , 13565-905, São Carlos, SP - Brasil, Tel.: 55-16-3361-8749 - São Carlos - SP - Brazil
E-mail: cadto@ufscar.br
rss_feed Acompanhe os números deste periódico no seu leitor de RSS
Acessibilidade / Reportar erro