Reliability of a questionnaire assessing daily practices of community mental health workers

Abstract

Objective

Bioethical questions have been raised among community health workers in terms of the perceptions and threats that they face during their daily labor practices. Thus, questionnaires are required for assessing the issues experienced by these workers and the psychological effects experienced by primary care workers. Therefore, this study demonstrates the reliability of a double scale based on a pilot study involving community health workers.

Methods

A scale-based and validated methodological investigation was developed by including 97 community health workers in the Brazilian cities of Riacho Fundo I and Riacho Fundo II, located within the administrative region of Brasilia.

Results

The perception scale’s internal consistency exhibited good Cronbach’s alpha values (0.76 overall, and >0.75 for the different dimensions). Furthermore, the factor analysis presented a 3-factor solution with ratio significance.

Conclusion

The scale exhibits good reliability and psychometric properties and has potential for use in future research.

Community health workers; Primary health care; Bioethics; Reproducibility of results; Questionnaires

Resumo

Objetivo

Questões bioéticas foram levantadas entre agentes comunitários de saúde sobre as percepções e ameaças enfrentadas por eles durante suas atividades diárias de trabalho. Deste modo, questionários são necessários para avaliar os problemas enfrentados por esses profissionais e os efeitos psicológicos enfrentados por trabalhadores na área de cuidados primários. Para isso, este estudo demonstra a confiabilidade de uma dupla escala baseada em um estudo piloto envolvendo agentes comunitários de saúde.

Métodos

Uma investigação metodologicamente validada e baseada em escala foi desenvolvida incluindo 97 agentes comunitários de saúde nas cidades brasileiras de Riacho Fundo I e Riacho Fundo II, localizadas na região administrativa de Brasília.

Resultados

A consistência interna da escala de percepção mostrou bons valores de coeficiente alfa de Cronbach (0,76 no geral, e >0,75 para as diferentes dimensões). Além disso, o fator de análise apresentou uma solução de 3 fatores com significância proporcional.

Conclusão

A escala mostra boa confiabilidade e boas propriedades psicométricas, e tem potencial para uso em pesquisas futuras.

Agentes comunitários de saúde; Atenção primária à saúde; Bioética; Reprodutibilidade dos testes; Questionários

Introduction

Mental health is a broad field of research that combines various sciences and types of professions in an attempt to observe, investigate, and understand human beings in a multifaceted manner involving familial, social, psychological, and psychopathological aspects.¹ Brazil’s Unified Health System (SUS) was created and based on this multidisciplinary principle that considers an integrated and regional perspective, with primary care as the first step for all patients’ health issues.

The concept of basic care is different from that of primary healthcare. According to the Brazilian Ministry of Health and the principles of SUS, there is no distinction between the areas of basic and primary healthcare. Primary care is defined as “a set of individual or collective measures performed at the primary care level of health systems that focused on fostering health, preventing illness, treatment, and rehabilitation.”²

Considering the magnitude of mental issues experienced by the Brazilian population, it is easy to see how primary care teams confront mental health issues in their daily routines. According to the World Health Report’s (2001) recommendation, the inclusion of mental health measures in primary care was a subject discussed at the Third National Mental Health Conference in Brazil (CNSM).³ The proposal that was approved recommends healthcare training for Family Health Program (PSF) teams, the inclusion of multi-professional mental health teams to work with Family Health Teams (ESFs), and the inclusion of a specific printed form for collecting data on individuals with psychological disorders who are treated by Family Health Teams.⁴

In this organizational context, psychiatric care reform in Brazil is in the initial stages of its inclusion in primary care. Psychiatric care reform is a complex political, social, and historical process including measures, institutions, and forces of different origins. It has only been successful in places where intensive and continued follow-up programs have been created within the community. Thus, the important interface between mental health networks is represented by substitute services such as Psycho-Social Care Centers (CAPS), Family Health Support Centers (NASF), residencies, social centers, and (ambulatory) healthcare centers, all of which represent primary healthcare.

With the positive result obtained in the northeastern region of Brazil, the Family Health Program (PSF) was expanded to the remaining country. This expansion caused an improvement in primary care and the establishment of Community Health Workers (CHWs).⁴ The Ministry of Health has stated on its website that in December 2012, the number of registered CHWs in Brasília was 849.⁵

CHWs’ daily activities involve issues that require to be understood well because these professionals depend on the support of other professionals for good performance and sustainability.⁶ Examples include CHWs’ entry into home environments, which requires involvement in families’ internal conflicts; the often unsuccessful attempt to mediate the interaction between the health team and individuals; measures to promote healthcare despite people’s resistance to changing modern unhealthy habits; CHWs’ personal beliefs regarding healthcare, which may be in conflict with scientific concepts; strategies to manage feelings of powerlessness despite the socioeconomic and cultural conditions that hinder patients’ healthy behaviors; and difficulties that arise at work because of difficulties in teams’ interpersonal relationships.⁷,⁸ CHWs must experience these and other issues in their daily tasks. They raise the bioethical questions experienced by these workers and require us to reflect upon the professionals’ (who work in Family Health Programs) practices. They reinforce the requirement for heightened sensitivity and ethical commitment on behalf of primary care professionals.⁹

To determine CHWs’ daily work experiences, it is important to consider workers’ opinions. Concerning public healthcare management, the major challenge is to maximize the quality of health services to society while respecting budgetary limitations. The population that largely utilizes public health services belongs to the marginalized strata of society and cannot seek paid treatment from physicians or private hospitals. Moreover, the resources available to public health are never sufficient to meet the growing demand. In addition to these financial issues, team training, appropriate work infrastructure, and employee skill and motivation represent important factors in the delivery of primary healthcare services.¹⁰

Thus, using questionnaires is a measure that aids in the understanding of individuals’ experiences regarding healthcare to develop strategic objectives and suggest measures to guide the interdisciplinary work of those engaged in public health.

However, a questionnaire should be duly developed such that it reliably reflects reality. Regardless of the type of study adopted, the philosophical basis is normally founded on the view that reality is constructed by individuals who interact with their social world; therefore, using empirical studies is key to the construction of reality.

To determine whether a questionnaire is reliable, statistical programs can be used, in which the variables studies can determine the research tool’s reliability. Cronbach’s alpha coefficient is an example of a test that relies on a factor to express the degree of responses’ reliability to a particular questionnaire.

Therefore, this study is aimed at demonstrating the reliability of a double scale constructed through a pilot study with CHWs.

Methods

This is a scale-based and validated methodological study. This study was preceded by another study that was qualitative in nature and was developed through interviews with community workers who were requested to respond to the following open question: “Objectively identify four issues involved in your mental health practice.” This study provided a set of proposals that we organized into two subscales with 16 items:

• Subscale 1: The response began with: “this occurs in my mental health practice.” The possible responses were expressed in the Likert format, varying from 1 (never occurs) to 6 (always occurs).

• Subscale 2: The response began with: “I see this as a threat to my mental health practice.” The possible responses were expressed in the Likert format, varying from 1 (never occurs) to 5 (always occurs).

In addition to these two subscales, the questionnaire contained sociodemographic data on gender, age, marital status, time working, and work location from the Family Health Program and CHW teams.

The sample included 97 CHWs undergoing training in mental healthcare in the Federal District’s (Brasilia, Brazil) regional health office. The inclusion criterion was participation in mental health training.

The survey was conducted in December 2012 in a private room at the health centers involved. On February 15, 2012, the Ethics Committee of the Science and Health Care Research Foundation (FEPECS/SES/DF) approved the project as No. 643/11. The 97 CHWs who agreed to participate in the study signed a free and informed consent form, which guaranteed their anonymity and their information’s confidentiality. To guarantee the study’s confidentiality, the names of the family health units that participated in the present study are not reported.

The data was analyzed using SPSS (version 19 for Windows). A p value of 0.05 was considered to be the critical significance level. Cronbach’s alpha was used to determine the questionnaire’s reliability.

Results and Discussion

The 97 CHWs’ sociodemographic data revealed that the majority were women (78.4%). Those between 30 and 39 years of age were found to constitute 40.2% of the group, and more than half of the workers interviewed were married. The main results are presented below and are organized by the previously defined objectives.

Bioethical issues regarding mental health workers’ practices - Validity and Reliability

The survey was analyzed using Cronbach’s alpha test (Table 1). On analyzing the effect of items’ correlation with the entire scale of the survey on the alpha coefficient, high correlations were obtained between almost all items and the entire scale. This finding demonstrates the entire survey’s correct functioning and contributes to the high alpha value (0.76 overall). According to the test, values of this magnitude support the conclusion that the questionnaire is within the expected reliability factor.

Table 1
Questions grouped into five factors by percentage of variance as confirmed by the scree plot

Factor	% Variance	Questions
1	24,425	4, 8, 11
2	14,866	6, 10, 12, 13
3	9,222	1, 5
4	8,162	14, 15
5	7,661	7, 9

Similar results were found in a study that sought to validate the self-confidence of the Portuguese version of the scale. High Cronbach’s alpha coefficients were found overall (0.92), and they were >0.83 for the different dimensions, reflecting the questionnaire’s reliability.

In a study by Aguiar et al.¹¹ seeking to determine the test-retest reliability of the Swedish “Demand-Control-Support Questionnaire” scale’s Portuguese version among a population of workers with low education levels, Cronbach’s alpha values were found to be >0.70 for most dimensions. These results reflect the questionnaire’s stability and enable its use in studies analyzing the association between work stress and health.¹¹

Factor analysis - Determining the frequency of the occurrence of bioethical issues

A principal components analysis was performed with the 16 items of the questionnaire by employing a varimax rotation with the sample of 97 CHWs. The factor analysis method’s suitability was confirmed using Bartlett’s test of sphericity (chi-square approximation = 457,358; df = 120; p < 0.0001). The test results were significant and indicated the data’s appropriateness for the method. The suitability of Kaiser-Meyer-Olkin (KMO) (with a value of 0.687) confirmed the factor analysis method’s adequacy. A commonality analysis revealed that the percentage of common variance in the data structure ranged from 46% to 75%.

A Cronbach’s alpha of 0.761 indicated that the questions attained a satisfactory degree of internal coherence and that the responses to the questions were reliable.

The initial analysis indicated that five components met the eigenvalue criterion of >1 and explained 64.336% of the variance. This indication was confirmed by the scree plot, which suggested five factors before the change in the curve’s steepness (Figure 1).

Figure 1
Scree plot showing the curve’s steepness before factor 5

The questions shown in table 1 are significant at the value of 0.05 with regard to each factor.

The questions were associated with each factor involved in the perception of frequency of occurrence. They were then grouped and labeled on the basis of the topic discussed in each question.

Factor 1, titled “Training in Mental Health,” included the following questions:

Q4-Is it difficult to work with patients who are dependent on medication?

Q8-Is it difficult to approach patients with mental illness?

Q11-Is diagnosis difficult?

These questions refer to issues involving the requirement for CHWs’ technical training to improve their approach toward patients receiving basic healthcare.

Factor 2, titled “Comprehensiveness of Treatment,” included the following four questions:

Q6-Do patients experience difficulty in getting urgent care or mental health clinics?

Q10-Is it difficult to provide patients with instructions?

Q12-Is it easy to manage a patient suffering from a mental illness?

Q13-Are you able to identify patients’ disorders?

Factor 3, titled “Resource Allocation,” included the following questions:

Q1-Are enough medications available to meet patients’ requirements?

Q5-Are you able to follow up with patient treatment?

Factor 4, titled “Lack of a Psycho-Social Network,” included the following questions:

Q14-Is the f the institution’s lack of support responsible?

Q15-Is there a lack of support for transferring patients?

Finally, factor 5, titled “Lack of a Psycho-Social Network” included the following questions:

Q7-Are you able to find mental health professionals when necessary?

Q9-Is it easy to guarantee the continuity of care?

These questions are correlated because of the psychosocial network’s inadequacies.

Perception of threats involving bioethical questions

A principal components analysis was performed using the questionnaire’s 16 questions by employing a varimax rotation with the sample of 97 CHWs. The factor analysis’s suitability was confirmed by Bartlett’s test of sphericity (chi-square approximation = 576,079, df = 120, p < 0.0001). The suitability of KMO was confirmed; the value was 0.822. A high KMO value and the significance of Bartlett’s test of sphericity indicated the data’s suitability for factor analysis. The communality analysis revealed that the percentage of common variance in the data structure ranged from 40% to 72%.

The value of Cronbach’s alpha was 0.883, a finding that indicates that the questions attained a high degree of internal consistency and reflected the responses’ reliability.

The initial analysis indicated that the four components met the criterion of an eigenvalue >1 and explained 60,651% of the variance, thus reflecting a multifactor structure (four values). However, this finding appears to be inconsistent with the scree plot, which suggests factor 1 before the change in the curve’s steepness (Figure 2).

Figure 2
Scree plot showing the curve’s decreased steepness after factor 1

The questions presented in table 2 are significant for each factor; the significance level was 0.05.

Table 2
Questions grouped into four factors by percentage of variance, as confirmed by the scree plot

Factor	% Variance	Questions
1	36,754	1, 5, 7, 9, 13
2	8,998	2, 6, 8, 10, 11
3	7,715	12, 14, 15, 16
4	7,185	3, 9

Table 2 shows that there was a large difference in the percentages of variance for the four factors. The first factor had a variance of 36.75%, which was very different from the others. This difference may explain the divergence between the criterion of an eigenvalue of >1, the four factors, and the scree plot, which suggests a single-factor solution. The estimated Cronbach’s alpha coefficients strengthen the single-factor solution because the coefficient for all the items is 0.883.

The factors’ nature was considered.

Factor 1, titled “Lack of a Psychosocial Network,” included the following questions:

Q1-Are medications available to meet patients’ requirements?

Q5-Am I able to follow up with patient treatment?

Q7-Can I find mental health professionals when necessary?

Q9-Is it easy to guarantee continuity of care?

Q13-Is it easy to identify patients’ disorders?

Factor 2, titled “Training in Mental Health,” includes the following questions:

Q2-Is there a lack of space in the locations indicated for patient care?

Q6-Do patients experience difficulty in getting urgent care or mental health clinics?

Q8-Is it difficult to approach a patient with a mental illness?

Q10-Is it difficult to provide patients with instructions?

Q11-Is it difficult to diagnose patients?

Factor 3, titled “Treatment Integrity,” included the following questions:

Q12-Is it easy to deal with a patient who suffers from a mental illness?

Q14-Is there a lack of support from the institution responsible?

Q15-Is there a lack of support for transferring patients?

Q16-Does the patient receive care in the private network and return to the public system to renew a prescription?

Factor 4, titled “Lack of a Psychosocial Network,” included the following questions:

Q3-Do patient families offer good support?

Q9-Is it easy to guarantee the continuity of care?

Cronbach’s alpha was 0.76. This value meets the standards established by Streiner (2003),¹² who suggests that the coefficient’s values must be >0.7 to be reliable.

These measures foster ethics and the protection of citizens. Such principles may be observed in the conditions proposed by Schramm and Kottow:¹³

“To consider protection each time, certain health objectives are publically accepted to be inevitable because they are indispensable; accepting public health programs involves the certainty, or the high probability, that the measures proposed are necessary and reasonably sufficient for prevention. Once accepted as pertinent, the principle of protection must be completely implemented without being disregarded for ulterior motives because a social need exists to exercise protection through predetermined measures; in other words, possible negative effects do not invalidate the program and health issues in question.”

Conclusion

The responses’ internal consistency obtained using a questionnaire among CHWs in the cities of Riacho Fundo I and II in Brasília revealed that the questionnaire presented reliability within the context in which it was applied. Thus, determining the obtained results’ reliability provides more relevance and robustness to this study regarding CHWs’ mental health labor practices. The study considered their perceptions and threats to their work and detected bioethical issues, particularly in terms of the principle of justice.

Therefore, the questionnaire was confirmed to be a relevant tool for future research to improve the association between basic healthcare and mental health. Identifying issues associated with CHWs, particularly those trained in mental healthcare, facilitates improvements in governmental measures. These improvements may particularly include the development of public policies for this group of patients. There is a requirement for providing better service. These changes may aid both service providers and patients treated using health programs in Brazil.

Acknowledgements

This study was performed at the PhD Program on Bioethics of the Faculty of Medicine of the University of Porto (Portugal)/Federal Council of Medicine (Brazil).

Referências

• ¹
  Ribeiro PRM. Saúde mental: dimensão histórica e campos de ação. São Paulo: EPU. 1996. p.18.
• ²
  Brasil. Ministério da Saúde. Definição da responsabilidade pela gestão da atenção primária nos sistemas municipais de saúde. Diario Oficial da União. Brasília (DF); Ministério da Saúde. 2004.
• ³
  World Health Organization. The World Health Report 2001: Mental health: new understanding, new hope. Geneva: World Health Organization. 2001.
• ⁴
  Medeiros EN. A prevalência de distúrbios mentais e perfil sócio-econômico dos usuários atendidos pelos serviços de saúde dos municípios do estado da Paraíba [tese]. João Pessoa: Universidade Federal da Paraíba. 2005.
• ⁵
  Número de equipes que atuam na Atenção Básica de Saúde aumenta em 19 estados. Ministério da Saúde. Available from: [Updated: 2014 29 July cited 2012 July 2012] http://www.brasil.gov.br/saude/2012/07/cresce-o-numero-de-equipes-que-atuam-na-atencao-basica-de-saude
  » http://www.brasil.gov.br/saude/2012/07/cresce-o-numero-de-equipes-que-atuam-na-atencao-basica-de-saude
• ⁶
  Kluthcovsky AC, Takayanagui AM, Santos CB, Kluthcovsky FA. Evaluation of overall quality of life of community health agents: the relative contribution of sociodemographic variables and domains of quality of life. Rev Psiquiatr RS. 2007; 29(2):176-183.
• ⁷
  Pedrosa JI, Teles JB. Consensus and differences in family health program teams. Rev Saúde Pública. 2001; 35(3):303-11.
• ⁸
  Nunes MO, Trad LB, Almeida BA, Homem CR, Melo MC. The community health worker: construction of the identity of this hybrid and polyphonic individual. Cad Saúde Pública. 2001; 18(6):1639-46.
• ⁹
  Zoboli EL, Fortes PA. Bioethics and primary care: a profile of ethical problems experienced by nurses and physicians of the Family Health Program. Cad Saúde Pública. 2004; 20(6):1690-99.
• ¹⁰
  Tajra SF. Strategic management in health: Thoughts and practices for an excellence-centered administration. São Paulo: Iátria. 2007.
• ¹¹
  Aguiar OB, Fonseca MJ, Valente JG. Reliability (test-retest) of the Swedish Questionnaire Demand-Control scale among industrial restaurant employees in the State of Rio de Janeiro. Rev Bras Epidemiol. 2010; 13(2): 212-22.
• ¹²
  Streiner DL. Being inconsistent about consistency: When coefficient alpha does and doesn´t matter. J Pers Assess. 2003; 80(3):217-22.
• ¹³
  Schramm FR, Kottow M. [Bioethical principles in public health: limitations and proposals]. Cad Saude Publica. 2001; 17(4):949-56. Portuguese.

Publication Dates

• Publication in this collection
  Nov-Dec 2016

History

• Received
  6 Dec 2016
• Accepted
  12 Dec 2016