Evaluation Indicators for the Psychosocial Care Centers Type III: results of a participatory design

ABSTRACT

Psychosocial Care Centers III (CAPs) are considered strategic in the reorientation of the care model in mental health. However, they still lack systematic evaluation mechanisms. This study presents a set of indicators developed in a participatory process for the Psychosocial Care Centers III of the state of São Paulo. Sixteen indicators, grouped into 8 themes were developed: Attention to crisis situations; Qualification of group meetings; Networking; Management of Psychosocial Care Centers; Continuing education; Individualization of care; Care for people with intellectual disabilities; and Use of medication. The indicators were tested in services and are presented as a potentially useful tool to support the assessment, monitoring and management of Psychosocial Care Centers III.

KEYWORDS:
Health evaluation; Mental health; Mental health services

RESUMO

Os Centros de Atenção Psicossocial III são considerados estratégicos na reorientação do modelo assistencial em saúde mental, contudo, ainda carecem de mecanismos de avaliação sistemáticos. O presente estudo apresenta um conjunto de indicadores desenvolvidos em processo participativo para os Centros de Atenção Psicossocial III do estado de São Paulo. Foram elaborados 16 indicadores, agrupados em 8 temas: Atenção à situação de crise; Qualificação dos atendimentos grupais; Trabalho em rede; Gestão dos Centros de Atenção Psicossocial; Educação permanente; Singularização da atenção; Atenção às pessoas com deficiência intelectual; e Uso de medicação. Os indicadores foram testados nos serviços e constituíram um conjunto potencialmente útil para subsidiar a avaliação, o monitoramento e a gestão dos Centros de Atenção Psicossocial III.

PALAVRAS-CHAVE:
Avaliação em saúde; Saúde mental; Serviços de saúde mental

Introduction

Despite their responsibility in almost all diagnostic and therapeutic procedures, as well as in individual care, health services are insufficiently considered in studies. Clinical and epidemiological research rarely takes into account the importance of services on risk factors for population morbidity and mortality or its effectiveness in patient care. Research on health services aims to overcome this insufficient systematic approach in services evaluation, and achieve a continuous qualification of health care. Evaluation procedures are one of the modalities of this type of research (WALKER, 2014).

The Psychosocial Care Centers (Caps, Centros de Atenção Psicossocial) stand out among the services that resulted from the psychiatric reform in Brazil. Considered as controllers of the mental healthcare network and, more recently, important components of the psychosocial care network, the Caps have shown high effectiveness in the care of people with severe psychiatric disorders (SALLES; BARROS, 2013). A study that addresses the current normative model of mental healthcare in Brazil presents the Caps as the main resource for policy orientation of the Psychosocial Attention Networks (RAPs) of the Ministry of Health (TRAPÉ, 2015).

The Caps originated in the second half of the 1980s as pioneer services in the city of São Paulo (SP) and, soon afterwards, became an optional network to the psychiatric hospital in the city of Santos (SP). This was an inflection point in care practices to people with severe mental disorders within the Unified Health System (SUS, Sistema Único de Saúde). The work of Pitta et al. (1995) and Libério (1999) on quality of care in the Caps, and Bandeira et al. (2000) on work overload of employees in these services are the first initiatives to systematically evaluate the new mental healthcare model in Brazil, since the 1990s.

A review study (DANTAS; ODA, 2014) shows a significant increase in scientific production about CSPs from the 2000s. For the authors, the number of what they consider qualitative research and participatory methodologies has hindered the establishment of mental health assessment indicators. However, in the last decades, different instruments than those focused by the authors have been offered by the Pan American Health Organization/World Health Organization (PAHO / WHO) for specific types of Caps and for mental health policies and systems in the country. Although aimed at encouraging the use of indicators in mental health, these tools were unsuccessful in promoting their effective appropriation and use.

With regard to SUS management, the SUS Performance Index (IDSUS) and the National Program for Access and Quality Improvement in Primary Care (PMAQ-AB) stand out as national initiatives for the evaluation of healthcare. In both cases, a series of indicators are applied to a service or a service network, in order to evaluate and induce practices. However, the IDSUS has no indicator related to mental health, and the PMAQ-AB, has four indicators of mental health practices, with exclusive focus on the abusive use of psychoactive substances. It is quite a narrow scope, given the complexity of mental health practices and the organization of services.

The WHO and its regional offices systematically release technical notes, reports and analyzes to support the implementation of common objectives and standards for sectoral policies (WHO, 2013), seeking to establish a consensus and foster the development of mental health indicators. The guidelines for the quality evaluation of mental healthcare and for the health system evaluation tool propose a set of indicators organized in different areas, such as mental health policies and programs, primary health care, human resources, user's rights, psychosocial rehabilitation and intersectoral actions, among several others. These instruments have an explicit intention of inducing good practices and, at the same time, promoting transparency and accountability of mental health initiatives. However, the distance between developers and potential users may lead to the abandonment of proposals or to their decontextualized use, or moreover, to the 'technification' of proposals, which implies ignoring origins, the development process and the intended use of a certain instrument, as stated by Ardila and Stolkiner (2009).

The purpose of this article is to present and discuss a set of 16 indicators directed to the monitoring, evaluation and potential qualification of the Caps III. The indicators were developed from the collaboration between evaluators from two universities and 58 Caps III employees and managers, in the State of São Paulo. Services were established as the central focus of the evaluation, favoring an effective exchange of knowledge between academics and workers, taking into account the different political and institutional contexts in which the Caps were inserted. The services' crucial issues were defined, seeking to overcome limitations, such as the distance between developers and users, and the risks of the technification of the evaluation instruments.

Methodology

The importance of including non-specialists in evaluation processes has been emphasized and implemented in the literature (BARON; MONNIER, 2003). The inclusion of people from the evaluated service in parts or during the whole evaluation process occurs either for pragmatic or for ideological reasons, aiming at increasing the usefulness of the results, or to enhance the effects of the evaluation process itself, referred to in the literature as 'process use' (PRESKILL; ZUCKERMAN; MATTHEWS, 2013). In this study, workers and managers of Caps III were included in the process of elaboration and definition of indicators, to effectively consider the issues that permeate workers' daily lives and to subsidize the incorporation and subsequent use of the indicators by the teams involved.

As a strategy to mediate participation and enable the elaboration of the indicators, a 120-hour course on mental health assessment was proposed, taught regularly over 11 months. The themes were selected based on the results of a previous study conducted by the same researchers that formulated and defined a good practices guide for the organization of the Caps III care: 1) Health assessment; 2) Attention to crisis; 3) Working with groups; 4) Territory and mental health; 5) Management in mental health; 6) Individual therapy project; 7) Intellectual deficiency; 8) Psychiatric medication; 9) Therapeutic home services; 10) Psychosocial rehabilitation; and 11) Continuing education. As a starting point, the groups were offered a set of indicators previously developed for the Caps III network of another city of São Paulo (FURTADO; ONOCKO-CAMPOS, 2008).

The definition of indicators based on issues from the Caps III was due to the strategic importance of these services as a consolidation of the psychiatric reform and the implementation of a care network, replacing the asylum model. Open 24 hours a day, 7 days a week, the Caps III serve 150 thousands inhabitants or more, attending to the continuous care of adults with severe mental disorders, including patients in crisis, and it covers issues inherent to the other types of Caps (I and II).

Two openings to attend the course were made available per service, one for a manager and one for an employee. The two persons were selected internally by the units, based on pre-defined criteria: university education (higher level), willingness and disposition to fully participate in the project, and more than six months of employment in the service. Five units requested extra spots, due to the interest of teams in participating, which were approved by the course coordinator, and 3 spots were made available to managers of the Health Department of São Paulo. The 58 managers and employees from 25 Caps III (out of 26, since 1 of the Caps enrolled but abandoned after the first meeting, alleging schedule difficulties) from São Paulo at the time, were divided into 2 groups, according to the proximity of their home town to the 2 participating universities.

After morning conversational discussions conducted by guests of recognized knowledge on one of the themes listed above, Shared Appreciation Groups (GAP) meeting were held, which were supported by masters and doctoral students of the universities' graduate programs. Each GAP included approximately ten attendees of the course, and was accompanied by the same supporter throughout the year. The groups' objective was to deepen the discussion on the topics covered in the morning and to outline indicators related to those same themes.

The GAPs arose from the awareness that a program or service should not be considered as the product of written documents or of what was advocated by those who conceive or manage them; other sources should be considered, including the informal communication. Jalbert et al. (1997) proposed and experimented with the GAPs (from the French language: Groupes d'Appréciation Partagée) in order to consolidate the feeling of participation, ensuring the manifestation of the greatest possible number of members around the reflection and evaluation of the action.

Through the GAPs, the aim is to involve different actors of the program or service (workers, volunteers, users etc.) in sharing their analysis of their performance in the service, thus contributing to the construction and improvement of this collective action. It seems to us it is an original way to encourage a reflection of the services and their self-assessment. (ZÚÑIGA, LULY, 2005, PAGE 11, NON-CERTIFIED TRANSLATION).

The preliminary definition of indicators was based on the convergence between the problems perceived by the agents in their work routine, the reflection induced by the thematic discussion within the GAP and the subsequent confrontation between the proposed indicators and the considerations of the original teams of the same agents, creating a feedback of the process. In addition, two large meetings were held in the universities with the 58 participants in the first and last months of the course, for the establishment of guidelines and, in the final meeting, for the alignment of proposed indicators previously designed and tested within the subgroups.

This research was submitted and approved by the Research Ethics Committee of the Faculty of Medical Sciences of the State University of Campinas (Unicamp), under the file 410/2011, and funded by the Foundation for Research Support of the State of São Paulo (Fapesp), process 2009/53130-3, and the International Development Research Center (IDRC), Canada, file 02 P-18737-2009.

Results

The indicators were grouped into eight themes: Attention to a crisis situation; Qualification of group meetings; Networking; Caps management; Continuing education; Individualization of care; Attention to people with intellectual disabilities; and Use of medication. The classes, the GAP and the proposal of indicators along the course were organized according to these thematic areas, identified as central in previous research (ONOCKO-CAMPOS et al., 2009). During the GAP, dozens of indicators were suggested, debated, and analyzed. At the end of the process, each of these themes was considered with one to three indicators. Each indicator had its central components dismantled and detailed, such as: the name of the indicator; its definition (the addressed problem); interpretation (the aspect it evaluates); data source (where to obtain the necessary data); measurement period (time interval between one measurement and the next); calculation method (numerator and denominator of the indicator); and additional observations for specific indicators.

Chart 1
Indicators for the Psicosocial Care Centers type III

Themes	Name of the Indicador	Definition	Interpretation	Data source	Frequency	Calculation Method	Observations
ATTENTION TO CRISES SITUATIONS	Assistance to crisis situations	N ° of cases of patients in crisis sent to other services	Capability of the Caps to be decisive in crisis situations	Book of duty, medical record.	Monthly	Number of referred patients in crisis situation / Total number of patients in crisis situation	Consider patients referred to emergency services, general hospitals and university hospitals as a result of crisis episodes.
	Attention to the family of patient in crisis	Therapeutic sessions offered to the family of the patient in a crisis situation	Capacity of the Caps to expand the clinic and care for the affective and familiar context of the users	Book of duty, medical record.	Monthly	Number of families of patients in crisis situations attended / Total n. of patients in crisis situation	Consider individual, group and shared care to the person responsible for group of the extended family of the user in crisis.
EVALUATION OF GROUP MEETINGS	Family group particiapation	Ratio between families of users participating in groups addressed to them and total number of patients of the service	Extension of the care beyond the patient	Group presence sheets	Quarterly	Number of families participating in the groups / Number of active patients	Consider group care with the presence of any member of the extended family group of the user of the service.
	Evaluation of group meetings	Analysis and discussion of groups by the team	Commitment of the team to monitor and qualify continuously its group practices	Book of minutes, technicians responsible for the group	Quarterly	N ° of general meetings in which discussions were held on the groups / N ° of general meetings of the unit
	Shared Single Therapeutic Project (PTS)	No. of PTSs developed in conjunction with other services and sectors	Capability of the Caps to perform collaborative work with other services and sectors	Caps reference technicians; Annotations of medical records; PTS	Semester	N° shared PTS / total n° of PTS	Consider PTS a project discussed by the reference team, with therapeutic offers oriented by the user's needs and its particularity with the participation of other institutions of the territory.
CAPS MANAGEMENT	Manager participation	Effective participation of the manager in formal management (councils, team meetings, assemblies, collegiate, supervisions)	Capacity of the service manager to participate in the formal discussion and deliberation spaces	Book of minutes, Manager schedule	Monthly	Number of spaces in which the service manager participates / Total spaces defined
	University level human resources	Proportion of number of hours of university level professionals for 100,000 inhabitants.	Investment in the Caps structure	Human Resources Worksheet	Semester	Number of hours of university professionals / 100,000 inhabitants
CONTINUING EDUCATION IN HEALTH	Investment in Continuing Education actions (CE)	Formal hours / month for CE in external activities of interest	Unit investment in the CE of its professionals	Attendance list and certificate	Anual	Number of working hours used for CE / Total workload in hours	Consider CE, a set of formal and non-formal education actions related to the work object of the Caps.
	Offer of clinical-institutional supervision	Average monthly hours of clinical-institutional supervision for the team	Provision of space for analysis and reflection of clinical-institutional practices by the team	Supervisor frequency sheet, service schedule	Quarterly	Total number of supervision hours in the trimester / 3
SINGULARIZATION OF ATTENTION	Design of Single Therapy Projects (STP)	Proportion of users who have STP in relation to registered users	Shows the ability of Caps to formulate a tailored care for its users	STP records and form	Quarterly	N° of STP / N° of active users	Consider as STP, a project discussed by the reference team, with therapeutic offers based on the user's needs and particularities.
	Systematic review of STP in the team	STP team discussion, in a given period, in relation to the total number of STPs of the same period	Indicates the team involvement in the process of continuous adaptation of the service to the evolution presented by the users	Meeting log books, Book of minutes and Records	Quarterly	Number of STP discussed in teams / Total number of users with STP
	Number of cases per university professional reference	Identifies the number of patients that each professional or mini-team cares for	Allows to verify the adequacy between number of patients followed and professionals	Records and RAAS (Register of Health Ambulatory Actions)	Quarterly	Number of Caps users / Number of reference professionals
INDICATOR OF USE OF MEDICATION IN CAPS	User adherence to medication	Medication not withdrawn and returned to the pharmacy or nursing station	User adherence to the prescribed medication	Pharmacy dispensing control and nursing station	Monthly	Amount of medication not withdrawn in the month / Amount of medication prescribed in the month
ATTENTION TO PEOPLE WITH DISABILITIES	Insertion of the user with Intellectual Disability (ID) in the Caps	Insertion of the user with ID in Caps	ID User accessibility to Caps	Screening, census, RAAS, medical records	Semester	Number of ID users sent to Caps / Total number of ID users inserted in Caps	Consider ID as primary diagnosis or comorbidity.
	Shared STP of ID users	Co-responsibility of service for ID users	STP collective construction of ID users inserted in Caps	Records, meeting records of team and/or mini-team (reference team)	Semester	Number of ID users who have STP shared with institutions that serve people with ID / Number of ID users inserted in Caps
	Insertion of people with ID in Home Therapeutic Services (HTS)	Proportion of Residents of HTS with ID	Specificity of work in HTS from the presence of residents with ID	Home census, Caps census, Areas of Protection of the Cultural Environment (Apac)	Annual	Number of ID users residing in annual HTS / Number of users residing in annual HTS	Not all units have connection to HTS.

Attention to a crisis situation

The crisis situation was one of the most debated topics by the participants in the GAP and showed different conceptions on the subject. The substitutive role of the Caps in general and of the Caps III in particular invariably navigates through the ability to cope with crisis situations of its users. For this reason, the number 1 indicator (crisis care) evaluates the service's ability to respond locally to acute conditions and their possible extreme variations, avoiding referrals to hospitals or psychiatric emergency rooms. The number 2 indicator (Attention to the family of patients in crisis) evaluates the ability of the Caps to operate on aspects beyond the case itself, considering the demands and needs of the family members, virtually sharing the attention.

Qualification of group meetings

The need to incorporate practices that go beyond individual sessions is a paradigm that supports the psychosocial model and the reorientation of the clinic for substitutive services. However, precisely because they are a consensus, group meetings are implemented without a systematic follow-up that would provide reformulations and their continuous qualification. The indicators defined here demonstrate the workers' concern to subject group sessions to discussion and possible revision by the teams, and also to expand their offer to the users' affective surroundings, i.e. to the family members.

Networking

The interrelation of institutions and professionals is the cornerstone of the mental health care model reorientation, and networking was envisioned as a need by almost all the members of the indicators' elaboration process. The group collectively decided that the relationship between the number of therapeutic projects conducted simultaneously by the Caps and other health services or other sectors with the total number of people receiving care should be use as indicator in order to evaluate the shared work. This indicator seeks to capture interprofessional and intersectoral work capacities, which translate into notion of networking.

Caps management

During the development of the indicators, the inseparability between clinic and management was reaffirmed by the majority of participants. Management organization has a direct impact on teams' orientations, corroborating the need for strategies that integrate management and the clinical practice (ONOCKO CAMPOS et al., 2009). One of the indicators for the management of the Caps seeks to capture the manager's participation in the various spaces of collective management and discussion, in which workers and users can directly participate in the decisions about the unit's direction. The second indicator seeks to demonstrate the effectiveness of the manager in providing the necessary structure for the operation of the service. The number of working hours of university professionals per 100 thousand inhabitants in the assigned area was selected as the parameter that would reflect the level of structuring of the Caps.

Continuing education

The clinical-institutional supervision, while enquiring the technical interventions (clinical practices) of each practitioner and team, should also address how the care model is understood and conducted in order to act on the health-disease process. Thus, supervisions have both a clinical and a political dimension, as they question aspects of both the work process and the institutional relations. The convergence of these two analyses derived from the employees' work is the basis for considering clinical-institutional supervision as an integral element of continuing education, together with the investment in institutional training outside the Caps. These were two elements (clinical-institutional supervision and external training) defined as indicators for the continuing training of workers.

Individualization of attention

The Individual Therapeutic Project was considered a decisive strategy for conducting the work according to the course of each user. For the evaluation of this strategy, three complementary indicators were elaborated. First, the number of patients that have a therapeutic plan is obtained; then, the frequency with which this plan is reviewed and adapted, considering eventual changes of the user and its context; and finally, the third indicator defines the number of people in charge of each technician or reference team, a ratio with strong impact in the effective individualization of attention.

Attention to people with intellectual disabilities

The borderline between intellectual disability and mental health is tenuous (SURJUS, ONOCKO-CAMPOS, 2014), especially in patients who were submitted to long psychiatric hospitalizations. The way in which the demands and needs related to intellectual disability are considered by the Caps were the focus of the first two indicators of this theme. The indicators investigate the insertion of this clientele in the service and the elaboration of a specific therapeutic plan for them. A third indicator is related to the insertion of people with intellectual disabilities into the Home Therapeutic Services.

Use of medication

The debate about the potentials and limitations of the use of psychiatric medication in mental health is controversial (ONOCKO-CAMPOS et al., 2013). Only one indicator was defined for the topic, addressing the interruption or maintenance of the prescribed medication, within the services. Patient consent, quality of prescriptions and level of user autonomy in medication intake did not reach the necessary consensus for the elaboration of respective indicators, even though they emphatically permeated the debate in many GAPs and in the plenary meetings.

Discussion

The operationalization of mental health indicators is a challenge. First, the tradition of indicators in this area is limited when compared to other health fields, such as basic care and hospital care, which have been in use for a long time and received support by national and international health agencies, and therefore, established the basis for follow-up criteria. In addition, the strong ethical and political character of the psychiatric reform and the consequent difficulty in establishing consensus around some parameters and indicators among agents, also have an impact. Moreover, the object of mental health workers, which is characterized by subjective issues involving the subjects they accompany, requires efforts to relativize and understand the singular matter, generating less permeability for objectification and numerical systematization of their practices around indicators.

During the construction process of these instruments, it was often difficult to objectify and synthesize actions of various natures developed within the Caps III. These difficulties were evidenced, among other characteristics, by the lack of parameters on conceptual aspects fundamental to the services. An illustrative example was the recurrent question regarding a definite conception of a 'crisis'. Extensive debates were required to establish a consensus on conceptions that were initially thought to be well established, given that some were faced daily by the teams. Such inquiries arouse when services' staff were presented with the preliminary version of the indicators. This situation seems to demonstrate the need for the development of a glossary to be used with the set of indicators discussed here.

The course allowed the consideration of the main themes that involve the daily routine, the management and the clinical work of Caps, according to researchers and workers. As part of the process, defined indicators were screened, ensuring 'utility' (meeting the practical information needs), 'feasibility' (requiring readily accessible information) and 'accuracy' (revealing technically appropriate information) characteristics. The collective analyzes of the indicators and their preliminary applications in the services corroborate their satisfactory performance. In addition, the obtained results demonstrate that these instruments are in accordance with the definition of a good evaluation by the American Association of Evaluation (FURTADO; LEPERRIERE, 2012).

The joint input of university researchers and workers from the services, on the one hand, allowed the consideration of issues that would be difficult to perceive from a scientific point of view, and the construction of effective instruments for observation and support of eventual changes. On the other hand, this joined proximity, while broadening the possibility of adoption and use of the indicators by the Caps teams, it simultaneously confers fragility to the indicators caused by the substantial changes in the service context. In addition, it should be emphasized that such an instrument should necessarily be combined with broader policies aimed at continuous qualification and monitoring of services. The fragility of management at various levels, lack of structure and stability of the SUS evaluative bodies and the inefficiency of information systems create a scenario that is unfavorable to the consolidation and expansion of initiatives such as the one presented in this study (TRAPÉ, 2015).

The participants from the Caps were essentially professionals with university education in health fields, which may explain the predominance of indicators for processes in relation to those for structure and result, or of 'soft' indicators as reported by Saraceno, Frattura and Betolote (1993). Important agents of the process, such as professionals with technical training, users, family members and medical professionals, collaborated only partially and indirectly in periods of deconcentration in the field locally, when of the validation or testing of a certain indicator. Thus, a demand arises for future initiatives that develop evaluative instruments from the perspective of these agents. We identify this as one of the weaknesses of the process registered here.

In spite of this, the approach described provide quality to the participation, as it allowed the appropriation of part of the theoretical reference of evaluation by a group unfamiliar with systematic discussions on the subject. The workers collaborated in the definition of issues, and in the preparation and preliminary application of the indicators, giving depth to their contribution, according to the proposal of Cousins and Chouirnard (2012). For these authors, the extent of participation is defined by the diversity of participants, and the depth by the level of influence in the process of formulating, researching and analyzing the procedures involved in the evaluation. Although we have just taken some steps towards evaluation - defining issues and instruments - and not performed the complete evaluation process, we guarantee effective collaboration in decisive stages of the process.

Conclusion

From an academic point of view, the process undertaken for the elaboration of the indicators combined efforts and effectively linked teaching, research and academic extension. From the point of view of services, the inclusive and formative methodological strategy fostered the critical thinking of workers and managers and the deepening of important and common issues, from which tools for objectifying and monitoring priority problems emerged. This objectification, however, was made within the context in question. After all, indicators are the expression of a concept, and their inseparability must necessarily be understood and considered by agents, overcoming the fetishization or technification of these instruments. The theoretical discussions and debates that preceded the formulation of the indicators proved effective in making the participants aware of the premises behind these indicators.

Effective participation and collaboration require time, appropriate spaces and connections between agents from different fields. The complex task of developing evaluative indicators in a health subarea without tradition in evaluation processes was made possible through the establishment of an institutional space, represented by the course, and a long period of time, represented by the several meetings over a year. This strategy allowed time breaks for the evolution of the debate and the establishment of trust bonds created during the GAP meetings that took place in singular spaces with the support of a single professional at all times.

The effective use of the indicators can contribute to the development of the evaluation culture - evaluation of the Caps and of the instruments themselves - as they can be confronted with diverse realities outside the state of São Paulo and with unforeseen situations. We present these indicators to the scientific community, without the pretension of them being unique or perfect. Instead, we expect to be contributing to the opening of a communication channel and a necessary construction in the current situation of the Brazilian psychiatric reform. Having developed and tested a set of evaluative indicators through an inclusive process of workers and managers represents the strength of our research.

The permeability of municipal managers, regional colleges, states and federation is always critical for the continuous improvement of the services studied herein, and obviously, for the support and expansion of the presented instruments.

• Financial support: Fundação de Amparo à Pesquisa do Estado de São Paulo (Fapesp), Process: 2009/53130-3 and International Development Research Centre (IDRC), Canada, Process: 02 P-18737-2009

Referências

• ARDILA, S.; STOLKINER, A. Estrategias de evaluación de programas y servicios de atención comunitaria en salud mental: consideraciones metodológicas. In: CONGRESSO INTERNACIONAL DEL INVESTIGACIÓN Y PRÁTICA PROFESIONAL EN PSICOLOGIA DEL MERCOSUR, 1., 2009, Buenos Aires. Anais... Buenos Aires: Acta Academica, 2009. p. 252-257. Disponível em: <http://www.aacademica.org/000-020/210>. Acesso em: 24 out. 2016.
  » http://www.aacademica.org/000-020/210
• BARON, G.; MONNIER, E. Une approche pluraliste et participative. Informations sociales, Paris, n. 110, p. 120-129, 2003.
• BANDEIRA, M.; PITTA, A. M. F.; MERCIER, C. Escalas brasileiras de avaliação de satisfação (SATIS-BR) e da sobrecarga (IMPACTO-BR) da equipe técnica em serviços de saúde mental. J. Bras. Psiquiatri, Rio de Janeiro, v. 49, n. 4, p. 105-115, abr. 2000.
• COUSINS, J. B.; CHOUIRNARD, J. A. Participatory evaluation up close: an integration of research-based knowledge Charlotte: Age Publishing, 2012.
• DANTAS, C. R.; ODA, A. M. G. R. Cartografia das pesquisas avaliativas de serviços de saúde mental no Brasil (2004-2013). Physis, Rio de Janeiro, v. 24, n. 4, p. 1127-1179, dez. 2014.
• FURTADO J. P.; LEPERRIERE, H. Parâmetros e paradigmas em meta-avaliação: uma revisão exploratória e reflexiva. Ciênci. Saúde Coletiva, Rio de Janeiro, v. 17, n. 3, p. 695-705, mar. 2012.
• FURTADO, J. P.; ONOCKO-CAMPOS, R. Participação, produção de conhecimento e pesquisa avaliativa: a inserção de diferentes atores em uma investigação em saúde mental. Cad. Saúde Pública, Rio de Janeiro, v. 24, n. 11, p. 2671-2680, nov. 2008.
• JALBERT, Y. et al. Epsilon: guide d'auto-évaluation des organismes communautaires. Montréal: COCQ-Sida, 1997.
• LIBÉRIO, M. M. Estudo da satisfação com o CAPS da cidade do Rio de Janeiro: ouvindo seus atores principais. 1999. Dissertação (Mestrado em Psiquiatria) - Instituto de Psiquiatria, Universidade Federal do Rio de Janeiro, Rio de Janeiro, 1999.
• WORLD HEALTH ORGANIZATION (WHO). The European Mental Health Action Plan Copenhagen: WHO, 2013.
• ONOCKO-CAMPOS, R. et al Avaliação da rede de centros de atenção psicossocial: entre a saúde coletiva e a saúde mental. Rev. Saúde Pública, São Paulo, v. 43, p. 16-22. 2009.
• ______. Gestão Autônoma da Medicação: uma intervenção analisadora de serviços em saúde mental. Ciênc. Saúde Coletiva, Rio de Janeiro, v. 18, n. 10, p. 2889-2898, out. 2013.
• PITTA, A. M. F. et al Determinantes da qualidade de serviços de saúde mental em municípios brasileiros: estudo da satisfação com os resultados das atividades desenvolvidas por pacientes, familiares e trabalhadores dos serviços. J. Bras. Psiquiatr, Rio de Janeiro, v. 44, n. 9, p. 441-452, 1995.
• PRESKILL, H.; ZUCKERMAN, B.; MATTHEWS, B. An exploratory study of process use: findings and implications for future research. American Journal of Evaluation, New Mexico, v. 24, n. 4, p. 423-442, 2013.
• SALLES, M. M.; BARROS, S. Transformações na atenção em saúde mental e na vida cotidiana de usuários: do hospital psiquiátrico ao Centro de Atenção Psicossocial. Saúde em Debate, Rio de Janeiro, v. 37, n. 97, p. 324-335, jun. 2013.
• SARACENO, B.; FRATTURA, L.; BERTOLOTE, J. M. Evaluation of psychiatric services: hard and soft indicators. Geneva: WHO, 1993.
• SURJUS, L. T. S.; ONOCKO-CAMPOS, R. Interface entre Deficiência Intelectual e Saúde Mental: revisão hermenêutica. Revista Saúde Pública, São Paulo, v. 48, n. 3, p. 532-540, jun. 2014.
• TRAPÉ, T. L. Redes de Atenção à Saúde Mental: Estudo Comparado Brasil-Catalunha. 120 f. 2015. Tese (Doutorado em Saúde Coletiva) - Faculdade de Ciências Médicas, Universidade Estadual de Campinas. Campinas, 2015.
• WALKER, D. M. An introduction to health services research Newbury Park: Sage, 2014.
• ZÚÑIGA, R.; LULY, M. H.; Savoir-faire et savoir-dire. Un guide d'évaluation communautaire. Montréal: COCQ-sida, 2005.

Publication Dates

• Publication in this collection
  Mar 2017

History

• Received
  Mar 2016
• Accepted
  Sept 2016