CO-PRODUCTION AND INNOVATION IN PUBLIC SERVICES’ ASSESSMENT: THE CASE OF THE CITIZEN AUDIT PROJECT

COPRODUÇÃO E INOVAÇÃO NA AVALIAÇÃO DE SERVIÇOS PÚBLICOS: O CASO DO PROJETO AUDITORIA CIDADÃ

ABSTRACT

Purpose:

This paper analyzes the Citizen Audit Project (PAC), in the light of public sector innovation theory, in order to identify challenges for introducing co-production in the evaluation process of public services by the internal audit of the Executive Power of the State of Santa Catarina, Brazil.

Design/methodology/approach:

We carried out the study as a qualitative research, and collected data through a documentary search and interviews with 21 key agents of public control or with those who interacted with PAC in the chosen operations. We used content analysis to examine the collected data.

Findings:

We analyze the main challenges inherent to co-production in the activity of public services’ assessment by internal audit. Results show an alignment between the identified dares and the theoretical basis of public innovation. These were barriers related to bureaucratic dysfunctions, openness of political agents and audit professionals to receive and consider proposals from society, time and resources needed, leadership style, citizens’ lack of motivation, and the evaluation model.

Originality/value/implications:

The paper contributes to the literature on public audit, public sector innovation, and co-production, by conducting an original study that identified challenges related to the implementation of co-production in audit process and analyzed them from the perspective of public sector innovation. Evidence brought by this research contributes to fill literature gaps regarding empirical studies on the convergence between these three topics. Regarding practical implications, it provides base for PAC consolidation and for its replication by governments interested in implementing innovations through the process of participatory audit.

Keywords:
internal audit; participatory audit; assessment of public services; co-production; innovation

RESUMO

Objetivo:

Este artigo analisa o Projeto Auditoria Cidadã (PAC), sob a luz da teoria da inovação no setor público, com o fim de identificar desafios para a introdução da coprodução no processo de avaliação de serviços públicos pela auditoria interna da Secretaria de Estado da Fazenda de Santa Catarina.

Desenho/metodologia/abordagem:

O estudo constou de uma pesquisa qualitativa, e os dados foram coletados por meio de pesquisa documental e entrevistas com 21 atores-chave do controle público ou aqueles que interagiram com o PAC nas operações examinadas. Utilizamos a análise de conteúdo para examinar os dados coletados.

Resultados:

O artigo analisa os principais desafios inerentes à coprodução, nas atividades de avaliação de serviços públicos pela auditoria interna. Os resultados demonstram um alinhamento entre os desafios identificados e a base teórica da inovação no setor público. Os principais desafios identificados foram barreiras relacionadas às disfunções da Burocracia, receptividade dos agentes políticos e profissionais da auditoria para receber e levar em consideração propostas da sociedade, tempo e recursos necessários, estilo de liderança, falta de motivação dos cidadãos e modelo de avaliação.

Originalidade/valor/implicações teóricas e práticas:

O artigo contribui para a literatura no campo da auditoria pública, da inovação no setor público e da coprodução, ao conduzir um estudo original que identifica desafios relacionados à implementação da coprodução no processo de auditoria e analisá-los sob a lente teórica da inovação no setor público. As evidências geradas contribuem para preencher lacunas identificadas na literatura relacionadas a pesquisas empíricas com foco na convergência desses três temas. Com relação às implicações práticas, o artigo traz elementos para a consolidação do PAC e sua replicação por governos interessados em implementar inovações por meio de processos de auditoria participativa.

Palavras-chave:
coprodução; inovação; auditoria interna; auditoria participativa; avaliação de serviços públicos

1 INTRODUCTION

It has been widely discussed that public administration should simultaneously provide better services and optimize costs, in order to fulfill its role without increasing tax burden, thus enabling nations’ social and economic development (Cavalcante, 2017; Ansell & Torfing, 2014; Torfing, Ferlie, Jukić & Ongaro, 2021). Public sector innovation is an alternative for dealing with this challenge, as new answers are necessary to address relevant problems (Torfing, 2013; Osborne & Strokosch, 2013; Ansell & Torfing, 2014; Crosby, Hart & Torfing, 2017; Barrutia & Echebarria, 2019; Lopes & Farias, 2020).

By assuming public sector innovation in services, the strategy of co-production plays a key role, since service is, by nature, co-produced (Osborne, 2013). This allows the incorporation of users' knowledge and their specific experience on the conditions and locations where public services are rendered, in order to reshape their design and the way of delivery (Boyle & Harris, 2009; Osborne & Strokosch, 2013). On the other hand, just as innovation requires co-production, this often generates demands for more innovation, since the needs that emerge from interacting with society may require non-existing processes and services from the public sector. In this perspective, the two concepts are interconnected.

To assist in the process of remodeling public services, public administration relies on internal audit, which provides objective and impersonal assessments and consulting for organizations to achieve their goals (Institute of Internal Auditors [IIA], 2016). In turn, for taking into account the contributions from citizen users, the audit process itself needs to innovate, by including interfaces that enable the co-production of evaluation (United Nations [UN], 2007; International Organization of Supreme Audit Institutions [INTOSAI], 2013a; World Bank, 2014; World Bank Group/ Civil Association for Equality and Justice [WBG/ACIJ], 2015).

In this sense, the participatory audit process has grown, based on a collaborative structure between civil society and audit bodies, where citizens and civil society organizations provide useful, qualified, and substantiated information on the quality of services rendered by state agencies (World Bank, 2014; WBG/ACIJ, 2015; Affiliated Network for Social Accountability in East Asia and the Pacific [ANSA-EAP], 2018, Mckenna, 2021). Participatory audit not only seeks to include co-production in the audit process, but also induces it in the assessed service itself.

Therefore, we observe that innovation capacity based on democratic principles implies the incorporation of collaborative practices (Moore & Hartley, 2010; Torfing, 2013; Ansell & Torfing, 2014; Klumb & Hoffmann, 2016; Torfing & Triantafillou, 2016; Crosby et al., 2017; Lopes & Farias, 2020; Torfing, Ferlie, Jukić, & Ongaro, 2021). However, the organization and coordination of collaborative interactions of multiple actors present several challenges and barriers (Torfing, 2013; Kozuch & Sienkiewicz-Malyjurek, 2016; Torfing & Triantafillou, 2016; Cavalcante, Camões, Cunha & Severo, 2017; Barrutia & Echebarria, 2019, Lopes & Farias, 2020). Thus, new frameworks are necessary to accommodate inter-organizational innovation (Moore & Hartley, 2010; Kinder, 2013; Kozuch & Sienkiewicz-Malyjurek, 2016; Crosby et al., 2017; Lopes & Farias, 2020).

This study follows the research agenda of Osborne and Strokosch (2013), who emphasize the importance of carrying out empirical research to test and refine these frameworks, and their contribution to the theory and practice of public administration. Similarly, Barrutia and Echebarria (2019) argue that empirical studies are necessary to deepen the different forms of collaboration in order to promote innovation in the public sector. Other authors discuss managerial and theoretical gaps that make up research topics that can further advance this one, regarding the need to understand barriers and challenges for fostering the implementation of public sector innovations, trough collaborative approaches in general (Crosby et al., 2017; Torfing, Sorensen & Roiseland, 2019; Lopes & Farias, 2020). In addition, many specifically point out challenges related to co-production, as a way to achieve public innovation (Voorberg, Bekkers & Tummers, 2014; Osborne, Radnor & Strokosch, 2016; Cavalcante et al., 2017; Brandsen, Steen, & Verschuere, 2018). However, few studies examine these issues in the field of public services’ assessment and the audit process. Mckenna (2021) observes that there are few examples of assessment co-production applied to audit, and this is an uncharted territory whose barriers are not clearly known. Therefore, we defined the following research question: What are the challenges for introducing co-production into the evaluation process of public services?

We carried out this study within the scope of the Citizen Audit Project (PAC), conceived by the General Audit Board (DIAG), an internal body of the Executive Power of the State of Santa Catarina (Brazil), to evaluate the provision of public services "under the citizen’s view". It is an unprecedented initiative, seen as an effort of democratic experimentation (Ansell, 2012) in the field of public control. By trying to introduce for the first time co-production into the internal audit evaluation service, we have assumed that the project brings challenges associated to public sector innovation. The implementation of innovative projects requires understanding these challenges (Osborne, 2013; Torfing; 2013; Ansell & Torfing, 2014; Torfing et al., 2019; Lopes & Farias, 2020).

Hence, this paper sought to analyze the Citizen Audit Project (PAC) in the light of the theory of public sector innovation, in order to identify challenges for introducing co-production into the evaluation process of public services by the internal audit of the Executive Power of the State of Santa Catarina, Brazil.

We carried out the research in two stages: (1) Characterization of PAC’s origin and its implementation process; and (2) Identification of the challenges for PAC implementation as a participatory audit process, in the perception of its stakeholders.

After this introduction, the next section presents the theoretical background on participatory audit, co-production, and public sector innovation. Section 3 addresses the methodological procedures of the research, section 4 presents and discusses the results, and section 5 shows the final considerations.

2 THEORETICAL BACKGROUND

2.1 Participatory Audit

Participatory audit is the result of audit evolution, in response to the current demands for State’s accountability, transparency and responsiveness. In fact, there are common ideas between the development of audit and the theoretical models that guide Public Administration. Thus, compliance audits are inherent to the conception of the Traditional or Bureaucratic Public Administration (Salm & Menegasso, 2009; Rocha, 2011a, 2011b). Performance audits are based on the philosophy of the New Public Management (Barzelay, 2000a, 2000b; Rocha, 2011a, 2011b); and participatory audit follows the visions of the New Public Service (Denhardt & Denhardt, 2007) and the New Public Governance (Osborne, 2006, 2010)¹.

In a traditional sense, audit is a check that compares usually accepted criteria with transactions actually carried out (Chemlinksy, 1985, apud Barzelay, 1997). This type of check or inferential practice is known as traditional, regularity or compliance audit (Power, 2000; Barzelay, 1997; Rocha, 2007). This way of acting has developed under the postulates of the bureaucratic model (Weber, 1978), from Woodrow Wilson's management theory (Behn, 1998; Denhardt & Denhardt, 2007), and from Frederick Winslow Taylor’s scientific management.

In the bureaucratic model, efficiency would result from rationality applied to an organization structured on hierarchy, with absolute separation of the planning and implementation functions of public policies (Barzelay, 1997). In this model, audit check examined acts, facts, and processes under the "exclusive" light of legal norms, which should be impersonal, general, and abstract (Weber, 1978). Therefore, the use of an "objective" parameter aimed at removing any aspect regarded as auditor’s subjectivism, who acted mainly ex-post, thus allowing a "neutral" evaluation of the object of analysis. Power (2000), however, warned that this artifice fostered a "myth of audit neutrality", because, although objective, the auditor's analysis was affected by elements of the audit body, or by the auditor itself.

However, effects called ‘bureaucratic dysfunctions’ questioned the bureaucratic model of management. These were a greater emphasis on the process rather than on its result, the impossibility of distinguishing between the technical and political action of public agents, lack of flexibility, excessive number of rules, and low efficiency (Merton, 1978; Behn, 1998; Denhardt & Denhardt, 2007; Salman & Menegasso, 2009). These effects were also present in traditional audits (Barzelay, 1997; Rocha, 2011a; Olivieri, 2011).

Criticisms to the bureaucratic doctrine, regarding its alleged inefficiency, and a period of budgetary restrictions (from the 1980s, and especially in the 1990s), led to reforms influenced by the New Public Management model, also known as ‘managerialism’. Its main grounds came from Hood (1995), Hood and Jackson (1991) and Osborne and Gaebler (1992), according to Barzelay (2000a), Denhardt and Denhardt (2007), and Rocha (2011). Barzelay (2000b) considers it a field of political and professional discussion on issues related to public management, which include public management policies, executive leadership, design of programmatic organizations, and government operations.

The New Public Management model emphasized that administration should improve the provision of public services (service-oriented) whose enhancements should meet customers’ needs (customer-driven services). Emphasis on "clients" also replaces the self-referenced centralism of state performance, by guiding government actions according to society demands.

Managers had higher discretion and focused on results; thus, it became necessary to measure their performance, their ability to deliver the expected services. This resulted in the "natural evolution of traditional audit" (Rocha, 2007, p. 12) - from checking the compliance with standards and the appropriate accounting record, for a comprehensive assessment of the organization's performance, with special emphasis on the economy, effectiveness, and efficiency of management (Rocha, 2007; Barzelay, 1997). This is the concept of performance audit or operational audit. Operational audit became an instrument of change in public administration; audit bodies acquired a broader view of public policy, which required interdisciplinary knowledge and a proactive performance from auditors (Barzelay, 1997).

According to Power (1994), as the idea of audit comprises a cluster of values that contemplate "independent validation, efficiency, rationality, visibility regardless of the mechanisms of practice, and ultimately the promise of control" (p. 13), there was a strong demand, which caused an "audit explosion". Especially in the public sector, the idea of audit was present in all areas, whether for public service efficiency, corporate governance, or management systems, so that “all these apparent virtues made audit become the central part of 'government reinvention’" (p. 14), referring to the New Public Management, more precisely to the paradigm shift proposed by Osborne and Gaebler (1992).

However, the New Public Management model also received criticisms for excessively addressing the organization itself, without considering the inter-relationships and inter-dependencies inherent to the development of modern societies, which are plural and complex (United Nations [UN], 2005; Osborne, 2006).

Hence, the New Public Governance model emerged, initially designed by Stephen Osborne (2006), who recognized the current situation of the pluralistic and diversified state that should deal with the complexity and fragmentation of society. Thus, it was necessary to promote the coordination of actors, institutions and interdependent sectors through a collaborative approach, where organizational borders became flexible for improving public services. The focus of state action shifted from intraorganizational to interorganizational, and performance metrics was no longer internal, regarding products or outputs, but focused on the outcomes effectively perceived in society.

Multilateral agencies and Supreme Audit Institutions (SAIs) started to adopt strategies for integrating users into the processes of monitoring and assessing public actions. This aimed to use the real feedback offered by the recipients of these actions, from their specific knowledge on the conditions and locations where public services were rendered (UN, 2007; INTOSAI, 2013a; World Bank, 2014; WBG/ACIJ, 2015). An example was the UN program Auditing for Social Change: A Strategy for Citizen Engagement in Public Sector Accountability, launched between 2005 and 2007; its main intention was to explore how audit could operate in a more open and participatory way, becoming a tool for citizens’ empowerment, for the promotion of transparency and to attain the millennium development goals.

In 2015, the World Bank, in partnership with the ’’Civil Association for Equality and Justice’’ (ACIJ) published the E-Guide ‘Engaging Citizens in the Audit Process’, with the purpose of assisting SAIs leaders and civil servants, along with representatives of the civil society, with tools and information for the establishment of participatory institutional policies and evaluations of their outcomes (WBG/ACIJ, 2015).

Therefore, the processes of participatory audit and social audit were defined; the first is run by institutional audit bodies, with the participation of citizens and civil society organizations, in their different stages, while the second is conducted by the civil society itself (WBG/ACIJ, 2015; World Bank, 2014; Grandvoinnet, Aslam, & Raha, 2015).

Although the participatory audit process involves external agents, given the audit principles of independence and objectivity, the official body has the responsibility for decision-making, which includes evaluating suggestions from interested parties, the audit analysis, and the resulting conclusion (INTOSAI, 2013b).

The inclusion of citizens in the process of participatory audit results in the co-production of the evaluation service; therefore, in order to analyze PAC, it is convenient to address the theoretical basis of this phenomenon.

2.2 Co-production

Osborne and Strokosch (2013) combined the literature on business services management with that on public administration, to create an understanding of the co-production of public services through the integration of these two perspectives. In business services management, they identified the attributes of the inseparability of production and consumption at the time of service delivery, which is the "cornerstone of co-production".

They also observed the intangible nature of services, which are more a user-lived experience than a fact objectively measured, as with material goods. It follows that co-production is inherent to service provision, being, therefore, an intrinsic element in the production of public services (Osborne & Strokosch, 2013). Based on the wide use of the self-service model in the private sector, which is co-production in practice, the authors formulated the ‘consumption co-production’ type, which engages the user at the operational level of the service production process, to balance expectations regarding the service experience, whose purpose is to empower him/her.

From the perspective of public administration, Osborne and Strokosch (2013) found that co-production challenges the traditional and orthodox speech that the design and delivery of services are state agents’ responsibility, and citizens just have to require, consume and evaluate them. By conceiving co-production as something "external" to public services, it should be added to the production process in order to increase their quality, through the adoption of participatory mechanisms at the strategic stage of planning and design. This is the participatory co-production type, whose purpose is user’s participation.

On the other hand, from the integration of business and public management perspectives, Osborne and Strokosch (2013) formulated the enhanced co-production type, which is the combination of the operational and strategic approaches. It challenges the current paradigm of service provision with innovative forms of public services guided by users (Osborne & Strokosch, 2013).

Likewise, the development of PAC as a participatory audit strategy implies the introduction of co-production mechanisms in the planning, execution, and monitoring phases, in order to strengthen the contributions of service evaluation by considering the experiences and knowledge of public policy makers and of the citizens for whom these services are intended. This means changing the role of public servants and citizens in the audit’s context, from "objects" evaluated, or passive interviewees, to active actors in all stages of the evaluation process. This new role includes proposing adjustments that range from framing the scope and design of the participatory evaluation to defining data collection method and analysis.

2.3 Public Sector Innovation

To implement innovation in the public sector, according to the principles of New Public Governance, Ansell and Torfing (2014) emphasize the concepts of collaboration and design. The strategy associated with these concepts is the union of complementary visions and knowledge in an open and applied design, to assist the achievement of new and creative solutions, defined by user needs (Crosby et al., 2017; Brandsen, Steen, & Verschuere, 2018; Cavalcante, 2019; Torfing et al., 2019)

Ansell and Torfing (2014) observe that the link between collaboration and innovation takes place through three mechanisms: synergy, learning, and commitment. Synergy means the combination of complementary resources, skills, and knowledge in a cooperative regime. "Learning is the process by which cognitive changes occur as a result of interaction with other stakeholders, which can change or reshape the collective sense of possibilities or generate new ideas" (ANSELL & TORFING, 2014, p. 11). And commitment is establishing consensus among the participants of a given project, feeling of belonging, and support to the conception and implementation of the intended public innovation. To the authors, without commitment the mechanisms of synergy and learning cannot do much.

Collaboration and design that result from the interaction of these mechanisms must cross the entire innovation cycle, which consists of four stages: idea generation, idea selection, implementation of new ideas, and dissemination of new practices (Torfing, 2013). Each stage of the innovation cycle must be strengthened with the cooperation of relevant actors and the use of methods and techniques associated with the design thinking approach (Alam, 2006, apud Osborne, 2013; Ansell & Torfing, 2014; Torfing, 2013; Trischler & Scott, 2016; Cavalcante, 2019; Dell´Era, Magistretti, Cautela, Verganti, & Zurlo, 2020). The use of design thinking removes the focus from the design of the new object, by providing insights and solutions that go beyond mere adaptations or only change the appearance of current practices (Ansell & Torfing, 2014).

However, the implementation of innovations in the public sector is not always easy and spontaneous; thus, it is convenient to adopt an intentional method directed to the development, adoption, and dissemination of new and creative ideas that defy conventional standards and bring a qualitative change to established practices (Torfing, 2013; Strokosch, 2013; Ansell & Torfing, 2014; Lopes & Farias, 2020).

In addition, since services have an intangible nature and consist of an experience undergone (just) by the user, only he/she is able to make a real evaluation of its quality (Osborne & Strokosch, 2013). Hence, instead of regarding users as passive individuals in the provision of public services, they should have active roles in evaluation and in suggesting ideas, thus inducing a collaboration for value co-creation with the state, through their valuable contribution (Voorberg et al., 2014; Osborne et al., 2016; Brandsen, Steen, & Verschuere, 2018; Cavalcante, 2019). Therefore, establishing partnerships between professionals and citizens is essential to improve public services, including the assessments carried out by the internal audit, as is the case of PAC.

Professionals also play an important role in identifying opportunities for improvement, capturing feedback from users, or detecting tacit knowledge to increase service quality (Kliksberg, 1999; Bovaird, 2007; Osborne & Strokosch, 2013; Brandsen, Steen, & Verschuere, 2018; Schuch & Hoffmann, 2021). This applies equally to service providers outside public administration, such as contractors, civil society organizations that provide services through partnerships, or civil servants, who are often absent from planning the activities they perform (Schommer, Rocha, Spaniol, Dahmer, & Souza, 2015). Thus, innovation that results from expanding co-production in public services is feasible only with changes in the governance structure, to enable an effective collaboration between professionals and users (Bovaird, 2007; Osborne, 2013; Brandsen, Steen, & Verschuere, 2018).

However, the discussion that refers to the challenges related to the implementation of innovations in the public sector points out, on the one hand, to the lack of motivation and training of citizens to participate or co-produce and, on the other hand, to the lack of openness and training of professionals for this new approach (Crosby et al., 2017; Struecker & Hoffmann, 2017; Brandsen, Steen, & Verschuere, 2018; Lopes & Farias, 2020). In this sense, Lopes & Farias (2020) argue about the need for incentive systems that motivate collaboration. Brandsen, Steen, & Verschuere (2018) highlight the importance of taylored incentives for citizen and taylored training for professionals.

Along with the importance of professionals, the need for leaders who act as agents promoting collaborative processes (Barrutia & Echebarria, 2019 ; Schuch & Hoffmann, 2021) and co-production (Brandsen, Steen, & Verschuere, 2018) is highlighted. Crosby et al. (2017) consider the leader as an orchestrator of collaboration, and argue that creating public value through collaborative innovation requires a special kind of leadership. In this meaning, for him, leaders must act as sponsors, champions, catalysts and implementers. They play an important role in creating effective methods that allow dialogue and deliberation in multi-stakeholder forums, in order to create shared understanding of public problems, to foster constructive conflict and collective creativity (Crosby et al. (2017), promoting trust (Kozuch & Sienkiewicz-Malyjurek, 2016; Brandsen, Steen, & Verschuere, 2018; Barrutia & Echebarria, 2019; Lopes & Farias, 2020) and to access external sources of knowledge and new sources of resources (Moore & Hartley, 2010; Barrutia & Echebarria, 2019).

Resources involve not just people and capital, but also time for collaborative processes (Kozuch & Sienkiewicz-Malyjurek, 2016; Struecker & Hoffmann, 2017). Time implies cost (Strucker & Hoffmann, 2017), and it can constitute a barrier for the organization to be open to social participation and co-production. In this sense, the domain of coordination mechanisms (Kozuch & Sienkiewicz-Malyjurek, 20 16; Lopes & Farias, 2020), the use of information and communication technologies (Struecker & Hoffmann, 2017) and digital technologies (Brandsen, Steen, & Verschuere, 2018) to create effective spaces of participation becomes a key challenge.

Lopes & Farias (2020) also highlight the importance of political support from senior management, and the need to introduce collaboration as part of the strategy so that it is possible, over time, to develop a culture of collaboration.

The discussion on inducers and barriers associated with collaborative innovation is sustained on the recognition of Bureaucracy dysfunctions (Weber, 1978; Merton, 1978) as a central challenge (Torfing, 2013; Agranoff, 2014; Ansell & Torfing, 2014; Kozuch & Sienkiewicz-Malyjurek, 2016; Brandão & Bruno-Faria, 2017; Crosby et al., 2017, Struecker & Hoffmann, 2017; Lopes & Farias, 2020). The commitment to ordered rule, predictable decision-making, procedural rationality, among other characteristics of Bureaucracy, are based on technical expertise, specialization, and rule-bound categorization and governance. This approach makes it difficult to view the problem holistically and from different angles. It also difficults the application of new repertories, the exploration of new opportunities, and the real assumption of citizens as potential partners and co-producers who have assets to effectively contribute, instead of treating them as clients or subjects that are expected to comply (Crosby et al., 2017).

The next section presents the methodological procedures, followed by the results and final remarks topics.

3 METHODOLOGICAL PROCEDURES

This study involved a qualitative research, through a case study that analyzed the Citizen Audit Project (PAC), carried out by the General Audit Board of the State Secretariat of Finance of Santa Catarina, from the perspective of the participating actors.

Although PAC is the unit of analysis, the study went beyond the General Audit Board limits, and even beyond those of the State Secretariat of Finance, since the internal audit activity is a means to enable Public Administration bodies to achieve their goals. That is, in order to check and understand the impacts of General Audit Board´s actions, we examined the field of services’ provision.

Keeping this in mind, in the first stage of the research we described PAC’s origin and its implementation process, by explaining the methodology of its design and actions, the operations carried out between 2016 and 2018, and the main mechanisms used to promote co-production through interaction with citizens, users, and professionals. We carried out a documentary search that comprised DIAG’s internal documents, especially those related to PAC, such as operation reports, working papers, checklists, questionnaires, and operations’ planning. It also involved the corresponding national and state legislations.

The second stage focused on the identification of the challenges faced for PAC implementation as a participatory audit process, in the perception of its stakeholders, under the lens of the theory of innovation in the public sector. To delimit the research, we chose as the object of analysis at that stage the operations related specifically to the school environment (operations "School Lunch" and "School Regularity"). This cutting sought to standardize the study object, so that stakeholders’ perceptions regarded the same phenomenon. We conducted semi-structured interviews with key agents of public control or those who had interacted with PAC in these operations. This sample comprised 21 people: four from DIAG, including its director; one from the Office of the Federal Controller General; two from the Accounting Court of the State of Santa Catarina; five Education managers - from the State Secretariat of Education and from schools, such as principals and teachers; one member of the Parents and Teachers' Association; two parents that are members of the School Deliberative Council at the unit where their children study; one student that is a member of this Council; four students; and a representative of an outsourced company that provides school lunch.

The choice of interviewees sought to include actors involved with the audit process in different ways [including DIAG internal controllers, representatives of the Office of the Federal Controller General (CGU) and the State Accounting Court], and with the provision of selected services (operations "School Lunch" and "School Regularity"). We conducted the interviews during the month of May 2018, with an average duration of 60 minutes each.

We recorded, transcribed and analyzed the collected data through content analysis (Gil, 2008). The contents of the interviews were classified into categories of analysis derived from the guidelines of co-production, as an innovation in the public service, and from the properties of the co-production mechanisms of control, in order to identify the challenges for introducing co-production in the process of evaluating public services carried out by PAC.

4 RESULTS AND DISCUSSION

4.1 Description of Citizen Audit Project origin and its implementation process

The Citizen Audit Project (PAC), conceived by the General Audit Board of the State Secretariat of Finance of Santa Catarina, started in 2016. It is defined as: “an audit work, in the form of inspection, whose objective is to assess, under the citizen’s view, if public service meets his/her needs, taking into account the principles that rule public services in general” (GESC, 2017a, p. 796; 2018, p. 813).

The project's operation model comprises thematic operations, according to the public service assessed. As the Inspection Report No. 0001/16 mentions, these operations aim to "reflect the real perception that citizens have of the services or products received, in addition to establishing the State’s presence, active and vigilant" (GESC/SEF/DIAG, 2016a).

In 2016, DIAG conducted five operations, called School Lunch, Emergency, Transfers, Environment, and APAEs (Association of Parents and Friends of Handicapped People), in which internal auditors made on-site inspections equipped with checklists prepared by the coordination team, based on previous audits, legislation, contracts, agreements, information systems, and related documents (GESC/SEF/DIAG, 2016a, 2016b, 2016c, 2016d, 2016e). After the visits, auditors ranged the results and made recommendations to the Management Secretariat of each public service.

As of 2017, DIAG adopted an electronic questionnaire, by using the Google Forms tool as a kind of citizen report card, in order to get answers directly from the users; in this case, they were higher education students, beneficiaries or potential beneficiaries of scholarships from the UNIEDU program (GESC/SEF/DIAG, 2017c). UNIEDU is a program of the State of Santa Catarina, carried out by the Secretariat of Education, which favors the inclusion of young people in higher education that have difficulties in carrying out their studies. This program aggregates all programs for attending higher education students, based on Articles 170 and 171 of the State Constitution. For the UNIEDU operation, they also used a focus group with representatives of Higher Education Institutions for planning the tasks, as well as the distribution of a brochure with information on the UNIEDU Program and PAC.

In turn, in 2018 the electronic questionnaire was addressed to the school units’ boards and, at the same time, to the School Deliberative Councils (CDEs), within the scope of the operation "School Regularity". The mobilization of CDEs called the attention of the school community, especially the parents, to the evaluation, and to the chosen items. According to a mother who is a CDE member at the school that her son attends, answering the questionnaire provided sharing responsibilities between public agents (principal, teachers), parents, and students, in seeking information and meeting the demand proposed by the internal audit. It also became evident, from the testimony of principals, teachers, and parents, that the simple act of questioning about the control of certain expenses, such as electricity, water, or waste collection rate, provided information that was until then unknown for most of the members of these bodies, who started to include these matters in their agenda.

For each new operation, new interfaces for interaction with citizens/users were added, in order to get their contribution for the evaluation process of public services. Initially limited to a satisfaction survey, the participation of users and other professionals increased, with the use of electronic forms (Citizen Report Cards), interaction with public policy councils and users’ councils, utilization of focus groups at the stage of operations design, and informative campaigns through flyers. If, on the one hand, such initiatives were an initial stage of the promotion of co-production (Bovaird, 2007), on the other hand there was an effort in applying iterative design for successive operations (Cavalcante, 2019; Dell´Era et al., 2020). This required adaptive management and experimental and collaborative ways to format the participatory audit process (Ansell, 2012).

4.2 Challenges associated to the implementation of the Citizen Audit Project as a participatory audit process

As the first challenges for the establishment of co-production at the audit process, we identified those related to the bureaucratic model of management. Data collection showed planning centralization; small openness for receiving external contributions; technical execution without discretion; procedural and normative rigidity; hierarchical, vertical and rigid organizational structure; division of labor in a tight manner; and risk aversion. The literature on public sector innovation considers all these elements as bureaucratic dysfunctions and barriers (Weber, 1978; Merton, 1978; Behn, 1998; Torfing, 2013; Agranoff, 2014; Ansell & Torfing, 2014; Kozuch & Sienkiewicz-Malyjurek, 2016; Brandão & Bruno-Faria, 2017; Crosby et al., 2017, Lopes & Farias, 2020). The predominance of such elements shows a relevant trend of the General Audit Board’s operation, according to the mental model of compliance audit (Barzelay, 1997; Rocha, 2011; Olivieri, 2011), as acknowledged by the General Audit Board director and a PAC coordinator.

One aspect repeatedly mentioned by the interviewees refers to the openness of political agents and audit professionals to receive and take into consideration proposals from society. Although recognized as a practice that increases the accountability of the political system and adds legitimacy to state actions, its implementation is not always adopted by manager's free will; although it adds quality to the service, it also restricts the free discretion of the manager who submits to them (Landry & Angeles, 2011; Fung, 2013). From the standpoint of a State Accounting Court representative, this resistance would be the result of the “organizational culture” of the public administration, where political and technical agents do not accept the participation of external actors in the planning and execution of public services. This result reflects the challenges pointed out by Crosby et al. (2017), Struecker and Hoffmann (2017), Brandsen, Steen, and Verschuere (2018), and Lopes and Farias (2020).

For the representative of the school’s food supplier, the company would like to be seen as a government partner, which is part of the vision and mission of its strategic planning. She highlighted the firm’s competence in providing school meals, mentioning that while the State Secretariat of Education (SED) has only 3 nutrition professionals, the company has 60. Despite that, its role is limited to meet the schedule (menu, amounts, foodstuff) defined by SED. "The state has the teaching vision, it has to think on the pedagogical part; but regarding the food issue, critical for students’ performance, we are specialists and we know the details, the safety of the process", she says.

Osborne and Strokosch (2013) observe that public service providers consider co-production as time and resource consuming, that it will hamper their leadership, and not necessarily achieve a consensus, besides "diverting attention from the 'real' task of providing effective services" (p. 35). In this sense, the CGU representative mentioned that civil servants prefer to stay in a “comfort zone”, due to the rigidity of bureaucratic performance, than to deal with the constant changes that arise from interactions with citizens. These issues are also discussed by Struecker and Hoffmann (2017) and Crosby et al. (2017). The discussion brought by Steen and Tuurnas (2018), on the changes imposed by co-production and the challenges it brings to professionals, goes in the same direction.

On the other hand, the interviews also showed citizens’ lack of motivation to participate actively in the co-production of public services (Crosby et al., 2017; Struecker & Hoffmann, 2017; Brandsen, Steen, & Verschuere, 2018; Lopes & Farias, 2020). Although we can assume that citizens have a natural interest in taking part in the co-production of public services, Gascó and van Eijk (2018) address the need to understand what really motivates them to participate. Similarly, Struecker and Hoffmann (2017) observe that a significant part of recent studies that focus on social participation mentions ways to motivate their participation and facilitate the process of approaching public agents. From the reports of parents and students interviewed, interest to participate depends on the availability of information and transparency, both on the service to be co-produced and the purpose that moves public administration to seek interaction with citizens.

Another identified challenge arises from the search for standardization of production, delivery, and documentation of services, and measurement of performance, regarding time and resources. These aspects may inhibit the innovative operation, due to the possibility of decreasing the quantitative index of services provided by the new model. According to a representative from the General Audit Board, "time is a challenge. Time control is not the same as in the citizen's life". This inhibits society’s participation, since it takes a long time to implement an appropriate answer to its contributions. In turn, according to a student's mother, the three-year cycle of high school is too short to encourage parents to contribute to school issues. As Torfing (2013) explains, in general, innovative practices do not have the same rigor in documenting activities, which can result in penalizing public agencies for innovating. Thus, as in any innovative process, the implementation of the participatory audit method may not attain the expected results, which demands some level of risk-taking (Bovaird, 2007; Torfing, 2013).

Interviews also indicated challenges related to the emphasis on cost evaluation, that is, on the efficiency aspect, rather than checking the quality of the service provided (effectiveness). This challenge is a sensitive issue for citizens' engagement in innovative collaboration, since the content or quality of public service is the main point of interest for citizens and external actors. Regarding this aspect, Abrucio (1997) observes that, along with operational efficiency, linked to increased cost awareness, adaptive efficiency should also be assessed and encouraged, regarding the issue of flexibility, which is extremely important for institutions that undergo changes. In times of fiscal constraint and new technologies, adaptive efficiency that enables a better response to internal and external demands is more valuable than operational efficiency, which, although relevant, attains limited results as an objective itself.

Another challenge that emerged from the interviews is the lack of capacity of both civil servants and citizens (Crosby et al., 2017; Struecker & Hoffmann, 2017; Brandsen, Steen, & Verschuere, 2018 e Lopes & Farias, 2020). On the state side, there is a need for skilled personnel, financial resources, and appropriate infrastructure, including information and communication technologies (ICTs), to ensure the conditions for co-production of public services’ evaluations (WBG/ACIJ, 2015, Struecker & Hoffmann, 2017; Brandsen, Steen, & Verschuere, 2018). Another element that increases the challenge of state agents’ lack of capacity, according to principals and teachers interviewed, refers to the high turnover of executives (Brandão & Bruno-Faria, 2017), which causes discontinuities in innovation processes and harms knowledge building.

On the citizens' side, social problems are complex, comprised by a tangle of norms, interests, and limitations. Hence, for a full and effective participation of citizens in the process of modulating, implementing, and evaluating public policies, it is convenient to know specific issues, often with strictly technical nuances, not always available to those interested. For a mother who is a member of the CDE council, the lack of information causes “blindness in the school community”, and limits the council's ability to act. This challenge that arises from the complexity of the subject for appropriate participation is a common factor mentioned by authors such as Koppenjan, Kars, and Van Der Voort (2009), Landry and Angeles (2011), Fung (2013) and Osborne (2013), as well as in UN (2005, 2007, 2011) and World Bank (2013, 2014) publications.

The communication challenge can add to this scenario. For auditors, there are many difficulties associated with this aspect, which demand care for its improvement. Students report lack of knowledge about important events in school, due to inefficient communication. In addition, principals and teachers emphasize the importance of communication for understanding how to do the evaluation and fill out the citizen report card, linking this challenge to the lack of qualification. The General Audit Board director and members of the State Accounting Court emphasize the importance of improving the communication process by using a language and a format that citizens can understand.

Communication has a relevant role in the innovation process, because it is a key factor for generating synergy, integrating the various participants and leveling their knowledge (Ansell & Torfing 2014, Lopes & Farias, 2020). Thus, the absence or limitation of communication, either by using an inappropriate language for the target audience or a restricted coverage of the recipients, is an important challenge for innovation, especially when it relates to the engagement of citizens in the evaluation of co-production of services (Goetz & Jenkins, 2001; UN, 2007; World Bank, 2013; Effective Institutions Platform, 2014). Moreover, in the relationship between auditors and managers, communication should be improved, as highlighted by Olivieri, Loureiro, Teixeira, and Abrucio (2015); dialogue between the parties and the existence of different forms of coordination (Kozuch & Sienkiewicz-Malyjurek, 2016; Lopes & Farias, 2020) are most important for control as an instrument of good public management.

Directly linked to the communication challenge, literature also highlights the challenge of information asymmetry as a limitation for innovation, especially if related to control co-production. According to the representative of the school’s food supplier, "I need to understand in order to meet the needs". In the view of a mother participating in the CDE “when people become aware of the needs, they are willing to help. If you don't have a partnership, you don't have collaboration”. In fact, for citizens and civil society organizations to make social control or contribute to it, state transparency and the availability of useful, accurate, and updated information are essential (Goetz & Jenkins, 2001; Olivieri et al., 2015, UN, 2011b; Rocha et al., 2012; Rocha, 2013; World Bank, 2013; Schommer et al., 2015; WBG/ACIJ, 2015; ANSA-EAP, 2018).

The challenges associated with communication lead to another barrier mentioned by interviewees: the difficulty to act in a network. The speech of a student’s mother shows no integration between the school, the health center, and the social assistance center. Although they are physically close, the lack of communication results in isolated activities, without articulation. According to a DIAG manager, the difficulty of network management is associated to formalization and standardization, which sometimes lead to the predominance of one or some organizations over the others.

The establishment of networks is a strategy aligned with the New Public Governance model (Osborne, 2006; Torfing, 2013), which contrasts the monopolistic view of public management with an interinstitutional approach. This approach takes into account the complexity of relationships and the interdependence of actors, to establish an arrangement based on collaboration and leadership, in order to solve the difficult problems of our multifaceted society. However, it faces the challenge of coordinating multiple actors, who often are in conflict and may require techniques for mediation and conciliation (Osborne, 2013; Torfing, 2013; Ansell & Torfing, 2014). The multiple interests that underlie the performance of each organization, as well as that of individuals, demand a coordination process of the network that enables the harmonization of proposals and pacification of the decision process (Goldstein, 2008; Koppenjan et al., 2009; Landry & Angeles, 2011; Fung, 2013; Amanatidou, Gritzas, & Kavoulakos, 2015).

Interviewees were unanimous in acknowledging the convenience of establishing a network operation for PAC. One of them, with experience in a local social observatory and practice within the Brazil Social Observatory Network, warned that network setting should follow some principles, with freedom for each party, without excessive hierarchy or standardization. For him, it is convenient to establish an internal regulation, with guidelines for coordination and governance of the network, without implying exclusive prerogatives, hierarchy, or predominance of one member over the others. The interviewee's thinking is in line with the governance orientation for the arrangements that comprise networks of third-sector organizations, according to which it is necessary to keep low centralization and high interconnectivity among the actors (Verschuere, Brandsen, & Pestoff, 2012).

The interviewees' impressions on the relevance of PAC current interaction with public policy and users’ councils are aligned with the view of Schommer and Tavares (2017), who consider these bodies as spaces of citizen participation, where the co-production of public services plays the role of catalyzing innovations in the relationship between civil society and the state structure.

Several other challenges were mentioned, such as lack of clarity in the division of roles (Bovaird, 2007); possibility of co-opting participants by state agents or offer of a merely symbolic involvement (Arnstein, 1969; Salm & Menegasso, 2010; UN, 2011c; World Bank, 2013; WBG/ACIJ, 2015); or, still, the predominance of political-party interests over private. It is also important to highlight the need to define an appropriate measure for external contributions, since the excessive empowerment of users or the excessive customization of services may lead to inefficiency and to a high public expenditure for meeting all demands (Osborne, 2013).

Among the challenges mentioned for citizen engagement in control co-production, two issues stand out: lack of commitment and lack of trust. School principals complain that, in general, the community does not have a sense of belonging towards the school; therefore, they are not committed to the solutions of local problems. According to a school principal, the lack of feedback undermines commitment and trust. There is a lack of understanding about the operation’s purpose: “Is it just to inspect or what? Do they only check compliance or seek to improve the quality of services provided by the school?"

Ansell and Torfing (2014) observe that without commitment, there is no synergy or learning, which removes from the process the three mechanisms that link collaboration to innovation. For Goetz and Gaventa (2001), sometimes users adopt a conservative attitude regarding co-production, because they are afraid of losing the services they have. Pestoff (2011, p. a 23) warns that citizens are not like jack-in-the-box puppets, just waiting for someone to push the button to release them.

On the other hand, according to the Citizen Participatory Audit Learning Guide, trust is the foundation of continuous dialogue and problem solving (ANSA-EAP, 2018). Thus, citizens and state agents should notice that a joint action could bring value to each of them, by setting channels of continuous communication and by developing new learning, with proactivity and collaborative spirit. According to a student's mother, “audit sounds like pressure”, highlighting the need for information about the role of audit to generate trust and commitment. This would generate shared understanding, commitment to the public good, and engagement in community service (Denhardt & Denhardt, 2007, 2015; UN, 2011c; Padley, 2013; Osborne, 2013; Effective Institutions Platform, 2014).

5 FINAL REMARKS

Operational audit has become a central element of the New Public Management movement of ‘government reinvention’, by assisting managers with evaluations of state performance under the perspective of economic and efficiency principles. Similarly, participatory audit emerges as a promising evaluation tool for public services in our contemporary society, marked by the complexity and plurality of agents and organizations involved in the governance process.

The establishment of a collaborative process of participatory audit requires overcoming the compliance audit paradigm, which should be limited to those appropriate situations (investigation of complaints, frauds), and the operational cost paradigm that is excessively focused on financial costs, in order to include interfaces that value users and professionals’ expertise. To do this, both the audit body and auditors should follow the guidelines and values of responsive governance and citizens’ engagement in the audit process, so that their results positively affect the design and delivery of the assessed public services.

Therefore, this paper sought to contribute to the literature on public audit, since the participatory audit model is restricted to the technical production of international organizations, such as the United Nations and the World Bank, in spite of its potential to improve evaluations, from the experience of those who are most affected by public sector performance. Once we protect the principles of independence and objectivity that guide audit bodies, taking into consideration the contributions of citizen users and other professionals involved in the provision of public services, this model offers unique knowledge and insights for the evaluation and improvement of these services.

The paper also contributes to the literature on public sector innovation and co-production of public services, especially in the area of control and audit, by identifying, through empirical evidence, the main challenges required to establish an innovative process of participatory audit. The results show an alignment between the challenges identified and the theoretical basis used as reference.

Regarding practical implications, it provides ground for PAC consolidation and institutionalization, and its replication by governments interested in implementing innovations through the process of participatory audit. It presents and analyzes the main challenges inherent to co-production in the activity of public services’ assessment by internal audit. The findings can also allow defining guidelines and skills that deserve development in order to implement this kind of initiative effectively.

However, both regarding contributions to the literature and practical implications, this research has the central limitation of being a case study on PAC experience. Therefore, future studies should focus on other contexts may contribute to expand the reach of our findings. According to Mckenna (2021), future studies could also explore this subject in the specific light of the co-assessment literature, rather than taking co-production in a broader sense as a reference. Studies that investigate the benefits, challenges, and how to apply co-assessment at each stage of the evaluation process are also relevant.

Similarly, other studies could be carried out focusing on the co-production of public service evaluation, through the lens of social innovation. This is a more specific look than the delimitation of this research, which considered the innovation literature in a more comprehensive way. For Evers and Ewert (2021), the current debate on co-production may benefit from social innovation research.

Finally, we highlight the importance of future studies that investigate how the digital transformation process can contribute to overcome barriers and promote key changes in the co-production of public service evaluation (Clark, 2021; Mckenna, 2021), and for co-creating innovative solutions and public value (Torfing et al., 2021).

ACKNOWLEDGEMENT

The authors thank the State Foundation for Research Support of Santa Catarina -FAPESC - for the financial aid to develop this study.

REFERENCES

• ABRUCIO, F. L. (1997). O impacto do modelo gerencial na Administração Pública: Um breve estudo sobre a experiência internacional recente. Cadernos ENAP, n. 10. http://repositorio.enap.gov.br/handle/1/556
  » http://repositorio.enap.gov.br/handle/1/556
• Affiliated Network for Social Accountability in East Asia and the Pacific [ANSA-EAP] (2018). CPA Learning Guidebook. Retrieved from http://www.ansa- eap.net/resources/thematic/cpa-learning-guidebooks/
  » http://www.ansa- eap.net/resources/thematic/cpa-learning-guidebooks/
• AGRANOFF, R. (2014). Reconstructing bureaucracy for service: Innovation in the governance era. In: C. Ansell & J. Torfing (Eds.), Public Innovation through Collaboration and Design (ch. 3). New York: Routledge.
• AMANATIDOU, E., GRITZAS, G., & KAVOULAKOS, K. I. (2015). Time banks, co-production and foresight: Intertwined towards an alternative future. Foresight: the Journal of Futures Studies, Strategic Thinking and Policy, 17(4), 308-331. https://doi.org/10.1108/FS-05-2014-0035
  » https://doi.org/https://doi.org/10.1108/FS-05-2014-0035
• ANSELL, C. (2012). What is a 'Democratic Experiment'? Contemporary Pragmatism, 9(2), 159-180. https://doi.org/10.1163/18758185-90000235
  » https://doi.org/https://doi.org/10.1163/18758185-90000235
• ANSELL, C., & TORFING, J. (Eds.) (2014). Public Innovation through Collaboration and Design . London: Routledge.
• ARNSTEIN, S. R. (1969). A ladder of citizen participation. Journal of the American Institute of Planners, 35(4), 216-224. https://doi.org/10.1080/01944366908977225
  » https://doi.org/https://doi.org/10.1080/01944366908977225
• BARRUTIA, J. M., & ECHEBARRIA, C. (2019). Drivers of exploitative and explorative innovation in a collaborative public-sector context. Public Management Review, 21(3), 446-472. https://doi.org/10.1080/14719037.2018.1500630
  » https://doi.org/https://doi.org/10.1080/14719037.2018.1500630
• BARZELAY, M. (1997). Central audit institutions and performance auditing: A comparative analysis of organizational strategies in the OECD. Governance, 10(3), 235-260. https://doi.org/10.1111/0952-1895.411997041
  » https://doi.org/https://doi.org/10.1111/0952-1895.411997041
• BARZELAY, M (2000a). Performance auditing and the new public management: Changing roles and strategies of central audit institutions. In T. Miyakawa (Ed.), The Science of Public Policy: Essential Readings in Policy Sciences II, vol. VII (pp. 52-90). London: Routledge .
• BARZELAY, M. (2000b). The new public management: A bibliographical essay for Latin American (and other) scholars. International Public Management Journal 3(2), 229 - 265. https://doi.org/10.1016/S1096-7494(00)00038-6
  » https://doi.org/https://doi.org/10.1016/S1096-7494(00)00038-6
• BEHN, R. D. (1998). O novo paradigma da gestão pública e a busca da accountability democrática. Revista do Serviço Público, 49(4), 5-45. Retrieved from http://repositorio.enap.gov.br/handle/1/1560
  » http://repositorio.enap.gov.br/handle/1/1560
• BOVAIRD, T. (2007). Beyond engagement and participation: User and community coproduction of public services. Public Administration Review, 67(5), 846-860. https://doi.org/10.1111/j.1540-6210.2007.00773.x
  » https://doi.org/https://doi.org/10.1111/j.1540-6210.2007.00773.x
• BOYLE, D., & HARRIS, M. (2009). The challenge of co-production: How equal partnerships between professionals and the public are crucial to improving public services [Discussion paper]. London: NESTA. Retrieved from http://b.3cdn.net/nefoundation/312ac8ce93a00d5973_3im6i6t0e.pdf
  » http://b.3cdn.net/nefoundation/312ac8ce93a00d5973_3im6i6t0e.pdf
• BRANDÃO, S. M., & BRUNO-FARIA, M. F. (2017). Barreiras à inovação em gestão em organizações públicas do governo federal brasileiro: Análise da percepção de dirigentes. In P. Cavalcante, M. Camões, B. Cunha, & W. Severo (Orgs.), Inovação no Setor Público: teoria, tendências e casos no Brasil (cap. 7). Brasília: IPEA. Retrieved from http://repositorio.ipea.gov.br/handle/11058/8795
  » http://repositorio.ipea.gov.br/handle/11058/8795
• BRANDSEN, T., STEEN, T., & VERSCHUERE, B. (2018). Co-production and co-creation. Engaging citizens in public services. New York: Routledge . http://library.oapen.org/handle/20.500.12657/25001
  » http://library.oapen.org/handle/20.500.12657/25001
• CAVALCANTE, P. L. C. (Org.) (2019). Inovação e políticas públicas: Superando o mito da ideia. Brasília: IPEA. http://repositorio.ipea.gov.br/handle/11058/9330
  » http://repositorio.ipea.gov.br/handle/11058/9330
• CAVALCANTE, P., CAMÕES, M., CUNHA, B., & SEVERO, W. (2017). Inovação no Setor Público: teoria, tendências e casos no Brasil . Brasília: ENAP/IPEA. http://repositorio.enap.gov.br/handle/1/2989
  » http://repositorio.enap.gov.br/handle/1/2989
• CLARK, B. Y. (2021). Co-assessment trough digital technologies. In E. Loeffler & T. Bovaird (Eds.). The Palgrave Handbook of Co-Production of Public Services and Outcomes (pp. 429-449). London, UK: Palgrave Mcmillan.
• CROSBY, B. C., HART, P., & TORFING, J. (2017). Public value creation through collaborative innovation. Public Management Review , 19(5), 655-669. https://doi.org/10.1080/14719037.2016.1192165
  » https://doi.org/https://doi.org/10.1080/14719037.2016.1192165
• DELL'ERA, C., MAGISTRETTI, S., CAUTELA, C., VERGANTI, R., & ZURLO, F. (2020). Four kinds of design thinking: From ideating to making, engaging, and criticizing. Creativity and Innovation Management, 29(2), 324-344. https://doi.org/10.1111/caim.12353
  » https://doi.org/https://doi.org/10.1111/caim.12353
• DENHARDT, R. B., & DENHARDT, J. V. (2007). The new public service: Serving, not steering (Expanded edition). New York: ME Sharpe.
• DENHARDT, R. B., & DENHARDT, J. V. (2015). The new public service revisited. Public Administration Review , 75(5), 664-672. https://doi.org/10.1111/puar.12347
  » https://doi.org/https://doi.org/10.1111/puar.12347
• Effective Institutions Platform [EIP] (2014). Supreme Audit Institutions and Stakeholder Engagement Practices: A Stocktaking Report. Retrieved from https://www.effectiveinstitutions.org/media/Stocktake_Report_on_Supreme_Audit_Institutions_and_Citizen_Engagement_.pdf
  » https://www.effectiveinstitutions.org/media/Stocktake_Report_on_Supreme_Audit_Institutions_and_Citizen_Engagement_.pdf
• EVERS A. EWERT, B. (2021). Unerstanding Co-Production as a Social Innovation. In E. Loeffler & T. Bovaird (Eds.). The Palgrave Handbook of Co-Production of Public Services and Outcomes (pp. 133-154). London, UK: Palgrave Mcmillan .
• FUNG, A. (2013). Infotopia: Unleashing the democratic power of transparency. Politics & Society, 41(2), 183-212. Retrieved from http://www.transparencypolicy.net/assets/FungInfotopia2013.pdf
  » http://www.transparencypolicy.net/assets/FungInfotopia2013.pdf
• GASCÓ, M., & VAN EIJK, C. (2018). Unravelling the co-producers: Who era they and what motivations do they have? In T. Brandsen, T. Steen, & B. Verschuere (Eds.), Co-production and co-creation. Engaging citizens in public services (pp. 63-76). New York: Routledge .
• GOETZ, A. M., & GAVENTA, J. (2001). Bringing citizen voice and client focus into service delivery [IDS Working Paper n. 138]. Brighton: Institute of Development Studies. Retrieved from https://opendocs.ids.ac.uk/opendocs/bitstream/handle/20.500.12413/3900/Wp138.pdf
  » https://opendocs.ids.ac.uk/opendocs/bitstream/handle/20.500.12413/3900/Wp138.pdf
• GOETZ, A. M., & JENKINS, R. (2001). Hybrid forms of accountability: citizen engagement in institutions of public-sector oversight in India. Public Management Review , 3(3), 363-383. https://doi.org/10.1080/14616670110051957
  » https://doi.org/https://doi.org/10.1080/14616670110051957
• GOLDSTEIN, B. E. (2008). Skunkworks in the embers of the Cedar Fire: Enhancing resilience in the aftermath of disaster. Human Ecology, 36(1), 15-28. https://doi.org/10.1007/s10745-007-9133-6
  » https://doi.org/https://doi.org/10.1007/s10745-007-9133-6
• Governo do Estado de Santa Catarina/Secretaria Estadual da Fazenda/Auditoria Geral [GESC/SEF/DIAG]. (2016a). Relatório de Inspeção Nº 0001/16 - Operação Merenda Escolar. Florianópolis: DIAG.
• GESC/SEF/DIAG. (2016b). Relatório de Inspeção Nº 0002/16 - Operação Emergência. Florianópolis: DIAG .
• GESC/SEF/DIAG. (2016c). Relatório de Inspeção Nº 0003/16 - Operação transferências. Florianópolis: DIAG .
• GESC/SEF/DIAG. (2016d). Relatório de Inspeção Nº 0004/16 - Operação Meio Ambiente. Florianópolis: DIAG .
• GESC/SEF/DIAG. (2016e). Relatório de Inspeção Nº 0036/16 - Operação APAE’s. Florianópolis: DIAG .
• GESC/SEF/DIAG. (2017a). Balanço Geral do Estado, Exercício de 2016. Vol. II. Retrieved from http://www.sef.sc.gov.br/arquivos_portal/relatorios/8/Volume_II.pdf
  » http://www.sef.sc.gov.br/arquivos_portal/relatorios/8/Volume_II.pdf
• GESC/SEF/DIAG. (2017b). Relatório de Inspeção Nº 0004/17 - Operação Merenda Escolar - Norte. Florianópolis: DIAG .
• GESC/SEF/DIAG. (2017c). Relatório de Inspeção Nº 0156/17 - Operação Uniedu. Florianópolis: DIAG .
• GESC/SEF/DIAG. (2018). Balanço Geral do Estado, Exercício de 2017. Vol. II. Retrieved from http://www.sef.sc.gov.br/arquivos_portal/relatorios/8/Balanco_Geral_Volume_2.pdf
  » http://www.sef.sc.gov.br/arquivos_portal/relatorios/8/Balanco_Geral_Volume_2.pdf
• GRANDVOINNET, H., ASLAM, G., & RAHA, S. (2015). Opening the Black Box: The Contextual Drivers of Social Accountability (New Frontiers of Social Policy Collection). Washington, DC: World Bank. Retrieved from https://www.openknowledge.worldbank.org/handle/10986/21686
  » https://www.openknowledge.worldbank.org/handle/10986/21686
• HOOD, C. (1995). The "new public management" in the 1980s: variations on a theme. Accounting, Organizations and Society, 20(2/3), 93-109. https://doi.org/10.1016/0361-3682(93)E0001-W
  » https://doi.org/https://doi.org/10.1016/0361-3682(93)E0001-W
• HOOD, C., & JACKSON, M. W. (1991). Administrative argument. London: Dartmouth Publishing Group.
• Institute of Internal Auditors [IIA] (2016). International Professional Practices Framework - IPPF. Retrieved from https://na.theiia.org/standards-guidance/Pages/Standards-and-Guidance-IPPF.aspx
  » https://na.theiia.org/standards-guidance/Pages/Standards-and-Guidance-IPPF.aspx
• International Organization of Supreme Audit Institutions [INTOSAI] (2004). GOV 9100 Guidelines for Internal Control Standards for the Public Sector. Retrieved from http://www.issai.org/en_us/site-issai/issai-framework/intosai-gov.htm
  » http://www.issai.org/en_us/site-issai/issai-framework/intosai-gov.htm
• INTOSAI (2013a). Report on the 22nd UN/INTOSAI Symposium on Government Audit. Vienna. Retrieved from http://www.intosai.org/fileadmin/downloads/downloads/5_events/symposia/2013/EN_22_UN-INT_SympReport_Attachments.pdf
  » http://www.intosai.org/fileadmin/downloads/downloads/5_events/symposia/2013/EN_22_UN-INT_SympReport_Attachments.pdf
• INTOSAI (2013b). Communicating and Promoting the Value and Benefits of SAIs: An INTOSAI Guideline. Retrieved from http://iniciativatpa.org/2012/wp-content/uploads/2014/06/Communicating-and-promoting-the-value-and-benefits-of-SAIs-Guideline.pdf
  » http://iniciativatpa.org/2012/wp-content/uploads/2014/06/Communicating-and-promoting-the-value-and-benefits-of-SAIs-Guideline.pdf
• KINDER, T. (2013). Innovation in an inter-organizational context. In S. P. Osborne & L. Brown (Eds.), Handbook of Innovation in Public Services (ch. 21). Cheltenham, UK: Edward Elgar.
• KLIKSBERG, B. (1999). Seis teses não-convencionais sobre participação. Revista de Administração Pública, 33(3), 7-37. Retrieved from http://bibliotecadigital.fgv.br/ojs/index.php/rap/article/view/7634/6168
  » http://bibliotecadigital.fgv.br/ojs/index.php/rap/article/view/7634/6168
• KLUMB, R., & HOFFMANN, M. G. (2016). Inovação no setor público e evolução dos modelos de administração pública: O caso do TRE-SC. Cadernos Gestão Pública e Cidadania, 21(69), 86-103. https://doi.org/10.12660/cgpc.v21n69.53902
  » https://doi.org/https://doi.org/10.12660/cgpc.v21n69.53902
• KOPPENJAN, J., KARS, M., & VAN DER VOORT, H. (2009). Vertical politics in horizontal policy networks: Framework setting as coupling arrangement. Policy Studies Journal, 37(4), 769-792. https://doi.org/10.1111/j.1541-0072.2009.00334.x
  » https://doi.org/https://doi.org/10.1111/j.1541-0072.2009.00334.x
• KOZUCH, B., & SIENKIEWICZ-MALYJUREK, K. (2016). Factors of effective inter-organizational collaboration: a framework for public management. Transylvanian Review of Administrative Sciences, 47, 97-115. Retrieved from http://rtsa.ro/tras/index.php/tras/article/view/474/463
  » http://rtsa.ro/tras/index.php/tras/article/view/474/463
• LANDRY, J., & ANGELES, L. (2011). Institutionalizing participation in municipal policy development: Preliminary lessons from a start-up process in Plateau-Mont-Royal. Canadian Journal of Urban Research, 20(1), 105-131. Retrieved from https://www.jstor.org/stable/26193857
  » https://www.jstor.org/stable/26193857
• LOPES, A. V., & FARIAS, J. S. (2020). How can governance support collaborative innovation in the public sector? A systematic review of the literature. International Review of Administrative Sciences, p. 1-17. https://doi.org/10.1177/0020852319893444
  » https://doi.org/https://doi.org/10.1177/0020852319893444
• MCKENNA, D. (2021). Co-assessment through citizens and service users in audit, inspection ad scrutinity. In E. Loeffler & T. Bovaird (Eds.). The Palgrave Handbook of Co-Production of Public Services and Outcomes (pp. 451-467). London, UK: Palgrave Mcmillan .
• MERTON, R. (1978). Estrutura burocrática e personalidade. In E. Campos (Org.) Sociologia da burocracia (4a ed.). Rio de Janeiro: Zahar.
• MOORE, M., & HARTLEY, J. (2010). Innovations in governance. In S. P. Osborne (Ed.), The new public Governance ? Emerging perspectives on the theory and practice of public governance. London: Routledge .
• OLIVIERI, C. (2011). Os controles políticos sobre a burocracia. Revista de Administração Pública , 45(5), 1395-1424. Retrieved from http://bibliotecadigital.fgv.br/ojs/index.php/rap/article/view/7041
  » http://bibliotecadigital.fgv.br/ojs/index.php/rap/article/view/7041
• OLIVIERI, C., LOUREIRO, M. R., TEIXEIRA, M. A. C., & ABRUCIO, F. L. (2015). Control and public management performance in Brazil: Challenges for coordination. International Business Research; 8(8), 181-190. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.832.3466&rep=rep1&type=pdf
  » http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.832.3466&rep=rep1&type=pdf
• OSBORNE, S. P. (2006). The new public governance? Public Management Review , 8(3), 377-388. https://doi.org/10.1080/14719030600853022.
  » https://doi.org/https://doi.org/10.1080/14719030600853022
• OSBORNE, S. P. (2010). Public governance and public services delivery: A research agenda for the future. In S. P. Osborne (Ed.), The new public governance? Emerging perspectives on the theory and practice of public governance. New York: Routledge .
• OSBORNE, S. P. (2013). A service-influence approach to public service innovation? In S.P. Osborne, & L. Brown (Eds.), Handbook of Innovation in Public Services . Cheltenham, UK: Edward Elgar .
• OSBORNE, D., & GAEBLER, T. (1992). Reinventing government: How the entrepreneurial spirit is transforming the public sector. New York: Penguin Books.
• OSBORNE, S. P., RADNOR, Z., & STROKOSCH, K (2016). Co-production and the co-creation of value in public sector: a suitable case for treatment? Public Management Review , 18(5), 639-653. https://doi.org/10.1080/14719037.2015.1111927
  » https://doi.org/https://doi.org/10.1080/14719037.2015.1111927
• OSBORNE, S. P., & STROKOSCH, K. (2013). It takes two to tango? Understanding the co-production of public services by integrating the services management and public administration perspectives. British Journal of Management, 24, S31-S47. https://doi.org/10.1111/1467-8551.12010
  » https://doi.org/https://doi.org/10.1111/1467-8551.12010
• PADLEY, M. (2013). Delivering localism: The critical role of trust and collaboration. Social Policy & Society, 12(3), 343-354. Retrieved from http://journals.cambridge.org/abstract_S1474746413000134
  » http://journals.cambridge.org/abstract_S1474746413000134
• PESTOFF, V. (2011). Co-production, new public governance and third sector social services in Europe. Ciências Sociais Unisinos, 47(1), 15-24. https://doi.org/10.4013/csu.2011.47.1.02
  » https://doi.org/https://doi.org/10.4013/csu.2011.47.1.02
• POWER, M. (1994). The audit explosion. London: Demos. Retrieved from https://www.demos.co.uk/files/theauditexplosion.pdf
  » https://www.demos.co.uk/files/theauditexplosion.pdf
• POWER, M. (2000). The audit society - Second thoughts. International Journal of Auditing, 4(1), 111-119. https://doi.org/10.1111/1099-1123.00306
  » https://doi.org/https://doi.org/10.1111/1099-1123.00306
• ROCHA, A. C. (2007). Auditoria de Gestão - Uma forma eficaz de promoção da Accountability. In Anais do 31º Encontro Nacional da Associação Nacional de Pós-Graduação e Pesquisa em Administração, Rio de Janeiro. Rio de Janeiro: ANPAD
• ROCHA, A. C. (2011a). Accountability na Administração Pública: Modelos teóricos e abordagens. Contabilidade, Gestão e Governança, 14(2), 82-97. Retrieved from https://www.revistacgg.org/contabil/article/view/314
  » https://www.revistacgg.org/contabil/article/view/314
• ROCHA, A. C. (2011b). Realização do potencial de accountability dos pareceres prévios do Tribunal de Contas do Estado de Santa Catarina (PhD dissertation). Universidade Federal da Bahia, Salvador. Retrieved from http://www.adm.ufba.br/sites/default/files/publicacao/arquivo/Arlindo%20Rocha%20-%20Tese%20completa%20-%20Vers%C3%A3o%20final%20em%20PDF.pdf
  » http://www.adm.ufba.br/sites/default/files/publicacao/arquivo/Arlindo%20Rocha%20-%20Tese%20completa%20-%20Vers%C3%A3o%20final%20em%20PDF.pdf
• ROCHA, A. C., & QUINTIERE, M. M. R (2011). Auditoria Governamental: Uma abordagem metodológica da Auditoria de Gestão. Curitiba: Juruá.
• ROCHA, A. C., SPANIOL, E. L., SCHOMMER, P. C., & SOUZA, A. D. E (2012). A coprodução do controle como bem público essencial à accountability. Anais do XXXVI Encontro Nacional da Associação de Pesquisa e Pós-Graduação em Administração, Rio de Janeiro. Rio de Janeiro: ANPAD.
• ROCHA, A. C. (2013). Accountability - Constituinte necessária das sociedades democráticas. ReFAE -Revista da Faculdade de Administração e Economia, 5(1), 81-100. Retrieved from http://bibliotecadigital.tse.jus.br/xmlui/handle/bdtse/2888
  » http://bibliotecadigital.tse.jus.br/xmlui/handle/bdtse/2888
• SALM, J. F., & MENEGASSO, M. E. (2009). Os modelos de Administração Pública como estratégias complementares para a coprodução do bem público. Revista de Ciências da Administração, 11(25), 97-120. https://doi.org/10.5007/2175-8077.2009v11n25p83
  » https://doi.org/https://doi.org/10.5007/2175-8077.2009v11n25p83
• SCHOMMER, P. C., ROCHA, A. C., SPANIOL, E. L., DAHMER, J., & SOUZA, A. D. (2015). Accountability and co-production of information and control: Social observatories and their relationship with government agencies. Revista de Administração Pública , 49(6), 1375-1400. http://dx.doi.org/10.1590/0034-7612115166
  » https://doi.org/http://dx.doi.org/10.1590/0034-7612115166
• SCHOMMER, P. C., & TAVARES, A. O. (2017). Gestão Social e Coprodução de Serviços Públicos. Curso de Gestão Social (Fascículo 4). Fortaleza: Fundação Demócrito Rocha/UANE/BID/STDS-CE. Retrieved from https://drive.google.com/file/d/1qxsytR4SNCBW8mkdsny8EUT5RGf13ONE/view
  » https://drive.google.com/file/d/1qxsytR4SNCBW8mkdsny8EUT5RGf13ONE/view
• SCHUCH, E. M., & HOFFMANN, M. G. (2021). Co-criação e design thinking: uma experiência de inovação no serviço público em um município brasileiro. Teoria e Prática em Administração, Ahead of Print, 1-13.
• STEEN, T., & TUURNAS, S. (2018). The roles of the professional in co-production and co-creation processes. In T. Brandsen, T. Steen, & B. Verschuere (Eds.). Co-production and co-creation. Engaging citizens in public services (ch. 8). New York: Routledge .
• STROKOSCH, K. (2013). Co-production and innovation in public services: Can co-production drive innovation? In S. P. Osborne, & L. Brown (Eds.). Handbook of Innovation in Public Services (pp. 375-389). Cheltenham, UK: Edward Elgar .
• STRUECKER, D., & HOFFMANN, M. G. (2017). Participação social nos serviços públicos: caracterização do estado da arte por meio da bibliometria e da revisão sistemática. REGE - Revista de Gestão, 24(4), 371 - 380. https://doi.org/10.1016/j.rege.2017.03.008
  » https://doi.org/https://doi.org/10.1016/j.rege.2017.03.008
• TORFING, J. (2013). Collaborative innovation in the public sector. In S. P. Osborne & L. Brown (Eds.). Handbook of Innovation in Public Services (pp. 301-316). Cheltenham, UK: Edward Elgar .
• TORFING, J., FERLIE, E., JUKIĆ, T. & ONGARO, E. (2021). A theoretical framework for studying the co-creation of innovative solutions and public value. Policy & Politics, 49 (2), 189-209. https://doi.org/10.1332/030557321X16108172803520
  » https://doi.org/https://doi.org/10.1332/030557321X16108172803520
• TORFING, J., SORENSEN, E., & ROISELAND, A. (2019). Transforming the public sector into an arena for co-creation: Barriers, drivers and ways forward. Administration and Society, 51(5), 795-825. https://doi.org/10.1177/0095399716680057
  » https://doi.org/https://doi.org/10.1177/0095399716680057
• TORFING, J., & TRIANTAFILLOU, P. (2016). Enhancing Public Innovation by Transforming Public Governance ? New York: Cambridge University Press.
• TRISCHLER, J., & SCOTT, D. R. (2016). Designing public services. The usefulness of three service design methods for identifying user experience. Public Management Review , 18(5), 718-739. https://doi.org/10.1080/14719037.2015.1028017
  » https://doi.org/https://doi.org/10.1080/14719037.2015.1028017
• United Nations [UN] (2005). Unlocking the human potential for public sector performance. World Public Sector Report 2005. New York: Department of Economic and Social Affairs. Retrieved from http://unpan1.un.org/intradoc/groups/public/documents/un/unpan021616.pdf
  » http://unpan1.un.org/intradoc/groups/public/documents/un/unpan021616.pdf
• United Nations (2007). Auditing for social change: A strategy for citizen engagement in public sector accountability. New York: Department of Economic and Social Affairs . Retrieved from https://digitallibrary.un.org/record/610334
  » https://digitallibrary.un.org/record/610334
• United Nations (2011a). Engaging citizens to enhance public sector accountability and prevent corruption in the delivery of public services [Expert Group Meeting report]. Vienna: Department of Economic and Social Affairs. Retrieved from http://unpan1.un.org/intradoc/groups/public/documents/un/unpan047647.pdf
  » http://unpan1.un.org/intradoc/groups/public/documents/un/unpan047647.pdf
• United Nations (2011b). A practical guide to social audit as a participatory tool to strengthen democratic governance, transparency, and accountability. New York: UNDP - Regional Bureau for Latin America and the Caribbean. Retrieved from http://www.undp-aciac.org/publications/ac/books/practicalguide-socialaudit-e.pdf
  » http://www.undp-aciac.org/publications/ac/books/practicalguide-socialaudit-e.pdf
• United Nations Department of Economic and Social Affairs [UNDESA] (2013). Citizen engagement practices by supreme audit institutions: Compendium of innovative practices of citizen engagement by supreme audit institutions for public accountability. New York: United Nations. Retrieved from http://www.intosai.org/fileadmin/downloads/downloads/4_documents/publications/eng_publications/Compendium_UNPAN92198.pdf
  » http://www.intosai.org/fileadmin/downloads/downloads/4_documents/publications/eng_publications/Compendium_UNPAN92198.pdf
• VERSCHUERE, B., BRANDSEN, T., & PESTOFF, V. (2012). Co-production: The state of the art in research and the future agenda. Voluntas: International Journal of Voluntary and Nonprofit Organizations, 23(4), 1083-1101. https:// doi.org/10.1007/s11266-012-9307-8
  » https://doi.org/https:// doi.org/10.1007/s11266-012-9307-8
• VOORBERG, W. H.; BEKKERS, V. J. J. M. & TUMMERS, L. G. (2014). A systematic review of co-creation and co-production: Embarking on the social innovation journey. Public Management Review , 17(9), 1333-1357. https://doi.org/10.1080/14719037.2014.930505
  » https://doi.org/https://doi.org/10.1080/14719037.2014.930505
• WEBER, M. (1978). Os fundamentos da organização burocrática: Uma construção do tipo ideal. In E. Campos (Org.), Sociologia da Burocracia (4a ed.). Rio de Janeiro: Zahar .
• World Bank (2014). Strategic framework for mainstreaming citizen engagement in World Bank Group operations. Retrieved from http://documents.worldbank.org/curated/en/266371468124780089/pdf/929570WP0Box380ategicFrameworkforCE.pdf
  » http://documents.worldbank.org/curated/en/266371468124780089/pdf/929570WP0Box380ategicFrameworkforCE.pdf
• World Bank Group/Asociación Civil por la Igualdad y la Justicia [WBG/ACIJ] (2015). E-Guide on engaging citizens in the audit process. Retrieved from http://www.e-participatoryaudit.org/index.html
  » http://www.e-participatoryaudit.org/index.html

• Plagiarism Check

  The ReA/UFSM maintains the practice of submitting all documents approved for publication to the plagiarism check, using specific tools, e.g.: CopySpider.

Publication Dates

• Publication in this collection
  13 Dec 2021
• Date of issue
  Oct-Dec 2021

History

• Received
  22 Jan 2021
• Accepted
  27 June 2021