Oversight Structures for Agentic AI in Public-Sector Organizations (2025)

Abstract: This paper finds that agentic AI systems intensify existing challenges to traditional public sector oversight mechanisms — which rely on siloed compliance units and episodic approvals rather than continuous, integrated supervision. We identify five governance dimensions essential for responsible agent deployment: cross-departmental implementation, comprehensive evaluation, enhanced security protocols, operational visibility, and systematic auditing. We evaluate the capacity of existing oversight structures to meet these challenges, via a mixed-methods approach consisting of a literature review and interviews with civil servants in AI-related roles. We find that agent oversight poses intensified versions of three existing governance challenges: continuous oversight, deeper integration of governance and operational capabilities, and interdepartmental coordination. We propose approaches that both adapt institutional mechanisms and design agent architectures compatible with public sector constraints.

## introduction artificial intelligence (ai) technologies, particularly large language model-based agents, hold the potential to fundamentally transform public sector organizations (psos) via efficiency enhancements and process automation @xcite @xcite . however, psos often lack the institutional capacity and agile governance 1 structures required for responsible adoption @xcite . these challenges are driven by characteristic features of bureaucratic organizations, such as procedural rigidity, hierarchical accountability, and dual demands for transparency and effectiveness, which complicate both 1 throughout this article, we use the term "governance" to refer specifically to tasks and structures for oversight of llm agents and other digital projects, rather than to the politicalscientific definition, which may encompass all pso activities. ai deployment and oversight @xcite . although recent work has proposed best practices for ai agent governance @xcite , it therefore remains unclear whether psos are structurally and procedurally equipped to implement these recommendations. the governance of agent systems likely requires both technical implementation and institutional adaptation -affecting processes, oversight mechanisms, and digital infrastructure @xcite . we ask the following research question: how do existing public sector ai governance structures meet the requirements for responsible agent deployment? based on interviews with civil servants, we find that agent governance requirements exasperate existing digital governance challenges in psos. we first outline the potential of ai agents in the public sector ( §2) and emerging agent governance tasks ( §3) . we derive a hypothesis on challenges these tasks pose to public-sector oversight structures from public administration literature ( §4), and describe our interview methodology ( §5). we summarize our findings on existing oversight structures ( §6) and, finally, chart the ways agent oversight intensifies existing governance challenges in psos ( §7). our core contribution is an evaluation of the feasibility of agent oversight in current real-world public-sector contexts. ## potential of agents in psos recent work demonstrates the application of llmbased agents to tasks in narrow domains. examples include implementations of agents designed to emulate domain experts such as academic researchers @xcite , office workers @xcite , developers @xcite or healthcare professionals @xcite . moreover, major llm providers started introducing their computer use agents for end-users in fall 2024 @xcite @xcite . while the ai community acknowledges the potential of such increasingly agentic systems @xcite , which integrate autonomous learning, tool use, and complex reasoning, @xcite emphasize their substantial current limitations, including susceptibility to benchmark overfitting, prohibitive operational costs, and unnecessarily complex computational architectures -which each scale with domain generality. narrow, domain-specific agentic systems may therefore represent a more practical and immediately valuable development pathway. the public sector is such a domain, as publicsector tasks possess ideal characteristics for automation: they are comprehensively documented, exhibit consistent structural patterns, and involve high-frequency repetition @xcite . focusing agent development on these wellstructured tasks provides a pragmatic pathway toward realizing the benefits of ai agents while mitigating their current limitations. previous waves of digitization standardized data formats, process explanations, and decision frameworks @xcite , contributing to bureaucratic rationality: the formal legibility and transparency of bureaucracies, from which they gain their legitimacy. llm-based agents potentially continue this process, e.g., by supporting documentation requirements in specific contexts @xcite . conversely, due to their well-known mechanistic inexplicability @xcite , ai systems including agents may also hinder, rather than enhance, the further rationalization of state entities. given the fundamental importance of correctlyattributed responsibility to bureaucratic legitimacy (cetina presuel and martinez sierra, 2024), successful integration of agentic systems therefore depends on the careful design of accountability mechanisms. the realization of this potential further depends on the resolution of myriad practical challenges commonly faced in psos: resource and capacity strain, varying levels of digitalization of existing processes, and growing internal and external reporting requirements, to name a few @xcite . ## agent governance challenges deploying llm-based agents in the public sector presents novel governance challenges over and above those introduced by non-agentic software. in this section, we identify five interdependent governance areas that collectively enable responsible agent deployment. ## public administration context this section draws on public administration theory to hypothesize that governing llm-based agents challenges existing governance structures in psos. psos are overwhelmingly weberian bureaucracies @xcite with strict delineation of responsibilities, specialization, hierarchical structure, and well-documented, impersonal processes @xcite . tasks are differentiated into secluded units @xcite with limited, formalized intra-unit communication, in the case of governance often in the form of approval processes. governance structures for ai, where existent, often inherit these features, characterized by episodic approvals and externalization @xcite . bureaucracies may arrive at these structures via "isomorphic" processes, in which they mimic similar organizations without considering whether these structures are well-suited for their individual case @xcite . the well-documented inefficiencies of these structures @xcite have led to the development of several more dynamic governance frameworks, notably new public management @xcite , digital-era governance @xcite , "governing by network" @xcite and derived meta-governance approaches @xcite . each of these have found limited success in practice, in part due to the persistence of external factors forcing rationalization, such as administrative law and freedom of information regulation @xcite . we therefore hypothesize that psos' existing governance structures are only partially compatible with the requirements posed by the integration of llm-based agents. while some novel agent governance tasks may be well-situated within existing bureaucratic oversight structures, holistically governing llm-based agents will require adaptations that challenge the typical segmentation, timing, and location of governance activities within psos. ## interview methodology in addition to our literature review, we conducted six semi-structured interviews with german officials across six agencies: three federal, two state, and one municipal agency. all participants were recruited through a certificate course on ai in the public sector coordinated by the first author. the interviews followed the interview guide in appendix a. we qualitatively analyzed the interviews using open coding @xcite . finally, we present the coreq checklist in appendix b @xcite following best practices in the field (adeoye-olatunde and olenik, 2021). ## findings on existing oversight structures here we summarize our interview findings into key attributes of existing governance structures for digital projects within psos. legal motivation: well-established governance generally only exists where legally mandated. in germany, ai systems as of today underlie no specific laws. as a result, in the surveyed organizations ai governance is largely handled through structures established for traditional digital and process governance. few organizations have aispecific governance, such as logging and oversight requirements, in place; only one interviewee cited an ai-specific governance body with formal authority in their organization. dedicated governance units: established digital governance responsibilities include data protection, cybersecurity, and accessibility. in the studied organizations, each of these are held by dedicated units or individuals, which frequently sit in standalone positions across the organization -in the case of data protection, this is legally required in the eu. these individuals wield significant authority, including veto rights on projects. event-triggered involvement: interviewees in implementing departments report proactively consulting governance teams to trigger compliance processes and obtain approval. these interactions are event-triggered when adapting projects in ways relevant to these teams. examples of named events were new projects, processing new forms of data, entering new stages of deployment, e.g., from poc to internal prototype, or adding new features to existing projects. governance is not generally considered a continuous process after approval is given or a solution is deployed. adversarial compliance processes: because compliance functions are rarely directly integrated in development processes, the relationship between compliance teams and project teams was often described as adversarial. projects scrutinized by compliance teams frequently require time-consuming iterative processes to redefine scope and structure before compliance approval is granted. some interviewees report "self-censorship" and reductions in the scope of projects before involving these units. limited employee capacity: interviewees lamented that outside "digital" units, employee capacity for digital skills is generally low. simultaneously, their workload and expected throughput is frequently very high, leading to the perception of additional duties, such as transition to new technologies, oversight of imperfectly functioning digital tools, or development of model evaluation metrics, as burdens rather than reliefs. success of breaking hierarchy: interviewees partaking in models of collaboration that break these hierarchical and adversarial structures report higher project success and smoother governance integration. examples include cross-functional expert steering teams that integrate governance functions in project management, and ad-hoc interest networks of ai specialists from many units across an organization. ## discussion we find strong evidence for our hypothesis that current governance structures face severe challenges in adapting to agent governance. in particular, our interviews reveal agent governance requirements may produce intensifications of governance challenges already familiar to psos from existing digitalization projects and implementations of nonagentic ai systems. here, we first aggregate three classes of shortcomings and conclude by highlighting promising paths forward for both agent architectures and pso governance structures. collectively, these developments imply successful agent governance is centrally coordinated, but diffused throughout psos. we visualize this transition in figure 1 . this matrix organization @xcite is evocative of governance models that iterate on strictly bureaucratic structures, such as those mentioned above ( §4). a promising direction of future research is therefore to evaluate which features of these neo-weberian governance approaches may be promising for agent governance -especially as the uptake of agentic systems drives more fundamental transformation of state structures @xcite . we propose three design principles for the tailoring of technical work on agent oversight to public-sector contexts. existing agent observability tooling, including logging of actions and tools, control over autonomy levels, and evaluation metrics, frequently interweaves subject-specific online oversight and technical, abstract offline oversight in combined interfaces and applications @xcite . our interviews show that psos frequently maintain organizational separation between these groups of tasks ( §6). oversight tooling should therefore be designed with delegation to these distinct groups in mind. non-technical public servants are usually illsuited and not enabled by the organization to translate technical oversight metrics into those required for performance of their duties. mechanisms by which they control and oversee agent actions must therefore reflect the requirements, logics, and language of the individual end user and the organization, rather than derive from the provided technology ( §3). beyond enabling non-expert oversight, this allows the attribution of responsibility for agent actions to humans. while the potential of agents ( §2), is enormous in the public sector, the ideal-state implementation relies on the prior end-toend digitalization of existing services, and is unlikely to be realized in psos with fragmented, often partially manual, processes across legacy systems ( §3). agent oversight systems must therefore allow for this variability. this includes designing systems that do not assume complete process observability, and incorporating human-in-the-loop fallback mechanisms for scenarios where automated oversight is insufficient.