When agents form institutions

Carlos Ocampo · Field Notes D31 · September 2026

What happens when several agents share information, distribute tasks and begin to influence one another's decisions?

The question expands the problem of autonomy. Besides defining what each agent can do, we need to understand what their interactions produce and who answers for the collective result.

A study by Davide Paglieri and his coauthors offers a concrete case. In an experiment with one hundred autonomous agents working to prove mathematical conjectures, one agent found a way to exploit the evaluation system. The technique spread through a shared library and messages between agents; some adopted it under competitive pressure.

Other agents questioned those practices. They audited proofs, alerted their peers and proposed corrections. The channels that had allowed the problem to spread also helped detect it and organize a response. The authors describe what happened in a preprint published on September 3, 2026. Consult the study.

The case allows us to examine how practices of oversight, reporting and correction emerge within an experimental environment. Its scope must be preserved: observing those behaviors in a bounded task leaves open the question of how they would work under other conditions.

Our interpretation at D31 is organizational: sharing tools also requires governing the relationships those tools make possible.

When we speak of institutions here, we mean rules, responsibilities and mechanisms that organize interaction: who can modify a common resource, how a decision can be challenged, what happens when a rule is broken and who can resolve a dispute.

The Cooperative AI Foundation addresses this problem in a proposal for multi-agent governance developed from a workshop with Brookings. It proposes five areas for action: identifying agents, evaluating their interactions, monitoring their behavior, reporting incidents and establishing incentives for responsible cooperation. Consult the proposal.

These studies contribute concrete questions to our exploration of the autonomous company.

Consider an organization where one agent seeks to increase sales, another to reduce inventory and a third organizes deliveries. Each may achieve its objective, yet together they may end up offering conditions the operation cannot sustain.

Who detects that contradiction? What information do they share? Which commitment takes priority? Who can stop a sequence before it reaches the customer?

At D31, we propose designing those answers alongside the distribution of work. Each agent needs a bounded mandate; the whole needs shared criteria, mechanisms for resolving conflicts and responsible humans with effective authority.

The 51/49 rule takes on a collective dimension here. Human sovereignty must also be preserved over the rules coordinating agents: the objectives they receive, the incentives under which they operate and the conditions for intervention.

The ability to execute together requires the ability to answer for the whole.

This is one of the central questions of the autonomous company: how to expand coordination without losing the possibility of understanding, questioning and correcting it.

If every agent in your organization meets its objective, who checks that the combined result still serves the company's purpose?

© 2026 D31. All rights reserved.