The Control Plane: Governing Agentic AI at Scale

This guide started as a whitepaper for engineering leaders at a major Canadian bank. They were not starting from zero. They had agents on LangGraph in pre-production: checkpointing to Postgres, subgraphs for specialist roles, interrupt() wired up for human review. They had read the docs and built real things. So when I walked in to talk about orchestration, a principal engineer asked the only question worth asking: “We already have LangGraph. It has persistence, it has human-in-the-loop, it has state. Why would we put anything else in the middle?”

There is a wrong way to answer that question, and most vendors take it. The wrong answer is tribal: my runtime versus your runtime, my logo on the architecture diagram. That answer deserves the eye-roll it gets. I should also disclose my position before going further: I work at Camunda, and one of the possible answers in this guide is the category of product my employer sells. Discount my framing accordingly. I will try to earn the trust back by being specific about trade-offs, by naming the alternatives honestly, and by telling you when you need none of this.

The right answer is a separation of concerns. Agent frameworks are excellent at the inner loop: the cycle of reasoning, tool selection, and memory that turns a goal into actions. They are structurally unsuited to owning the outer loop: the set of obligations an institution carries whether or not an LLM is involved. The answer the bank needed was not rip-and-replace. It was a control plane that orchestrates agents built on any framework, and a clear line about which loop owns what. This guide is that whitepaper, generalized.

The inner loop and the outer loop

The inner loop is what a framework runs between receiving a task and producing a result. Assemble the prompt, reason about the goal, pick a tool, call it, read the output, update memory, decide whether to loop again. LangGraph, CrewAI, and the OpenAI Agents SDK are genuinely good at this, and they are improving fast. I have no interest in replacing them, and neither should you. If your team is productive in LangGraph, that is an asset.

The outer loop is everything the institution is obligated to wrap around that cycle. It is not intelligence. It is accountability. The two loops have different owners, different lifetimes, and different audiences.

Inner loop (owned by the agent framework)	Outer loop (owned by the control plane)
Reasoning and planning	Audit trail that exists without custom logging
Tool selection and invocation	Human approval with identity and authorization
Prompt assembly and context management	SLAs, escalations, and durable timers
Short-term and episodic memory	Policy as data, owned by compliance
Self-correction and retry on bad output	Versioning and migration of long-running work
Structured output and artifact production	Incident investigation in one place
Model choice and token economics	Atomic cancellation and compensation

The important claim is not that frameworks lack these features today. It is that they are the wrong home for them structurally. A framework runs inside your application process. Its state store exists so the graph can resume, not so an auditor can attest. Its authorization context is whatever your application passed in. Its lifetime is the lifetime of a deploy. The outer loop’s requirements are institutional: they have to outlive processes, deploys, team reorganizations, and sometimes the framework itself. A LangGraph checkpointer is a fine piece of engineering for resumption. It was never designed to be evidence. Guardrails in the OpenAI Agents SDK run in the same process as the agent they guard, which is exactly where a control should not live. To be fair, the hosted platforms around these frameworks (LangSmith, LangGraph Platform, and their equivalents) add real observability and ops tooling. But what they add is developer-shaped and framework-bound. The outer loop’s audience includes people who will never open a trace viewer: compliance officers, internal audit, operations, and eventually a regulator.

The five obligations of the outer loop

When I wrote the original whitepaper, I kept cutting the list until only the obligations that survived contact with real reviews remained. Five did.

1. An audit trail that exists without custom logging

The test is simple: can you reconstruct what happened without any developer having remembered to log it? If the answer depends on logger.info() calls sprinkled through agent code, you do not have an audit trail. You have a diary, kept by volunteers.

An auditor’s questions are specific. Who initiated this case? What did the system know at each step? Which version of which policy was evaluated, with what inputs? Who approved the action, and under what authority? Application logs answer none of these reliably. They are sampled, rotated on retention schedules measured in days, mutated by pipeline config, and shaped by whatever a developer thought was interesting at 4 p.m. on a Thursday. Reconstructing a case from them is archaeology, and archaeology is what you do when the civilization failed to keep records.

A control plane inverts this. When every step of a process is a state transition in an engine, the history is a byproduct of execution. The trail cannot drift from reality because the trail is how reality was executed. This is the single strongest argument for the pattern, and it costs nothing extra at development time. You get it by putting the work inside the thing that records, instead of asking the thing that works to also remember to write.

2. Human-in-the-loop with identity and authorization

Every framework now ships a pause primitive. LangGraph has interrupt(). Others have equivalents. A pause is not an approval. An approval is a pause plus an authorization model, and the authorization model is the hard part.

Real institutional approval means: a defined set of people who may approve this specific action, resolved from a directory, not hardcoded. Four-eyes (maker-checker) where the person who initiated the case cannot be the person who approves it. Amount-based routing, where a $2,000 exception goes to a team lead and a $200,000 exception goes to a named officer. Delegation when the approver is on vacation, and escalation when nobody acts. And an immutable record of exactly who clicked, when, on which version of the case data.

None of that is a framework’s job, and frameworks correctly do not attempt it. But it has to be someone’s job. The dangerous middle state is a pause primitive wired to whatever channel was convenient, which is how approvals end up as Slack messages. I will come back to that failure below, because I have watched it happen.

3. Time as a first-class citizen

Agent frameworks think in turns. Institutions think in days. A dispute case waits eleven days for a merchant response. A loan exception sits with an underwriter over a long weekend. A KYC refresh has a regulatory clock attached to it. In the life of one of these cases, the context window is long gone, the pod has restarted twice, the model behind the agent got a version bump, and the application deployed four times. The outer loop has to survive all of that without anyone noticing.

That means durable timers, not setTimeout() and not a cron job reading a table you wrote on a Friday. It means SLAs modeled as data with escalation chains attached: if the approval is not granted in 48 hours, notify the manager; at 72, reroute. It means a case can sleep for three weeks at zero cost and wake up exactly where it was. Durable execution engines and BPMN engines both do this natively, because both were built by people who got paged when the homemade version broke. Frameworks are getting better at persistence, but persistence for resumption is a smaller promise than time as a modeling primitive with escalation semantics.

4. Policy as data, owned by compliance

Here is the line I have repeated in more meetings than any other: the budget check does not live inside the model.

If your spending limit is a sentence in a system prompt (“never approve refunds above $500”), you do not have a control. You have a request, made politely, to a stochastic system, in a channel that adversarial input also writes to. Prompt injection is the dramatic failure, but the mundane failure is worse: even when the model complies every single time, you cannot prove it. “We told the model” is not a sentence you want to say to internal audit.

Review comment: a policy you cannot show being evaluated is indistinguishable from a policy you do not have. Auditors figured this out long before LLMs existed.

The pattern is policy as data, evaluated outside the model, before or after the agent acts but never inside it. Decision tables (DMN if you are in the BPMN world, but a rules service or even a well-tested policy module also qualifies) that compliance can read, version, and change without a code deploy. The agent proposes: “refund $650, here is my reasoning.” The policy disposes: amount over threshold, route to human approval. The division of labor is clean. The model provides judgment. The table provides limits. When the limit changes, compliance edits a row, the change is versioned, and no prompt engineer is involved.

5. Operability

The least glamorous obligation, and the one that decides whether the system survives year two. Four capabilities, all boring, all essential.

Pause and resume without losing context: not just the agent’s memory, the whole case, including its timers and pending approvals. Migration of in-flight instances: you deployed version 3 of the process, but 4,000 cases are still mid-flight on version 1, and some of them will run for weeks; you need tooling to map old state onto new definitions or to let versions drain side by side, deliberately. Atomic cancellation: when a case must die, everything dies with it, compensating actions fire for the steps that already committed, and no orphaned agent keeps calling tools into the void. And one place to investigate: when operations asks “where is case 8842 and why is it stuck,” the answer should be one screen, not a grep across four services and a Slack thread.

Frameworks have almost nothing to say about any of this, and that is fine. It is not their concern. It has to be yours.

The control plane is a role, not a product

Everything above describes a role. At least three families of technology can fill it, and I have seen all three work. The honest comparison matters more than the category my employer happens to be in.

Option A: a BPMN engine

Camunda, Flowable, and their relatives. The strengths map almost one-to-one onto the five obligations, because workflow engines were built for exactly this class of problem two decades before agents existed. The execution history is native and complete. Human tasks with identity, candidate groups, and four-eyes are built in. Timers and escalations are model elements you can point at. Policy externalization has a first-class answer in decision tables. Instance migration tooling exists because long-running processes made it necessary. And the process model is readable by non-engineers, which sounds soft until the day compliance signs off on the actual executable artifact instead of a Confluence page that drifted from it.

Two more properties matter at institutional size, and they are the reason this category keeps winning the regulated-industry evaluations I sit in. First, scale here is proven rather than projected: engines in this family already run core processes at some of the world’s largest banks, insurers, and retailers, and the current generation was re-architected precisely for horizontally scaled, high-throughput workloads, which means the agent traffic you are planning for is smaller than the payment traffic these engines already carry. Second, the governance surface is a product, not a project: operations tooling for investigating a stuck instance, replaying an incident, or migrating ten thousand in-flight cases exists off the shelf, with a vendor on the hook for it. When an agent misbehaves at 2am, the difference between “open the operations console” and “grep the event store” is the difference between an incident and an outage.

The costs are real. You run an engine, and engines have operational weight. BPMN has a learning curve, and some developers have an allergic reaction to anything that looks like diagrams-as-code. Modeling discipline is required; a sloppy BPMN model rots exactly like sloppy code, just more visibly.

Option B: a durable execution runtime

Temporal, Restate, and the cloud-native equivalents. These are excellent pieces of engineering with the strongest durability semantics in the industry: event-sourced histories, deterministic replay, timers and multi-week sleeps as first-class primitives, and versioning APIs for evolving workflows. If your organization is code-first and allergic to visual models, this is a legitimate way to fill the role, and teams I respect have done it.

The gaps are institutional rather than technical. The event history exists to make replay correct, not to make an auditor’s life easy; it is developer-shaped, and a compliance officer cannot read a Go workflow. Human task management with directory-resolved identity, four-eyes, and delegation is yours to build. Policy externalization is yours to build (most teams bolt on OPA or a rules service, which works but adds an integration you now govern too). You can absolutely construct the outer loop on Temporal. Just budget for the fact that you are constructing it, not configuring it.

Option C: a disciplined homegrown state machine

Postgres, a state column, workers, an outbox table. Do not let anyone tell you this is always wrong. For one process with modest volume and a stable team, it is often the correct first step, and I have seen it done well. No new infrastructure, total control, every line understood.

The trade-off is that the discipline is load-bearing and the discipline decays. You will re-implement durable timers, escalation, and versioning, each about 80% correctly. The audit trail is only as good as your weakest code reviewer. And the killer is in-flight migration: the failure mode is not day one, it is month eighteen, when version 3 of the process meets 4,000 instances still running version 1 and there is no tooling, only a very long incident channel.

Obligation	BPMN engine	Durable execution	Homegrown
Audit trail without custom logging	Native	Partial (replay history, dev-shaped)	Build it
Human tasks with identity, four-eyes	Native	Build it	Build it
Durable timers, SLAs, escalation	Native	Native (escalation: build it)	Build it
Policy as data, non-dev editable	Native (decision tables)	Bolt on (OPA, rules service)	Build it
In-flight instance migration	Tooling exists	APIs exist, effort required	The month-18 incident
Readable by compliance	Yes	No	No
Operational and learning cost	Engine + BPMN curve	Cluster + determinism rules	Lowest to start, compounds

Pick based on who your organization is, not on this table alone. But pick deliberately, because the default (the outer loop nowhere, smeared across application code) is the one option that always fails.

The integration pattern: orchestrate, don’t replace

The bank’s fear, reasonable on its face, was that adding a control plane meant rewriting their agents. It does not, and any architecture that requires it is wrong. The pattern that works keeps every agent in its native framework and puts the process model above them.

The process is the agent, not a call to one on the side.

That sentence is the whole design. From the institution’s point of view, the unit that matters is the end-to-end process: resolve the dispute, open the account, examine the letter of credit. That process is what gets audited, what carries the SLA, what a regulator asks about. The LLM loop is one activity inside it, not the container around it. Invert that (application code with an agent at the center, calling out to a workflow occasionally) and every outer-loop obligation lands back in application code, which is where this guide started.

Concretely, for a dispute case:

1. The control plane starts a case instance with a business key and correlation ID. From this moment there is a history.
2. A deterministic step gathers account data. No agent. It is a lookup.
3. The control plane invokes the triage agent as a task. The agent lives in its own runtime (LangGraph, CrewAI, whatever the team chose) and is reached over A2A as a peer agent, or its capabilities are exposed as tools over MCP. The inner loop runs entirely inside the framework: reasoning, tool calls, memory. The control plane sees a task in progress, then an artifact coming back.
4. The agent’s proposal lands in process variables. A decision table evaluates it: amount, customer tier, fraud score. Below threshold, auto-proceed. Above it, a human task is created for the right approver group, with four-eyes enforced and a 48-hour escalation timer armed.
5. Approved, the execution step runs. Every transition, including the agent’s task and the approver’s identity, is already in the history. Nobody logged anything.

The protocol boundary is what makes this framework-agnostic. A2A gives the control plane a standard way to hand a task to an agent it does not host and receive artifacts back, with the agent’s team keeping full ownership of their stack; I wrote up the mechanics, card discovery, and failure modes in the A2A field guide. MCP covers the tool-shaped cases. Teams keep their frameworks, their evals, their deploy pipelines. The institution gets one spine.

This pattern is also where agents pay for themselves. In one trade finance proof of concept, moving document examination into an agent step took letter-of-credit examination from hours to seconds. The agent produced that number. The control plane is what made the number usable in production: every examined document left an audit event, every exception routed to a person with authority, and the two-week cases survived every deploy in between.

When you don’t need a control plane

This section is why you should trust the rest. A control plane is not free. You are adding an engine to operate, a model or workflow artifact to version, latency at every boundary, and a second system your on-call rotation has to understand. Do not pay that for a chatbot.

You do not need a control plane when a human reviews 100% of the output before anything happens (a drafting assistant is not an autonomous system, whatever the pitch deck says). You do not need one for internal developer tooling, prototypes, or anything whose worst failure is a bad paragraph. You do not need one for single-shot generation with no side effects. In all of these, the framework alone, or no framework at all, is the right amount of architecture.

One step earlier, ask whether you need an agent in the first place. My heuristic, from a piece I wrote on open banking, is to count the tools and the deviation in the path: “when you have that level of deviation where you can use three tools versus 10 tools, that’s where agentic AI is a better fit.” Three tools and a predictable path is a program. Write code, or model a deterministic workflow, and get determinism for free. Ten tools, high variance in what each case needs, judgment about which to use in what order: that is where the inner loop earns its tokens. I have watched teams deploy an agent, with all its evaluation burden, on a flow that was three if statements wearing a trench coat.

The control plane becomes necessary when the outer-loop obligations stop being theoretical. Money or commitments move without a human touching each one. Work outlives a context window and a deploy. A regulator, or your own audit function, can ask “why did the system do this” and expect an answer. More than one team’s agents have to compose into one process. Two or more of those, and the question is no longer whether you have an outer loop, only whether it is deliberate or accidental.

What breaks when the outer loop lives inside the framework

Three failures, generalized from real systems I have reviewed. Details altered, structure intact.

The audit trail that died with the pod

A team built an agent pipeline that assembled its full reasoning trace in memory and wrote one consolidated record at the end of each run. Clean design, easy to read, until a pod was OOM-killed mid-run after the action had already been taken. Result: an executed operation with no record of why. The checkpoint store had the graph state needed to resume, but resumption state is not attestation, and the retention policy on that store had been set by an infra default nobody had read. The fix was not better logging. It was moving each step’s completion into an engine where recording is not optional, so a crash can lose the future but never the past.

The approval that was a Slack emoji

Human-in-the-loop, implemented as a message to a channel; a thumbs-up from anyone in the channel resumed the graph. It demoed beautifully. Then the questions: who exactly approved case 4417? Was that person authorized at that amount? Was the message edited after the reaction? What happens when the approver leaves and their account is deactivated? And the one that ended the design review: the workspace retention policy deleted messages after 90 days, which meant the approval evidence had a shorter lifespan than the loans it approved. Approval is an authorization event with an identity attached. A reaction emoji is neither.

The budget check that lived in the system prompt

A refund agent with its limit expressed as a system-prompt instruction. A red-team exercise got past it inside a day with an unremarkable injection. But the finding that mattered was the counterfactual: even if the model had held the line every time, the institution could not demonstrate enforcement. There was no artifact to show, no evaluation log, no versioned policy. Moving the limit into a decision table evaluated after the agent’s proposal turned an unprovable behavior into a control with evidence. The prompt still mentions the limit, as a courtesy to the model. The table is what enforces it.

Review comment: every one of these systems worked in the demo. The outer loop is invisible in demos, because demos do not crash, do not get audited, and do not run for three weeks.

What the regulators are actually asking for

None of the documents that matter here says “agent.” They do not need to. Read them as requirements on the outer loop and the mapping is direct. This is a measured reading, not a scare; most of these expectations are things a well-run system wants anyway.

OSFI’s updated Guideline E-23 on model risk management (final in September 2025, effective May 1, 2027, for all federally regulated financial institutions in Canada) explicitly pulls AI and machine learning into scope and asks for lifecycle governance: inventory, design, review, deployment, monitoring, and clear accountability, proportional to risk. An agentic system that lives entirely inside application code makes almost every one of those expectations a manual documentation project. A control plane makes most of them a byproduct: the inventory is the set of deployed process and decision definitions, the lifecycle is the versioned artifact history, the monitoring is the engine’s own operational data.

The EU AI Act puts creditworthiness assessment of natural persons in Annex III, which makes a credit-decisioning agent a high-risk system; those obligations became enforceable on August 2, 2026. Article 12 requires automatically generated event logs, which is obligation one of this guide in regulatory language (“automatic” is the operative word; volunteer logging does not qualify). Article 14 requires effective human oversight, which is obligation two, including the part where oversight means a person with authority and the ability to intervene, not a person cc’d on a channel.

SR 11-7, the Federal Reserve’s model risk guidance, has been the baseline for US banks since 2011, and its definition of a model (a quantitative approach that processes inputs into estimates, applied to decisions) comfortably covers an LLM-driven agent. Its core demand is effective challenge: independent validation, documentation, and inventories. You cannot effectively challenge a system whose decision path exists only in application logs and prompt files. You can challenge a process model, a decision table, and an execution history, because they are inspectable objects with versions.

The pattern across all three: regulators are converging on lifecycle governance, automatic records, and accountable human oversight. Those are exactly the things the inner loop cannot provide and the outer loop exists to provide. Institutions that separate the loops will find compliance mostly falls out of the architecture. Institutions that do not will run a parallel paperwork process that describes a system it cannot actually see.

The one-paragraph version

Agent frameworks (LangGraph, CrewAI, the OpenAI Agents SDK) are excellent at the inner loop: reasoning, tool selection, prompt assembly, memory. They are structurally unsuited to owning the outer loop: the audit trail that exists without custom logging, human approval with identity and four-eyes, time as a first-class citizen, policy as data owned by compliance, and the operability of long-running work. The answer is not rip-and-replace; it is a control plane that orchestrates agents in whatever framework they were built, reached over A2A or MCP, with the process model as the governed artifact. The process is the agent, not a call to one on the side. Fill the role with a BPMN engine, a durable execution runtime, or a disciplined homegrown state machine, with eyes open about what each makes native and what each makes your problem. Skip all of it for prototypes and human-reviewed tools, and skip the agent entirely when three tools and a predictable path mean you should just write code. Adopt it before the first time an auditor, a regulator, or your own incident review asks a question your logs cannot answer.

References

• OSFI, Guideline E-23: Model Risk Management (2027), final September 2025, effective May 1, 2027.
• EU AI Act, Annex III: high-risk AI systems and Article 12: record-keeping; Annex III obligations enforceable from August 2, 2026.
• Federal Reserve, SR 11-7: Guidance on Model Risk Management (2011).
• Camunda blog, Agentic AI Puts Open Banking to Work, source of the three-tools-versus-ten heuristic quoted above.
• Camunda blog, Using A2A to Achieve Your Business Goals, part 1 and part 2, my co-authored writeup of the agent-to-agent integration pattern.
• A2A protocol specification and Model Context Protocol, the two interop layers referenced in the integration pattern.
• LangGraph documentation (persistence and interrupt()), OpenAI Agents SDK, and Temporal documentation, for the inner-loop and durable-execution capabilities discussed.
• Related on this site: the A2A field guide.