Most teams already know how to ship software: write code, test it, put it live, watch it break, fix it. Agents need that same loop - plus a few habits ordinary unit tests do not cover. Agentic Operation (AO) is the name we give those habits: how an organization keeps agents secure, deployed, and available, from one agent up to a whole population of them.
This page is the plain-language sketch of Agentic Operation. It is not the full Agentic Operation Manual (AOM), not a scored assessment of any company, and not a product pitch. For how we run evaluation work in practice, see Evaluations and the interactive demo figures. For how the same grammar lands on healthcare workloads, see the Harborlight method series that starts at the device-ladder anatomy. Engagement shape lives on Services and Consultants.
What Agentic Operation is
Think of three columns that have to line up:
- People - who builds, who judges the work, who decides what gets shared.
- Lifecycle - the cycle itself: build → test → deploy → monitor → iterate.
- Stack - the tools and runtimes that cycle actually configures.
Under those columns sits a fourth row that never becomes a sixth lifecycle step: governance, security and compliance, cost awareness (FinOps), and the freedom to swap a host or a harness when the market moves. AO is the operating model that keeps the columns and that row talking to each other so agents improve on the runs they make, instead of failing quietly in production.
A unit test still asserts code. An eval (a dataset plus checks) is what scores whether the answer was any good. That distinction is the whole reason AO exists. The eval is not a one-time gate at the end; it drives the run and comes back around after deploy:
Seats: who does what
People sit on a spectrum, not in a single org-chart box:
- AI engineers build and run the shared stack (and often sit on the team that owns a given agent).
- Department leads coordinate which lessons leave a team and become shared standards.
- Team members do the real work and supply most of what “good” looks like for evals.
Early on, the same few people wear every hat. Later, a hub keeps the shared stack while most evals still stay with the team that did the work. A job title on a website is not evidence that a seat is staffed. A blank stays a blank until someone is actually doing the job.
Empty seats stay empty
Some capabilities are named in the model but not filled on our stack yet - for example a productized spend ledger, an automatic code-fix loop, or a one-click human-approval pause mid-run. We call those empty seats.
The honesty rule is simple: name the seat, leave it blank, and do not invent a number or a checkbox that looks like progress. Filling an empty seat with a rival manual’s diagram, a vendor slide, or a guess is how assessments go soft. Hollow is hollow.
Naming an empty seat is not a product claim and not a roadmap promise. It puts the capability in the shared vocabulary so a buyer or operator can ask for it; it does not mean we ship it, sell it, or have a date to fill it. If that seat must be filled, that is scoped work inside your governed environment - not something these pages advertise as ready.
Estimate vs scored
A typed monthly cost, a vendor claim, or “as good as model X” is a criterion or an estimate. It is not a scored run.
A scored result needs a real eval: stored or live runs, checks that return a score, and (for cost) usage read from the trace. An estimate stays marked as an estimate until measurement replaces it. Writing a nice sentence in a deck does not create that measurement. That same split shows up in how we talk about evaluation engagements: demo figures on /evals/demo teach the shape; a measured result only appears when your runs and your checks exist inside your governed environment.
Operating tenets
The bottom row under every phase is four concerns. They are not four extra one-time spreadsheets. Secure, then deployed, then available, and the cost of a run assessed again, are the service properties. A mark is enforced where an object exists, and it is named where the seat is empty.
Governance
Governance is who owns the workload, who may change a source, and who accepts an estimate becoming a measurement.
- On a run, that is usually the engineer on the team.
- The department lead coordinates which eval leaves the team and becomes shared.
- A standard that binds every team goes to a small committee on the hub.
A form that collects a typed cost, a vendor name, and a sensitivity label is not the practice. Those fields do not staff the seat of who accepts a figure, and they do not create a trace or an eval. A shared floor is what every team shares; a stage label on a slide does not mean a team relies on the run.
Security and compliance
Security and compliance stays with the people who already hold that duty. Class the agent by what it is allowed to do:
| Class | The work | What you can enforce today |
|---|---|---|
| Informs | Answers from trusted material, or prepares a draft a person sends. | The eval scores the answer before the run is trusted. The trace is the record. A standing filter on what enters and leaves the model call is still an empty seat. |
| Writes or sends | Changes a record people use, or sends something a customer will see. | The toolset and the sandbox limit what can be written. A timeout and a token cap bound the run. A mid-run pause for a person to approve the write is an empty seat. |
| Continues, or stakes are high | Long runs, regulated actions, hand-offs between agents. | Hostile tasks before deploy, code stays in the sandbox, a named owner on the team. Grouping those failures, and a compliance packet that assembles the evidence, are empty seats. |
Ordinary software already stops a hostile query and a retry that never ends. An agent adds failures those controls do not score: a next step that trusts a wrong one, a drifted goal, data leaving the machine, a bad tool call, a prompt that steers the next step. Those land on the trace and the eval - or on the sandbox network policy - when the object exists. Where the place to enforce a commitment is empty, say so; do not pretend a principle is a control.
FinOps
FinOps is the cost of the run: tokens, usage, and the host. This section names the concern. It does not price it. No amount is stated.
The first cost choice is the host: a public API or a rented GPU (hosted - the model call leaves the machine) versus own-the-box (you operate the hardware). Tokens are consumed either way. The trace records usage either way. Hosted versus own-the-box changes where the call goes and who operates the machine; it does not, by itself, create a ledger.
What this stack can show today is usage on the model calls inside a trace, plus a token cap and a timeout on a rollout. What stays empty: a productized spend ledger, an alert on money, and a hard block at a money limit. Cost per interaction belongs next to whether the answer was right - a cost overrun is a failure mode of the run, in the same family as a plausible wrong answer. You see it by reading usage on that interaction’s model calls, not on a ledger this tree does not have.
Three levers, said plainly: route the right model to the right step (a judge can use a different model than the agent; an automatic cheap-route is empty), cache repeated work when you can (a dedicated skip-cache is empty), and bound what usage is allowed at all (caps and timeouts yes; a money ledger no).
Interoperability
Interoperability is the freedom to swap a host, a harness, or a trace. A stack that looks fast on day one and expensive on day three hundred is a stack you cannot leave. Agents that run only on one runtime, store prompts only in one registry, and emit traces only in one format are agents you cannot move.
What keeps the columns swappable:
- People - portable skills (prompt design, evals, asking), not a certificate from one vendor.
- Lifecycle - the cycle itself (build, test, deploy, monitor, iterate), not a second product.
- Stack - another harness program, another model endpoint, tools as MCP servers, a runtime you can move, and a trace you can export.
What stays behind when you cannot leave: the security duty (if it was only a vendor feature), the record (prompts, evals, and traces that cannot be exported), and the next model or tool (you cannot point the same eval somewhere else). Neutrality is also how you gain: a better model can show up on short notice; because the environment is the task outside the loop, you point the run at the new model and keep the eval.
How the practice usually grows
The practice does not arrive as a tool. It arrives when the same people can say who owns the agent, which eval accepts the next version, and which object holds the record. Different efforts sit at different degrees at the same time; the degrees are not a score for the organization.
- One team - one engineer is building and is also the hub. Department leads and team members are not in the work yet.
- A run the team relies on - something has shipped. The eval may still be only that engineer’s. A department lead or team member starts saying what good looks like.
- A check on the live path - a stored trace is scored before the next change, and a live rollout is scored on that path. Cost is read from usage on the model call. The spend ledger is still empty.
- More than one department - the hub is its own seat. Most evals still stay on the team. An eval moves only when the lesson changes the stack, a standard, or a failure other teams will meet.
Place an effort on a degree only when every condition of that degree’s sentence holds. Width of use, a device count, or two committee speeches do not place it. The gap between “we have traces” and “someone on the team agreed to the eval” is usually the work.
Why a written manual matters
Slides age. Chat threads scatter. A short written manual does three useful jobs:
- It gives the organization a shared vocabulary (eval, seat, empty seat, estimate vs score).
- It says what the practice is without pretending every organization already has it.
- It keeps the improvement loop visible: failure → cause → change → eval again.
When you later map a sector’s workloads onto hardware tiers and public evals - as in our healthcare agent-anatomy demo - you are using the same grammar: name the use case, say which eval stands in for it, and leave gaps empty when nothing on disk proxies them. The companion eval figures and system-swarm anatomy push the same honesty into seating and population views.
In this series
This page is the operating-vocabulary stop in the Harborlight method series. Path: device ladder → system swarm → ladder eval figures → this sketch → Evaluations.
Harborlight is also the specimen name on /evals/demo. That page is a different artifact: a proprietary-shaped 12-agent intake-to-discharge suite. The anatomy publications are seating and population charts over public evals. Same fictional health system name; not the same pack.
Learn more
- Evaluations - how an evaluation engagement runs, and what we score end to end.
- Evals demo figures - proprietary 12-agent Harborlight suite demo plots (not the anatomy seating pack; not measured).
- Healthcare agent anatomy · device ladder - series start: use cases, stand-in evals, nested hardware seating.
- Device-ladder eval figures - seating matrix, radar, and cost-against-success on one eval at a time.
- Healthcare system swarm - a population of roles, composite tasks, and who burns the tokens.
- Services · Consultants - how senior consultants join the work, by the hour or under a scoped statement of work.