Hub / Guides / Pilot to production
Pilot to production
Take your agent pilot to production
Yes. Your pilot can carry real work once it has an owner, a review gate, audit, and a budget, and we set those up and keep running them after launch. BMC frames the shift as pilots to governed production; the steps here are the vendor-neutral version of that path.
Why do agent pilots stall?
A pilot proves the model can do the task. It doesn't prove anyone can run it: who gets paged, who approves output, what it costs at volume. BMC names "fragmented AI pilots" as the enterprise pain. The stall is rarely the model; it's the missing operating layer.
What does governed production mean?
Five things are true and written down:
- An owner — a named person per agent. The job around that owner is AI fleet ops.
- A review gate — output is checked before it merges, ships, or reaches a customer. Human approval sits where risk is highest (Runflow).
- Audit — every run and approval is logged (IBM, BMC, Runflow).
- Identity and budget — own credentials, metered spend (BMC, Microsoft). See spend and policy.
- Evaluation — quality re-checked on a schedule (IBM, Microsoft).
What are the steps from pilot to production?
- Week 0 — Scout. Inventory the pilot: inputs, tools, keys, cost per run, failure cases.
- Week 1 — Stand up. Give it an owner and one queue. Every task gets a done-check. The house path for that queue is standing up an agent team.
- Add the gate. Pick an independent reviewer (person or agent) who passes or fails work. In 0xRay this is the Strict gate — use any name.
- Split identity. Move the agent onto its own credentials and set a spend alert.
- Turn on traces and audit. Record tool calls and approvals.
- First gated ship. One real piece of work goes through the full path, with evidence.
- Retain. Weekly evaluation and a short report: shipped, failed, spent.
Who runs the agent after launch?
This is the question pilots skip. Someone owns the queue, reviews failures, and decides when to add the next agent. That can be your team, or a retained operator; AI Fleet OPS offers Retain for that reason.
What fails on the way to production?
- Scaling the pilot before the gate exists.
- One shared key for every agent.
- Announcing early. Public claims should follow the first gated ship, not the demo.
- No exit for loops. An agent repeating tool calls needs a rule that stops it (Snowflake). That rule lives on the control plane.
What does a gated first ship look like?
Take a pilot agent that drafts replies to customer emails. In the demo it did well on twenty hand-picked messages. Here is how the first real ship runs.
The eng lead moves the agent onto its own credentials and limits it to the inbox folder it needs. They set a spend alert at a level the owner agreed to, and turn on traces so each tool call and model call is recorded.
The owner writes the rule for the gate: replies about billing or refunds need human approval; general questions can go out after review by an independent reviewer. The owner also names the measure of success — for example, how many drafts the reviewer sends with no edits.
The reviewer takes the first batch of real drafts. Some pass. Some fail because the agent guessed at a policy it didn't have. Each fail goes back with a note, and the eng lead adds the missing policy text to the agent's context.
After the batch, the team has evidence: what shipped, what failed, what it cost per finished reply, and who approved each one. That evidence is what makes the next step — more volume, or a second agent — a decision instead of a guess.
How do I know the pilot is ready for more volume?
- The gate has run on real work, not only demo inputs, and the pass rate holds over at least a few cycles.
- Spend per finished task is known and within what the owner agreed to.
- Failures have a path. Every failed item has an owner and a next step, and the same failure doesn't repeat for the same reason.
- The audit trail answers questions on its own. Someone outside the pilot team can see who approved what without asking.
- Someone owns it after launch. There is a named operator for the queue next month, not only this week.
If any of these is missing, hold volume where it is and fix that item first. Adding volume to a pilot that hasn't passed these checks spreads the same problems across more work.
How do you request a consult?
Scout, Stand-up, or Retain.
What else do operators ask?
How long does it take?
It depends on the stack; the steps above are a week-one shape, not a promise.
Do I need new tools?
Not always. Many gaps are rules and owners, not software.
Can an agent be the reviewer?
Yes, if it is independent from the agent that did the work and a human owns the gate.
What if we have several pilots?
Run one through first, then reuse the path.