Before I build an AI agent, I want to see the workflow

If a business asks me for an AI agent, my first question is what it needs done. What authority that system needs, what evidence it can use, and what happens when it is wrong will shape the finished product more than which agent framework we choose.

That is not reluctance to use AI. “Handle our inbox” sounds like one task. It might include reading messages, finding missing information, updating records, deciding prices, and sending replies. I would not give all those actions the same permissions or the same release criteria.

At CodeAspen, the approach I want to defend is straightforward: choose the smallest amount of automation that completes the job reliably, and add complexity when the evidence justifies it.

The first job has to be narrow enough to test

Consider a hypothetical freight broker whose coordinators copy shipment details from emails into a quoting process.

The emails vary. Some contain a clear pickup address and date. Others say “same as last time.” A forwarded thread may include an old request below the current one. An attachment may contradict the message.

I would define the first job narrowly: prepare a quote-request record, show where the information came from, flag missing fields, and wait for staff approval before exporting it.

That gives us something concrete to test. It also keeps pricing and customer commitments outside the first release.

“Prepare the record correctly” and “run the quoting operation” are very different projects.

Three ways to build it, and the simplest deserves the first test

Ordinary automation follows explicit rules. It works well when inputs and decisions are predictable.

An AI-assisted workflow gives a model a bounded interpretation task while the application controls the sequence. An agent lets the model choose some of its next actions and tools.

Anthropic’s explanation of workflows and agents makes this architectural distinction clearly. Its recommendation to start simply is useful even as the available tooling changes.

For our quote-request example:

I would test the first credible option before assuming the third is necessary.

If customers send a consistent spreadsheet, a parser may be sufficient. If the difficult part is reading variable language, a bounded model call may be enough. If the useful next step changes substantially between requests, an agent becomes worth evaluating.

Keep the rules out of the model’s hands

The application should enforce required fields, valid dates, customer access, and permitted status changes.

A model can propose an extracted value. It should not make an invalid value acceptable because its explanation sounds convincing.

For an ambiguous phrase such as “next Friday,” the system needs relevant context or a clarification. For “same as last time,” it needs permission to inspect the appropriate history and a reliable way to identify which record the customer means.

Sometimes the correct output is a blank field with a question attached.

I would also keep the source visible beside the draft. Reviewers need to check the evidence quickly. A polished paragraph summarizing what the model thinks happened can make that harder.

Autonomy needs a reason for each permission

Suppose the bounded workflow cannot resolve enough requests because essential details are spread across messages and prior records.

An agent could investigate within a defined boundary. It might read the current thread, search that customer’s approved history, and assemble a draft. That flexibility may be valuable.

But read access does not imply edit access. Drafting a reply does not imply permission to send it. Finding a previous price does not imply permission to quote that price again.

I would specify the tools, accessible accounts, stopping conditions, and approval requirements individually. I would also cap execution time and tool use so a confused request cannot run indefinitely.

If the agent reaches conflicting evidence, I want it to stop with a useful explanation. I do not want persistence to turn into guessing.

I do not want persistence to turn into guessing.

A useful evaluation has to be able to reject the agent

I would collect appropriately authorized examples from the actual workflow: ordinary requests, revisions, missing fields, duplicates, and unsupported attachments. A domain expert should label the expected values and the cases that need clarification.

Keep development examples separate from the final comparison set. Related email threads should stay together so the test is not accidentally made easier by near-duplicate information.

The sample size depends on variation and the consequences of failure. A small pilot can reveal obvious problems, but it cannot establish that rare, serious errors will never happen.

I would compare the manual process and the simplest credible automation using these measures:

• Records accepted without correction.
• Errors in important fields such as addresses and dates.
• Missing information correctly left unresolved.
• Human review and correction time.
• Duplicate or unauthorized actions.
• Cost per completed, accepted request.

I would inspect failures individually, too. Five correct fields and one dangerous guess can produce an attractive average score and an unusable result.

The review time can change the business case

It is easy to time how quickly a model produces an answer. It takes more care to measure how long someone needs to trust or correct it.

That second number belongs in the economics.

For a purely illustrative calculation, imagine 1,000 monthly requests and a verified reduction of three minutes of handling time per request. That releases 50 hours. At an assumed loaded labor rate of $35 per hour, the theoretical capacity value is $1,750 before operating costs.

Those are example inputs, not CodeAspen results. They also do not establish cash savings. The business may use the time for other work rather than reduce payroll, and there may not be enough demand to use all the released capacity.

Model usage, hosting, review, rework, maintenance, and implementation costs all belong in the comparison. I would divide recurring costs by accepted tasks, not by attempts. An inexpensive attempt that needs substantial repair is not an inexpensive completion.

I do not treat confidence as evidence

A model saying it is highly confident does not establish how often it is correct.

I would rather check a supporting passage, an authoritative customer record, or a validated date. If confidence scores are used for routing, they need calibration against labeled outcomes.

A second model agreeing is also not proof. Both can make the same mistake. A reviewer should see the evidence behind the answer rather than two versions of the same assurance.

Authority should expand one step at a time

First, test historical requests. Then run in shadow mode, preparing drafts without changing live records. Next, let a small team use the drafts with review required.

Observe where staff correct the system, where work gets stuck, and whether the manual fallback remains practical. Expand permissions only when there is a specific reason and enough evidence.

I would be comfortable shipping a narrow workflow that saves a coordinator useful effort while leaving judgment with the team. It does not need to look autonomous to be worth buying.

If you are considering an agent, bring the awkward requests alongside the clean ones. The email with a revised date buried in the thread will tell us more about the right design than a demo built around a perfect input.