The AI demo works. I still have questions before we ship.

A good AI demo tells us a feature is possible. The feature on screen is only part of what a client is buying, because they also need to know who can access the data, what the system is allowed to do, and how the business keeps operating when it fails.

Seeing a system read a document and answer a difficult question makes the opportunity tangible. I like that moment. Then I want to try the question it cannot answer. I want to sign in as a different customer. I want to remove a source document, interrupt a tool call, and see whether the interface admits that something went wrong.

That is the point where a prototype starts becoming a product.

For CodeAspen, those questions belong in the scope from the beginning. They affect the architecture and the cost too much to leave until launch week.

A promise we can test beats a headline we cannot

Take a hypothetical internal document assistant. “Ask anything about your business” is an appealing headline, but I would not use it as a specification.

A more useful promise is that employees can ask questions about an approved collection of procedures, see supporting passages, and send unresolved questions to a responsible person.

That tells us which documents matter and what a successful answer looks like. It also tells us what the assistant cannot do. It cannot invent policy, make a technical diagnosis, or approve an exception simply because the user asks confidently.

Before discussing models, I want a named audience, a data owner, supported tasks, excluded actions, and someone responsible for escalation.

An information tool and an action-taking system need different controls. Giving the assistant a button that writes to another system changes the project.

The cases that could delay a launch are the ones worth testing

A question the demo already answers well is useful for explanation. It is not much of a release test.

For the document assistant, I would include questions with missing answers, contradictory sources, old documents, ambiguous terminology, and information the user is not allowed to access. I would also include ordinary questions written badly. People do not stop being rushed because the interface uses AI.

Expected behavior should be written down before judging the answer. Sometimes the right result is a direct response. Sometimes it is a clarification, a conflict warning, or an admission that the sources do not support an answer.

I want retrieval and answer quality evaluated separately. If the system found the wrong document, a more polished response is not progress.

Keep consequential failures visible. An average quality score should never hide cross-customer disclosure. Version the model configuration, prompts, retrieval settings, and document-processing changes so regressions can be traced and reproduced.

A prompt is not an access-control system

The application needs to know who the user is and what they can read before passing documents to the model.

Telling the model “only use this customer’s information” is not a substitute for enforcing that boundary in retrieval and storage. The same rules need to apply when someone opens a source link, downloads an export, or triggers a background task.

Caches and diagnostic logs deserve attention here. A feature can enforce permissions on its main screen while leaking information through a less obvious path.

For a multi-customer product, I want deliberate attempts to cross those boundaries using distinct test organizations. Search is one route to test. Direct record identifiers and copied source links are others.

If document-level permissions make the first release too complex, narrow the initial product. A single approved staff collection can still be useful. The limitation should be explicit rather than hidden behind an optimistic permission prompt.

Assume a document can contain hostile instructions

An uploaded document might contain text telling the assistant to ignore its instructions, expose information, or call a tool. A webpage or email can do the same.

OWASP’s prompt-injection guidance describes these direct and indirect attacks and recommends layered defenses. I would not treat a suspicious-phrase filter as proof that the problem is solved.

My next question is what the system could do if the model followed the wrong instruction.

A read-only tool with a small authorized document set has a different exposure from an assistant that can browse arbitrary destinations, edit customer records, and send email.

Restrict tools, validate their inputs, and enforce authorization on the server. Keep secrets out of model-visible content. Require trusted application approval for consequential actions. A line in a retrieved document saying “the user approved this” is still just document content.

Test whether the application contains attempted misuse, not just whether the model produces a reassuring refusal.

Review needs enough context to be worth doing

I do not want to put an Approve button on a screen and call the oversight problem solved.

The reviewer should see the proposed action, its target, the evidence, and the unresolved issue. For an outgoing message, show the recipient and exact content. For a record change, show the values before and after.

Approval should apply to that specific action. If the target or content changes, the previous approval should not silently carry over.

There is also a staffing question. Who checks the queue, and when? If the answer is “someone in operations,” that needs to become more specific before the product depends on it.

Review can become careless when people are asked to approve the same obvious action repeatedly. I would measure correction patterns and observe how the screen is used. The interface needs to make meaningful review practical.

This is where AI ethics becomes product work

I am more interested in the decisions a user experiences than a broad claim that a system is ethical.

Does it identify generated content clearly? Can someone inspect the source? Can they correct a mistake or reach a person? Does the visual design make a tentative answer look like approved policy?

Testing should include the language and accessibility needs of the actual audience. If the tool only works for carefully phrased questions, it may fail the people who need it most during a busy shift.

Data practices also need specifics. Decide what is collected, where it goes, how long it stays, and who can inspect it. That includes source files, extracted text, model requests, and logs. Verify the provider configuration and contractual requirements before making privacy promises.

None of these decisions should be delegated to a vague “AI policy.” Give them owners and make their effects visible in the product.

What happens on a bad Friday?

Suppose the model provider becomes unavailable while employees are working. Can they still reach the original documents? Does the interface explain the outage? Who gets notified?

Suppose costs suddenly rise. Can we trace the requests and limit execution? Suppose a document is replaced. How do we know the search index stopped serving the old version?

Operational monitoring should answer those questions. Request identifiers, deployed configuration, validation outcomes, tool activity, latency, and disposition are useful. Sensitive content in logs needs deliberate access and retention controls.

For write actions, uncertain results need reconciliation. A timeout after an update may mean the update succeeded. Repeating it without checking can make the incident worse.

Rollback cannot undo everything

Restoring an earlier prompt changes future behavior. It does not unsend a message or reverse a decision someone made after reading a wrong answer.

I separate restoring the application from repairing its effects. We need enough history to identify affected records, assign follow-up, and help the user. Prefer reversible actions where the task allows them.

Before a pilot, I want someone to demonstrate how to disable the feature and continue the work manually. Recovery instructions that nobody has tried are still assumptions.

Recovery instructions that nobody has tried are still assumptions.

My release questions are fairly plain: did the held-out evaluation pass the agreed criteria, did access tests hold, are consequential actions controlled, and can the team operate the fallback?

The acceptable standard depends on what a mistake can do. Suggesting an internal search term and sending a customer commitment do not deserve the same threshold.

A working demo gives us a reason to continue. I am ready to put it in customers’ hands when we can also explain its limits, detect the failures that matter, and take care of the person affected by one.