By Greg Nowak. Updated 23 October 2026.
An AI workflow approves the wrong request, uses an outdated policy or sends an unsuitable instruction to another system. Every API call succeeds. The dashboard stays green. Your operations team still has a customer problem to resolve.
Here is a useful test: pick one poor outcome and ask your team to explain what happened, from the original request to the final action. If that means comparing timestamps across several suppliers’ logs, your AI observability has a gap. You need a connected investigation trail that a business owner and an engineer can review together.
Start with the business decision
Consider a hypothetical refund workflow. It retrieves a policy, asks a model to assess eligibility and calls a payment service. A successful payment response proves that the payment service accepted the instruction. It does not prove that the refund was appropriate.
Before adding instrumentation, define the acceptable outcome: which policy applies, what requires approval and who handles exceptions. Record technical status separately from business review status. A completed run might later be marked incorrect, reversed or awaiting investigation.
The goal is to preserve observable evidence of what the system received, selected and did. A trace cannot expose a model’s private reasoning or prove why it produced a particular answer.
What evidence should an AI workflow keep?
Use this checklist to agree what your team needs before choosing another dashboard. These are proposed investigation fields; some require application instrumentation beyond standard telemetry.
| Stage | Minimum useful evidence | Investigation question |
|---|---|---|
| Trigger | Run ID, trace ID, time, workflow and prompt-template versions | What started this run, under which configuration? |
| Retrieval | Source and document IDs, revisions, filters, records selected for model input | Was the relevant, current information supplied? |
| Model | Provider, requested and returned model identifiers when available, duration, token usage | Which model call actually ran? |
| Tool and approval | Tool name, call ID, validation result, approval reference, retry attempt | What action was permitted and attempted? |
| Destination | External request ID, status, affected-record reference | Was the change accepted and completed? |
| Review | Outcome category, reviewer reference, correction and test result | Was the result acceptable, and did the fix help? |
A document ID alone is insufficient if its contents change. Retain an immutable revision reference and distinguish search results from the material actually passed to the model. Otherwise, you may investigate today’s document instead of yesterday’s evidence.
Connect the steps across team and supplier boundaries
A trace connects timed operations, called spans, across services. Use the W3C Trace Context standard for HTTP propagation through traceparent and, where needed, tracestate. Let your tracing library extract incoming context, create child spans and inject outgoing context. Keep sensitive business data out of these headers.
Test queues, retries and background jobs explicitly. Use parent relationships or span links appropriate to the messaging pattern. Where a supplier cannot continue your trace, record its request or job ID on your side. Keep a workflow run ID to connect later reviews and callbacks, even when they belong to separate traces.
OpenTelemetry’s GenAI conventions describe agent, workflow and tool spans and shared attributes. They remain in development. Pin instrumentation and convention versions, document your custom business fields, and check dashboards when upgrading.
Keep useful evidence without copying every payload
Start with an allowlist of identifiers, versions, categories, counts and statuses. Avoid collecting full prompts, retrieved documents, tool arguments and responses by default. Even identifiers can reveal customer relationships or sensitive activity, so they still need access controls.
If content capture is necessary, define its purpose, scope, storage location, access and deletion period. Where possible, retain protected references to existing records rather than copying their contents into telemetry. Metadata alone will not answer every content question; make that tradeoff explicit.
The OpenTelemetry Collector redaction processor supports attribute allowlists and value masking. Filter before export and verify which fields your configuration covers. Redaction is one control, not a guarantee that arbitrary payloads are safe.
Follow OWASP’s logging guidance on excluding secrets and sensitive information and sanitising untrusted event data. Test with representative tokens and personal data, including error paths.
Do not sample away the case you need
A random sample of successful requests can miss the run a customer disputes tomorrow. OpenTelemetry distinguishes head sampling from tail sampling: the former decides early, while the latter can consider later spans and attributes. A downstream sampler cannot recover evidence already discarded upstream.
Choose retention around when problems become visible. Consider keeping a minimal business outcome record for every consequential action, alongside sampled diagnostic traces. Tail sampling can retain flagged failures, but a complaint arriving days later may fall outside its decision window.
Prove the investigation works before expanding it
- Choose one workflow. Pick an outcome that creates meaningful rework or customer impact.
- Assign ownership. Name the business reviewer and the technical owner across internal teams and agencies.
- Run a controlled failure. Confirm that someone can find the run and explain the sources, approvals, retries and downstream result.
- Test the correction safely. Use a sandbox or stub action tools so replay cannot send messages, issue payments or change production records. Define expected behaviour; identical model wording is unnecessary.
- Make it a regression case. Repeat the check after changes to prompts, models, retrieval or integrations.
Bring one workflow to the review
I can help map the workflow, agree the evidence worth retaining and coordinate implementation across your operations team, developers and suppliers. A focused first deliverable can be a workflow map, an evidence checklist and an investigation walkthrough for one controlled case.
If your team struggles to explain a bad AI decision, talk to me about reviewing one workflow. Bring an example outcome and the systems involved; we can use those to identify the practical gaps.
Related on GrN.dk
- AI automations need a spend dashboard before the first runaway bill
- AI Agents Need a Spending Brake, Not Just a Billing Dashboard
- Before an AI Assistant Issues a Refund, Put Approval in Its Path
Need help with this kind of work?
Review your AI workflow with Greg Get in touch with Greg.
