No workflow lives in isolation. A customer complaint arrives. An insurance claim gets submitted. A warranty request comes in. A healthcare case gets opened. The system is expected to do more than generate a response. It has to understand the business context, classify the work, identify the underlying issue, decide whether a human needs to review it, route the work to the correct team, and create a record another system can actually use. Those are business decisions, not language tasks, and when any one of them is wrong, work slows down, customers wait longer, compliance risk goes up, and so does operational cost.
Almost every evaluation I found was still asking whether the response sounded intelligent or looked complete. That is a useful question. It was not the one I needed answered.
Building the tool I wished existed
So instead of building another chatbot, I built a small enterprise AI workbench. One side runs an actual business workflow. The other measures whether that workflow completed successfully. That distinction sounds subtle, but it completely changes what gets measured.
The model became one component inside the workflow instead of the workflow becoming an extension of the model. That single design decision changed how I think about enterprise AI evaluation.
The workbench treats the model as one component inside the process and evaluates the workflow from intake to outcome.
The first experiment
The first workflow I implemented was banking complaint triage. A complaint enters as free-form text. The workflow asks the model to determine the banking product, identify the underlying issue, assess urgency, decide whether the case requires human review, recommend the correct operational queue, and create an auditable case record.
Notice what the workflow never asks the model to do.
It never asks for a beautiful answer.
It asks for operational decisions.
I built a balanced dataset of 400 banking complaint cases across credit cards, debt collection, checking and savings, and credit reporting, each with a known expected outcome. I ran a 10-case smoke test first, then a 100-case sample against Qwen 2.5 7B Instruct, a small model running locally, to make sure I wasn’t reading noise.
The number that looked fine, and the ones that weren’t
Across the 100-case sample, Enterprise score reached 63.8 percent. Schema valid, Fields present, and Case record were all 100 percent. Review match was 80 percent. If I had stopped there, I would have concluded the model was closing in on production readiness for assisted banking triage.
Then I looked underneath those numbers. Product match was 36 percent. Queue match was 30 percent. Issue match, the model’s ability to correctly identify what the complaint was actually about, was 1 percent.
Not low. Effectively zero.
The 10-case smoke test had already hinted at the same pattern. The larger sample just confirmed it.
A blended score can make the workflow look healthy while the operational decisions underneath are failing.
A model can be perfectly formatted and still be wrong about the one thing the workflow exists to determine.
Looking for the weak link
My first assumption was that the model was sounding convincing while quietly making mistakes. That is not what I found when I went back into the individual cases.
It was disciplined about structure. It consistently produced complete operational records. It was surprisingly good at recognizing when a case should be escalated.
The weakness sat somewhere more specific, in the enterprise taxonomy itself: which product a complaint belonged to, what the precise issue was, where the work should actually be routed.
Those are not formatting problems; they are business understanding problems.
One complaint about a debt collector pursuing a debt the customer did not owe came back correctly classified, correctly flagged for review, and correctly routed to the debt collection team with a clean recommendation attached.
The capability is there. The problem is consistency, and that is a different engineering challenge than an incapable model.
The evaluation lab made the weak link visible: structure looked strong, but product, issue, and queue matching exposed the real workflow failure.
The agent studio shows the workflow as an operational system: intake, classification, routing, human review, execution trace, and case record creation.
Two kinds of intelligence
This is the part I hadn’t fully appreciated going in. There are two different kinds of intelligence inside an enterprise workflow.
One is procedural: can the system generate valid structure, produce complete records, and follow the mechanics of the workflow.
The other is operational: can it understand enough business context to classify work correctly, route it to the right team, and make reliable business decisions.
This model handled the procedural side well and struggled with the operational side and improving one does not automatically improve the other. That also changes what actually needs to get fixed.
Not simply a bigger model, but the classification layer itself: better taxonomies, better prompting, targeted fine-tuning where it earns its cost, or a hybrid workflow that leans on deterministic rules for the parts a language model keeps getting wrong.
The lesson, and what’s next
I started this project believing I needed a better way to evaluate language models. I finished this experiment realizing I needed a better way to evaluate workflows. Those are not the same thing.
A blended score can tell you whether the system looks healthy. It rarely tells you where it actually failed.
I’ve written before that the future enterprise won’t be defined by its applications, it will be defined by its ability to generate new capabilities, and that the workflow is becoming the product because the differentiator was never the model sitting inside the stack.
This experiment made me realize the next piece of that argument. If the workflow is becoming the product, the workflow also has to become the unit of evaluation. Measuring answer quality alone is no longer enough.
The remaining banking cases are next, and after that, the same test against other enterprise processes: insurance claims, automotive diagnostics, healthcare administration, customer support, field service. The workflows will change. The evaluation philosophy will not.
I no longer think enterprise AI should be evaluated as a chatbot. It should be evaluated as a workflow participant.
Enterprises do not buy answers.
They buy completed work.
If your evaluation only reports one blended score, would it have told you that issue identification, not case formatting, was the real reason the workflow failed?
#AgenticAI #EnterpriseAI #WorkflowEvaluation
