AI voice agents made our core product obsolete. We launched a new product alongside it.

XOIP provides phone numbers to small and medium businesses with a simple call-configuration layer: routing, voicemail, office hours, and blocked numbers. The product works because SME owners can set it up themselves, without a telecoms engineer.

AI voice agents change the job. An agent can handle the routing XOIP already offered, then answer questions from a knowledge base, book appointments, look up customer data, and speak naturally in another language. That was previously out of reach for most SMEs.

The question was not whether to build it. It was whether to add it to the existing product or treat it as a new product.

I tested the idea of adding the voice agent to the existing Call Plan as an AI layer. I designed that path to see if it held up. It did not.

The existing product is simple because it limits the choices. SME owners configure a finite set of rules. A voice agent behaves in open-ended ways. Putting that product inside a limited rule editor would make it look like a smarter IVR, which was the wrong promise.

We launched it as a separate product with its own onboarding and pricing, plus a migration path from classic subscriptions. There was pushback because separate products take more work to maintain. A live demo settled the question.

The first design was a conventional form. SME owners filled in fields to configure the agent: cards, inputs, and dropdowns. It worked, but it treated the agent like a settings object.

Asking someone to configure an AI agent through a settings panel is like making a touchscreen that only works with a keyboard. If the agent can answer calls naturally, the owner should be able to instruct it naturally too.

Early testing found a consistent failure. When someone configured a routing rule, the agent asked for some required details and missed others. A request to "Transfer billing questions to the Billing Department" might capture the phone number but omit office hours, after-hours behavior, or the voicemail address.

The fix combined prompt changes with guardrails in the orchestration layer. Each routing rule now has a required-field checklist, and the agent must complete it before confirming the rule.

Manual review of voice sessions does not scale. Every call contains useful evidence, but without a structure for capturing it, the lesson disappears. I built an automated evaluation layer that runs on every session.

It runs in two passes. The first checks required fields and whether the agent confirmed the configuration correctly. The second uses a judge model to review the transcript and decide whether the user achieved their goal, with evidence for the verdict.

The output is a set of prioritized eval cards with specific findings and next steps. The next step is to send those cards to coding agents for remediation. The product should be able to find recurring gaps and help close them without a human reviewing every session.

The chat configuration UI works. The voice interface is operational, the routing, knowledge, and agent views are complete, and the eval system runs on live sessions. Next are the customer launch, pricing validation, and the loop from eval cards to automated fixes.

I would design the completeness guardrails before the first user test. The checklist for each routing rule became obvious while watching people test the system. It should have been part of the prompt design from the start.

I would also invest in the agent persona earlier. Testing showed that SME owners gave more detailed instructions to a named, characterful agent. We added the persona picker in response, but it would have been stronger at the start of onboarding instead of inside settings.