Two years ago a voice agent was a phone tree with better manners. Today the good ones listen and talk at the same time, handle interruptions, hold a dialect, and finish real tasks inside your systems while the caller is still on the line. That change is real, and it is why so many teams are now asking whether they should put one on their own lines.
The answer is often yes. But the gap between a convincing demo and a line you would put your customers on is still wide, and it is not mostly about the model. This is the list we walk through with every client before we agree a scope.
What changed
Three things, and each of them matters on a phone line specifically.
- Speech-to-speech models. The agent no longer has to wait for a full transcript, think, then speak. It can respond while you are still finishing the sentence, which is the difference between a conversation and a queue.
- Interruption handling. A caller can talk over the agent and the agent stops, listens, and picks up correctly. This used to be the single most common way for a call to fall apart.
- Dialects and accents. With models trained on the right recordings, Gulf Arabic, Indian English and a Texan drawl on a bad mobile connection are workable. Not solved. Workable.
What a voice agent can do well
The pattern is the same as everywhere else in AI: bounded conversations with a clear end state, connected to the systems that hold the truth.
- Appointments. Book, move, confirm and cancel, against the real calendar, with the rules about buffers and no-shows built in.
- Verification and status. Confirm identity against the fields you allow, then read back an order, a claim, a delivery, a balance.
- Collections and reminders. Outbound calls that say the right thing, log the outcome, and hand over the moment the conversation turns into a negotiation.
- First-line questions with a knowledge base. Opening hours, policies, "do you do X", answered from your documents rather than from the model's imagination.
- Hand-off. Knowing when to stop, warm-transferring to a person with a summary, and never leaving a caller in the dark.
What it still cannot do
We are direct about this because the failures are expensive and public.
- Distress. A caller who is frightened, grieving or furious needs a person. An agent can recognise it and transfer fast. It should not attempt the conversation.
- Judgement calls the business has not written down. If your team argues about a refund rule in the kitchen, the agent will argue with the customer about it on the line.
- Negotiation. Anything where the outcome depends on reading the other party is not a job for an agent yet.
- Anything that needs a decision above its pay grade. An agent can gather everything a manager needs. It should not pretend to be the manager.
The part nobody talks about: the line itself
Most of the disappointing voice agents we are asked to look at were built on clean studio audio and deployed on real telephony. A phone call is narrow-band, compressed, and often noisy. The caller is in a car, on speaker, in a market. A model that scored beautifully in a browser demo can fall apart on a G.711 line from a warehouse.
This is why we train speech models on the actual line quality and, where we can, on the client's own recordings. It is also why we run our own numbers and trunks: when something sounds wrong, we want to be able to look at the whole path, not open a ticket with a carrier.
The hand-off is the product. A voice agent that knows exactly when to stop is worth more than one that can talk about anything.
What to measure
Pick these before the pilot, and put today's values next to them.
| Measure | What it tells you |
|---|---|
| Containment | Calls finished without a transfer, for the tasks in scope |
| Transfer quality | Did the person receiving the call get a usable summary |
| Task completion | The booking exists, the record changed, the payment was logged |
| Abandonment | Callers who hung up mid-conversation, and where |
| Silence and talk-over | Long pauses and interruptions per call, from the audio logs |
Containment on its own is a trap. An agent can contain a call by exhausting the caller. Track it next to completion and abandonment or it will lie to you.
A short checklist before you go live
- Every task in scope has an end state in a real system.
- The transfer path is tested with the people who will receive the calls.
- The agent has been listened to on the worst line you have, not the best.
- Someone reviews a sample of calls every day for the first month.
- The number you are trying to move is written at the top of the page.
If those five are true, a voice agent on your line is a normal engineering project, with normal risks. If they are not, no model will save it.