What I'd Do Differently Building My First Voice Agent

The first mistake I made building a voice agent was assuming the hard part was getting the model to say smart things.

It wasn't.

The hard part was making the conversation feel normal.

If a caller has to wait too long, gets talked over, or cannot reach a human when the call goes sideways, they do not care how elegant your prompt stack is. The product already failed.

On my first pass I overbuilt the plumbing, under-designed the fallback path, and treated latency like something I could clean up later. I would do almost all of that in the reverse order now.

I Started By Solving The Wrong Problem

I spent too much time trying to make the architecture feel future-proof.

Custom wrappers. Provider abstraction. More flexibility than I actually needed on day one. It felt responsible. It also slowed me down and pushed the real problem further away.

What I should have done was trust the library earlier and get a single path working end to end.

Pipecat Flows makes this point pretty clearly: structured conversations work better when each step has one job, the model only sees the tools it needs right now, and conversation state is explicit instead of implied. That is a much better starting point than one giant prompt with a pile of functions hanging off it.

This is especially true in voice, where there is no patience for "pretty good." The caller is not grading your architecture. They are deciding whether the thing feels broken.

If I were doing it again, I would start with one transport, one speech stack, one narrow use case, and the smallest possible call flow that still solves a real problem.

Latency Is The Product

This is the lesson I would put in giant letters over the whole build.

Latency is not a performance optimization for later. In voice, latency is the product.

Dead air on a phone call feels like failure way faster than a slow web app does. A caller will forgive a weird phrasing choice before they forgive that awkward "did this thing die?" pause after every turn.

Twilio's latency guide for AI voice agents is one of the better sanity checks I have seen on this. Their November 2025 launch benchmarks put a straightforward median mouth-to-ear turn gap around 1.1 seconds, with separate budgets for STT, LLM first-token time, and TTS. You do not need to obsess over the exact numbers. You do need to respect the shape of the problem.

Once I started thinking that way, a lot of decisions got easier:

• keep services close together regionally
• avoid unnecessary provider hops
• stream responses instead of waiting for the whole answer
• measure turn gap from the caller's perspective, not just your internal logs

If you are building realtime voice and you do not have a latency budget in the spec, you are basically hoping the product experience works out by accident.

The Fallback Path Comes Before The Happy Path

My first instinct was to make the "good demo" work.

Caller asks a clear question. Agent gives a clean answer. Everybody smiles. Great.

That is not the real product.

The real product is what happens when the caller interrupts halfway through a sentence, asks for a human, changes the subject, gives a bad phone number, or says something your knowledge base cannot answer cleanly.

Twilio's guide on token streaming and interruption handling gets at a subtle but important point: the agent's memory has to match what the caller actually heard, not the full response the model intended to finish saying. That sounds obvious once you hear it. It was not obvious enough to me the first time.

The same thing shows up in OpenAI's realtime prompting guide: keep turns short, keep pacing fast, and define explicit escalation rules up front. Not after the launch. Up front.

So now I would design these before I wrote a clever opening prompt:

• when the caller should be handed to a human
• when the system should stop trying and collect a callback instead
• when the agent is allowed to answer from the knowledge base
• when the agent should say "I do not know" and route the issue
• what summary gets handed to the human on transfer

That is boring compared to prompt tweaks. It is also the part that makes the thing usable.

A Better Voice Agent Looks More Like A Call Flow

I used to think I could prompt my way out of a bad system design.

You cannot.

If one agent is trying to be a receptionist, scheduler, FAQ bot, lead qualifier, and escalation router all at once, the prompt is not your main issue. The scope is.

This is basically the same rule I wrote about in Things I've Learned Building AI Agents: start as a workflow, then add agent behavior only where the state is actually unknown.

For voice, that usually means the system should feel more like a smart call flow than a fully autonomous robot employee. I mean that as a compliment.

Something like this is usually enough for a first version:

1. Greet the caller and identify intent.
2. Answer simple grounded questions if the answer is high confidence.
3. Collect the lead or intake details that matter.
4. Route, transfer, or create a callback task.
5. Hand a human a clean summary instead of a raw transcript.

That is already useful. More importantly, it is debuggable.

What I'd Actually Build Now

If I were starting over tomorrow and wanted a version a business could trust, I would not sell "AI that handles your phones."

I would sell one narrow pilot:

• after-hours intake for a local service business
• basic FAQ coverage
• lead capture
• urgent-call routing
• human handoff rules

No billing. No wide-open support. No pretending version one should do everything.

If you want the plain-English pricing version of this conversation, I wrote that up separately in What Does a Voice AI Agent Actually Cost in 2026?.

The business plan is pretty simple:

1. Pick one lane where missed calls cost real money. After-hours HVAC, plumbing, electrical, med spa, legal intake. Something with obvious value.
2. Define success in business terms. Calls answered, leads captured, callback tasks created, urgent issues escalated, average response gap, and how many calls still needed a human.
3. Build the flow before the personality. Greeting, intent detection, grounded answers, lead capture, transfer rules, and fallback behavior.
4. Run it as a pilot for two weeks and review every bad call manually. This is where the real product work happens.
5. Only add scheduling, CRM writes, or broader autonomy after the first lane becomes boring and reliable.

That is also why the current Voice Agent Pilot on the site is scoped the way it is.

It is not "an AI receptionist that does everything." It is a real-time phone or web voice pilot focused on intake, routing, after-hours coverage, knowledge grounding, and handoff rules. That scope exists because I trust narrow systems that survive real calls a lot more than broad systems that only survive demos.

If this stuff is interesting to you, the follow-up post is probably going to be the deeper technical version: why I use Pipecat instead of building voice from scratch, where it helps, and where it definitely does not save you.

Voice taught me the same lesson production software usually does: the impressive part is not making it talk. The impressive part is making it recover gracefully when reality shows up.

Written from home, with a lot more respect for dead air than I had the first time.