An AI Voice Agent Needs More Than a Phone Number and a Realtime Model
By Greg Nowak. Last updated 2026-08-30.
Connecting an AI voice model to a business number is now relatively straightforward. Operating that number without losing callers, leads, or context is the harder job.
A production voice agent needs a defined purpose, reliable call routing, a route to a human, and sensible behavior when speech recognition, an internal system, or the AI itself fails. Those decisions should be made before anyone spends days polishing the prompt or choosing a pleasant voice.
Start with the architecture, not the voice
OpenAI documents two broad voice-agent architectures. Speech-to-speech sessions process live audio directly and suit conversations where low latency, natural turn-taking, and interruption matter. Chained pipelines separate speech-to-text, text reasoning, and text-to-speech, giving the application more control over each stage.
That distinction also helps clarify two commonly confused telephony options. A SIP provider can send a call directly into OpenAI Realtime, while Twilio ConversationRelay sends transcribed caller input to your application over a WebSocket and converts the application's text responses back into speech. They are alternative integration patterns, not components every project must combine.
| Approach | Best fit | Operational advantage | Main trade-off |
|---|---|---|---|
| Direct SIP to OpenAI Realtime | Natural, fast conversations with barge-in and realtime tools | OpenAI exposes call acceptance, monitoring, transfer, rejection, and hang-up controls | Your team must operate the SIP flow, server control channel, tools, and fallbacks |
| Twilio ConversationRelay | Structured qualification, support, and routing workflows | Twilio manages speech recognition, speech synthesis, and the live call bridge | Your application still owns conversation logic, WebSocket reliability, and escalation |
| Explicit chained pipeline | Approval-heavy or regulated workflows needing visible intermediate text | Each transcription, reasoning, policy, and speech stage can be inspected or replaced | More components can mean additional latency and integration work |
Design the call flow before the prompt
Write down what the agent is allowed to accomplish. “Answer the phone” is too broad. “Identify the caller's service need, collect a postcode and preferred callback time, then route or create a lead” is testable.
For every branch, define an outcome: continue with the agent, transfer to a person, arrange a callback, reject the call, or play a fallback message. Include silence, repeated misunderstanding, unsupported languages, unavailable staff, failed CRM requests, and a caller explicitly asking for a human. A voice agent that cannot exit gracefully is not ready for a published number.
With OpenAI's SIP path, the provider points its trunk at OpenAI's SIP endpoint. An incoming call produces a realtime.call.incoming webhook containing a call_id. The application can then accept or reject the call. Once accepted, it can monitor the session over wss://api.openai.com/v1/realtime?call_id={call_id}, transfer it using the refer endpoint, or hang up. These controls should map to written business rules rather than model improvisation.
Keep tools and private rules on the server
Calendar availability, CRM lookups, lead scoring, account data, and transfer rules belong in server-side code. OpenAI's sideband design allows the caller connection and an application-server connection to participate in the same Realtime session. The server can monitor events, update instructions, and answer tool calls without exposing credentials or internal logic to the caller-facing connection.
This separation also makes failures easier to diagnose. A tool timeout, failed WebSocket, invalid response, and poor model decision are different problems. Record them separately, give each one a fallback, and avoid treating every unsuccessful call as a prompting issue.
Specify the phone experience explicitly
ConversationRelay exposes controls for the welcome greeting, interruptions, interruption sensitivity, speech timeouts, keypad detection, languages, voices, and whether input is reported while the agent is speaking. Configure the settings you depend on rather than trusting defaults. For example, Twilio changed the default for reportInputDuringAgentSpeech from any to none in May 2025, and the default transcription provider can differ for older accounts.
Test interruption behavior with real callers. An agent that never yields feels frustrating; one that stops whenever it hears a short acknowledgement or background noise feels broken. Also decide whether the welcome greeting may be interrupted and whether keypad input provides an alternative when speech recognition struggles.
Multilingual support needs its own test plan. ConversationRelay can configure speech recognition and synthesis separately for each language. Its automatic multi mode currently requires Deepgram for transcription and ElevenLabs for speech synthesis; unsupported combinations end the session. Test accents, names, addresses, language switching, and the human-transfer path rather than relying on a polished bilingual demo.
Operate the number with evidence
For every call, capture the route taken, completion or transfer reason, tool failures, session status, duration, and the minimum transcript data the business genuinely needs. Twilio can send session details to the <Connect> action callback when ConversationRelay ends, including failure or handoff information. OpenAI reports token usage in response.done events, while enabled input transcription has separate usage events and billing.
Cost control starts with scope. Short qualification calls with a quick handoff are easier to predict than open-ended assistants. Stable instructions and tool definitions also preserve prompt-cache efficiency. For unusually long sessions, OpenAI provides token-window and truncation controls, including a retention ratio, but reducing memory can affect later responses. Treat that as a measured optimisation, not a default shortcut.
A practical launch checklist
- Give the agent one measurable call objective.
- Document acceptance, rejection, transfer, callback, and failure rules.
- Keep tools, credentials, and routing logic on the server.
- Configure interruption, timeout, keypad, language, and callback behavior explicitly.
- Test noisy calls, silence, unsupported requests, tool failures, and human escalation.
- Monitor completion, transfer reasons, technical failures, latency, and cost from day one.
The model is only one part of the service. The real deliverable is a phone operation that callers can understand and the business can monitor, change, and trust.
If you are deciding between direct SIP, ConversationRelay, or a chained workflow, Greg can help turn the requirements into a practical call flow and implementation plan. Talk to Greg about the project.
Related on GrN.dk
- A Voice Agent Is Only Ready When the Human Handoff Works
- Not Every AI Job Needs an Instant Answer: Batch the Backlog
- OpenAI’s Assistants API Shuts Down in Ten Days. Is Your App Ready?
Need help with this kind of work?
Plan your AI voice project with Greg Get in touch with Greg.