Live calls

One websocket per call. You push caller audio and play what comes back. Voice activity detection, turn taking, barge-in, transcription, the LLM and text-to-speech all run server side. You do not orchestrate anything.

Connect

wss://api.latency.cam/ws/stream?api_key=sk-lat-...

The key goes in the query string, not a header. Browsers cannot set headers on a WebSocket, so a header-only scheme would rule out browser clients entirely. A bad key gets a JSON error frame with code: "unauthorized" and then close code 1008, so you see a reason instead of an opaque handshake failure.

1. Configure, first frame

Audio sent before this is refused.

{
  "type": "configure",
  "language": "hi",
  "script": "roman",
  "audio_format": "mulaw_8000",
  "voice": "Hindi (Female)",
  "llm": "fast",
  "system_prompt": "You are a delivery helpdesk. Reply in Hindi, under 12 words.",
  "silence_ms": 500,
  "interrupt_min_words": 3,
  "use_fillers": true,
  "greeting": true
}

You get back {"type":"configured","config":{...}} echoing everything that was applied, so you can assert on it.

2. Stream caller audio

Binary frames, 20 milliseconds each — 160 bytes for 8kHz mulaw. Send them roughly one every 20ms, in real time. Do not dump a whole file at once: turn detection would see it as one enormous utterance and never fire.

3. Read events

A normal turn produces these, in order:

speech_started      caller began talking
turn_end            500ms of silence, turn closed        { turn, audio_ms }
filler_played       cached audio already sent to you     { id, category }
transcript.final    { text, turn, language, stt_ms, model }
llm.reply           { text, turn, llm_ms }
bot_audio_started   { turn, ttfa_ms, text }   binary frames start now
bot_audio_done      synthesis for this turn finished
turn_complete       { turn, total_ms, stt_ms, llm_ms, tts_ms }

Binary frames arriving between bot_audio_started and bot_audio_done are raw audio in your configured format, with no header. Write them straight to the call.

Barge-in

interrupted        { turn, reason }   the caller has taken the turn
speech_abandoned   { turn, reason: "barge_in" }   old synthesis stopped

Barge-in fires about 200ms after the caller starts speaking, not after they finish. That distinction matters: waiting for end-of-turn means the bot talks over the caller for their whole sentence plus the silence timeout, which is the most obvious way a voice agent feels broken.

Every event carries a turn number. When you see interrupted, drop any audio you have buffered for the previous turn — that is what the number is for.

Other events

Frames you can send

The turn budget, measured

From the caller finishing their sentence, measured over wss next to the service, so excluding the round trip from India:

    3ms   filler starts playing        <- what the caller actually perceives
   40ms   transcription done           language identified for free
  627ms   reply ready                  587ms of it, and the one stage that leaves our machines
  628ms   numbers rewritten            "15250" -> "pandrah hazaar do sau pachas"
  724ms   first bot audio out          time to first audio 96ms, real-time factor 0.83

Measured wait 724ms. Perceived wait 3 milliseconds, because the filler is already playing. That gap is the entire trick, and it is why use_fillers defaults to on.

A complete client

import asyncio, json, os, websockets

KEY = os.environ["LATENCY_API_KEY"]
URL = f"wss://api.latency.cam/ws/stream?api_key={KEY}"
FRAME = 160          # 20ms of 8kHz mulaw

async def main():
    async with websockets.connect(URL, max_size=None) as ws:
        await ws.send(json.dumps({
            "type": "configure",
            "language": "hi",
            "script": "roman",
            "audio_format": "mulaw_8000",
            "system_prompt": "You are a delivery helpdesk. Reply in Hindi, under 12 words.",
        }))

        current_turn, abandoned = 0, set()

        async def pump():
            # In production this is your telephony provider's media stream.
            with open("caller.mulaw", "rb") as fh:
                while chunk := fh.read(FRAME):
                    await ws.send(chunk)
                    await asyncio.sleep(0.02)      # real-time pacing matters

        asyncio.create_task(pump())

        async for msg in ws:
            if isinstance(msg, bytes):
                if current_turn not in abandoned:
                    play_to_caller(msg)            # raw audio, no header
                continue

            ev = json.loads(msg)
            if ev["type"] == "bot_audio_started":
                current_turn = ev["turn"]
            elif ev["type"] == "interrupted":
                abandoned.add(current_turn)        # stop the old reply
                stop_playback()
            elif ev["type"] == "transcript.final":
                print("caller:", ev["text"])
            elif ev["type"] == "error":
                say_fallback_line()

asyncio.run(main())

Known limitations

• Live captions have a warm-up. transcript.partial arrives roughly every 400ms while the caller is still speaking, but not for the first second: below about a second of audio there is too little to identify the language from, and a partial in the wrong script is worse than no partial. Naming your languages in allowed_languages removes even that.
• Speech detection struggles in noisy places. It calibrates on the first few hundred milliseconds of each call, which makes turn taking reliable, but it does not reliably tell speech from sustained background noise — a loud shop or a television in the room will trigger it. Our most common real-world failure, and next on the list.
• The server decides when a turn ended. That keeps the protocol simple and means a naive client cannot break turn taking, but you cannot override it.