Live calls

One websocket per call. You push caller audio and play what comes back. Voice activity detection, turn taking, barge-in, transcription, the LLM and text-to-speech all run server side. You do not orchestrate anything.

Connect

wss://api.latency.cam/ws/stream?api_key=sk-lat-...

The key goes in the query string, not a header. Browsers cannot set headers on a WebSocket, so a header-only scheme would rule out browser clients entirely. A bad key gets a JSON error frame with code: "unauthorized" and then close code 1008, so you see a reason instead of an opaque handshake failure.

1. Configure, first frame

Audio sent before this is refused.

{
  "type": "configure",
  "language": "hi",
  "script": "roman",
  "audio_format": "mulaw_8000",
  "voice": "Hindi (Female)",
  "llm": "nova-lite",
  "system_prompt": "You are a delivery helpdesk. Reply in Hindi, under 12 words.",
  "silence_ms": 500,
  "interrupt_min_words": 3,
  "use_fillers": true,
  "greeting": true
}

You get back {"type":"configured","config":{...}} echoing everything that was applied, so you can assert on it.

2. Stream caller audio

Binary frames, 20 milliseconds each — 160 bytes for 8kHz mulaw. Send them roughly one every 20ms, in real time. Do not dump a whole file at once: turn detection would see it as one enormous utterance and never fire.

3. Read events

A normal turn produces these, in order:

speech_started      caller began talking
turn_end            500ms of silence, turn closed        { turn, audio_ms }
filler_played       cached audio already sent to you     { id, category }
transcript.final    { text, turn, language, stt_ms, model }
llm.reply           { text, turn, llm_ms }
bot_audio_started   { turn, ttfa_ms, text }   binary frames start now
bot_audio_done      synthesis for this turn finished
turn_complete       { turn, total_ms, stt_ms, llm_ms, tts_ms }

Binary frames arriving between bot_audio_started and bot_audio_done are raw audio in your configured format, with no header. Write them straight to the call.

Barge-in

interrupted        { turn, reason }   the caller has taken the turn
speech_abandoned   { turn, reason: "barge_in" }   old synthesis stopped

Barge-in fires about 200ms after the caller starts speaking, not after they finish. That distinction matters: waiting for end-of-turn means the bot talks over the caller for their whole sentence plus the silence timeout, which is the most obvious way a voice agent feels broken.

Every event carries a turn number. When you see interrupted, drop any audio you have buffered for the previous turn — that is what the number is for.

Other events

Frames you can send

The turn budget, measured

From the caller finishing their sentence, measured over wss from inside AWS, so excluding the ~238ms India round trip:

    1ms   filler starts playing        <- what the caller actually perceives
  166ms   transcription done           faster-whisper large-v3-turbo int8
  553ms   LLM reply ready              nova-lite via Bedrock, ~387ms
 1346ms   first bot audio out          svara-TTS via vLLM, TTFA ~793ms

Measured wait about 1.5 seconds. Perceived wait about 1 millisecond, because the filler is already playing. That gap is the entire trick, and it is why use_fillers defaults to on.

A complete client

import asyncio, json, os, websockets

KEY = os.environ["LATENCY_API_KEY"]
URL = f"wss://api.latency.cam/ws/stream?api_key={KEY}"
FRAME = 160          # 20ms of 8kHz mulaw

async def main():
    async with websockets.connect(URL, max_size=None) as ws:
        await ws.send(json.dumps({
            "type": "configure",
            "language": "hi",
            "script": "roman",
            "audio_format": "mulaw_8000",
            "system_prompt": "You are a delivery helpdesk. Reply in Hindi, under 12 words.",
        }))

        current_turn, abandoned = 0, set()

        async def pump():
            # In production this is your telephony provider's media stream.
            with open("caller.mulaw", "rb") as fh:
                while chunk := fh.read(FRAME):
                    await ws.send(chunk)
                    await asyncio.sleep(0.02)      # real-time pacing matters

        asyncio.create_task(pump())

        async for msg in ws:
            if isinstance(msg, bytes):
                if current_turn not in abandoned:
                    play_to_caller(msg)            # raw audio, no header
                continue

            ev = json.loads(msg)
            if ev["type"] == "bot_audio_started":
                current_turn = ev["turn"]
            elif ev["type"] == "interrupted":
                abandoned.add(current_turn)        # stop the old reply
                stop_playback()
            elif ev["type"] == "transcript.final":
                print("caller:", ev["text"])
            elif ev["type"] == "error":
                say_fallback_line()

asyncio.run(main())

Known limitations

• No streaming partial transcripts. Whisper transcribes complete utterances, so transcript.final arrives once per turn. Partials would save roughly 70ms of a 1500ms turn, which is why they are not built yet.
• Voice activity detection is energy based, calibrated on the first ~300ms of the call. It handles turn taking well and does not reliably tell speech from sustained background noise. Silero VAD would be better and is a known gap.
• The server decides when a turn ended. That keeps the protocol simple and means a naive client cannot break turn taking, but you cannot override it.