Limits and known gaps

What does not work, stated plainly. This page exists so nobody discovers these the hard way, and so an AI assistant reading these docs gives honest answers.

There are no rate limits

No rate limits, no quotas, no per-model gating, no concurrency capsVerified by audit, not asserted: 216 requests including 20-way and 30-way parallel bursts and parallel hits on every GPU endpoint returned zero HTTP 429s and no rate-limit headers of any kind. Requests queue when the hardware is busy. They are never rejected.

You will never see a 429 from this API. If a request is slow under load it is because it is waiting for a GPU, and the response tells you so — /v1/translate even reports queued_ms separately from inference_ms so slow is never mistaken for stuck.

What actually constrains you

Constraint	The real number
One A10G GPU, 23GB	About 6 concurrent synthesis requests run comfortably. Measured: 6 in 13.9s against 83.7s when serialised. Beyond that, requests queue and each gets slower.
VRAM budget	vLLM permanently holds ~12.6GB because that is what keeps time-to-first-audio at 793ms. The optional large speech-to-text models load into what is left and may evict each other.
Network round trip	The box is in Virginia. ~238ms round trip from India, versus ~25ms if it were in Mumbai. For Indian callers this is often the largest single number in your budget.
Translation throughput	CPU only, serialised per worker, two workers. 12 concurrent requests all completed in 14.5s total.
Bedrock	`/v1/chat/completions` is backed by AWS Bedrock, whose own service quotas apply and are not controlled here.

Not built

Voice cloning

/v1/clone returns 501. The model exists and is Apache-2.0 — kenpath/svara-tts-voiceclone-beta, ungated, 19 languages, and it keeps the same speaker IDs as the model already in use. The blocker is VRAM: it is 6.6GB and there is about 6.8GB free once vLLM has taken its share. Fitting it means either a second GPU or replacing the default text-to-speech model with the beta one, which would put a beta model on the path every live call depends on.

Meanwhile the 38 preset voices cover 19 languages in both genders and all of them are verified working.

Streaming partial transcripts

Whisper transcribes complete utterances, so the websocket emits transcript.final once per turn rather than a growing partial. This is a deliberate deferral, not an oversight: partials would save roughly 70ms of a ~1500ms turn, which is the smallest remaining win on the list.

Voice activity detection is energy based

It calibrates on the first ~300ms of the call and adapts to the line. It handles turn taking well and does not reliably distinguish speech from sustained background noise — a loud shop or a TV in the room will trigger it. Silero VAD would be better and is a known gap.

Token-by-token LLM streaming

stream: true is accepted on /v1/chat/completions and ignored; the reply arrives whole. On a call this matters less than it sounds, because synthesis is the long pole and the filler cache already covers the wait.

Languages we can hear but cannot speak

Odia, Urdu, Kashmiri, Konkani, Manipuri, Santali and Sindhi. All seven can be transcribed and translated, but svara-TTS has no voice for them, so they cannot be spoken mid-call. If you need to reply in one of these, translate into a language that has a voice, or fall back to text.

Quality is not flat across 22 languages

Measured on real human speech (FLEURS) with the default model: Hindi word error rate 0.104, Telugu 0.219, Tamil 0.277. Hindi, Tamil, Telugu, Bengali and Marathi are good. Bodo, Santali, Dogri, Manipuri, Sanskrit, Kashmiri, Sindhi and Konkani are weak — for everyone, including large commercial vendors, because the training data does not exist.

Listing 22 languages is honest. Promising equal quality across 22 is not.

The 8kHz telephony penalty, now measured

This used to be the largest open question on this page. It has been measured: word error rate goes from 0.204 at 16kHz to 0.252 after a round trip through 8kHz mulaw, and character error rate from 0.099 to 0.121.

So telephony costs about 5 points of WER — real, but considerably less than most people fear. You do not need to plan around it.

Language detection has one hard failure

Hindi versus Urdu cannot be reliably separated by audio. They are near-identical when spoken and differ mainly in script and formal vocabulary. Published accuracy for that family is 58.7%. Do not build a feature that depends on it.

Telugu versus Tamil is fine — different family, clearly different sounds, 85.9% published. Always pass restrict_to when you can narrow the candidate set; it is the single biggest accuracy lever on that endpoint.

Models that stayed out

Model	Why not
`indicconformer-600m`	Gated on Hugging Face. Needs an `HF_TOKEN` from an account that accepted the licence. Approval is automatic — it is one click, but somebody has to click it.
`IndicTrans2` official repos	Also gated now. Translation runs on the ungated distilled checkpoints published by IndicTrans2's own lead author instead, which are the same MIT licence.
`VibeVoice-Realtime-0.5B`	English only, research-use-only licence, and it embeds an audible "generated by AI" disclaimer plus a watermark into every output.
`indic-mio`, `indicf5`	No stated licence on the model card. Fine for internal use; not something to put in a customer contract until the authors confirm terms in writing.
`claude-haiku-4-5`	Available as an id, but Bedrock returns `ResourceNotFoundException` until the Anthropic use-case form is submitted for this AWS account. `nova-lite` works today and is the default.

Infrastructure

Single box, no autoscaling. The AWS credentials in use have no IAM permissions, so there is no load balancer and no auto-scaling group. Fine for development; it will not survive production traffic.
Virginia, not Mumbai. No GPU quota in ap-south-1 on this account. Adds ~238ms round trip for Indian callers and breaks any data-residency claim.
Starting the GPU can fail. us-east-1 GPU capacity is tight and InsufficientInstanceCapacity happens. Retry with backoff.
The websocket does not pass through Vercel, which cannot hold long-lived connections. It goes to the box directly, so when the GPU is off it fails rather than returning a clean error. Poll /health first.
Cold start is about 40 seconds. Models warm automatically after boot. Before that, requests work but the first is slow.

Compliance not yet handled

DPDP Act 2023. Voice recordings are personal data. Fine for synthetic test audio; needs a real answer before customer calls flow through.
DLT registration and DNC scrubbing. Required for outbound calling in India. Not built.
Voice cloning consent. Every model licence in this stack forbids non-consensual cloning, so /v1/clone requires an explicit consent flag and logs every request. It is the one place where "no restrictions" does not apply, because removing the restriction would breach the licences the whole stack depends on.

What would move the needle fastest

A GPU in ap-south-1. Removes ~238ms from every request for Indian callers, which is larger than any remaining model-level win.
Accept the Hugging Face licences for IndicConformer and IndicTrans2. Minutes of clicking, and it unlocks a purpose-built Indic speech model.
Submit the Anthropic use-case form so Claude Haiku becomes available.
Swap the energy VAD for Silero, which fixes turn detection in noisy environments.
Start recording real calls. Fine-tuning on your own traffic is the only durable advantage here, and it cannot be bought.