Our take on integrating OpenAI, Anthropic, and your in-house models without the prototype-to-prod cliff.
Most AI features die between the demo and the deploy. The model was never the hard part — the plumbing around it is. This is the checklist we run on every engagement, in the order we run it.
All model traffic — OpenAI, Anthropic, or the open-weights model on your own hardware — goes through a single gateway with a virtual key per app. Swap providers with a config change, cap spend per team, and revoke one key without redeploying anything. The moment two services call two SDKs directly, you've lost the ability to see, budget, or switch.
An ungrounded chatbot will invent your pricing and apologise later. Retrieval comes first: your handbook, docs, and policies — chunked, embedded, and cited in every answer. Guardrails sit in front to keep the conversation on topic and inside policy. If the context doesn't cover it, the right answer is "I'm not sure", not improvisation.
The prototype works on the happy path; production is all the other paths. Retries with backoff and jitter, hard timeouts, per-user rate limits, and streaming with a graceful fallback when the stream drops. None of this demos well, and all of it is load-bearing — it's the difference between an incident and a retry nobody noticed.
Prompts live in git, not in a dashboard textbox. Every completion is logged with the model, prompt version, and context that produced it. When a provider ships a model update and behaviour shifts — and it will — you can diff what changed and roll back in minutes, with an audit trail your compliance team can actually read.
A golden set of real questions with expected behaviours, run automatically on every prompt or model change. Not academic benchmarks — your questions, your edge cases, your tone. "It feels better" is not a regression test; a failing eval that blocks the deploy is.
Start on a hosted API if that's fastest — but keep the seam clean, so moving to open-weights models on your own hardware is a config change, not a rewrite. We run production workloads on self-hosted models behind the same gateway pattern, and nothing we build assumes a vendor. Like everything else we ship: it's yours.