Semantic cachingIn progress
Match prompts by meaning instead of exact text, so a reworded question can return an answer that was already generated.
Currently in development
Make every AI request smarter.
An intelligent gateway between your applications and LLM providers. Reduce redundant requests, optimize latency, and gain visibility into your AI infrastructure.
Not publicly available yet. Everything in the window below is a simulation.
Served from cache, about 6 msForwarded to provider, about 1.2 sIllustrative timings, not measurements
Request traceSimulated
- normalize
- “I forgot my password, how can I change it?”
- embed
- Prompt converted to a vector
- nearest
- Closest stored prompt
“How do I reset my password?” - similarity
- score 0.962threshold 0.90
- decision
- Cache hit. Stored answer returned.HIT≈ 6 ms this request≈ 1,240 ms if uncached
Simulated cache hit. The closest stored prompt scored above the threshold.
app.ts
// Planned API shape. The gateway is not available yet. import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://gateway.example/v1", apiKey: process.env.SG_KEY, }); // Reworded repeats can now be answered from cache. const res = await client.chat.completions.create({ model: "gpt-4o-mini", messages: [{ role: "user", content: "How do I reset my password?" }], });
Simulated
Cut repeat calls. Keep the visibility.
The gateway handles the cache lookup, the provider call, and the record of what happened. These are the planned capabilities and where each one stands today.
pgvector + Redis storageIn progress
Cached responses and their vectors are stored in PostgreSQL with pgvector; Redis handles hot-cache and infrastructure concerns.
LLM provider integrationIn progress
Send requests to one endpoint. Cache misses are forwarded to the provider you configure.
API key managementPreview
Issue scoped keys per workspace, see the secret once, and revoke them at any time.
Multi-tenant architectureIn progress
Keys, cache entries, and logs are scoped to a tenant, so one team never receives another team's answers.
Request logs and analyticsPreview
Designed to show every request with its hit or miss result, similarity score, latency, and token count. The read APIs are not built yet.
Status labels describe the build, not a release date. See every capability.
One request, two possible paths
Every request takes the same first steps. What happens next depends on how close it is to something already answered.
Your app calls the gateway
Point your client at the gateway endpoint and authenticate with a gateway key instead of a provider key.
The prompt is embedded
The gateway turns the prompt into a vector and searches the scoped cache namespace for the nearest stored entry.
A hit or miss is decided
Above the similarity threshold, the stored answer is returned in milliseconds. Below it, the request goes to the provider.
The result is recorded
Misses are cached for next time. Every request is logged with its score, latency, and tokens.
A gateway you can reason about
A small number of moving parts, each with one job. The diagram shows the planned request flow, not a running system.
The diagram scrolls horizontally on narrow screens.
Where the build stands
Phases run in order. No dates are promised yet. This page will change as the work does.
Foundations
Gateway service skeleton, Tenant and API key modelIn progressSemantic cache core
Embedding pipeline, pgvector index, scopes and retentionIn progressProvider integration
OpenAI provider adapter, OpenRouter provider adapterIn progressObservability
Request log read API, Usage analytics APIPlannedPrivate preview
Invite-only access, Public documentation and SDKPlanned
Follow the build
Semantic Gateway is not open to the public yet. Leave an email to be considered for the private preview.
Demo form. Nothing is submitted, stored, or sent.