Skip to content
Semantic Gateway
Sign inGet early access

Architecture overview

The planned shape of the system: a stateless gateway, PostgreSQL with pgvector for semantic storage, Redis for short-lived state, and provider adapters behind a single request format.

A small number of moving parts

Each component has one job. The diagram is a design reference, not a running deployment.

Semantic Gateway request architectureApplications call the gateway, which authenticates the key, embeds the prompt, looks up the nearest entry in PostgreSQL with pgvector, and either returns a cached response or forwards the request to an LLM provider. This is the planned request flow, not a running system.Your applicationsSDK or plain HTTPgateway keySemantic GatewaypreviewAuthkey → tenantEmbedprompt → vectorLookupnearest entryDecidescore ≥ thresholdPostgreSQL + pgvectorvector index + stored responsesLLM providersyour configuredprovider · previewmisshit: stored response returned in milliseconds

The diagram scrolls horizontally on narrow screens.

Stateless gateway
Gateway instances hold no request state. Cache records and logs live in PostgreSQL; Redis provides a best-effort hot cache.
Scoped cache entries
The backend supports private (per key, default), tenant, and opt-in global cache scopes. Every lookup is fingerprinted by scope, models, and generation parameters.
Explicit threshold
SEMANTIC_THRESHOLD is a deployment setting today, defaulting to 0.92. Per-workspace tuning tools are labelled In progress.
Retention controls
Prompts are not stored by default. Cache entries and request logs have configurable retention windows, and a daily cleanup job enforces them.
Observable by default
Every request records hit or miss, similarity, latency, tokens, and failure stage. The read APIs that expose this in the dashboard are still planned.
Provider-agnostic
OpenAI and OpenRouter adapters sit behind one request shape. Streaming passthrough has not been implemented yet.

Built with

  • PostgreSQL + pgvector
  • Redis
  • NestJS
  • TypeScript
  • OpenAI
  • OpenRouter

Authentication, tenant membership, API key management, and the gateway query path exist in the backend today. The dashboard screens that read logs, analytics, and cache entries are a preview backed by sample data until those read endpoints are built.

Follow the build

Semantic Gateway is not open to the public yet. Leave an email to be considered for the private preview.

Demo form. Nothing is submitted, stored, or sent.