Skip to content
Semantic Gateway
Sign inGet early access

How a request moves through the gateway

Try the simulated trace, then see the four steps behind it.

Request traceSimulated
normalize
“I forgot my password, how can I change it?”
embed
Prompt converted to a vector
nearest
Closest stored prompt
“How do I reset my password?”
similarity
score 0.962threshold 0.90
decision
Cache hit. Stored answer returned.HIT
≈ 6 ms this request≈ 1,240 ms if uncached

Simulated cache hit. The closest stored prompt scored above the threshold.

A local demonstration. No request leaves this page and no provider is called.

The request lifecycle

The first two steps always happen. What comes next depends on how close the prompt is to something already answered.

  1. Your app calls the gateway

    Point your client at the gateway endpoint and authenticate with a gateway key instead of a provider key.

  2. The prompt is embedded

    The gateway turns the prompt into a vector and searches the scoped cache namespace for the nearest stored entry.

  3. A hit or miss is decided

    Above the similarity threshold, the stored answer is returned in milliseconds. Below it, the request goes to the provider.

  4. The result is recorded

    Misses are cached for next time. Every request is logged with its score, latency, and tokens.

HIT

Served from the cache

If the nearest stored entry scores at or above the configured threshold, its response is returned without calling a provider. The request log records the similarity score and the latency.

MISS

Forwarded to the provider

Streaming planned

Below the threshold, the prompt is sent to the configured LLM provider and the response is returned. A cache record is written best-effort for the next matching prompt. Streaming passthrough and threshold tuning tools are still being completed.

Follow the build

Semantic Gateway is not open to the public yet. Leave an email to be considered for the private preview.

Demo form. Nothing is submitted, stored, or sent.