One appliance on the LAN. It indexes the internal corpus once, retrieves and compresses per request on a local model, and forwards a 4,000-token curated context instead of a 14,400-token pile. Modelled at 63.3% lower monthly cost than cloud-side RAG — with the appliance's own hardware inside that figure.
The gateway does not make your cloud model smarter. It changes what the cloud model is asked to read.
Answer length is a few hundred tokens; prompt length is tens of thousands. At pro-tier list price, unmatched input runs 4.50 CNY per million tokens against 13.50 CNY for output, and cached input is 30× cheaper than uncached. Cutting the prompt is the only lever with this much travel.
If 2,000 tokens are what the task needs and the call carries 14,400, only 25.0% of the context is on point. The gateway lifts that to 50.0% — less spend and a lower chance of the model anchoring on the wrong passage.
Retrieval, ranking and summarisation run on the appliance. The raw documents, mail and repositories never leave the network — only the curated context does. For teams that cannot ship a whole corpus to an API, this is the difference between using an LLM and not using one.
Placement of the gateway. Only the curated context leaves the LAN; the raw corpus never does.
One team, one knowledge base, one set of questions. 19,800 requests per month at pro-tier pricing; the gateway scheme carries its own hardware and power.
| Scheme | Context / turn | Turns | Input / call | Input / month | Total / month |
|---|---|---|---|---|---|
| No orchestration | 32,000 tok | 2.2 | 70,400 tok | 1,393.9 M tok | 6,078 CNY |
| Cloud-side RAG | 8,000 tok | 1.8 | 14,400 tok | 285.1 M tok | 1,372 CNY |
| On-device gateway | 4,000 tok | 1.4 | 5,600 tok | 83.2 M tok | 503 CNY |
Cloud-side RAG is the baseline we quote against, because it is what most teams already run. Comparing a good design to no design would produce a larger, less honest number.
Total monthly run cost — cloud plus the gateway appliance. The right-hand bar is the on-device scheme.
The four mechanisms interact, so their effects cannot be summed. The meaningful figure is the penalty for removing one while the other three stay on.
| Mechanism | Cost if disabled | Share of total |
|---|---|---|
| Stable prefixes — cache hit 10% → 65% | +163 CNY/mo | 32% |
| Context compression — 8,000 → 4,000 tok | +139 CNY/mo | 27% |
| Local offload — 25% resolved on the appliance | +112 CNY/mo | 22% |
| Turn convergence — 1.8 → 1.4 turns | +96 CNY/mo | 19% |
The biggest single lever is not compression — it is cache hit rate. At 32% it outweighs context compression at 27%, because cached input is 30× cheaper than uncached. A prompt prefix that never changes is worth more than a shorter one, which is why prefix stability is specified as a deliverable rather than left as an implementation detail.
Compression ratio is the assumption most likely to differ in your environment. Here it is scanned rather than asserted.
| Context / turn | vs cloud-side RAG | Relevant share | Total / month | Saving |
|---|---|---|---|---|
| 2,000 tok | 4.0× | 100.0% | 433 CNY | 68.4% |
| 3,000 tok | 2.7× | 66.7% | 468 CNY | 65.9% |
| 4,000 tok — specified | 2.0× | 50.0% | 503 CNY | 63.3% |
| 6,000 tok | 1.3× | 33.3% | 572 CNY | 58.3% |
| 8,000 tok — no compression | 1.0× | 25.0% | 642 CNY | 53.2% |
| 16,000 tok | 0.5× | 12.5% | 920 CNY | 32.9% |
Even with compression switched off entirely, local offload and turn convergence still carry roughly half the saving. The value of the appliance does not rest on aggressive compression alone.
The gateway is a productised configuration of our RK182X edge inference hardware, not a bare board.
RK182X-class edge inference host sized for the workload, with the BSP, thermal validation and enclosure. Throughput reference: 102.01 tok/s decode on a 3B model, 215.86 tok/s on 0.5B for routing alone.
Corpus ingestion and incremental index build, retrieval and re-ranking, context compression, prefix freezing, per-request token accounting and a local-only fallback path.
An OpenAI-compatible endpoint that sits in front of whichever cloud provider you already use. Existing client code keeps its interface; only the base URL changes.
The cost model above re-run with your token logs, plus a written baseline of signal-to-noise measured on a sample of your own requests.
Stated honestly: the appliance is not free. 150 CNY/month of amortised hardware and 17 CNY/month of power sit inside the 503 CNY figure. The first full pass over a large corpus is prefill-bound and must be amortised into the index — a configuration that re-embeds the whole corpus per request will lose more in latency than it saves in tokens.
| # | Step | Output |
|---|---|---|
| 1 | Token accounting on the current setup | Spend attributed per feature, split input / cached / output |
| 2 | Corpus survey and index design | What gets indexed, refresh cadence, exclusion list |
| 3 | Appliance build and indexing | Gateway running on the LAN, first full index complete |
| 4 | Retrieval and compression tuning | Compressed context still answers the sampled questions correctly |
| 5 | Cut over and measure | Before / after cost and signal-to-noise on your own traffic |
Steps 1 and 2 are done on your side with our model; steps 3 to 5 are ours. We will not quote a saving before step 1 is complete — the number depends on your token logs, not on ours.
No. It sits in front of the provider you already use and reduces what is sent to it. You keep the provider relationship, the model choice and the account.
A single appliance covers a team of the size modelled here (30 people). Below roughly a dozen users the appliance's own amortisation outweighs the token saving, and we would say so.
Modelled at 25% of requests answered without a cloud call. That share is a measurement, not a claim — step 1 of the deployment establishes it for your traffic.
Only the curated context. Documents, mail and repositories are read on the appliance; the raw corpus is never transmitted. The index itself also stays on the appliance.
Class and task routing can run on a 215.86 tok/s 0.5B model; retrieval and summarisation on a 102.01 tok/s 3B model, which is the specified configuration. Larger on-device models are available at lower throughput.
Then the result is worse, and the sensitivity table above says by how much. At 8,000 tokens of context — no compression at all — the saving still reaches 53.2%.
Cascade AI cards over PCIe or USB 3.0 — on-device LLM capacity to the 27B class with OpenAI-compatible serving.
AI-native 3-tier power and environment monitoring for broadcast and telecom sites.
Carrier board and enclosure design, BSP, thermal validation and mass production.
[measured] Byte-to-token conversion was measured with a 200k-vocabulary tokeniser on this study's own corpora: 3.0 bytes/token CJK-dominant, 3.2 English with markup, 3.7 English plain text. Our internal memory stack holds 284,222 bytes = 95,303 tokens and injects 8,766 bytes = 3,004 tokens per session — a measured 31.7× compression. [published] Pro-tier list prices, CNY per million tokens, quiet-hours band: unmatched input 4.50, cached input 0.15, output 13.50. On-device decode throughput for 0.5B / 3B / 4B / 8B: 215.86 / 102.01 / 90 / 61.11 tok/s. [assumed] Team size, turn counts, cache hit rates, output length, electricity price and hardware amortisation. [derived] Costs, savings, signal-to-noise and latency. The model claims a compression ratio of 2.0× where our own measurement is 31.7× — roughly a quarter of the observed value.
Send us a month of usage records and the shape of your corpus. We will return the cost model with your numbers in place of our assumptions.
Inquire