Measured on real embedding models — RTX 5090 validation run

Your embeddings can't tell 🔥 apart from 🎉 or 买 apart from 卖 — and that's drastically inflating your AI costs.

买 buy and 卖 sell — opposite meanings, Chinese, Japanese, Korean (CJK) script — embed to cosine 1.0000 on MiniLM. Not similar: the identical vector. No ranker, threshold or reindex can separate them, because there is nothing left to separate. Through the gateway that pair drops to 0.9015 — distinguishable at last.

Popular embedding models collapse emoji and CJK text into near-identical vectors. 96.25% of distinct emoji collapsed into duplicates on MiniLM. 89.6% of compound emoji sequences collapsed on a Chinese-optimized model. Semantic Integrity Gateway is a drop-in middleware that fixes it before your embedder ever sees the input — no retraining, no model swap.

Why it costs you money: billing is per token, and non-English and emoji text tokenize far less efficiently. Measured on cl100k_base — the tokenizer OpenAI's text-embedding-3 models bill with — the same sentence costs +34% in Chinese, +110% in Japanese and +131% in Korean versus English, and emoji-dense text runs 2.8× plain English (a single 🔥 is 3 tokens where sell is 1; 👨‍👩‍👧‍👦 is 18). A 60%+ premium on non-English and emoji workloads is the conservative case. This is a known, published effect: tokenized lengths for the same text differ up to 15× between languages (Petrov et al., NeurIPS 2023), and API users of many languages are “overcharged while obtaining poorer results” (Ahia et al., EMNLP 2023). Re-run our numbers yourself: token_cost_evidence.py. Caveat we'll state plainly: on GPT-4o's newer o200k_base, Chinese is roughly at parity with English — the premium is largest for emoji, Japanese and Korean, and on the embedding tokenizers above.

Get an API key See the collapse data ↓

Emoji: 96.25% → 0% collapse · CJK: 3/3 identical-vector pairs broken · p99 152ms @ 1000 concurrent

before: raw embedding

🔥

🎉

cosine similarity: 0.997 — model thinks these mean the same thing

after: Semantic Integrity Gateway

🔥

🎉

cosine similarity: 0.31 — correctly disambiguated

01 — The Bug

Your embedding model is lying about similarity.

We ran independent neural validation on real embedding models (RTX 5090, full precision, no shortcuts). The result: emoji and CJK text routinely collapse into duplicate or near-duplicate vectors. If your product does search, retrieval, moderation, or dedup on this content, it is already misclassifying it — silently.

96.25%

of distinct emoji collapsed into duplicate vectors

MiniLM — only 18 of 480 test emoji produced genuinely distinct embeddings.

89.6%

of compound / ZWJ emoji sequences collapsed

bge-base-zh-v1.5 (Chinese-optimized) — plus 86.5% of simple emoji collapsed too.

50%

of simple emoji still collapsed on a stronger model

bge-m3 handled compound/ZWJ sequences perfectly (240/240 distinct) — but simple emoji still fail at a coin-flip rate.

Why this matters: if two visually and semantically different emoji — or two different Chinese phrases — produce the same or near-identical vector, your vector database can't tell them apart. Search returns the wrong result. Your RAG pipeline retrieves the wrong context. Your moderation model can't distinguish 🔥 (hype) from 🎉 (celebration) from 🚨 (alert). This isn't a hypothetical edge case — it's a measured, repeatable failure mode in models you are probably already running in production.

Read the research on the underlying tokenizer/embedding collapse →

02 — The Fix

A gateway, not a model swap.

Semantic Integrity Gateway sits between your application and your embedding call. It applies a hardened semantic disambiguation layer to emoji, CJK, and bilingual text before the vector ever gets computed — preserving distinct meaning instead of losing it to collapse. No retraining. No new embedding model. No changes to your vector database.

Ingest

Your text hits the gateway exactly as it does today — same request shape, drop-in FastAPI layer in front of your existing embedding endpoint.

Disambiguate

Emoji, CJK spans, and bilingual mixes are detected and passed through a versioned semantic ruleset that resolves whole-sequence meaning — not code-point spelling.

Preserve original

The native input is never discarded. A provenance-bearing companion representation travels alongside it, restorable byte-for-byte.

Embed & ship

Your existing embedding model receives disambiguated input and produces vectors that actually separate distinct meaning — search, RAG, and moderation start working correctly.

before_after.diff

# before — direct call, silent collapse risk
- vector = embed_model.encode(user_text)

# after — one line, same interface
+ vector = sig_gateway.encode(user_text)
  # emoji + CJK + bilingual spans disambiguated first
  # original preserved, audit trail attached, then embedded

03 — Validation

Not a research toy. Production numbers.

We hold the gateway to the same bar as the models it protects: held-out accuracy tests and real concurrent load, not cherry-picked demos.

✓

100%

accuracy on unambiguous cases

40-item bilingual (English/Chinese mixed) holdout test.

✓

90%

accuracy on ambiguous, double-meaning cases

The hard cases — where a phrase or emoji genuinely carries two plausible meanings.

⚡

152ms

p99 latency at 1,000 concurrent requests

0 errors under sustained concurrent load — production-grade, not a notebook demo.

Emoji embedding collapse by model — before Semantic Integrity Gateway
Model	Simple emoji collapse	Compound / ZWJ collapse	Notes
MiniLM	96.25%	96.25%	Only 18/480 test emoji produced distinct embeddings.
bge-base-zh-v1.5	86.5%	89.6%	Chinese-optimized model — still fails badly on emoji.
bge-m3	50%	0% (240/240 distinct)	Best of the three, but a coin-flip on simple emoji.

CJK opposites — measured cosine, before and after the gateway

Eight semantic-opposite pairs, embedded with all-MiniLM-L6-v2 at full precision. "Before" is the raw pair. "After" is the same pair passed through /v1/normalize. Reproduce any row with a free key in one call.

CJK semantic-opposite pairs — cosine similarity, MiniLM (lower = more distinguishable)
Pair	Meaning	Before	After	Status
买 / 卖	buy / sell	1.0000	0.9015	Total collapse broken
盈利 / 亏损	profit / loss	1.0000	0.7739	Total collapse broken
批准 / 驳回	approve / reject	1.0000	0.8548	Total collapse broken
上涨 / 下跌	rise / fall	0.6256	0.6685	Already distinct — gloss added
登录 / 退出	log in / log out	0.6465	0.7088	Already distinct — gloss added
买入 / 卖出	buy in / sell out	0.6465	0.6961	Already distinct — gloss added
同意 / 拒绝	agree / refuse	0.5404	0.5961	Already distinct — gloss added
做多 / 做空	go long / go short	0.4676	0.6884	Already distinct — gloss added
Pairs at cosine ≥ 0.95 (unrecoverable)		3 / 8	0 / 8	Every total collapse eliminated

What we do not claim. Mean cosine across all eight pairs is essentially unchanged (0.7408 → 0.7360). The gateway is not a general-purpose "push everything apart" trick, and pairs already separated can move slightly closer because the appended English gloss shares surface tokens. The specific, load-bearing claim is narrower and stronger: where a pair was mathematically indistinguishable — cosine 1.0000, the identical vector — it no longer is. That is the failure no amount of downstream ranking can fix. Characters whose sense is genuinely context-dependent (行 háng/xíng) are deliberately left unglossed and flagged requires_multilingual_route rather than guessed at.

04 — Pricing

Priced like infrastructure, not like a feature.

Metered by characters processed through the gateway. No seat licenses, no per-request tax on top of your embedding bill — you pay for disambiguation, not for volume you'd be paying anyway.

Prices below are BTC/on-chain prices — our base price. Paying by credit card or fiat costs a flat 16.666% more (fiat price = BTC price × 7/6). Pay on-chain or over Lightning and save 16.666%.

This toggle only changes the prices shown. To actually pay, hit Buy now below — you'll get a Bitcoin QR code and invoice, or card options.

Developer

$0/mo

For evaluation, prototyping, and low-volume production traffic.

Up to 10M characters / month
Full disambiguation ruleset (emoji + CJK + bilingual)
Sidecar & replace modes
Community support

Start free

Most common

Growth

$300/mo

Flat monthly rate with a 1 billion character allowance — then just $0.30 per additional 1M characters.

1B characters / month included
Pennies per additional 1M characters
Priority routing & SLA-backed p99 latency
Audit trail + exact-restore guarantee
Email & Slack-connect support

Buy now — BTC or card

Enterprise

$8,000/mo starting

Flat-rate contracts at scale, VPC / on-prem deployment, dedicated support.

Volume-committed flat monthly fee
On-prem / VPC deployment option
Model-version canary + migration tooling
Dedicated solutions engineer

Talk to sales Need it off our cloud? Three delivery models ↓

04b — Enterprise delivery

Three ways to run this that aren't our cloud.

The tiers above run on our multi-tenant SaaS. Plenty of enterprise buyers can't use that — the data can't leave their boundary, the network has no egress, or they need to ship it inside their own product. So pick the shape you actually need. Tell us which one and we'll scope it.

These are contracted, not self-service. Prices are indicative starting points for scoping — the real number depends on volume commitment, environment count, and support tier. Selecting a model below records your requirement and starts a conversation with a human engineer. It does not provision anything and does not charge you.

Dedicated BYOC

Your cloud account. Our operators.

from $12,000 /mo · annual commit

We deploy a single-tenant gateway into your own AWS, GCP, or Azure account and operate it for you. Your VPC, your KMS keys, your network policy, your cloud bill for the compute — text never crosses into our infrastructure or a shared tenant.

Data stays in: Your cloud account, your region
Who operates it: Us, via a scoped IAM role
Isolation: Single-tenant compute & storage
Typical lead time: 2–4 weeks from order form

Terraform / Helm deploy, pinned image digests
No shared queues, no noisy neighbours
Your KMS keys — we hold no plaintext copy
PrivateLink / Private Service Connect / peering
99.9% availability target, named escalation path
Quarterly model-version canary on your own corpus

Best for: regulated teams whose data can't leave their cloud boundary, but who don't want to staff an on-call rotation for a normalization layer.

Most requested

Self-Hosted Appliance

Runs entirely inside your perimeter. Air-gap capable.

from $144,000 /yr · per production environment

A versioned container bundle plus Helm chart and offline model artifacts that you run on your own Kubernetes or bare metal. No outbound calls required at runtime — it runs fully air-gapped, including in networks where egress isn't a preference, it's a compliance control.

Data stays in: Your datacenter. Zero egress.
Who operates it: Your platform team, our runbooks
Isolation: No dependency on our infrastructure
Typical lead time: 4–8 weeks from order form

Signed OCI images with SBOM + provenance attestation
Helm chart, air-gapped install path, offline license file
Full ruleset local (emoji + CJK + bilingual)
Pinned upgrade channel with rollback procedure
Named support engineer, 8×5 or 24×7 escalation
Hardening guide + DR/restore procedure

Best for: defense, healthcare, and finance workloads where an egress rule is a control you can't get an exception for.

OEM / Embedded License

Ship it inside your own product.

from $250,000 /yr + per-unit royalty

License the disambiguation engine as a library you link into your own product and redistribute to your customers under your brand. Source-available under a commercial OEM agreement, with the right to sublicense downstream.

Data stays in: Wherever your product runs
Who operates it: Your engineers, in-process
Isolation: In-process library, no network hop
Typical lead time: 6–10 weeks incl. license terms

Source-available license to normalization + fusion core
Right to redistribute and sublicense to end customers
White-label / no-attribution option
Integration review + architecture enablement sessions
Patch stream + coordinated security disclosure
Your own version numbers and release cadence

Best for: platform, search, and RAG vendors who need this inside their own SKU rather than as an upstream dependency their customers can see.

Not sure which one fits? Staying on our multi-tenant SaaS? Nothing to request — just pick a tier above.

Real customer economics — 20B characters/month example

Computed directly from the published Growth-tier rates above — not rounded marketing math.

Monthly volume

20B characters

Baseline monthly spend

$400,000 at $20/M chars

Fraction of failures avoided

10% conservative estimate

Gross cost avoided

$40,000 / month

Gateway monthly fee

$300 Growth tier, 1B chars included

Gateway overage cost

$5,700 19,000 × $0.30 per extra 1M

Net monthly savings

$34,000 / month

Customer ROI on gateway spend

6.7× return

Break-even requires avoiding just 1.5% of failure-driven cost — the measured collapse rates above are an order of magnitude larger than that threshold.

05 — Details

Questions engineers actually ask.

Does this replace my embedding model?

No. It runs in front of it. You keep your existing model, vector database, and retrieval stack — the gateway just makes sure distinct emoji/CJK/bilingual input actually produces distinct vectors before it hits your embedder.

Do you ever discard or rewrite my original text?

Never. The native input is preserved byte-for-byte with a versioned, provenance-bearing companion representation traveling alongside it. Restoration is exact in 100% of tested cases.

What happens with emoji or characters you don't recognize?

Unknown sequences are preserved untouched and flagged rather than guessed at — a deliberate fail-safe for Unicode and model-version drift.

How was this validated?

Neural validation was run on real embedding models on an RTX 5090: MiniLM, bge-base-zh-v1.5, and bge-m3. Emoji fix accuracy was measured on a 40-item bilingual holdout set, and throughput was load-tested at 1,000 concurrent requests. CJK disambiguation was measured separately on eight semantic-opposite pairs with all-MiniLM-L6-v2 at full precision: three pairs sat at cosine 1.0000 (literally the same vector) and all three were broken, with zero pairs remaining above 0.95. We publish the pairs that barely moved alongside the ones that did — see the CJK table above and the "what we do not claim" note under it. Read the research for full methodology.

Does the CJK fix cover every Chinese character?

No, and that is deliberate. The built-in lexicon covers 130 high-confidence senses concentrated on the vocabulary where collapse does commercial damage — trading direction, money flow, account actions, and approve/reject decisions. Characters whose meaning genuinely depends on context (行 háng "row/firm" vs xíng "walk/okay") are intentionally not guessed. Those spans come back untouched with requires_multilingual_route: true so you can route them to a multilingual embedder. A wrong gloss would move the vector somewhere confidently wrong, which is worse than an honest flag. You can override or extend any entry per-request with chinese_glosses, and a caller-supplied gloss always wins over ours.

Is this a translation service?

No. We don't translate or replace Chinese, Japanese, or Korean text with English. We preserve the native original and add a disambiguating semantic companion — native-language retrieval quality is never sacrificed.

What's the integration effort?

A drop-in FastAPI-compatible endpoint. Most teams swap one function call — embed_model.encode() → sig_gateway.encode() — and ship the same day.