See the tokens before you embed

Different text.
Identical tokens.

πŸ”₯, πŸŽ‰, δΉ° buy, and 卖 sell each produce [101, 100, 102] in the BERT-base-uncased reference tokenizer. Four different inputs; the same token-ID sequence. This is a measured tokenizer example, not a test of your model.

Check emoji and Chinese, Japanese, Korean (CJK) text before committing it to an embedding pipeline. Run a free reference audit on a sample, then use Semantic Integrity Gateway to evaluate added text glosses with your own model.

Run free audit See the API example

Try your sample or a bundled example. The report counts exact token collisions and unknown tokens under the BERT-base-uncased reference tokenizer. It does not run your production model or measure search quality.

Open the free audit β€” no email or API key β†’
No email gateΒ·Submitted text is not retained by the audit service
Reference tokenizer Β· original text
πŸ”₯
[101, 100, 102]
πŸŽ‰
[101, 100, 102]
Both contain the unknown-token ID 100.
Gateway Β· augment mode
πŸ”₯
[emoji: fire]
πŸŽ‰
[emoji: party popper]
Original emoji retained; different glosses added. These examples produce distinct reference token IDs after normalization. Retrieval quality still needs testing.
Built for teams running
RAG pipelines Semantic search Content moderation Dedup & clustering Bilingual support tooling
01 β€” What the audit measures

Find distinctions lost at tokenization.

A tokenizer converts text into model inputs. When distinct records become the same token-ID sequence, that sequence no longer carries the distinction. Whether it matters depends on your application and the rest of your pipeline.

500
records per free audit
Paste one record per line, or choose a bundled example.
100k
characters per sample
Up to 20,000 characters per record. Oversized input is rejected, not silently shortened.
1
reference tokenizer
BERT-base-uncased WordPiece. Other listed models are explicitly marked as not evaluated.

A collision is a finding to investigate. Case folding or accent removal may be intentional. Check which distinctions your users need, and reproduce the finding with your actual tokenizer and preprocessing settings.

Read the research, methods and limitations β†’
02 β€” Try normalization

Normalize text, then embed it in your application.

The API returns text and audit metadata. Pass normalized_text to your existing embedder, and retain original_text with the audit metadata. Evaluate the result before changing an index.

HTTP example Β· set SIG_API_KEY to your Developer or paid key
curl --fail-with-body https://api.semantic-integrity.com/v1/normalize \
  -H "X-API-Key: $SIG_API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{"text":"δΉ° πŸ”₯","mode":"augment"}'

Example text result: δΉ° [Chinese: buy] πŸ”₯ [emoji: fire]. The response also includes the original text, span audits, rule identifiers and usage information. It does not return an embedding vector.

API reference β†’

Usage accounting starts September 14, 2026 at 23:29:00 UTC. The current-month allowance starts fresh at this time. Counts for this first month cover only the period since that start. Subsequent months use the full UTC calendar month. How characters, limits and retries are counted β†’

Get a free Developer keyStart with the free audit
03 β€” Validate the benefit

Different tokens are the first check.

These four inputs were checked locally with the deployed normalization implementation and independently cross-checked against the BERT-base-uncased reference tokenizer. Original inputs share the token-ID sequence [101, 100, 102]; each augmented output below has a distinct sequence. This does not measure embeddings, retrieval accuracy or customer savings.

Measured normalization examples Β· augment mode
InputNormalized text
πŸ”₯πŸ”₯ [emoji: fire]
πŸŽ‰πŸŽ‰ [emoji: party popper]
δΉ°δΉ° [Chinese: buy]
卖卖 [Chinese: sell]

Before a rollout: use a free Developer key to compare original and normalized inputs on your actual model and representative labeled queries. Include cases that already work, inspect regressions, and compare latency and processing cost. Glosses can change similarity in either direction; they do not guarantee better rankings.

04 β€” Pricing

Choose an allowance after you evaluate.

The reference audit is free. API plans provide a monthly character allowance for normalization. Your embedding service and its charges remain separate.

Prices below are BTC/on-chain prices β€” our base price. Paying by credit card or fiat costs a flat 16.667% more (fiat price = BTC price Γ— 7/6). Pay the base price on-chain or over Lightning.

This toggle only changes the prices shown. To actually pay, hit Buy now below β€” you'll get a Bitcoin QR code and invoice, or card options.

Developer

$0/mo

For evaluating normalization with your own model and queries.

  • Up to 10M characters / month
  • Emoji glosses and supported CJK senses
  • Original text and span audit metadata
  • 10M-character monthly allowance
Start free
Higher allowance

Growth

$300/mo

Includes 1 billion characters per month. Automatic overage processing is unavailable; requests can be rejected at configured limits.

  • 1B characters / month included
  • Usage metadata in API responses
  • Same normalization API as Developer
  • Original text and span audit metadata
  • Evaluate model compatibility before upgrading
Buy now β€” BTC or card

Enterprise

$8,000/mo starting

Discuss volume, deployment and support requirements before agreeing to a contract.

  • Custom volume requirements
  • Deployment requirements review
  • Model and integration evaluation
  • Scope and terms agreed before purchase
Talk to sales

Plan around the included allowance. Growth includes 1B characters per month. The API price catalog also lists an overage rate, but that is not an enabled automatic billing or processing path. Discuss higher volume before relying on it.

Usage accounting starts September 14, 2026 at 23:29:00 UTC. The current-month allowance starts fresh at this time. Counts for this first month cover only the period since that start. Subsequent months use the full UTC calendar month. How characters, limits and retries are counted β†’

04 β€” Before you integrate

What to expect.

Does the free audit test my embedding model?

It checks the BERT-base-uncased reference tokenizer. The report identifies that scope and does not measure your production model, retrieval quality or financial loss.

What happens to submitted audit text?

The audit service processes the sample to show a report in your browser, then keeps only aggregate audit metadata. It does not retain the submitted text or store the report as a permalink. Use a sample without secrets or personal data.

Does normalization replace my embedder?

No. It returns text that you can pass to your own embedder. In augment mode, available glosses are added alongside the original input. Keep the original text and review the audit metadata.

Does it cover every CJK character?

No. Available glosses cover a limited set of senses. Inspect requires_multilingual_route and the span audits when text needs another route. A preserved character is not proof that your embedder understands it.

Will it improve search quality?

That needs a test on your data. Compare both improvements and regressions on labeled queries. Use a consistent, evaluated preprocessing strategy for documents and queries; changing indexed representations may require re-embedding.

Can I evaluate before paying?

Yes. Run the reference audit without an email or key, then use a free Developer key for up to 10M characters per month through the normalization API. Choose a paid allowance after the evaluation supports it.

Test normalization in your own pipeline.

Get a free Developer key, compare original and normalized text, and measure whether the change helps your users.

Read the research

Developer tier is free β€” no credit card or payment. Growth uses the allowance shown above. Payment methods, prices and access timing are displayed in checkout.