Semantic Integrity Gateway · Reference tokenizer check
Try the reference check

See what a reference tokenizer preserves.

Paste a sample to check the BERT-base-uncased reference tokenizer, or explore a bundled example. This check does not run your production embedding model or measure search quality. No email gate or API key is required, and the service does not retain the submitted text.

Examples use our reference corpus. Unsupported models are labelled as not evaluated.

Use a representative sample. Do not paste secrets or personal data.

What the report answers

  • Screen: exact collision and unknown-token counts on the selected corpus.
  • Explain: when distinct texts become identical inputs to the reference tokenizer.
  • Estimate: which effects were measured and which would need business assumptions.
  • Next: what to verify with your actual tokenizer, model, and retrieval pipeline.

Text retention: the service does not retain your submitted text. You can save the result to your device after the audit; it is not stored as a permalink.