README.md

---
title: TokenizerBench
emoji: 🤗
colorFrom: yellow
colorTo: gray
sdk: gradio
sdk_version: 6.11.0
app_file: app.py
pinned: false
license: mit
short_description: Evaluate & compare tokenizers
---

# TokenizerBench

Evaluate any Hugging Face or tiktoken tokenizer against the **TokenizerBench** dataset — covering multilingual text, programming languages, scientific formulas, and edge cases.

## Features

- **🧪 Playground** — type any text and see live tokenization (token IDs, fertility, compression, fidelity check)
- **📊 Evaluate** — run a full evaluation on a single tokenizer with heatmap, language bar chart, and scatter plot
- **⚖️ Compare** — compare two tokenizers side-by-side with grouped bar charts and a leaderboard

## Dataset categories

| Category | Subcategories |
|----------|--------------|
| 🌍 Human languages | English, Hindi, Chinese, Arabic, Japanese, German, Russian, Korean |
| 💻 Programming languages | Python, JavaScript, SQL, Rust |
| 🧮 Scientific formulas | Algebra, Calculus, Physics, Statistics |
| ⚠️ Edge cases | Whitespace, Long tokens, Mixed scripts |

## Metrics

| Metric | Better | Notes |
|--------|--------|-------|
| `avg_fertility` | Lower | Tokens per word. Near 1.0 = ideal. ≥4 = poor. |
| `avg_compression_ratio` | Lower | Tokens per character. |
| `avg_byte_compression` | Lower | Tokens per UTF-8 byte. Language-agnostic. |
| `fidelity_pass_rate` | 1.0 | Must be 1.0 — any failure indicates a problem. |

## Supported tokenizer types

- **HuggingFace AutoTokenizer** — any model from the Hub, e.g. `bert-base-multilingual-cased`, `xlm-roberta-base`, `google/mt5-base`
- **tiktoken** — OpenAI encodings: `cl100k_base`, `o200k_base`, `p50k_base`