A new technical tool has been developed to generate "token-space fonts," where every token from a specific LLM tokenizer is rendered with an equal width. By incorporating a tokenizer's logic into the font's shaping rules, the tool enables users to visually represent text such that each token occupies a consistent amount of horizontal space.
New tool generates token-space fonts with constant width for LLM tokens
The tool supports various popular tokenizers, including DeepSeek, OpenAI, Kimi, Qwen, GLM, Llama 3, and Gemini. Users can upload their own TTF, OTF, WOFF, or WOFF2 files to use as a base or provide a Hugging Face repository link to download a specific tokenizer. The resulting output is a single TTF file that does not require a separate script for display.
The implementation uses Pyodide and fontTools for local compilation. While the tool aims to provide a visual approximation of token boundaries, the developers note that browser shaping, line breaks, and complex script normalization can still affect the exact visual result. The project includes a whitespace audit to compare different font binaries against local tokenizer references to ensure accuracy.
Sources
- Generate fonts where every LLM token is the same width （Hacker News Frontpage, 2026-09-26）