Token & Embedding Explorer

Type a sentence to see how it gets broken into tokens, then explore a toy "vector space" where related words sit near each other -- the same two ideas behind how language models actually read text.

How this works: the tokenizer is a simplified greedy longest-match splitter over a curated ~350-piece vocabulary, not the learned byte-pair-encoding a real model uses. The word map's positions are hand-arranged by category, not computed from a real trained embedding -- but the core idea it illustrates (similar meaning → nearby location) is the real one.

Tokenizer

Dashed chips are pieces that weren't in the toy vocabulary as a whole word, so the tokenizer fell back to a smaller chunk. Solid chips with a highlight border are clickable, they also exist in the word map on the right.

Word map

Type a word above, or click a point on the map.

Why this matters

Tokens, not words

Language models never see whole words directly -- they see tokens, and a token can be a whole word, a fragment of one, or a single character. Common words are usually one token; rare, long, or made-up words get split into several. Try typing a long invented word above and watch it fragment.

Embeddings, not lookup tables

Each token gets mapped to a vector -- a long list of numbers -- and that vector is positioned so that tokens used in similar contexts end up near each other in that space. This toy map fakes that by hand-assigning categories, but real embeddings learn this arrangement automatically from how words actually appear together across huge amounts of text.

Why this matters for how models "think"

Once text becomes vectors, everything downstream -- attention, prediction, generation -- is really just math on those vectors. "Meaning" in a language model is entirely a byproduct of geometry: words end up close together because they were used similarly, not because the model looked up a definition anywhere.