Skip to content
BuildTuber

Token

Also called tokens · tokenisation · tokenization

The unit a language model reads and writes in, and the unit providers bill by. Roughly a short word or a fragment of a longer one.

In more detail

Models do not process text as characters or as whole words but as tokens, produced by splitting text on statistical boundaries. Common words are usually one token; longer or unusual words break into several. Providers publish separate prices for input tokens (what you send) and output tokens (what the model generates), and output is normally the more expensive of the two.

Example

A rough working figure for English prose is about 0.75 words per token, so a 15,000-word script is on the order of 20,000 output tokens. Because that is a ratio rather than a rule, the only reliable count is the one the provider reports back on the response.

Why it matters

Token counts are what turn a vague sense of "this got expensive" into an arithmetic you can check. Once you know a provider's input and output rates, cost per script stops being a mystery and becomes multiplication. It also becomes obvious that re-sending a long context on every call, which is how many multi-step pipelines work, is often the thing actually driving the bill.

Get the build guides as they go out.

One email when a new guide ships. The full method, not a teaser. Unsubscribe whenever.