What is an AI token?

If you've spent any time with Large Language Models (LLMs) like Llama 3, GPT-4 or Claude, you've likely encountered the term "Token." Whether it's in a pricing table ("$0.01 per 1M tokens") or a context window limit ("128k tokens"), tokens are the fundamental currency of AI.

But what exactly is a token? Is it a word? A character? A piece of a word?

The short answer

A token is the atomic unit of text that an AI model processes. You can think of it as a chunk of text. Depending on the tokenization method, a token can be as short as a single character or as long as a whole word.

On average, for English text, 1,000 tokens is roughly equal to 750 words.

The deep dive: how tokenization works

Computers cannot "read" letters or words the way humans do. They only understand numbers. To bridge this gap, LLMs use a process called Tokenization.

1. From text to IDs

The journey from your prompt to the AI's brain looks like this: Raw Text $\rightarrow$ Tokens $\rightarrow$ Token IDs (Numbers) $\rightarrow$ Vectors (Embeddings)

2. Why not just use words?

Using whole words is inefficient. If a model had to learn every single variation of a word (e.g., "walk," "walking," "walked," "walker"), its vocabulary would be astronomical.

3. Why not just use characters?

Using single characters (a, b, c...) is too granular. The model would have to spend too much computational "effort" just to realize that h-e-l-l-o forms a single concept.

4. The solution: subword tokenization (BPE)

Most modern AIs use Byte Pair Encoding (BPE). This method splits common words into single tokens and rare words into multiple sub-tokens.

Example:

• The word apple is common $\rightarrow$ 1 token [apple]
• The word tokenization might be split $\rightarrow$ 2-3 tokens [token] [iz] [ation]

This allows the model to understand the root of a word and its suffixes, enabling it to handle words it has never seen before by combining known pieces.

Why tokens matter to you

1. The context window

Every model has a "context window" (e.g., 8k, 32k, 128k tokens). This is the model's "short-term memory." Once your conversation exceeds this limit, the model starts "forgetting" the earliest parts of the chat to make room for new tokens.

2. Cost and performance

Since AI providers charge per token, the efficiency of the tokenizer directly impacts your bill.

• Inefficient tokenization (splitting simple words into many pieces) = Higher cost & slower response.
• Efficient tokenization = Lower cost & faster speed.

3. The "math" problem

You might notice some LLMs struggle with simple math or spelling. This is often because of tokenization. If a model sees the number 12345 as two tokens [12] and [345], it isn't seeing the digits individually, which can lead to calculation errors.

Summary table

Next time you see a token count, remember: you're looking at the fragmented, numerical puzzle that the AI uses to reconstruct human thought.