What Do Words, Poems and AI Tokens Have in Common?

In short

A token is a small thing that stands for something bigger. A word is one. A poem is one. An NFT is one. An AI token is a different kind, and the difference is worth understanding.

AI models hold patterns from a huge amount of human writing, with real limits. The skill that makes that useful to you is old: noticing a pattern in the field you know and carrying it into a field that has not seen it. This article walks through the idea in plain words and ends with a way to try it this week.

A word you already use

Token comes from an old English word for a sign or a mark. You have held tokens. A subway coin is a small piece of metal that stands for a ride. A ticket stands for a seat. A badge stands for a right to walk through a door. The object is small and cheap, and what it stands for is bigger.

That is the whole idea. A token is a small stand-in that points to something larger, and it works because the people using it agree on what it points to.

Words are tokens

Say the word home. For you, it points to a whole set of rooms, smells, people and evenings. Someone who grew up elsewhere has their own set. The word works between you because your experiences overlap enough. It does not carry your home across to them. It points, and they fill it in from theirs.

This works in any language. Home, 家, hogar and heimat are different tokens that point at overlapping pieces of human life. Where two people have not shared an experience, the word comes across thin. Try explaining a first snowfall to someone who has never seen snow. The word is there. The weight behind it is not.

A metaphor is a token that points to a longer story in a few short words. The linguist George Lakoff and the philosopher Mark Johnson made the case in their 1980 book Metaphors We Live By that we think in these. Take 'time is money'. We spend time, save time, waste time and run out of it. Three words import a whole pattern from the world of money into how we handle our days.

A poem pushes this as far as it goes. A few lines point at something large, and the reader has to rebuild it from their own life. Two readers rebuild two different things, and neither is wrong. The compression is deliberate, and the reader finishes the work.

Tokens in computing

Computing uses the same idea in three ways.

A login token is a short string of characters that stands for the fact that you proved who you are. The system checks the token instead of asking for your password again. It stands for permission.

An NFT, short for non-fungible token, is a record on a blockchain that stands for ownership of one particular thing. The record is often small. It usually holds an ID and a link, and the picture or file it refers to is typically stored somewhere else. So an NFT is a stand-in for something bigger in the same way a ticket is, and it works because many people agree to treat that record as proof.

An AI token is the odd one out. It is a piece of text that a language model reads and writes. In English it is often a word or part of a word, and a rough rule of thumb is that a token is about three quarters of a word. 'Unbelievable' may be split into a few tokens.

Tokens are practical to know about. A model can only read so many at once, which is its context window, so a very long conversation can push your earlier words out of view. Many services also charge by the token, so a long request costs more than a short one. Those two facts explain a lot of everyday AI behaviour.

Where the meaning lives

Here is the difference. A subway coin, a login token and an NFT stand for something. An AI token, by itself, stands for nothing. It is a chunk of text. 'Un' means nothing alone.

So where does the meaning come from? The linguist J.R. Firth put the answer in one line in 1957: you shall know a word by the company it keeps. Ludwig Wittgenstein, a few years earlier, argued that for many words, meaning is use: what a word means is how people use it. A model learns from the company tokens keep across a very large amount of writing. The meaning lives in the pattern of relations between tokens, not in any single piece.

Linguists already had a word for this split. The philosopher Charles Sanders Peirce, in the early 1900s, separated a type, the word as an abstract unit, from a token, each single occurrence of it. When someone counts how often 'the' appears on a page, they are counting tokens of one type. AI borrowed the name, and the old distinction still helps.

What AI models keep, and what they miss

I want to give credit where it is due. A modern language model holds patterns from an enormous amount of written human knowledge, across medicine, law, music, farming, code, poetry and many languages. You can ask in ordinary words and get back the vocabulary of a field you have never studied. A person could not read that much in many lifetimes. That is a real achievement, and it is built on the work of everyone who wrote things down.

It is also lossy. The model compresses what it read, and compression drops detail. It can only keep what was written and put online, which leaves out a great deal: knowledge passed on by showing, not telling, the know-how of a craftsperson, and the languages and communities that have little text on the web. Its coverage is uneven, and it is thinnest where the least has been written.

There is a deeper limit, which the researcher Stevan Harnad named the symbol grounding problem in 1990. Your words connect to things you have lived. A model's tokens connect to other tokens. Whether that counts as meaning is still argued about, and I do not think the argument is settled. What is safe to say is that a model has not lived what the words point at, and it can sound sure when it is wrong. Check anything that matters.

The skill: carry a pattern into a new field

A metaphor moves a pattern from one field into another. 'Time is money' moves the shape of spending and saving into how we think about days. That move is old, and it is behind many useful ideas.

A well-known example is Velcro. As the story is told, the Swiss engineer George de Mestral noticed burrs clinging to his dog's coat after a walk in 1941, looked closely at the tiny hooks, and carried that pattern into a new kind of fastener. Another is the nose of Japan's 500 Series bullet train, which by the usual account was shaped after the kingfisher's beak to reduce the noise at tunnel exits. In both, someone saw a pattern in one field and used it in a field that had not looked at it.

AI models make that move easier to attempt, because they hold patterns from many fields at once. You supply what the model cannot: a pattern you have seen in your own work, and a sense of where it might help. The model supplies the other field's vocabulary, so you do not have to start from zero.

The area does not have to be new to the world. It only has to be new to your team, your group or your organisation. That is easier to find and easier to test.

Try it this week

1. Name a pattern from your field in plain words. Leave out the jargon. 'Things queue up, then a small fix at the front clears the line' is a pattern. 'Throughput bottleneck' is a label for it.

2. Ask an AI model where else that shape appears. For example: 'In my work, [the pattern]. Which other fields have the same shape, and what do they call it?' You will get names and ideas from fields you have not studied.

3. Pick one place your team has not looked at this way and run the smallest test that could show whether the pattern holds.

4. Check it with a person who knows that field. The model has read about it. They have worked in it. They will tell you what the model missed, and that is where you learn the most.

I keep coming back to this idea in my own work. If you would like to hear how I think about it, watch my TEDxCSTU talk, One Skill Behind Every Breakthrough. For the habit of asking clearly, which supports all of this, read How to Do Things with Words?.

A question to take with you

Where could the pattern you know best help someone who has never seen it?

Tools will keep changing. The habit of noticing a pattern, naming it plainly and carrying it somewhere new will not.

Glossary

Token. A small thing that stands for something bigger, such as a coin, a ticket, a word, or in AI, a piece of text.

Tokenization. Splitting text into the pieces, called tokens, that a language model reads.

Context window. The amount of text, counted in tokens, that a model can take in at one time.

LLM (large language model). An AI model trained on a very large amount of text to read and write language.

NFT (non-fungible token). A record on a blockchain that stands for ownership of one particular thing. The thing itself is usually stored elsewhere.

Metaphor. A short phrase that carries a pattern from one area of life into another, such as 'time is money'.

Type and token. In linguistics, a type is a word as an abstract unit, and a token is one occurrence of it.

Symbol grounding. The question of how symbols such as words connect to the things they stand for.

Where these ideas come from

If you want to read the sources behind this article, here they are. Most are free.

Metaphor as thinking: George Lakoff and Mark Johnson, Metaphors We Live By (1980).

Meaning is use: Ludwig Wittgenstein's Philosophical Investigations (1953), introduced in the Stanford Encyclopedia of Philosophy entry on Wittgenstein.

Words get meaning from their company: J.R. Firth's 1957 line is the root of distributional semantics, the idea behind how language models learn. Background on the man: J.R. Firth.

Type and token: Charles Sanders Peirce's distinction, explained in the Stanford Encyclopedia entry Types and Tokens.

Symbol grounding: Stevan Harnad's 1990 paper The Symbol Grounding Problem, with an overview on Wikipedia.

NFTs: the technical standard behind most of them is ERC-721, which shows that a token holds an ID and a pointer to its details.

Old words and sign: the history of the word is in the Online Etymology Dictionary entry for token, at etymonline.com.

Pattern transfer in the wild: Velcro and the 500 Series Shinkansen.

My own thinking on this is in my TEDxCSTU talk, One Skill Behind Every Breakthrough.

Notes

This is one practitioner's plain-language reading of ideas from linguistics and philosophy, not a scholarly survey. The Velcro and bullet-train accounts are the commonly told versions of those stories. Token sizes and prices vary by model and change over time, so check your provider. Names belong to their owners.