AI Primer: Inference, Tokens, and Weights

[Download the full illustrated primer (PDF) →](assets/uploads/AI-Primer-Inference-Tokens-Weights.pdf)

AI can seem far more mysterious than it really is.

The conversation around artificial intelligence is full of terms such as tokens, weights, parameters, training, inference, context windows, latency, and throughput.

They sound technical — and they are — but the underlying ideas can be understood without mathematics or a computer-science degree.

This primer focuses on three ideas that explain a surprising amount of how modern AI works:

Tokens. Weights. Inference.

Understand those three, and discussions about AI models, inference hardware, prompting, hallucinations, context windows, model size, and even the enormous computing infrastructure behind AI start becoming much easier to follow.

No math. No formulas. No mysticism.

Just a practical mental model for understanding what happens between the moment you type a prompt and the moment an AI answers you.

The Difference Between Training vs. inference

Before talking about tokens and weights, there is one distinction that makes everything else easier:

Training teaches the model. Inference uses the model.

Training

Training is when the model is being built and adjusted.

It is exposed to enormous amounts of examples and repeatedly changes its internal settings so that it becomes better at recognizing patterns and producing useful outputs.

Think:

learning, adjustment, construction.

Inference

Inference happens after training, when the trained model is actually used.

You give it an input. It applies what it learned and produces an output.

Think:

using, responding, generating.

During an ordinary conversation with an AI, the model is not normally retraining itself every time you ask a question. It is using a trained collection of internal values to respond to the information you have just provided.

That distinction also matters economically.

Training is generally the expensive build phase. Inference is the everyday operating phase.

Once a model becomes popular, inference may happen millions or billions of times. At that scale, response speed, memory use, power consumption, hardware efficiency, and operating cost become enormous practical concerns.

What is a token?

A token is a small chunk of input or output that the model works with.

People often say an AI “reads words,” but a language model does not usually receive text exactly as you see it.

Before the model processes a sentence, the text is broken into smaller units called tokens.

A token might be:

• a whole word
• part of a word
• punctuation
• a number
• a symbol
• another small unit of text

A short and common word might fit into a single token. A long or unusual word might be split into several.

Names, slang, invented terms, programming code, mixed languages, and even typos can still be represented because the system can work with pieces rather than requiring every possible word to exist as one fixed entry.

The exact split varies from one tokenizer to another.

A useful way to remember it is:

Tokens are the bite-sized pieces the model reads and writes.

Tokenization and context

Before a model can respond, your input must first become something it can process.

That preparation step is called tokenization.

In simplified form:

Tokenization itself is not intelligence. It is more like preparing material for the model to work with.

The context window

The model can only work with a limited amount of active information at one time.

That working space is commonly called the context window.

Depending on the system, it may contain your current prompt, previous parts of the conversation, instructions, documents, retrieved information, tool results, and other material relevant to the current task.

A larger context window means the model can consider more material at once.

It does not automatically mean the model understands every detail perfectly, and it is not the same thing as permanent memory.

A useful mental picture is:

Context is the model's workbench. Tokens are the things placed on that workbench for the current task.

What are weights?

If tokens are what the model is given, weights are what the model learned.

During training, the model is adjusted again and again.

Those adjustments gradually shape an enormous network of numerical relationships. The resulting learned values are called weights.

In plain English, weights represent the model's learned internal tendencies: which patterns matter, which concepts tend to relate, what structures tend to follow other structures, and which kinds of responses tend to fit particular situations.

Weights are not:

• a folder full of sentences
• a neat encyclopedia
• a list of every fact the model has ever encountered
• a traditional database containing one row for every possible answer

They are closer to:

• learned tendencies
• pattern relationships
• internal adjustments
• the accumulated result of training

So the shorthand becomes:

Tokens are what the model sees. Weights are what the model learned.

Weights, parameters, and the model

You will also often hear AI companies describe models as having millions, billions, or even more parameters.

In everyday AI discussion, parameters and weights are sometimes used almost interchangeably.

Technically, parameter is the broader term. Weights are the main type of learned parameter people are usually referring to.

Does more parameters mean better AI?

Not necessarily.

A larger model has more internal capacity, but capability also depends on things such as model architecture, training data, the training process, post-training, context handling, tools, and many other design decisions.

A smaller model designed well for a particular job can outperform a much larger model at that task.

So where is the knowledge?

Much of what a model learned is distributed throughout its weights rather than stored as neat, human-readable facts.

There generally is not one particular weight you can point toward and say:

“That one contains Paris.”

A better analogy is a musician.

A musician does not store every acquired skill as a written sentence somewhere in the brain. Years of experience become instincts, associations, habits, expectations, and learned relationships.

Weights are closer to learned instincts than pages inside a filing cabinet.

What is inference?

Inference is the moment the trained model actually does its job.

You give the model something.

It processes that input using its trained parameters.

Then it produces something back.

For a language model:

Prompt → answer

For an image model:

Description → image

For speech recognition:

Audio → transcript

For a vision system:

Image → description or decision

Every ordinary chatbot response is an inference run.

Every AI-generated image involves inference.

Every time a speech model converts audio into words, inference is taking place.

Training builds the capability. Inference spends that capability on a task.

What happens when you ask an AI a question?

Consider a simple prompt:

“Explain inference in plain English.”

A heavily simplified version of what happens looks like this:

1. You type the prompt.
2. The text is tokenized. Your sentence becomes model-ready pieces.
3. The model processes those tokens. Its learned weights help determine relationships between the words, concepts, instructions, and context.
4. It estimates what should come next. There may be several plausible next tokens.
5. A token is selected.
6. That new token becomes part of the context.
7. The model predicts again.
8. The reply grows token by token.

What appears to you as a finished paragraph is being constructed piece by piece.

That leads to an important distinction:

The model is not normally uncovering a finished paragraph hidden somewhere inside itself. It is constructing a response as it goes.

How tokens, weights, and inference fit together

These are not three unrelated pieces of AI jargon.

They are parts of the same process.

Tokens → Weights → Inference

You provide tokens.

The model processes those tokens using relationships encoded in its trained weights.

Inference is the live computational process of doing that and generating new tokens as output.

The shortest version worth remembering is:

Tokens are the pieces. Weights are the learned tendencies. Inference is the act of using those tendencies on those pieces.

Or another way:

Training creates the weights. Your prompt becomes tokens. Inference runs those tokens through the trained model to generate new tokens.

Why AI does not simply “look up” an answer

A language model is not merely a search engine with better grammar.

When you ask a trained language model a question, it generally generates its reply from patterns encoded in its weights combined with the context you supplied.

It is not necessarily consulting a hidden encyclopedia and copying a stored paragraph.

That distinction explains a lot of AI behavior.

It explains flexibility

The model can express the same concept in many different ways and combine ideas into new forms.

It explains creativity

A response can contain combinations that were never stored as one exact pre-written answer.

It also explains hallucinations

A continuation can fit the learned pattern extremely well while still being factually wrong.

And it explains why verification matters

Fluent language is not the same thing as guaranteed truth.

Modern AI systems can also use search engines, databases, code execution, external tools, or retrieved documents.

When they do, retrieval and generation work together.

But the model is still performing inference to determine how to interpret and use the retrieved information.

Why prompts and context matter

During an ordinary conversation, the trained weights remain essentially the same.

What changes is the input.

Compare these prompts:

“Tell me about AI.”

and:

“Explain inference, tokens, and weights for a beginner. Use plain English, no formulas, and include a real-world analogy.”

The second gives the model much more information about the topic, audience, format, and level of detail you want.

You did not retrain the model.

You changed the active tokens and context it has available.

That is why a useful way to think about prompting is:

Prompting is steering, not rebuilding.

A prompt chooses a route through a very large learned landscape. It does not normally rewrite that landscape.

Why the same prompt can produce different answers

Generative AI is often designed to choose among several plausible continuations rather than always producing one fixed sentence.

At many points during generation, more than one next token may make sense.

Depending on how the system is configured, it can make very conservative choices or permit more variation.

That is why repeating the same prompt can produce replies that are similar in meaning but different in wording, structure, examples, or tone.

This is not automatically evidence that the model is confused.

Sometimes there are simply many reasonable ways to answer.

But it also means that an open-ended generative model should not be treated exactly like a calculator.

For important factual work:

consistency, confidence, and fluency are not substitutes for verification.

What does an inference chip actually do?

An inference chip is hardware designed to run trained AI models efficiently.

The chip is not the intelligence itself.

The model's learned behavior comes from its trained architecture and parameters. The hardware performs the enormous amount of computation required to apply those parameters to incoming information and generate outputs.

Inference hardware can help provide:

• faster responses
• lower latency
• greater throughput
• support for many simultaneous requests
• lower power consumption
• more efficient movement of model data

But the chip does not, by itself, contain:

• a personality
• the meaning of your prompt
• a hidden library of finished answers
• a tiny person sitting inside deciding what to say

A useful analogy is:

The model is the trained brain. The inference chip is the high-performance engine running it.

Why “performance per watt” matters

At enormous scale, simply being fast is not enough.

An AI system also has to be efficient.

If an AI service handles huge numbers of requests, even small improvements in processing speed, energy consumption, memory efficiency, or hardware utilization can translate into substantial operational savings.

Power matters

AI hardware consumes electricity and produces heat.

More electricity can mean more cooling, more infrastructure, and more operating expense.

A processor capable of doing the same useful AI work with less energy may therefore be more valuable than another processor that is merely faster in isolation.

Throughput matters

Throughput describes how much work a system can process over time.

For an AI service, that can mean how many requests or tokens the infrastructure can handle while still providing acceptable response times.

Which gives us the TGP shorthand:

Don't work harder. Work smarter. Performance per watt.

And no, before Jensen's lawyers arrive — that's our running joke, not a Jensen Huang quote. 😆

The same basic pattern appears everywhere

The specific technologies vary, but the broad relationship appears across many AI applications.

A chatbot

Your question becomes tokens. The model applies its trained weights during inference and generates output tokens that become its reply.

An image generator

Your description is encoded into something the model can process. The system uses learned patterns to generate an image.

That generation is inference.

Speech recognition

Audio is converted into model-friendly representations. The trained system infers which words are most likely being spoken and outputs a transcript.

A coding assistant

Your code, comments, and instructions become context. The model applies learned programming patterns to infer a completion, explanation, or proposed patch.

A recommendation system

The system receives signals about the current situation and applies learned parameters to infer which item, ranking, or action may be most appropriate.

Different applications. Same broad principle:

Input comes in. Learned parameters are applied. Output comes out.

Common misunderstandings

There are a few shortcuts that make AI sound easier to explain but ultimately create the wrong mental model.

❌ “A token is a word.”

Sometimes — but not always. Tokens can be whole words, parts of words, punctuation, symbols, and other units.

❌ “Weights are stored facts.”

Not quite. Weights are better understood as learned internal values encoding distributed patterns and relationships.

❌ “Inference means the AI is learning while answering me.”

Ordinary inference usually operates using fixed trained weights. Training is a separate process.

❌ “The chip is the AI.”

No. The chip is hardware used to run the model. The learned behavior comes from the architecture and trained parameters.

❌ “If it sounds confident, it knows.”

Nope.

Fluency is not certainty. An answer can be articulate, detailed, and completely wrong.

A better overall view is:

AI is a trained generative system. It receives structured input, applies learned patterns, and produces an output through inference.

The mental model worth keeping

If most of the technical vocabulary disappears from memory tomorrow, keep these four ideas:

Tokens = ingredients

The individual pieces available for the current task.

Weights = learned instincts

The accumulated skills, associations, tendencies, and experience acquired during training.

Inference = cooking

The live process that uses what is available and what has been learned to produce something.

Training = culinary education

The long process that created those instincts in the first place.

No analogy is perfect, but if you can explain those four boxes to somebody else, you understand the relationship well enough to follow most mainstream conversations about modern AI.

Or, in one sentence:

AI takes your input, turns it into model-ready pieces, processes those pieces through learned internal patterns, and generates an output.

That's it.

Not simple technology.

But a much simpler mental model than the vocabulary initially makes it sound.

Quick reference

Architecture — the overall design of a model and how its major parts are arranged and interact.

Context — the active information available to the model for the current task.

Context window — the amount of active tokenized information the model can consider at once.

Inference — using a trained model to produce an output from an input.

Inference chip — hardware optimized to run trained AI models efficiently.

Latency — how long you wait for a response, or part of one.

Model — the trained computational system that transforms inputs into outputs.

Parameter — a learned internal value in a model; weights are the main parameters people usually mean.

Prompt — the instructions or input you give the model.

Throughput — how much work a system can process over time.

Token — a small unit of input or output used by the model.

Tokenization — converting raw text into tokens.

Training — the process that adjusts the model's parameters so that useful patterns are learned.

Weights — learned internal settings that shape how the model responds.

Same questions. A brighter tomorrow.

[Download the full illustrated primer (PDF) →](assets/uploads/AI-Primer-Inference-Tokens-Weights.pdf)