Ask a chatbot a question and it answers in fluent sentences, in any language, on nearly any topic. The trick behind this apparent intelligence is both simpler and stranger than it looks. A large language model does not “think” in words the way a person does. At its core, it performs one operation, over and over: given a sequence of text, predict the most likely next piece of text. Everything else — the essays, the code, the conversation — emerges from repeating that single step thousands of times.

Tokens: The Model’s Real Units of Work

Models do not read raw characters or whole words. They work with tokens — sub-word chunks produced by a tokenizer, most commonly using an algorithm called Byte-Pair Encoding (BPE). Common words may become a single token; rare or invented words get split into pieces. For example, “unbelievable” might be broken into “un”, “believ”, and “able”.

Tokens are the model’s true unit of accounting: as a rule of thumb, one token is about three-quarters of an English word, and 1,000 English words typically become around 1,300 tokens. Each token is converted into a unique number, because neural networks compute with numbers, not letters. This is also why AI providers bill per token rather than per word — tokens are what the model actually processes.

The Transformer: An Architecture Built on Attention

Nearly every model behind today’s chatbots is built on the same architecture: the Transformer, introduced in the 2017 paper “Attention Is All You Need” by Vaswani and colleagues at Google. Before it, language models processed text one word at a time, like reading down a telephone chain. The Transformer processes an entire sequence in parallel — more like a group chat where everyone hears everyone at once.

Its core mechanism is self-attention: for each token, the model weighs how relevant every other token in the input is. In the sentence “The cat sat on the mat because it was tired,” self-attention helps the model link “it” to “cat” rather than “mat.” Mathematically, every token is projected into three vectors — called query, key, and value — and the model computes weighted combinations of them across multiple “attention heads” in parallel, each head learning different kinds of relationships, from grammar to meaning.

Since attention itself is blind to word order, the model also receives positional encodings — signals that tell it where each token sits in the sequence. Stacked dozens of times over, these layers build an ever-richer representation of the input, which is then used to predict the next token.

Training: From Prediction to Helpfulness

An LLM’s abilities are shaped in stages. Pre-training exposes the model to internet-scale text and asks it to do one thing: predict the next token, billions of times. Through this, it absorbs grammar, facts, and patterns of reasoning — encoded in its billions or trillions of learned parameters, or “weights.”

But a pure prediction machine is not a helpful assistant. Fine-tuning on instructions and examples teaches the base model to follow requests and answer questions. A further stage, often reinforcement learning from human feedback (RLHF), has human reviewers rank the model’s responses so it learns to prefer helpful, accurate, and safe answers. It is this alignment process, layered on top of raw prediction, that makes a model feel conversational.

Choosing the Next Token: Temperature and Sampling

For each step, the model outputs a probability distribution over its entire vocabulary — its best guess at how likely each possible next token is. How the final token is chosen is controlled by decoding parameters:

• Temperature: a single number that sharpens or flattens the distribution. A low temperature (near 0) makes the model almost always pick the top token — deterministic, factual, sometimes dry. A high temperature (near 1 or above) gives lower-probability tokens a real chance — more creative, more surprising, and more error-prone. Technically, the temperature divides the model’s raw scores (logits) before probabilities are computed.
• Top-k: only the k most likely tokens are considered, discarding the long tail of bizarre options entirely.
• Top-p (nucleus sampling): instead of a fixed number, the model considers the smallest set of tokens whose probabilities add up to p (often around 0.9 or 0.95) — dynamically adapting the pool to the model’s confidence.

Once a token is chosen, it is appended to the input, and the whole process repeats — one token at a time, until the model emits an end-of-sequence signal. Generation is thus a loop: predict, append, predict again.

Why Models Hallucinate

The prediction-loop design explains the most famous weakness of LLMs: hallucinations — fluent, confident statements that are simply wrong. The model does not check facts; it samples tokens that are statistically plausible. At higher temperatures, an unlikely token can slip through, and because every subsequent token is conditioned on everything before it, one wrong choice can steer the whole answer down a path disconnected from reality. Errors compound.

This is also why the same question asked twice can get different answers: sampling introduces randomness by design. Lower temperatures and narrower sampling pools reduce that randomness and the hallucination risk, at the cost of variety — a trade-off engineers tune for each use case, from cautious legal drafting to open-ended storytelling.

FAQs

What is a token in an AI language model?
A token is the chunk of text a model actually reads and writes — a sub-word unit produced by a tokenizer. Common words may be one token each; rare words get split into pieces. Roughly speaking, one token equals about three-quarters of an English word, and AI services bill by tokens because tokens are the model’s real unit of work.

What is a transformer in AI?
The Transformer is the neural-network architecture behind almost all modern language models, introduced by Vaswani et al. in the 2017 paper “Attention Is All You Need.” It processes text in parallel using a mechanism called self-attention, which lets every word weigh its relationship to every other word — replacing the older approach of reading words one at a time.

How do LLMs generate text if they only predict the next token?
Generation is a loop: the model predicts a probability distribution over the next token, one token is sampled from it (influenced by settings like temperature), that token is appended to the input, and the model predicts again. A paragraph of text is thousands of these tiny predictions chained together.

What is temperature in AI text generation?
Temperature controls how risky the model is when picking the next token. Low temperature (near 0) makes the model almost always choose the most probable token — deterministic and reliable. High temperature flattens the probabilities, letting unlikely tokens through — more creative but more likely to hallucinate.

Why do AI models sometimes make things up?
Because they generate statistically plausible text rather than retrieving verified facts. If a low-probability token gets sampled, every later token builds on that mistake, so errors compound. This is called hallucination, and it is more likely at high temperature settings or with very open-ended prompts.

Compiled by the Khabar 24h Editorial Desk from publicly available sources.