How a language model works

Model weights: what a trained model is#

A trained language model is a huge set of numbers, called weights, plus code that uses them to predict the next word of a text.12 The weights hold what the model learned in training; they are stored in files, and some developers release those files for people to download.342

A language model is a kind of neural network: data passes through layers of units called neurons, which are loosely modeled on the cells of a brain.5 Each neuron gives every input it receives a weight, a number that sets how much that input counts; training is what sets these numbers.56 Weights are also called parameters; the 2020 paper on GPT-3, a language model, described a model with 175 billion of them.76 Training adjusts the weights, while running the model, called inference, only reads them.67 So in normal use, a model adapts only through its prompt, the text it is given, such as instructions or examples; lasting changes usually mean retraining the weights, as in fine-tuning.8736

In February 2023, Meta released its LLaMA model by request only, as a package that people in the AI community could ask to access.19 On March 3, 2023, soon after Meta started taking requests, someone posted a downloadable copy of the model on 4chan.91 The copy then spread across AI communities and set off a debate about how best to share new research.9 Because the weights hold everything the model learned in training, anyone who downloaded that copy had the trained model itself.39

Tokens: how models read and bill text#

A token is a small chunk of text, such as a word, part of a word, a character or a punctuation mark, that a language model reads and writes.10 Each token takes computing power to process, so companies selling pay-as-you-go model access price usage in tokens.10111213

A program called a tokenizer first splits text into pieces from a fixed vocabulary.141516 A common method, byte-pair encoding, keeps frequent words whole and builds rare words from smaller subword pieces.1514 In May 2023, researchers including Aleksandar Petrov found some languages need up to 15 times more tokens for the same text, raising costs.17 Providers can price tokens sent in and tokens written out differently.1012 Models split text differently and write different amounts, so a lower per-token rate does not guarantee a lower bill.10

By February 2023, researchers Jessica Rumbelow and Matthew Watkins had reported strange tokens in the GPT-2 and GPT-3 language models, such as SolidGoldMagikarp.18 The models largely could not repeat these strings, answered in odd ways and reliably insulted one of the researchers.18 Later work traced such glitch tokens to vocabulary entries that almost never appear in training text.16 The case shows that models read tokens, not letters: each string was one indivisible unit the model had barely learned, with no spelling to fall back on.181416

In September 2026, Anthropic released Claude Sonnet 5.5 at unchanged prices of $2 per million input tokens and $10 per million output tokens.19 In October 2026, coverage of the launch reported Anthropic's statement that per-task costs fell because the model finished tasks with fewer tokens.20

Embeddings: turning words into numbers#

A model splits text into tokens and looks up each one in a learned table of vectors, lists of numbers called embeddings, stored in its weights.212223 Training on very large amounts of text sets these numbers so that text with related meanings gets similar vectors.2123

You can picture each vector as a point in a space with one direction for each of its numbers, so vectors with similar numbers sit close together.2422 Training can also turn a relationship between words into a consistent direction from one point to another.21 Search systems apply the same idea to whole sentences, documents and even photos, turning each into one vector and treating the closest vectors as matches.25222627

In 2013, Tomas Mikolov, Scott Wen-tau Yih and Geoffrey Zweig studied the word vectors learned by a language model.21 Taking the vector for king, subtracting the vector for man number by number, and adding the vector for woman gave a vector very close to queen.21 Nobody told the model about gender; training learned that relationship on its own, which is why the arithmetic works.21

The context window: how much a model can see at once#

The context window is a model's working memory: all the text, counted in tokens, that it can refer to while writing a reply, including the reply.2829 Text past that limit is invisible to the model, so systems drop or condense older text to make room.28

The model's weights stay fixed while it runs, so anything it did not learn in training must fit in the window.303128 Chat interfaces such as claude.ai can drop the oldest turns first, and long-running agent systems use server-side compaction to condense earlier text.28 Long windows cost more: standard attention, which relates each token to the others, needs memory and computing that grow with the square of input length.3233 More text is not always better: Anthropic's documentation says accuracy and recall, how reliably a model finds facts in its input, degrade as token count grows, which it calls context rot.28 Models can find one planted fact in a long input yet do much worse on harder long-input tasks, and they often use facts in the middle of a long input worse than facts at either end.3435

In 2024, in the report that introduced the model, Google tested Gemini 1.5 Pro on Kalamang, a language with very few speakers and a single grammar manual.3637 With the grammar manual and example sentences in its window, Google reported, the model translated English into Kalamang about as well as a person who learned from the same content.383731

Mixture of experts: big models that use only part of themselves#

A mixture-of-experts (MoE) model splits parts of its network into many smaller sub-networks, called experts, and runs only a few of them for each token.3940 All experts count toward its total parameters, but only the chosen ones are active, so each token costs about what a much smaller model would.4041

An ordinary dense model uses all of its parameters for every token.42 An MoE layer adds a small trained router that scores the experts and sends each token to the top few.3943 This choice happens inside one model, at each of its expert layers, unlike model routing, which picks between whole models.4445 Having many experts but running only a few raises the model's capacity, its room to absorb knowledge, without a matching rise in computing work per token.3942 The tradeoffs are that every expert must still sit in memory, and training must balance the load so a few experts do not take all the work.4647

DeepSeek-V3, from the Chinese AI lab DeepSeek, has 671 billion parameters but activates only 37 billion for each token.4849 So each token uses only about one-eighteenth of the model's parameters.48 Its technical report says the expert design, together with other choices, was meant to make training and running the model cheaper.50

In August 2026, Tencent released Hy4 Preview, a mixture-of-experts model with 770 billion total parameters and 49 billion active per token.51