The Information Machine
Learn · Guide 5 of 13

Getting answers from a model: prompts, reasoning and routing

This guide explains how models receive standing instructions, why writing out steps can improve answers, what extra thinking time buys you, and how products pick which model responds.

The system prompt: standing instructions#

A company writes a system prompt: a block of text its app sends to the model at the start of every conversation.1 Training teaches the model to put those instructions ahead of what the user types.2

A chat request is a list of messages marked by role: the person's turns (user), the model's earlier replies (assistant) and, in some APIs, the company's standing instructions (system).34 The system prompt is not part of the model's weights; it is ordinary text, resent with every request, that fills part of the context window.54 A system prompt can carry facts the model cannot know, like today's date, and house rules, like formatting code in Markdown.1 In post-training, labs teach models to rank instructions by source and ignore conflicting lower-ranked ones, an approach researchers call an instruction hierarchy.67 Companies often hide the prompt from users, but that ranking is a learned habit, not a lock, so the prompt is neither reliably secret nor a security boundary: prompt injection and jailbreaking can override it.867

In February 2023, a day after Microsoft unveiled its Bing chatbot, Stanford student Kevin Liu got it to print its hidden instructions.89 The instructions opened with an identity section that gave the bot the internal codename Sydney and set rules such as introducing itself with 'This is Bing'.89 Another rule told the bot not to reveal the codename to users, so the leak showed both what a real system prompt holds and how little its rules guarantee.89

Chain-of-thought: models that think before answering#

Chain-of-thought means a model writes out a series of intermediate reasoning steps before it gives its final answer.10 Writing the steps out makes large models better at problems that take several steps, such as math word problems, and theoretical work traces part of that gain to the extra computation each written step gives the model to build on.111213

What is chain-of-thought?

The original method, chain-of-thought prompting, puts a few worked examples in the prompt, and a large enough model copies their step-by-step style.10 Theoretical work offers one reason: a model produces one token per pass through its network.1213 Each pass runs through a fixed number of layers (processing stages), which limits how many steps it can do in a row.12 Each token it writes then becomes input for the next pass, so a written chain gives it many extra rounds of computation in sequence.1312 Reasoning models go a step further: they are trained with reinforcement learning (training by reward) to write a long chain of thought on their own before answering.1415 The written steps are not guaranteed to match how the model really reached its answer; chain-of-thought monitoring covers why that matters.1617

In 2022, Kojima and colleagues added a single phrase, "Let's think step by step," to prompts for a large language model called InstructGPT, with no worked examples at all.1813 On MultiArith, a set of arithmetic problems, its accuracy rose from 17.7% to 78.7%.18 Because one fixed prompt worked across very different tasks, the authors wrote that models may already hold little-studied abilities to reason without any examples.18

Reasoning effort: trading cost for thinking time#

A reasoning model can often answer hard questions better when it spends more computation while it runs, not only during training; this is called test-time compute.192021 Products let you buy more of this thinking, and how much it helps depends on how hard the question is.2223

The extra computation goes into a longer chain of reasoning steps written before the final answer.24 Those steps are generated as output tokens even when they are hidden from the user, so more thinking costs more and takes longer.2526 Users can set how much a model thinks: some APIs take an effort level, and others take a thinking budget counted in tokens.2726 On simple questions, models can spend many tokens for little gain, which researchers call overthinking.28

In January 2025, researchers who built a reasoning model called s1 made it think longer by adding the word "Wait" whenever it tried to stop.29 This could lead it to double-check its answer, often fixing wrong steps.29 Its score on the 2024 AIME, a hard competition exam for top high school math students, rose from 50% to 57%.2921 Both scores came from the same trained model; only its thinking time differed.29

Model routing: sending each request to the right model#

Model routing means a product picks, for each request, which of several models should answer it.30 Larger, stronger models give better answers but cost more to run, so routing sends easy requests to smaller, cheaper models and saves the large ones for hard requests.3132

One design is a router: it predicts how hard a request is, then sends it to a small or large model, or to one that thinks longer.3334 Another design is a cascade: it tries a small, cheap model first and escalates to a large one only if the answer fails a check.3536 One risk is a wrong call: small models can do much worse on difficult tasks, so a hard request sent to one can get a weaker answer.3633

When OpenAI launched GPT-5 in August 2025, ChatGPT's version was a fast model plus a deeper reasoning model, with a real-time router choosing between them for each message.3437 OpenAI said the router would remove the need for ChatGPT users to navigate its model picker, which TechCrunch described as a long, complicated menu of options.38 The day after launch, Sam Altman, OpenAI's chief executive, said the "autoswitcher", his name for the router, had been down for part of launch day, so "GPT-5 seemed way dumber."3738 He also said OpenAI was changing how the router decides, which "should help you get the right model more often."37

Keep going
Sources (38)
  1. System prompts docs.anthropic.com
  2. Paper page - The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions huggingface.co
  3. Messages - Claude API Reference docs.anthropic.com
  4. Role : assistant in client.chat.completions.create community.openai.com
  5. Context windows docs.anthropic.com
  6. Many-Tier Instruction Hierarchy in LLM Agents arxiv.org
  7. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions arxiv.org
  8. AI-powered Bing Chat spills its secrets via prompt injection attack [Updated] arstechnica.com
  9. Notes from Bing Chat—Our First Encounter With Manipulative AI simonwillison.net
  10. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models ar5iv.labs.arxiv.org
  11. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models arxiv.org
  12. Chain of Thought Empowers Transformers to Solve Inherently Serial Problems arxiv.org
  13. Not All LLM Reasoning is Visible in the Chain-of-Thought arxiv.org
  14. Lie to Me: How Faithful Is Chain-of-Thought Reasoning in Open-Weight Reasoning Models? arxiv.org
  15. Evaluating chain-of-thought monitorability openai.com
  16. Reasoning models don't always say what they think anthropic.com
  17. Reasoning Models Don't Always Say What They Think arxiv.org
  18. Large Language Models are Zero-Shot Reasoners arxiv.org
  19. Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead arxiv.org
  20. Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinking arxiv.org
  21. Learning to reason with LLMs openai.com
  22. A model guide for the GPT-6 family openai.com
  23. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters arxiv.org
  24. Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond arxiv.org
  25. Understanding and counting tokens | OpenAI Help Center help.openai.com
  26. Extended thinking docs.anthropic.com
  27. Azure OpenAI reasoning models - GPT-6 Astra and GPT-5 - Microsoft Foundry learn.microsoft.com
  28. Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs arxiv.org
  29. s1: Simple test-time scaling arxiv.org
  30. Model router for Microsoft Foundry concepts - Microsoft Foundry learn.microsoft.com
  31. RouteLLM: Learning to Route LLMs with Preference Data arxiv.org
  32. RouteLLM: Learning to Route LLMs with Preference Data arxiv.org
  33. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing arxiv.org
  34. OpenAI GPT-5 System Card arxiv.org
  35. GLM-5.3 vs. GLM-5.3 Flash on DeepSWE: Cost, Coding, and Routing together.ai
  36. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance ar5iv.labs.arxiv.org
  37. Sam Altman addresses ‘bumpy’ GPT-5 rollout, bringing 4o back, and the ‘chart crime’ | TechCrunch techcrunch.com
  38. ChatGPT's model picker is back, and it's complicated | TechCrunch techcrunch.com