How ChatGPT Actually Works
The 5 papers that explain exactly how ChatGPT was built. Read in order: architecture → scale → alignment → refinement → tradeoffs.
Ashish Vaswani et al.
Transformers revolutionize AI by ditching recurrence and convolutions, shining with sheer parallelizable efficiency.
Why this paper
Start here. Transformers are the architecture inside every modern LLM — this paper invented them.
gpt-3 — coming soon
Scale changes everything. GPT-3 showed that making models bigger produces surprisingly emergent capabilities.
Jared Kaplan et al.
Larger language models offer more sample efficiency, enabling better results with smaller datasets and fixed compute resources.
Why this paper
The science of scale — why bigger isn't always better and how OpenAI decides how much compute to spend.
instructgpt — coming soon
This is the exact technique that turned GPT-3 into ChatGPT. RLHF from human feedback changed everything.
Yuntao Bai et al.
Train AI with its own feedback to reduce need for human labels and increase precision in behavior control.
Why this paper
Anthropic's refinement: teaching models to self-critique reduces the cost and bias of human feedback.
Unlock the full analysis for each paper
Deep-dive articles, expert annotations, PM action plans, and interactive experiments — all for $6/mo.
Go Pro — $6/mo