How AI Models Work
How AI Models Work
The models behind ChatGPT, Claude, and Gemini are not magic — they are specific mathematical architectures trained on specific data with specific objectives. Understanding the mechanics makes you a far better builder and a much harder person to fool by AI hype.
What a Language Model Is Doing
At its core, a large language model (LLM) is doing one thing: given a sequence of tokens, predict the next most likely token. That's it. The emergent capabilities — reasoning, coding, summarization, translation — are not programmed in. They arise from training this simple prediction task on several trillion tokens of human-generated text.
When you send a prompt to Claude and get a response, the model is generating one token at a time, with each new token conditioned on every previous token in the context. This is why longer responses cost more — both in compute and in latency.
The Transformer Architecture
The breakthrough paper was "Attention is All You Need" (Vaswani et al., 2017). Before transformers, the best NLP models processed text sequentially — each word depended only on the words before it. Transformers process all tokens in the context simultaneously using a mechanism called self-attention.
Self-attention lets the model answer: "For the current token I am predicting, how relevant is every other token in the context?" It assigns attention weights to all tokens, then computes a weighted representation. This is why transformers can handle long-range dependencies — the model can attend to a word 10,000 tokens back with the same mechanism as attending to the previous word.
The cost: self-attention is O(n²) in the length of the sequence. Doubling the context window quadruples the compute. This is why extending context windows is expensive and why most production systems use chunking and retrieval rather than stuffing everything into one prompt.
Tokens and Why They Matter
Every API call you make to an LLM is priced in tokens. At $3 per million input tokens (rough GPT-4 pricing), running 10,000 API calls with 1,000-token prompts costs $30. At 10,000-token prompts, it's $300. Token efficiency is not a theoretical concern — it is a unit economics decision.
Token counts also affect latency. Models generate at roughly 30–80 tokens per second. A 1,000-token response takes 12–33 seconds at those rates. For user-facing applications, streaming (sending tokens as they are generated) is essential for perceived responsiveness.
Embeddings and Semantic Search
Embeddings convert text into vectors — lists of ~1,500 numbers that encode meaning. Two sentences that mean the same thing have similar vectors even if they share no words. "The car broke down" and "the vehicle stopped functioning" will be embedded close together.
This property powers:
- Semantic search: find documents by meaning, not keyword match
- RAG (Retrieval-Augmented Generation): fetch relevant knowledge before generating a response
- Clustering: group similar customer feedback, tickets, or documents automatically
- Recommendation systems: find items similar to what a user liked
In practice: you embed your knowledge base once (or nightly), store vectors in a database like Pinecone or pgvector, then at query time embed the user's question and retrieve the closest vectors. The retrieved text goes into the prompt as context.
The Prompting Layer
Prompt engineering is the interface between human intent and model behavior. The model cannot read your mind — it responds to the exact tokens you send. Effective prompting means:
- Role definition — tell the model what persona and constraints to operate under
- Task specification — be precise about format, length, and output requirements
- Examples (few-shot) — showing the model 2–3 examples of desired input-output pairs dramatically improves consistency
- Chain-of-thought — asking the model to "think step by step" before answering improves accuracy on reasoning tasks by 30–50% on benchmarks
What Models Cannot Do
LLMs do not have: persistent memory across calls, access to real-time information (without tools), the ability to run code reliably without verification, or true understanding of their own confidence. They are pattern-matching systems trained to produce plausible text. Every output should be treated as a strong first draft that requires review, not a ground truth.