How AI Really Works

The core loop: guess → measure error → adjust

Learning is just a repetitive loop. The network makes a guess, you measure how wrong it was, and you nudge every weight slightly in the direction that makes it less wrong. Repeat millions of times.

Loss functions

import numpy as np

# === Mean Squared Error (regression tasks) ===
def mse_loss(predictions, targets):
    return np.mean((predictions - targets) ** 2)

# === Cross-Entropy Loss (classification tasks) ===
# "How surprised was the model?" — penalizes confident wrong answers hard
def cross_entropy(pred_prob, true_class):
    return -np.log(pred_prob[true_class])

# Example: model predicted [0.1, 0.8, 0.1], correct class is 1
# Loss = -log(0.8) ≈ 0.22  (low — model was right and confident)

# Example: model predicted [0.1, 0.1, 0.8], correct class is 1
# Loss = -log(0.1) ≈ 2.3  (high — model was confidently wrong)

Gradient Descent — finding the bottom of the hill

# learning_rate controls step size
# too large: you overshoot and bounce around
# too small: training takes forever
learning_rate = 0.01

for epoch in range(1000):
    # Forward pass — get predictions
    predictions = forward(X, weights)

    # Compute how wrong we are
    loss = mse_loss(predictions, y_true)

    # Backward pass — compute gradient (how loss changes per weight)
    gradients = backward(loss, weights)

    # Update: move weights in the opposite direction of gradient
    weights = weights - learning_rate * gradients

What backpropagation actually is

Backpropagation is just the chain rule from calculus, applied systematically across all layers. It answers: "if I change this weight by a tiny amount, how much does the total loss change?"

In PyTorch/TensorFlow, calling loss.backward() does all of this automatically. You never write backprop by hand — but knowing it exists is crucial to debugging.

The sequence problem

Language, audio, and time-series data are sequences. The order matters. "The dog bit the man" and "The man bit the dog" have the same words but completely different meanings.

A regular neural network (feedforward) takes a fixed-size input and produces a fixed-size output. It has no concept of order or time. So how do you handle sequences?

Recurrent Neural Networks (RNNs)

The idea: process one word at a time, and keep a "hidden state" — a memory vector that carries information forward as you read each word.

import numpy as np

def rnn_cell(x_t, h_prev, W_x, W_h, b):
    """
    x_t    : current word's vector representation
    h_prev : hidden state from previous step (memory)
    W_x    : weights for input
    W_h    : weights for previous hidden state
    b      : bias
    """
    # Combine current input + previous memory
    z = W_x @ x_t + W_h @ h_prev + b
    
    # Squash through tanh to keep values bounded
    h_new = np.tanh(z)
    
    return h_new

# Process the sentence "The cat sat"
h = np.zeros(128)  # initial memory: all zeros

for word_vector in sentence_embeddings:
    h = rnn_cell(word_vector, h, W_x, W_h, b)
    # h now "remembers" everything up to this word

The vanishing gradient problem

Here's the fundamental issue with RNNs: when you backpropagate through time (through 50+ steps), the gradients get multiplied together at each step. If each multiplication makes them slightly smaller... after 50 steps they're essentially zero.

LSTMs — the engineering fix

Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) added explicit memory management. Instead of one hidden state, an LSTM has a cell state (long-term memory) and three gates that control what to remember, forget, and output.

The cell state (long-term memory) flows through with minimal modification — so gradients can now travel further back without vanishing.

LSTMs were dominant from ~2013–2017 and powered Google Translate, Siri, and many production NLP systems. But they were still fundamentally sequential — you had to process word 1 before word 2. You couldn't parallelize.

The Encoder-Decoder architecture

For translation (English → French), the approach was: use one LSTM to encode the entire input sentence into a fixed vector, then use another LSTM to decode that vector into the output sentence.

In 2015, Bahdanau et al. introduced attention as an add-on to fix this bottleneck. The decoder could now "look back" at all encoder hidden states, not just the last one. This was the precursor to everything.

Three fundamental problems with RNNs/LSTMs

The word embedding problem — inputs as vectors

Before understanding Attention, you need to know how words become numbers. Neural networks only understand numbers.

# Each word maps to a dense vector of floats (e.g., 512 dimensions)
# These vectors are LEARNED during training, not hand-crafted

from torch import nn

# vocab_size: how many unique tokens in our vocabulary (e.g. 50,000)
# embed_dim: how many numbers represent each word (e.g. 512)
embedding = nn.Embedding(vocab_size=50000, embedding_dim=512)

# Word ID → vector
word_id = 342   # e.g., token for "cat"
vector = embedding(word_id)
# vector is now shape [512] — a point in 512D space

# A sentence "The cat sat" becomes a matrix:
# shape = [3 words × 512 dimensions]
# This matrix is the input to the Transformer

The core insight that changed everything

What if instead of passing a hidden state left-to-right through a sequence, we let every word look at every other word directly — and decide how much to "pay attention" to each one?

This means the word "it" in "The animal didn't cross because it was tired" can directly look at "animal" and compute a high attention score — without needing to relay information through 4 other words.

The query-key-value intuition

Attention is built around three vectors per token: Query (Q), Key (K), and Value (V). Each is computed from the token's embedding using learned weight matrices.

In Python — from scratch

import numpy as np

def attention(Q, K, V):
    """
    Q: [seq_len, d_k]  — queries (what each token looks for)
    K: [seq_len, d_k]  — keys   (what each token offers)
    V: [seq_len, d_v]  — values (what each token contains)
    """
    d_k = Q.shape[-1]

    # How much does each query match each key?
    # scores[i][j] = "how much should token i attend to token j?"
    scores = Q @ K.T / np.sqrt(d_k)
    # shape: [seq_len, seq_len]

    # Convert to probabilities (each row sums to 1)
    def softmax(x):
        e = np.exp(x - x.max(axis=-1, keepdims=True))
        return e / e.sum(axis=-1, keepdims=True)

    attn_weights = softmax(scores)
    # Each row = "what % of attention goes to each other token?"

    # Weighted blend of value vectors
    output = attn_weights @ V
    # output[i] = a mix of all tokens' values, weighted by relevance

    return output, attn_weights

Try it: which word does "it" attend to?

Click any token to see which other tokens it would most strongly attend to in this sentence. (Scores are illustrative — but they reflect what real attention heads actually learn.)

Multi-Head Attention

One attention operation can only capture one type of relationship. Multi-head attention runs the attention mechanism multiple times in parallel (e.g., 8 or 16 heads), each with its own learned W_Q, W_K, W_V matrices.

"Do we tell each head what to look for?"

Short answer: no. Each head's W_Q, W_K, W_V matrices start as small random numbers. Nothing in the architecture says "head 3 = pronouns." During training, the only signal the model gets is the next-token loss, propagated back through every weight via backpropagation.

So why do heads specialize? Because the architecture forces them to:

The original paper — "Attention Is All You Need"

The 2017 Google paper (Vaswani et al., arXiv:1706.03762) stacked attention layers into a full architecture. No recurrence. No convolutions. Just attention and feed-forward layers.

Walking through every stage

Here's exactly what happens to a sentence as it flows through the architecture above, top to bottom:

Positional Encoding — solving "no order" problem

Attention has no concept of order. If you shuffle the input words, you get the same attention scores. So the paper adds positional encodings — vectors that encode each token's position in the sequence — and add them to the embeddings.

import torch
import math

def positional_encoding(seq_len, d_model):
    """
    seq_len: number of tokens in the sequence
    d_model: embedding dimension (e.g., 512)
    Returns positional vectors to ADD to token embeddings
    """
    PE = torch.zeros(seq_len, d_model)
    position = torch.arange(0, seq_len).unsqueeze(1)

    # Use sine/cosine waves of different frequencies
    # Each position gets a unique "fingerprint" vector
    div_term = torch.exp(torch.arange(0, d_model, 2) *
                       (-math.log(10000.0) / d_model))

    PE[:, 0::2] = torch.sin(position * div_term)
    PE[:, 1::2] = torch.cos(position * div_term)

    return PE

# Add to your embeddings before feeding to Transformer
x = token_embeddings + positional_encoding(seq_len, d_model)

Layer Norm and Feed-Forward — the other components

After each attention block, two things happen:

Add & LayerNorm: The input is added back to the output (residual connection), then normalized. Residuals prevent vanishing gradients in deep networks and enable extremely deep architectures.

Feed-Forward Network (FFN): Two linear layers with ReLU in between. Applied independently to each token position. This is where the network stores and retrieves factual knowledge — each FFN layer acts like a key-value memory store.

import torch.nn as nn

class TransformerBlock(nn.Module):
    def __init__(self, d_model=512, n_heads=8, d_ff=2048):
        super().__init__()
        # Multi-head self-attention
        self.attention = nn.MultiheadAttention(d_model, n_heads)
        # Layer normalizations
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        # Feed-forward network (applied to each position independently)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Linear(d_ff, d_model)
        )

    def forward(self, x):
        # Self-attention + residual connection + normalization
        attn_out, _ = self.attention(x, x, x)
        x = self.norm1(x + attn_out)   # "Add & Norm"

        # Feed-forward + residual + normalization
        ffn_out = self.ffn(x)
        x = self.norm2(x + ffn_out)   # "Add & Norm"

        return x

GPT vs BERT — two ways to use the Transformer

Stage 1: Pre-training — next token prediction

The base LLM is trained on a massive corpus (trillions of tokens from the internet, books, code). The task is brutally simple: given all previous tokens, predict the next token.

for batch in training_data:
    # batch = tokenized text like: [42, 8931, 102, 5531, 99, ...]

    # Input: all tokens except the last
    # Target: all tokens except the first (shifted by 1)
    input_ids  = batch[:-1]  # "The cat sat on the"
    target_ids = batch[1:]   # "cat sat on the mat"

    # Model predicts probability distribution over vocab for each position
    logits = model(input_ids)
    # logits shape: [seq_len, vocab_size]  e.g., [512, 50000]

    # Cross-entropy loss: how surprised was the model by the real next word?
    loss = cross_entropy(logits.view(-1, vocab_size), target_ids.view(-1))

    loss.backward()   # backprop through ALL layers of the Transformer
    optimizer.step()  # update all weights (billions of them)

Stage 2: Supervised Fine-Tuning (SFT)

A base model is just a text completion engine. It will complete "How do I make a bomb?" by continuing in the style of whatever it was trained on. To make it a helpful assistant, you fine-tune on instruction-response pairs.

{
  "instruction": "Summarize this text in 3 bullet points",
  "input":       "... long article ...",
  "output":      "• Key point 1\n• Key point 2\n• Key point 3"
}

Thousands-to-millions of such examples teach the model to produce outputs that match this instruction-response format. The result: a model that follows instructions instead of just continuing text.

Stage 3: RLHF — making it actually good

SFT gets you an instruction-following model. RLHF (Reinforcement Learning from Human Feedback) makes it aligned — honest, helpful, harmless.

Tokenization — what the model actually sees

Models don't see words. They see tokens — sub-word chunks. "unhappiness" → ["un", "happi", "ness"]. This lets the model handle words it has never seen.

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3-8B")

text = "The telecom network had 5G latency issues"
tokens = tokenizer.encode(text)
# → [450, 14584, 3904, 2400, 29871, 29945, 29954, 21162, 5626]

# Decode to see the actual chunks:
decoded = [tokenizer.decode([t]) for t in tokens]
# → ['The', ' tele', 'com', ' network', ' had', ' 5', 'G', ' lat', 'ency', ' issues']

# This is why domain-specific terms fragment strangely —
# "nrcelldu" would tokenize into many small pieces
# This is the exact reason your schema synonym enrichment matters!

Inference: how a model generates text

During generation, the model outputs one token at a time. Each token becomes part of the input for the next step (autoregressive generation). The model predicts a probability distribution over the entire vocabulary, and you sample from it.

import torch
import torch.nn.functional as F

def generate(model, prompt_tokens, max_new_tokens=100, temperature=0.7):
    tokens = prompt_tokens.copy()

    for _ in range(max_new_tokens):
        # Forward pass: get logits for ALL positions
        logits = model(tokens)

        # Take only the LAST position's logits (the next token prediction)
        next_logits = logits[-1] / temperature

        # Convert to probabilities
        probs = F.softmax(next_logits, dim=-1)

        # Sample (don't just take argmax — that's too rigid)
        next_token = torch.multinomial(probs, 1).item()

        if next_token == EOS_TOKEN:
            break

        tokens.append(next_token)

    return tokens

What to explore next

You now have the foundation. The next chapter covers everything that has happened since the Transformer — and the modern stack you'll actually build on.

The chronological map

The Transformer (2017) was the architectural breakthrough. Everything since has been about scaling it up, making it cheaper, connecting it to tools, and letting it think and act. This timeline only includes things that are actually used in production today — no hype, no toys.

Concept 1 · RAG (Retrieval-Augmented Generation)

An LLM's knowledge is frozen at training time. RAG fixes this without retraining: when a question comes in, retrieve relevant documents from a vector database and stuff them into the prompt. The model now "knows" your private data.

Concept 2 · Context, the Context Window, and KV Cache

"Context" is just everything you put into the prompt: system message, conversation history, retrieved RAG chunks, tool outputs. The context window is the hard limit on how many tokens fit.

Concept 3 · Quantization & Local Models (llama.cpp)

A 70B-parameter model in float16 is 140GB. You can't fit that in consumer RAM. Quantization compresses each weight from 16 bits to 8, 5, 4, or even 2 bits — with surprisingly small quality loss.

Concept 4 · Mixture of Experts (MoE)

A standard Transformer activates every parameter for every token. MoE replaces the FFN with N experts and a router that picks just 2 of them per token. You get a model with the knowledge capacity of all N experts, but the inference cost of just 2.

Examples: Mixtral 8×7B (open), GPT-4 (rumored MoE), DeepSeek V3, Grok-1. The downside: you still need RAM for all experts because the router can pick any of them next token.

Concept 5 · Multimodal Models

A multimodal model accepts more than text — typically images, but also audio and video. The architectural trick is simple: convert every modality into tokens, then feed them into the same Transformer.

Examples: GPT-4o, Claude 3.5, Gemini 1.5, Llama 3.2 Vision, Qwen-VL. The same recipe works for audio (whisper-style tokens) and video (frame-by-frame ViT tokens).

Concept 6 · Tool Use & Function Calling

By itself an LLM can't access the internet, query a database, or run code. Function calling teaches it to emit structured JSON describing what tool it wants to invoke. Your code runs the tool, feeds the result back, and the model continues.

// You provide tool definitions in the system prompt:
{
  "name": "get_weather",
  "description": "Get current weather for a city",
  "parameters": { "city": "string" }
}

// User: "What's the weather in Tokyo?"
// Model emits this instead of plain text:
{
  "tool_call": {
    "name": "get_weather",
    "arguments": { "city": "Tokyo" }
  }
}

// Your code runs the function, feeds result back:
{ "tool_result": { "temp": 22, "condition": "clear" } }

// Model now produces the final answer:
"It's 22°C and clear in Tokyo right now."

Concept 7 · MCP — Model Context Protocol

Function calling is per-application. Every team re-invents tool plumbing. MCP (Anthropic, Nov 2024) standardizes it: a client-server protocol where any LLM can talk to any tool / data source / prompt-library as long as both speak MCP. Think "USB for AI" — and by 2026 it has won as the de facto standard, the way LSP won for editors.

MCP exposes four primitive types over JSON-RPC:

Plus newer 2025 additions: Roots (which directories/URIs the host exposes to a server) and Elicitation (server asks the user a question mid-flow — "which env should I deploy to?"). These are what turned MCP from "function calling 2.0" into a real two-way agent protocol.

Concept 8 · Agents & the Harness

An agent is an LLM in a loop: think → act → observe → think. The model decides what tool to call, your code runs it, the result goes back into the prompt, and the model decides the next step. The loop terminates when the model emits a "done" signal.

The harness is the surrounding code that makes the loop work: tool registry, prompt assembly, output parsing, retries, safety checks, token-budget management, and the UI. Frontier-class agentic products are 90% harness engineering on top of an off-the-shelf model.

Concept 9 · Mixture of Agents (MoA)

If one agent is good, multiple specialist agents in collaboration can be better. Mixture of Agents (Wang et al., 2024) stacks layers of agents: each layer's outputs become the next layer's inputs.

Concept 10 · Test-Time Compute & Reasoning Models

For 7 years the recipe was: spend more compute at training time → get a smarter model at inference. In Sep 2024, OpenAI's o1 flipped the script: spend more compute at inference time (let the model think longer) and you get the same gain. This is the biggest shift since RLHF.

Concept 11 · GRPO & RLVR — how reasoning models are actually trained

RLHF needs humans to label "which answer is better." That doesn't scale to math problems with 10,000-token solutions. RLVR (Reinforcement Learning with Verifiable Rewards) replaces the human reward model with a program: run the unit tests, check the math, parse the SQL — if it passes, reward = 1.

GRPO (Group Relative Policy Optimization, DeepSeek) is the lightweight RL algorithm that made this practical. Instead of training a separate value/critic network like PPO, it samples multiple candidate solutions per prompt and uses their relative rewards as the signal.

Concept 12 · Hybrid architectures: Mamba & State Space Models

Pure attention is O(n²). For 1M-token contexts that becomes painful. State Space Models (Mamba, Dec 2023) revisit RNN-style linear-time sequence processing, but with selective gating that — empirically — competes with attention on quality.

By 2025, the winning recipe turned out to be hybrid: a few Transformer layers (for precise recall) interleaved with many Mamba/SSM layers (for cheap long context).

Production-deployed hybrids (2025–26): AI21 Jamba 1.5, NVIDIA Nemotron-H, TII Falcon-Mamba, Zyphra Zamba. You won't write Mamba code by hand — but if a model card says "hybrid" or "SSM," now you know what's inside and why it's faster on long inputs.

Concept 13 · Speculative Decoding — the silent 2× speedup

Generation is sequential: predict token, append, predict next. Each step needs a full forward pass through 70B parameters. Speculative decoding uses a small "draft" model to propose k tokens at once, then the big model verifies all k in a single parallel pass. Accepted tokens are kept; the first rejected one resets the draft.

Where you'll see it: vLLM, TGI, llama.cpp, SGLang, Ollama, every commercial API. By 2025–26, it's not optional — every production inference stack runs it by default. Variants: EAGLE, Medusa, self-speculation (no separate draft model needed).

Concept 14 · Production inference stacks — beyond llama.cpp

llama.cpp is for local single-user inference. For serving thousands of concurrent users, you need a different stack — one that batches requests, manages KV cache memory, and supports speculative decoding at scale.

When to use what — the practical decision matrix

The hardest part of using all this isn't understanding any single concept — it's knowing which tool to reach for. Here's the rough decision tree you'll actually use:

How everything fits together