27. Transformer Architecture¶

Understand the architecture that transformed modern Deep Learning by replacing recurrent sequence processing with self-attention, and learn how embeddings, positional information, multi-head attention, feed-forward networks, residual connections, normalization, encoder-decoder structures, and causal masking work together to form modern Transformer systems.

🎯 Learning Objectives¶

After completing this chapter, you will be able to:

Explain why Transformer architecture was introduced
Understand the limitations of recurrent sequence models
Explain the overall Transformer architecture
Understand Transformer encoder and decoder components
Explain token embeddings
Understand positional information
Explain self-attention inside a Transformer
Understand Query, Key, and Value projections
Explain scaled dot-product attention
Understand multi-head attention
Explain the role of feed-forward networks
Understand residual connections
Explain Layer Normalization
Understand the Transformer encoder block
Understand the Transformer decoder block
Explain masked self-attention
Understand cross-attention
Explain the original encoder-decoder Transformer
Understand encoder-only, decoder-only, and encoder-decoder Transformers
Understand Transformer tensor shapes
Understand Transformer computational complexity
Understand PyTorch Transformer components
Build a Transformer encoder classifier
Build a simple Transformer architecture
Understand causal language modeling
Understand autoregressive generation
Understand KV caching conceptually
Understand the evolution from Transformer to LLMs
Understand production considerations for Transformer systems

📖 Overview¶

The Transformer is one of the most important architectures in modern Artificial Intelligence.

Before Transformers, sequence modeling was dominated by:

RNN
 ↓
LSTM
 ↓
GRU

These architectures process sequences recurrently.

The Transformer introduced a fundamentally different approach:

Self-Attention
+
Feed-Forward Networks
+
Residual Connections
+
Normalization
+
Positional Information

Instead of processing tokens one by one through a recurrent state, Transformers allow tokens to interact directly through attention.

🧠 Why Were Transformers Introduced?¶

RNN-based architectures have several limitations:

Sequential Computation
        ↓
Limited Parallelism
        ↓
Long Training Time

and:

Long Sequence
      ↓
Long Information Path
      ↓
Difficulty Learning Long-Range Relationships

Transformers address these limitations using attention.

🧠 RNN vs Transformer¶

RNN¶

x₁
 ↓
h₁
 ↓
x₂
 ↓
h₂
 ↓
x₃
 ↓
h₃
 ↓
x₄
 ↓
h₄

Transformer¶

x₁ ─────────┐
x₂ ─────────┤
x₃ ─────────┼──► Self-Attention
x₄ ─────────┤
             │
             ▼
       Contextual Output

The Transformer can process relationships between many positions simultaneously.

🧠 The Original Transformer¶

The original Transformer architecture introduced an:

Encoder
+
Decoder

architecture.

Conceptually:

Input Sequence
      ↓
   Encoder
      ↓
Contextual Representations
      ↓
   Decoder
      ↓
Output Sequence

🧠 High-Level Transformer Architecture¶

flowchart LR

    INPUT["Input Tokens"]

    EMBED["Token Embeddings"]

    POS["Positional Information"]

    ENCODER["Transformer Encoder"]

    DECODER["Transformer Decoder"]

    OUTPUT["Output Tokens"]

    INPUT --> EMBED
    POS --> ENCODER

    EMBED --> ENCODER
    ENCODER --> DECODER
    DECODER --> OUTPUT

🧠 Transformer Architecture Landscape¶

Modern Transformer architectures evolved into three major patterns:

Transformer
     │
     ├───────────────┬────────────────
     ▼               ▼               ▼
Encoder-only    Decoder-only    Encoder-Decoder
     │               │               │
     ▼               ▼               ▼
Classification      LLMs       Translation / Generation

Examples of tasks:

Encoder-only
→ Classification
→ Embeddings
→ Sequence Understanding

Decoder-only
→ Text Generation
→ Code Generation
→ Conversational AI

Encoder-Decoder
→ Translation
→ Summarization
→ Sequence-to-Sequence Generation

🧠 Transformer Building Blocks¶

A Transformer is built from several core components:

Token Embeddings
       ↓
Positional Information
       ↓
Multi-Head Attention
       ↓
Feed-Forward Network
       ↓
Residual Connections
       ↓
Layer Normalization
       ↓
Repeated Transformer Blocks

🧠 Transformer Block¶

A simplified Transformer block looks like:

Input
  │
  ▼
Multi-Head Self-Attention
  │
  ▼
Add & Norm
  │
  ▼
Feed-Forward Network
  │
  ▼
Add & Norm
  │
  ▼
Output

🧠 Transformer Encoder Block¶

flowchart TD

    INPUT["Input Representation"]

    ATTENTION["Multi-Head Self-Attention"]

    ADD1["Residual Connection"]

    NORM1["Layer Normalization"]

    FFN["Feed-Forward Network"]

    ADD2["Residual Connection"]

    NORM2["Layer Normalization"]

    OUTPUT["Encoder Output"]

    INPUT --> ATTENTION
    INPUT --> ADD1
    ATTENTION --> ADD1
    ADD1 --> NORM1

    NORM1 --> FFN

    NORM1 --> ADD2
    FFN --> ADD2

    ADD2 --> NORM2
    NORM2 --> OUTPUT

🧠 Token Embeddings¶

Neural networks operate on numerical representations.

Text starts as:

"The cat sleeps"

The processing pipeline becomes:

Text
 ↓
Tokenization
 ↓
Token IDs
 ↓
Embedding
 ↓
Dense Vectors

For example:

"The"   → [0.12, -0.21, ...]
"cat"   → [0.51,  0.33, ...]
"sleeps"→ [-0.17, 0.81, ...]

🧠 Embedding Matrix¶

If:

Vocabulary Size = V
Embedding Dimension = D

then the embedding matrix has shape:

[ V \times D ]

Each token maps to one row of this matrix.

🧠 Positional Information¶

Self-attention does not inherently understand:

First
Second
Third
...

Therefore Transformer inputs need positional information.

Conceptually:

Token Embedding
      +
Position Representation
      ↓
Transformer Input

🧠 Transformer Input¶

The Transformer input can be represented as:

[ X=E+P ]

where:

E = Token Embeddings
P = Positional Representation
X = Transformer Input

🧠 Positional Information¶

Different Transformer architectures can use different positional mechanisms:

Sinusoidal Positional Encoding
        ↓
Learned Positional Embeddings
        ↓
Relative Position Methods
        ↓
Rotary Position Representations

The exact mechanism depends on the model architecture.

🧠 Self-Attention¶

The central operation inside a Transformer is self-attention.

Given input:

the model computes:

Q = XWQ
K = XWK
V = XWV

🧠 Scaled Dot-Product Attention¶

The attention operation is:

[ Attention(Q,K,V) = softmax \left( \frac{QK^T}{\sqrt{d_k}} \right)V ]

This consists of:

QKᵀ
 ↓
Similarity Scores
 ↓
Scale
 ↓
Optional Mask
 ↓
Softmax
 ↓
Attention Weights
 ↓
Weighted Values

🧠 Attention Inside Transformer¶

flowchart LR

    X["Input"]

    Q["Query Projection"]

    K["Key Projection"]

    V["Value Projection"]

    SCORE["QKᵀ"]

    SCALE["Scale"]

    SOFTMAX["Softmax"]

    WEIGHTS["Attention Weights"]

    OUTPUT["Weighted Values"]

    X --> Q
    X --> K
    X --> V

    Q --> SCORE
    K --> SCORE

    SCORE --> SCALE
    SCALE --> SOFTMAX
    SOFTMAX --> WEIGHTS

    WEIGHTS --> OUTPUT
    V --> OUTPUT

🧠 Multi-Head Attention¶

Transformers do not usually rely on a single attention operation.

Instead:

Input
 ↓
Head 1
Head 2
Head 3
...
Head H
 ↓
Concatenate
 ↓
Output Projection

🧠 Multi-Head Attention Formula¶

[ MultiHead(Q,K,V) = Concat(head_1,\ldots,head_h)W^O ]

Each attention head is:

[ head_i= Attention(QW_i^Q,KW_iK,VW_i^V) ]

🧠 Multi-Head Attention Architecture¶

flowchart TD

    INPUT["Input"]

    H1["Attention Head 1"]
    H2["Attention Head 2"]
    H3["Attention Head 3"]
    H4["Attention Head H"]

    CONCAT["Concatenate Heads"]

    PROJECTION["Output Projection"]

    OUTPUT["Multi-Head Output"]

    INPUT --> H1
    INPUT --> H2
    INPUT --> H3
    INPUT --> H4

    H1 --> CONCAT
    H2 --> CONCAT
    H3 --> CONCAT
    H4 --> CONCAT

    CONCAT --> PROJECTION
    PROJECTION --> OUTPUT

🧠 Why Multiple Heads?¶

Different heads can learn different relationships.

For example:

Head 1
→ Local Relationships

Head 2
→ Syntactic Relationships

Head 3
→ Semantic Relationships

Head 4
→ Long-Range Relationships

These interpretations are conceptual rather than guaranteed fixed roles.

🧠 Attention Head Dimensions¶

Suppose:

Model Dimension = 512
Number of Heads = 8

Then:

[ d_{head}=\frac{512}{8}=64 ]

The heads operate in separate lower-dimensional subspaces before their outputs are concatenated.

🧠 Feed-Forward Network¶

Attention determines:

Which information should interact?

The Feed-Forward Network transforms each token representation independently.

A standard Transformer FFN is:

[ FFN(x)=\sigma(xW_1+b_1)W_2+b_2 ]

where:

W₁ = First Projection
W₂ = Second Projection
σ  = Activation Function

Modern architectures may use different activation functions and FFN variants.

🧠 Feed-Forward Network Architecture¶

Input
  ↓
Linear Projection
  ↓
Activation
  ↓
Linear Projection
  ↓
Output

🧠 Why Does the Transformer Need an FFN?¶

Attention primarily mixes information across positions.

The FFN then performs nonlinear transformation on each position.

Conceptually:

Self-Attention
      ↓
Mix Information Across Tokens
      ↓
Feed-Forward Network
      ↓
Transform Each Token Representation

🧠 Attention + FFN¶

flowchart LR

    INPUT["Token Representations"]

    ATTENTION["Self-Attention"]

    FFN["Feed-Forward Network"]

    OUTPUT["Contextual Representations"]

    INPUT --> ATTENTION
    ATTENTION --> FFN
    FFN --> OUTPUT

🧠 Residual Connections¶

Transformers use residual connections around major sublayers.

The basic idea is:

[ y=x+F(x) ]

Instead of forcing the layer to learn an entirely new representation, the network learns a transformation on top of the existing representation.

🧠 Residual Connection¶

          ┌────────────────────┐
          │                    │
Input ────┼──► Transformer ────┼──► Add
          │         Block       │
          │                    │
          └────────────────────┘
                    │
                    ▼
                  Output

🧠 Why Residual Connections?¶

Residual connections help:

Gradient Flow
      ↓
Deep Network Training
      ↓
Stable Optimization

They are especially important when many Transformer blocks are stacked.

🧠 Layer Normalization¶

Transformer architectures use normalization to stabilize activations.

Layer Normalization normalizes features within an individual example rather than across the batch.

Conceptually:

Token Representation
        ↓
Layer Normalization
        ↓
Normalized Representation

🧠 Layer Normalization¶

For a feature vector:

[ \hat{x}=\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}} ]

A learnable scale and bias are generally applied afterward.

🧠 Why LayerNorm?¶

Layer normalization can help:

Stable Activations
+
Stable Gradient Flow
+
Reliable Deep Training

It is particularly suitable for sequence models because it does not depend on batch statistics in the same way BatchNorm does.

🧠 Add & Norm¶

A simplified Transformer sublayer can be visualized as:

Input
 │
 ├─────────────────┐
 │                 │
 ▼                 │
Sublayer           │
 │                 │
 └──────────► Add ◄┘
               │
               ▼
        Layer Normalization
               │
               ▼
             Output

🧠 Post-Norm vs Pre-Norm¶

Two common arrangements are:

Post-Norm¶

x
 ↓
Sublayer
 ↓
Add
 ↓
LayerNorm

Pre-Norm¶

x
 ↓
LayerNorm
 ↓
Sublayer
 ↓
Add

Modern large Transformer architectures commonly use pre-normalization or related variants because of training-stability considerations.

🧠 Transformer Encoder Block¶

A conceptual encoder block can be represented as:

Input
 ↓
LayerNorm
 ↓
Multi-Head Self-Attention
 ↓
Residual Add
 ↓
LayerNorm
 ↓
Feed-Forward Network
 ↓
Residual Add
 ↓
Output

🧠 Encoder Block¶

flowchart TD

    X["Input"]

    N1["LayerNorm"]

    ATT["Multi-Head Self-Attention"]

    ADD1["Residual Add"]

    N2["LayerNorm"]

    FFN["Feed-Forward Network"]

    ADD2["Residual Add"]

    Y["Output"]

    X --> N1
    N1 --> ATT
    X --> ADD1
    ATT --> ADD1

    ADD1 --> N2
    N2 --> FFN

    ADD1 --> ADD2
    FFN --> ADD2

    ADD2 --> Y

🧠 Stacking Transformer Encoder Blocks¶

A Transformer rarely uses only one block.

Instead:

Input
 ↓
Encoder Block 1
 ↓
Encoder Block 2
 ↓
Encoder Block 3
 ↓
...
 ↓
Encoder Block N
 ↓
Output

🧠 Deep Transformer¶

flowchart TD

    INPUT["Input Embeddings"]

    B1["Transformer Block 1"]

    B2["Transformer Block 2"]

    B3["Transformer Block 3"]

    BN["Transformer Block N"]

    OUTPUT["Encoder Representation"]

    INPUT --> B1
    B1 --> B2
    B2 --> B3
    B3 --> BN
    BN --> OUTPUT

🧠 Transformer Decoder¶

The original Transformer decoder contains:

Masked Self-Attention
+
Cross-Attention
+
Feed-Forward Network

with residual connections and normalization around the sublayers.

🧠 Decoder Block¶

Input
 ↓
Masked Self-Attention
 ↓
Add & Norm
 ↓
Cross-Attention
 ↓
Add & Norm
 ↓
Feed-Forward Network
 ↓
Add & Norm
 ↓
Output

🧠 Transformer Decoder Architecture¶

flowchart TD

    INPUT["Decoder Input"]

    MASKED["Masked Self-Attention"]

    ADD1["Residual + Norm"]

    CROSS["Cross-Attention"]

    ADD2["Residual + Norm"]

    FFN["Feed-Forward Network"]

    ADD3["Residual + Norm"]

    OUTPUT["Decoder Output"]

    INPUT --> MASKED
    MASKED --> ADD1

    ADD1 --> CROSS
    CROSS --> ADD2

    ADD2 --> FFN
    FFN --> ADD3

    ADD3 --> OUTPUT

🧠 Masked Self-Attention¶

The decoder's self-attention is masked so the model cannot see future target tokens.

For:

I love machine learning

when predicting:

machine

the model can use:

I
love

but not future tokens.

🧠 Decoder Causal Mask¶

       Token

       1  2  3  4

1      ✓  ✗  ✗  ✗
2      ✓  ✓  ✗  ✗
3      ✓  ✓  ✓  ✗
4      ✓  ✓  ✓  ✓

This prevents information leakage during autoregressive generation.

🧠 Cross-Attention in Decoder¶

The decoder can attend to encoder outputs.

Decoder Query
       ↓
Cross-Attention
       ↑
Encoder Keys + Values

This allows the decoder to retrieve relevant information from the encoded source sequence.

🧠 Full Encoder-Decoder Transformer¶

flowchart LR

    INPUT["Source Tokens"]

    EMBED1["Source Embedding + Position"]

    ENC["Encoder Stack"]

    MEMORY["Encoder Representations"]

    TARGET["Target Tokens"]

    EMBED2["Target Embedding + Position"]

    DEC["Decoder Stack"]

    HEAD["Linear + Softmax"]

    OUTPUT["Output Tokens"]

    INPUT --> EMBED1
    EMBED1 --> ENC
    ENC --> MEMORY

    TARGET --> EMBED2
    EMBED2 --> DEC

    MEMORY --> DEC

    DEC --> HEAD
    HEAD --> OUTPUT

🧠 Encoder-Only Transformer¶

Encoder-only models use:

Input
 ↓
Encoder Blocks
 ↓
Contextual Representations
 ↓
Task Head

Typical tasks:

Text Classification
Named Entity Recognition
Embedding Generation
Semantic Similarity

🧠 Encoder-Only Architecture¶

flowchart TD

    INPUT["Input Tokens"]

    EMBED["Embedding + Position"]

    ENCODER["Encoder Stack"]

    REPRESENTATION["Contextual Representation"]

    HEAD["Task Head"]

    OUTPUT["Prediction"]

    INPUT --> EMBED
    EMBED --> ENCODER
    ENCODER --> REPRESENTATION
    REPRESENTATION --> HEAD
    HEAD --> OUTPUT

🧠 Decoder-Only Transformer¶

Decoder-only architectures use:

Causal Self-Attention
+
Feed-Forward Networks

They are particularly suited to autoregressive generation.

Pipeline:

Prompt
 ↓
Decoder Blocks
 ↓
Next-Token Probabilities
 ↓
Selected Token
 ↓
Append Token
 ↓
Repeat

🧠 Decoder-Only Architecture¶

flowchart TD

    PROMPT["Prompt Tokens"]

    EMBED["Embedding + Position"]

    DECODER["Decoder-Only Transformer"]

    LMHEAD["Language Model Head"]

    LOGITS["Next-Token Logits"]

    TOKEN["Next Token"]

    PROMPT --> EMBED
    EMBED --> DECODER
    DECODER --> LMHEAD
    LMHEAD --> LOGITS
    LOGITS --> TOKEN

🧠 Encoder-Decoder Transformer¶

Encoder-decoder architectures use:

Encoder
+
Decoder

They are particularly useful for sequence-to-sequence tasks.

Examples:

Translation
Summarization
Text Transformation

🧠 Transformer Architecture Types¶

Architecture	Main Mechanism	Typical Use
Encoder-only	Bidirectional Self-Attention	Understanding
Decoder-only	Causal Self-Attention	Generation
Encoder-decoder	Encoder + Cross-Attention Decoder	Sequence-to-sequence

🧠 Transformer Data Flow¶

A simplified Transformer pipeline:

Raw Input
   ↓
Tokenizer
   ↓
Token IDs
   ↓
Embedding
   ↓
Positional Information
   ↓
Transformer Blocks
   ↓
Contextual Representation
   ↓
Task / Language Model Head
   ↓
Output

🧠 Transformer Token Processing¶

For:

"Machine learning is powerful"

the process becomes:

Tokens
 ↓
[Machine, learning, is, powerful]
 ↓
Token IDs
 ↓
Embeddings
 ↓
Position Information
 ↓
Self-Attention
 ↓
Contextual Representations

After multiple Transformer layers, each token representation incorporates information from the relevant context.

🧠 Contextual Embeddings¶

A static embedding:

bank
 ↓
One Vector

does not necessarily capture the meaning of every context.

Transformer representations are contextual:

bank + river
      ↓
Contextual Representation A

bank + finance
      ↓
Contextual Representation B

The same token can therefore have different contextual representations depending on surrounding information.

🧠 Transformer as Context Builder¶

Token
  +
Surrounding Tokens
  ↓
Self-Attention
  ↓
Contextual Representation

Repeated across layers:

Context
 ↓
More Context
 ↓
Higher-Level Representation

🧠 Transformer Layer Processing¶

A Transformer layer can be understood as:

Input Representation
       ↓
Attention
       ↓
Information Mixing
       ↓
Feed-Forward Transformation
       ↓
Output Representation

Repeated many times:

Layer 1
 ↓
Layer 2
 ↓
Layer 3
 ↓
...
Layer N

🧠 Transformer Tensor Shapes¶

Suppose:

Batch Size = B
Sequence Length = T
Model Dimension = D

The input tensor is:

[ X\in\mathbb{R}^{B\times T\times D} ]

For example:

B = 32
T = 128
D = 512

Input shape:

[32, 128, 512]

🧠 Attention Tensor Shapes¶

For:

Batch = B
Heads = H
Sequence Length = T
Head Dimension = Dh

the projected tensors become:

Q → [B, H, T, Dh]
K → [B, H, T, Dh]
V → [B, H, T, Dh]

The attention score matrix is:

[B, H, T, T]

🧠 Why T × T Matters¶

The attention matrix contains relationships between every pair of sequence positions.

For:

T = 4

we get:

4 × 4

For:

T = 1024

we get:

1024 × 1024

which is:

[ 1,048,576 ]

attention positions per head for one sequence.

⚠ Transformer Complexity¶

Standard self-attention has approximately quadratic complexity with respect to sequence length:

[ O(T^2D) ]

where:

T = Sequence Length
D = Model Dimension

This becomes a major consideration for long-context systems.

🧠 Transformer Complexity Visualization¶

Sequence Length
      │
      │
      │                 █
      │
      │          █
      │
      │      █
      │
      │   █
      │ █
      └────────────────────
          Attention Cost

Conceptually:

Short Context
    ↓
Low Attention Cost

Long Context
    ↓
Rapidly Increasing Cost

🧠 Why Transformers Scale Well During Training¶

Although attention has quadratic sequence complexity, Transformer training can perform many operations in parallel using matrix operations.

Compared with RNNs:

RNN

Time Step 1
    ↓
Time Step 2
    ↓
Time Step 3
    ↓
Time Step 4

Transformer:

Tokens
 ↓
Large Matrix Operations
 ↓
GPU Parallelism

This is one of the key reasons Transformers became dominant for large-scale sequence modeling.

🧠 Autoregressive Generation¶

Decoder-only Transformers generate tokens one at a time during inference.

Example:

Prompt:
"The weather is"

      ↓

Token 1:
"good"

      ↓

"The weather is good"

      ↓

Token 2:
"today"

      ↓

"The weather is good today"

The process continues until:

EOS

or another stopping condition.

🧠 Autoregressive Generation¶

flowchart LR

    PROMPT["Prompt"]

    MODEL["Transformer"]

    LOGITS["Next Token Logits"]

    SELECT["Token Selection"]

    APPEND["Append Token"]

    NEXT["Updated Sequence"]

    PROMPT --> MODEL
    MODEL --> LOGITS
    LOGITS --> SELECT
    SELECT --> APPEND
    APPEND --> NEXT
    NEXT --> MODEL

🧠 Language Model Head¶

The Transformer hidden representation is projected into vocabulary space.

If:

Hidden Dimension = D
Vocabulary Size = V

then the language model head produces:

[B, T, V]

logits.

The probability distribution is:

[ P(token_i|context)=softmax(logits) ]

🧠 Next Token Prediction¶

The model estimates:

P(token₁ | context)
P(token₂ | context)
P(token₃ | context)
...
P(tokenV | context)

Then a decoding strategy selects the next token.

Common strategies include:

Greedy Decoding
Sampling
Temperature
Top-k
Top-p
Beam Search

These are covered further in modern generative AI systems.

🧠 KV Cache¶

During autoregressive generation, the model repeatedly processes an expanding sequence.

Without caching:

Token 1
 ↓
Recompute

Token 1 + Token 2
 ↓
Recompute

Token 1 + Token 2 + Token 3
 ↓
Recompute

KV caching stores previously calculated:

Keys
+
Values

so they can be reused.

🧠 KV Cache Concept¶

flowchart LR

    CURRENT["Current Token"]

    Q["Current Query"]

    CACHE["Cached Keys + Values"]

    ATTENTION["Attention"]

    OUTPUT["Next Token Representation"]

    CURRENT --> Q
    CACHE --> ATTENTION
    Q --> ATTENTION
    ATTENTION --> OUTPUT

🧠 Why KV Cache Matters¶

KV caching improves autoregressive generation efficiency by avoiding unnecessary recomputation of previous Key and Value representations.

It is especially important for:

LLM Serving
Long Conversations
High Throughput Inference
Interactive AI

🧠 Transformer Training vs Inference¶

Training¶

Many Tokens
     ↓
Parallel Matrix Operations
     ↓
GPU
     ↓
Efficient Training

Autoregressive Inference¶

Token 1
 ↓
Token 2
 ↓
Token 3
 ↓
Token 4

Generation remains sequential at the token level.

KV caching reduces repeated computation but does not make autoregressive generation fully parallel.

🧠 Transformer Training¶

A typical training flow:

Dataset
 ↓
Tokenization
 ↓
Batching
 ↓
Transformer
 ↓
Logits
 ↓
Loss
 ↓
Backpropagation
 ↓
Optimizer
 ↓
Parameter Update

🧠 Language Model Training¶

For next-token prediction:

Input:

The cat is

Target:

cat is sleeping

The model learns:

P(cat | The)
P(is | The cat)
P(sleeping | The cat is)

using causal masking.

🧠 Cross-Entropy Loss¶

For classification or next-token prediction, cross-entropy is commonly used.

For a target class:

[ L=-\log P(y|x) ]

Higher probability assigned to the correct target produces lower loss.

🧠 Transformer Training Loop¶

for batch in train_loader:

    input_ids = batch["input_ids"]
    labels = batch["labels"]

    optimizer.zero_grad()

    logits = model(
        input_ids
    )

    loss = criterion(
        logits,
        labels
    )

    loss.backward()

    optimizer.step()

In production training, additional components are commonly required:

Mixed Precision
Gradient Clipping
Learning Rate Scheduling
Checkpointing
Distributed Training
Experiment Tracking
Validation
Monitoring

🐍 Part I — PyTorch Transformer¶

PyTorch provides Transformer components such as:

torch.nn.Transformer
torch.nn.TransformerEncoder
torch.nn.TransformerEncoderLayer
torch.nn.TransformerDecoder
torch.nn.MultiheadAttention

These can be used to construct Transformer-based models.

🧪 Transformer Encoder Layer¶

A simple encoder layer can be created using:

import torch.nn as nn


encoder_layer = nn.TransformerEncoderLayer(
    d_model=512,
    nhead=8,
    batch_first=True
)

🧪 Transformer Encoder¶

encoder = nn.TransformerEncoder(
    encoder_layer,
    num_layers=6
)

Conceptually:

Input
 ↓
Encoder Layer 1
 ↓
Encoder Layer 2
 ↓
Encoder Layer 3
 ↓
...
 ↓
Encoder Layer 6
 ↓
Output

🧪 Transformer Encoder Classifier¶

class TransformerClassifier(
    nn.Module
):

    def __init__(
        self,
        vocab_size,
        d_model,
        nhead,
        num_layers,
        num_classes
    ):

        super().__init__()

        self.embedding = nn.Embedding(
            vocab_size,
            d_model
        )

        encoder_layer = (
            nn.TransformerEncoderLayer(
                d_model=d_model,
                nhead=nhead,
                batch_first=True
            )
        )

        self.encoder = (
            nn.TransformerEncoder(
                encoder_layer,
                num_layers=num_layers
            )
        )

        self.fc = nn.Linear(
            d_model,
            num_classes
        )

    def forward(
        self,
        input_ids
    ):

        x = self.embedding(
            input_ids
        )

        x = self.encoder(
            x
        )

        pooled = x[:, 0]

        return self.fc(
            pooled
        )

🧠 Transformer Classifier Architecture¶

flowchart TD

    TOKENS["Token IDs"]

    EMBED["Embedding"]

    ENCODER["Transformer Encoder Stack"]

    POOL["Sequence Representation"]

    FC["Classification Head"]

    OUTPUT["Class Prediction"]

    TOKENS --> EMBED
    EMBED --> ENCODER
    ENCODER --> POOL
    POOL --> FC
    FC --> OUTPUT

🧪 Transformer Configuration¶

Example:

model = TransformerClassifier(
    vocab_size=30000,
    d_model=256,
    nhead=8,
    num_layers=6,
    num_classes=3
)

This configuration means:

Vocabulary = 30,000
Model Dimension = 256
Attention Heads = 8
Encoder Layers = 6
Classes = 3

🧠 Attention Head Dimension¶

For:

d_model = 256
nhead = 8

we get:

[ d_{head}=\frac{256}{8}=32 ]

🧪 Transformer Mask¶

For causal modeling, a causal mask can be created.

seq_len = 128

mask = (
    torch.triu(
        torch.ones(
            seq_len,
            seq_len
        ),
        diagonal=1
    ).bool()
)

This identifies future positions that should be blocked.

🧠 Transformer Attention Masks¶

Production Transformer systems may need multiple masks:

Padding Mask
+
Causal Mask
+
Application-Specific Mask

The exact masking strategy depends on the architecture.

🧠 Transformer Encoder vs Decoder¶

Encoder¶

Input
 ↓
Bidirectional Self-Attention
 ↓
FFN
 ↓
Output

Decoder¶

Input
 ↓
Causal Self-Attention
 ↓
Cross-Attention
 ↓
FFN
 ↓
Output

🧠 Architecture Comparison¶

Component	Encoder	Decoder
Self-Attention	Yes	Yes
Causal Mask	Usually No	Yes for autoregressive decoding
Cross-Attention	No	Yes in encoder-decoder architecture
FFN	Yes	Yes
Residual Connections	Yes	Yes
LayerNorm	Yes	Yes

🧠 Original Transformer vs Modern LLMs¶

The original Transformer was an:

Encoder
+
Decoder

architecture.

Modern LLMs often use:

Decoder-Only Transformer

with:

Causal Self-Attention
+
Feed-Forward Networks
+
Positional Representation
+
Residual Connections
+
Normalization

🧠 Modern LLM Architecture¶

flowchart TD

    INPUT["Prompt Tokens"]

    EMBED["Token Embeddings"]

    POS["Positional Representation"]

    BLOCK1["Transformer Block"]

    BLOCK2["Transformer Block"]

    BLOCKN["Transformer Block N"]

    LMHEAD["Language Model Head"]

    LOGITS["Vocabulary Logits"]

    TOKEN["Next Token"]

    INPUT --> EMBED
    POS --> BLOCK1
    EMBED --> BLOCK1

    BLOCK1 --> BLOCK2
    BLOCK2 --> BLOCKN

    BLOCKN --> LMHEAD
    LMHEAD --> LOGITS
    LOGITS --> TOKEN

🧠 Transformer → LLM¶

A Large Language Model is not simply:

Transformer
+
More Layers

A production LLM ecosystem also involves:

Large-Scale Pretraining
+
Massive Datasets
+
Distributed Training
+
Optimization
+
Tokenizer
+
Evaluation
+
Alignment / Post-Training
+
Inference Infrastructure
+
Safety / Governance

🧠 Transformer Evolution¶

Original Transformer
        ↓
Encoder Models
        ↓
Decoder Models
        ↓
Large-Scale Pretraining
        ↓
Foundation Models
        ↓
Large Language Models
        ↓
Multimodal Models
        ↓
Modern Generative AI

🧠 Transformer Architecture Landscape¶

                     Transformer
                          │
            ┌─────────────┼─────────────┐
            ▼             ▼             ▼
       Encoder-only   Decoder-only   Encoder-Decoder
            │             │             │
            ▼             ▼             ▼
       Understanding   Generation   Seq2Seq
            │             │             │
            ▼             ▼             ▼
       Embeddings        LLMs       Translation
       Classification    Code       Summarization
       Retrieval         Chat       Transformation

🏢 Enterprise Transformer Architecture¶

A production Transformer system may look like:

Client
  ↓
API Gateway
  ↓
Inference Service
  ↓
Tokenizer
  ↓
Model Runtime
  ↓
Transformer
  ↓
Post Processing
  ↓
Response

🏢 Production Transformer Architecture¶

flowchart TD

    CLIENT["Client Application"]

    API["API Gateway"]

    SERVICE["Inference Service"]

    TOKENIZER["Tokenizer"]

    RUNTIME["Model Runtime"]

    TRANSFORMER["Transformer Model"]

    POST["Post Processing"]

    RESPONSE["Response"]

    CLIENT --> API
    API --> SERVICE
    SERVICE --> TOKENIZER
    TOKENIZER --> RUNTIME
    RUNTIME --> TRANSFORMER
    TRANSFORMER --> POST
    POST --> RESPONSE
    RESPONSE --> CLIENT

🏢 Production Transformer Concerns¶

A production Transformer system must consider:

Latency
Throughput
GPU Memory
Context Length
Batch Size
Model Size
Quantization
KV Cache
Concurrency
Autoscaling
Observability
Model Versioning
Cost
Security

🏢 GPU Memory¶

Transformer inference can require substantial memory because of:

Model Parameters
+
Activations
+
Attention Buffers
+
KV Cache
+
Batch Size

Therefore:

Long Context
+
Large Batch
+
Large Model

can rapidly increase GPU memory requirements.

🏢 KV Cache and Serving¶

For decoder-only LLMs:

Prompt
 ↓
Prefill
 ↓
KV Cache
 ↓
Token Generation
 ↓
Reuse KV Cache

This creates two important inference phases:

Prefill
+
Decode

🧠 Prefill¶

During prefill:

Entire Prompt
      ↓
Transformer
      ↓
Compute Prompt Representations
      ↓
KV Cache

The prompt can generally be processed in parallel.

🧠 Decode¶

During decoding:

Current Token
      ↓
Transformer
      ↓
Next Token
      ↓
Repeat

The process is sequential at the token level.

KV caching avoids recomputing previous Key and Value representations.

🧠 Prefill vs Decode¶

flowchart LR

    PROMPT["Prompt"]

    PREFILL["Prefill"]

    CACHE["KV Cache"]

    DECODE["Decode"]

    TOKEN["Next Token"]

    PROMPT --> PREFILL
    PREFILL --> CACHE
    CACHE --> DECODE
    DECODE --> TOKEN
    TOKEN --> DECODE

🏢 Transformer Observability¶

Production monitoring should cover:

Infrastructure¶

GPU Utilization
GPU Memory
CPU Utilization
Memory
Network

Model¶

Latency
Throughput
Token Generation Rate
Error Rate
Output Quality

Request¶

Input Tokens
Output Tokens
Total Tokens
Context Length
Batch Size

Serving¶

Queue Time
Prefill Latency
Decode Latency
P50 Latency
P95 Latency
P99 Latency

🏢 Cost Monitoring¶

For LLM workloads, cost is often related to:

Input Tokens
+
Output Tokens
+
Model Size
+
GPU Time

Therefore production teams should track:

Tokens per Request
GPU Seconds per Request
Requests per Second
Cost per Request
Cost per Token

🏢 Model Versioning¶

A production Transformer deployment should version:

Model
Tokenizer
Vocabulary
Prompt Templates
Configuration
Weights
Quantization
Runtime
Evaluation Dataset

For LLM applications also consider:

System Prompt
Tool Configuration
Retrieval Configuration
Safety Policies

🏢 Deployment Strategies¶

Transformer systems can be deployed using:

Dedicated GPU Servers
+
Managed ML Platforms
+
Containerized Inference
+
Model Serving Platforms
+
Serverless / Specialized Inference

The choice depends on:

Latency
Traffic
Model Size
Cost
Scaling Requirements
Operational Complexity

🏢 Transformer Scaling¶

At enterprise scale:

Client Requests
       ↓
Load Balancer
       ↓
Inference Workers
       ↓
GPU Pool
       ↓
Transformer Models

Autoscaling can respond to:

Request Rate
Queue Depth
GPU Utilization
Latency

🧠 Transformer Architecture Decision¶

When designing a Transformer system, ask:

What is the task?
      ↓
Understanding or Generation?
      ↓
Encoder-only?
Decoder-only?
Encoder-decoder?
      ↓
What is the context length?
      ↓
What latency is required?
      ↓
What hardware is available?
      ↓
What model size is appropriate?
      ↓
What serving strategy is required?

🧪 Practical Exercise 1 — Build Transformer Encoder¶

Create:

Embedding
+
Positional Information
+
Transformer Encoder
+
Classification Head

Train it on a sequence classification dataset.

🧪 Practical Exercise 2 — Inspect Attention¶

Capture attention weights and visualize:

Token × Token

attention matrices.

Analyze:

Which tokens interact?
Which heads behave differently?

🧪 Practical Exercise 3 — Causal Transformer¶

Build a decoder-style Transformer with:

Causal Mask

Verify that:

Current Token

cannot access:

Future Tokens

🧪 Practical Exercise 4 — Positional Encoding¶

Implement:

Sinusoidal Positional Encoding

and compare it with:

Learned Positional Embedding

🧪 Practical Exercise 5 — Multi-Head Attention¶

Configure:

d_model = 256
heads = 8

and verify:

head dimension = 32

🧪 Practical Exercise 6 — Transformer Depth¶

Compare:

2 Layers
4 Layers
6 Layers
8 Layers

Measure:

Accuracy
Training Time
Parameter Count
Memory

🧪 Practical Exercise 7 — Context Length¶

Benchmark:

128 tokens
256 tokens
512 tokens
1024 tokens

Measure:

GPU Memory
Attention Cost
Latency

🧪 Practical Exercise 8 — RNN vs Transformer¶

Train:

LSTM

and:

Transformer Encoder

on the same dataset.

Compare:

Accuracy
Training Time
Inference Latency
Memory
Long-Range Dependency Performance

🧪 Practical Exercise 9 — KV Cache Concept¶

Build a simplified autoregressive decoder.

Measure generation time:

Without KV Cache

vs:

With KV Cache

Observe how caching affects repeated computation.

🧪 Practical Exercise 10 — Production Benchmark¶

Benchmark a Transformer under different:

Batch Sizes
Context Lengths
Sequence Lengths
Model Sizes

Record:

P50 Latency
P95 Latency
Throughput
GPU Memory
Cost

🧠 Interview Questions¶

Beginner¶

1. What is a Transformer?¶

A Transformer is a neural network architecture based primarily on attention mechanisms rather than recurrent sequence processing.

2. Why were Transformers introduced?¶

They were introduced to improve sequence modeling by enabling stronger parallelism and direct modeling of relationships between sequence positions.

3. What are the main components of a Transformer?¶

Embeddings
Positional Information
Attention
Feed-Forward Networks
Residual Connections
Normalization

4. What is self-attention?¶

Self-attention allows tokens within the same sequence to directly interact through Query-Key-Value attention.

5. What is multi-head attention?¶

Multi-head attention performs several attention operations in parallel and combines their outputs.

Intermediate¶

6. What is the Transformer attention equation?¶

[ Attention(Q,K,V) = softmax \left( \frac{QK^T}{\sqrt{d_k}} \right)V ]

7. Why does a Transformer need positional information?¶

Because self-attention by itself does not inherently encode the order of tokens.

8. What is the role of the FFN?¶

The FFN applies nonlinear transformations independently to each token representation after attention mixes contextual information.

9. What is a residual connection?¶

A shortcut that adds the input representation to the output of a sublayer.

[ y=x+F(x) ]

10. What is LayerNorm?¶

A normalization technique that normalizes feature representations within individual examples.

11. What is causal attention?¶

Attention that prevents a token from accessing future positions.

12. What is cross-attention?¶

Attention where Queries come from one representation and Keys/Values come from another.

Advanced¶

13. Why are Transformers more parallelizable than RNNs?¶

Transformers can process sequence relationships using matrix operations without requiring each time step to wait for the previous hidden state.

14. What is the complexity of standard self-attention?¶

Approximately:

[ O(T^2D) ]

where T is sequence length and D is model dimension.

15. Why does attention become expensive for long contexts?¶

Because every token can attend to every other token, producing an approximately T × T attention matrix.

16. What is the difference between encoder-only and decoder-only Transformers?¶

Encoder-only models are generally optimized for contextual understanding, while decoder-only models use causal attention for autoregressive generation.

17. What is the purpose of causal masking?¶

To prevent future-token information from leaking into predictions during autoregressive training.

18. What is KV caching?¶

A technique that stores previously computed Key and Value representations during autoregressive generation so they do not need to be recomputed.

19. What are prefill and decode phases?¶

Prefill processes the prompt and builds the KV cache. Decode generates new tokens sequentially using the cached context.

20. Why are Transformers effective for long-range relationships?¶

Self-attention provides direct paths between distant sequence positions rather than requiring information to propagate through many recurrent time steps.

21. Why are residual connections important?¶

They improve information and gradient flow through deep Transformer stacks.

22. Why is LayerNorm preferred over BatchNorm in many Transformers?¶

LayerNorm operates independently of batch statistics and works naturally with token-level sequence representations.

🏢 Enterprise Perspective¶

The Transformer is not just another neural network architecture.

It represents a fundamental change in how Deep Learning systems process context:

RNN Era
   ↓
Sequential Memory
   ↓
LSTM / GRU
   ↓
Attention
   ↓
Direct Context Interaction
   ↓
Transformer
   ↓
Large-Scale Pretraining
   ↓
Foundation Models

This architecture now underpins a large portion of modern:

Generative AI
Large Language Models
Code Models
Vision Transformers
Multimodal Models
Speech Models
Embedding Models

🏢 Production Transformer Stack¶

A modern enterprise AI platform may look like:

                    Client
                      │
                      ▼
                API Gateway
                      │
                      ▼
             AI Application Service
                      │
          ┌───────────┴───────────┐
          ▼                       ▼
      Retrieval               Tools / APIs
          │                       │
          └───────────┬───────────┘
                      ▼
                 Prompt Builder
                      │
                      ▼
                  Tokenizer
                      │
                      ▼
             Transformer Runtime
                      │
                      ▼
                  GPU Cluster
                      │
                      ▼
                Model Output
                      │
                      ▼
              Post Processing
                      │
                      ▼
                 Response

🏢 Transformer + RAG¶

A production RAG system commonly combines:

User Query
     ↓
Embedding
     ↓
Retriever
     ↓
Relevant Documents
     ↓
Context Construction
     ↓
Transformer / LLM
     ↓
Generated Response

The Transformer performs contextual reasoning/generation, while the retrieval system supplies external knowledge.

🏢 Transformer + Agentic AI¶

A modern agentic architecture may extend the Transformer with:

LLM
 ↓
Reasoning / Planning
 ↓
Tool Selection
 ↓
Tool Execution
 ↓
Observation
 ↓
Next Model Call
 ↓
Final Response

The Transformer provides the model intelligence, while orchestration components manage the surrounding workflow.

🏢 Production Optimization Areas¶

For enterprise Transformer workloads, optimization often happens across:

Model
 ↓
Quantization
 ↓
Attention Kernel
 ↓
KV Cache
 ↓
Batching
 ↓
GPU Utilization
 ↓
Serving Runtime
 ↓
Autoscaling

🏢 GPU-Aware Transformer Design¶

Production performance depends heavily on:

GPU Architecture
+
Memory Bandwidth
+
GPU Memory
+
Kernel Efficiency
+
Batch Size
+
Sequence Length

Therefore:

Transformer architecture and infrastructure architecture cannot be treated independently at production scale.

Production Insight

The Transformer is an architectural pattern, not the complete production AI system.

A production-grade Transformer application requires multiple engineering layers:

Model Architecture
      ↓
Model Weights
      ↓
Tokenization
      ↓
Inference Runtime
      ↓
GPU Infrastructure
      ↓
Serving Layer
      ↓
API / Microservice
      ↓
Observability
      ↓
Security & Governance

For large-scale AI systems, the most important engineering questions are not only:

"Which model should we use?"

but also:

How much context?
How much latency?
How much throughput?
How much GPU memory?
How much does each request cost?
How do we monitor it?
How do we version it?
How do we scale it?

📌 Key Takeaways¶

Transformers replaced recurrence as the dominant architecture for many modern sequence-modeling workloads.
The original Transformer uses an encoder-decoder architecture.
Modern Transformer systems commonly use encoder-only, decoder-only, or encoder-decoder configurations.
Token embeddings convert token IDs into dense vector representations.
Positional information provides sequence-order information.
Self-attention allows tokens to directly interact with other tokens.
Query, Key, and Value projections form the foundation of attention.
Scaled dot-product attention computes contextual representations.
Multi-head attention allows multiple attention subspaces to operate in parallel.
Feed-Forward Networks provide nonlinear transformation after attention.
Residual connections improve information and gradient flow.
Layer Normalization helps stabilize deep Transformer training.
Encoder blocks use self-attention and feed-forward networks.
Decoder blocks use masked self-attention and, in encoder-decoder architectures, cross-attention.
Causal masking prevents future-token information leakage.
Encoder-only models are commonly used for understanding and representation tasks.
Decoder-only models are widely used for autoregressive generation and LLMs.
Encoder-decoder models are useful for sequence-to-sequence tasks.
Standard self-attention has approximately quadratic complexity with sequence length.
KV caching improves autoregressive inference efficiency.
Transformer training is highly parallelizable compared with recurrent architectures.
Autoregressive generation remains sequential at the token level.
Production Transformer systems require careful attention to GPU memory, latency, throughput, context length, batching, and cost.
Transformers provide the architectural foundation for many modern foundation models and Generative AI systems.

📚 Further Reading¶

Continue with:

The next chapter explores how Transformer architectures are applied across NLP, computer vision, speech, multimodal AI, Generative AI, embeddings, and enterprise systems.

➡️ Next Chapter¶

28. Transformer Applications

Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.