13. TensorFlow and Keras Fundamentals¶

Learn the foundations of TensorFlow and Keras, understand tensors, computation, models, layers, datasets, training, evaluation, GPU acceleration, and the complete workflow for building Deep Learning models.

🎯 Learning Objectives¶

After completing this chapter, you will be able to:

Explain what TensorFlow is
Explain what Keras is
Understand the relationship between TensorFlow and Keras
Understand tensors and their dimensions
Work with TensorFlow tensors
Understand tensor shapes, ranks, and data types
Perform basic tensor operations
Understand broadcasting
Understand TensorFlow computation
Understand automatic differentiation at a high level
Understand Keras layers and models
Build models using Keras
Compile and train Keras models
Evaluate and predict using Keras models
Understand the Keras training workflow
Work with datasets using tf.data
Build input pipelines
Understand batching, shuffling, caching, and prefetching
Understand callbacks
Save and load Keras models
Use GPUs for TensorFlow training
Understand CPU vs GPU execution
Understand model parameters and trainable parameters
Understand inference vs training mode
Build classification and regression models using Keras
Understand when to use Sequential vs Functional API
Prepare for advanced Keras topics covered in later chapters

📖 Overview¶

TensorFlow is an open-source framework for numerical computation and Machine Learning.

It provides:

Tensor operations
Automatic differentiation
Neural network primitives
GPU and accelerator support
Data pipelines
Model training
Model serialization
Distributed training capabilities

Keras is a high-level Deep Learning API that provides a simpler interface for building and training neural networks.

The modern TensorFlow ecosystem can be viewed as:

TensorFlow
    │
    ├── Tensor Operations
    ├── Automatic Differentiation
    ├── GPU / Accelerator Execution
    ├── tf.data
    └── Keras
          │
          ├── Layers
          ├── Models
          ├── Losses
          ├── Metrics
          ├── Optimizers
          └── Training APIs

🧠 What Is TensorFlow?¶

TensorFlow is a framework for numerical computation and Machine Learning developed around tensor-based computation.

The fundamental data structure is the:

Tensor

A tensor is a multidimensional array with a defined shape and data type.

Examples:

Scalar
   ↓
Vector
   ↓
Matrix
   ↓
3D Tensor
   ↓
Higher-Dimensional Tensor

📐 Tensor Ranks¶

Tensor rank represents the number of dimensions of a tensor.

Rank 0 — Scalar¶

Rank 1 — Vector¶

[1, 2, 3]

Rank 2 — Matrix¶

[
  [1, 2],
  [3, 4]
]

Rank 3¶

[
  [
    [1, 2],
    [3, 4]
  ],
  [
    [5, 6],
    [7, 8]
  ]
]

Conceptually:

flowchart LR

    S["Scalar<br>Rank 0"]
    V["Vector<br>Rank 1"]
    M["Matrix<br>Rank 2"]
    T["Tensor<br>Rank 3+"]

    S --> V
    V --> M
    M --> T

🧮 Tensor Terminology¶

Three concepts are particularly important:

Rank
Shape
Data Type

For example:

tensor.shape

might return:

(32, 224, 224, 3)

This could represent:

Batch Size = 32
Height     = 224
Width      = 224
Channels   = 3

🐍 Creating TensorFlow Tensors¶

import tensorflow as tf


scalar = tf.constant(10)

vector = tf.constant(
    [1, 2, 3]
)

matrix = tf.constant(
    [
        [1, 2],
        [3, 4]
    ]
)

tensor = tf.constant(
    [
        [
            [1, 2],
            [3, 4]
        ],
        [
            [5, 6],
            [7, 8]
        ]
    ]
)

🔍 Inspecting a Tensor¶

x = tf.constant(
    [
        [1, 2, 3],
        [4, 5, 6]
    ]
)

print(x)
print(x.shape)
print(x.dtype)
print(tf.rank(x))

Example:

Shape:
(2, 3)

Rank:
2

Data Type:
int32

🧠 Tensor Shape¶

Tensor shape describes the size of each dimension.

For:

x.shape

returning:

(2, 3)

the tensor contains:

2 rows
3 columns

For an image batch:

(32, 224, 224, 3)

the dimensions commonly represent:

Batch
Height
Width
Channels

🖼️ Image Tensor Representation¶

A typical RGB image can be represented as:

Height × Width × Channels

For example:

224 × 224 × 3

A batch of 32 images becomes:

32 × 224 × 224 × 3

flowchart LR

    B["Batch<br>32"]
    H["Height<br>224"]
    W["Width<br>224"]
    C["Channels<br>3"]

    B --> H
    H --> W
    W --> C

🧠 Tensor Data Types¶

TensorFlow supports multiple numeric data types.

Common examples include:

float32
float64
int32
int64
uint8
bool

Deep Learning models commonly use:

float32

while image data may initially be:

uint8

🔄 Type Conversion¶

image = tf.constant(
    [0, 128, 255],
    dtype=tf.uint8
)

image = tf.cast(
    image,
    tf.float32
)

This is important when preparing data for neural networks.

🧮 Tensor Operations¶

TensorFlow supports mathematical operations directly on tensors.

a = tf.constant(
    [1, 2, 3]
)

b = tf.constant(
    [4, 5, 6]
)

print(a + b)
print(a - b)
print(a * b)
print(a / b)

✖️ Matrix Multiplication¶

Matrix multiplication is fundamental to neural networks.

a = tf.constant(
    [
        [1, 2],
        [3, 4]
    ],
    dtype=tf.float32
)

b = tf.constant(
    [
        [5, 6],
        [7, 8]
    ],
    dtype=tf.float32
)

result = tf.matmul(
    a,
    b
)

Mathematically:

[ C=AB ]

📐 Tensor Reshaping¶

Tensor shape can often be changed without changing the underlying values.

x = tf.constant(
    [
        [1, 2, 3],
        [4, 5, 6]
    ]
)

y = tf.reshape(
    x,
    (3, 2)
)

Original:

2 × 3

New shape:

3 × 2

🧩 Flattening¶

Flattening converts multiple dimensions into one.

x = tf.constant(
    [
        [1, 2],
        [3, 4]
    ]
)

flat = tf.reshape(
    x,
    [-1]
)

Result:

[1, 2, 3, 4]

This is frequently used when transitioning from convolutional layers to dense layers.

🔄 Broadcasting¶

TensorFlow supports broadcasting for compatible shapes.

For example:

x = tf.constant(
    [
        [1, 2],
        [3, 4]
    ],
    dtype=tf.float32
)

y = tf.constant(
    [10, 20],
    dtype=tf.float32
)

result = x + y

Conceptually:

[1, 2]       [10, 20]
[3, 4]   +   [10, 20]

Result:

[11, 22]
[13, 24]

🧠 Why Tensors Matter in Deep Learning¶

Neural networks operate primarily on tensors.

flowchart LR

    DATA["Raw Data"]
    TENSOR["Tensor"]
    LAYER["Neural Network Layer"]
    OUTPUT["Output Tensor"]

    DATA --> TENSOR
    TENSOR --> LAYER
    LAYER --> OUTPUT

Examples:

Images      → 4D tensors
Text        → 2D / 3D tensors
Audio       → 2D / 3D tensors
Video       → 5D tensors
Tabular     → 2D tensors

🧠 What Is Keras?¶

Keras is a high-level Deep Learning API designed to make neural network development easier and more readable.

It provides abstractions for:

Layers
Models
Loss functions
Optimizers
Metrics
Callbacks
Training
Evaluation
Prediction

Instead of implementing every training operation manually, Keras provides a structured workflow.

🏗 Keras Model Architecture¶

flowchart TD

    INPUT["Input"]
    L1["Layer 1"]
    L2["Layer 2"]
    L3["Layer 3"]
    OUTPUT["Output"]

    INPUT --> L1
    L1 --> L2
    L2 --> L3
    L3 --> OUTPUT

A Keras model is essentially a composition of layers connected according to an architecture.

🧱 Keras Layers¶

Common Keras layers include:

Dense
Conv2D
MaxPooling2D
Flatten
Dropout
BatchNormalization
Embedding
LSTM
GRU
MultiHeadAttention

Example:

dense = tf.keras.layers.Dense(
    128,
    activation="relu"
)

🧠 Dense Layer¶

A Dense layer performs an affine transformation followed by an optional activation.

[ y=f(Wx+b) ]

where:

(x) = input
(W) = weights
(b) = bias
(f) = activation function

🏗 Sequential Model¶

The simplest Keras model is the Sequential model.

model = tf.keras.Sequential([

    tf.keras.layers.Dense(
        128,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        64,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        10,
        activation="softmax"
    )
])

This represents:

Input
  ↓
Dense 128
  ↓
Dense 64
  ↓
Dense 10
  ↓
Output

The Sequential API is useful when the model is a simple linear stack of layers.

🧠 Input Shape¶

A model can explicitly define its input shape.

model = tf.keras.Sequential([

    tf.keras.Input(
        shape=(784,)
    ),

    tf.keras.layers.Dense(
        128,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        10,
        activation="softmax"
    )
])

Here:

Input = 784 features

🧠 Keras Model Summary¶

Keras provides a useful model summary.

model.summary()

Typical information includes:

Layer
Output Shape
Number of Parameters

For example:

Dense
(32, 128)
101,248 parameters

🧮 Trainable Parameters¶

For a Dense layer:

[ Parameters = InputFeatures\times OutputFeatures + OutputFeatures ]

For example:

Input  = 784
Output = 128

Then:

[ 784\times128+128 = 100480 ]

This represents:

Weights = 100352
Biases  = 128
Total   = 100480

🧠 Build → Compile → Fit → Evaluate → Predict¶

The basic Keras workflow is:

flowchart LR

    BUILD["Build Model"]
    COMPILE["Compile"]
    FIT["Fit"]
    EVAL["Evaluate"]
    PRED["Predict"]

    BUILD --> COMPILE
    COMPILE --> FIT
    FIT --> EVAL
    EVAL --> PRED

This is one of the most important workflows to remember.

⚙️ Compile¶

Before training, configure:

Optimizer
Loss
Metrics

Example:

model.compile(
    optimizer=tf.keras.optimizers.Adam(
        learning_rate=0.001
    ),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

🏋️ Fit¶

Training is performed using:

history = model.fit(
    X_train,
    y_train,
    epochs=20,
    batch_size=32,
    validation_data=(
        X_val,
        y_val
    )
)

📊 What Happens During `fit()`?¶

Conceptually:

flowchart TD

    BATCH["Training Batch"]

    BATCH --> FORWARD["Forward Pass"]

    FORWARD --> LOSS["Calculate Loss"]

    LOSS --> BACK["Backpropagation"]

    BACK --> GRAD["Gradients"]

    GRAD --> OPT["Optimizer"]

    OPT --> UPDATE["Update Weights"]

    UPDATE --> NEXT["Next Batch"]

    NEXT --> BATCH

Keras handles much of this training loop automatically.

📉 Training History¶

The object returned by fit() contains training history.

history.history.keys()

Typical values:

loss
accuracy
val_loss
val_accuracy

📈 Plotting Training Curves¶

import matplotlib.pyplot as plt


plt.figure(figsize=(10, 6))

plt.plot(
    history.history["loss"],
    label="Training Loss"
)

plt.plot(
    history.history["val_loss"],
    label="Validation Loss"
)

plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.title("Training vs Validation Loss")

plt.legend()
plt.grid(True)

plt.show()

Training curves help identify:

Underfitting
Overfitting
Good Convergence
Training Instability

🧪 Evaluate¶

After training:

results = model.evaluate(
    X_test,
    y_test
)

This evaluates the model on unseen data.

🔮 Predict¶

Prediction:

predictions = model.predict(
    X_test
)

For a classification model using Softmax:

Output:

[
    0.02,
    0.01,
    0.91,
    0.06
]

The largest probability can be selected as the predicted class.

🧠 Classification Example¶

For a 10-class classification problem:

model = tf.keras.Sequential([

    tf.keras.Input(
        shape=(784,)
    ),

    tf.keras.layers.Dense(
        128,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        64,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        10,
        activation="softmax"
    )
])

Compile:

model.compile(
    optimizer="adam",
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

🧠 Regression Example¶

For regression, the output layer commonly has one unit with no classification activation.

model = tf.keras.Sequential([

    tf.keras.Input(
        shape=(10,)
    ),

    tf.keras.layers.Dense(
        64,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        32,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        1
    )
])

Compile:

model.compile(
    optimizer="adam",
    loss="mse",
    metrics=["mae"]
)

🧠 Classification vs Regression¶

Problem	Output Layer	Typical Loss
Binary Classification	1 + Sigmoid	Binary Cross-Entropy
Multi-Class Classification	N + Softmax	Categorical Cross-Entropy
Integer-Labeled Multi-Class	N + Softmax	Sparse Categorical Cross-Entropy
Regression	1 or more linear outputs	MSE / MAE

📦 TensorFlow `tf.data`¶

TensorFlow provides the tf.data API for building efficient input pipelines.

A basic pipeline:

dataset = tf.data.Dataset.from_tensor_slices(
    (X_train, y_train)
)

dataset = dataset.shuffle(
    buffer_size=10000
)

dataset = dataset.batch(
    32
)

dataset = dataset.prefetch(
    tf.data.AUTOTUNE
)

🔄 `tf.data` Pipeline¶

flowchart LR

    RAW["Raw Data"]
    CREATE["Dataset"]
    SHUFFLE["Shuffle"]
    BATCH["Batch"]
    PREFETCH["Prefetch"]
    MODEL["Model"]

    RAW --> CREATE
    CREATE --> SHUFFLE
    SHUFFLE --> BATCH
    BATCH --> PREFETCH
    PREFETCH --> MODEL

🔀 Shuffle¶

Shuffling helps prevent the model from learning unwanted ordering patterns.

dataset = dataset.shuffle(
    buffer_size=10000
)

For training data, shuffling is generally useful.

For evaluation datasets, deterministic ordering is usually preferred.

📦 Batch¶

Batching groups examples together.

dataset = dataset.batch(
    32
)

Conceptually:

Dataset
   ↓
32 examples
   ↓
Model
   ↓
Gradient Update

⚡ Prefetch¶

Prefetching allows the input pipeline to prepare future batches while the model processes the current batch.

dataset = dataset.prefetch(
    tf.data.AUTOTUNE
)

Conceptually:

flowchart LR

    DATA["Input Pipeline"]

    DATA --> B1["Batch N"]
    DATA --> B2["Prepare Batch N+1"]

    B1 --> GPU["GPU Training"]

    B2 --> GPU

This can improve hardware utilization by reducing input pipeline waiting time.

💾 Cache¶

Caching can reduce repeated data-loading work.

dataset = dataset.cache()

However, caching the entire dataset in memory may not be appropriate for large datasets.

Possible approaches include:

Memory Cache
Disk Cache
No Cache

Choose based on dataset size and infrastructure.

🧠 Complete `tf.data` Pipeline¶

A common training pipeline:

train_ds = (
    tf.data.Dataset
    .from_tensor_slices(
        (X_train, y_train)
    )
    .shuffle(10000)
    .batch(32)
    .prefetch(
        tf.data.AUTOTUNE
    )
)

Validation:

val_ds = (
    tf.data.Dataset
    .from_tensor_slices(
        (X_val, y_val)
    )
    .batch(32)
    .prefetch(
        tf.data.AUTOTUNE
    )
)

🧠 Data Pipeline Performance¶

A production pipeline should aim for:

Storage
   ↓
Data Loading
   ↓
Preprocessing
   ↓
Batching
   ↓
Prefetch
   ↓
GPU

The goal is to prevent:

GPU waiting for data

🔌 Keras Callbacks¶

Callbacks allow additional behavior during training.

Common callbacks include:

EarlyStopping
ModelCheckpoint
ReduceLROnPlateau
LearningRateScheduler
TensorBoard

⏹️ EarlyStopping¶

early_stopping = tf.keras.callbacks.EarlyStopping(
    monitor="val_loss",
    patience=5,
    restore_best_weights=True
)

Use:

model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=100,
    callbacks=[
        early_stopping
    ]
)

💾 ModelCheckpoint¶

checkpoint = tf.keras.callbacks.ModelCheckpoint(
    "best_model.keras",
    monitor="val_loss",
    save_best_only=True
)

This ensures that the best validation model can be preserved.

📉 ReduceLROnPlateau¶

reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(
    monitor="val_loss",
    factor=0.5,
    patience=3,
    min_lr=1e-6
)

This reduces the learning rate when validation performance stops improving.

📊 TensorBoard¶

TensorBoard provides tools for inspecting:

Training metrics
Loss curves
Learning rates
Histograms
Model graphs
Profiling information

Example:

tensorboard = tf.keras.callbacks.TensorBoard(
    log_dir="./logs"
)

🧠 Complete Callback Configuration¶

callbacks = [

    tf.keras.callbacks.EarlyStopping(
        monitor="val_loss",
        patience=5,
        restore_best_weights=True
    ),

    tf.keras.callbacks.ModelCheckpoint(
        "best_model.keras",
        monitor="val_loss",
        save_best_only=True
    ),

    tf.keras.callbacks.ReduceLROnPlateau(
        monitor="val_loss",
        factor=0.5,
        patience=3
    )
]

💾 Saving a Keras Model¶

Modern Keras supports saving complete models.

model.save(
    "my_model.keras"
)

Load:

loaded_model = tf.keras.models.load_model(
    "my_model.keras"
)

A complete model artifact can include:

Architecture
Weights
Training Configuration
Optimizer State

depending on the saving configuration.

💾 Saving Weights Only¶

model.save_weights(
    "model.weights.h5"
)

Load:

model.load_weights(
    "model.weights.h5"
)

This is useful when the architecture is reconstructed separately.

🧠 Model Serialization Strategy¶

flowchart TD

    MODEL["Trained Model"]

    MODEL --> FULL["Complete Model Artifact"]
    MODEL --> WEIGHTS["Weights Only"]

    FULL --> DEPLOY["Load for Inference"]
    WEIGHTS --> ARCH["Recreate Architecture"]
    ARCH --> DEPLOY

🖥️ CPU vs GPU¶

TensorFlow can execute operations on different devices.

Common devices include:

CPU
GPU
TPU

For Deep Learning workloads:

CPU
 ↓
General Purpose

GPU
 ↓
Highly Parallel Tensor Computation

🚀 Checking GPU Availability¶

gpus = tf.config.list_physical_devices(
    "GPU"
)

print(gpus)

If a GPU is available, TensorFlow can often place supported operations on it automatically.

🔍 Inspecting Devices¶

tf.config.list_physical_devices()

Possible output:

[
    PhysicalDevice(
        name="/physical_device:CPU:0",
        device_type="CPU"
    ),
    PhysicalDevice(
        name="/physical_device:GPU:0",
        device_type="GPU"
    )
]

🧠 Explicit Device Placement¶

TensorFlow allows explicit device contexts.

with tf.device("/GPU:0"):

    x = tf.random.normal(
        (1000, 1000)
    )

    y = tf.matmul(
        x,
        x
    )

In most applications, explicit placement is not necessary because TensorFlow's runtime can handle device placement.

⚡ GPU Training Pipeline¶

flowchart LR

    DATA["Dataset"]
    CPU["CPU Input Pipeline"]
    GPU["GPU"]
    MODEL["Neural Network"]
    GRAD["Gradients"]
    UPDATE["Parameter Update"]

    DATA --> CPU
    CPU --> GPU
    GPU --> MODEL
    MODEL --> GRAD
    GRAD --> UPDATE
    UPDATE --> MODEL

The CPU may prepare and feed data while the GPU performs tensor-heavy computation.

🧠 GPU Memory¶

GPU memory is a critical resource.

Memory is consumed by:

Model Parameters
+
Gradients
+
Optimizer State
+
Activations
+
Input Batches

Therefore, increasing batch size may increase GPU memory consumption significantly.

🧠 Training vs Inference¶

Training requires:

Forward Pass
+
Loss
+
Gradients
+
Optimizer State

Inference usually requires:

Forward Pass

Therefore, training generally consumes considerably more memory.

flowchart LR

    TRAIN["Training"]

    TRAIN --> FORWARD["Forward"]
    FORWARD --> LOSS["Loss"]
    LOSS --> BACK["Backward"]
    BACK --> UPDATE["Update"]

    INFER["Inference"]

    INFER --> PRED["Forward"]

🧠 Trainable Parameters¶

Keras exposes model parameters through:

model.trainable_variables

You can inspect them:

for variable in model.trainable_variables:

    print(
        variable.name,
        variable.shape
    )

🔒 Freezing Layers¶

Layers can be frozen by setting:

layer.trainable = False

This is particularly important for Transfer Learning.

Example:

for layer in base_model.layers:

    layer.trainable = False

Transfer Learning is covered in:

21. Transfer Learning and Fine-Tuning

🧠 Keras Functional API Preview¶

The Functional API allows non-linear model structures.

For example:

inputs = tf.keras.Input(
    shape=(784,)
)

x = tf.keras.layers.Dense(
    128,
    activation="relu"
)(inputs)

x = tf.keras.layers.Dense(
    64,
    activation="relu"
)(x)

outputs = tf.keras.layers.Dense(
    10,
    activation="softmax"
)(x)

model = tf.keras.Model(
    inputs=inputs,
    outputs=outputs
)

The Functional API is covered in detail in:

14. Keras Sequential and Functional API

🧠 Custom Training Loops Preview¶

Keras also allows custom training loops.

Conceptually:

flowchart TD

    BATCH["Batch"]

    BATCH --> FORWARD["Forward Pass"]
    FORWARD --> LOSS["Loss"]
    LOSS --> GRAD["GradientTape"]
    GRAD --> OPT["Optimizer"]
    OPT --> UPDATE["Update Variables"]

    UPDATE --> BATCH

A simplified TensorFlow example:

with tf.GradientTape() as tape:

    predictions = model(
        X_batch,
        training=True
    )

    loss = loss_fn(
        y_batch,
        predictions
    )

gradients = tape.gradient(
    loss,
    model.trainable_variables
)

optimizer.apply_gradients(
    zip(
        gradients,
        model.trainable_variables
    )
)

Custom models and training loops are covered in:

15. Custom Layers, Models and Training Loops

🧠 Keras Model Lifecycle¶

A practical Keras lifecycle is:

flowchart LR

    DATA["Data"]
    BUILD["Build"]
    COMPILE["Compile"]
    TRAIN["Train"]
    VALIDATE["Validate"]
    SAVE["Save"]
    DEPLOY["Deploy"]
    MONITOR["Monitor"]

    DATA --> BUILD
    BUILD --> COMPILE
    COMPILE --> TRAIN
    TRAIN --> VALIDATE
    VALIDATE --> SAVE
    SAVE --> DEPLOY
    DEPLOY --> MONITOR

🧪 Complete Example — Classification¶

import tensorflow as tf


# Model
model = tf.keras.Sequential([

    tf.keras.Input(
        shape=(784,)
    ),

    tf.keras.layers.Dense(
        128,
        activation="relu"
    ),

    tf.keras.layers.Dropout(
        0.2
    ),

    tf.keras.layers.Dense(
        64,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        10,
        activation="softmax"
    )
])


# Compile
model.compile(
    optimizer=tf.keras.optimizers.AdamW(
        learning_rate=3e-4,
        weight_decay=1e-4
    ),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)


# Callbacks
callbacks = [

    tf.keras.callbacks.EarlyStopping(
        monitor="val_loss",
        patience=5,
        restore_best_weights=True
    ),

    tf.keras.callbacks.ModelCheckpoint(
        "best_model.keras",
        monitor="val_loss",
        save_best_only=True
    )
]


# Train
history = model.fit(
    X_train,
    y_train,
    validation_data=(
        X_val,
        y_val
    ),
    epochs=50,
    batch_size=64,
    callbacks=callbacks
)


# Evaluate
model.evaluate(
    X_test,
    y_test
)

🧪 Complete Example — Regression¶

import tensorflow as tf


model = tf.keras.Sequential([

    tf.keras.Input(
        shape=(10,)
    ),

    tf.keras.layers.Dense(
        128,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        64,
        activation="relu"
    ),

    tf.keras.layers.Dense(
        1
    )
])


model.compile(
    optimizer=tf.keras.optimizers.AdamW(
        learning_rate=3e-4
    ),
    loss="mse",
    metrics=["mae"]
)


history = model.fit(
    X_train,
    y_train,
    validation_data=(
        X_val,
        y_val
    ),
    epochs=50,
    batch_size=64
)

🧪 Complete Example — `tf.data` + Keras¶

train_ds = (
    tf.data.Dataset
    .from_tensor_slices(
        (X_train, y_train)
    )
    .shuffle(10000)
    .batch(64)
    .prefetch(
        tf.data.AUTOTUNE
    )
)


val_ds = (
    tf.data.Dataset
    .from_tensor_slices(
        (X_val, y_val)
    )
    .batch(64)
    .prefetch(
        tf.data.AUTOTUNE
    )
)


model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=50
)

🧠 Production-Oriented TensorFlow Architecture¶

A production Deep Learning application should separate:

Data
 ↓
Preprocessing
 ↓
Dataset Pipeline
 ↓
Model
 ↓
Training
 ↓
Evaluation
 ↓
Artifact
 ↓
Serving

A possible architecture:

flowchart TD

    DATA["Raw Data"]

    PRE["Preprocessing"]

    PIPE["tf.data Pipeline"]

    MODEL["Keras Model"]

    TRAIN["Training"]

    EVAL["Evaluation"]

    ART["Model Artifact"]

    SERVE["Model Serving"]

    MON["Monitoring"]

    DATA --> PRE
    PRE --> PIPE
    PIPE --> MODEL
    MODEL --> TRAIN
    TRAIN --> EVAL
    EVAL --> ART
    ART --> SERVE
    SERVE --> MON

🏢 Enterprise Perspective¶

TensorFlow and Keras provide more than APIs for building neural networks.

They can serve as components of an enterprise Deep Learning platform:

Data Engineering
       ↓
TensorFlow Data Pipeline
       ↓
Keras Model
       ↓
Training Infrastructure
       ↓
GPU / TPU
       ↓
Model Artifact
       ↓
Model Registry
       ↓
Serving
       ↓
Monitoring

For enterprise systems, the important concerns include:

Reproducibility
Dataset versioning
Experiment tracking
Model versioning
Hardware utilization
Training cost
Model quality
Deployment
Monitoring
Governance

Production Insight

A Keras model is only one component of a production Deep Learning system.

A reliable system should connect:

Versioned Data
     ↓
Reproducible Training
     ↓
Versioned Model
     ↓
Validated Artifact
     ↓
Deployment
     ↓
Monitoring

Optimizing only the model while ignoring the data pipeline, hardware utilization, artifact management, and observability can create a fragile production system.

⚠ Common Mistakes¶

Avoid these common mistakes:

Confusing TensorFlow with Keras
Ignoring tensor shapes
Mixing incompatible tensor data types
Forgetting to normalize input data
Using incorrect output activation functions
Using the wrong loss function
Forgetting validation data
Training without monitoring validation metrics
Ignoring GPU memory
Using excessively large batch sizes
Loading an entire large dataset into memory unnecessarily
Failing to use efficient tf.data pipelines
Applying augmentation incorrectly to validation/test data
Saving only weights when the complete model artifact is required
Forgetting to save the best checkpoint
Not versioning the training configuration
Assuming GPU usage automatically guarantees efficient training
Ignoring CPU input-pipeline bottlenecks
Treating training and inference as identical workloads
Hardcoding hyperparameters throughout the codebase

🧠 Interview Questions¶

Beginner¶

1. What is TensorFlow?¶

TensorFlow is a framework for tensor-based numerical computation and Machine Learning, providing automatic differentiation, neural-network primitives, hardware acceleration, and other ML capabilities.

2. What is Keras?¶

Keras is a high-level Deep Learning API that provides abstractions for building, training, evaluating, and deploying neural networks.

3. What is a tensor?¶

A tensor is a multidimensional array with a defined shape and data type.

4. What is tensor rank?¶

Tensor rank represents the number of dimensions of a tensor.

5. What is the difference between shape and rank?¶

Rank tells you how many dimensions a tensor has, while shape tells you the size of each dimension.

Intermediate¶

6. What are the main steps in a Keras workflow?¶

Build
 ↓
Compile
 ↓
Fit
 ↓
Evaluate
 ↓
Predict

7. What does `model.compile()` do?¶

It configures the optimizer, loss function, and metrics used by the training process.

8. What does `model.fit()` do?¶

It executes the training process over the supplied dataset for the specified number of epochs.

9. What is `tf.data`?¶

tf.data is TensorFlow's API for constructing input pipelines.

10. Why is `prefetch()` useful?¶

It allows future batches to be prepared while the current batch is being processed, potentially improving hardware utilization.

11. What is a Keras callback?¶

A callback is an object that can execute actions at specific points during training.

12. Why use `ModelCheckpoint`?¶

It allows the best or selected model state to be saved during training.

Advanced¶

13. Why is tensor shape important?¶

Neural-network layers expect inputs with compatible dimensions. Incorrect shapes can cause runtime errors or produce incorrect model behavior.

14. Why can GPU training still be slow?¶

Potential bottlenecks include:

Input Pipeline
CPU Preprocessing
GPU Memory
Small Batches
Data Transfer
Model Architecture
I/O

15. Why is `tf.data` important for production?¶

It can provide efficient batching, shuffling, caching, prefetching, and transformation pipelines that help keep training hardware utilized.

16. What is the difference between training and inference?¶

Training requires forward propagation, loss computation, backpropagation, and parameter updates. Inference generally requires only the forward pass.

17. Why is optimizer state important?¶

Optimizers such as Adam and AdamW maintain additional tensors, increasing memory requirements during training.

18. When should you use the Functional API instead of Sequential?¶

Use the Functional API when the architecture contains branching, multiple inputs/outputs, skip connections, shared layers, or other non-linear graph structures.

19. Why would you use a custom training loop?¶

Custom loops provide control over training behavior when the standard fit() workflow is insufficient for a particular research or production requirement.

20. How would you optimize a TensorFlow training pipeline?¶

Inspect:

Input Pipeline
Batch Size
Prefetching
Caching
GPU Utilization
Mixed Precision
Model Complexity
Data Transfer

Then measure the actual bottleneck before optimizing.

🧪 Practical Exercises¶

Exercise 1 — Tensor Fundamentals¶

Create tensors of:

Rank 0
Rank 1
Rank 2
Rank 3

For each tensor, print:

Value
Shape
Rank
Data Type

Exercise 2 — Tensor Operations¶

Implement:

Addition
Subtraction
Multiplication
Matrix Multiplication
Reshape
Transpose
Reduction
Broadcasting

Verify the resulting shapes.

Exercise 3 — Build a Classification Model¶

Create a Keras model with:

Input
 ↓
Dense
 ↓
ReLU
 ↓
Dense
 ↓
ReLU
 ↓
Dense
 ↓
Softmax

Train it on a classification dataset.

Track:

Training Loss
Validation Loss
Training Accuracy
Validation Accuracy

Exercise 4 — Build a Regression Model¶

Create a regression network using:

Input
 ↓
Dense
 ↓
ReLU
 ↓
Dense
 ↓
ReLU
 ↓
Linear Output

Evaluate using:

MSE
MAE

Exercise 5 — Build a `tf.data` Pipeline¶

Create a pipeline using:

from_tensor_slices()
shuffle()
batch()
prefetch()

Measure whether the training throughput changes when prefetching is enabled.

Exercise 6 — GPU Training¶

Check:

tf.config.list_physical_devices(
    "GPU"
)

Train the same model using:

CPU
GPU

Compare:

Training Time
Throughput
Memory Usage

Exercise 7 — Callbacks¶

Train a model using:

EarlyStopping
ModelCheckpoint
ReduceLROnPlateau
TensorBoard

Inspect the resulting training behavior.

📌 Key Takeaways¶

TensorFlow provides tensor-based computation and Deep Learning infrastructure.
Keras provides a high-level API for building and training neural networks.
Tensors are the fundamental data structure used by Deep Learning systems.
Tensor rank represents the number of dimensions.
Tensor shape describes the size of each dimension.
Data type determines how tensor values are represented.
Matrix multiplication is fundamental to neural-network computation.
Keras models are composed of layers.
The basic Keras workflow is Build → Compile → Fit → Evaluate → Predict.
tf.data provides efficient input-pipeline capabilities.
Shuffling is generally useful for training data.
Batching controls the number of examples processed per optimizer update.
Prefetching can improve hardware utilization.
Caching can reduce repeated data-loading work when used appropriately.
Callbacks provide control over training behavior.
Checkpointing preserves useful model states.
Early Stopping can prevent unnecessary training.
TensorFlow can use CPUs, GPUs, and other accelerators.
GPU memory is consumed by parameters, gradients, optimizer state, activations, and input batches.
Training generally requires more memory than inference.
Sequential models are useful for simple linear stacks.
More complex architectures should use the Functional API or custom model implementations.
Efficient Deep Learning requires optimizing the complete pipeline, not only the neural network.

📚 Further Reading¶

Continue with:

The next chapter goes deeper into the two major Keras model-building approaches: the Sequential API and Functional API.

➡️ Next Chapter¶

14. Keras Sequential and Functional API

Enterprise AI Engineering Handbook
Building Production-Grade Enterprise AI Systems — One Chapter at a Time.

13. TensorFlow and Keras Fundamentals¶

🎯 Learning Objectives¶

📖 Overview¶

🧠 What Is TensorFlow?¶

📐 Tensor Ranks¶

Rank 0 — Scalar¶

Rank 1 — Vector¶

Rank 2 — Matrix¶

Rank 3¶

🧮 Tensor Terminology¶

🐍 Creating TensorFlow Tensors¶

🔍 Inspecting a Tensor¶

🧠 Tensor Shape¶

🖼️ Image Tensor Representation¶

🧠 Tensor Data Types¶

🔄 Type Conversion¶

🧮 Tensor Operations¶

✖️ Matrix Multiplication¶

📐 Tensor Reshaping¶

🧩 Flattening¶

🔄 Broadcasting¶

🧠 Why Tensors Matter in Deep Learning¶

🧠 What Is Keras?¶

🏗 Keras Model Architecture¶

🧱 Keras Layers¶

🧠 Dense Layer¶

🏗 Sequential Model¶

🧠 Input Shape¶

🧠 Keras Model Summary¶

🧮 Trainable Parameters¶

🧠 Build → Compile → Fit → Evaluate → Predict¶

⚙️ Compile¶

🏋️ Fit¶

📊 What Happens During fit()?¶

📉 Training History¶

📈 Plotting Training Curves¶

🧪 Evaluate¶

🔮 Predict¶

🧠 Classification Example¶

🧠 Regression Example¶

🧠 Classification vs Regression¶

📦 TensorFlow tf.data¶

🔄 tf.data Pipeline¶

🔀 Shuffle¶

📦 Batch¶

⚡ Prefetch¶

💾 Cache¶

🧠 Complete tf.data Pipeline¶

🧠 Data Pipeline Performance¶

🔌 Keras Callbacks¶

⏹️ EarlyStopping¶

💾 ModelCheckpoint¶

📉 ReduceLROnPlateau¶

📊 TensorBoard¶

🧠 Complete Callback Configuration¶

💾 Saving a Keras Model¶

💾 Saving Weights Only¶

🧠 Model Serialization Strategy¶

🖥️ CPU vs GPU¶

🚀 Checking GPU Availability¶

🔍 Inspecting Devices¶

🧠 Explicit Device Placement¶

⚡ GPU Training Pipeline¶

🧠 GPU Memory¶

🧠 Training vs Inference¶

🧠 Trainable Parameters¶

🔒 Freezing Layers¶

🧠 Keras Functional API Preview¶

🧠 Custom Training Loops Preview¶

🧠 Keras Model Lifecycle¶

🧪 Complete Example — Classification¶

🧪 Complete Example — Regression¶

🧪 Complete Example — tf.data + Keras¶

🧠 Production-Oriented TensorFlow Architecture¶

🏢 Enterprise Perspective¶

⚠ Common Mistakes¶

🧠 Interview Questions¶

Beginner¶

1. What is TensorFlow?¶

2. What is Keras?¶

📊 What Happens During `fit()`?¶

📦 TensorFlow `tf.data`¶

🔄 `tf.data` Pipeline¶

🧠 Complete `tf.data` Pipeline¶

🧪 Complete Example — `tf.data` + Keras¶

7. What does `model.compile()` do?¶

8. What does `model.fit()` do?¶

9. What is `tf.data`?¶

10. Why is `prefetch()` useful?¶

12. Why use `ModelCheckpoint`?¶

15. Why is `tf.data` important for production?¶

Exercise 5 — Build a `tf.data` Pipeline¶