4  Bases of Deep Learning

5 Tensors in pytorch

(Adapted from PyTorch Tutorials)

Tensors are a specialized data structure that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters.

5.1 Warm-up: numpy

Tensors are similar to NumPy’s ndarrays, except that tensors can run on GPUs or other hardware accelerators. In fact, tensors and NumPy arrays can often share the same underlying memory, eliminating the need to copy data (see Section 5.18).

Tensors are also optimized for automatic differentiation (we’ll see more on that Chapter 6).

\(\Leftarrow\) If you’re familiar with ndarrays, you’ll be right at home with the Tensor API.

5.2 Initializing a Tensor

import torch
import numpy as np

Tensors can be initialized in various ways.

  • from data
  • from NumPy arrays
  • from other tensors
  • with random or constant values

5.3 Initializing a Tensor: from data

Tensors can be created directly from data. The data type is automatically inferred.

data = [[1, 2],[3, 4]]
x_data = torch.tensor(data)

5.4 Initializing a Tensor: from NumPy

Tensors can be created from NumPy arrays (and vice versa - see Section 5.18).

np_array = np.array(data)
x_np = torch.from_numpy(np_array)

5.5 Initializing a Tensor: from another tensor

The new tensor retains the properties (shape, datatype) of the argument tensor, unless explicitly overridden.

# retains the properties of x_data
x_ones = torch.ones_like(x_data) 
print(f"Ones Tensor: \n {x_ones} \n")

# overrides the datatype of x_data
x_rand = torch.rand_like(x_data, dtype=torch.float)
print(f"Random Tensor: \n {x_rand} \n")
Ones Tensor: 
 tensor([[1, 1],
        [1, 1]]) 

Random Tensor: 
 tensor([[0.0029, 0.9247],
        [0.3090, 0.3638]]) 

5.6 Initializing a Tensor: with random or constant values

shape is a tuple of tensor dimensions. In the functions below, it determines the dimensionality of the output tensor.

shape = (2,3,)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)
zeros_tensor = torch.zeros(shape)

print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Ones Tensor: \n {ones_tensor} \n")
print(f"Zeros Tensor: \n {zeros_tensor}")
Random Tensor: 
 tensor([[0.7485, 0.0997, 0.0122],
        [0.0076, 0.9352, 0.2850]]) 

Ones Tensor: 
 tensor([[1., 1., 1.],
        [1., 1., 1.]]) 

Zeros Tensor: 
 tensor([[0., 0., 0.],
        [0., 0., 0.]])

5.7 Attributes of a Tensor

Tensor attributes describe their shape, datatype, and the device on which they are stored.

tensor = torch.rand(3,4)

print(f"Shape of tensor: {tensor.shape}")
print(f"Datatype of tensor: {tensor.dtype}")
print(f"Device tensor is stored on: {tensor.device}")
Shape of tensor: torch.Size([3, 4])
Datatype of tensor: torch.float32
Device tensor is stored on: cpu

5.8 Operations on Tensors

Over 100 tensor operations, including arithmetic, linear algebra, matrix manipulation (transposing, indexing, slicing), sampling and more are comprehensively described here.

Each of these operations can be run on the GPU (at typically higher speeds than on a CPU).

5.9 Operations on Tensors: move to GPU

By default, tensors are created on the CPU. We need to explicitly move tensors to the GPU using .to method (after checking for GPU availability). Keep in mind that copying large tensors across devices can be expensive in terms of time and memory!

5.10 To check for GPU availability

# We move our tensor to the GPU if available
if torch.cuda.is_available():
    tensor = tensor.to("cuda")
    devname = torch.cuda.get_device_name()
    print(f"Device tensor is stored on: {devname}")
Device tensor is stored on: NVIDIA A10

Try out some of the operations from the list.

5.11 Standard numpy-like indexing and slicing

tensor = torch.ones(4, 4)
print(f"First row: {tensor[0]}")
print(f"First column: {tensor[:, 0]}")
print(f"Last column: {tensor[..., -1]}")
tensor[:,1] = 0
print(tensor)
First row: tensor([1., 1., 1., 1.])
First column: tensor([1., 1., 1., 1.])
Last column: tensor([1., 1., 1., 1.])
tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]])

5.12 Joining tensors, cat

You can use torch.cat to concatenate a sequence of tensors along a given dimension.

t1 = torch.cat([tensor, tensor, tensor], dim=1)
print(t1)
tensor([[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
        [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
        [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
        [1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.]])

5.13 Joining tensors, stack

See also torch.stack, another tensor joining operator that is subtly different from torch.cat.

t2 = torch.stack([tensor, tensor, tensor], dim=2)
print(t2)
tensor([[[1., 1., 1.],
         [0., 0., 0.],
         [1., 1., 1.],
         [1., 1., 1.]],

        [[1., 1., 1.],
         [0., 0., 0.],
         [1., 1., 1.],
         [1., 1., 1.]],

        [[1., 1., 1.],
         [0., 0., 0.],
         [1., 1., 1.],
         [1., 1., 1.]],

        [[1., 1., 1.],
         [0., 0., 0.],
         [1., 1., 1.],
         [1., 1., 1.]]])

5.14 Arithmetic operations

This computes the matrix multiplication between two tensors. y1, y2, y3 will have the same value

tensor.T returns the transpose of a tensor

y1 = tensor @ tensor.T
y2 = tensor.matmul(tensor.T)

y3 = torch.rand_like(y1)
torch.matmul(tensor, tensor.T, out=y3)
tensor([[3., 3., 3., 3.],
        [3., 3., 3., 3.],
        [3., 3., 3., 3.],
        [3., 3., 3., 3.]])

This computes the element-wise product. z1, z2, z3 will have the same value

z1 = tensor * tensor
z2 = tensor.mul(tensor)

z3 = torch.rand_like(tensor)
torch.mul(tensor, tensor, out=z3)
tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]])

5.15 Single-element tensors

If you have a one-element tensor, for example by aggregating all values of a tensor into one value, you can convert it to a Python numerical value using item():

agg = tensor.sum()
agg_item = agg.item()
print(agg_item, type(agg_item))
12.0 <class 'float'>

5.16 In-place operations

Operations that store the result into the operand are called in-place. They are denoted by a _ suffix. For example: x.copy_(y), x.t_(), will change x.

print(f"{tensor} \n")
tensor.add_(5)
print(tensor)
tensor([[1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.],
        [1., 0., 1., 1.]]) 

tensor([[6., 5., 6., 6.],
        [6., 5., 6., 6.],
        [6., 5., 6., 6.],
        [6., 5., 6., 6.]])

5.17 In-place operations: caution

Note

In-place operations save some memory, but can be problematic when computing derivatives because of an immediate loss of history. Hence, their use is discouraged.

5.18 Bridge with NumPy

Tensors on the CPU and NumPy arrays can share their underlying memory locations, and changing one will change the other.

5.19 Bridge with NumPy: Tensor to NumPy array

A change in the tensor reflects in the NumPy array.

t = torch.ones(5)
print(f"t: {t}")
n = t.numpy()
print(f"n: {n}")
t: tensor([1., 1., 1., 1., 1.])
n: [1. 1. 1. 1. 1.]
t.add_(1)
print(f"t: {t}")
print(f"n: {n}")
t: tensor([2., 2., 2., 2., 2.])
n: [2. 2. 2. 2. 2.]

5.20 Bridge with NumPy: NumPy array to Tensor

Changes in the NumPy array reflects in the tensor.

n = np.ones(5)
t = torch.from_numpy(n)
np.add(n, 1, out=n)
print(f"t: {t}")
print(f"n: {n}")
t: tensor([2., 2., 2., 2., 2.], dtype=torch.float64)
n: [2. 2. 2. 2. 2.]

6 Automatic Differentiation, Theory and practice

6.1 Definition

Wikipedia:

Automatic differentiation (AD), also called algorithmic differentiation or computational differentiation, is a set of techniques to numerically evaluate the derivative of a function specified by a computer program… derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.

6.2 Numeric differentiation

\[\frac{df(x)}{dx} = \lim_{\epsilon \to 0} \frac{f(x + \epsilon) - f(\epsilon)}{\epsilon}\]

  • (almost) trivial implementation
  • \(O(n)\) evaluations of \(f\) for gradients of \(n\) dimensions
  • Truncation error1 + rounding error2.

6.3 Symbolic differentiation

\[\begin{align} \frac{d(u(x) + v(x))}{dx} & = & \frac{du(x)}{dx} + \frac{dv(x)}{dx},\\ \frac{d(u(x) v(x))}{dx} & = & u(x) \frac{dv(x)}{dx} + \frac{du(x)}{dx} v(x)\\ & \textrm{etc.} & \end{align} \]

6.4 Benefits of symbolic differentiation

  • Implemented in all symbolic calculation software (Mathematica, Maxima, Sage etc.)
  • Extremely efficient for moderately complex functions

6.5 Drawbacks of symbolic differentiation

  1. What to do with “constructed functions” such as constructors, structures, etc.

\[\frac{d(\textrm{make-point}(x,y))}{dx}\]

  1. If \(f\) is a product, then \(df\) and \(f\) contain common calculations

\[\frac{d(u(x) v(x))}{dx} = u(x) \frac{dv(x)}{dx} + \frac{du(x)}{dx} v(x)\]

likewise \(u\) and \(du\) etc. ⇒ exponential growth of symbolic code

6.6 Forward mode

Automatic differentiation in foward mode = “calculated” symbolic differentiation that proceeds in parallel with the main calculation and the “derived” calculation, while sharing the intermediate results between the two at each stage.

6.7 And what about deep learning?

(Adapted from PyTorch Tutorials)

When training neural networks, the most frequently used algorithm is back propagation. In this algorithm, parameters (model weights) are adjusted according to the gradient of the loss function with respect to the given parameter.

\(\Leftarrow\) To compute those gradients, PyTorch has a built-in automatic differentiation engine called torch.autograd. It supports automatic computation of gradient for any computational graph.

6.8 A Simple Example

Consider the simplest one-layer neural network, with input x, parameters w and b, and some loss function. It can be defined in PyTorch in the following manner:

import torch

x = torch.ones(5)  # input tensor
y = torch.zeros(3)  # expected output
w = torch.randn(5, 3, requires_grad=True)
b = torch.randn(3, requires_grad=True)
z = torch.matmul(x, w)+b
loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)

6.9 Tensors, Functions and Computational graph

This code defines the following computational graph:

Computational graph example

In this network, w and b are parameters, which we need to optimize. Thus, we need to be able to compute the gradients of loss function with respect to those variables. In order to do that, we set the requires_grad property of those tensors.

Note

You can set the value of requires_grad when creating a tensor, or later by using x.requires_grad_(True) method.

A function that we apply to tensors to construct computational graph is in fact an object of class Function. This object knows how to compute the function in the forward direction, and also how to compute its derivative during the backward propagation step.

A reference to the backward propagation function is stored in grad_fn property of a tensor. You can find more information of Function in the documentation.

print(f"Gradient function for z = {z.grad_fn}")
print(f"Gradient function for loss = {loss.grad_fn}")
Gradient function for z = <AddBackward0 object at 0x7fd04b3eba90>
Gradient function for loss = <BinaryCrossEntropyWithLogitsBackward0 object at 0x7fd04b3ea410>

6.10 Computing Gradients

To optimize weights of parameters in the neural network, we need to compute the derivatives of our loss function with respect to parameters, namely, we need \(\frac{\partial loss}{\partial w}\) and \(\frac{\partial loss}{\partial b}\) under some fixed values of x and y.

To compute those derivatives, we call loss.backward(), and then retrieve the values from w.grad and b.grad:

loss.backward()
print(w.grad)
print(b.grad)
tensor([[0.1977, 0.0475, 0.2733],
        [0.1977, 0.0475, 0.2733],
        [0.1977, 0.0475, 0.2733],
        [0.1977, 0.0475, 0.2733],
        [0.1977, 0.0475, 0.2733]])
tensor([0.1977, 0.0475, 0.2733])
Note
  • We can only obtain the grad properties for the leaf nodes of the computational graph, which have requires_grad property set to True. For all other nodes in our graph, gradients will not be available.
  • We can only perform gradient calculations using backward once on a given graph, for performance reasons. If we need to do several backward calls on the same graph, we need to pass retain_graph=True to the backward call.

6.11 Disabling Gradient Tracking

By default, all tensors with requires_grad=True are tracking their computational history and support gradient computation.

However, there are some cases when we do not need to do that, for example, when we have trained the model and just want to apply it to some input data, i.e. we only want to do forward computations through the network.

We can stop tracking computations by surrounding our computation code with torch.no_grad() block:

z = torch.matmul(x, w)+b
print(z.requires_grad)

with torch.no_grad():
    z = torch.matmul(x, w)+b
print(z.requires_grad)
True
False

Another way to achieve the same result is to use the detach() method on the tensor:

z = torch.matmul(x, w)+b
z_det = z.detach()
print(z_det.requires_grad)
False

6.12 Why would we want to disable gradient tracking?

There are reasons you might want to disable gradient tracking:

  • To mark some parameters in your neural network as frozen parameters.
  • To speed up computations when you are only doing forward pass, because computations on tensors that do not track gradients would be more efficient.
  • You want to manually control the updates to the weights in the network.

6.13 More on Computational Graphs : DAGs

Conceptually, autograd keeps a record of data (tensors) and all executed operations (along with the resulting new tensors) in a directed acyclic graph (DAG) consisting of Function objects.

In this DAG, leaves are the input tensors, roots are the output tensors. By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule.

In a forward pass, autograd does two things simultaneously:

  • run the requested operation to compute a resulting tensor
  • maintain the operation’s gradient function in the DAG.

The backward pass kicks off when .backward() is called on the DAG root. autograd then:

  • computes the gradients from each .grad_fn,
  • accumulates them in the respective tensor’s .grad attribute
  • using the chain rule, propagates all the way to the leaf tensors.
DAGs are dynamic in PyTorch

An important thing to note is that the graph is recreated from scratch; after each .backward() call, autograd starts populating a new graph. This is exactly what allows you to use control flow statements in your model; you can change the shape, size and operations at every iteration if needed.

6.14 Tensor Gradients and Jacobian Products

In many cases, we have a scalar loss function, and we need to compute the gradient with respect to some parameters. However, there are cases when the output function is an arbitrary tensor. In this case, PyTorch allows you to compute so-called Jacobian product, and not the actual gradient.

For a vector function \(\vec{y}=f(\vec{x})\), where \(\vec{x}=\langle x_1,\dots,x_n\rangle\) and \(\vec{y}=\langle y_1,\dots,y_m\rangle\), a gradient of \(\vec{y}\) with respect to \(\vec{x}\) is given by Jacobian matrix:

\[ \begin{align}J=\left(\begin{array}{ccc} \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{1}}{\partial x_{n}}\\ \vdots & \ddots & \vdots\\ \frac{\partial y_{m}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}} \end{array}\right)\end{align} \]

Instead of computing the Jacobian matrix itself, PyTorch allows you to compute Jacobian Product \(v^T\cdot J\) for a given input vector \(v=(v_1 \dots v_m)\). This is achieved by calling backward with \(v\) as an argument. The size of \(v\) should be the same as the size of the original tensor, with respect to which we want to compute the product.

6.15 Illustration of Jacobian Product

inp = torch.eye(4, 5, 
    requires_grad=True)
out = (inp+1).pow(2).t()
out.backward(torch.ones_like(out), 
    retain_graph=True)
inp.grad
tensor([[4., 2., 2., 2., 2.],
        [2., 4., 2., 2., 2.],
        [2., 2., 4., 2., 2.],
        [2., 2., 2., 4., 2.]])
out.backward(torch.ones_like(out), 
    retain_graph=True)
inp.grad
tensor([[8., 4., 4., 4., 4.],
        [4., 8., 4., 4., 4.],
        [4., 4., 8., 4., 4.],
        [4., 4., 4., 8., 4.]])
inp.grad.zero_()
out.backward(torch.ones_like(out), 
    retain_graph=True)
inp.grad
tensor([[4., 2., 2., 2., 2.],
        [2., 4., 2., 2., 2.],
        [2., 2., 4., 2., 2.],
        [2., 2., 2., 4., 2.]])

Notice that when we call backward for the second time with the same argument, the value of the gradient is different.

This happens because when doing backward propagation, PyTorch accumulates the gradients, i.e. the value of computed gradients is added to the grad property of all leaf nodes of computational graph.

If you want to compute the proper gradients, you need to zero out the grad property before. In real-life training an optimizer helps us to do this.

6.16 Computing the Jacobian with respect to some input

Note

Previously we were calling backward() function without parameters. This is essentially equivalent to calling backward(torch.tensor(1.0)), which is a useful way to compute the gradients in case of a scalar-valued function, such as loss during neural network training.

6.17 Accumulating the gradient

As seen in the previous exemple, each call to the backward() function adds the computed gradient to the grad property of the tensor. If you want to compute the proper gradients, you need to zero out the grad property before. In real-life training an optimizer helps us to do this.

7 Gradient Descent

7.1 Algorithm

In this section, we assume \(f\) to be a smooth function and our aim is to introduce the gradient descent method to find its (or one of its) minima.

7.2 Fixed step gradient descent

\begin{algorithm} \caption{Fixed step gradient descent} \begin{algorithmic} \Input $\nabla f, x_{init}, \gamma,$ maxiter, $\epsilon$ \State $x = x_{init}$ \Comment{Initialization} \For{$i=1, \dots, $maxiter} \State $g\leftarrow \nabla f(x)$ \Comment{Gradient evaluation in the current iteration} \If{$\|g\|^2 \leq \epsilon^2$ } \Comment{Characterization of an extremum} \State Break \Else \State $x \leftarrow x - \gamma g$ \Comment{Step in the opposite direction to the gradient} \EndIf \EndFor \Output $x$ \end{algorithmic} \end{algorithm}

7.3 Example setting

We’re interested in minimizing the following function:

\(f(x_1, x_2) = (x_1^2 + x_2 - 11)^2 + (x_1 + x_2^2 - 7)^2\)

def f(x):
    x1, x2 = x[0],x[1]
    return (x1**2 + x2 - 11)**2 + (x1 + x2**2 - 7)**2

7.4 Surface plot

import numpy as np
from python_utils.interactive_plot import init_plot, plot_surface, plot_grad_descent, plot_contour, plot_surface

X1, X2 = np.meshgrid(np.linspace(-5.5, 5.5, 50),
                     np.linspace(-5.5, 5.5, 50))
Z = f(np.array([X1, X2]))

plot_surface(X1,X2,Z).show()

7.5 Contour plot

plot_contour(X1,X2,Z).show()

7.6 Gradient Definition

\[ \nabla f(x_1, x_2) = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \end{bmatrix} = \begin{bmatrix} 2 \left(-7 + x_1 + x_2^2 + 2x_1(-11 + x_1^2 + x_2)\right) \\ 2 \left(-11 + x_1^2 + x_2 + 2x_2(-7 + x_1 + x_2^2)\right) \end{bmatrix} \]

def f_grad(x):
    x1, x2 = x
    df_x1 = 2 * (-7 + x1 + x2**2 + 2 * x1 * (-11 + x1**2 + x2))
    df_x2 = 2 * (-11 + x1**2 + x2 + 2 * x2 * (-7 + x1 + x2**2))
    return np.array([df_x1, df_x2])

7.7 Gradient descent code

Complete the following function for gradient descent with constant step size, it should return the list of iterates.

def grad_descent(f,x0,step_size,max_iter=0):
    """Gradient descent with constant step size"""
    x = x0 # starting point
    xs = [x]
    for k in range(max_iter):
        # to complete
    return xs

7.8 Plotting the iterates

from python_utils.interactive_plot import init_plot, plot_grad_descent

x0 = np.array([0.5, -4])
xs = grad_descent(f,x0,0.01,max_iter=20)
init_plot(X1,X2,Z).add_scatter(x=xs[0],y=xs[1],mode='markers+lines').show()

7.9 Interactive plot

from python_utils import interactive_plot

plot_grad_descent(grad_descent,f,X1,X2,Z)

7.10 And now with pytorch

We will use the automatic differentiation feature of pytorch to compute the gradient of \(f\), and convert the gradient descent algorithm to pytorch.

Tips
  • To store the iterates as numpy arrays, we will use the detach method of pytorch tensors (get the values out of the computational graph).
  • We also have to take care of the synchronization between numpy and pytorch, we will use the copy() method of numpy arrays.
  • Moreover as we do everything inplace in our main loop, we have to not need to compute the gradient of the iterates, we will use the no_grad context manager to disable gradient computation. Alternatively, we could use the the <op>_ internal in place operation.

7.11 Gradient descent code with pytorch

import torch

def torch_grad_descent(f,x0,step_size,max_iter=0):
    """Gradient descent with constant step size"""
    x = torch.tensor(x0, requires_grad=True)
    xs = ...
    for k in range(max_iter):
        # to complete
    return xs

7.12 Plotting the iterates

x0 = np.array([0.5, -4])
xs = torch_grad_descent(f,x0,0.01,max_iter=20)
init_plot(X1,X2,Z).add_scatter(x=xs[0],y=xs[1],
    mode='markers+lines').show()

7.13 Interactive plot

plot_grad_descent(torch_grad_descent,f,X1,X2,Z)

7.14 With another function

\(f(x_1, x_2) = x_1^2 + sin(2x_2) + (cos(2x_1) + x_2^2 - 7)^2 + 2\)

7.15 Surface plot

Code
X1, X2 = np.meshgrid(np.linspace(-5.5, 5.5, 50),
                     np.linspace(-5.5, 5.5, 50))
Z = f(torch.tensor([X1, X2])).detach().numpy()

plot_surface(X1,X2,Z)
/tmp/ipykernel_1739982/1510781993.py:3: UserWarning:

Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1680572619157/work/torch/csrc/utils/tensor_new.cpp:245.)

7.16 Contour plot

7.17 Interactive plot

plot_grad_descent(torch_grad_descent,f,X1,X2,Z)

8 Optimizers

(Adapted from this guide)

8.1 Introduction

In this tutorial, we will go through PyTorch optimizers which are used to reduce the error rate while training the neural networks. We will first understand what is optimization and then we will see different types of optimizers in PyTorch with their syntax and examples of usage.

8.2 What is Optimization?

During neural network training, its weights are randomly initialized initially and then they are updated in each epoch in a manner such that they increase the overall accuracy of the network.

8.3 Optimization Process

In each epoch, the output of the training data is compared to actual data with the help of the loss function to calculate the error and then the weight is updated accordingly But how do we know how to update the weight such that it increases the accuracy?

This is actually a problem of optimization where the goal is to optimize the loss function and get the ideal weights. And the method used for optimization is called Optimizer.

8.4 Optimization Process, Illustration

Illustration of the optimization process

8.5 PyTorch Optimizers Examples

Gradient Descent is the most commonly known optimizer but for practical purposes, there are many other optimizers. You will find many of these Optimizers in PyTorch library as well.

  • SGD
  • Adam
  • Adadelta
  • Adagrad
  • AdamW
  • Adamax
  • RMSProp

8.6 Setting up the environment

import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
import plotly.express as px
import pandas as pd

8.7 Generating Data

We’ll use a simple linear function \(y = Ax + b\) to generate some data. We’ll also add some noise to the data to make it more realistic.

# Let's make some data for a linear regression.
A = 5.1213945
b = 3.2047893
error = 2.0
N = 100 # number of data points

# Data
X = torch.randn(N, 1)

# (noisy) Target values that we want to learn.
t = A * X + b + torch.randn(N, 1) * error

8.8 Plotting the data

8.9 Linear Regression

from sklearn.linear_model import LinearRegression
reg = ... # fit the regressin
pred = ... # predictions
regloss = ... # MSE loss
print(f"R2 Score : {reg.score(X, t)}, MSELoss : {regloss}")
px.scatter(x=X[:,0],y=t[:,0]) \
    .add_traces(px.line(x=X[:,0],y=pred[:,0]) \
    .update_traces(line_color='red').data)

8.10 PyTorch Gradient Descent Optimizer

Define our moder and loss function

# Creating a model, defining loss
model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()

8.11 Defining the optimizer loop

def runOptimizer(model,opt,niter):
    model.reset_parameters()
    df = pd.DataFrame({},columns=['index','Data','Value'])
    for i in range(0, 30):
1        opt.zero_grad()
2        predictions = model(X)
        loss = loss_fn(predictions, t)
        df.loc[len(df)] = [i,'learned A',list(model.parameters())[0].data[0,0].item()]
        df.loc[len(df)] = [i,'learned b',list(model.parameters())[1].data[0].item()]
        df.loc[len(df)] = [i,'loss',loss.data.item()]
3        loss.backward()
4        opt.step()
    return df
1
Zero the gradients
2
Forward pass
3
Backward pass
4
Update parameters

8.12 SGD

The SGD or Stochastic Gradient Optimizer is an optimizer in which the weights are updated for each training sample or a small subset of data.

8.13 SGD, Syntax

torch.optim.SGD(params, lr=<required parameter>, momentum=0, dampening=0, weight_decay=0, nesterov=False)

Parameters:

  • params (iterable) — These are the parameters that help in the optimization.
  • lr (float) — This parameter is the learning rate
  • momentum (float, optional) — Here we pass the momentum factor
  • weight_decay (float, optional) — This argument is containing the weight decay
  • dampening (float, optional) — To dampen the momentum, we use this parameter

8.14 SGD, Example

8.15 Adam Optimizer

Adam Optimizer uses both momentum and adaptive learning rate for better convergence. This is one of the most widely used optimizer for practical purposes for training neural networks.

8.16 Adam, Syntax

torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)

Parameters

  • params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
  • lr (float) – Learning rate helping optimization (default: 1e-3)
  • betas (Tuple[float, float]) – This parameter is used for calculating and running the averages for gradient (default: (0.9, 0.999))
  • beta3 (float) – Smoothing coefficient (default: 0.9999)
  • eps (float) – For improving the numerical stability (default: 1e-8)
  • weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)

8.17 Adam, Example

8.18 Adagrad Optimizer

Adaptive Gradient Algorithm (Adagrad) is an algorithm for gradient-based optimization where each parameter has its own learning rate that improves performance on problems with sparse gradients.

8.19 Adagrad, Syntax

torch.optim.Adagrad(params, lr=0.01, lr_decay=0, weight_decay=0, initial_accumulator_value=0, eps=1e-10)

Parameters

  • params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
  • lr (float) – Learning rate helping optimization (default: 1e-3)
  • lr_decay (float, optional) – learning rate decay (default: 0)
  • eps (float) – For improving the numerical stability (default: 1e-8)
  • weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)

8.20 Adagrad, Example

8.21 Adadelta Optimizer

Adadelta is an extension of the Adagrad optimizer which adapts learning rate based on a moving window of gradient updates instead of accumulating all previous gradients.

8.22 Adadelta, Syntax

torch.optim.Adadelta(params, lr=1.0, rho=0.9, eps=1e-06, weight_decay=0)

Parameters

  • params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
  • lr (float) – Learning rate helping optimization (default: 1e-3)
  • betas (Tuple[float, float]) – This parameter is used for calculating and running the averages for gradient (default: (0.9, 0.999))
  • beta3 (float) – Smoothing coefficient (default: 0.9999)
  • eps (float) – For improving the numerical stability (default: 1e-8)
  • weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)

8.23 Adadelta, Example

8.24 AdamW Optimizer

AdamW optimizer is a variation of Adam optimizer that performs the optimization of both weight decay and learning rate separately. It is supposed to converge faster than Adam in certain scenarios.

8.25 AdamW, Syntax

torch.optim.AdamW(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01, amsgrad=False)

Parameters

  • params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
  • lr (float) – Learning rate helping optimization (default: 1e-3)
  • betas (Tuple[float, float], optional) – It helps in calculation of averages for the gradient squares (default: (0.9, 0.999))
  • eps (float) – For improving the numerical stability (default: 1e-8)
  • weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)
  • amsgrad (boolean, optional) – This parameter helps in deciding whether AMSgrad algorithm will be used or not(default: False)

8.26 AdamW, Example

8.27 Adamax Optimizer

Adamax optimizer is a variant of Adam optimizer that uses infinity norm. Though it is not used widely in practical work some research shows Adamax results are better than Adam optimizer.

8.28 Adamax, Syntax

torch.optim.Adamax(params, lr=0.002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)

Parameters

  • params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
  • lr (float) – Learning rate helping optimization (default: 1e-3)
  • betas (Tuple[float, float], optional) – It helps in calculation of averages for the gradient squares (default: (0.9, 0.999))
  • eps (float) – For improving the numerical stability (default: 1e-8)
  • weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)

8.29 Adamax, Example

8.30 RMSProp

RMSProp applies stochastic gradient with mini-batch and uses adaptive learning rates which means it adjusts the learning rates over time.

8.31 RMSProp, Syntax

torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, centered=False)

Parameters

  • params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
  • lr (float) – Learning rate helping optimization (default: 1e-3)
  • betas (Tuple[float, float], optional) – It helps in calculation of averages for the gradient squares (default: (0.9, 0.999))
  • eps (float) – For improving the numerical stability (default: 1e-8)
  • momentum (float, optional) – momentum factor (default: 0)
  • alpha (float, optional) – smoothing constant (default: 0.99)
  • centered (bool, optional) – if True, compute the centered RMSProp, the gradient is normalized by an estimation of its variance
  • weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)

8.32 RMSProp, Example

8.33 Conclusion

A lot of choices, parameters and no obvious winner.

Some optimizers struggle even with a simple linear regression problem, but are better for more complex problems.

9 Stochastic Gradient Descent

9.1 Introduction

Stochastic Gradient Descent is a stochastic variant of gradient descent, which uses an efficient algorithm to minimize cost functions in the form of a sum:

\[ Q(\mathbf{w}) = \sum_{i=1}^{d} Q_i(\mathbf{w}) \; , \]

where \(\mathbf{w}\) is a vector of parameters (or weights) to be optimized. The \(Q_i\) component is the contribution of the \(i\)th sample to the total cost \(Q\), which is precisely what we aim to minimize using a training set of \(d\) samples.

9.2 Why stochastic?

Using standard gradient descent, \(Q\) can be minimized by the next iteration:

\[\begin{align*} \mathbf{w}_{t+1} &=& \mathbf{w}_t - \eta \nabla Q \\ &=& \mathbf{w}_t - \eta \sum_{i=1}^{d} \nabla Q_i(\mathbf{w}_t) \; ,\\ \end{align*} \]

where \(\eta > 0\) is the step size. This batch iteration method calculates the total cost at each step with a computation time that is proportional to the size \(d\) of the training set.

Similarly, in stochastic gradient descent, \(Q\) is minimized using

\[ \mathbf{w}_{t+1} = \mathbf{w}_t - \eta \nabla Q_i (\mathbf{w}_t) \; , \]

by updating the \(\mathbf{w}\) weights at each iteration using just a randomly chosen \(i\) sample from the training set. This is extremely efficient for large sample sizes, because it makes the computation time of each iteration independent of \(d\). Another advantage is that it allows samples to be processed on the fly, like an online learning task.

9.3 The step size is the key

In practice, instead of a fixed \(\eta\), the algorithm decreases the step size \(\eta_t\) to improve convergence. That’s the job of the optimizer to chose the relevant step size at each iteration.

9.4 Example setting

We’re going to implement a version with constant step.

In this implementation \(Q_i\) has the form

\[ Q_i(\mathbf{w}) = \left\Vert \mathbf{y}_i - f_{\mathbf{w}}(\mathbf{x}_i) \right\Vert \; , \]

where \(f_{\mathbf{w}} : \mathbb{R}^n \to \mathbb{R}^m\) is a model function for our dataset, parameterized by \(\mathbf{w}\); \(\mathbf{x}_i \in \mathbb{R}^n\) and \(\mathbf{y}_i \in \mathbb{R}^m\) are the input/output pair for the \(i\)th sample of the training set. Find the parameters \(\mathbf{w}\) that minimize \(Q(\mathbf{w}) = \sum_{i=1}^{d} Q_i (\mathbf{w})\) and fit the model function \(f_{\mathbf{w}}\) to our data.

9.5 Python imports

import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from tqdm.notebook import tqdm 

We will test this with curve fitting on the following:

\[f_{\mathbf{w}} (x) = w_1 x^2 + w_2 x + w_3\]

def f(w,x):
    return w[0] * x * x + w[1] * x + w[2]

Let’s start by defining our dataset

wt = [0.3,-1.7,2.5] # Ce sont les paramètres que l'on va essayer de retrouver
xtrain = np.arange(-10,10,0.01,dtype='f')
ytrain = (lambda x : f(wt,x))(xtrain)

9.6 Plot

import plotly.express as px
px.line(x=xtrain,y=ytrain)

9.7 Preparing the dataset

To process our training set, we need to put together inputs and outputs(target) as columns using the concatenate function..

xtrainnp = np.array([xtrain])
ytrainnp = np.array([ytrain])
dataTconcat = np.concatenate((xtrainnp,ytrainnp))
points = np.transpose(dataTconcat)
points
array([[-10.      ,  49.5     ],
       [ -9.99    ,  49.423027],
       [ -9.98    ,  49.346115],
       ...,
       [  9.970457,  15.373228],
       [  9.980457,  15.416082],
       [  9.990458,  15.458996]], dtype=float32)

9.8 Initialization

We initialize the \(w\) parameters

w0 = torch.zeros(3,requires_grad=True)

9.9 First “naive” version

In a first version of the naive SGD, we’ll iterate over all the samples and increment our weight vector on each of them.

We’ll have the function:

def sgd(f,w,t,eta=0.00001, epsilon=0.001,nepoch=100):

where eta is the step size, epsilon is the convergence limit of the cost, and nepoch is the maximum number of times all samples will be traversed.

9.10 Exercise 1

Complete the following code to perform a gradient descent

def sgd(f,w,t,eta=0.0001, epsilon=0.01,nepoch=20):
    bar = tqdm(total=nepoch) # 
    epoch = 0
    x = t[:,0]
    y = t[:,1]
    while(epoch < nepoch):
        loss = ... 
        ...
        bar.update() # Displaying the change
        bar.set_description("loss %f" % loss.item())
        epoch += 1
        if (loss.item() < epsilon):
            print("convergence au bout de %i epochs." % epoch)
            break
w0 = torch.zeros(3,requires_grad=True)
sgd(f,w0,torch.tensor(points),eta=0.0002,epsilon=0.1,nepoch=100)

We check that we are closer to the model parameters.

Code
def compareLoss(w):
    return nn.MSELoss()(f(w,torch.tensor(points[:,0])),
                        torch.tensor(points[:,1]))
print(f"Initial loss : {compareLoss(torch.zeros(3))}")
print(f"Final loss : {compareLoss(w0)}")
Initial loss : 333.13409423828125
Final loss : 9.24051570892334

9.11 Exercice 2

Slight simplification, use the ready-made cost function torch.nn.MSELoss().

criterion = torch.nn.MSELoss()

Write the version of sgd with this “ready-to-use” cost function

w0 = torch.zeros(3,requires_grad=True)
sgd(f,w0,torch.tensor(points),eta=0.0002,epsilon=0.1,nepoch=100)

9.12 Mini-batch

We’re now going to set up the real minibatch version of the SGD. To do this, we make direct use of pytorch’s DataLoader, which combines a dataset and a sampler.

fakedata = torch.tensor(np.repeat(np.arange(10),2).reshape(10,2))
trainloader = DataLoader(fakedata,batch_size=3,shuffle=True)

9.13 Exercice 3

list all the items in each batch with the following printout

Code
for i, data in enumerate(trainloader):
    print(f"batch {i} : {data}")
batch 0 : tensor([[4, 4],
        [6, 6],
        [7, 7]])
batch 1 : tensor([[8, 8],
        [1, 1],
        [2, 2]])
batch 2 : tensor([[3, 3],
        [0, 0],
        [5, 5]])
batch 3 : tensor([[9, 9]])

9.14 Exercice 4

Rewrite the mini-batch version of gradient descent using the trainloader. Note that for cost calculation, you now need to cumulate over all batches at each epoch. Think carefully about the right strategy.

w0 = torch.zeros(3,requires_grad=True)
sgd(f,w0,torch.tensor(points),eta=0.0002,epsilon=0.1,nepoch=50)
convergence after 48 epochs.
print(f"Initial loss : {compareLoss(torch.zeros(3))}")
print(f"Final loss : {compareLoss(w0)}")
Initial loss : 333.13409423828125
Final loss : 0.09605718404054642

9.15 Optimizers

Now we just need to use the optimizers in torch.optim like Adam, RMSProp etc.

the optimizer must be called with the parameters \(w\)

optimizer = torch.optim.Adam([w], lr=lr)

After each backpropagation, you can call optimizer.step(): this will directly perform the descent “step” and reset the gradients, so you don’t need to do it again.

9.16 Exercice 5

Version of sgd with RMSprop optimizer.

w0 = torch.zeros(3,requires_grad=True)
sgd(f,w0,torch.tensor(points),epsilon=1e-7,nepoch=50)
convergence after 33 iterations.

Compare the final loss

print(f"Initial loss : {compareLoss(torch.zeros(3))}")
print(f"Final loss : {compareLoss(w0)}")
Initial loss : 333.13409423828125
Final loss : 9.98525848672216e-08

9.17 Exercice 6

Try sgd again with the shuffle=False option in the trainloader. What do you find? Why?

9.18 Exercice 7

  • Compare in the sgd the cost function \(L(X,Y)\) on the set of points with the sum of the batch cost functions \(\sum{L(X_i,Y_i)}\). How to explain the difference? Is it serious, Doctor? (hint: think of triangular inequality).
  • Same question, but for the gradient norm.

10 Layers and non linearities

Last but not least, we have the notion of layers, hence the “Deep” in learning, and non linearities which are a key component of neural networks.

Code
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
import plotly.graph_objs as go
import plotly.express as px
from sklearn.metrics import confusion_matrix
import numpy as np
from torch.utils.tensorboard import SummaryWriter
from torchvision.utils import make_grid

10.1 Affine maps

One of the core workhorses of deep learning is the affine map, which is a function \(f(x)\) where

\[ f(x) = Ax + b \]

for a matrix \(A\) and vectors \(x, b\). The parameters to be learned here are \(A\) and \(b\). Often, \(b\) is refered to as the bias term.

10.2 Dialectal considerations

PyTorch and most other deep learning frameworks do things a little differently than traditional linear algebra.

It maps the rows of the input instead of the columns. That is, the \(i\)’th row of the output below is the mapping of the \(i\)’th row of the input under \(A\), plus the bias term. Look at the example below.

10.3 Example

lin = nn.Linear(5, 3)  # maps from R^5 to R^3, parameters A, b
# data is 2x5.  A maps from 5 to 3... can we map "data" under A?
data = torch.randn(2, 5)
print(lin(data))  # yes
tensor([[-0.6200, -0.2753, -0.6522],
        [ 0.4319,  0.5096, -0.4357]], grad_fn=<AddmmBackward0>)
lin.weight
Parameter containing:
tensor([[ 0.2227, -0.0270, -0.1261, -0.2731, -0.2658],
        [ 0.1531, -0.1252, -0.2985, -0.0013, -0.0438],
        [ 0.2553, -0.2737, -0.0079, -0.0471,  0.1810]], requires_grad=True)
lin.bias
Parameter containing:
tensor([ 0.1400, -0.1005, -0.3270], requires_grad=True)

10.4 Stack them up

If we compose affine maps, that is we just “stack” them on top of each, we’re still only computing affine maps.

Suppose we have two affine maps \(f(x) = Ax + b\) and \(g(x) = Cx + d\). What is \(f(g(x))\)?

\[ f(g(x)) = A(Cx + d) + b = ACx + (Ad + b) \]

\(AC\) is a matrix and \(Ad + b\) is a vector, so we see that composing affine maps gives you an affine map.

10.5 Non-linearities, or activation functions

From this, you can see that if you wanted your neural network to be long chains of affine compositions, that this adds no new power to your model than just doing a single affine map.

There are a few core non-linearities. \(\tanh(x), \sigma(x), \text{ReLU}(x)\) are the most common. You are probably wondering: “why these functions? I can think of plenty of other non-linearities.” The reason for this is that they have gradients that are easy to compute, and computing gradients is essential for learning.

\[ \frac{d\sigma}{dx} = \sigma(x)(1 - \sigma(x)) \]

The well known sigmoid function has a particularly nice derivative and a nice interpretation as a saturating “activation” signal in neurons. But it is not very efficient in deep networks, because of the vanishing gradient problem.

Code
x = np.linspace(-10, 10, 100)
y = 1 / (1 + np.exp(-x))

fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y, name='Sigmoid'))
fig.update_layout(title='Sigmoid Function', xaxis_title='x', yaxis_title='y')
fig.show()

This is because the gradient vanishes very quickly as the absolute value of the argument grows. Small gradients means it is hard to learn. Most people default to tanh or ReLU.

Code
x = np.linspace(-10, 10, 100)
y = np.maximum(0, x)

fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y, name='ReLU'))
fig.update_layout(title='Rectified Linear Unit (ReLU) Function', xaxis_title='x', yaxis_title='y')
fig.show()

10.6 Build the network

We define our neural network by subclassing nn.Module, and initialize the neural network layers in __init__. Every nn.Module subclass implements the operations on input data in the forward method.

class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
1        self.flatten = nn.Flatten()
2        self.linear_relu_stack = nn.Sequential(
3            nn.Linear(28*28, 512),
4            nn.ReLU(),
5            nn.Linear(512, 512),
            nn.ReLU(),
6            nn.Linear(512, 10),
        )

    def forward(self, x):
7        x = self.flatten(x)
8        logits = self.linear_relu_stack(x)
        return logits
1
Flatten just flattens the input tensor into a vector
2
Sequential is a container for the layers
3
28*28 is the size of the input image
4
ReLU is the activation function
5
512 is the size of the output of the previous layer
6
10 is the size of the output of the previous layer
7
flatten the input
8
logits is the output of the network before applying the softmax function

10.7 Instantiate the network

We create an instance of NeuralNetwork, and move it to the device, and print its structure.

model = NeuralNetwork().to("cuda")
print(model)
NeuralNetwork(
  (flatten): Flatten(start_dim=1, end_dim=-1)
  (linear_relu_stack): Sequential(
    (0): Linear(in_features=784, out_features=512, bias=True)
    (1): ReLU()
    (2): Linear(in_features=512, out_features=512, bias=True)
    (3): ReLU()
    (4): Linear(in_features=512, out_features=10, bias=True)
  )
)

10.8 Better visualization with tensorboard

writer = SummaryWriter('torchlogs/')
writer.add_graph(model, torch.rand(64, 28*28).to("cuda"))
writer.close()

and see it with the tensorboard on the dir torchlogs:

Tensorboard Launch from jupyterlab

Tensorboard Graph

10.9 Model Layers

Let’s break down the layers in the FashionMNIST model. To illustrate it, we will take a sample minibatch of 3 images of size 28x28 and see what happens to it as we pass it through the network.

input_image = torch.rand(3,28,28)
print(input_image.size())
torch.Size([3, 28, 28])

10.10 nn.Flatten

We initialize the nn.Flatten layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values ( the minibatch dimension (at dim=0) is maintained).

flatten = nn.Flatten()
flat_image = flatten(input_image)
print(flat_image.size())
torch.Size([3, 784])

10.11 nn.Linear

The linear layer is a module that applies a linear transformation on the input using its stored weights and biases.

layer1 = nn.Linear(in_features=28*28, out_features=20)
hidden1 = layer1(flat_image)
print(hidden1.size())
torch.Size([3, 20])

10.11.1 nn.ReLU

In this model, we use nn.ReLU between our linear layers, but there’s other activations to introduce non-linearity in your model to introduce non linearity in our model.

print(f"Before ReLU: {hidden1}\n\n")
hidden1 = nn.ReLU()(hidden1)
print(f"After ReLU: {hidden1}")
Before ReLU: tensor([[ 0.2991,  0.1975,  0.1709, -0.6827,  0.3192,  0.2041, -0.0016,  0.2832,
         -0.3922,  0.2677, -0.1190, -0.3733,  0.2332,  0.0607,  0.7411,  0.5116,
          0.2189,  0.3999, -0.2244,  0.3730],
        [ 0.5172, -0.2357,  0.2192, -0.7395,  0.2954,  0.2465,  0.2205,  0.0700,
         -0.0016,  0.5997, -0.2390, -0.4861,  0.8645,  0.3372,  0.9774,  0.5403,
          0.3724,  0.3531, -0.0121, -0.0739],
        [ 0.2557,  0.6357,  0.2762, -0.6375,  0.0539,  0.2871,  0.1220,  0.2155,
         -0.3984,  0.4774, -0.1118, -0.1641, -0.0050,  0.3453,  0.9153,  0.5326,
          0.5245,  0.1613, -0.0555,  0.1386]], grad_fn=<AddmmBackward0>)


After ReLU: tensor([[0.2991, 0.1975, 0.1709, 0.0000, 0.3192, 0.2041, 0.0000, 0.2832, 0.0000,
         0.2677, 0.0000, 0.0000, 0.2332, 0.0607, 0.7411, 0.5116, 0.2189, 0.3999,
         0.0000, 0.3730],
        [0.5172, 0.0000, 0.2192, 0.0000, 0.2954, 0.2465, 0.2205, 0.0700, 0.0000,
         0.5997, 0.0000, 0.0000, 0.8645, 0.3372, 0.9774, 0.5403, 0.3724, 0.3531,
         0.0000, 0.0000],
        [0.2557, 0.6357, 0.2762, 0.0000, 0.0539, 0.2871, 0.1220, 0.2155, 0.0000,
         0.4774, 0.0000, 0.0000, 0.0000, 0.3453, 0.9153, 0.5326, 0.5245, 0.1613,
         0.0000, 0.1386]], grad_fn=<ReluBackward0>)

10.12 nn.Sequential

nn.Sequential is an ordered container of modules. The data is passed through all the modules in the same order as defined. You can use sequential containers to put together a quick network like seq_modules.

seq_modules = nn.Sequential(
    flatten,
    layer1,
    nn.ReLU(),
    nn.Linear(20, 10)
)
input_image = torch.rand(3,28,28)
logits = seq_modules(input_image)

10.13 nn.Softmax

The last linear layer of the neural network returns logits - raw values in \([-\infty, \infty]\) - which are passed to the nn.Softmax module. The logits are scaled to values \([0, 1]\) representing the model’s predicted probabilities for each class. dim parameter indicates the dimension along which the values must sum to 1.

softmax = nn.Softmax(dim=1)
pred_probab = softmax(logits)

10.14 Model Parameters

Many layers inside a neural network are parameterized, i.e. have associated weights and biases that are optimized during training. Subclassing nn.Module automatically tracks all fields defined inside your model object, and makes all parameters accessible using your model’s parameters() or named_parameters() methods.

In this example, we iterate over each parameter, and print its size and a preview of its values.

print(f"Model structure: {model}\n\n")

for name, param in model.named_parameters():
    print(f"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \n")
Model structure: NeuralNetwork(
  (flatten): Flatten(start_dim=1, end_dim=-1)
  (linear_relu_stack): Sequential(
    (0): Linear(in_features=784, out_features=512, bias=True)
    (1): ReLU()
    (2): Linear(in_features=512, out_features=512, bias=True)
    (3): ReLU()
    (4): Linear(in_features=512, out_features=10, bias=True)
  )
)


Layer: linear_relu_stack.0.weight | Size: torch.Size([512, 784]) | Values : tensor([[-0.0011,  0.0036, -0.0158,  ...,  0.0299, -0.0061,  0.0115],
        [-0.0295, -0.0215, -0.0321,  ...,  0.0058,  0.0018, -0.0251]],
       device='cuda:0', grad_fn=<SliceBackward0>) 

Layer: linear_relu_stack.0.bias | Size: torch.Size([512]) | Values : tensor([-0.0228, -0.0109], device='cuda:0', grad_fn=<SliceBackward0>) 

Layer: linear_relu_stack.2.weight | Size: torch.Size([512, 512]) | Values : tensor([[-0.0008,  0.0102,  0.0376,  ..., -0.0059,  0.0174, -0.0057],
        [ 0.0354, -0.0045,  0.0153,  ...,  0.0393, -0.0301,  0.0434]],
       device='cuda:0', grad_fn=<SliceBackward0>) 

Layer: linear_relu_stack.2.bias | Size: torch.Size([512]) | Values : tensor([-0.0332,  0.0050], device='cuda:0', grad_fn=<SliceBackward0>) 

Layer: linear_relu_stack.4.weight | Size: torch.Size([10, 512]) | Values : tensor([[-0.0271, -0.0068, -0.0369,  ...,  0.0203, -0.0393,  0.0267],
        [ 0.0225,  0.0083, -0.0204,  ...,  0.0135, -0.0117,  0.0170]],
       device='cuda:0', grad_fn=<SliceBackward0>) 

Layer: linear_relu_stack.4.bias | Size: torch.Size([10]) | Values : tensor([-0.0399,  0.0277], device='cuda:0', grad_fn=<SliceBackward0>) 

10.15 Saving and loading model weights

PyTorch models store the learned parameters in an internal state dictionary, called state_dict. These can be persisted via the torch.save method:

torch.save(model.state_dict(), 'model_weights.pth')

To load model weights, you need to create an instance of the same model first, and then load the parameters using load_state_dict() method.

model2 = NeuralNetwork()
model2.load_state_dict(torch.load('model_weights.pth'))
<All keys matched successfully>

10.16 Loading the dataset

The torchvision.datasets module contains Dataset objects for many real-world vision data like CIFAR, COCO (full list here). In this tutorial, we use the FashionMNIST dataset. Every TorchVision Dataset includes two arguments: transform and target_transform to modify the samples and labels respectively.

# Download training data from open datasets.
training_data = datasets.FashionMNIST(
    root="data",
    train=True,
    download=True,
    transform=transforms.ToTensor(),
)
# Download test data from open datasets.
test_data = datasets.FashionMNIST(
    root="data",
    train=False,
    download=True,
    transform=transforms.ToTensor(),
)

We pass the Dataset as an argument to DataLoader. This wraps an iterable over our dataset, and supports automatic batching, sampling, shuffling and multiprocess data loading. Here we define a batch size of 64, i.e. each element in the dataloader iterable will return a batch of 64 features and labels.

batch_size = 64

# Create data loaders.
train_dataloader = DataLoader(training_data, batch_size=batch_size)
test_dataloader = DataLoader(test_data, batch_size=batch_size)

for X, y in test_dataloader:
    print(f"Shape of X [N, C, H, W]: {X.shape}")
    print(f"Shape of y: {y.shape} {y.dtype}")
    break
Shape of X [N, C, H, W]: torch.Size([64, 1, 28, 28])
Shape of y: torch.Size([64]) torch.int64

10.17 Setting up the loss function and optimizer

As the loss function, as we are in multiclass classification problem, we will use the cross entropy loss. 

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

10.18 Exercise

Implement the training and testing functions.

def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    model.train()
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to("cuda"), y.to("cuda")

        # Compute prediction error
        pred = ...
        loss = ...

        # Backpropagation
        ...

        if batch % 100 == 0:
            loss, current = loss.item(), (batch + 1) * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")
def test(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to("cuda"), y.to("cuda")
            pred = ...
            test_loss += ...
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()
    test_loss /= num_batches
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

10.19 Launch the training and testing

epochs = 5
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train(train_dataloader, model, loss_fn, optimizer)
    test(test_dataloader, model, loss_fn)
print("Done!")
Epoch 1
-------------------------------
loss: 2.313305  [   64/60000]
loss: 2.302577  [ 6464/60000]
loss: 2.279602  [12864/60000]
loss: 2.269845  [19264/60000]
loss: 2.248892  [25664/60000]
loss: 2.214803  [32064/60000]
loss: 2.236776  [38464/60000]
loss: 2.194689  [44864/60000]
loss: 2.193618  [51264/60000]
loss: 2.169196  [57664/60000]
Test Error: 
 Accuracy: 36.0%, Avg loss: 2.161442 

Epoch 2
-------------------------------
loss: 2.176624  [   64/60000]
loss: 2.167854  [ 6464/60000]
loss: 2.116770  [12864/60000]
loss: 2.130357  [19264/60000]
loss: 2.072228  [25664/60000]
loss: 2.017969  [32064/60000]
loss: 2.051332  [38464/60000]
loss: 1.970719  [44864/60000]
loss: 1.973851  [51264/60000]
loss: 1.918556  [57664/60000]
Test Error: 
 Accuracy: 51.9%, Avg loss: 1.908704 

Epoch 3
-------------------------------
loss: 1.940778  [   64/60000]
loss: 1.915648  [ 6464/60000]
loss: 1.805761  [12864/60000]
loss: 1.847780  [19264/60000]
loss: 1.730446  [25664/60000]
loss: 1.680535  [32064/60000]
loss: 1.707684  [38464/60000]
loss: 1.604326  [44864/60000]
loss: 1.627919  [51264/60000]
loss: 1.536645  [57664/60000]
Test Error: 
 Accuracy: 59.1%, Avg loss: 1.544043 

Epoch 4
-------------------------------
loss: 1.610138  [   64/60000]
loss: 1.577930  [ 6464/60000]
loss: 1.428814  [12864/60000]
loss: 1.503949  [19264/60000]
loss: 1.380030  [25664/60000]
loss: 1.366920  [32064/60000]
loss: 1.389339  [38464/60000]
loss: 1.307619  [44864/60000]
loss: 1.340239  [51264/60000]
loss: 1.250525  [57664/60000]
Test Error: 
 Accuracy: 63.2%, Avg loss: 1.269593 

Epoch 5
-------------------------------
loss: 1.349123  [   64/60000]
loss: 1.330799  [ 6464/60000]
loss: 1.163287  [12864/60000]
loss: 1.273368  [19264/60000]
loss: 1.145072  [25664/60000]
loss: 1.160446  [32064/60000]
loss: 1.191180  [38464/60000]
loss: 1.121791  [44864/60000]
loss: 1.156709  [51264/60000]
loss: 1.080867  [57664/60000]
Test Error: 
 Accuracy: 64.7%, Avg loss: 1.096680 

Done!

10.20 Displaying images in tensorboard

from torch.utils.tensorboard import SummaryWriter
from torchvision.utils import make_grid

writer = SummaryWriter('torchlogs/')
a_batch = next(iter(train_dataloader))[0]
writer.add_images('a batch of images',a_batch,dataformats='NCHW')
writer.close()

10.21 List of labels

Label Description
0 T-shirt/top
1 Trouser
2 Pullover
3 Dress
4 Coat
5 Sandal
6 Shirt
7 Sneaker
8 Bag
9 Ankle boot
labels = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
          'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

10.22 Getting confusion matrix

model.eval()

y_pred = model(test_data.data.float().to("cuda")).argmax(1).cpu()
cm = confusion_matrix(test_data.targets,y_pred)

10.23 Plotting the confusion matrix

Code
def plot_confusion_matrix(cm, labels, title):
    # cm : confusion matrix list(list)
    # labels : name of the data list(str)
    # title : title for the heatmap
    data = go.Heatmap(z=cm, y=labels, x=labels)
    annotations = []
    for i, row in enumerate(cm):
        for j, value in enumerate(row):
            annotations.append(
                {
                    "x": labels[j],
                    "y": labels[i],
                    "font": {"color": "white"},
                    "text": str(value),
                    "xref": "x1",
                    "yref": "y1",
                    "showarrow": False
                }
            )
    layout = {
        "title": title,
        "xaxis": {"title": "Predicted value"},
        "yaxis": {"title": "Real value"},
        "annotations": annotations
    }
    fig = go.Figure(data=data, layout=layout)
    fig.update_yaxes(autorange="reversed")
    fig.update_xaxes(tickangle=-45)
    return fig
plot_confusion_matrix(cm,labels,'Confusion matrix')

10.24 Plotting the misclassified images

Code
import numpy as np
def get_samples(dataloader,class_id,n_samples=10):
    imgs = np.zeros((10, 28, 28))
    predicted = np.zeros(10)
    model.eval()
    s = 0
    done = False
    for X, y in dataloader:
        for i in range(len(y)):
            if done:
                break
            if y[i] == class_id:
                imgs[s] = X[i]
                predicted[s] = model(X[i].unsqueeze(0).to("cuda")).argmax(1)
                s += 1
            if s == 10:
                done = True
    return imgs,predicted

10.25 Displaying 10 misclassified images on specific class

Code
import plotly.express as px

def plot_predclass(class_id):
    imgs,predicted = get_samples(test_dataloader,class_id)
    fig = px.imshow(imgs, facet_col=0, facet_col_wrap=5, binary_string=True)
    for i, pred in enumerate(predicted):
        fig.layout.annotations[i]['text'] = labels[int(pred)]
        if int(pred) != class_id:
            # Make the misclassified facet's background red
            fig.layout.annotations[i]['font']['color'] = 'red'
    fig.show()
plot_predclass(labels.index('Ankle boot'))

  1. \(\epsilon{}\neq{}0\)↩︎

  2. the storage of \(f(x + \epsilon)\) and \(f(x)\) during the calculation up to the final subtraction makes you “forget” the low-order bits that are nevertheless significant for the result.↩︎