import torch
import numpy as np4 Bases of Deep Learning
5 Tensors in pytorch
(Adapted from PyTorch Tutorials)
Tensors are a specialized data structure that are very similar to arrays and matrices. In PyTorch, we use tensors to encode the inputs and outputs of a model, as well as the model’s parameters.
5.1 Warm-up: numpy
Tensors are similar to NumPy’s ndarrays, except that tensors can run on GPUs or other hardware accelerators. In fact, tensors and NumPy arrays can often share the same underlying memory, eliminating the need to copy data (see Section 5.18).
Tensors are also optimized for automatic differentiation (we’ll see more on that Chapter 6).
\(\Leftarrow\) If you’re familiar with ndarrays, you’ll be right at home with the Tensor API.
5.2 Initializing a Tensor
Tensors can be initialized in various ways.
- from data
- from NumPy arrays
- from other tensors
- with random or constant values
5.3 Initializing a Tensor: from data
Tensors can be created directly from data. The data type is automatically inferred.
data = [[1, 2],[3, 4]]
x_data = torch.tensor(data)5.4 Initializing a Tensor: from NumPy
Tensors can be created from NumPy arrays (and vice versa - see Section 5.18).
np_array = np.array(data)
x_np = torch.from_numpy(np_array)5.5 Initializing a Tensor: from another tensor
The new tensor retains the properties (shape, datatype) of the argument tensor, unless explicitly overridden.
# retains the properties of x_data
x_ones = torch.ones_like(x_data)
print(f"Ones Tensor: \n {x_ones} \n")
# overrides the datatype of x_data
x_rand = torch.rand_like(x_data, dtype=torch.float)
print(f"Random Tensor: \n {x_rand} \n")Ones Tensor:
tensor([[1, 1],
[1, 1]])
Random Tensor:
tensor([[0.0029, 0.9247],
[0.3090, 0.3638]])
5.6 Initializing a Tensor: with random or constant values
shape is a tuple of tensor dimensions. In the functions below, it determines the dimensionality of the output tensor.
shape = (2,3,)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)
zeros_tensor = torch.zeros(shape)
print(f"Random Tensor: \n {rand_tensor} \n")
print(f"Ones Tensor: \n {ones_tensor} \n")
print(f"Zeros Tensor: \n {zeros_tensor}")Random Tensor:
tensor([[0.7485, 0.0997, 0.0122],
[0.0076, 0.9352, 0.2850]])
Ones Tensor:
tensor([[1., 1., 1.],
[1., 1., 1.]])
Zeros Tensor:
tensor([[0., 0., 0.],
[0., 0., 0.]])
5.7 Attributes of a Tensor
Tensor attributes describe their shape, datatype, and the device on which they are stored.
tensor = torch.rand(3,4)
print(f"Shape of tensor: {tensor.shape}")
print(f"Datatype of tensor: {tensor.dtype}")
print(f"Device tensor is stored on: {tensor.device}")Shape of tensor: torch.Size([3, 4])
Datatype of tensor: torch.float32
Device tensor is stored on: cpu
5.8 Operations on Tensors
Over 100 tensor operations, including arithmetic, linear algebra, matrix manipulation (transposing, indexing, slicing), sampling and more are comprehensively described here.
Each of these operations can be run on the GPU (at typically higher speeds than on a CPU).
5.9 Operations on Tensors: move to GPU
By default, tensors are created on the CPU. We need to explicitly move tensors to the GPU using .to method (after checking for GPU availability). Keep in mind that copying large tensors across devices can be expensive in terms of time and memory!
5.10 To check for GPU availability
# We move our tensor to the GPU if available
if torch.cuda.is_available():
tensor = tensor.to("cuda")
devname = torch.cuda.get_device_name()
print(f"Device tensor is stored on: {devname}")Device tensor is stored on: NVIDIA A10
Try out some of the operations from the list.
5.11 Standard numpy-like indexing and slicing
tensor = torch.ones(4, 4)
print(f"First row: {tensor[0]}")
print(f"First column: {tensor[:, 0]}")
print(f"Last column: {tensor[..., -1]}")
tensor[:,1] = 0
print(tensor)First row: tensor([1., 1., 1., 1.])
First column: tensor([1., 1., 1., 1.])
Last column: tensor([1., 1., 1., 1.])
tensor([[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.]])
5.12 Joining tensors, cat
You can use torch.cat to concatenate a sequence of tensors along a given dimension.
t1 = torch.cat([tensor, tensor, tensor], dim=1)
print(t1)tensor([[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.],
[1., 0., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1.]])
5.13 Joining tensors, stack
See also torch.stack, another tensor joining operator that is subtly different from torch.cat.
t2 = torch.stack([tensor, tensor, tensor], dim=2)
print(t2)tensor([[[1., 1., 1.],
[0., 0., 0.],
[1., 1., 1.],
[1., 1., 1.]],
[[1., 1., 1.],
[0., 0., 0.],
[1., 1., 1.],
[1., 1., 1.]],
[[1., 1., 1.],
[0., 0., 0.],
[1., 1., 1.],
[1., 1., 1.]],
[[1., 1., 1.],
[0., 0., 0.],
[1., 1., 1.],
[1., 1., 1.]]])
5.14 Arithmetic operations
This computes the matrix multiplication between two tensors. y1, y2, y3 will have the same value
tensor.T returns the transpose of a tensor
y1 = tensor @ tensor.T
y2 = tensor.matmul(tensor.T)
y3 = torch.rand_like(y1)
torch.matmul(tensor, tensor.T, out=y3)tensor([[3., 3., 3., 3.],
[3., 3., 3., 3.],
[3., 3., 3., 3.],
[3., 3., 3., 3.]])
This computes the element-wise product. z1, z2, z3 will have the same value
z1 = tensor * tensor
z2 = tensor.mul(tensor)
z3 = torch.rand_like(tensor)
torch.mul(tensor, tensor, out=z3)tensor([[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.]])
5.15 Single-element tensors
If you have a one-element tensor, for example by aggregating all values of a tensor into one value, you can convert it to a Python numerical value using item():
agg = tensor.sum()
agg_item = agg.item()
print(agg_item, type(agg_item))12.0 <class 'float'>
5.16 In-place operations
Operations that store the result into the operand are called in-place. They are denoted by a _ suffix. For example: x.copy_(y), x.t_(), will change x.
print(f"{tensor} \n")
tensor.add_(5)
print(tensor)tensor([[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.],
[1., 0., 1., 1.]])
tensor([[6., 5., 6., 6.],
[6., 5., 6., 6.],
[6., 5., 6., 6.],
[6., 5., 6., 6.]])
5.17 In-place operations: caution
In-place operations save some memory, but can be problematic when computing derivatives because of an immediate loss of history. Hence, their use is discouraged.
5.18 Bridge with NumPy
Tensors on the CPU and NumPy arrays can share their underlying memory locations, and changing one will change the other.
5.19 Bridge with NumPy: Tensor to NumPy array
A change in the tensor reflects in the NumPy array.
t = torch.ones(5)
print(f"t: {t}")
n = t.numpy()
print(f"n: {n}")t: tensor([1., 1., 1., 1., 1.])
n: [1. 1. 1. 1. 1.]
t.add_(1)
print(f"t: {t}")
print(f"n: {n}")t: tensor([2., 2., 2., 2., 2.])
n: [2. 2. 2. 2. 2.]
5.20 Bridge with NumPy: NumPy array to Tensor
Changes in the NumPy array reflects in the tensor.
n = np.ones(5)
t = torch.from_numpy(n)np.add(n, 1, out=n)
print(f"t: {t}")
print(f"n: {n}")t: tensor([2., 2., 2., 2., 2.], dtype=torch.float64)
n: [2. 2. 2. 2. 2.]
6 Automatic Differentiation, Theory and practice
6.1 Definition
Automatic differentiation (AD), also called algorithmic differentiation or computational differentiation, is a set of techniques to numerically evaluate the derivative of a function specified by a computer program… derivatives of arbitrary order can be computed automatically, accurately to working precision, and using at most a small constant factor more arithmetic operations than the original program.
6.2 Numeric differentiation
\[\frac{df(x)}{dx} = \lim_{\epsilon \to 0} \frac{f(x + \epsilon) - f(\epsilon)}{\epsilon}\]
6.3 Symbolic differentiation
\[\begin{align} \frac{d(u(x) + v(x))}{dx} & = & \frac{du(x)}{dx} + \frac{dv(x)}{dx},\\ \frac{d(u(x) v(x))}{dx} & = & u(x) \frac{dv(x)}{dx} + \frac{du(x)}{dx} v(x)\\ & \textrm{etc.} & \end{align} \]
6.4 Benefits of symbolic differentiation
- Implemented in all symbolic calculation software (Mathematica, Maxima, Sage etc.)
- Extremely efficient for moderately complex functions
6.5 Drawbacks of symbolic differentiation
- What to do with “constructed functions” such as constructors, structures, etc.
\[\frac{d(\textrm{make-point}(x,y))}{dx}\]
- If \(f\) is a product, then \(df\) and \(f\) contain common calculations
\[\frac{d(u(x) v(x))}{dx} = u(x) \frac{dv(x)}{dx} + \frac{du(x)}{dx} v(x)\]
likewise \(u\) and \(du\) etc. ⇒ exponential growth of symbolic code
6.6 Forward mode
Automatic differentiation in foward mode = “calculated” symbolic differentiation that proceeds in parallel with the main calculation and the “derived” calculation, while sharing the intermediate results between the two at each stage.
6.7 And what about deep learning?
(Adapted from PyTorch Tutorials)
When training neural networks, the most frequently used algorithm is back propagation. In this algorithm, parameters (model weights) are adjusted according to the gradient of the loss function with respect to the given parameter.
\(\Leftarrow\) To compute those gradients, PyTorch has a built-in automatic differentiation engine called torch.autograd. It supports automatic computation of gradient for any computational graph.
6.8 A Simple Example
Consider the simplest one-layer neural network, with input x, parameters w and b, and some loss function. It can be defined in PyTorch in the following manner:
import torch
x = torch.ones(5) # input tensor
y = torch.zeros(3) # expected output
w = torch.randn(5, 3, requires_grad=True)
b = torch.randn(3, requires_grad=True)
z = torch.matmul(x, w)+b
loss = torch.nn.functional.binary_cross_entropy_with_logits(z, y)6.9 Tensors, Functions and Computational graph
This code defines the following computational graph:

In this network, w and b are parameters, which we need to optimize. Thus, we need to be able to compute the gradients of loss function with respect to those variables. In order to do that, we set the requires_grad property of those tensors.
You can set the value of requires_grad when creating a tensor, or later by using x.requires_grad_(True) method.
A function that we apply to tensors to construct computational graph is in fact an object of class Function. This object knows how to compute the function in the forward direction, and also how to compute its derivative during the backward propagation step.
A reference to the backward propagation function is stored in grad_fn property of a tensor. You can find more information of Function in the documentation.
print(f"Gradient function for z = {z.grad_fn}")
print(f"Gradient function for loss = {loss.grad_fn}")Gradient function for z = <AddBackward0 object at 0x7fd04b3eba90>
Gradient function for loss = <BinaryCrossEntropyWithLogitsBackward0 object at 0x7fd04b3ea410>
6.10 Computing Gradients
To optimize weights of parameters in the neural network, we need to compute the derivatives of our loss function with respect to parameters, namely, we need \(\frac{\partial loss}{\partial w}\) and \(\frac{\partial loss}{\partial b}\) under some fixed values of x and y.
To compute those derivatives, we call loss.backward(), and then retrieve the values from w.grad and b.grad:
loss.backward()
print(w.grad)
print(b.grad)tensor([[0.1977, 0.0475, 0.2733],
[0.1977, 0.0475, 0.2733],
[0.1977, 0.0475, 0.2733],
[0.1977, 0.0475, 0.2733],
[0.1977, 0.0475, 0.2733]])
tensor([0.1977, 0.0475, 0.2733])
- We can only obtain the
gradproperties for the leaf nodes of the computational graph, which haverequires_gradproperty set toTrue. For all other nodes in our graph, gradients will not be available. - We can only perform gradient calculations using
backwardonce on a given graph, for performance reasons. If we need to do severalbackwardcalls on the same graph, we need to passretain_graph=Trueto thebackwardcall.
6.11 Disabling Gradient Tracking
By default, all tensors with requires_grad=True are tracking their computational history and support gradient computation.
However, there are some cases when we do not need to do that, for example, when we have trained the model and just want to apply it to some input data, i.e. we only want to do forward computations through the network.
We can stop tracking computations by surrounding our computation code with torch.no_grad() block:
z = torch.matmul(x, w)+b
print(z.requires_grad)
with torch.no_grad():
z = torch.matmul(x, w)+b
print(z.requires_grad)True
False
Another way to achieve the same result is to use the detach() method on the tensor:
z = torch.matmul(x, w)+b
z_det = z.detach()
print(z_det.requires_grad)False
6.12 Why would we want to disable gradient tracking?
There are reasons you might want to disable gradient tracking:
- To mark some parameters in your neural network as frozen parameters.
- To speed up computations when you are only doing forward pass, because computations on tensors that do not track gradients would be more efficient.
- You want to manually control the updates to the weights in the network.
6.13 More on Computational Graphs : DAGs
Conceptually, autograd keeps a record of data (tensors) and all executed operations (along with the resulting new tensors) in a directed acyclic graph (DAG) consisting of Function objects.
In this DAG, leaves are the input tensors, roots are the output tensors. By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule.
In a forward pass, autograd does two things simultaneously:
- run the requested operation to compute a resulting tensor
- maintain the operation’s gradient function in the DAG.
The backward pass kicks off when .backward() is called on the DAG root. autograd then:
- computes the gradients from each
.grad_fn, - accumulates them in the respective tensor’s
.gradattribute - using the chain rule, propagates all the way to the leaf tensors.
An important thing to note is that the graph is recreated from scratch; after each .backward() call, autograd starts populating a new graph. This is exactly what allows you to use control flow statements in your model; you can change the shape, size and operations at every iteration if needed.
6.14 Tensor Gradients and Jacobian Products
In many cases, we have a scalar loss function, and we need to compute the gradient with respect to some parameters. However, there are cases when the output function is an arbitrary tensor. In this case, PyTorch allows you to compute so-called Jacobian product, and not the actual gradient.
For a vector function \(\vec{y}=f(\vec{x})\), where \(\vec{x}=\langle x_1,\dots,x_n\rangle\) and \(\vec{y}=\langle y_1,\dots,y_m\rangle\), a gradient of \(\vec{y}\) with respect to \(\vec{x}\) is given by Jacobian matrix:
\[ \begin{align}J=\left(\begin{array}{ccc} \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{1}}{\partial x_{n}}\\ \vdots & \ddots & \vdots\\ \frac{\partial y_{m}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}} \end{array}\right)\end{align} \]
Instead of computing the Jacobian matrix itself, PyTorch allows you to compute Jacobian Product \(v^T\cdot J\) for a given input vector \(v=(v_1 \dots v_m)\). This is achieved by calling backward with \(v\) as an argument. The size of \(v\) should be the same as the size of the original tensor, with respect to which we want to compute the product.
6.15 Illustration of Jacobian Product
inp = torch.eye(4, 5,
requires_grad=True)
out = (inp+1).pow(2).t()
out.backward(torch.ones_like(out),
retain_graph=True)
inp.gradtensor([[4., 2., 2., 2., 2.],
[2., 4., 2., 2., 2.],
[2., 2., 4., 2., 2.],
[2., 2., 2., 4., 2.]])
out.backward(torch.ones_like(out),
retain_graph=True)
inp.gradtensor([[8., 4., 4., 4., 4.],
[4., 8., 4., 4., 4.],
[4., 4., 8., 4., 4.],
[4., 4., 4., 8., 4.]])
inp.grad.zero_()
out.backward(torch.ones_like(out),
retain_graph=True)
inp.gradtensor([[4., 2., 2., 2., 2.],
[2., 4., 2., 2., 2.],
[2., 2., 4., 2., 2.],
[2., 2., 2., 4., 2.]])
Notice that when we call backward for the second time with the same argument, the value of the gradient is different.
This happens because when doing backward propagation, PyTorch accumulates the gradients, i.e. the value of computed gradients is added to the grad property of all leaf nodes of computational graph.
If you want to compute the proper gradients, you need to zero out the grad property before. In real-life training an optimizer helps us to do this.
6.16 Computing the Jacobian with respect to some input
Previously we were calling backward() function without parameters. This is essentially equivalent to calling backward(torch.tensor(1.0)), which is a useful way to compute the gradients in case of a scalar-valued function, such as loss during neural network training.
6.17 Accumulating the gradient
As seen in the previous exemple, each call to the backward() function adds the computed gradient to the grad property of the tensor. If you want to compute the proper gradients, you need to zero out the grad property before. In real-life training an optimizer helps us to do this.
7 Gradient Descent
7.1 Algorithm
In this section, we assume \(f\) to be a smooth function and our aim is to introduce the gradient descent method to find its (or one of its) minima.
7.2 Fixed step gradient descent
7.3 Example setting
We’re interested in minimizing the following function:
\(f(x_1, x_2) = (x_1^2 + x_2 - 11)^2 + (x_1 + x_2^2 - 7)^2\)
def f(x):
x1, x2 = x[0],x[1]
return (x1**2 + x2 - 11)**2 + (x1 + x2**2 - 7)**27.4 Surface plot
import numpy as np
from python_utils.interactive_plot import init_plot, plot_surface, plot_grad_descent, plot_contour, plot_surface
X1, X2 = np.meshgrid(np.linspace(-5.5, 5.5, 50),
np.linspace(-5.5, 5.5, 50))
Z = f(np.array([X1, X2]))
plot_surface(X1,X2,Z).show()7.5 Contour plot
plot_contour(X1,X2,Z).show()7.6 Gradient Definition
\[ \nabla f(x_1, x_2) = \begin{bmatrix} \frac{\partial f}{\partial x_1} \\ \frac{\partial f}{\partial x_2} \end{bmatrix} = \begin{bmatrix} 2 \left(-7 + x_1 + x_2^2 + 2x_1(-11 + x_1^2 + x_2)\right) \\ 2 \left(-11 + x_1^2 + x_2 + 2x_2(-7 + x_1 + x_2^2)\right) \end{bmatrix} \]
def f_grad(x):
x1, x2 = x
df_x1 = 2 * (-7 + x1 + x2**2 + 2 * x1 * (-11 + x1**2 + x2))
df_x2 = 2 * (-11 + x1**2 + x2 + 2 * x2 * (-7 + x1 + x2**2))
return np.array([df_x1, df_x2])7.7 Gradient descent code
Complete the following function for gradient descent with constant step size, it should return the list of iterates.
def grad_descent(f,x0,step_size,max_iter=0):
"""Gradient descent with constant step size"""
x = x0 # starting point
xs = [x]
for k in range(max_iter):
# to complete
return xs7.8 Plotting the iterates
from python_utils.interactive_plot import init_plot, plot_grad_descent
x0 = np.array([0.5, -4])
xs = grad_descent(f,x0,0.01,max_iter=20)
init_plot(X1,X2,Z).add_scatter(x=xs[0],y=xs[1],mode='markers+lines').show()7.9 Interactive plot
from python_utils import interactive_plot
plot_grad_descent(grad_descent,f,X1,X2,Z)7.10 And now with pytorch
We will use the automatic differentiation feature of pytorch to compute the gradient of \(f\), and convert the gradient descent algorithm to pytorch.
- To store the iterates as numpy arrays, we will use the
detachmethod of pytorch tensors (get the values out of the computational graph). - We also have to take care of the synchronization between numpy and pytorch, we will use the
copy()method of numpy arrays. - Moreover as we do everything inplace in our main loop, we have to not need to compute the gradient of the iterates, we will use the
no_gradcontext manager to disable gradient computation. Alternatively, we could use the the<op>_internal in place operation.
7.11 Gradient descent code with pytorch
import torch
def torch_grad_descent(f,x0,step_size,max_iter=0):
"""Gradient descent with constant step size"""
x = torch.tensor(x0, requires_grad=True)
xs = ...
for k in range(max_iter):
# to complete
return xs7.12 Plotting the iterates
x0 = np.array([0.5, -4])
xs = torch_grad_descent(f,x0,0.01,max_iter=20)
init_plot(X1,X2,Z).add_scatter(x=xs[0],y=xs[1],
mode='markers+lines').show()7.13 Interactive plot
plot_grad_descent(torch_grad_descent,f,X1,X2,Z)7.14 With another function
\(f(x_1, x_2) = x_1^2 + sin(2x_2) + (cos(2x_1) + x_2^2 - 7)^2 + 2\)
7.15 Surface plot
Code
X1, X2 = np.meshgrid(np.linspace(-5.5, 5.5, 50),
np.linspace(-5.5, 5.5, 50))
Z = f(torch.tensor([X1, X2])).detach().numpy()
plot_surface(X1,X2,Z)/tmp/ipykernel_1739982/1510781993.py:3: UserWarning:
Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /home/conda/feedstock_root/build_artifacts/pytorch-recipe_1680572619157/work/torch/csrc/utils/tensor_new.cpp:245.)
7.16 Contour plot
7.17 Interactive plot
plot_grad_descent(torch_grad_descent,f,X1,X2,Z)8 Optimizers
(Adapted from this guide)
8.1 Introduction
In this tutorial, we will go through PyTorch optimizers which are used to reduce the error rate while training the neural networks. We will first understand what is optimization and then we will see different types of optimizers in PyTorch with their syntax and examples of usage.
8.2 What is Optimization?
During neural network training, its weights are randomly initialized initially and then they are updated in each epoch in a manner such that they increase the overall accuracy of the network.
8.3 Optimization Process
In each epoch, the output of the training data is compared to actual data with the help of the loss function to calculate the error and then the weight is updated accordingly But how do we know how to update the weight such that it increases the accuracy?
This is actually a problem of optimization where the goal is to optimize the loss function and get the ideal weights. And the method used for optimization is called Optimizer.
8.4 Optimization Process, Illustration

8.5 PyTorch Optimizers Examples
Gradient Descent is the most commonly known optimizer but for practical purposes, there are many other optimizers. You will find many of these Optimizers in PyTorch library as well.
- SGD
- Adam
- Adadelta
- Adagrad
- AdamW
- Adamax
- RMSProp
8.6 Setting up the environment
import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
import plotly.express as px
import pandas as pd8.7 Generating Data
We’ll use a simple linear function \(y = Ax + b\) to generate some data. We’ll also add some noise to the data to make it more realistic.
# Let's make some data for a linear regression.
A = 5.1213945
b = 3.2047893
error = 2.0
N = 100 # number of data points
# Data
X = torch.randn(N, 1)
# (noisy) Target values that we want to learn.
t = A * X + b + torch.randn(N, 1) * error8.8 Plotting the data
8.9 Linear Regression
from sklearn.linear_model import LinearRegression
reg = ... # fit the regressin
pred = ... # predictions
regloss = ... # MSE loss
print(f"R2 Score : {reg.score(X, t)}, MSELoss : {regloss}")
px.scatter(x=X[:,0],y=t[:,0]) \
.add_traces(px.line(x=X[:,0],y=pred[:,0]) \
.update_traces(line_color='red').data)8.10 PyTorch Gradient Descent Optimizer
Define our moder and loss function
# Creating a model, defining loss
model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()8.11 Defining the optimizer loop
def runOptimizer(model,opt,niter):
model.reset_parameters()
df = pd.DataFrame({},columns=['index','Data','Value'])
for i in range(0, 30):
1 opt.zero_grad()
2 predictions = model(X)
loss = loss_fn(predictions, t)
df.loc[len(df)] = [i,'learned A',list(model.parameters())[0].data[0,0].item()]
df.loc[len(df)] = [i,'learned b',list(model.parameters())[1].data[0].item()]
df.loc[len(df)] = [i,'loss',loss.data.item()]
3 loss.backward()
4 opt.step()
return df- 1
- Zero the gradients
- 2
- Forward pass
- 3
- Backward pass
- 4
- Update parameters
8.12 SGD
The SGD or Stochastic Gradient Optimizer is an optimizer in which the weights are updated for each training sample or a small subset of data.
8.13 SGD, Syntax
torch.optim.SGD(params, lr=<required parameter>, momentum=0, dampening=0, weight_decay=0, nesterov=False)
Parameters:
- params (iterable) — These are the parameters that help in the optimization.
- lr (float) — This parameter is the learning rate
- momentum (float, optional) — Here we pass the momentum factor
- weight_decay (float, optional) — This argument is containing the weight decay
- dampening (float, optional) — To dampen the momentum, we use this parameter
8.14 SGD, Example
8.15 Adam Optimizer
Adam Optimizer uses both momentum and adaptive learning rate for better convergence. This is one of the most widely used optimizer for practical purposes for training neural networks.
8.16 Adam, Syntax
torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)
Parameters
- params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
- lr (float) – Learning rate helping optimization (default: 1e-3)
- betas (Tuple[float, float]) – This parameter is used for calculating and running the averages for gradient (default: (0.9, 0.999))
- beta3 (float) – Smoothing coefficient (default: 0.9999)
- eps (float) – For improving the numerical stability (default: 1e-8)
- weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)
8.17 Adam, Example
8.18 Adagrad Optimizer
Adaptive Gradient Algorithm (Adagrad) is an algorithm for gradient-based optimization where each parameter has its own learning rate that improves performance on problems with sparse gradients.
8.19 Adagrad, Syntax
torch.optim.Adagrad(params, lr=0.01, lr_decay=0, weight_decay=0, initial_accumulator_value=0, eps=1e-10)
Parameters
- params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
- lr (float) – Learning rate helping optimization (default: 1e-3)
- lr_decay (float, optional) – learning rate decay (default: 0)
- eps (float) – For improving the numerical stability (default: 1e-8)
- weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)
8.20 Adagrad, Example
8.21 Adadelta Optimizer
Adadelta is an extension of the Adagrad optimizer which adapts learning rate based on a moving window of gradient updates instead of accumulating all previous gradients.
8.22 Adadelta, Syntax
torch.optim.Adadelta(params, lr=1.0, rho=0.9, eps=1e-06, weight_decay=0)
Parameters
- params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
- lr (float) – Learning rate helping optimization (default: 1e-3)
- betas (Tuple[float, float]) – This parameter is used for calculating and running the averages for gradient (default: (0.9, 0.999))
- beta3 (float) – Smoothing coefficient (default: 0.9999)
- eps (float) – For improving the numerical stability (default: 1e-8)
- weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)
8.23 Adadelta, Example
8.24 AdamW Optimizer
AdamW optimizer is a variation of Adam optimizer that performs the optimization of both weight decay and learning rate separately. It is supposed to converge faster than Adam in certain scenarios.
8.25 AdamW, Syntax
torch.optim.AdamW(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.01, amsgrad=False)
Parameters
- params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
- lr (float) – Learning rate helping optimization (default: 1e-3)
- betas (Tuple[float, float], optional) – It helps in calculation of averages for the gradient squares (default: (0.9, 0.999))
- eps (float) – For improving the numerical stability (default: 1e-8)
- weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)
- amsgrad (boolean, optional) – This parameter helps in deciding whether AMSgrad algorithm will be used or not(default: False)
8.26 AdamW, Example
8.27 Adamax Optimizer
Adamax optimizer is a variant of Adam optimizer that uses infinity norm. Though it is not used widely in practical work some research shows Adamax results are better than Adam optimizer.
8.28 Adamax, Syntax
torch.optim.Adamax(params, lr=0.002, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)
Parameters
- params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
- lr (float) – Learning rate helping optimization (default: 1e-3)
- betas (Tuple[float, float], optional) – It helps in calculation of averages for the gradient squares (default: (0.9, 0.999))
- eps (float) – For improving the numerical stability (default: 1e-8)
- weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)
8.29 Adamax, Example
8.30 RMSProp
RMSProp applies stochastic gradient with mini-batch and uses adaptive learning rates which means it adjusts the learning rates over time.
8.31 RMSProp, Syntax
torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, centered=False)
Parameters
- params (Union[Iterable[Tensor], Iterable[Dict[str, Any]]]) – These are the iterable parameters that help in optimization
- lr (float) – Learning rate helping optimization (default: 1e-3)
- betas (Tuple[float, float], optional) – It helps in calculation of averages for the gradient squares (default: (0.9, 0.999))
- eps (float) – For improving the numerical stability (default: 1e-8)
- momentum (float, optional) – momentum factor (default: 0)
- alpha (float, optional) – smoothing constant (default: 0.99)
- centered (bool, optional) – if True, compute the centered RMSProp, the gradient is normalized by an estimation of its variance
- weight_decay (float) – For adding the weight decay (L2 penalty) (default: 0)
8.32 RMSProp, Example
8.33 Conclusion
A lot of choices, parameters and no obvious winner.
Some optimizers struggle even with a simple linear regression problem, but are better for more complex problems.
9 Stochastic Gradient Descent
9.1 Introduction
Stochastic Gradient Descent is a stochastic variant of gradient descent, which uses an efficient algorithm to minimize cost functions in the form of a sum:
\[ Q(\mathbf{w}) = \sum_{i=1}^{d} Q_i(\mathbf{w}) \; , \]
where \(\mathbf{w}\) is a vector of parameters (or weights) to be optimized. The \(Q_i\) component is the contribution of the \(i\)th sample to the total cost \(Q\), which is precisely what we aim to minimize using a training set of \(d\) samples.
9.2 Why stochastic?
Using standard gradient descent, \(Q\) can be minimized by the next iteration:
\[\begin{align*} \mathbf{w}_{t+1} &=& \mathbf{w}_t - \eta \nabla Q \\ &=& \mathbf{w}_t - \eta \sum_{i=1}^{d} \nabla Q_i(\mathbf{w}_t) \; ,\\ \end{align*} \]
where \(\eta > 0\) is the step size. This batch iteration method calculates the total cost at each step with a computation time that is proportional to the size \(d\) of the training set.
Similarly, in stochastic gradient descent, \(Q\) is minimized using
\[ \mathbf{w}_{t+1} = \mathbf{w}_t - \eta \nabla Q_i (\mathbf{w}_t) \; , \]
by updating the \(\mathbf{w}\) weights at each iteration using just a randomly chosen \(i\) sample from the training set. This is extremely efficient for large sample sizes, because it makes the computation time of each iteration independent of \(d\). Another advantage is that it allows samples to be processed on the fly, like an online learning task.
9.3 The step size is the key
In practice, instead of a fixed \(\eta\), the algorithm decreases the step size \(\eta_t\) to improve convergence. That’s the job of the optimizer to chose the relevant step size at each iteration.
9.4 Example setting
We’re going to implement a version with constant step.
In this implementation \(Q_i\) has the form
\[ Q_i(\mathbf{w}) = \left\Vert \mathbf{y}_i - f_{\mathbf{w}}(\mathbf{x}_i) \right\Vert \; , \]
where \(f_{\mathbf{w}} : \mathbb{R}^n \to \mathbb{R}^m\) is a model function for our dataset, parameterized by \(\mathbf{w}\); \(\mathbf{x}_i \in \mathbb{R}^n\) and \(\mathbf{y}_i \in \mathbb{R}^m\) are the input/output pair for the \(i\)th sample of the training set. Find the parameters \(\mathbf{w}\) that minimize \(Q(\mathbf{w}) = \sum_{i=1}^{d} Q_i (\mathbf{w})\) and fit the model function \(f_{\mathbf{w}}\) to our data.
9.5 Python imports
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from tqdm.notebook import tqdm We will test this with curve fitting on the following:
\[f_{\mathbf{w}} (x) = w_1 x^2 + w_2 x + w_3\]
def f(w,x):
return w[0] * x * x + w[1] * x + w[2]Let’s start by defining our dataset
wt = [0.3,-1.7,2.5] # Ce sont les paramètres que l'on va essayer de retrouver
xtrain = np.arange(-10,10,0.01,dtype='f')
ytrain = (lambda x : f(wt,x))(xtrain)9.6 Plot
import plotly.express as px
px.line(x=xtrain,y=ytrain)9.7 Preparing the dataset
To process our training set, we need to put together inputs and outputs(target) as columns using the concatenate function..
xtrainnp = np.array([xtrain])
ytrainnp = np.array([ytrain])
dataTconcat = np.concatenate((xtrainnp,ytrainnp))
points = np.transpose(dataTconcat)
pointsarray([[-10. , 49.5 ],
[ -9.99 , 49.423027],
[ -9.98 , 49.346115],
...,
[ 9.970457, 15.373228],
[ 9.980457, 15.416082],
[ 9.990458, 15.458996]], dtype=float32)
9.8 Initialization
We initialize the \(w\) parameters
w0 = torch.zeros(3,requires_grad=True)9.9 First “naive” version
In a first version of the naive SGD, we’ll iterate over all the samples and increment our weight vector on each of them.
We’ll have the function:
def sgd(f,w,t,eta=0.00001, epsilon=0.001,nepoch=100):where eta is the step size, epsilon is the convergence limit of the cost, and nepoch is the maximum number of times all samples will be traversed.
9.10 Exercise 1
Complete the following code to perform a gradient descent
def sgd(f,w,t,eta=0.0001, epsilon=0.01,nepoch=20):
bar = tqdm(total=nepoch) #
epoch = 0
x = t[:,0]
y = t[:,1]
while(epoch < nepoch):
loss = ...
...
bar.update() # Displaying the change
bar.set_description("loss %f" % loss.item())
epoch += 1
if (loss.item() < epsilon):
print("convergence au bout de %i epochs." % epoch)
breakw0 = torch.zeros(3,requires_grad=True)
sgd(f,w0,torch.tensor(points),eta=0.0002,epsilon=0.1,nepoch=100)We check that we are closer to the model parameters.
Code
def compareLoss(w):
return nn.MSELoss()(f(w,torch.tensor(points[:,0])),
torch.tensor(points[:,1]))print(f"Initial loss : {compareLoss(torch.zeros(3))}")
print(f"Final loss : {compareLoss(w0)}")Initial loss : 333.13409423828125
Final loss : 9.24051570892334
9.11 Exercice 2
Slight simplification, use the ready-made cost function torch.nn.MSELoss().
criterion = torch.nn.MSELoss()Write the version of sgd with this “ready-to-use” cost function
w0 = torch.zeros(3,requires_grad=True)
sgd(f,w0,torch.tensor(points),eta=0.0002,epsilon=0.1,nepoch=100)9.12 Mini-batch
We’re now going to set up the real minibatch version of the SGD. To do this, we make direct use of pytorch’s DataLoader, which combines a dataset and a sampler.
fakedata = torch.tensor(np.repeat(np.arange(10),2).reshape(10,2))
trainloader = DataLoader(fakedata,batch_size=3,shuffle=True)9.13 Exercice 3
list all the items in each batch with the following printout
Code
for i, data in enumerate(trainloader):
print(f"batch {i} : {data}")batch 0 : tensor([[4, 4],
[6, 6],
[7, 7]])
batch 1 : tensor([[8, 8],
[1, 1],
[2, 2]])
batch 2 : tensor([[3, 3],
[0, 0],
[5, 5]])
batch 3 : tensor([[9, 9]])
9.14 Exercice 4
Rewrite the mini-batch version of gradient descent using the trainloader. Note that for cost calculation, you now need to cumulate over all batches at each epoch. Think carefully about the right strategy.
w0 = torch.zeros(3,requires_grad=True)
sgd(f,w0,torch.tensor(points),eta=0.0002,epsilon=0.1,nepoch=50)convergence after 48 epochs.
print(f"Initial loss : {compareLoss(torch.zeros(3))}")
print(f"Final loss : {compareLoss(w0)}")Initial loss : 333.13409423828125
Final loss : 0.09605718404054642
9.15 Optimizers
Now we just need to use the optimizers in torch.optim like Adam, RMSProp etc.
the optimizer must be called with the parameters \(w\)
optimizer = torch.optim.Adam([w], lr=lr)After each backpropagation, you can call optimizer.step(): this will directly perform the descent “step” and reset the gradients, so you don’t need to do it again.
9.16 Exercice 5
Version of sgd with RMSprop optimizer.
w0 = torch.zeros(3,requires_grad=True)
sgd(f,w0,torch.tensor(points),epsilon=1e-7,nepoch=50)convergence after 33 iterations.
Compare the final loss
print(f"Initial loss : {compareLoss(torch.zeros(3))}")
print(f"Final loss : {compareLoss(w0)}")Initial loss : 333.13409423828125
Final loss : 9.98525848672216e-08
9.17 Exercice 6
Try sgd again with the shuffle=False option in the trainloader. What do you find? Why?
9.18 Exercice 7
- Compare in the sgd the cost function \(L(X,Y)\) on the set of points with the sum of the batch cost functions \(\sum{L(X_i,Y_i)}\). How to explain the difference? Is it serious, Doctor? (hint: think of triangular inequality).
- Same question, but for the gradient norm.
10 Layers and non linearities
Last but not least, we have the notion of layers, hence the “Deep” in learning, and non linearities which are a key component of neural networks.
Code
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
import plotly.graph_objs as go
import plotly.express as px
from sklearn.metrics import confusion_matrix
import numpy as np
from torch.utils.tensorboard import SummaryWriter
from torchvision.utils import make_grid10.1 Affine maps
One of the core workhorses of deep learning is the affine map, which is a function \(f(x)\) where
\[ f(x) = Ax + b \]
for a matrix \(A\) and vectors \(x, b\). The parameters to be learned here are \(A\) and \(b\). Often, \(b\) is refered to as the bias term.
10.2 Dialectal considerations
PyTorch and most other deep learning frameworks do things a little differently than traditional linear algebra.
It maps the rows of the input instead of the columns. That is, the \(i\)’th row of the output below is the mapping of the \(i\)’th row of the input under \(A\), plus the bias term. Look at the example below.
10.3 Example
lin = nn.Linear(5, 3) # maps from R^5 to R^3, parameters A, b
# data is 2x5. A maps from 5 to 3... can we map "data" under A?
data = torch.randn(2, 5)
print(lin(data)) # yestensor([[-0.6200, -0.2753, -0.6522],
[ 0.4319, 0.5096, -0.4357]], grad_fn=<AddmmBackward0>)
lin.weightParameter containing:
tensor([[ 0.2227, -0.0270, -0.1261, -0.2731, -0.2658],
[ 0.1531, -0.1252, -0.2985, -0.0013, -0.0438],
[ 0.2553, -0.2737, -0.0079, -0.0471, 0.1810]], requires_grad=True)
lin.biasParameter containing:
tensor([ 0.1400, -0.1005, -0.3270], requires_grad=True)
10.4 Stack them up
If we compose affine maps, that is we just “stack” them on top of each, we’re still only computing affine maps.
Suppose we have two affine maps \(f(x) = Ax + b\) and \(g(x) = Cx + d\). What is \(f(g(x))\)?
\[ f(g(x)) = A(Cx + d) + b = ACx + (Ad + b) \]
\(AC\) is a matrix and \(Ad + b\) is a vector, so we see that composing affine maps gives you an affine map.
10.5 Non-linearities, or activation functions
From this, you can see that if you wanted your neural network to be long chains of affine compositions, that this adds no new power to your model than just doing a single affine map.
There are a few core non-linearities. \(\tanh(x), \sigma(x), \text{ReLU}(x)\) are the most common. You are probably wondering: “why these functions? I can think of plenty of other non-linearities.” The reason for this is that they have gradients that are easy to compute, and computing gradients is essential for learning.
\[ \frac{d\sigma}{dx} = \sigma(x)(1 - \sigma(x)) \]
The well known sigmoid function has a particularly nice derivative and a nice interpretation as a saturating “activation” signal in neurons. But it is not very efficient in deep networks, because of the vanishing gradient problem.
Code
x = np.linspace(-10, 10, 100)
y = 1 / (1 + np.exp(-x))
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y, name='Sigmoid'))
fig.update_layout(title='Sigmoid Function', xaxis_title='x', yaxis_title='y')
fig.show()This is because the gradient vanishes very quickly as the absolute value of the argument grows. Small gradients means it is hard to learn. Most people default to tanh or ReLU.
Code
x = np.linspace(-10, 10, 100)
y = np.maximum(0, x)
fig = go.Figure()
fig.add_trace(go.Scatter(x=x, y=y, name='ReLU'))
fig.update_layout(title='Rectified Linear Unit (ReLU) Function', xaxis_title='x', yaxis_title='y')
fig.show()10.6 Build the network
We define our neural network by subclassing nn.Module, and initialize the neural network layers in __init__. Every nn.Module subclass implements the operations on input data in the forward method.
class NeuralNetwork(nn.Module):
def __init__(self):
super().__init__()
1 self.flatten = nn.Flatten()
2 self.linear_relu_stack = nn.Sequential(
3 nn.Linear(28*28, 512),
4 nn.ReLU(),
5 nn.Linear(512, 512),
nn.ReLU(),
6 nn.Linear(512, 10),
)
def forward(self, x):
7 x = self.flatten(x)
8 logits = self.linear_relu_stack(x)
return logits- 1
- Flatten just flattens the input tensor into a vector
- 2
- Sequential is a container for the layers
- 3
- 28*28 is the size of the input image
- 4
- ReLU is the activation function
- 5
- 512 is the size of the output of the previous layer
- 6
- 10 is the size of the output of the previous layer
- 7
- flatten the input
- 8
- logits is the output of the network before applying the softmax function
10.7 Instantiate the network
We create an instance of NeuralNetwork, and move it to the device, and print its structure.
model = NeuralNetwork().to("cuda")
print(model)NeuralNetwork(
(flatten): Flatten(start_dim=1, end_dim=-1)
(linear_relu_stack): Sequential(
(0): Linear(in_features=784, out_features=512, bias=True)
(1): ReLU()
(2): Linear(in_features=512, out_features=512, bias=True)
(3): ReLU()
(4): Linear(in_features=512, out_features=10, bias=True)
)
)
10.8 Better visualization with tensorboard
writer = SummaryWriter('torchlogs/')
writer.add_graph(model, torch.rand(64, 28*28).to("cuda"))
writer.close()and see it with the tensorboard on the dir torchlogs:


10.9 Model Layers
Let’s break down the layers in the FashionMNIST model. To illustrate it, we will take a sample minibatch of 3 images of size 28x28 and see what happens to it as we pass it through the network.
input_image = torch.rand(3,28,28)
print(input_image.size())torch.Size([3, 28, 28])
10.10 nn.Flatten
We initialize the nn.Flatten layer to convert each 2D 28x28 image into a contiguous array of 784 pixel values ( the minibatch dimension (at dim=0) is maintained).
flatten = nn.Flatten()
flat_image = flatten(input_image)
print(flat_image.size())torch.Size([3, 784])
10.11 nn.Linear
The linear layer is a module that applies a linear transformation on the input using its stored weights and biases.
layer1 = nn.Linear(in_features=28*28, out_features=20)
hidden1 = layer1(flat_image)
print(hidden1.size())torch.Size([3, 20])
10.11.1 nn.ReLU
In this model, we use nn.ReLU between our linear layers, but there’s other activations to introduce non-linearity in your model to introduce non linearity in our model.
print(f"Before ReLU: {hidden1}\n\n")
hidden1 = nn.ReLU()(hidden1)
print(f"After ReLU: {hidden1}")Before ReLU: tensor([[ 0.2991, 0.1975, 0.1709, -0.6827, 0.3192, 0.2041, -0.0016, 0.2832,
-0.3922, 0.2677, -0.1190, -0.3733, 0.2332, 0.0607, 0.7411, 0.5116,
0.2189, 0.3999, -0.2244, 0.3730],
[ 0.5172, -0.2357, 0.2192, -0.7395, 0.2954, 0.2465, 0.2205, 0.0700,
-0.0016, 0.5997, -0.2390, -0.4861, 0.8645, 0.3372, 0.9774, 0.5403,
0.3724, 0.3531, -0.0121, -0.0739],
[ 0.2557, 0.6357, 0.2762, -0.6375, 0.0539, 0.2871, 0.1220, 0.2155,
-0.3984, 0.4774, -0.1118, -0.1641, -0.0050, 0.3453, 0.9153, 0.5326,
0.5245, 0.1613, -0.0555, 0.1386]], grad_fn=<AddmmBackward0>)
After ReLU: tensor([[0.2991, 0.1975, 0.1709, 0.0000, 0.3192, 0.2041, 0.0000, 0.2832, 0.0000,
0.2677, 0.0000, 0.0000, 0.2332, 0.0607, 0.7411, 0.5116, 0.2189, 0.3999,
0.0000, 0.3730],
[0.5172, 0.0000, 0.2192, 0.0000, 0.2954, 0.2465, 0.2205, 0.0700, 0.0000,
0.5997, 0.0000, 0.0000, 0.8645, 0.3372, 0.9774, 0.5403, 0.3724, 0.3531,
0.0000, 0.0000],
[0.2557, 0.6357, 0.2762, 0.0000, 0.0539, 0.2871, 0.1220, 0.2155, 0.0000,
0.4774, 0.0000, 0.0000, 0.0000, 0.3453, 0.9153, 0.5326, 0.5245, 0.1613,
0.0000, 0.1386]], grad_fn=<ReluBackward0>)
10.12 nn.Sequential
nn.Sequential is an ordered container of modules. The data is passed through all the modules in the same order as defined. You can use sequential containers to put together a quick network like seq_modules.
seq_modules = nn.Sequential(
flatten,
layer1,
nn.ReLU(),
nn.Linear(20, 10)
)
input_image = torch.rand(3,28,28)
logits = seq_modules(input_image)10.13 nn.Softmax
The last linear layer of the neural network returns logits - raw values in \([-\infty, \infty]\) - which are passed to the nn.Softmax module. The logits are scaled to values \([0, 1]\) representing the model’s predicted probabilities for each class. dim parameter indicates the dimension along which the values must sum to 1.
softmax = nn.Softmax(dim=1)
pred_probab = softmax(logits)10.14 Model Parameters
Many layers inside a neural network are parameterized, i.e. have associated weights and biases that are optimized during training. Subclassing nn.Module automatically tracks all fields defined inside your model object, and makes all parameters accessible using your model’s parameters() or named_parameters() methods.
In this example, we iterate over each parameter, and print its size and a preview of its values.
print(f"Model structure: {model}\n\n")
for name, param in model.named_parameters():
print(f"Layer: {name} | Size: {param.size()} | Values : {param[:2]} \n")Model structure: NeuralNetwork(
(flatten): Flatten(start_dim=1, end_dim=-1)
(linear_relu_stack): Sequential(
(0): Linear(in_features=784, out_features=512, bias=True)
(1): ReLU()
(2): Linear(in_features=512, out_features=512, bias=True)
(3): ReLU()
(4): Linear(in_features=512, out_features=10, bias=True)
)
)
Layer: linear_relu_stack.0.weight | Size: torch.Size([512, 784]) | Values : tensor([[-0.0011, 0.0036, -0.0158, ..., 0.0299, -0.0061, 0.0115],
[-0.0295, -0.0215, -0.0321, ..., 0.0058, 0.0018, -0.0251]],
device='cuda:0', grad_fn=<SliceBackward0>)
Layer: linear_relu_stack.0.bias | Size: torch.Size([512]) | Values : tensor([-0.0228, -0.0109], device='cuda:0', grad_fn=<SliceBackward0>)
Layer: linear_relu_stack.2.weight | Size: torch.Size([512, 512]) | Values : tensor([[-0.0008, 0.0102, 0.0376, ..., -0.0059, 0.0174, -0.0057],
[ 0.0354, -0.0045, 0.0153, ..., 0.0393, -0.0301, 0.0434]],
device='cuda:0', grad_fn=<SliceBackward0>)
Layer: linear_relu_stack.2.bias | Size: torch.Size([512]) | Values : tensor([-0.0332, 0.0050], device='cuda:0', grad_fn=<SliceBackward0>)
Layer: linear_relu_stack.4.weight | Size: torch.Size([10, 512]) | Values : tensor([[-0.0271, -0.0068, -0.0369, ..., 0.0203, -0.0393, 0.0267],
[ 0.0225, 0.0083, -0.0204, ..., 0.0135, -0.0117, 0.0170]],
device='cuda:0', grad_fn=<SliceBackward0>)
Layer: linear_relu_stack.4.bias | Size: torch.Size([10]) | Values : tensor([-0.0399, 0.0277], device='cuda:0', grad_fn=<SliceBackward0>)
10.15 Saving and loading model weights
PyTorch models store the learned parameters in an internal state dictionary, called state_dict. These can be persisted via the torch.save method:
torch.save(model.state_dict(), 'model_weights.pth')To load model weights, you need to create an instance of the same model first, and then load the parameters using load_state_dict() method.
model2 = NeuralNetwork()
model2.load_state_dict(torch.load('model_weights.pth'))<All keys matched successfully>
10.16 Loading the dataset
The torchvision.datasets module contains Dataset objects for many real-world vision data like CIFAR, COCO (full list here). In this tutorial, we use the FashionMNIST dataset. Every TorchVision Dataset includes two arguments: transform and target_transform to modify the samples and labels respectively.
# Download training data from open datasets.
training_data = datasets.FashionMNIST(
root="data",
train=True,
download=True,
transform=transforms.ToTensor(),
)# Download test data from open datasets.
test_data = datasets.FashionMNIST(
root="data",
train=False,
download=True,
transform=transforms.ToTensor(),
)We pass the Dataset as an argument to DataLoader. This wraps an iterable over our dataset, and supports automatic batching, sampling, shuffling and multiprocess data loading. Here we define a batch size of 64, i.e. each element in the dataloader iterable will return a batch of 64 features and labels.
batch_size = 64
# Create data loaders.
train_dataloader = DataLoader(training_data, batch_size=batch_size)
test_dataloader = DataLoader(test_data, batch_size=batch_size)
for X, y in test_dataloader:
print(f"Shape of X [N, C, H, W]: {X.shape}")
print(f"Shape of y: {y.shape} {y.dtype}")
breakShape of X [N, C, H, W]: torch.Size([64, 1, 28, 28])
Shape of y: torch.Size([64]) torch.int64
10.17 Setting up the loss function and optimizer
As the loss function, as we are in multiclass classification problem, we will use the cross entropy loss.
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)10.18 Exercise
Implement the training and testing functions.
def train(dataloader, model, loss_fn, optimizer):
size = len(dataloader.dataset)
model.train()
for batch, (X, y) in enumerate(dataloader):
X, y = X.to("cuda"), y.to("cuda")
# Compute prediction error
pred = ...
loss = ...
# Backpropagation
...
if batch % 100 == 0:
loss, current = loss.item(), (batch + 1) * len(X)
print(f"loss: {loss:>7f} [{current:>5d}/{size:>5d}]")def test(dataloader, model, loss_fn):
size = len(dataloader.dataset)
num_batches = len(dataloader)
model.eval()
test_loss, correct = 0, 0
with torch.no_grad():
for X, y in dataloader:
X, y = X.to("cuda"), y.to("cuda")
pred = ...
test_loss += ...
correct += (pred.argmax(1) == y).type(torch.float).sum().item()
test_loss /= num_batches
correct /= size
print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")10.19 Launch the training and testing
epochs = 5
for t in range(epochs):
print(f"Epoch {t+1}\n-------------------------------")
train(train_dataloader, model, loss_fn, optimizer)
test(test_dataloader, model, loss_fn)
print("Done!")Epoch 1
-------------------------------
loss: 2.313305 [ 64/60000]
loss: 2.302577 [ 6464/60000]
loss: 2.279602 [12864/60000]
loss: 2.269845 [19264/60000]
loss: 2.248892 [25664/60000]
loss: 2.214803 [32064/60000]
loss: 2.236776 [38464/60000]
loss: 2.194689 [44864/60000]
loss: 2.193618 [51264/60000]
loss: 2.169196 [57664/60000]
Test Error:
Accuracy: 36.0%, Avg loss: 2.161442
Epoch 2
-------------------------------
loss: 2.176624 [ 64/60000]
loss: 2.167854 [ 6464/60000]
loss: 2.116770 [12864/60000]
loss: 2.130357 [19264/60000]
loss: 2.072228 [25664/60000]
loss: 2.017969 [32064/60000]
loss: 2.051332 [38464/60000]
loss: 1.970719 [44864/60000]
loss: 1.973851 [51264/60000]
loss: 1.918556 [57664/60000]
Test Error:
Accuracy: 51.9%, Avg loss: 1.908704
Epoch 3
-------------------------------
loss: 1.940778 [ 64/60000]
loss: 1.915648 [ 6464/60000]
loss: 1.805761 [12864/60000]
loss: 1.847780 [19264/60000]
loss: 1.730446 [25664/60000]
loss: 1.680535 [32064/60000]
loss: 1.707684 [38464/60000]
loss: 1.604326 [44864/60000]
loss: 1.627919 [51264/60000]
loss: 1.536645 [57664/60000]
Test Error:
Accuracy: 59.1%, Avg loss: 1.544043
Epoch 4
-------------------------------
loss: 1.610138 [ 64/60000]
loss: 1.577930 [ 6464/60000]
loss: 1.428814 [12864/60000]
loss: 1.503949 [19264/60000]
loss: 1.380030 [25664/60000]
loss: 1.366920 [32064/60000]
loss: 1.389339 [38464/60000]
loss: 1.307619 [44864/60000]
loss: 1.340239 [51264/60000]
loss: 1.250525 [57664/60000]
Test Error:
Accuracy: 63.2%, Avg loss: 1.269593
Epoch 5
-------------------------------
loss: 1.349123 [ 64/60000]
loss: 1.330799 [ 6464/60000]
loss: 1.163287 [12864/60000]
loss: 1.273368 [19264/60000]
loss: 1.145072 [25664/60000]
loss: 1.160446 [32064/60000]
loss: 1.191180 [38464/60000]
loss: 1.121791 [44864/60000]
loss: 1.156709 [51264/60000]
loss: 1.080867 [57664/60000]
Test Error:
Accuracy: 64.7%, Avg loss: 1.096680
Done!
10.20 Displaying images in tensorboard
from torch.utils.tensorboard import SummaryWriter
from torchvision.utils import make_grid
writer = SummaryWriter('torchlogs/')
a_batch = next(iter(train_dataloader))[0]
writer.add_images('a batch of images',a_batch,dataformats='NCHW')
writer.close()10.21 List of labels
| Label | Description |
|---|---|
| 0 | T-shirt/top |
| 1 | Trouser |
| 2 | Pullover |
| 3 | Dress |
| 4 | Coat |
| 5 | Sandal |
| 6 | Shirt |
| 7 | Sneaker |
| 8 | Bag |
| 9 | Ankle boot |
labels = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']10.22 Getting confusion matrix
model.eval()
y_pred = model(test_data.data.float().to("cuda")).argmax(1).cpu()
cm = confusion_matrix(test_data.targets,y_pred)10.23 Plotting the confusion matrix
Code
def plot_confusion_matrix(cm, labels, title):
# cm : confusion matrix list(list)
# labels : name of the data list(str)
# title : title for the heatmap
data = go.Heatmap(z=cm, y=labels, x=labels)
annotations = []
for i, row in enumerate(cm):
for j, value in enumerate(row):
annotations.append(
{
"x": labels[j],
"y": labels[i],
"font": {"color": "white"},
"text": str(value),
"xref": "x1",
"yref": "y1",
"showarrow": False
}
)
layout = {
"title": title,
"xaxis": {"title": "Predicted value"},
"yaxis": {"title": "Real value"},
"annotations": annotations
}
fig = go.Figure(data=data, layout=layout)
fig.update_yaxes(autorange="reversed")
fig.update_xaxes(tickangle=-45)
return figplot_confusion_matrix(cm,labels,'Confusion matrix')10.24 Plotting the misclassified images
Code
import numpy as np
def get_samples(dataloader,class_id,n_samples=10):
imgs = np.zeros((10, 28, 28))
predicted = np.zeros(10)
model.eval()
s = 0
done = False
for X, y in dataloader:
for i in range(len(y)):
if done:
break
if y[i] == class_id:
imgs[s] = X[i]
predicted[s] = model(X[i].unsqueeze(0).to("cuda")).argmax(1)
s += 1
if s == 10:
done = True
return imgs,predicted10.25 Displaying 10 misclassified images on specific class
Code
import plotly.express as px
def plot_predclass(class_id):
imgs,predicted = get_samples(test_dataloader,class_id)
fig = px.imshow(imgs, facet_col=0, facet_col_wrap=5, binary_string=True)
for i, pred in enumerate(predicted):
fig.layout.annotations[i]['text'] = labels[int(pred)]
if int(pred) != class_id:
# Make the misclassified facet's background red
fig.layout.annotations[i]['font']['color'] = 'red'
fig.show()plot_predclass(labels.index('Ankle boot'))