Jovian
⭐️
Sign In

WHAT IS PYTORCH?

It’s a Python-based scientific computing package targeted at two sets of audiences:

A replacement for NumPy to use the power of GPUs

A deep learning research platform that provides maximum flexibility and speed

In [2]:
import torch
In [ ]:
x = torch.rand(5, 3)
print(x)
In [ ]:
x = torch.zeros(5, 3, dtype=torch.long)
In [ ]:
print(x)
In [ ]:
x = torch.rand(5, 3)
print(x)
In [ ]:
x = x.new_ones(5, 3, dtype=torch.double)      # new_* methods take in sizes
print(x)

x = torch.randn_like(x, dtype=torch.float)    # override dtype!
print(x)                                      # result has the same size
In [ ]:
y = torch.rand(5, 3)
print(x + y)
In [ ]:
print(torch.add(x, y))
In [ ]:
result = torch.empty(5, 3)
torch.add(x, y, out=result)
print(result)
In [ ]:
# adds x to y
y.add_(x)
print(y)
In [ ]:
# Any operation that mutates a tensor in-place is post-fixed with an _. For example: x.copy_(y), x.t_(), will change x.

# You can use standard NumPy-like indexing with all bells and whistles!

print(x[:, 1])
In [6]:
# See how changing the np array changed the Torch Tensor automatically

import numpy as np
import matplotlib as plt
a = np.ones(5)
b = torch.from_numpy(a)
np.add(a, 1, out=a)
print(a)
print(b)
[2. 2. 2. 2. 2.] tensor([2., 2., 2., 2., 2.], dtype=torch.float64)

NumPy Bridge

Converting a Torch Tensor to a NumPy array and vice versa is a breeze.

The Torch Tensor and NumPy array will share their underlying memory locations (if the Torch Tensor is on CPU), and changing one will change the other.

Converting a Torch Tensor to a NumPy Array

In [10]:
a = torch.ones(5)
print(a)
tensor([1., 1., 1., 1., 1.])
In [11]:
b = a.numpy()
print(b)
[1. 1. 1. 1. 1.]
In [12]:
# See how the numpy array changed in value.

a.add_(1)
print(a)
print(b)
tensor([2., 2., 2., 2., 2.]) [2. 2. 2. 2. 2.]
In [13]:
b = a.numpy()
print(b)
[2. 2. 2. 2. 2.]

CUDA Tensors

Tensors can be moved onto any device using the .to method.

In [21]:
# let us run this cell only if CUDA is available
# We will use ``torch.device`` objects to move tensors in and out of GPU

if torch.cuda.is_available():
    device = torch.device("cuda")          # a CUDA device object
    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU
    x = x.to(device)                       # or just use strings ``.to("cuda")``
    z = x + y
    print(z)
    print(z.to("cpu", torch.double))       # ``.to`` can also change dtype together!

AUTOGRAD: AUTOMATIC DIFFERENTIATION

Central to all neural networks in PyTorch is the autograd package. Let’s first briefly visit this, and we will then go to training our first neural network.

The autograd package provides automatic differentiation for all operations on Tensors. It is a define-by-run framework, which means that your backprop is defined by how your code is run, and that every single iteration can be different.

Let us see this in more simple terms with some examples.

Tensor torch.Tensor is the central class of the package. If you set its attribute .requires_grad as True, it starts to track all operations on it. When you finish your computation you can call .backward() and have all the gradients computed automatically. The gradient for this tensor will be accumulated into .grad attribute.

To stop a tensor from tracking history, you can call .detach() to detach it from the computation history, and to prevent future computation from being tracked.

To prevent tracking history (and using memory), you can also wrap the code block in with torch.no_grad():. This can be particularly helpful when evaluating a model because the model may have trainable parameters with requires_grad=True, but for which we don’t need the gradients.

There’s one more class which is very important for autograd implementation - a Function.

Tensor and Function are interconnected and build up an acyclic graph, that encodes a complete history of computation. Each tensor has a .grad_fn attribute that references a Function that has created the Tensor (except for Tensors created by the user - their grad_fn is None).

If you want to compute the derivatives, you can call .backward() on a Tensor. If Tensor is a scalar (i.e. it holds a one element data), you don’t need to specify any arguments to backward(), however if it has more elements, you need to specify a gradient argument that is a tensor of matching shape

In [22]:
x = torch.ones(2, 2, requires_grad=True)
print(x)
tensor([[1., 1.], [1., 1.]], requires_grad=True)
In [30]:
# Do a tensor operation:
y = x + 2
print(y)

tensor([[3., 3.], [3., 3.]], grad_fn=<AddBackward0>)
In [31]:
a = torch.randn(2, 2)
a = ((a * 3) / (a - 1))
print(a.requires_grad)
a.requires_grad_(True)
print(a.requires_grad)
b = (a * a).sum()
print(b.grad_fn)
False True <SumBackward0 object at 0x000001E7D53A0408>
In [32]:
out.backward()

--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-32-617965056ac0> in <module> ----> 1 out.backward() AttributeError: 'str' object has no attribute 'backward'
In [ ]:
 
In [ ]: