Learn data science and machine learning by building real-world projects on Jovian

PyTorch Introductory - Assignment 1

A brief walkthrough with Shripal ...

... to understand 5 of the functions that can be applied to tensors - torch.Tensor objects.

A short introduction about PyTorch and about the chosen functions.

  • function 1 - new_empty()
  • function 2 - add()
  • function 3 - sqrt()
  • function 4 - bernoulli()
  • function 5 - sin()
In [1]:
# Import torch and other required modules
import torch

Function 1 - torch.Tensor.new_empty()

  • returns a new empty Tensor, with specified size (from arguments) with uninitialized elements and attributes (torch.dtype and torch.device) same as that of the mentioned tensor.
In [2]:
# Example 1 - working

t1 = torch.zeros(())
print("t1 =", t1, "\nt1.dtype = ", t1.dtype, ";   t1.device = ", t1.device)
print()

t2 = t1.new_empty((2,4))
print("t1 =", t1)
print("t2 =", t2, "\nt2.dtype = ", t2.dtype, ";   t2.device = ", t2.device)

t1 = tensor(0.) t1.dtype = torch.float32 ; t1.device = cpu t1 = tensor(0.) t2 = tensor([[7.5510e-34, 4.5623e-41, 7.5510e-34, 4.5623e-41], [4.4842e-44, 0.0000e+00, 8.9683e-44, 0.0000e+00]]) t2.dtype = torch.float32 ; t2.device = cpu
Explanation about example:

By default, the dtype of t1 is torch.float32 so, dtype of t2 is also torch.float32 since it is not explicitly mentioned to something else.

Same is for device = cpu

In [3]:
# Example 2 - working

t3 = t1.new_empty((2,3), dtype=torch.int8)
print("t3 =", t3, "\nt3.dtype = ", t3.dtype, ";   t3.device = ", t3.device)
print("t1 =", t1, "\nt1.dtype = ", t1.dtype, ";   t1.device = ", t1.device)
t3 = tensor([[ 112, 118, -110], [ 70, 101, 85]], dtype=torch.int8) t3.dtype = torch.int8 ; t3.device = cpu t1 = tensor(0.) t1.dtype = torch.float32 ; t1.device = cpu
Explanation about example:

When explicitly mentioned, the dtype of the returned t3 is set to torch.int8
While, that of t1 is still torch.float32

In [4]:
# Example 3 - breaking - not working when notebook is run on CPU only mode, without any accelerator/GPU.

t4 = t1.new_empty((2,5), dtype=torch.int8, device=torch.device("cuda:0"))
print("t3 =", t3, "\nt3.dtype = ", t3.dtype, ";   t3.device = ", t3.device)
print("t1 =", t1, "\nt1.dtype = ", t1.dtype, ";   t1.device = ", t1.device)
--------------------------------------------------------------------------- AssertionError Traceback (most recent call last) <ipython-input-4-1274903458ea> in <module> 1 # Example 3 - breaking - not working when notebook is run on CPU only mode, without any accelerator/GPU. 2 ----> 3 t4 = t1.new_empty((2,5), dtype=torch.int8, device=torch.device("cuda:0")) 4 print("t3 =", t3, "\nt3.dtype = ", t3.dtype, "; t3.device = ", t3.device) 5 print("t1 =", t1, "\nt1.dtype = ", t1.dtype, "; t1.device = ", t1.device) /srv/conda/envs/notebook/lib/python3.7/site-packages/torch/cuda/__init__.py in _lazy_init() 147 raise RuntimeError( 148 "Cannot re-initialize CUDA in forked subprocess. " + msg) --> 149 _check_driver() 150 if _cudart is None: 151 raise AssertionError( /srv/conda/envs/notebook/lib/python3.7/site-packages/torch/cuda/__init__.py in _check_driver() 52 Found no NVIDIA driver on your system. Please check that you 53 have an NVIDIA GPU and installed a driver from ---> 54 http://www.nvidia.com/Download/index.aspx""") 55 else: 56 # TODO: directly link to the alternative bin that needs install AssertionError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx
Explanation about example:

Providing the keyword argument device = torch.device("cuda:0") throws a RunTimeError since there is hardware incompatibility.

Function 2 - torch.add()

  • it adds the tensor with another tensor or a scalar object
  • it is required that the provided (adder) tensor be of the same size as that of the (addend) tensor,
    or, that it can be broadcasted to be the same as the size of the (addend) tensor.
In [5]:
# Example 1 - working

t5 = torch.ones((2,3))
print(t5)
print()

t6 = torch.add(t5, 5)
print(t6)
tensor([[1., 1., 1.], [1., 1., 1.]]) tensor([[6., 6., 6.], [6., 6., 6.]])
Explanation about example:
  • Above example demonstrates how to add a Python scalar to a torch.Tensor
In [6]:
# Example 2 - working

t7 = torch.ones((2,3), dtype=torch.int)
print(t7)
print()

t8 = torch.tensor([[8.99, 8.99, 8.99]], dtype=torch.float64)
print("t7.shape =", t7.shape)
print("t8.shape =", t8.shape)
print()

t9 = t7.add(t8)
print(t9)
tensor([[1, 1, 1], [1, 1, 1]], dtype=torch.int32) t7.shape = torch.Size([2, 3]) t8.shape = torch.Size([1, 3]) tensor([[9.9900, 9.9900, 9.9900], [9.9900, 9.9900, 9.9900]], dtype=torch.float64)
Explanation about example:

Above example shows that

  • the torch.Size of t7 and t8 tensors are different, but it was automatically broadcasted to match each other and then added to form t9 tensor.
  • also, the dtype of t7 is torch.int but it was automatically typecasted to torch.float64 since t8 has the higher dtype; and t9 gets the dtype = torch.float64
In [7]:
# Example 3 - breaking - when shape doesn't match & can't be matched even using broadcasting

t10 = t7.add(torch.tensor([1, 2, 3, 4]))
--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-7-482ebf4c5394> in <module> 1 # Example 3 - breaking - when shape doesn't match & can't be matched even using broadcasting 2 ----> 3 t10 = t7.add(torch.tensor([1, 2, 3, 4])) RuntimeError: The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 1
Explanation about example:
  • The size of the tensors don't match! Moreover, they are not even broadcast-able to match.
    Thus, it failed.

Function 3 - torch.sqrt()

  • returns a tensor with the same size as the original.
  • but, all the elements are the square roots of those in the original tensor.
In [8]:
# Example 1 - working

t11 = torch.tensor([[100., 64],
                    [   9, 49]])
t12 = t11.sqrt()
t12
Out[8]:
tensor([[10.,  8.],
        [ 3.,  7.]])
Explanation about example:
  • the function returns a tensor with the square roots of all the elements in the t11 tensor
In [9]:
# Example 2 - working

t13 = torch.randn(2,3)
print(t13, "\n")
t14 = torch.sqrt(t13)
print(t14)
tensor([[-0.3703, -0.5108, -1.0514], [-0.4055, -0.5981, 1.7822]]) tensor([[ nan, nan, nan], [ nan, nan, 1.3350]])
Explanation about example:
  • the square root of a negative number is a complex mathematical entity.
    Hence, it successfully returns a tensor with nan's wherever a real square root is not possible.
In [10]:
# Example 3 - breaking - sqrt() doesn't work for tensors with dtype=torch.int

t15 = torch.ones((2,2), dtype=torch.int)
t16 = torch.sqrt(t15)
--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-10-b80610a36a8a> in <module> 2 3 t15 = torch.ones((2,2), dtype=torch.int) ----> 4 t16 = torch.sqrt(t15) RuntimeError: sqrt_vml_cpu not implemented for 'Int'
Explanation about example:
  • torch.sqrt() doesn't work for Short, Int, Long dtype tensors

Function 4 - torch.bernoulli()

  • Draws binary random numbers (0 or 1) from a Bernoulli distribution.
  • The i-th element of the output tensor will draw a value 1 according to the i-th probability value given in input.
output_i ∼ Bernoulli( p = input_i )
In [11]:
# Example 1 - working

t17 = torch.empty(2, 3).uniform_(0, 1)  # generates a uniform random matrix with range [0, 1]
print(t17)

t18 = torch.bernoulli(t17)
print(t18)
tensor([[0.1917, 0.8109, 0.3409], [0.8578, 0.1723, 0.5195]]) tensor([[0., 1., 0.], [1., 0., 1.]])
Explanation about example:
  • the input tensor t17 provides input values for the Bernoulli() function.
  • then generates the output tensor t18 with output values (1 or 0) based on the corresponding input probability of generating a 1 (otherwise 0)
In [12]:
# Example 2 - working

t19 = torch.tensor([[0.1, 0.8, 0.9], [0.7, 0.2, 0.5]])

t20 = torch.bernoulli(input=t19, out=torch.rand(2,2))
print(t20)

tensor([[1., 1., 1.], [0., 0., 0.]])
Explanation about example:

As per the pytorch documentation, torch.bernoulli() can also be run using the above method.
is same as before, but takes another tensor and is used to compute the bernoulli function on the input.

In [13]:
# Example 3 - breaking - the input tensor should always contain elements ranging from 0 through 1.

t21 = torch.randn(2,3)
print(t21)

t22 = torch.bernoulli(t21)
tensor([[-0.5437, -1.5096, 0.5433], [ 0.2914, -1.5694, -0.2815]])
--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-13-f6fbe500ac77> in <module> 4 print(t21) 5 ----> 6 t22 = torch.bernoulli(t21) RuntimeError: Expected p_in >= 0 && p_in <= 1 to be true, but got false. (Could this error message be improved? If so, please report an enhancement request to PyTorch.)
Explanation about example:
  • torch.bernoulli() function takes input tensor as a probability of getting a 1 (otherwise 0) and hence, input tensor elements should always be a number ranging from 0 through 1.

Additional Comments:

I checked the documentation as well as tried to go through a few other references to find a better understanding of the poorly documented arguments - * and generator in the function - torch.bernoulli()

It'd be great if you can help me understand that better!

Here is the official github issues page related to this: https://github.com/pytorch/pytorch/issues/21567

Function 5 - torch.sin()

  • returns a tensor that contains elementwise (trigonometric) sine values of the input tensor.
In [14]:
!pip install matplotlib
Requirement already satisfied: matplotlib in /srv/conda/envs/notebook/lib/python3.7/site-packages (3.2.1) Requirement already satisfied: cycler>=0.10 in /srv/conda/envs/notebook/lib/python3.7/site-packages (from matplotlib) (0.10.0) Requirement already satisfied: python-dateutil>=2.1 in /srv/conda/envs/notebook/lib/python3.7/site-packages (from matplotlib) (2.8.1) Requirement already satisfied: kiwisolver>=1.0.1 in /srv/conda/envs/notebook/lib/python3.7/site-packages (from matplotlib) (1.2.0) Requirement already satisfied: numpy>=1.11 in /srv/conda/envs/notebook/lib/python3.7/site-packages (from matplotlib) (1.18.1) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /srv/conda/envs/notebook/lib/python3.7/site-packages (from matplotlib) (2.4.7) Requirement already satisfied: six in /srv/conda/envs/notebook/lib/python3.7/site-packages (from cycler>=0.10->matplotlib) (1.14.0)
In [15]:
import numpy as np
pi = np.pi

import matplotlib.pyplot as plt
%matplotlib inline
In [16]:
# Example 1 - working

t23 = torch.tensor([[0   , pi    , 2*pi], 
                    [pi/2, 3*pi/2, pi/6]])
print(t23)
print()

t24 = t23.sin()
print(t24)
print()

t25 = t24.round()
print("Rounding off t24 for simplicity:\n", t25)
tensor([[0.0000, 3.1416, 6.2832], [1.5708, 4.7124, 0.5236]]) tensor([[ 0.0000e+00, -8.7423e-08, 1.7485e-07], [ 1.0000e+00, -1.0000e+00, 5.0000e-01]]) Rounding off t24 for simplicity: tensor([[ 0., -0., 0.], [ 1., -1., 0.]])

Explanation about example

In [17]:
# Example 2 - working

np26 = np.arange(-pi, pi+pi/6, pi/6)
t26 = torch.tensor(np26)
print(t26)
print()

t27 = t26.sin()
print(t27)

plt.bar(t26, t27, color='purple')
tensor([-3.1416e+00, -2.6180e+00, -2.0944e+00, -1.5708e+00, -1.0472e+00, -5.2360e-01, -8.8818e-16, 5.2360e-01, 1.0472e+00, 1.5708e+00, 2.0944e+00, 2.6180e+00, 3.1416e+00], dtype=torch.float64) tensor([-1.2246e-16, -5.0000e-01, -8.6603e-01, -1.0000e+00, -8.6603e-01, -5.0000e-01, -8.8818e-16, 5.0000e-01, 8.6603e-01, 1.0000e+00, 8.6603e-01, 5.0000e-01, 1.8988e-15], dtype=torch.float64)
Out[17]:
<BarContainer object of 13 artists>
Notebook Image
Explanation about example:
  • in the above demonstration, first we got a range of equidistant elements from to π with a step-size = π/6
  • then we created a tensor from that numpy.ndarray.
  • then, applied function torch.sin() on that tensor.
  • then we passed both the tensors into a matplotlib.pyplot function as arguments to plot a graph of sine curve.
In [18]:
# Example 3 - breaking - torch.sin() function doesn't apply on tensors with dtypes other than torch.float32 or torch.float64

t29 = torch.zeros((2,3), dtype=torch.int)
t30 = torch.sin(t29)
--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-18-7f866c15b079> in <module> 2 3 t29 = torch.zeros((2,3), dtype=torch.int) ----> 4 t30 = torch.sin(t29) RuntimeError: sin_vml_cpu not implemented for 'Int'
Explanation about example:
  • torch.sin() function doesn't apply on tensors with dtypes other than torch.float32 or torch.float64

Conclusion

To summarize, in this notebook, we covered a brief description and a few examples of executing 5 selected functions on torch.Tensor object.

the functions selected and discussed in this notebook are:

  • new_empty()
  • torch.add()
  • torch.sqrt()
  • torch.bernoulli()
  • torch.sin()

Reference Links

Provide links to your references and other interesting articles about tensors

In [19]:
!pip install jovian --upgrade --quiet
In [20]:
import jovian
In [ ]:
jovian.commit()
[jovian] Attempting to save notebook..
In [ ]: