Jovian
⭐️
Sign In

2 years ago
NotebookFilesCollaborators
Version 2

Managing Data Science Projects with Jovian

Jovian is a platform that helps data scientists and ML engineers

  • track & reproduce data science projects
  • collaborate easily with friends/colleagues, and
  • automate repetitive tasks in their day-to-day workflow.

Uploading your work to Jovian

It's really easy to get started with Jovian!

To follow along with this tutorial, click the 'Run' button above or Click Here to start a Jupyter notebook server (hosted by mybinder.org)

Step 1: Install the jovian python library

You can do this from the terminal, or directly within a Jupyter notebook.

In [1]:
!pip install jovian -q --upgrade

Step 2: Import the library

In [2]:
import jovian

Step 3: Run jovian.commit

After writing some code, running some experiments, training some models and plotting some charts, you can save and commit your Jupyter notebook.

In [3]:
jovian.commit()
[jovian] Saving notebook..
[jovian] Creating a new notebook on https://jvn.io/ [jovian] Please enter your API key ( from https://jvn.io/ ): API Key:········
[jovian] Error: The current API key is invalid or expired.
[jovian] Please enter your API key ( from https://jvn.io/ ): API Key:········ [jovian] Uploading notebook.. [jovian] Capturing environment.. [jovian] Committed successfully! https://jvn.io/liqia0qia0/100d7e86757b43f0a89f8944a19d249a

Here's what jovian.commit does:

  • It saves and uploads the Jupyter notebook to your Jovian account.
  • It captures and uploads the python virtual environment containing the list of libraries required to run your notebook.
  • It returns a link that you can use to view and share your notebook with friends or colleagues.

NOTE: When you run jovian.commit for the first time, you'll be asked to provide an API, which you can find on your Jovian account.

Reproducing uploaded notebooks

Once a notebook is uploaded to Jovian, anyone (including you) can download the notebook and it's Python dependencies by running jovian clone <notebook_id> command on the Linux/Mac terminal or Windows Command Prompt. Try clicking the 'Clone' button at the top of this page to copy the command (including notebook ID) to clipboard.

pip instal jovian --upgrade
jovian clone 903a04b17036436b843d70443ef5d7ad

Once cloned, you can enter the directly and setup the virtual environment using jovian install.

cd jovian-demo
jovian install

Jovian uses conda internally, so make sure you have it installed before running the above commands. Once the libraries are installed, you can activate the environment and start Jupyter in the usual way:

conda activate jovian-demo
jupyter notebook

In this way, Jovian seamlessly ensures the end-to-end reproducibility of your Jupyter notebooks.

Updating existing notebooks

Updating existing notebooks is really easy too! Just run jovian.commit once again, and Jovian will automatically identify and update the current notebook on your Jovian account.

In [ ]:
# Updating the notebook
jovian.commit()

Jovian keeps track of existing notebooks using a .jovianrc file next to your notebook. If you don't want to update the current notebook, but create a new notebook instead, simply delete the .jovianrc file. Note that if you rename your notebook, Jovian will upload a new notebooko when you commit, instead of updating the old one.

If you run into issues with updating a notebook, or want to replace a notebook in your account using a new/renamed notebook, you can provide the notebook_id argument to jovian.commit.

In [ ]:
jovian.commit(notebook_id="903a04b17036436b843d70443ef5d7ad")

Getting new changes on cloned notebooks

Once a notebook has been updated, the new changes can be retrieved at any cloned location using the jovian pull command.

cd jovian-demo # Enter cloned directory
jovian pull    # Pull the latest changes

Uploading additional files and artifacts

You can also include additional files like Python scripts and output files while committing a notebook to Jovian, using the files and artifacts arguments.

  • Use files to include python scripts, input CSVs and anything else you need to execute your notebook
  • Use artifacts to include the outputs of your notebooks (trained models, output images, CSVs etc.)

Let's look at an example. I'm going to use a function called sigmoid imported from a Python script called utils.py.

In [3]:
import numpy as np
from utils import sigmoid

inputs = np.array([1, 2, 3, 4, 5, 6, 7, 8])

outputs = sigmoid(inputs)
print(outputs)

np.savetxt("outputs.csv", outputs, delimiter=",")
[0.73105858 0.88079708 0.95257413 0.98201379 0.99330715 0.99752738 0.99908895 0.99966465]
In [4]:
!cat utils.py
import numpy as np #sigmoid = lambda x: 1 / (1 + np.exp(-x)) def sigmoid(x): return (1 / (1 + np.exp(-x)))
In [5]:
!cat outputs.csv
7.310585786300048960e-01 8.807970779778823145e-01 9.525741268224333647e-01 9.820137900379084517e-01 9.933071490757152677e-01 9.975273768433653432e-01 9.990889488055993972e-01 9.996646498695336280e-01
In [ ]:
jovian.commit(files=['utils.py'], artifacts=['outputs.csv'])
[jovian] Saving notebook..
In [ ]: