Important links:
This is the starter notebook for the course project for Data Analysis with Python: Zero to Pandas. You will pick a real-world dataset of your choice and apply the concepts learned in this course to perform exploratory data analysis. Use this starter notebook as an outline for your project . Focus on documentation and presentation - this Jupyter notebook will also serve as a project report, so make sure to include detailed explanations wherever possible using Markdown cells.
Your submission will be evaluated using the following criteria:
Follow this step-by-step guide to work on your project.
opendatasets
Python libraryHere's some sample code for downloading the US Elections Dataset:
import opendatasets as od
dataset_url = 'https://www.kaggle.com/tunguz/us-elections-dataset'
od.download('https://www.kaggle.com/tunguz/us-elections-dataset')
You can find a list of recommended datasets here: https://jovian.ml/forum/t/recommended-datasets-for-course-project/11711
jovian.commit
.Refer to these projects for inspiration:
Analyzing your browser history using Pandas & Seaborn by Kartik Godawat
WhatsApp Chat Data Analysis by Prajwal Prashanth
Understanding the Gender Divide in Data Science Roles by Aakanksha N S
NOTE: Remove this cell containing the instructions before making your submission. You can do using the "Edit > Delete Cells" menu option.
TODO - Write some introduction about your project here: describe the dataset, where you got it from, what you're trying to do with it, and which tools & techniques you're using. You can also mention about the course Data Analysis with Python: Zero to Pandas, and what you've learned from it.
This is an executable Jupyter notebook hosted on Jovian.ml, a platform for sharing data science projects. You can run and experiment with the code in a couple of ways: using free online resources (recommended) or on your own computer.
The easiest way to start executing this notebook is to click the "Run" button at the top of this page, and select "Run on Binder". This will run the notebook on mybinder.org, a free online service for running Jupyter notebooks. You can also select "Run on Colab" or "Run on Kaggle".
Install Conda by following these instructions. Add Conda binaries to your system PATH
, so you can use the conda
command on your terminal.
Create a Conda environment and install the required libraries by running these commands on the terminal:
conda create -n zerotopandas -y python=3.8
conda activate zerotopandas
pip install jovian jupyter numpy pandas matplotlib seaborn opendatasets --upgrade
jovian clone notebook-owner/notebook-id
cd directory-name
and start the Jupyter notebook.jupyter notebook
You can now access Jupyter's web interface by clicking the link that shows up on the terminal or by visiting http://localhost:8888 on your browser. Click on the notebook file (it has a .ipynb
extension) to open it.
TODO - add some explanation here
Instructions for downloading the dataset (delete this cell)
- Find an interesting dataset on this page: https://www.kaggle.com/datasets?fileType=csv
- The data should be in CSV format, and should contain at least 3 columns and 150 rows
- Download the dataset using the
opendatasets
Python library
!pip install jovian opendatasets --upgrade --quiet
Let's begin by downloading the data, and listing the files within the dataset.
# Change this
dataset_url = 'https://www.kaggle.com/sudalairajkumar/covid19-in-india'
import opendatasets as od
od.download(dataset_url)
Please provide your Kaggle credentials to download this dataset. Learn more: http://bit.ly/kaggle-creds
Your Kaggle username: abhijeetraj22
Your Kaggle Key: ········
100%|██████████| 147k/147k [00:00<00:00, 33.3MB/s]
Downloading covid19-in-india.zip to ./covid19-in-india
The dataset has been downloaded and extracted.
# Change this
data_dir = './covid19-in-india'
import os
os.listdir(data_dir)
['covid_19_india.csv', 'StatewiseTestingDetails.csv']
Let us save and upload our work to Jovian before continuing.
project_name = "zerotopandas-course-project-covid19-in-india" # change this (use lowercase letters and hyphens only)
!pip install jovian --upgrade -q
import jovian
jovian.commit(project=project_name)
[jovian] Attempting to save notebook..
[jovian] Please enter your API key ( from https://jovian.ml/ ):
API KEY: ········
[jovian] Updating notebook "abhijeetraj22/zerotopandas-course-project-covid19-in-india" on https://jovian.ml/
[jovian] Uploading notebook..
[jovian] Capturing environment..
[jovian] Committed successfully! https://jovian.ml/abhijeetraj22/zerotopandas-course-project-covid19-in-india
TODO - Write some explanation here.
Instructions (delete this cell):
- Load the dataset into a data frame using Pandas
- Explore the number of rows & columns, ranges of values etc.
- Handle missing, incorrect and invalid data
- Perform any additional steps (parsing dates, creating additional columns, merging multiple dataset etc.)
import pandas as pd
import numpy as np
#import datetime
covid_india_df = pd.read_csv(data_dir + '/covid_19_india.csv')
covid_india_df
covid_india_df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 7086 entries, 0 to 7085
Data columns (total 9 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Sno 7086 non-null int64
1 Date 7086 non-null object
2 Time 7086 non-null object
3 State/UnionTerritory 7086 non-null object
4 ConfirmedIndianNational 7086 non-null object
5 ConfirmedForeignNational 7086 non-null object
6 Cured 7086 non-null int64
7 Deaths 7086 non-null int64
8 Confirmed 7086 non-null int64
dtypes: int64(4), object(5)
memory usage: 498.4+ KB
covid_india_df.describe()
#Drop Lat & Long
covid_india_df.drop(['ConfirmedIndianNational', 'ConfirmedForeignNational','Sno'], inplace=True, axis=1)
#Rename Cured to Recovered
covid_india_df.rename(columns = {'Cured':'Recovered'}, inplace = True)
#Rename State/UnionTerritory to Province_State
covid_india_df.rename(columns = {'State/UnionTerritory':'Province_State'}, inplace = True)
#Date format change
covid_india_df['Date'] = pd.to_datetime(covid_india_df['Date'],format='%d/%m/%y').dt.date
#Add New Cases
covid_india_df['Prev_Confirmed'] = covid_india_df.groupby('Province_State')['Confirmed'].shift(1)
covid_india_df['New Cases'] = covid_india_df['Confirmed'] - covid_india_df['Prev_Confirmed']
covid_india_df.drop('Prev_Confirmed',inplace = True,axis=1)
#Add New Recovered Cases
covid_india_df['Prev_Recovered'] = covid_india_df.groupby('Province_State')['Recovered'].shift(1)
covid_india_df['New Recovered'] = covid_india_df['Recovered'] - covid_india_df['Prev_Recovered']
covid_india_df.drop('Prev_Recovered',inplace = True,axis=1)
#Add New Deaths Cases
covid_india_df['Prev_Deaths'] = covid_india_df.groupby('Province_State')['Deaths'].shift(1)
covid_india_df['New Deaths'] = covid_india_df['Deaths'] - covid_india_df['Prev_Deaths']
covid_india_df.drop('Prev_Deaths',inplace = True,axis=1)
#check Null
covid_india_df.isnull().sum()
Date 0
Time 0
Province_State 0
Recovered 0
Deaths 0
Confirmed 0
New Cases 42
New Recovered 42
New Deaths 42
dtype: int64
#few Null data So, Remove it
covid_india_df['New Cases'].fillna(0, inplace=True)
covid_india_df['New Recovered'].fillna(0, inplace=True)
covid_india_df['New Deaths'].fillna(0, inplace=True)
#Type Check
covid_india_df['New Deaths'].dtypes
dtype('float64')
#Type Change into integer
covid_india_df['New Deaths'] = covid_india_df['New Deaths'].astype(int)
covid_india_df['New Cases'] = covid_india_df['New Cases'].astype(int)
covid_india_df['New Recovered'] = covid_india_df['New Recovered'].astype(int)
covid_india_df['Province_State'].unique()
array(['Kerala', 'Telengana', 'Delhi', 'Rajasthan', 'Uttar Pradesh',
'Haryana', 'Ladakh', 'Tamil Nadu', 'Karnataka', 'Maharashtra',
'Punjab', 'Jammu and Kashmir', 'Andhra Pradesh', 'Uttarakhand',
'Odisha', 'Puducherry', 'West Bengal', 'Chhattisgarh',
'Chandigarh', 'Gujarat', 'Himachal Pradesh', 'Madhya Pradesh',
'Bihar', 'Manipur', 'Mizoram', 'Andaman and Nicobar Islands',
'Goa', 'Unassigned', 'Assam', 'Jharkhand', 'Arunachal Pradesh',
'Tripura', 'Nagaland', 'Meghalaya', 'Dadar Nagar Haveli',
'Cases being reassigned to states', 'Sikkim', 'Daman & Diu',
'Dadra and Nagar Haveli and Daman and Diu', 'Telangana',
'Telangana***', 'Telengana***'], dtype=object)
#Remove non State data
#Drop 'Cases being reassigned to states' & 'Unassigned'
df_dump = covid_india_df['Province_State']=='Unassigned'
covid_india_df.drop(covid_india_df[df_dump].index,inplace=True)
df_dump = covid_india_df['Province_State']=='Cases being reassigned to states'
covid_india_df.drop(covid_india_df[df_dump].index,inplace=True)
def change_state_name(state):
if state == 'Odisha':
return 'Orissa'
elif state == 'Telengana':
return 'Telangana'
return state
covid_india_df['Province_State'] = covid_india_df.apply(lambda x: change_state_name(x['Province_State']), axis=1)
last_date = covid_india_df.Date.max()
state_cases = covid_india_df.copy()
state_cases = state_cases[state_cases['Date']==last_date]
state_cases.drop(['Date','Time', 'New Cases','New Recovered','New Deaths'],inplace = True,axis=1)
#Add Active, Deaths/Recovered, Mortality & Recovered Rate(per 100)
state_cases['Active'] = state_cases['Confirmed'] - (state_cases['Deaths']+state_cases['Recovered'])
state_cases['Active'] = state_cases['Active'].astype(int)
state_cases["Mortality Rate(per 100)"] = np.round(100*state_cases["Deaths"]/state_cases["Confirmed"],2)
state_cases["Recovered Rate(per 100)"] = np.round(100*state_cases["Recovered"]/state_cases["Confirmed"],2)
state_cases.reset_index(drop=True,inplace=True)
state_cases.head()
#Day wise Copy Data
covid_india_dayswise_df = covid_india_df.groupby('Date').sum().reset_index()
covid_india_dayswise_df.head()
#check Null
covid_india_dayswise_df.isnull().sum()
Date 0
Recovered 0
Deaths 0
Confirmed 0
New Cases 0
New Recovered 0
New Deaths 0
dtype: int64
#Add Deaths/Recovered, Mortality & Recovered Rate(per 100)
covid_india_df["Mortality Rate(per 100)"] = np.round(100*covid_india_df["Deaths"]/covid_india_df["Confirmed"],2)
covid_india_df["Recovered Rate(per 100)"] = np.round(100*covid_india_df["Recovered"]/covid_india_df["Confirmed"],2)
#check Null
covid_india_df.isnull().sum()
Date 0
Time 0
Province_State 0
Recovered 0
Deaths 0
Confirmed 0
New Cases 0
New Recovered 0
New Deaths 0
Mortality Rate(per 100) 7
Recovered Rate(per 100) 7
dtype: int64
#few Null data So, Remove it
covid_india_df['Mortality Rate(per 100)'] = covid_india_df['Mortality Rate(per 100)'].replace(np.nan, 0)
covid_india_df['Recovered Rate(per 100)'] = covid_india_df['Recovered Rate(per 100)'].replace(np.nan, 0)
#check Null
covid_india_df.isnull().sum()
Date 0
Time 0
Province_State 0
Recovered 0
Deaths 0
Confirmed 0
New Cases 0
New Recovered 0
New Deaths 0
Mortality Rate(per 100) 0
Recovered Rate(per 100) 0
dtype: int64
import jovian
jovian.commit()
[jovian] Attempting to save notebook..
[jovian] Updating notebook "abhijeetraj22/zerotopandas-course-project-covid19-in-india" on https://jovian.ml/
[jovian] Uploading notebook..
[jovian] Capturing environment..
[jovian] Committed successfully! https://jovian.ml/abhijeetraj22/zerotopandas-course-project-covid19-in-india
TODO - write some explanation here.
Instructions (delete this cell)
- Compute the mean, sum, range and other interesting statistics for numeric columns
- Explore distributions of numeric columns using histograms etc.
- Explore relationship between columns using scatter plots, bar charts etc.
- Make a note of interesting insights from the exploratory analysis
Install plotly-express
!pip install plotly-express
Requirement already satisfied: plotly-express in /srv/conda/envs/notebook/lib/python3.8/site-packages (0.4.1)
Requirement already satisfied: statsmodels>=0.9.0 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from plotly-express) (0.12.0)
Requirement already satisfied: numpy>=1.11 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from plotly-express) (1.19.1)
Requirement already satisfied: plotly>=4.1.0 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from plotly-express) (4.11.0)
Requirement already satisfied: patsy>=0.5 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from plotly-express) (0.5.1)
Requirement already satisfied: pandas>=0.20.0 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from plotly-express) (1.1.2)
Requirement already satisfied: scipy>=0.18 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from plotly-express) (1.5.2)
Requirement already satisfied: six in /srv/conda/envs/notebook/lib/python3.8/site-packages (from plotly>=4.1.0->plotly-express) (1.15.0)
Requirement already satisfied: retrying>=1.3.3 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from plotly>=4.1.0->plotly-express) (1.3.3)
Requirement already satisfied: pytz>=2017.2 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from pandas>=0.20.0->plotly-express) (2020.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /srv/conda/envs/notebook/lib/python3.8/site-packages (from pandas>=0.20.0->plotly-express) (2.8.1)
Let's begin by importingmatplotlib.pyplot
and seaborn
.
import seaborn as sns
import matplotlib
import matplotlib.pyplot as plt
import plotly_express as px
%matplotlib inline
sns.set_style('darkgrid')
matplotlib.rcParams['font.size'] = 14
matplotlib.rcParams['figure.figsize'] = (9, 5)
matplotlib.rcParams['figure.facecolor'] = '#00000000'
total_cases = state_cases.copy()
total_cases.drop(['Province_State','Mortality Rate(per 100)','Recovered Rate(per 100)'],inplace = True,axis=1)
total = total_cases.sum()
total.name = "Total"
df_t = pd.DataFrame(total,dtype=float).transpose()
df_t["Mortality Rate(per 100)"] = np.round(100*df_t["Deaths"]/df_t["Confirmed"],2)
df_t.style.background_gradient(cmap='Purples',axis=1).format("{:.2f}").format("{:.0f}",subset=["Confirmed","Deaths","Recovered","Active"])
india_confirmed = total.Confirmed
india_active = total.Active
india_recovered = total.Recovered
india_deaths = total.Deaths
labels = ['Active','Recovered','Deaths']
sizes = [india_active,india_recovered,india_deaths]
color= ['#66b3ff','green','red']
explode = []
for i in labels:
explode.append(0.02)
plt.figure(figsize= (15,5))
plt.pie(sizes, labels=labels, autopct='%1.1f%%', startangle=9, explode =explode,colors = color)
centre_circle = plt.Circle((0,0),0.30,fc='white')
fig = plt.gcf()
fig.gca().add_artist(centre_circle)
plt.title('India COVID-19 Cases',fontsize = 20)
plt.axis('equal')
plt.tight_layout()
state_cases_df = state_cases.copy()
state_cases_df = state_cases_df.set_index("Province_State")
state_cases_df.sort_values('Confirmed', ascending= False).fillna(0).style.background_gradient(cmap='Blues',subset=["Confirmed"])\
.background_gradient(cmap='Reds',subset=["Deaths"])\
.background_gradient(cmap='Greens',subset=["Recovered"])\
.background_gradient(cmap='Purples',subset=["Active"])\
.background_gradient(cmap='YlOrBr',subset=["Mortality Rate(per 100)"])\
.background_gradient(cmap='Greens_r',subset=["Recovered Rate(per 100)"])
fig, ax = plt.subplots(figsize=(15, 10))
ax.clear()
#fig = sns.lineplot(x="Date",y="Active",data = covid_india_dayswise_df ,color="y",label="Active")
fig = sns.lineplot(x="Date",y="Recovered",data = covid_india_dayswise_df ,color="green",label="Recovered")
fig = sns.lineplot(x="Date",y="Deaths",data = covid_india_dayswise_df ,color="r",label="Deaths")
fig = sns.lineplot(x="Date",y="Confirmed",data = covid_india_dayswise_df ,color="c",label="Confirmed")
fig.set_xlabel('Date\n',size=15,color='#4bb4f2')
fig.set_ylabel('Number of Cases\n',size=15,color='#4bb4f2')
fig.set_title('India Covid-19 Cases',size=25,color='navy')
fig.ticklabel_format(style='plain', axis='y',useOffset=False)
df1 = covid_india_dayswise_df.melt(id_vars='Date', value_vars=['New Cases','New Deaths','New Recovered'],
var_name='Cases', value_name='Cases Count')
fig = px.line(df1, x="Date", y="Cases Count",color='Cases')
fig.update_layout(title="India Covid-19 Daily Cases", xaxis_title="", yaxis_title="")
fig.show()
TODO - Explore one or more columns by plotting a graph below, and add some explanation about it
fig = px.line(covid_india_df,x='Date', y='Confirmed', color='Province_State',title='India growth COVID19 Cases ')
fig.show()
TODO - Explore one or more columns by plotting a graph below, and add some explanation about it
def plot_hbar(df, col, n, hover_data=[]):
fig = px.bar(df.sort_values(col).tail(n),
x=col, y="Province_State", color='Province_State',
text=col, orientation='h', width=700, hover_data=hover_data,
color_discrete_sequence = px.colors.qualitative.Dark24)
fig.update_layout(title=col, xaxis_title="", yaxis_title="",
yaxis_categoryorder = 'total ascending',
uniformtext_minsize=8, uniformtext_mode='hide')
fig.show()
plot_hbar(state_cases, 'Confirmed', 10)
plot_hbar(state_cases, 'Deaths', 10)
plot_hbar(state_cases, 'Recovered', 10)
plot_hbar(state_cases, 'Active', 10)
plot_hbar(state_cases, 'Mortality Rate(per 100)', 10)
plot_hbar(state_cases, 'Recovered Rate(per 100)', 10)
TODO - Explore one or more columns by plotting a graph below, and add some explanation about it
fig = px.scatter(state_cases.sort_values('Deaths', ascending=False).iloc[:20, :],
x='Confirmed', y='Deaths', color='Province_State', size='Confirmed',
height=700, text='Province_State', log_x=True, log_y=True,
title='Confirmed vs Deaths (Scale in log10)')
fig.update_traces(textposition='top center')
fig.update_layout(showlegend=False)
fig.update_layout(xaxis_rangeslider_visible=True)
fig.show()
Let us save and upload our work to Jovian before continuing
import jovian
jovian.commit()
[jovian] Attempting to save notebook..
TODO - write some explanation here.
Instructions (delete this cell)
- Ask at least 5 interesting questions about your dataset
- Answer the questions either by computing the results using Numpy/Pandas or by plotting graphs using Matplotlib/Seaborn
- Create new columns, merge multiple dataset and perform grouping/aggregation wherever necessary
- Wherever you're using a library function from Pandas/Numpy/Matplotlib etc. explain briefly what it does
print("The total number of active COVID-19 cases in India is {a}.".format(a = india_active))
The total number of active COVID-19 cases in India is 907883.
india_active
907883
print("A total of {a} COVID-19 confirmed cases have been reported in Indian \
as from {date} with {b} active cases ({ap:.1f}%),{c} cured/discharged ({rp:.1f}%),\
and {d} deaths ({dp:.1f}%).".format(a = india_confirmed,date = last_date.strftime("%d. %B %Y"),b = india_active,ap=(india_active/india_confirmed)*100 \
,c = india_recovered,rp=(india_recovered/india_confirmed)*100 \
,d = india_deaths,dp=(india_deaths/india_confirmed)*100))
Let us save and upload our work to Jovian before continuing.
import jovian
jovian.commit()
TODO - Write some explanation here: a summary of all the inferences drawn from the analysis, and any conclusions you may have drawn by answering various questions.
import jovian
jovian.commit()
TODO - Write some explanation here: ideas for future projects using this dataset, and links to resources you found useful.
Submission Instructions (delete this cell)
- Upload your notebook to your Jovian.ml profile using
jovian.commit
.- Make a submission here: https://jovian.ml/learn/data-analysis-with-python-zero-to-pandas/assignment/course-project
- Share your work on the forum: https://jovian.ml/forum/t/course-project-on-exploratory-data-analysis-discuss-and-share-your-work/11684
- Share your work on social media (Twitter, LinkedIn, Telegram etc.) and tag @JovianML
(Optional) Write a blog post
- A blog post is a great way to present and showcase your work.
- Sign up on Medium.com to write a blog post for your project.
- Copy over the explanations from your Jupyter notebook into your blog post, and embed code cells & outputs
- Check out the Jovian.ml Medium publication for inspiration: https://medium.com/jovianml
import jovian
jovian.commit()