Jovian
⭐️
Sign In

Project Whatsapp Message analysis

Write some introduction about your project here: describe the dataset, where you got it from, what you're trying to do with it, and which tools & techniques you're using. You can also mention about the course, and what you've learned from it.

As a first step, let's upload our Jupyter notebook to Jovian.ml.

In [3]:
!pip install jovian --upgrade --quiet
!pip install numpy --upgrade --quiet
!pip install pandas --upgrade --quiet
!pip install matplotlib --upgrade --quiet
!pip install seaborn --upgrade --quiet
!pip install wordcloud --upgrade --quiet
!pip install emoji  --upgrade --quiet

In [4]:
project_name = "whatsapp-chat-analysis-course-project-try"

In [5]:
import jovian
In [4]:
jovian.commit(project=project_name,enviroment=None,files=["Chat.txt"])
[jovian] Attempting to save notebook.. [jovian] Please enter your API key ( from https://jovian.ml/ ): API KEY: ········ [jovian] Updating notebook "edsenmichaelcy/whatsapp-chat-analysis-course-project-try" on https://jovian.ml/ [jovian] Uploading notebook.. [jovian] Capturing environment.. [jovian] Uploading additional files... [jovian] Committed successfully! https://jovian.ml/edsenmichaelcy/whatsapp-chat-analysis-course-project-try

Data Preparation and Cleaning

In [28]:
import os
import pandas as pd
import re
import datetime as time
import jovian
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import emoji
import re
In [7]:
whatsapp_df = pd.read_fwf('Chat.txt', header = None)

whatsapp_df
Out[7]:
In [8]:
whatsapp_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 23330 entries, 0 to 23329 Data columns (total 3 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 0 23177 non-null object 1 1 23087 non-null object 2 2 788 non-null object dtypes: object(3) memory usage: 546.9+ KB

After that we will use the info() that provided by the pandas to understand the datatype in the dataframe. As you can see we need to do some clearning such as the date and the Media omitted. (re-explain)

In [9]:
whatsapp_df.shape
Out[9]:
(23330, 3)
In [10]:
def txtTodf(txt_file):
    '''Convert WhatsApp chat log text file to a Pandas dataframe.'''
    
    # some regex to account for messages taking up multiple lines
    pat = re.compile(r'^(\d\d\/\d\d\/\d\d\d\d.*?)(?=^^\d\d\/\d\d\/\d\d\d\d|\Z)', re.S | re.M)
    with open(txt_file) as file:
        data = [m.group(1).strip().replace('\n', ' ') for m in pat.finditer(file.read())]

    user     = []; 
    message  = []; 
    datetime = []
    
    for row in data:

        # timestamp is before the first dash
        datetime.append(row.split(' - ')[0])

        # sender is between am/pm, dash and colon
        try:
            s = re.search('m - (.*?):', row).group(1)
            user.append(s)
        except:
            user.append('')

        # message content is after the first colon
        try:
            message.append(row.split(': ', 1)[1])
        except:
            message.append('')

    df = pd.DataFrame(zip(datetime, user, message), columns=['datetime', 'user', 'message'])
    df['datetime'] = pd.to_datetime(df.datetime, format='%d/%m/%Y, %I:%M %p')

    # remove events not associated with a sender
    df = df[df.user != ''].reset_index(drop=True)
    
    return df

whatsapp_df = txtTodf('Chat.txt')
In [11]:
whatsapp_df.head(10)
Out[11]:
In [12]:
whatsapp_df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 22701 entries, 0 to 22700 Data columns (total 3 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 datetime 22701 non-null datetime64[ns] 1 user 22701 non-null object 2 message 22701 non-null object dtypes: datetime64[ns](1), object(2) memory usage: 532.2+ KB
In [25]:
jovian.commit(project=project_name)
[jovian] Attempting to save notebook.. [jovian] Updating notebook "edsenmichaelcy/whatsapp-chat-analysis-course-project-try" on https://jovian.ml/ [jovian] Uploading notebook.. [jovian] Capturing environment.. [jovian] Committed successfully! https://jovian.ml/edsenmichaelcy/whatsapp-chat-analysis-course-project-try

Clearning the image data

In [13]:
# To understand the number od the image data
img = whatsapp_df[whatsapp_df['message'] == "<Media omitted>" ]
img.shape

Out[13]:
(1182, 3)

Since we want to do analysis on the text rather than image so we have to clean the image data in the text file. In here we have 11k of image in the three row

In [14]:
# We will drop all the image file by using the Drop functions
whatsapp_df.drop(img.index, inplace=True)

So now we will drop all the img to make the dataset more clean. Moreover, we want to make sure it will not copy a new dataset that why we will use "inplace == True"

In [15]:
whatsapp_df.head(10)
Out[15]:

As you can see now the dataset is clean from the media format.But we have a problem because after we did the clearning the index of the dataset had been off-order. So now we have to clean the data by using the reset_index().

In [16]:
whatsapp_df.reset_index(inplace=True, drop=True)
whatsapp_df.shape
Out[16]:
(21519, 3)

So after the data is clean we have left 21519 data in our dataset. So now we are able to perform the data driven decision making!

Let get started on the data we are going analysis

  1. Which users have the most Chat/messages in the group?
  2. Which emojis use the most by which users?
  3. The most usage of whatsapp during the time and day?
  4. Which month have the highest messages and also the busiest month?
  5. What time did the users usually start chatting and sleep?
In [ ]:
jovian.commit(project=project_name)
[jovian] Attempting to save notebook..

1. Which users have the most Chat/messages in the group?

In [17]:
#Understand how many user and messages in this chat first
totalNumberofMessage = whatsapp_df.message.count()
username   = whatsapp_df["user"].unique()

print('The total of the number of message:',totalNumberofMessage)
print('User name that involve in the chat:',username)
The total of the number of message: 21519 User name that involve in the chat: ['Ed' 'Rohit' 'Pei Yin']
In [21]:
userNumber = whatsapp_df.user.unique()
print("The total number of message from each of the users:\n")
for i in range(len(userNumber)):
    #Get one particular user name
    user_df = whatsapp_df[whatsapp_df['user'] == userNumber[i]]
    
    #user_df will show the user message 
    name = print(f'User name: {userNumber[i]}')
    
    #Get the total number of each user send
    messages = print('Messages', user_df.shape[0])
    
    print()
The total number of message from each of the users: User name: Ed Messages 6991 User name: Rohit Messages 10268 User name: Pei Yin Messages 4260
In [23]:
whatsapp_df.head(5)
Out[23]:
In [40]:
whatsapp_df1 = whatsapp_df.copy()
whatsapp_df1['Number_of_messages'] = [1]* whatsapp_df1.shape[0]
whatsapp_df1.drop(columns = 'datetime', inplace = True)
whatsapp_df1 = whatsapp_df1.groupby('user')['message'].count().sort_values(ascending = False).reset_index() 
whatsapp_df1
Out[40]:
In [69]:
#Formating
sns.set_style("darkgrid")
plt.rcParams['font.size'] = 15
plt.rcParams['figure.figsize'] = (12, 8)

#Creating a bar chart
pal = sns.color_palette("Greens_d", len(whatsapp_df1))
rank = whatsapp_df1["message"].argsort().argsort() 
barplot = sns.barplot(x = whatsapp_df1.user,y = whatsapp_df1.message,data=whatsapp_df1,hue="user",palette=np.array(pal[::-1])[rank],dodge=False)

for index, row in whatsapp_df1.iterrows():
    barplot(row.user, row.message , round(row.rank))

plt.show()
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-69-868919f60e86> in <module> 10 11 for index, row in whatsapp_df1.iterrows(): ---> 12 barplot(row.user, row.message , round(row.rank)) 13 14 plt.show() TypeError: type method doesn't define __round__ method
In [ ]:
jovian.commit(project=project_name,enviroment=None,files=["Chat.txt"])




[jovian] Attempting to save notebook..
In [ ]:
# Improving Default Styles using Seaborn
sns.set_style("darkgrid")

# For better readablity;
import matplotlib
matplotlib.rcParams['font.size'] = 10
matplotlib.rcParams['figure.figsize'] = (12, 8)

# A bar plot for top 10 days
sns.barplot(top10days.date, top10days.message_count, palette="hls");

# Saving the plots
plt.savefig('top10_days.svg', format = 'svg')
In [ ]:
 
In [25]:
  

Out[25]:
user
Ed          6991
Pei Yin     4260
Rohit      10268
Name: message, dtype: int64
In [ ]:
 
In [ ]:
 
In [ ]: