Jovian
⭐️
Sign In
In [1]:
import re
import jovian
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from wordcloud import WordCloud, STOPWORDS
import emoji
from collections import Counter
In [ ]:
## Data Preprocessing

* Regex cheatsheet
    * https://www.rexegg.com/regex-quickstart.html
* Datetime format
    * http://strftime.org/
    
```
def rawToDf(file):
    with open(file, 'r') as raw_data:
        raw_string = ' '.join(raw_data.read().split('\n')) # converting the list split by newline char. as one whole string as there can be multi-line messages
        user_msg = re.split('\d{1,2}/\d{1,2}/\d{2,4},\s\d{1,2}:\d{2}\s[APap][mM]\s-\s', raw_string) [1:] # splits at all the date-time pattern, resulting in list of all the messages with user names
        date_time = re.findall('\d{1,2}/\d{1,2}/\d{2,4},\s\d{1,2}:\d{2}\s[APap][mM]\s-\s', raw_string) # finds all the date-time patterns
        
        df = pd.DataFrame({'date_time': date_time, 'user_msg': user_msg}) # exporting it to a df
   
    # converting date-time pattern which is of type String to type datetime, format is to be specified for the whole string where the placeholders are extracted by the method    
    try:
        df['date_time'] = pd.to_datetime(df['date_time'], format='%m/%d/%y, %I:%M %p - ') #10/20/19, 10:24 pm - 
    except:
        df['date_time'] = pd.to_datetime(df['date_time'], format='%d/%m/%Y, %I:%M %p - ') #20/10/2019, 10:24 pm -
    
    # split user and msg 
    usernames = []
    msgs = []
    for i in df['user_msg']:
        a = re.split('([\w\W]+?):\s', i) # lazy pattern match to first {user_name}: pattern and spliting it aka each msg from a user
        if(a[1:]): # user typed messages
            usernames.append(a[1])
            msgs.append(a[2])
        else: # other notifications in the group(eg: someone was added, some left ...)
            usernames.append("grp_notif")
            msgs.append(a[0])

    # creating new columns         
    df['user'] = usernames
    df['msg'] = msgs

    # dropping the old user_msg col.
    df.drop('user_msg', axis=1, inplace=True)
    
    return df
```
In [ ]:
me = "Prajwal Prashanth"

Data

No. of Images, group notifications and dropping them

Q 1)

Who is the most active member of the group. Who is the least active. Is it same on weekday weekend?

https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DatetimeIndex.weekday.html

Q 2)

Count of all the emoticons that i have used till date.

Q 3)

What can my activity say about my sleep cycle?

Q 4)

What is the difference in Weekend vs Weekday usage pattern?

How many words do I type on average on weekday vs weekend?

Q 5)

Most Usage - Time of Day

Q 6)

In any group, do I have any inclination towards responding to someone?

Q 7)

Which are the most common words?

In [ ]:
comment_words = ' '
stopwords = STOPWORDS.update(['lo', 'ge', 'Lo', 'illa', 'yea', 'ella', 'en', 'na', 'En', 'yeah', 'alli', 'ide', 'okay', 'ok', 'will'])
  
for val in df.msg.values: 
    val = str(val) 
    tokens = val.split() 
        
    for i in range(len(tokens)): 
        tokens[i] = tokens[i].lower() 
          
    for words in tokens: 
        comment_words = comment_words + words + ' '
  
  
wordcloud = WordCloud(width = 800, height = 800, 
                background_color ='black', 
                stopwords = stopwords, 
                min_font_size = 10).generate(comment_words) 

In [ ]:
plt.figure(figsize = (8, 8), facecolor = None) 
plt.imshow(wordcloud) 
plt.axis("off") 
plt.tight_layout(pad = 0) 
  
plt.show() 
In [ ]:
wordcloud.to_image()

Know What They Know(atleast by little

Assingment-kind

  • 1way ya 2way, check for a response time between two people
In [15]:
f=open('WhatsApp Chat with Xyz.txt',encoding='utf-8')
In [7]:
pwd
Out[7]:
'C:\\Users\\DELL\\Desktop\\kwtk-whatsapp-blank'
In [8]:
cd ..
C:\Users\DELL\Desktop
In [16]:
x=f.read()
In [17]:
x
Out[17]:
'2/27/17, 8:38 PM - Aru added Pradeep\n2/28/17, 4:49 PM - Pradeep changed the subject from "😊" to "Xyz"\n2/28/17, 4:50 PM - Pradeep: I am free next Saturday... Not Sunday\n2/28/17, 4:58 PM - CG: Hey, share the expenses details\n2/28/17, 5:02 PM - Pradeep: 55euros to chetan\n2/28/17, 5:03 PM - Pradeep: You can return my money in local currency.. no issues\n2/28/17, 5:03 PM - Pradeep: 😌😌😌\n2/28/17, 5:04 PM - CG: Ull not get it back\n2/28/17, 5:10 PM - Aru: 😜😜\n2/28/17, 9:43 PM - Pradeep: Sharat don\'t have WhatsApp\n2/28/17, 9:44 PM - Pradeep: Who paid in MTR?\n2/28/17, 9:45 PM - CG: Sharath\n2/28/17, 9:45 PM - Pradeep: 1200rs Sharat paid then\n2/28/17, 9:45 PM - CG: Yup\n2/28/17, 9:45 PM - Pradeep: Movie how much it was\n2/28/17, 9:46 PM - CG: 590\n2/28/17, 9:47 PM - CG: Le, U calculate and tell\n2/28/17, 9:47 PM - CG: I\'m sleeping..😴\n2/28/17, 9:48 PM - CG: I should wake up at 3 30😐\n2/28/17, 9:48 PM - CG: Bye\n2/28/17, 10:57 PM - Aru changed this group\'s icon\n2/28/17, 9:48 PM - Pradeep: For u me and Manasa have to pay 240... For Sharat 480...\n2/28/17, 9:49 PM - Pradeep: I will be paying Sharat 720... Directly... You pay 240 less to sharat\n2/28/17, 10:20 PM - Sharath: Pradeep est pay madida\n2/28/17, 10:20 PM - Sharath: Dinner ge\n2/28/17, 10:20 PM - Sharath: How much I shld tranfer\n2/28/17, 10:20 PM - Sharath: ?\n2/28/17, 10:21 PM - Pradeep: Dinner is paid by manasa\n2/28/17, 10:21 PM - Pradeep: It was her treat she said\n2/28/17, 10:21 PM - Pradeep: 😌\n2/28/17, 10:21 PM - Sharath: Y treat\n2/28/17, 10:22 PM - Pradeep: Ad nangu gottilla🤔\n2/28/17, 10:23 PM - Sharath: Add her expense also\n2/28/17, 10:24 PM - Pradeep: It\'s a treat\n2/28/17, 10:26 PM - Sharath: Adee yav kushi ge anta\n2/28/17, 10:26 PM - Pradeep: @919620177843 you have been summoned\n2/28/17, 10:28 PM - Manasa: Ayyo sumne😄\n2/28/17, 10:31 PM - Sharath: Sumne ella beda\n2/28/17, 10:31 PM - Sharath: 😁\n2/28/17, 10:31 PM - Pradeep: 😂\n2/28/17, 10:31 PM - Manasa: Soon I will find some reason\n2/28/17, 10:31 PM - Manasa: 😂\n2/28/17, 10:32 PM - Sharath: Hangadre reason sikmele kodsu.. Iga beda 😝\n2/28/17, 10:32 PM - Manasa: Illa very soon I will Tel u d reason\n2/28/17, 10:33 PM - Manasa: 😂\n2/28/17, 10:33 PM - Sharath: Noo\n2/28/17, 10:37 PM - Pradeep: I m booked this weekend\n2/28/17, 10:37 PM - Manasa: 🙄🙄🙄\n2/28/17, 10:38 PM - Manasa: Busy heladru helu...Booked ante\n2/28/17, 10:39 PM - Pradeep: That\'s how I talk.. To have funny conversation\n2/28/17, 10:39 PM - Pradeep: 😬😬😬\n2/28/17, 10:39 PM - Pradeep: You should know it\n2/28/17, 10:39 PM - Pradeep: 😌😌\n2/28/17, 10:40 PM - Pradeep: Someone add dp\n2/28/17, 10:40 PM - Sharath: Nan mobile clean krishnanna agide\n2/28/17, 10:40 PM - Manasa: <Media omitted>\n2/28/17, 10:40 PM - Sharath: No photos\n2/28/17, 10:40 PM - Manasa: <Media omitted>\n3/2/17, 11:00 PM - Pradeep: @918722030931  a/c details kals pa\n3/3/17, 8:30 AM - Sharath: 915010035097718\n3/3/17, 8:30 AM - Sharath: Sharath\n3/3/17, 8:30 AM - Sharath: Axis bank\n3/3/17, 8:31 AM - Sharath: IFSC code- UTIB0000300\n3/3/17, 8:45 AM - Pradeep: Guys... Manasa got into IBS Hyderabad\n3/3/17, 8:46 AM - Sharath: Wow\n3/3/17, 8:46 AM - Sharath: Congratulations👏🏼👏🏼\n3/3/17, 8:46 AM - Pradeep: Next time nu avladde party\n3/3/17, 8:46 AM - Pradeep: 😛\n3/3/17, 8:48 AM - Manasa: Thank u🙂\n3/3/17, 6:33 PM - CG: Congrats😃\n3/3/17, 8:54 PM - Manasa: Thank u Chetan🙂\n3/4/17, 1:20 AM - Aru: Congrats dea😃\n3/4/17, 6:46 AM - Manasa: Thank u🙂\n3/5/17, 7:42 AM - Pradeep: <Media omitted>\n3/5/17, 7:42 AM - Pradeep: <Media omitted>\n3/5/17, 8:28 AM - CG: Bagew😂\n3/5/17, 8:29 AM - Manasa: So cute🤢🤢\n3/5/17, 8:30 AM - Manasa: U photo bombed me😩\n3/5/17, 8:38 AM - Aru: 😂😂\n3/5/17, 9:26 AM - Pradeep: En nadsir pa ella\n3/5/17, 9:29 AM - Sharath: Huhhaahha😝🤣\n3/5/17, 9:30 AM - Sharath: Iga edde\n3/5/17, 9:30 AM - Pradeep: @918722030931 ... What were u planning to do to Manasa😛\n3/5/17, 9:30 AM - Sharath: She told already\n3/5/17, 9:31 AM - Pradeep: Psycho killer thara expression\n3/5/17, 9:32 AM - Sharath: Le psycho killer nagtana yavagadro\n3/5/17, 9:32 AM - Sharath: 😄\n3/5/17, 9:32 AM - Pradeep: Look at those eyes\n3/5/17, 9:32 AM - Pradeep: It is haunting her till today😅😅\n3/5/17, 9:32 AM - Sharath: 🤘🏼🤘🏼\n3/5/17, 9:35 AM - Manasa: Lol🤣🤣🤣\n7/1/17, 7:31 PM - Pradeep: <Media omitted>\n7/1/17, 7:31 PM - Pradeep: @bayyappanalli station\n7/1/17, 7:34 PM - Sharath: 🤣🤣\n7/1/17, 8:50 PM - Aru: 😂😂\n7/13/17, 12:14 PM - CG: <Media omitted>\n7/13/17, 12:46 PM - Sharath: How can I forget this🤣🤣\n7/13/17, 12:46 PM - Sharath: Flop show from pradeep\n7/13/17, 12:47 PM - Aru: 😜😜😜\n7/13/17, 12:48 PM - CG: 😂😂\n7/13/17, 12:50 PM - Manasa: 🤣🤣\n7/13/17, 12:50 PM - Sharath: We shall go to college once\n7/13/17, 12:50 PM - Sharath: 😄\n7/13/17, 12:51 PM - CG: Pradeep Na kel pa??🙄\n7/13/17, 12:51 PM - CG: Nav ready\n7/13/17, 12:51 PM - Sharath: Good joke 😬\n7/13/17, 12:53 PM - Pradeep: For what?\n7/13/17, 12:54 PM - CG: Ronaldo is here⚽\n7/13/17, 12:54 PM - Sharath: To meet ur UPK\n7/13/17, 11:26 PM - Manasa: That was a punctured ball\n7/13/17, 11:27 PM - Manasa: <Media omitted>\n7/13/17, 11:45 PM - Sharath: What do you think?😝\n7/13/17, 11:45 PM - Pradeep: 🙄\n7/13/17, 11:46 PM - Manasa: I don\'t think so😅\n7/13/17, 11:47 PM - Pradeep: Listen to audio\n7/14/17, 1:19 AM - CG: Exactly🤣\n8/14/17, 10:25 AM - Pradeep: pradeeplh.sarahah.com\n8/14/17, 10:34 AM - Sharath: ??\n8/14/17, 10:35 AM - Pradeep: 🙄sarahah\n8/14/17, 10:38 AM - Sharath: App like Facebook?\n8/14/17, 10:38 AM - Pradeep: No\n8/14/17, 10:39 AM - Pradeep: App where u can give contrustive feedback to anyone anonymously\n8/14/17, 11:12 AM - Sharath: 🤦🏼\u200d♂\n8/14/17, 11:28 AM - Pradeep: 🙄\n8/19/17, 9:02 AM - Pradeep: <Media omitted>\n8/19/17, 9:16 AM - Sharath: 😣😣\n8/19/17, 9:16 AM - Pradeep: Bhaal miss maadkotidien avn?\n8/19/17, 9:16 AM - Pradeep: 😛\n9/11/17, 11:01 AM - Pradeep: <Media omitted>\n9/11/17, 11:04 AM - Sharath: 🤣🤣🤣🤣🤣\n9/11/17, 11:04 AM - Sharath: Yes\n9/11/17, 11:04 AM - Sharath: I said Great job\n9/11/17, 11:04 AM - Sharath: Njoy\n9/11/17, 11:05 AM - Sharath: 🤣\n9/11/17, 11:05 AM - Pradeep: 😂😂😂😂😂😂😂\n9/11/17, 11:05 AM - Sharath: HSM\n9/11/17, 11:05 AM - Sharath: Is he gone mad?\n9/11/17, 11:05 AM - Pradeep: Totally\n9/11/17, 11:05 AM - Pradeep: Obviously he somehow wants his money back\n9/11/17, 11:05 AM - Pradeep: So doing all these circus\n9/11/17, 11:06 AM - Sharath: Very difficult at this moment\n9/11/17, 11:06 AM - Sharath: Cos everyone is aware of what it is\n9/11/17, 4:38 PM - CG: Exactly😂\n9/17/17, 5:26 PM - Manasa: Please take 1 minute of your time and fill this form.Its a small survey regarding coaching institutes ...it  would be of great help in completion of my project...please\n9/17/17, 5:26 PM - Manasa: https://docs.google.com/forms/d/e/1FAIpQLSe3Sgil-P_IXJQjB0A6_iMagHi4EaQU-hipxIXTGocAClRXYw/viewform?usp=sf_link\n10/1/17, 10:10 AM - Pradeep: <Media omitted>\n10/1/17, 10:12 AM - Sharath: 🤣🤣🤣🤣🤣\n10/1/17, 10:12 AM - Sharath: Egjactly\n10/1/17, 10:12 AM - Sharath: Even I got that msg\n10/5/17, 8:04 AM - Pradeep: Hello guys,\n\nThis is regarding SOUMYA BHAT . I know this is a bit inappropriate but I don\'t want anyone else to go through what I am going through right now. So PLEASE READ THIS COMPLETELY.\n\nSoumya is doing a business named QNET. I don\'t blame the business but I blame the people who are working for it in a wrong way and she is one of them. She will call you and tell you that it\'s a job opportunity BUT ITS NOT. It\'s a business in which she will convince you to take up a huge amount of loan (3 lakhs for me) and buy the products through which she will get a whooping profit of 20000 per person.\n\nAt this point, any person would just leave telling some reason. I gave my reason telling that I will have to talk to my mom. They didn\'t let me do it. I was surrounded by 5 strangers(all men) and I was shivering and crying but still some IDIOT NAMED AVINASH told me to open my netbanking. I was reluctant at first and then they said they will just check the interest rates and see and because of my nervousness I had to open, without asking me the loan was taken and even the money got deducted.\n\nAt this moment, I was so damn scared that I wanted to do something to myself like literally but these people didn\'t care. Thankfully my mom supported me and there are enough proofs to lodge an FIR against Soumya. SHE HERSELF HAS ACCEPTED THE MISTAKE.\n\nTrust me guys, I am not the only person who has gone through this. There are many girls who are in the same situation like mine. And I DON\'T WANT ANYONE ELSE TO GET INTO THIS. FIRSTLY PLEASE AVOID SUCH "JOB" OPPORTUNITY CALLS N MESSAGES FROM SOUMYA and be safe.\n\nRegards,\nShruti\n10/5/17, 8:56 AM - Sharath: 🙆🏼\u200d♂🙆🏼\u200d♂\n10/5/17, 8:56 AM - Sharath: Who is she?\n10/5/17, 8:56 AM - Sharath: Shruti\n10/5/17, 9:19 AM - CG: Case registered against her?\n10/5/17, 9:37 AM - Pradeep: Shruti is some Mumbai working girl\n10/5/17, 9:38 AM - Pradeep: @919620177843 it\'s the same Soumya Bhat who was there in PUC with us\n10/5/17, 9:48 AM - CG: Jakati ge hel le,, hushar ag irlike 😂\n10/5/17, 9:49 AM - Pradeep: I hope Jakati won\'t go this far\n10/5/17, 9:49 AM - Pradeep: He will say in the end.... Hatrrr le... That\'s all\n10/5/17, 9:50 AM - CG: Who knows, we didnt expect he will get into Qnet\n10/5/17, 9:51 AM - Sharath: 😂😂\n10/5/17, 10:00 AM - Pradeep: http://www.moneylife.in/article/qnet-ldquoit-is-a-chain-where-a-person-is-fooled-and-then-he-is-trained-to-fool-others-to-earn-moneyrdquo-says-bombay-hc-order/46857.html\n10/5/17, 10:00 AM - Pradeep: Bombay high court itself is saying its a scam\n10/8/17, 8:42 AM - Aru: Hello\n10/8/17, 8:52 AM - Pradeep: Everyone\'s sleeping\n10/8/17, 8:57 AM - Aru: Wake up\n10/8/17, 9:51 AM - CG: Hel pa\n10/8/17, 9:52 AM - Pradeep: Wake up Sharath\n10/8/17, 10:16 AM - Sharath: Heloo\n10/8/17, 10:17 AM - Pradeep: Ready aag pa\n10/8/17, 10:17 AM - Sharath: What\'s the plan\n10/8/17, 10:17 AM - Sharath: ?\n10/8/17, 10:19 AM - Pradeep: We r coming to ur pg\n10/8/17, 10:27 AM - Sharath: Baa baa\n10/8/17, 10:34 AM - Pradeep: Guys\n10/8/17, 10:34 AM - Pradeep: Where to meet\n1/21/18, 6:23 PM - Pradeep: <Media omitted>\n1/21/18, 6:23 PM - Pradeep: <Media omitted>\n1/21/18, 6:23 PM - Pradeep: <Media omitted>\n1/21/18, 6:23 PM - Pradeep: <Media omitted>\n1/21/18, 6:23 PM - Pradeep: <Media omitted>\n1/21/18, 6:23 PM - Pradeep: <Media omitted>\n1/21/18, 6:23 PM - Pradeep: <Media omitted>\n1/21/18, 6:24 PM - Pradeep: 😜😜😜😜\n1/21/18, 6:27 PM - Sharath: Ababaa\n1/21/18, 6:27 PM - Sharath: 🤣\n1/21/18, 6:29 PM - Pradeep: 😌😌😌\n1/21/18, 6:30 PM - Sharath: DSLR?\n1/21/18, 6:31 PM - Pradeep: Haan... But camera problem kodta ittu... Ond pic nu focus aagilla\n1/21/18, 6:31 PM - Pradeep: So session got postponed\n1/21/18, 6:31 PM - Sharath: Next session yavaga\n1/21/18, 6:31 PM - Sharath: ?\n1/21/18, 6:31 PM - Sharath: 😛\n1/21/18, 6:33 PM - Pradeep: Date decide aagilla\n1/21/18, 6:34 PM - Pradeep: Sirsi nal ond session pakka... Bike togond hodte\n1/21/18, 6:34 PM - Manasa: Jatkak kodalana\n1/21/18, 6:34 PM - Pradeep: 😑\n1/21/18, 6:34 PM - Pradeep: Hudagrige jaatka Tagata.. kodadalla🙄\n1/21/18, 6:34 PM - Sharath: 🤣🤣\n1/21/18, 6:35 PM - Sharath: Igle tagond hogbyad le\n1/21/18, 6:35 PM - Manasa: Iga Kala badlajala\n1/21/18, 6:35 PM - Manasa: So\n1/21/18, 6:35 PM - Pradeep: Btw.... Sirsi jaatre from 27th Feb to 6th March... I am planning for March 3rd to 6th..\n1/21/18, 6:36 PM - Pradeep: Ella leave haakro\n1/21/18, 6:36 PM - Sharath: 3rd yavattu\n1/21/18, 6:36 PM - Sharath: ?\n1/21/18, 6:36 PM - Pradeep: @919620177843  and  @917795358568 you guys too😬😬\n1/21/18, 6:36 PM - Pradeep: Saturday\n1/21/18, 6:36 PM - Manasa: Lets see\n1/21/18, 6:37 PM - Sharath: 👍🏼\n3/12/18, 9:38 PM - Pradeep: <Media omitted>\n3/12/18, 10:09 PM - Aru left\n3/12/18, 10:09 PM - You\'re now an admin\n3/17/18, 5:28 PM - Sharath: <Media omitted>\n3/17/18, 5:28 PM - Sharath: <Media omitted>\n3/17/18, 5:28 PM - Sharath: <Media omitted>\n3/17/18, 5:29 PM - Sharath: Bhaal light weight ansatte\n3/17/18, 5:29 PM - Sharath: Full fiber\n3/17/18, 5:38 PM - Pradeep: 🙄🙄🙄\n3/17/18, 5:38 PM - Sharath: What\n3/17/18, 5:38 PM - Pradeep: 162kerb weight\n3/17/18, 5:38 PM - Sharath: Houdo illo\n3/17/18, 5:39 PM - Pradeep: Last weekend manig hogbekidre 147touch maadini... Swalpa nu shake aaglilla\n3/17/18, 5:40 PM - Sharath: Ktm 150\n3/17/18, 5:40 PM - Pradeep: Duke or rc?\n3/17/18, 5:40 PM - Sharath: Shake agalla but risky\n3/17/18, 5:40 PM - Sharath: Fibre body\n3/17/18, 5:40 PM - Pradeep: Nange risky ansilla illitanka\n3/17/18, 5:41 PM - Pradeep: Ella haange irtadyo\n3/17/18, 5:41 PM - Pradeep: KTM nu same\n3/17/18, 5:41 PM - Sharath: Adee helid\n3/17/18, 5:41 PM - Sharath: I don\'t like KTM\n3/17/18, 5:42 PM - Sharath: But look ultimatteeee ide RS200\n3/17/18, 5:47 PM - Pradeep: KTM Duke 250was good but it doesn\'t have abs\n3/17/18, 5:50 PM - Sharath: Hmm\n3/18/18, 12:52 PM - CG: <Media omitted>\n3/18/18, 12:52 PM - CG: <Media omitted>\n3/18/18, 12:52 PM - CG: <Media omitted>\n3/18/18, 12:52 PM - CG: <Media omitted>\n3/18/18, 12:52 PM - CG: <Media omitted>\n3/18/18, 12:53 PM - Pradeep: Party\n3/18/18, 12:56 PM - Sharath: Chaluvaa\n3/18/18, 1:07 PM - Pradeep: Ning suit aagalla le aa bike\n3/18/18, 2:29 PM - Manasa: Nice bike🤩\n3/18/18, 2:56 PM - Pradeep: Dodd mandi\n3/18/18, 3:07 PM - Pradeep: Next weekend trip? 😐 even bhanu is free\n3/18/18, 5:19 PM - CG: Where?\n3/18/18, 5:19 PM - Pradeep: Somewhere\n3/18/18, 5:20 PM - Pradeep: 50-60kms\n3/20/18, 11:36 PM - Pradeep: Barti illa?\n3/21/18, 12:12 AM - CG: Im not sure.. heltini\n3/21/18, 12:12 AM - Pradeep: 😒\n3/21/18, 12:12 AM - CG: Ni first elli anta hel\n3/21/18, 12:13 AM - Pradeep: @918722030931 what was that place which we talked last time\n3/21/18, 12:58 AM - Pradeep: <Media omitted>\n3/21/18, 5:59 AM - Sharath: Somewhere near tumkur\n3/21/18, 6:00 AM - Sharath: Dd hills\n3/21/18, 8:10 AM - Pradeep: Somewhere Mysore road we talked\n3/21/18, 8:10 AM - Pradeep: We talked about going via your house route\n3/22/18, 8:16 PM - Sharath: <Media omitted>\n3/22/18, 8:33 PM - Pradeep: 🙄ningen aat pa\n3/22/18, 8:34 PM - Sharath: Hengee😎\n3/22/18, 8:41 PM - Pradeep: Nee ello irbekitle sharat\n3/22/18, 8:53 PM - Sharath: Gottid\n3/22/18, 9:15 PM - Manasa: Why Aruna left the group\n3/22/18, 9:15 PM - Manasa: Chetan add her back\n4/1/18, 7:20 PM - You added Aru\n8/21/18, 11:25 PM - Sharath left\n10/22/18, 10:57 PM - Manasa left\n11/2/18, 6:18 PM - Pradeep left\n12/31/18, 7:28 AM - You added Harsha C and Pradeep\n12/31/18, 7:30 AM - CG: Share the expenses details @918884882928\n12/31/18, 8:17 AM - Pradeep: 1700 food 1200 movie\n12/31/18, 8:19 AM - Pradeep: Naale bartiren pa ibru\n12/31/18, 8:19 AM - Pradeep: Moovaroo*\n12/31/18, 10:39 AM - CG: En plan\n12/31/18, 10:39 AM - CG: ?\n12/31/18, 10:41 AM - Harsha C: Guys\n\nI can\'t come tomorrow \n\nHv some work\n12/31/18, 10:46 AM - Pradeep: Jana aadre plan\n12/31/18, 10:46 AM - Pradeep: Else nothing\n1/2/19, 11:59 AM - Pradeep: @918147874935 send pics..\n2/14/19, 6:25 PM - Pradeep: Sunday bike ride?\n2/14/19, 6:26 PM - CG: U r free??🙄🤨\n2/14/19, 6:26 PM - Pradeep: Mostly\n2/14/19, 6:26 PM - Pradeep: 🤭\n2/14/19, 6:26 PM - Pradeep: Not 100% sure\n2/14/19, 6:26 PM - Pradeep: But mostly possible\n2/14/19, 6:26 PM - Pradeep: Add Sharat\n2/14/19, 6:27 PM - You added Sharath\n2/15/19, 7:56 AM - Pradeep: @918867126930 @918147874935 @917795358568\n2/15/19, 7:56 AM - Pradeep: Helripa\n2/15/19, 7:56 AM - Pradeep: Nand Sunday 100% fix\n2/15/19, 9:56 AM - Aru: M ready\n2/15/19, 11:02 AM - CG: Fix... 💯%👍\n2/15/19, 11:03 AM - CG: Sharath , harsha helripa\n2/15/19, 11:03 AM - Sharath: Nandu badminton tournament ide\n2/15/19, 11:03 AM - CG: Le, v r going on sunday\n2/15/19, 11:03 AM - Sharath: So can\'t confirm\n2/15/19, 11:03 AM - CG: Nind naale ala?\n2/15/19, 11:04 AM - Sharath: Houduu\n2/23/19, 8:04 PM - CG: <Media omitted>\n2/23/19, 8:07 PM - Sharath: Handsome guy\n2/23/19, 8:07 PM - CG: Yes👌👌👌\n2/23/19, 8:07 PM - Sharath: 😎😎\n2/23/19, 8:10 PM - CG: <Media omitted>\n2/23/19, 8:12 PM - Sharath: With  background sounds in between\n2/23/19, 8:12 PM - CG: Hun😂\n2/23/19, 8:13 PM - CG: <Media omitted>\n2/23/19, 8:14 PM - Sharath: Stop Sharing pics\n2/23/19, 8:14 PM - CG: 😂😂\n2/23/19, 8:14 PM - Sharath: 🤮\n2/23/19, 8:19 PM - Aru: 😜😜\n6/10/19, 10:50 AM - Pradeep: This Saturday bike trip?\n6/10/19, 10:51 AM - Aru: Gng to tirupati\n6/10/19, 10:51 AM - Pradeep: Bye bye🤭\n6/10/19, 10:51 AM - Aru: 😒\n6/10/19, 11:10 AM - CG: Where?\n6/10/19, 11:11 AM - CG: I have exam on sunday.. But ill try to finish reading for that asap..\n6/10/19, 11:11 AM - Pradeep: Somewhere nearby I need to take bus to Sirsi on Saturday night\n6/10/19, 11:12 AM - CG: @918722030931 and @918867126930 ?\n6/10/19, 11:16 AM - Sharath: I m not sure...\n6/10/19, 11:16 AM - Sharath: Swalpa kelsa baaki ide\n6/10/19, 11:25 AM - Harsha C: I have class man on both sat and Sunday.. \n\nEven after 4 years from graduation,  my studies are not getting over.  👶🏼\n6/10/19, 11:30 AM - Pradeep: 🙄\n7/9/19, 12:58 AM - Sharath: Guys\n7/9/19, 12:59 AM - Sharath: Is it possible to meet on 11th july ?\n7/9/19, 7:15 AM - CG: 🙄\n7/9/19, 7:15 AM - CG: En plan\n7/9/19, 7:29 AM - Aru: I can meet in mrng\n7/9/19, 8:01 AM - Pradeep: Even I can meet in the morning... Dont know exact time\n7/9/19, 9:44 AM - Harsha C: <Media omitted>\n7/9/19, 9:44 AM - Harsha C: In India?  Or Paris?\n7/9/19, 11:20 AM - Sharath: Good one😃👍🏼\n7/9/19, 11:20 AM - Sharath: India\n7/9/19, 11:20 AM - Sharath: Huchha is leaving on 12tg\n7/9/19, 11:20 AM - Sharath: So ondsari meet agana antaa\n7/9/19, 11:21 AM - CG: Ohh\n7/9/19, 11:31 AM - Aru: Done\n7/9/19, 11:55 AM - Pradeep: 🙄\n7/9/19, 11:55 AM - Pradeep: Yaar pa\n9/11/19, 9:38 PM - Pradeep: Dec 7th bike ride?\n9/11/19, 9:38 PM - Aru: Done\n9/11/19, 9:38 PM - Pradeep: You don\'t have a bike\n9/11/19, 9:46 PM - Aru: 😼\n9/11/19, 9:49 PM - Harsha C: Sure.  \nIn Toronto?  Or Montreal?\n9/11/19, 9:49 PM - Pradeep: Sakalaeshapura road🤣\n9/11/19, 9:50 PM - Harsha C: The only reason you didn\'t get UK scolding is bcoz it is WhatsApp grp\n9/11/19, 9:50 PM - Harsha C: 😂\n9/11/19, 9:53 PM - Harsha C: Offend aadi alo pradeepa Canada hogi 😂\n9/11/19, 9:54 PM - Pradeep: Weekdays naag enne haaki en pa\n9/11/19, 9:58 PM - Harsha C: Kindly check separate message 😄\n9/12/19, 8:21 PM - Harsha C: <Media omitted>\n9/12/19, 8:22 PM - Pradeep: IPhone 11 max pro🤷🏻\u200d♂\n9/12/19, 8:24 PM - Harsha C: You can learn from this data and expected to give result. \n\n#Regression problem. \nSupervised learning.. 😂😂\n9/12/19, 8:25 PM - Pradeep: Data is incorrect.... Not cleansed properly... Back to data warehousing team for inspection on Data Quality\n9/12/19, 8:27 PM - Sharath: Ok\n9/12/19, 8:27 PM - Harsha C: 😂😂\n9/12/19, 8:29 PM - Harsha C: Make sure that your output should be 3.\nOtherwise effort from data warehouse team goes waste. \n(Since we already know the result here😅)\n'
In [25]:
raw_string = ' '.join(f.read().split('\n')) # converting the list split by newline char. as one whole string as there can be multi-line messages
user_msg = re.split('\d{1,2}/\d{1,2}/\d{2,4},\s\d{1,2}:\d{1,2}\s[APap][mM]\s-\s', x) [1:] # splits at all the date-time pattern, resulting in list of all the messages with user names
date_time = re.findall('\d{1,2}/\d{1,2}/\d{2,4},\s\d{1,2}:\d{1,2}\s[APap][mM]\s-\s', x) # finds all the date-time patterns
        
df = pd.DataFrame({'date_time': date_time, 'user_msg': user_msg})
In [26]:
df
Out[26]:
In [29]:
try:
    df['date_time'] = pd.to_datetime(df['date_time'], format='%m/%d/%y, %I:%M %p - ') #10/20/19, 10:24 pm - 
except:
    df['date_time'] = pd.to_datetime(df['date_time'], format='%d/%m/%Y, %I:%M %p - ')
In [30]:
usernames = []
msgs = []
for i in df['user_msg']:
    a = re.split('([\w\W]+?):\s', i) # lazy pattern match to first {user_name}: pattern and spliting it aka each msg from a user
    if(a[1:]): # user typed messages
        usernames.append(a[1])
        msgs.append(a[2])
    else: # other notifications in the group(eg: someone was added, some left ...)
        usernames.append("grp_notif")
        msgs.append(a[0])

    # creating new columns         
df['user'] = usernames
df['msg'] = msgs

    # dropping the old user_msg col.
df.drop('user_msg', axis=1, inplace=True)
In [33]:
df.head()
Out[33]:
In [34]:
images = df[df['msg']=="<Media omitted> "] #no. of images, images are represented by <media omitted>
images.shape
Out[34]:
(0, 3)
In [36]:
grp_notif = df[df['user']=="grp_notif"] #no. of grp notifications
grp_notif.shape
Out[36]:
(11, 3)
In [37]:
df.drop(images.index, inplace=True) #removing images
df.drop(grp_notif.index, inplace=True)
In [38]:
df.reset_index(inplace=True, drop=True)
df.shape
Out[38]:
(347, 3)
In [39]:
df.groupby("user")["msg"].count().sort_values(ascending=False)
Out[39]:
user
Pradeep     136
Sharath     105
CG           50
Manasa       28
Aru          15
Harsha C     13
Name: msg, dtype: int64
In [40]:
import jovian
In [ ]:
jovian.commit()
[jovian] Saving notebook..
In [ ]: