Hacker News Posts

In this project, we'll work with a data set of submissions to popular technology site Hacker News [https://news.ycombinator.com/].

Hacker News is a site started by the startup incubator Y Combinator, where user-submitted stories (known as "posts") are voted and commented upon, similar to reddit. Hacker News is extremely popular in technology and startup circles, and posts that make it to the top of Hacker News' listings can get hundreds of thousands of visitors as a result.

You can find the data set here [https://www.kaggle.com/hacker-news/hacker-news-posts]

Below are descriptions of the columns:

id: The unique identifier from Hacker News for the post.

title: The title of the post.

url: The URL that the posts links to, if it the post has a URL.

num_points: The number of points the post acquired, calculated as the total number of upvotes minus the total number of downvotes.

num_comments: The number of comments that were made on the post.

author: The username of the person who submitted the post.

created_at: The date and time at which the post was submitted.

In [1]:
from csv import*
open_file=open("hacker_news.csv")
read_file=reader(open_file)
hn=list(read_file)
print(hn[0:3])
[['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'], ['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30']]
In [2]:
header=hn[0]
hn=hn[1:]
print(header)
print(hn[0])
['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']
['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52']
In [3]:
ask_posts=[]
show_posts=[]
other_posts=[]
for row in hn:
    title=row[1]
    if title.startswith("Ask HN"):
        ask_posts.append(row)
    elif title.startswith("Show HN"):
        show_posts.append(row)
    else:
        other_posts.append(row)
print(len(ask_posts),len(show_posts),len(other_posts))
    
1742 1161 17197

Here we seperated posts into Ask posts, Show posts,and Other posts based on beginning of title.

In [4]:
total_ask_comments=0
for comment in ask_posts:
    element=int(comment[4])
    total_ask_comments+=element
avg_ask_comments=total_ask_comments/len(ask_posts)
print("Average Ask Comments : ",avg_ask_comments)

total_show_comments=0
for comment in show_posts:
    element=int(comment[4])
    total_show_comments+=element
avg_show_comments=total_show_comments/len(show_posts)
print("Average Show Comments : ",avg_show_comments)
    
    
    
    
    
Average Ask Comments :  14.044776119402986
Average Show Comments :  10.324720068906116

Here we calculated average of ask and show comments. On an average Ask post received more comments.

Since Ask posts are more likely to receive comments, we'll focus our remaining analysis just on these posts.

Now we will calculate number of Ask post created in each hour of day, along with number of comments received per hour.

In [5]:
import datetime as dt

result_list=[]

for element in ask_posts:
    created_at=element[6]
    comment=int(element[4])
    result_list.append([created_at, comment])

print(len(result_list)) 
print(result_list[:5])

counts_by_hour={}
comments_by_hour={}
for element in result_list:
    date_hour=element[0]
    comment=element[1]
    date_hour=dt.datetime.strptime(date_hour,"%m/%d/%Y %H:%M")
    hour=dt.datetime.strftime(date_hour,"%H")
    if hour not in counts_by_hour:
        counts_by_hour[hour]=1
        comments_by_hour[hour]=comment
    else:
        counts_by_hour[hour]+=1
        comments_by_hour[hour]+=comment
      
        
print(counts_by_hour)
print(comments_by_hour)
        

    
    

    



           

            
                
            
    
        
        
            
        
        
        
    
    


    
    
1742
[['8/16/2016 9:55', 6], ['11/22/2015 13:43', 29], ['5/2/2016 10:14', 1], ['8/2/2016 14:20', 3], ['10/15/2015 16:38', 17]]
{'14': 107, '05': 46, '20': 80, '22': 71, '23': 68, '03': 54, '00': 54, '01': 60, '13': 85, '12': 73, '18': 108, '16': 108, '10': 59, '15': 116, '21': 109, '19': 110, '02': 58, '07': 34, '04': 47, '06': 44, '17': 100, '08': 48, '11': 58, '09': 45}
{'14': 1416, '05': 464, '20': 1722, '22': 479, '23': 543, '03': 421, '00': 439, '01': 683, '13': 1253, '12': 687, '18': 1430, '16': 1814, '10': 793, '15': 4477, '21': 1745, '19': 1188, '02': 1381, '07': 267, '04': 337, '06': 397, '17': 1146, '08': 492, '11': 641, '09': 251}

we started by initiating empty list result_list

loop over ask_posts and append date-time and comment corresponding to each post in result_list, now result_list is nested list, each list with two elements.

Create two empty dictionaries counts_by_hour : for number of posts per hour comment_by_hour : for number of comments per hour

Then loop through each row of result_list assign datetime to variable date_hour and comment to variable comment

As we have to calculate number of posts and comments for each hour, we used if statement and filled above two dictionaries

In [6]:
avg_by_hour=[]
for hour in comments_by_hour:
    avg_by_hour.append([hour,comments_by_hour[hour]/counts_by_hour[hour]])
print(avg_by_hour)
[['14', 13.233644859813085], ['05', 10.08695652173913], ['20', 21.525], ['22', 6.746478873239437], ['23', 7.985294117647059], ['03', 7.796296296296297], ['00', 8.12962962962963], ['01', 11.383333333333333], ['13', 14.741176470588234], ['12', 9.41095890410959], ['18', 13.24074074074074], ['16', 16.796296296296298], ['10', 13.440677966101696], ['15', 38.5948275862069], ['21', 16.009174311926607], ['19', 10.8], ['02', 23.810344827586206], ['07', 7.852941176470588], ['04', 7.170212765957447], ['06', 9.022727272727273], ['17', 11.46], ['08', 10.25], ['11', 11.051724137931034], ['09', 5.5777777777777775]]

we calculated the average number of comments for posts created during each hour of the day, and stored the results in a list of lists named avg_by_hour

In [7]:
swap_avg_by_hour=[]
for i in avg_by_hour:
    x=i[0]
    y=i[1]
    swap_avg_by_hour.append([y,x])
print(swap_avg_by_hour)
[[13.233644859813085, '14'], [10.08695652173913, '05'], [21.525, '20'], [6.746478873239437, '22'], [7.985294117647059, '23'], [7.796296296296297, '03'], [8.12962962962963, '00'], [11.383333333333333, '01'], [14.741176470588234, '13'], [9.41095890410959, '12'], [13.24074074074074, '18'], [16.796296296296298, '16'], [13.440677966101696, '10'], [38.5948275862069, '15'], [16.009174311926607, '21'], [10.8, '19'], [23.810344827586206, '02'], [7.852941176470588, '07'], [7.170212765957447, '04'], [9.022727272727273, '06'], [11.46, '17'], [10.25, '08'], [11.051724137931034, '11'], [5.5777777777777775, '09']]
In [8]:
sorted_swap=sorted(swap_avg_by_hour, reverse=True)
for i in sorted_swap[:5]:
    hour=i[1]
    avg=i[0]
    hour=dt.datetime.strptime(hour,"%H")
    top5="{} : {:.2f} average comments per post.".format(hour,avg)
    print(top5)
1900-01-01 15:00:00 : 38.59 average comments per post.
1900-01-01 02:00:00 : 23.81 average comments per post.
1900-01-01 20:00:00 : 21.52 average comments per post.
1900-01-01 16:00:00 : 16.80 average comments per post.
1900-01-01 21:00:00 : 16.01 average comments per post.

Here, we sorted swap_avg_by_hour in descending order. And then printed top 5 hours for ask posts comments. The most comments on post are between 15:00 UTC i.e, 3pm to 2:00 UTC (2 am)