Title: Guilded Project: Exploring Hacker News Posts

Introduction

Hacker News is a site started by the startup incubator Y Combinator, where user-submitted stories (known as "posts") are voted and commented upon, similar to reddit. Hacker News is extremely popular in technology and startup circles, and posts that make it to the top of Hacker News' listings can get hundreds of thousands of visitors as a result.

We're specifically interested in posts whose titles begin with either Ask HN or Show HN. For titles with Ask HN, users submit posts to ask the Hacker News community a specific question. Likewise, for titles with Show HN users submit posts to show the Hacker News community a project, product, or just generally something interesting.

Purpose

We'll compare these two types of posts to determine the following:

  • Do Ask HN or Show HN receive more comments on average?
  • Do posts created at a certain time receive more comments on average?

Methodology

The data set can be found here, but note that it has been reduced from almost 300,000 rows to approximately 20,000 rows by removing all submissions that did not receive any comments, and then randomly sampling from the remaining submissions. We will use this data set as a sampling size for our data analysis.

Let's start by opening the set of data and display the first five rows:

In [1]:
from csv import reader

opened_file = open ('hacker_news.csv')
read_file = reader (opened_file)
hn = list(read_file)
hn_test = hn[0:5]

print(hn_test)
[['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'], ['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01']]

Notice that the first list in the inner lists contains the column headers, and the lists after contain the data for one row. In order to analyze our data, we need to first remove the row containing the column headers. Let's remove that first row next.

In [2]:
headers = hn[0]
hn = hn [1:]
print (headers)
['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']

Now that we've removed the headers from hn, we're ready to filter our data. Since we're only concerned with post titles beginning with Ask HN or Show HN, we'll create new lists of lists containing just the data for those titles.

To find the posts that begin with either Ask HN or Show HN, we'll use the string method startswith. Given a string object, say, string1, we can check if starts with, say, dq by inspecting the output of the object string1.startswith('dq'). If string1 starts with dq, it will return True, otherwise it will return False.

In [3]:
ask_posts = []
show_posts = []
other_posts = []

for row in hn:
    title = row[1]
    title = title.lower()
    if title.startswith('ask hn') == True:
        ask_posts.append(row)
    elif title.startswith('show hn') == True:
        show_posts.append(row)
    else:
        other_posts.append(row)

print ('Number of Ask HN Posts: ', len(ask_posts))
print ('Number of Show HN Posts: ', len(show_posts))
print ('Number of Show Other Posts: ', len(other_posts))
Number of Ask HN Posts:  1744
Number of Show HN Posts:  1162
Number of Show Other Posts:  17194

In the last screen, we separated the "ask posts" and the "show posts" into two list of lists named ask_posts and show_posts. Next, let's determine if ask posts or show posts receive more comments on average.

In [4]:
total_ask_comments = 0

for row in ask_posts:
    num_comments = int(row[4])
    total_ask_comments = total_ask_comments + num_comments

avg_ask_comments = total_ask_comments / len(ask_posts)

print ('Average Number of Comments for Ask HN Posts: ', avg_ask_comments)

total_show_comments = 0

for row in show_posts:
    num_comments = int(row[4])
    total_show_comments = total_show_comments + num_comments

avg_show_comments = total_show_comments / len(show_posts)

print ('Average Number of Comments for Show HN Posts: ', avg_show_comments)
Average Number of Comments for Ask HN Posts:  14.038417431192661
Average Number of Comments for Show HN Posts:  10.31669535283993

The data shows the average number of received comments for Ask HN posts is greater than the received comments for Show HN posts. Since ask posts are more likely to receive comments, we'll focus our remaining analysis just on these posts.

Next, we'll determine if ask posts created at a certain time are more likely to attract comments. We'll use the following steps to perform this analysis:

1.Calculate the amount of ask posts created in each hour of the day, along with the number of comments received.

2.Calculate the average number of comments ask posts receive by hour created.

In [38]:
import datetime as dt

result_list = []

for row in ask_posts:
    created_at = row[6]
    num_of_comments = row[4]
    row = [created_at, num_of_comments]
    result_list.append(row)

counts_by_hour = {}
comments_by_hour = {}

date_hour_format = '%m/%d/%Y %H:%M'

for rows in result_list:
    date_hour = rows[0]
    comments = rows[1]
    extract_hour = dt.datetime.strptime(date_hour, date_hour_format).time()
    extract_date = dt.datetime.strptime(date_hour, date_hour_format).date()
    #print (extract_hour)
    #print (extract_date)
    time = extract_hour.strftime("%H")
    #print (time)
    if time not in counts_by_hour:
        counts_by_hour[time] = 1
        comments_by_hour[time] = int(comments)
    else:
        counts_by_hour[time] += 1
        comments_by_hour[time] += int(comments)

print ('Counts by Hours:')
print (counts_by_hour)
print ("\n")
print ('Comments by Hours:')
print (comments_by_hour)
Counts by Hours:
{'02': 58, '09': 45, '06': 44, '16': 108, '14': 107, '10': 59, '00': 55, '23': 68, '03': 54, '21': 109, '22': 71, '19': 110, '04': 47, '13': 85, '11': 58, '05': 46, '20': 80, '08': 48, '07': 34, '17': 100, '15': 116, '18': 109, '12': 73, '01': 60}


Comments by Hours:
{'02': 1381, '09': 251, '06': 397, '16': 1814, '14': 1416, '10': 793, '00': 447, '23': 543, '03': 421, '21': 1745, '22': 479, '19': 1188, '04': 337, '13': 1253, '11': 641, '05': 464, '20': 1722, '08': 492, '07': 267, '17': 1146, '15': 4477, '18': 1439, '12': 687, '01': 683}

In the above, we created two dictionaries:

  • counts_by_hour: contains the number of ask posts created during each hour of the day.

  • comments_by_hour: contains the corresponding number of comments ask posts created at each hour received.

Next, we'll use these two dictionaries to calculate the average number of comments for posts created during each hour of the day.

In [47]:
avg_by_hour = []

for hour in comments_by_hour:
    avg_by_hour.append([hour, comments_by_hour[hour]/counts_by_hour[hour]])

print ('Average number of comments for posts created during each hour of the day:')
print('\n')
print (avg_by_hour)
Average number of comments for posts created during each hour of the day:


[['02', 23.810344827586206], ['09', 5.5777777777777775], ['06', 9.022727272727273], ['16', 16.796296296296298], ['14', 13.233644859813085], ['10', 13.440677966101696], ['00', 8.127272727272727], ['23', 7.985294117647059], ['03', 7.796296296296297], ['21', 16.009174311926607], ['22', 6.746478873239437], ['19', 10.8], ['04', 7.170212765957447], ['13', 14.741176470588234], ['11', 11.051724137931034], ['05', 10.08695652173913], ['20', 21.525], ['08', 10.25], ['07', 7.852941176470588], ['17', 11.46], ['15', 38.5948275862069], ['18', 13.20183486238532], ['12', 9.41095890410959], ['01', 11.383333333333333]]

In the last screen, we calculated the average number of comments for posts created during each hour of the day, and stored the results in a list of lists named avg_by_hour.

Although we now have the results we need, this format makes it hard to identify the hours with the highest values. Let's finish by sorting the list of lists and printing the five highest values in a format that's easier to read.

In [75]:
swap_avg_by_hour = []

for rows in avg_by_hour:
    avg_comments = rows [1]
    avg_post = rows [0]
    swap = [avg_comments, avg_post]
    swap_avg_by_hour.append(swap)

#print (swap_avg_by_hour)

sorted_swap = sorted(swap_avg_by_hour, reverse = True)

#print (sorted_swap)

print('Top 5 Hours for Ask Posts Comments:')
#print(sorted_swap[0:5])


for row in sorted_swap[0:5]:
    avg = row[0]
    hour = row [1]
    hour_format ="%H"
    hour = dt.datetime.strptime(hour, hour_format)
    hour = hour.strftime('%H:%M')
    hour_avg_string = '{h}: {a:.2f} average comments per post.'.format(h=hour, a=avg)
    print (hour_avg_string)
    
Top 5 Hours for Ask Posts Comments:
15:00: 38.59 average comments per post.
02:00: 23.81 average comments per post.
20:00: 21.52 average comments per post.
16:00: 16.80 average comments per post.
21:00: 16.01 average comments per post.

Based on