Exploring Hacker News Posts

Hacker News is a site started by the startup incubator Y Combinator, where user-submitted stories (known as "posts") are voted and commented upon, similar to reddit. Hacker News is extremely popular in technology and startup circles.

In this project, we'll compare two different types of posts from Hacker News, those which begin with either Ask HN or Show HN.

We'll specifically compare these two types of posts to determine the following:

*Do Ask HN or Show HN receive more comments on average?
*Do posts created at a certain time receive more comments on average?

Reading the dataset and removing the header

In [2]:
from csv import reader
open_file = open('hacker_news.csv')
read_file = reader(open_file)
hacker_news = list(read_file)
hacker_news[:5]
Out[2]:
[['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'],
 ['12224879',
  'Interactive Dynamic Video',
  'http://www.interactivedynamicvideo.com/',
  '386',
  '52',
  'ne0phyte',
  '8/4/2016 11:52'],
 ['10975351',
  'How to Use Open Source and Shut the Fuck Up at the Same Time',
  'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/',
  '39',
  '10',
  'josep2',
  '1/26/2016 19:30'],
 ['11964716',
  "Florida DJs May Face Felony for April Fools' Water Joke",
  'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/',
  '2',
  '1',
  'vezycash',
  '6/23/2016 22:20'],
 ['11919867',
  'Technology ventures: From Idea to Enterprise',
  'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429',
  '3',
  '1',
  'hswarna',
  '6/17/2016 0:01']]
In [5]:
headers = hacker_news[1]
hacker_news = hacker_news[1:]
print(headers)
print(hacker_news[:5])

['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52']
[['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]

Separating Ask HN and Show HN posts

First, we'll identify posts that begin with either Ask HN or Show HN and separate the data for those two types of posts into different lists.

In [7]:
ask_posts = []
show_posts = []
other_posts = []
for row in hacker_news:
    title = row[1]
    if title.lower().startswith('ask hn'):
        ask_posts.append(row)
    elif title.lower().startswith('show hn'):
        show_posts.append(row)
    else:
        other_posts.append(row)
print(len(ask_posts))
print(len(show_posts))
print(len(other_posts))

1744
1162
17194

Calculating the Average Number of Comments for Ask HN and Show HN Posts

In [8]:
total_ask_comments = 0
for row in ask_posts:
    total_ask_comments+=int(row[4])
avg_ask_comments = total_ask_comments/len(ask_posts)
print(avg_ask_comments)

14.038417431192661
In [9]:
total_show_comments = 0
for row in show_posts:
    total_show_comments+=int(row[4])
avg_show_comments = total_show_comments/len(show_posts)
print(avg_show_comments)

10.31669535283993

Finding the Amount of Ask Posts and Comments by Hour Created

In [14]:
import datetime as dt
result_list = []
for row in ask_posts:
    result_list.append([row[6],int(row[4])])
counts_by_hour = {}
comments_by_hour = {}
for row in result_list:
    date = row[0]
    comment = row[1]
    hour = dt.datetime.strptime(date,"%m/%d/%Y %H:%M").strftime('%H')
    if hour not in counts_by_hour:
        counts_by_hour[hour] = 1
        comments_by_hour[hour] = comment
    else:
        counts_by_hour[hour]+=1
        comments_by_hour[hour]+=comment
print(counts_by_hour)
print(comments_by_hour)

{'01': 60, '16': 108, '12': 73, '21': 109, '02': 58, '15': 116, '00': 55, '04': 47, '23': 68, '07': 34, '22': 71, '17': 100, '11': 58, '14': 107, '20': 80, '05': 46, '09': 45, '13': 85, '08': 48, '06': 44, '10': 59, '18': 109, '19': 110, '03': 54}
{'01': 683, '16': 1814, '12': 687, '21': 1745, '02': 1381, '15': 4477, '00': 447, '04': 337, '23': 543, '07': 267, '22': 479, '17': 1146, '11': 641, '14': 1416, '20': 1722, '05': 464, '09': 251, '13': 1253, '08': 492, '06': 397, '10': 793, '18': 1439, '19': 1188, '03': 421}

Calculating the Average Number of Comments for Ask HN Posts by Hour

In [15]:
avg_by_hour = []

for hr in comments_by_hour:
    avg_by_hour.append([hr, comments_by_hour[hr] / counts_by_hour[hr]])

avg_by_hour
Out[15]:
[['01', 11.383333333333333],
 ['16', 16.796296296296298],
 ['12', 9.41095890410959],
 ['21', 16.009174311926607],
 ['02', 23.810344827586206],
 ['15', 38.5948275862069],
 ['00', 8.127272727272727],
 ['04', 7.170212765957447],
 ['23', 7.985294117647059],
 ['07', 7.852941176470588],
 ['22', 6.746478873239437],
 ['17', 11.46],
 ['11', 11.051724137931034],
 ['14', 13.233644859813085],
 ['20', 21.525],
 ['05', 10.08695652173913],
 ['09', 5.5777777777777775],
 ['13', 14.741176470588234],
 ['08', 10.25],
 ['06', 9.022727272727273],
 ['10', 13.440677966101696],
 ['18', 13.20183486238532],
 ['19', 10.8],
 ['03', 7.796296296296297]]

Sorting and Printing Values from a List of Lists

In [17]:
swap_avg_by_hour = []
for row in avg_by_hour:
    swap_avg_by_hour.append([row[1],row[0]])
print(swap_avg_by_hour)
sorted_swap = sorted(swap_avg_by_hour,reverse=True)
sorted_swap

[[11.383333333333333, '01'], [16.796296296296298, '16'], [9.41095890410959, '12'], [16.009174311926607, '21'], [23.810344827586206, '02'], [38.5948275862069, '15'], [8.127272727272727, '00'], [7.170212765957447, '04'], [7.985294117647059, '23'], [7.852941176470588, '07'], [6.746478873239437, '22'], [11.46, '17'], [11.051724137931034, '11'], [13.233644859813085, '14'], [21.525, '20'], [10.08695652173913, '05'], [5.5777777777777775, '09'], [14.741176470588234, '13'], [10.25, '08'], [9.022727272727273, '06'], [13.440677966101696, '10'], [13.20183486238532, '18'], [10.8, '19'], [7.796296296296297, '03']]
Out[17]:
[[38.5948275862069, '15'],
 [23.810344827586206, '02'],
 [21.525, '20'],
 [16.796296296296298, '16'],
 [16.009174311926607, '21'],
 [14.741176470588234, '13'],
 [13.440677966101696, '10'],
 [13.233644859813085, '14'],
 [13.20183486238532, '18'],
 [11.46, '17'],
 [11.383333333333333, '01'],
 [11.051724137931034, '11'],
 [10.8, '19'],
 [10.25, '08'],
 [10.08695652173913, '05'],
 [9.41095890410959, '12'],
 [9.022727272727273, '06'],
 [8.127272727272727, '00'],
 [7.985294117647059, '23'],
 [7.852941176470588, '07'],
 [7.796296296296297, '03'],
 [7.170212765957447, '04'],
 [6.746478873239437, '22'],
 [5.5777777777777775, '09']]
In [19]:
print('Top 5 Hours for Ask Posts Comments')
for avg,hr in sorted_swap[:5]:
    print('{}: {:.2f} average comments per post'.format(dt.datetime.strptime(hr, "%H").strftime("%H:%M"),avg))

Top 5 Hours for Ask Posts Comments
15:00: 38.59 average comments per post
02:00: 23.81 average comments per post
20:00: 21.52 average comments per post
16:00: 16.80 average comments per post
21:00: 16.01 average comments per post
In [ ]: