Analysing Comment Responses in Hacker News (HN)

We are going to analyse comment response in Hacker News data set specifically on 'Ask HN'and 'Show HN'postings. We want to know which one receive more comments on average. We also want to know whether posts created on certain time receive more comments on average? To do this, we will download the data sets from here. The data set has been reduced from almost 300,000 rows to approximately 20,000 rows by removing all postings that did not receive any comments and then randomly sampling from the remaining submmissions.

The descriptions of the columns are as follow:

* id: The unique identifier from HN for the post
* title: The title of the post
* url : The URL that the posts link to, if the post has the URL
* num_points : The number of points the post acquired, calculated as the      total number of upvotes minus the total number of downvotes
* num_comments: The number of comments that were made on the post
* author : The username of the person who submitted the post
* created_at: The data and time at which the post was submitted
In [17]:
opened_file = open('hacker_news.csv')
from csv import reader
read_file = reader(opened_file)
hn = list(read_file)
headers = hn[0]
hn = hn[1:]

# separating headers and the rest of the rows

print(headers)
hn[:5]
['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']
Out[17]:
[['12224879',
  'Interactive Dynamic Video',
  'http://www.interactivedynamicvideo.com/',
  '386',
  '52',
  'ne0phyte',
  '8/4/2016 11:52'],
 ['10975351',
  'How to Use Open Source and Shut the Fuck Up at the Same Time',
  'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/',
  '39',
  '10',
  'josep2',
  '1/26/2016 19:30'],
 ['11964716',
  "Florida DJs May Face Felony for April Fools' Water Joke",
  'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/',
  '2',
  '1',
  'vezycash',
  '6/23/2016 22:20'],
 ['11919867',
  'Technology ventures: From Idea to Enterprise',
  'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429',
  '3',
  '1',
  'hswarna',
  '6/17/2016 0:01'],
 ['10301696',
  'Note by Note: The Making of Steinway L1037 (2007)',
  'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0',
  '8',
  '2',
  'walterbell',
  '9/30/2015 4:12']]

Separating 'Ask HN' and 'Show HN' lists from the rest of the Data Sets

First thing to do, is to separate the postings that we are going to analyse from the rest of the data sets. To do that we are going to create three empty lists called: ask_posts, show_posts and other_post.

We will calculate the length of posting that start with Ask HN, Show HN and Others.

In [18]:
ask_posts = []
show_posts = []
other_posts = []

for row in hn:
    title = row[1]
    title = title.lower() # the output will be in lowercase
    if title.startswith('ask hn'):
        ask_posts.append (row)
    elif title.startswith('show hn'):
        show_posts.append(row)
    else:
        other_posts.append(row)
print('Number of posts that start with ask hn:', len(ask_posts))
print('Number of posts that start with show hn:',len(show_posts))
print('Number of post that start with other:', len(other_posts))
Number of posts that start with ask hn: 1744
Number of posts that start with show hn: 1162
Number of post that start with other: 17194

Let's see the first five rows of the ask_posts list of lists:

In [19]:
ask_posts[:5]
Out[19]:
[['12296411',
  'Ask HN: How to improve my personal website?',
  '',
  '2',
  '6',
  'ahmedbaracat',
  '8/16/2016 9:55'],
 ['10610020',
  'Ask HN: Am I the only one outraged by Twitter shutting down share counts?',
  '',
  '28',
  '29',
  'tkfx',
  '11/22/2015 13:43'],
 ['11610310',
  'Ask HN: Aby recent changes to CSS that broke mobile?',
  '',
  '1',
  '1',
  'polskibus',
  '5/2/2016 10:14'],
 ['12210105',
  'Ask HN: Looking for Employee #3 How do I do it?',
  '',
  '1',
  '3',
  'sph130',
  '8/2/2016 14:20'],
 ['10394168',
  'Ask HN: Someone offered to buy my browser extension from me. What now?',
  '',
  '28',
  '17',
  'roykolak',
  '10/15/2015 16:38']]

And we can also see the first five rows of show_posts list of lists as follow:

In [20]:
show_posts[:5]
Out[20]:
[['10627194',
  'Show HN: Wio Link  ESP8266 Based Web of Things Hardware Development Platform',
  'https://iot.seeed.cc',
  '26',
  '22',
  'kfihihc',
  '11/25/2015 14:03'],
 ['10646440',
  'Show HN: Something pointless I made',
  'http://dn.ht/picklecat/',
  '747',
  '102',
  'dhotson',
  '11/29/2015 22:46'],
 ['11590768',
  'Show HN: Shanhu.io, a programming playground powered by e8vm',
  'https://shanhu.io',
  '1',
  '1',
  'h8liu',
  '4/28/2016 18:05'],
 ['12178806',
  'Show HN: Webscope  Easy way for web developers to communicate with Clients',
  'http://webscopeapp.com',
  '3',
  '3',
  'fastbrick',
  '7/28/2016 7:11'],
 ['10872799',
  'Show HN: GeoScreenshot  Easily test Geo-IP based web pages',
  'https://www.geoscreenshot.com/',
  '1',
  '9',
  'kpsychwave',
  '1/9/2016 20:45']]

Now we will determine between ask posts and show posts which one receive more comments on average.

In [21]:
total_ask_comments = 0
for row in ask_posts:
    num_comments = int(row[4])
    total_ask_comments += num_comments
    avg_ask_comments = total_ask_comments/len(ask_posts)
print(avg_ask_comments)
    
14.038417431192661

There is 14.0384 average number of comments for Ask HN postings. Let's check the show_posts list of lists:

In [22]:
total_show_comments = 0
for row in show_posts:
    num_comments = int(row[4])
    total_show_comments += num_comments
    avg_show_comments = total_show_comments/len(show_posts)
print(avg_show_comments)
10.31669535283993

So, there is 10.3167 average number of comments for Show HN postings. Hence, it appears that the Ask HN receive more comments than the Show HN.

Knowing ask posts receives more comments, we will be focusing our analysis on ask posts and at this time our attention would be centered on the timing when ask posts is created and how many comments is generated at each time it is created. To do this, we will do the following steps:

1. Calculate the number of ask posts created at each hour of the days 
along with the number of comments received
2. Calculate the average number of comments ask posts receive by hour    created.
In [24]:
import datetime as dt # importing the datetime module
result_list = []
for row in ask_posts:
    created_at = row[6]
    comments_at = int(row[4])
    result_list.append(created_at)
    result_list.append(comments_at)
print(result_list[:1])

counts_by_hour = {}
comments_by_hour = {}
date_format = '%m/%d/%Y %H:%M'
for value in result_list:
    date = value[0]
    comment = value[1]
    time = dt.datetime.strptime(creation_date, date_format).strftime('%H')
    if time in counts_by_hour:
        comments_by_hour[time] += comment
        counts_by_hour[time] += 1
    else:
        comments_by_hour[time] = comment
        coounts_by_hour[time] = 1
print(comments_by_hour)
print(counts_by_hour)
    
    
    
['8/16/2016 9:55']
ValueErrorTraceback (most recent call last)
<ipython-input-24-7d69c3b9ef60> in <module>()
     14     date = value[0]
     15     comment = value[1]
---> 16     time = dt.datetime.strptime(creation_date, date_format).strftime('%H')
     17     if time in counts_by_hour:
     18         comments_by_hour[time] += comment

/usr/lib/python3.4/_strptime.py in _strptime_datetime(cls, data_string, format)
    498     """Return a class cls instance based on the input string and the
    499     format string."""
--> 500     tt, fraction = _strptime(data_string, format)
    501     tzname, gmtoff = tt[-2:]
    502     args = tt[:6] + (fraction,)

/usr/lib/python3.4/_strptime.py in _strptime(data_string, format)
    335     if not found:
    336         raise ValueError("time data %r does not match format %r" %
--> 337                          (data_string, format))
    338     if len(data_string) != found.end():
    339         raise ValueError("unconverted data remains: %s" %

ValueError: time data '8' does not match format '%m/%d/%Y %H:%M'