The Stack Exchange Websites welcomes every type of questions related to the subject in matter. For example, the Stack Exchange Website related to Data Science is populated with questions and queries related to the subject of Data Science. They vary from how to plot a graph, to questions related to a programming language such as Python.¶
The range of subdivisions on the Data Science Stack Exchange (DSSE) go from popular libraries like Pandas, Keras, Tensorflow to the programming language Python, passing trough Statistics, Predictive-modeling, Time-series, regression, etc.¶
Apart from all this information, we can also explore all the Questions that were made in the website, and some data related to them, such as how many answers each question had, votes, views.¶
Maybe we could take the approach of using as a proxy the most popular - most viewed, most answered, and most voted - questions/tags as one of the hottest topics for Data Science enthusiasts.¶
After drilling down into the several tables available in the Stack Exchange Data Explorer (SEDE) - https://data.stackexchange.com/datascience/query/new - we found that the most valuable ones for our strategy would be both the Posts as the Tags. Combining both tables we can get a good insight into the most popular Data Science topics, the ones with most comments, favorites, and most answered ones, and combine that data with the Tags being more frequently used.¶
After running the following query:¶
SELECT PostTypeId as type_of_posts, COUNT(*) as nº_of_posts
FROM Posts
GROUP BY PostTypeId
ORDER BY nº_of_posts DESC;
One could check that the two most numerous post types are the 2 and the 1, Answer and Question. Let's now focus on the Questions side.¶
Now, pulling the data for the Posts related only to the recent Questions - from 2019 onwards:¶
SELECT Id,
PostTypeId,
CreationDate,
Score,
ViewCount,
Tags,
AnswerCount,
FavoriteCount
FROM Posts
WHERE PostTypeId = 1 AND CreationDate >= '2019-01-01'
ORDER BY CreationDate;
Now lets read into the file created from the query above:¶
In [1]:
import numpy as np
import pandas as pd
posts = pd.read_csv('2019_questions.csv', parse_dates=['CreationDate'])
# Creating a sample of five rows from the newly created posts DataFrame:
posts.sample(5)
Out[1]:
In [2]:
posts.info()
In [3]:
posts['FavoriteCount'].value_counts(dropna=False)
Out[3]:
In [4]:
posts['Tags'].value_counts()
Out[4]:
We have a tremendous amount of missing (NaN) datapoints in our posts Dataframe, mainly in our FavoriteCount column. Where 7432 datapoints, out of 8839, are NaN's. Apart from this particular col we do not have any other missing values in our Dataframe.¶
From a quick search we found out that this missing values might correspond to Questions that have never been favorited, hence felt into the forgotten category. And the difference between these and the 0.0 values, in the FavoriteCount column, is that the last were once favorited Questions that lost their favorited pedigree to the extent of 0. - https://meta.stackexchange.com/questions/327680/why-do-some-questions-have-a-favorite-count-of-0-while-others-have-none¶
We have two solutions to fix this missing values in our Dataframe. The first solution would be to simply erase the rows, the second solution would be to assign to all the rows that have missing values, and based on our findings, the value of 0, implying that these rows have zero favorite votes. Due to the size of the missing values in our Dataframe we would suggest to opt for the second solution, thus not losing a lot of data.¶
Regarding our Tags col we could, in order to favour the analysis, and to smooth the results, further group the column, or we could also treat each group of tags as an individual tag. But first of all we should separate each tag properly, i e, with a comma (,).¶
First let's fill in the missing values in our FavoriteCount column with zeros (0). And then changing the col type from a float to a integer one:¶
In [5]:
#resorting to the fillna method:
posts['FavoriteCount'] = posts['FavoriteCount'].fillna(0)
posts.sample(5)
Out[5]:
In [6]:
posts['FavoriteCount'] = posts['FavoriteCount'].astype(int)
posts.info()
Its time now to clean the Tags column and separate each different tag by a comma (,):¶
In [7]:
posts['Tags'] = posts['Tags'].str.replace('><', ',').copy()
posts['Tags'] = posts['Tags'].str.replace('<', ''). copy()
posts['Tags'] = posts['Tags'].str.replace('>', '').copy()
posts.head()
Out[7]:
Now lets check, from the different Tags, which are the most popular ones, using two distinct popularity proxies: first we will count the number of times that each Tag was used, and come up with a top5. Then we will use the parameter ViewCount to grasp, among the top5 most used Tags, which one of the five was most viewed.¶
We are going to calculate the number of times each Tag was used, first by spliting on the commas (,) each individual Tag from their counterparts, then by stacking the full Dataframe that resulted from that, and finally counting the number of times that each different Tag was used:¶
In [8]:
tags_count = posts['Tags'].str.split(',', expand=True).stack().value_counts()
tags_top5 = tags_count.head(5)
print(tags_top5)
Plotting the above results:¶
In [9]:
import matplotlib.pyplot as plt
%matplotlib inline
#Plotting the tags_top5 DataFrame in an horizontal bar graph:
tags_top5_graph = tags_top5.plot.barh(
edgecolor='none',
color = [(255/255,188/255,121/255),
(162/255,200/255, 236/255),
(207/255,207/255,207/255),
(200/255,82/255,0/255),
(255/255,194/255,10/255)])
#ENHANCING PLOT AESTHETICS:
#Removing all the 4 spines with a for loop from our graph figure:
for key, spine in tags_top5_graph.spines.items():
spine.set_visible(False)
#Removing the ticks:
tags_top5_graph.tick_params(
bottom='off', top='off', left='off', right='off')
#Setting a graph title:
tags_top5_graph.set_title('Top5 Tags by Number of Usage')
#Setting an average graph line:
tags_top5_graph.axvline(tags_top5.mean(),
alpha=.8, linestyle='--', color='grey')
# Displaying the graph:
plt.show()
In [10]:
posts[posts['Tags'].str.contains('machine-learning')]['ViewCount'].sum()
Out[10]:
Now lets check, from the top5 Tags calculated above, which one is the most viewed. We will do so using the str.contains() method combined with a mask to filter only the top5 Tags in our posts Dataframe, and then, concentrating on the ViewCount col, adding all the times each tag was viewed:¶
In [11]:
#creating the tags_top5_views DataFrame(DF):
tags_top5_views = pd.DataFrame(
columns=tags_top5.index,
index=['Total Views'])
for r in range(0,5):
df = tags_top5
n_views = posts[posts['Tags'].str.contains(df.index[r])]['ViewCount'].sum()
print(str(df.index[r]) + '__total-views:',n_views)
#filling in the tags_top5_views DF with the name of the columns and respective
#number of views:
col = tags_top5_views.columns[r]
tags_top5_views[col] = [n_views]
Plotting the above results:¶
In [12]:
#Plotting the tags_top5_views DataFrame in a bar graph:
tags_top5_views_graph = tags_top5_views.plot.bar(edgecolor='none',
color = [(255/255,188/255,121/255),
(162/255,200/255, 236/255),
(207/255,207/255,207/255),
(200/255,82/255,0/255),
(255/255,194/255,10/255)])
#ENHANCING PLOT AESTHETICS:
#Removing all the 4 spines with a for loop from our graph figure:
for key, spine in tags_top5_views_graph.spines.items():
spine.set_visible(False)
#Removing the ticks from the graph:
tags_top5_views_graph.tick_params(
top ='off',
bottom = 'off',
right = 'off',
left = 'off')
# Setting up a legend box for our bar graph:
tags_top5_views_graph.legend(
loc='upper right',
labels=(tags_top5_views.columns),
ncol=1, fancybox=True, framealpha=.6,
prop={'size': 10})
#Rotating the xtick labels:
plt.xticks(rotation='horizontal')
tags_top5_views_graph.axhline(n_views.mean(),
color='grey',
alpha=.8,
linestyle=':')
plt.show()
It is clear that among our top5 Tags there are two that stand out: Machine-Learning (ML) and Python (Py). Not only are these two Tags the most used (ML-2693 times; Py-1814 times) but also the most viewed ones (Py-541691 views; ML-398666 views).¶
We've got two pretty good potential candidates for our assignment, and two compimentlary ones that can even be combined into a major one: Python and Machine-Learning.¶
In [13]:
posts[posts['Tags'].apply(
lambda tags: True if 'python' and 'tensorflow' in tags else False)]
Out[13]:
The use of pairs or an even bigger group of tags grouped together, along our Dataframe, is probably related to their intrinsic relationship, dependency in one another, and common usage. For example, keras is a library used 'inside' Pythons ecosystem, such as scikit-learn. Tensorflow is used with Python through a dedicated API. And all these tools, apart from being used along with Python, are being used to 'train'/'teach'/'feed' algorithms, or in other words, to do machine-learning/deep-learning. So now we can understand why so many of these tags are grouped together in our Dataframe.¶
Prior to summarizing our findings let's dig deeper into Deep Learning and check whether or not this trend as come to stay.¶
In [14]:
all_questions = pd.read_csv('all_questions.csv', parse_dates=['CreationDate'])
all_questions.head()
Out[14]:
Doing the same process as we've done above, and cleaning the Tags column separating each different tag by a comma (,):¶¶
In [15]:
all_questions['Tags'] = all_questions['Tags'].str.replace('><', ',')
all_questions['Tags'] = all_questions['Tags'].str.replace('<', '')
all_questions['Tags'] = all_questions['Tags'].str.replace('>', '')
all_questions.sample(5)
Out[15]:
We will define a cut-off in terms of what we consider a deep-learning question or not, and we are only assuming questions related to the deep-learning topic as question with one, and one only deep-learning Tag in it.¶
In [16]:
deep_learning = all_questions[all_questions['Tags'] == 'deep-learning']
#Sorting the Dataframe per Date:
deep_learning.sort_values(by='CreationDate', inplace=True)
deep_learning.head()
Out[16]:
In [17]:
deep_learning.tail()
Out[17]:
It's clear that the range of our deep-learning sample varies from 2015 and 2020. Although for 2020 the size of the sample is too small, only contaning one month.¶
Now it is time to group together our deep-learning data, based on its year:¶
In [18]:
deep_learning_grp = deep_learning.groupby(
deep_learning.CreationDate.dt.year).sum().sort_values(by='Id')
deep_learning_grp.head(10)
Out[18]:
Applying the same process as above, to the all_questions Dataframe:¶
In [19]:
all_questions_grp = all_questions.groupby(
all_questions.CreationDate.dt.year).sum()
all_questions_grp.head(10)
Out[19]:
Lets now merge the two Dataframes into one, for the sake of our analysis, and proceed with some comparisons and conclusions:¶
In [20]:
deep_all = pd.merge(all_questions_grp, deep_learning_grp, how='left',
left_index=True, right_index=True)
deep_all = deep_all.rename(
columns={'Id_x':'all_questions','Id_y':'deep_learning' })
deep_all.head(10)
Out[20]:
In order to adjust both samples and to make the comparison cleaner we will drop rows 2014, since there is no data for the deep-learning Tag for this period, and also the row 2020, due to the lack of sufficient data for these year in particular. This way our comparisons are more robust and consistent:¶
In [21]:
deep_all = deep_all.drop([2014, 2020], axis=0)
deep_all.head(10)
Out[21]:
Let us now make another test, and compare the number of deep_learning Tags against all the questions made in the Stack Exchange website, in order to demonstrate and validate its growth in terms of all the questions ever made. Making a kind of common-size analysis:¶
In [22]:
deep_all['%_deep_learning'] = (deep_all[
'deep_learning']/deep_all['all_questions'])*100
deep_all['date'] = deep_all.index
#dropping index (other way could be df.index.name=None):
deep_all.reset_index(drop=True, inplace=True)
deep_all = deep_all[['date', 'all_questions', 'deep_learning', '%_deep_learning']]
deep_all.head(10)
Out[22]:
From a first glimpse we can observe an upward trend along the years, in the use of the deep_learning Tags. In 2015 their number were around 12000, as in 2019 these numbers climbed to figures around 1600000. This is an impressive growth.¶
In terms of their percentage, among all questions made in the Stack Exchange website, the growth trend is also there, albeit not so strong and not so linear. Now let's visualize it.¶
Plotting the results:¶
In [23]:
fig = plt.figure(figsize=(6,7))
ax_spines = ['right', 'left', 'bottom', 'top']
ax1 = fig.add_subplot(3,1,1)
ax2 = fig.add_subplot(3,1,2)
ax3 = fig.add_subplot(3,1,3)
x = deep_all['date']
xi = list(range(len(x)))
plt.xlabel=('x')
plt.xticks(xi, x)
ax1.plot(deep_all['all_questions'], color='green', linestyle='-.')
#Setting the yticks for ax1:
ax1.set_yticks([deep_all['all_questions'].min(), deep_all['all_questions'].max()/2,
deep_all['all_questions'].max()])
ax2.plot(deep_all['deep_learning'], color = 'orange', linestyle='-.')
#Setting the yticks for ax2:
ax2.set_yticks([deep_all['deep_learning'].min(), deep_all['deep_learning'].max()/2,
deep_all['deep_learning'].max()])
ax3.plot(deep_all['%_deep_learning'], color='grey', linestyle='-.')
#Setting the yticks for ax3:
ax3.set_yticks([0, 0.25, 0.5])
# Giving each graph a title:
ax1.set_title('All Questions')
ax2.set_title('Deep Learning Questions')
ax3.set_title('% of Deep Learning Questions')
#ENHANCING PLOT AESTHETICS:
#Removing the ticks from the graph:
for i in range(3):
ax = fig.add_subplot(3,1,i+1)
ax.tick_params(
top ='off',
bottom = 'off',
right = 'off',
left = 'off')
#Removing all the 4 spines with a for loop from our graph figure:
for i in range(0,4):
ax.spines[ax_spines[i]].set_visible(False)
#Removing unnecessary x labels:
for i in range(1,3):
ax = fig.add_subplot(3,1,i)
ax.tick_params(labelbottom=False)
plt.show()
