Clustering with Scikit-learn in Python¶

A tutorial by Thomas Jurczyk, CERES (Bochum)

This notebook includes the code used in my clustering tutorial for the Programming Historian. Please note that you need to save the datasets DNP_ancient_authors.csv and RELIGION_abstracts.csv in the a folder called data. The datasets are available in my GitHub repository.

If you have any questions or comments, please contact me via email.

DNP Ancient Authors Data¶

1. Loading the Datasets & Exploratory Data Analysis¶

In the first step, we will load the DNP_ancient_authors.csv dataset into our program and look at some overview statistics.

In the next step, we print out the first five rows and look at some information and overview statistcs about each dataset using pandas' info() and describe() methods.

As we can see when using describe() on the df_authors dataset, we have an overall huge standard deviation in almost every column and a huge difference between the 75% percentil value and the maxim value. This indicates that we might have some serious outliers in our dataset, and it might make sense to get rid of them before we continue with our analysis. Therefore, we only keep those data points in our dataframe with a word_count within the 95% percentil range.

2. Analyzing the DNP Ancient Authors Dataset¶

We start our analysis by clustering the DNP_ancient_authors.csv dataset. Before we start with the actual clustering process, we first import all the necessary library and write a couple of functions that will help us to plot our results during the analysis. We will also use these functions and imports in the second part of our analysis.

Before continuing with the following part, you need to install yellowbrick first if you haven't done so already. You can either do this via

pip install yellowbrick

or

conda install yellowbrick

when using the Anaconda distribution.

3. Standardizing the DNP Ancient Authors Dataset¶

Next, we initialize scikit-learn's StandardScaler() to standardize our data.

4. Feature Selection¶

This next section is a little fuzzy. Selecting features in an unsupervised learning context is difficult. I have decided to implement a rather "simple" approach (see the function progressiveFeatureSelection). This attempt is based on this algorithm presented on stackexchange. Of course, there are more elaborate attempts to select features in an unsupervised learning context, such as presented in this article. However, the method implemented here turned out to be quite successful in the context of the datasets used in this tutorial.

For the sake of simplicity and since we only have ten features in the DNP_ancient_authors.csv dataset, we are looking for the ideal three features to cluster our data.

It turns out that the features known_works_standardized, commentaries_standardized, and modern_editions_standardized might be worth using when trying to cluster our data. We are creating a new dataframe with only these three features.

5. Figuring Out the Right Amount of Clusters¶

We will now apply the elbow method and then use silhouette plots to get an impression of how many clusters we should choose to analyse our dataset. We will check for two to ten clusters. Note, however, that the feature selection was also done with a pre-defined k-means algorithm using n=3 clusters. Thus, our three selected features might already have a tendency towards this amount of clusters, since they turned out to be the best choice under the parametric circumstances of n=3 cluster.

Looking at the elbow plot indeed shows us that an "elbow" at n=3 as well as n=5 clusters. However, it's still pretty difficult to decide whether to use three, four, five or even six clusters. Therefore, we should also look at the silhouette plots.

Looking at the silhouette scores underlines our previous intuition that a selection of n=3 or n=5 seems to be the right choice of clusters. The silhouette plot with n=3 clusters in particular has a relatively high average silhouette score. Yet, since the two other clusters are far below the average silhouette score for n=3 clusters, we decide to analyze the dataset with k-means using n=5 clusters. However, the different sizes of the “knives” and their sharp form in both n=3 and n=5 clusters indicate a single dominant cluster and a couple of rather small and less cohesive clusters.

6. n=5 K-Means Analysis of the DNP Ancient Authors Dataset¶

Let us now train an k-means instance with n=5 clusters and plot the results using seaborn. Since we are focusing the plotting part in this tutorial on two dimensional plots, we wil us PCA() (Principal Component Analysis) to reduce the dimensionality to two dimensions for plotting reasons only.

This looks interesting! Even though the data is not perfectly clustered (which is something that we could already see during the evaluation of the silhouette plot), we can clearly separate at least one to two clusters. However, the two smaller ones still include quite a few outliers, which is a common problem of k-means.

Let us now look at the entries in the three clusters and see if this actually delivered any valuable insights into our data.

This provided us with quite some interesting results. Note, however, that many of the known authors are missing due to our initial cut off of 10% of the data. In an actual research article, this might be a bad idea, however, it facilitated our clustering process.

Religion Abstracts Data¶

The second part of this tutorial is going to deal with textual data, namely all abstracts scraped from the Religion (journal) website. We will try to cluster the abstracts based on their word features in the form of TF-IDF vectors (which is short for "Text Frequency - Inverted Document Frequency").

1. Loading the Dataset & Exploratory Data Analysis¶

Similar to the analysis of the DNP_ancient_authors.csv dataset, we will first load the RELIGION_abstracts.csv into our program and look at some descriptive statistics.

2. TF-IDF Vectorization¶

In order to process the textual data with clustering algorithms, we need to convert the texts to vectors. For this purpose, we are using the scikit-learn implementation of TF-IDF vectorization. For a good introduction to how TF-IDF works, see this great tutorial by Melanie Walsh.

As an optional step, I have implemented a function called lemmatizeAbstracts() that lemmatizes the abstracts for us. Since we are not interested in stylistic similiarities between the abstracts, this might help us to reduce the overall amount of features (words) in our dataset. As part of the function, we are also cleaning the text of all punctuation and other noise such as brackets etc. In the following part, we are only working with the lemmatized version of the abstracts. However, if you feel like it, you can also continue using the original texts.

Optional Step: Lemmatization of Abstracts (using spacy)¶

# lemmatization (optional step)
import spacy
import re
nlp = spacy.load("en_core_web_sm")

def lemmatizeAbstracts(x):
        doc = nlp(x)
        new_text = []
        for token in doc:
            new_text.append(token.lemma_)
        text_string = " ".join(new_text)
        # getting rid of non-word characters
        text_string = re.sub(r"[^\w\s]+", "", text_string)
        text_string = re.sub(r"\s{2,}", " ", text_string)
        return text_string

df_abstracts["abstract_lemma"] = df_abstracts["abstract"].apply(lemmatizeAbstracts)
df_abstracts.to_csv("data/RELIGION_abstracts_lemmatized.csv")

I recommend saving the new lemmatized version of the dataset so that we do not have to redo the lemmatization each time we restart our notebook.

The first step is to instantiate our TF-IDF model by passing it the argument to ignore stop words, such as "the," "a," etc. The second step is pretty similar to the training of our K-Means instance in the previous part: We are passing the abstracts from our dataset to the vectorizer in order to convert them to machine-readable vectors. For the moment, we are not passing any additional arguments. Finally, we create a new pandas DataFrame object based on the TF-IDF matrix of our textual data.

Our initial matrix is huge and includes over 8,000 words from the overall vocabulary of the 701 abstracts. This is obviously too much, not only from a computational perspective, but also because clustering algorithms such as k-means loose much of their power due to the so-called curse of dimensionality. We will thus need to significantly reduce the number of features. To do so, we first create a new instance of the df_abstracts_tfidf with reduced maximum features to 250. We also tell the model to only consider words from the vocabulary that appear in at least five different documents but not more than 100 times. We also add the possibility to not only include single words, but bigrams as well (such as "19th century"). Finally, we tell our model to get rid of any potential accents.

Secondly, we are also using the Principal Component Analysis (PCA), a technique that is often applied to reduce the dimensionality of datasets.

PCA allows us to reduce the dimensionality of the original data substantially while retaining most of the salient information. On the PCA-reduced feature set, other machine learning algorithms—downstream in the machine learning pipeline—will have an easier time separating the data points in space (to perform tasks such as anomaly detection and clustering) and will require fewer computational resources. (quote from the online version of Ankur A. Patel: Hands-On Unsupervised Learning Using Python, O'Reilly Media 2020)

3. Dimensionality Reduction Using PCA¶

Due to the curse of dimensionality when using k-means, let us next use PCA() to caste the dimension from d=250 to d=10.

4. Applying K-Means Clustering on Textual Data¶

Next, we try to figure out if we can find any clusters in the abstracts using k-means. As we did in case of the DNP_ancient_authors.csv dataset, we will start by looking for the right amount of cluster using the elbow method and the silhouette score.

Works pretty well!

5. Applying DBSCAN Clustering on Textual Data¶

Even though the k-means clustering of our data already provided us with some valuable results from a clustering perspective, it might still be interesting to apply a different clustering algorithm such as DBSCAN. As explained above, DBSCAN allows for outliers in our data, meaning that it focuses on those regions in our data that may rightfully be called dense.

We will be using the d=10 reduced version of our RELIGION_abstracts.csv dataset, which allows us to keep euclidean distance as a metric. If we were to use the initial TF-IDF matrix with 250 vectors, we should maybe change the underlying metric to cosine distance, which is better when dealing with sparse matrices such as in this example.

The first step will be to figure out which eps value is most suitable for our data.

Using 0.2 as eps value seems to be reasonable.

6. Visualizing DBSCAN Results¶