Created by Nathan Kelber and Ted Lawless for JSTOR Labs under Creative Commons CC BY License
For questions/comments/improvements, email nathan.kelber@ithaka.org.
___
Latent Dirichlet Allocation (LDA) Topic Modeling
Description: This notebook demonstrates how to do topic modeling. The following processes are described:
tdm_client
to retrieve a datasetpyldavis
Use Case: For Researchers (Less explanation, better for research pipelines)
Difficulty: Intermediate
Completion time: 30 minutes
Knowledge Required:
Knowledge Recommended:
Data Format: JSON Lines (.jsonl)
Libraries Used:
pandas
to load a preprocessing listcsv
to load a custom stopwords listpyldavis
to visualize our topic modelResearch Pipeline
Topic modeling is a machine learning technique that attempts to discover groupings of words (called topics) that commonly occur together in a body of texts. The body of texts could be anything from journal articles to newspaper articles to tweets.
Topic modeling is an unsupervised, clustering technique for text. We give the machine a series of texts that it then attempts to cluster the texts into a given number of topics. There is also a supervised, clustering technique called Topic Classification, where we supply the machine with examples of pre-labeled topics and then see if the machine can identify them given the examples.
Topic modeling is usually considered an exploratory technique; it helps us discover new patterns within a set of texts. Topic Classification, using labeled data, is intended to be a predictive technique; we want it to find more things like the examples we give it.
We'll use the tdm_client library to automatically retrieve the dataset in the JSON file format.
Enter a dataset ID in the next code cell.
If you don't have a dataset ID, you can:
# Creating a variable `dataset_id` to hold our dataset ID
# The default dataset is Shakespeare Quarterly, 1950-present
dataset_id = "7e41317e-740f-e86a-4729-20dab492e925"
Next, import the tdm_client
, passing the dataset_id
as an argument using the get_dataset
method.
# Importing your dataset with a dataset ID
import tdm_client
# Pull in the dataset that matches `dataset_id`
# in the form of a gzipped JSON lines file.
dataset_file = tdm_client.get_dataset(dataset_id)
If you completed pre-processing with the "Exploring Metadata and Pre-processing" notebook, you can use your CSV file of dataset IDs to automatically filter the dataset. Your pre-processed CSV file must be in the root folder.
# Import a pre-processed CSV file of filtered dataset IDs.
# If you do not have a pre-processed CSV file, the analysis
# will run on the full dataset and may take longer to complete.
import pandas as pd
import os
pre_processed_file_name = f'data/pre-processed_{dataset_id}.csv'
if os.path.exists(pre_processed_file_name):
df = pd.read_csv(pre_processed_file_name)
filtered_id_list = df["id"].tolist()
use_filtered_list = True
print('Pre-Processed CSV found. Successfully read in ' + str(len(df)) + ' documents.')
else:
use_filtered_list = False
print('No pre-processed CSV file found. Full dataset will be used.')
# Load a custom data/stop_words.csv if available
# Otherwise, load the nltk stopwords list in English
# The filename of the custom data/stop_words.csv file
stopwords_list_filename = 'data/stop_words.csv'
if os.path.exists(stopwords_list_filename):
import csv
with open(stopwords_list_filename, 'r') as f:
stop_words = list(csv.reader(f))[0]
print('Custom stopwords list loaded from CSV')
else:
# Load the NLTK stopwords list
from nltk.corpus import stopwords
stop_words = stopwords.words('english')
print('NLTK stopwords list loaded')
def process_token(token):
token = token.lower()
if token in stop_words:
return
if len(token) < 4:
return
if not(token.isalpha()):
return
return token
# Limit to n documents. Set to None to use all documents.
limit = 500
n = 0
documents = []
for document in tdm_client.dataset_reader(dataset_file):
processed_document = []
document_id = document["id"]
if use_filtered_list is True:
# Skip documents not in our filtered_id_list
if document_id not in filtered_id_list:
continue
unigrams = document.get("unigramCount", [])
for gram, count in unigrams.items():
clean_gram = process_token(gram)
if clean_gram is None:
continue
processed_document.append(clean_gram)
if len(processed_document) > 0:
documents.append(processed_document)
n += 1
if (limit is not None) and (n >= limit):
break
Build a gensim dictionary corpus and then train the model. More information about parameters can be found at the Gensim LDA Model page.
import gensim
dictionary = gensim.corpora.Dictionary(documents)
doc_count = len(documents)
num_topics = 7 # Change the number of topics
# Remove terms that appear in less than 10% of documents and more than 75% of documents.
dictionary.filter_extremes(no_below=10 * .10, no_above=0.75)
bow_corpus = [dictionary.doc2bow(doc) for doc in documents]
# Train the LDA model.
model = gensim.models.LdaModel(
corpus=bow_corpus,
id2word=dictionary,
num_topics=num_topics
)
Print the most significant terms, as determined by the model, for each topic.
for topic_num in range(0, num_topics):
word_ids = model.get_topic_terms(topic_num)
words = []
for wid, weight in word_ids:
word = dictionary.id2token[wid]
words.append(word)
print("Topic {}".format(str(topic_num).ljust(5)), " ".join(words))
Visualize the model using pyLDAvis
. This visualization can take a while to generate depending on the size of your dataset.
import pyLDAvis.gensim
pyLDAvis.enable_notebook()
pyLDAvis.gensim.prepare(model, bow_corpus, dictionary)