MIS 670 Final Project¶

Jace Bothner

Watch the video presentation here:
https://youtu.be/EF8a5sYS6Co

If you're not able to view the interactive visuals try viewing the project on my github page:
https://github.com/jbothner21/MIS_670_Final_Project

Table of Contents

1. Project Introduction and Problem Definition¶

a. Purpose¶

The purpose of this project is to explore how lyrical content in popular music has changed over time.

Music, while not only an art form and medium of expression, is also a multi-billion dollar a year industry. Revenue aside, one of music's main purposes is serving as cultural expression. Therefore, the most popular songs within a given time are representative of that time. While many can identify musical styles of certain generations, it's worth asking how the lyrical content of these periods differ. In this project, we'll explore how lyrical content has evolved and how it impacts the future of lyricism.

b. Hypotheses¶

While exploring lyrical content, we'll test several common claims made about the evolution of pop music. These claims are:

1. Song lyrics have become more repetitive over time
2. Lyrics in songs have become more verbose, in other words more text heavy
3. Song lyrics have become sadder, angrier, or generally more negative over time

2. Data Identification and Collection¶

To gather song lyrics that are most representative of their time period we'll use a dataset from data scientist Steve Hawtin. This dataset features the top 5000 songs of all time. In order to compile and rank this list, Hawtin compiled chart positions from 230 popular music charts like Billboard, RIAA, and Rolling Stone. Then he assigned each song an overall score by incorporating appearances on each chart weighted by the importance of the chart. For more information on this process and methodology, visit the link below.
https://tsort.info/music/faq_site_generation.htm

For this project, we'll focus on certain distinct periods of music. The easiest way to group these periods is by decades. We'll start with the 1950s and continue until the 2010s, the most recent complete decade. Therefore, we'll filter out any songs before or after these time periods.

The top 5000 songs dataset contains the artist's name, the song name, and the year. In order to get the lyrics for each song, we'll have to create links to crawl and scrape information. In the cell below, we're creating the song name portion of the URL by hyphenating each word in the song title.

Here we'll use this same process for the artist portion of the URL.

Now we can combine these elements to create links using the MetroLyrics URL. Each URL on the metro lyrics site has its domain name, the name of the song, a lyrics tag, the name of the artist, and an html tag.

Now using the links we've creating, we'll crawl through each link to scrape the song's lyrics.

3. Data Cleaning and Text Preprocessing¶

Now that we have our lyrical information, it's important to clean the text before performing any analysis. This process involves removing any markup included in the text after scraping, any non-numeric or English characters, and other similar processes that would affect our analysis.

After cleaning the text we can check for any null values in the dataset. These null values are a result of scraping the lyrics from MetroLyrics' website. For example, some songs may be instrumental and not feature lyrics. Other songs may not have lyrical information because MetroLyrics is not authorized to display the lyrics. Another common occurrence is a discrepancy between an artist or a song name in the dataset and how it's featured on MetroLyrics' site. After dropping these null values, there are still 3120 with lyrics which will provide a reasonable sample size to analyze and draw conclusions from.

Here we'll define a function to process text that we can apply as needed. This function will tokenize the text, a process of breaking the text into separate words. It will also remove stopwords, the most common words in a language which won't be meaningful in our analysis. It's also worth noting that additional stop words have been set to remove non-lexical vocables, a music term for a nonsensical syllable used to sing a melody. Similarly, the processing function will remove short words under three characters since these words aren't meaningful in our analysis. The function also converts the tokens to lowercase since python is case sensitive. Lastly, the function will 'stem' words, a process of breaking a word down to its root and removing the suffix.

Now we can separate each song into its respective decade based on the year of its release. It's worth noting that songs aren't equally represented by decade in the dataset. Therefore, decades with a larger sample size might have a higher amount of credibility in our analysis. Although, the reasons for stark differences in sample size by decade are likely intuitive. Music in the 1950s likely wasn't marketed as aggressively as in subsequent decades and therefore is less represented on the top songs of all time. On the other end of the spectrum, songs in the 2010s have had less time to gain popularity and move up on the list of all time songs. After separating songs into decades we can apply the text processing function to process the lyrics for each decade.

4. Descriptive Analytics¶

a. Word frequency analysis¶

Word frequency analysis will show which words are most prevalent in the song lyrics. This type of analysis not only will show which words are used the most, but can provide an idea of what songs are about based on most frequent words. For example, the word frequency for the entire dataset (all decades) is shown below. The most popular word used in all songs is 'love' meaning that 'love songs' are likely most prevalent in pop songs.

Now we'll separate word frequency by decade to see how word frequency has changed over time. The word frequency for songs from the 1950s is shown below. It's already apparent based on the frequency of the word 'love' and similar words such as 'heart' and 'baby' how prevalent 'love songs' were in the 1950s, especially compared to the frequency of all lyrics.

The word frequency for the 1960s shown below tells a similar story to the 1950s. The word 'love' is by far most prevalent, and 'while' heart is no longer in the top 20 words, the word 'baby' is still one of the most prevalent words.

Word frequency for song lyrics in the 1970s shown below starts to shift compared to previous decades. While 'love' is still the most frequent, the margin between the first and second words is starting to close.

Word frequency for the 1980s tells a fairly similar story to the 1970s, and the gap between the word 'love' and the second most frequent closes ever so slightly.

Word frequency for the 1990s shown below makes the trend quite apparent. While 'love' is still the most frequent, it's becoming less frequent compared to other words.

Word frequency for the 2000s shown below, is very meaningful as it represents the first decade where the word 'love' does not appear most frequently. In fact, the word 'love' is only the third most frequent.

Word frequency for the 2010s continues the trend for the decline of 'love songs' as the word 'love' is only the fifth most frequent word in this decade. The trend is quite apparent: 'love songs' are declining in popularity. The reason for this trend is not as apparent. It's possible that 'love songs' are seen as too cliche, and seen as a thing of the past. Or it's possible that 'love songs' are still popular, but they're not as superficial or overstated as in past generations. It could be that 'love songs' of the present and the future are more ambiguous or open to interpretation, and less likely to overtly use the word 'love.' While this is conjecture, it is likely, however, that this trend will continue, and it should be expected that love songs, or at least the word 'love' in songs, won't be as popular as in the past.

b. Word cloud¶

Word clouds are a way to visualize word frequency. More frequent words will appear larger and those that are less frequent will appear smaller. In this context, word clouds can be useful to see the most prevalent words at a glance or can show slang or colloquialism of each generation, or even cultural events after closer inspection. The word cloud for the entire dataset (all decades) is shown below.

The word cloud for the 1950s is shown below. The word 'ditty' is an example of a popular slang term from this time.

The word cloud for the 1970s is shown below. 'Boogie' is a word characterized by the rise of disco in this decade and the word 'war' can be explained by social commentary on the war in Vietnam.

In the word cloud for the 2000s below, the word 'shawty' is an example of popular slang from this decade.

c. Descriptive statistics¶

1. Number of words per song¶

In this section, we'll examine how the number of words per song has changed over time. We can split the lyrics into words and take the average of those word counts to compare them between decades.

The dataframe above and the visualization of it in the bar graph below illustrates that there is indeed a positive correlation between average words per song and decade meaning that songs are indeed becoming more verbose. This doesn't necessarily mean that songs are getting longer though, just that the songs are more densely packed with lyrical content. In fact, one of the reasons for this trend is the adoption of rap into mainstream music. Rap music entered the mainstream pop scene in the late 1970s/early 1980s and has grown in popularity since. Although, rap is fairly saturated in pop music currently, which is explained by the similar average numbers between the 2000s and 2010s. In the future, I would expect this number to remain stagnant or increase slightly if any.

2. Song repetitiveness using Lempel-Ziv¶

This section will test the claim that pop songs have become more repetitive over time. To measure this we'll use an algorithm called Lempel-Ziv complexity. This compression algorithm forms the basis of many compression formats, including GIF, PNG, and ZIP. But this idea of compression can also be applied to song lyrics to illustrate repetitiveness. Data scientist Collin Morris explains why this is the case which you can read in full here. Essentially the idea is that the Lempel-Ziv algorithm finds repeated patterns in a string and removes them, leaving only the unique portions. Therefore, the amount of compression applied by the algorithm correlates with the repetitiveness of the song's lyrics. To see a demonstration of how this works in context see the link here.
https://pypi.org/project/lempel-ziv-complexity/
https://pudding.cool/2017/05/song-repetition/
http://colinmorris.github.io/pop-compression/

The dataframe below shows the five most repetitive pop songs of all time according to the Lempel-Ziv algorithm. This algorithm does appear to function well since the top song in the dataframe below is Daft Punk's 'Around the World' where the words 'around the world' are repeated 120 times and are the only lyrics in the song.

The dataframe above and the visualization of it in the bar graph below illustrates that there is indeed a positive correlation between compression amount (repetitiveness) and decade meaning that songs are indeed becoming more repetitive.

The correlation between year and amount of compression (repetitiveness) is shown below. In fact, there is a moderate amount of positive correlation between year and compression shown by the correlation coefficient, meaning that as time goes on songs are likely to become more repetitive. So there appears to be truth to the claim that lyrics in pop music have become more repetitive. I would expect this trend to continue in the future and song lyrics will most likely continue to become more repetitive.

5. Content Analytics¶

a. Sentiment analysis¶

For this project, sentiment analysis will identify the song's lyrical tone: whether a song's lyrical content is positive or negative. Sentiments are scored on a spectrum between -1 and 1. A song that has a sentiment score of -1 will be a negative song with an angry or sadder tone. A song with a sentiment score of 1 will have a positive sentiment and a happier tone. For this sentiment analysis, we'll use a python package called vaderSentiment. Like other methods and lexicons for sentiment analysis, vaderSentiment provides polarity and subjectivity scores for the given text. However, what sets vaderSentiment apart from other methods and lexicons is its ability to process not only common dictionary words, but also slang, and acronyms. Because of this, vaderSentiment is the perfect fit for analyzing lyrical text in pop music.

Here are the ten songs with the most positive lyrics in the dataset.

These are the ten songs with the most negative lyrics in the dataset.

Now we can use these sentiment scores and find the averages by decade to see how the sentiment of songs have changed over time.

The dataframe above and the visualization of it in the bar graph below illustrates that there is really not much correlation between sentiment scores and decade. There are interesting aspects to point out though, like how songs in the 1950s were quite a bit more positive than other decades. Similarly, songs in the 2000s were a bit more negative than in other decades. While there isn't really a trend to find with sentiment scores, the one takeaway from this information is that pop music is generally positive. Therefore, I would expect most pop songs to chart in the future to have a higher positive sentiment score.

b. Text classification¶

Using text classification we'll train a model to learn and determine whether a song fits into a particular category or not. The goal is for the algorithm to classify each song as accurately as possible. In this particular case, we'll divide songs into two time periods, before 1985 and after 1985. Because the algorithm needs labeled data to train itself, a 1 will denote that a song was made before 1985 and a 0 means that the song was made after. Once the model is trained, we'll apply it using unlabeled songs.

In order to prevent overfitting, we'll use split validation. In other words, we'll separate the data into a train/test split. The training set will be used to train the algorithm, and the testing set will test the accuracy of the model.

For this project, we'll use Pipeline to build our classification model. The benefit of using Pipeline is that it streamlines the model building process while also building several models based on several parameters to select the most accurate model.

In this case, the most accurate model that Pipeline could build is around 69% accurate.

Displayed below is the confusion matrix for the Pipeline classification model. A confusion matrix is a visualization of the accuracy of a classification model. The goal of a confusion matrix is to maximize the true rates, the numbers diagonal from top left to bottom right. Naturally, when maximizing the true rates we'll want to minimize the rest which are called false rates.

Now we can use this classification model to test unlabeled data. I've gather lyrics from ten songs that were not able to be collected through scraping. These tens songs are:

• Chubby Checker - Let's Twist Again - 1961
• The Righteous Brothers - Unchained Melody - 1965
• Archies - Sugar Sugar - 1969
• Simon & Garfunkel - Bridge Over Troubled Water - 1970
• Eagles - Hotel California - 1977
• Hanson - Mmmbop - 1997
• Lou Bega - Mambo No 5 - 1999
• TLC - No Scrubs - 1999
• Timbaland - Apologize - 2007
• Kesha - Tik Toc - 2010

With the unlabeled data, the text classification model correctly classified 7 out of the 10 songs for an accuracy score of 70%, very similar to the overall model accuracy. While this accuracy score isn't incredibly high, it's still quite a bit more accurate than just a flip of a coin.

c. Topic modeling¶

Topic modeling is similar to text classification, however, the main difference between the two is that topic modeling is unsupervised machine learning while text classification is supervised meaning that topic modeling does not need labeled data. One popular algorithm for topic modeling is called latent dirichlet allocation (LDA). The goal of LDA is to find topics from texts that may be hidden or unobserved and cluster them together. For this project, we'll attempt to see what topics are prevalent in popular music song lyrics.

Before building the model we'll need to determine how many topics are present in the data by finding the optimal k-value. According to the graph below, the optimal k-value is 2, signaling that there isn't a great deal of diversity in topics.

The document-topic distribution is the combination of topics within a document or set of texts. Most documents will have more than one topic associated with it and will have probability information for association with a topic. Here we are determining the highest topic probability for each document to assign the document a topic.

The topic distribution below shows that most topics belong to the topic with the index 0.

Now we can visualize the topic model to find out more information about each topic.

• Topic 1 (index 0) in the visual below seems to reflect the 'love song' phenomenon referenced earlier in the project. Words like 'baby,' 'heart,' and 'cry' are reflective of this topic.
• Topic 2 (index 1) seems to be more about dancing, partying, and enjoying the music itself. Words like 'dance,' 'rock,' 'shake,' and 'music' point to this.

The topics were identified based on the highest probability for each document. But we could also take into account the probabilities to figure out which topic is most prevalent based on document-topic distribution. The result is fairly similar to the topic distribution, the vast majority of songs are topic 1 (index 0).

Now we can use the same unlabeled songs we used previously to apply to the topic model.

The unlabeled songs are predicted to represent the following topics:

• Chubby Checker - Let's Twist Again    ==>    predicted as topic 2 (dance/party song)
• The Righteous Brothers - Unchained Melody    ==>    predicted as topic 1 (love song)
• Archies - Sugar Sugar    ==>    predicted as topic 2 (dance/party song)
• Simon & Garfunkel - Bridge Over Troubled Water    ==>    predicted as topic 1 (love song)
• Eagles - Hotel California    ==>    predicted as topic 1 (love song)
• Hanson - Mmmbop    ==>    predicted as topic 1 (love song)
• Lou Bega - Mambo No 5    ==>    predicted as topic 2 (dance/party song)
• TLC - No Scrubs    ==>    predicted as topic 2 (dance/party song)
• Timbaland - Apologize    ==>    predicted as topic 1 (love song)
• Kesha - Tik Toc    ==>    Predicted as topic 2 (dance/party song)

6. Network Analytics¶

Network analytics can visualize and reveal relationships between certain groups. For this project, we'll use the artists in the top 5000 songs dataset to visualize and reveal relationships. The goal is to create relationships between artists who released music in the same decade. The output will then show artists who have had both success and longevity in their careers.