Finding Similar Songs - Part 1: Distance Based Search¶

The first part of this tutorial series demonstrates a distance based similarity search approach, based on extracted content descriptors and distance metrics.

Part 1 - Overview¶

1. Introductions & Requirements
2. Loading data
3. Preprocess data
4. Define the Similarity Model
5. Optimize the Model

Short Introduction to Music Similarity Retrieval¶

The objective of Music Similarity estimation or retrieval is to estimate the notion of similarity between two given tracks. A central part of such an approaches is the definition of a measure for similarity which is further affected by the approach taken to extract the relevant information. One approach is to analyze contextual data such as user generated listening behaviour data (e.g. play/skip-counts, user-tags, ratings, etc.). The approach followed by this tutorial is based on the music content itself and largely focuses on the notion of acoustic similarity. Music features are extracted from the audio content. The resulting music descriptors are high-dimensional numeric vectors and the accumulation of all feature vectors of a collection forms a vector-space. The general principle of content based similarity estimations is based on the assumption that numerical differences are an expression of perceptual dissimilarity. Different metrics such as the Manhattan (L1) or the Euclidean Distance (L2) or non-metric similarity functions such as the Kullback-Leibler divergence are used to estimate the numerical similarity of the feature vectors.

Requirements¶

Please follow the instructions on the tutorial's Github page to install the following dependencies to run this tutorial:

Loading Data¶

Before we can train our models we first have to get some data.

Load Feature Data¶

load feature data from numpy pickle

prepare feature-metadata for alignment with dataset meta-data

Load Metadata¶

load meta-data from csv-file.

Align featuredata with metadata

Add Media-Player to Metadata

Use jupyters display functions to enable HTML5 audio in pandas dataframe

Example

Aggregate Feature Space - Early Fusion¶

Align Meta-data and Feature-data¶

Sort Metadata by Feature-space ID => metadata is aligned to feature-data

Select feature-data for this metadata

Normalize feature data¶

The feature vectors are composed of differnt feature-sets. All of them with different value ranges. While features such as Acousticness and Danceability are scaled between 0 and 1, the BPM values of the tempo feature ranges around 120 or higher. We apply Standard Score or Zero Mean and Unit Variance normalization to uniformly scale the value ranges of the features.

Calculate Similarities¶

This section describes the fundamentals of the content-based audio similarity search approach followed in this tutorial. Audio features are descriptive numbers calculated from the audio spectrum of a track. A good example is the Spectral Centroid, which can be interpreted as the center of gravity of an audio recording. It describes the average frequency weighted by its intensity and distinguishes brighter from darker sounds. Such features are usually calculated for several intervals of a track and finally aggregated into a single vector representation. The latter step, which is a requirement for many machine/statistical learning tasks, is accomplished by calculating statistical measures such as mean, standard deviation, etc.

In the following example, the Spectral Centroids of 10 different tracks are provided using their mean and standard deviation aggregations. Thus, the Spectral Centroid feature(-set) is represented by a two-dimensional feature vector such as the following example:

ID   Mean                  Standard Deviation
0    1517.5993814237531    291.1855836731788

In this example the center frequency is 1518 Hz and it deviates by 291 Hz. These numbers already describe the audio content and can be used to find similar tracks. The common approach to calcualte music similarity from audio content is based on vector difference. The assumption is, that similar audio feature-values correspond with similar audio content. Thus, feature vectors with smaller vector differences correspond to more similar tracks. The following data represents the extracted Spectral Centroids of our 10-tracks collection:

ID   Mean                  Standard Deviation
0    1517.5993814237531    291.1855836731788
1    1659.1988993873124    327.64811981777865
2    1507.4617047141264    340.8830079395701
3    1597.6019371942953    507.1007933367403
4    1498.8531206911534    288.3780838480238
5    535.5910732230583     89.90893994909047
6    2261.4032345595674    353.5971736260454
7    2331.881852844861     406.33517225264194
8    1868.690426450363     342.7489751514078
9    2204.6324484864085    328.94334883095553

The tracks have unique identifiers and we are using the track with ID 5 to search for similar items. This step requires a similarity metric, which defines how the vector distance has to be calculated as a single numeric value. The most common choices are the Manhattan (L1) and Euclidean (L2) distance measures. The Euclidean Distance is the square root of the sum of squared differences of two vectors. To calculate the Euclidean Distance between track 5 and track 0:

ID   Mean                  Standard Deviation
0    1517.5993814237531    291.1855836731788
5    535.5910732230583     89.90893994909047

we first compute the difference between the values of each vectors

982.008308           201.276644

square them to get the absolute magnitude:

964340.317375        40512.287309

and take the sum of these values:

1004852.6046840245

Per definition the square root has to be calculated from the sum, but this step is normally skipped because it does not alter the ranking and is processing intensive. By calculating the distance for all items in the collection, we retrieve a list of distance values where the smaller distances correspond to more similar audio content and the higher values should sound more dissimilar.

ID   Distance
0    1004852.6046840245
1    1319014.4646621975
2    1007520.5071585375
3    1301916.1177259558
4    967263.7731724023
5    0.0
6    3047959.100796666
7    3326786.1254441254
8    1841081.968976167
9    2842836.5609704787

To retrieve a ranked list of similar sounding tracks, the list of vector distances has to be ordered ascendingly.

ID   Distance
5    0.0
4    967263.7731724023
0    1004852.6046840245
2    1007520.5071585375
3    1301916.1177259558
1    1319014.4646621975
8    1841081.968976167
9    2842836.5609704787
6    3047959.100796666
7    3326786.1254441254

This so called vector space model is predominant in content based multimedia retrieval. The most crucial and problematic part is feature crafting, meaning that in the case in which the extracted numbers do not describe the audio well enough, the vector based similarity will also fail to provide results that are perceived as similar. The described approach requires the availability of all feature vectors of all items of a collection. Thus, the feature vectors must be stored. No matter which retrieval approach (pre-calculated / indexed / on demand) will be chosen, all features will be required at a certain time. Given that the feature extraction is an computationally expensive task (in terms of processing resources and total time), the extracted features are stored and made accessible using a common data format.

Euclidean Distance¶

In the final part of this tutorial we wil use the Euclidean Distance to calculate similarities between tracks. As mentioned above, the Euclidean Distance is a metric to calculate the distance between two vectors and thus is a function of dissimilarity. This means, vectors with smaller distance values are more similar than those with higher distances.

For the rest of the tutorial we will use this song to demonstrate the results of the approach:

The following lines of code implement the approach described above. First, the distances between the query vector and all other vectors of the collection are calculated. Then the distances are sorted ascnedingly to get the simlar tracks. Because the metric distance of identical vectors is 0, the top-most entry of the sorted list is always the query track.

Scaled Eucledian Distance¶

The approach taken to combine the different feature-sets is refered to as early fusion. The problem with the approach described in the previous step is, that larger feature-sets dominate the calculated distance values. The aggregated MFCC and Chroma features have 24 dimensions each. Together they have more dimensions as the remaining features which are mostly single dimensional features. Thus, the distances are unequally dominated by the two feature sets.

To avoid such a bias, we scale the feature-space such that feature-sets and single-value features have euqal the same weights and thus euqal influence on the resulting distance.

Example result

Feature Weighting¶

As explained above, the vanilla Eucliden Distance in an early fusion approach is dominated by large feature-sets. Through scaling the feature-space we achieved equal influence for all feature-sets and features. Now, equal influence is not always the best choice fo music similarity. For example, the year and popularity feature we included into our feature vector are not an intrinsic music property. We just added them to cluster recordings of the same epoch together. Currently this feature has the same impact on the estimated similarity as timbre, rhythm and harmonics. When using many features it is commonly a good choice to apply different weights to them. Estimating these weights is generally achieved empirically.

Example result: