1. Abstract

The HVT package is a collection of R functions to facilitate building topology preserving maps for rich multivariate data analysis. Tending towards a big data preponderance, a large number of rows. A collection of R functions for this typical workflow is organized below:

1. Data Compression: Vector quantization (VQ), HVQ (hierarchical vector quantization) using means or medians. This step compresses the rows (long data frame) using a compression objective.

2. Data Projection: Dimension projection of the compressed cells to 1D,2D or Interactive surface plot with the Sammons Non-linear Algorithm. This step creates topology preserving map (also called embedding ) coordinates into the desired output dimension.

3. Tessellation: Create cells required for object visualization using the Voronoi Tessellation method, package includes heatmap plots for hierarchical Voronoi tessellations (HVT). This step enables data insights, visualization, and interaction with the topology preserving map. Useful for semi-supervised tasks.

4. Scoring: Scoring new data sets and recording their assignment using the map objects from the above steps, in a sequence of maps if required.

5. Temporal Analysis and Visualization: A Collection of new functions that leverages the capacity of the HVT package by analyzing time series data for its underlying patterns, calculation of transitioning probabilities and the visualizations for the flow of data over time.

2. Import Code Modules

Here is the guide to install the HVT package. This helps user to install the most recent version of the HVT package.

###direct installation###
#install.packages("HVT")

#or

###git repo installation###
#library(devtools)
#devtools::install_github(repo = "Mu-Sigma/HVT")

NOTE: At the time documenting this vignette, the updated changes were not still in CRAN, hence we are sourcing the scripts from the R folder directly to the session environment.

# Sourcing required code scripts for HVT
script_dir <- "../R"
r_files <- list.files(script_dir, pattern = "\\.R$", full.names = TRUE)
invisible(lapply(r_files, function(file) { source(file, echo = FALSE); }))

3. Example : HVT with the Torus dataset

In this section, we will see how we can use the package to visualize multidimensional data by projecting them to two dimensions using Sammon’s projection and further used for Scoring.

Data Understanding

First of all, let us see how to generate data for torus. We are using a library geozoo for this purpose. Geo Zoo (stands for Geometric Zoo) is a compilation of geometric objects ranging from three to 10 dimensions. Geo Zoo contains regular or well-known objects, eg cube and sphere, and some abstract objects, e.g. Boy’s surface, Torus and Hyper-Torus.

Here, we will generate a 3D torus (a torus is a surface of revolution generated by revolving a circle in three-dimensional space one full revolution about an axis that is coplanar with the circle) with 12000 points.

Raw Torus Dataset

The torus dataset includes the following columns:

• x: This column represents the X-coordinate of each point in the torus.
• y: This column represents the Y-coordinate of each point in the torus.
• z: This column represents the Z-coordinate of each point in the torus.

Lets, explore the torus dataset containing 12000 points. For the sake of brevity we are displaying first 6 rows.

set.seed(240)
# Here p represents dimension of object, n represents number of points
torus <- geozoo::torus(p = 3,n = 12000)
torus_df <- data.frame(torus$points)
colnames(torus_df) <- c("x","y","z")
torus_df <- torus_df %>% round(4)
Table(head(torus_df), scroll = FALSE)

Now let’s have a look at structure of the torus dataset.

str(torus_df)

## 'data.frame':    12000 obs. of  3 variables:
##  $ x: num  -2.63 -1.42 -1.03 1.88 -1.95 ...
##  $ y: num  0.566 -0.89 1.107 0.19 -2.251 ...
##  $ z: num  -0.725 0.946 -0.873 0.994 0.207 ...

Data distribution

This section displays four objects.

Variable Histograms: The histogram distribution of all the features in the dataset.

Box Plots: Box plots for all the features in the dataset. These plots will display the median and Interquartile range of each column at a panel level.

Correlation Matrix: This calculates the Pearson correlation which is a bivariate correlation value measuring the linear correlation between two numeric columns. The output plot is shown as a matrix.

Summary EDA: The table provides descriptive statistics for all the features in the dataset.

• variable: The features/columns of the dataset
• min: Minimum value of that feature/column
• 1st Quartile: The value that splits the lower 25% of the data when arranged in ascending order
• median: Middle value in the ascendingly ordered dataset
• mean: Sum of all values in the dataset divided by the total number of values
• sd: Measure of the dispersion of dataset relative to its mean.
• 3rd Quartile: The value that splits the lower 75% of the data when arranged in ascending order
• max: Maximum value of that feature/column
• hist: The basic barchart of the data distribution of a feature/column
• n_row: Number of rows for that feature/column
• n_missing: Number of missing values/NAs for that feature/column

It uses an inbuilt function called edaPlots to display the above-mentioned four objects.

edaPlots(torus_df, output_type = "summary", n_cols = 3)

edaPlots(torus_df, output_type = "histogram", n_cols = 3)

edaPlots(torus_df, output_type = "boxplot", n_cols = 3)

edaPlots(torus_df, output_type = "correlation", n_cols = 3)

Train - Test Split

Let us split the torus dataset into train and test. We will randomly select 80% of the torus dataset as train and remaining as test.

smp_size <- floor(0.80 * nrow(torus_df))
set.seed(279)
train_ind <- sample(seq_len(nrow(torus_df)), size = smp_size)
torus_train <- torus_df[train_ind, ]
torus_test <- torus_df[-train_ind, ]

Training Dataset

Now, lets have a look at the selected training dataset containing (9600 data points). For the sake of brevity we are displaying first six rows.

rownames(torus_train) <- NULL
Table(head(torus_train), scroll= FALSE)

Now lets have a look at structure of the training dataset.

str(torus_train)

## 'data.frame':    9600 obs. of  3 variables:
##  $ x: num  1.796 0.712 1.929 1.018 -0.274 ...
##  $ y: num  -0.4204 -2.3528 1.2034 0.0344 1.1298 ...
##  $ z: num  -0.988 -0.889 0.962 -0.189 -0.546 ...

Data Distribution

edaPlots(torus_train, output_type = "summary", n_cols = 3)

edaPlots(torus_train,output_type = "histogram", n_cols = 3)

edaPlots(torus_train, output_type = "boxplot", n_cols = 3)

edaPlots(torus_train, output_type = "correlation", n_cols = 3)

Testing Dataset

Now, lets have a look at testing dataset containing(2400 data points).For the sake of brevity we are displaying first six rows.

rownames(torus_test) <- NULL
Table(head(torus_test), scroll = FALSE)

Now lets have a look at structure of the testing dataset.

str(torus_test)

## 'data.frame':    2400 obs. of  3 variables:
##  $ x: num  -2.628 2.747 -2.445 -2.649 -0.268 ...
##  $ y: num  0.566 -0.999 -1.653 -0.575 -1.08 ...
##  $ z: num  -0.725 -0.385 0.31 0.704 -0.461 ...

Data Distribution

edaPlots(torus_test, output_type = "summary", n_cols = 3)

edaPlots(torus_test,output_type = "histogram", n_cols = 3)

edaPlots(torus_test, output_type = "boxplot", n_cols = 3)

edaPlots(torus_test, output_type = "correlation", n_cols = 3)

4. Map A : Base Compressed Map

Let us try to visualize the compressed Map A from the diagram below.

Figure 1: Data Segregation with highlighted bounding box in red around compressed map A

This package can perform vector quantization using the following algorithms -

• Hierarchical Vector Quantization using k−means
• Hierarchical Vector Quantization using k−medoids

For more information on vector quantization, refer the following link.

The trainHVT function constructs highly compressed hierarchical Voronoi tessellations. The raw data is first scaled and this scaled data is supplied as input to the vector quantization algorithm. The vector quantization algorithm compresses the dataset until a user-defined compression percentage rate is achieved using a parameter called quantization error which acts as a threshold and determines the compression percentage. It means that for a given user-defined compression percentage we get the ‘n’ number of cells, then all of these cells formed will have a quantization error less than the threshold quantization error.

Let’s try to comprehend the trainHVT first before moving ahead.

trainHVT(
  data,
  min_compression_perc,
  n_cells,
  depth,
  quant.err,
  normalize,
  distance_metric = c("L1_Norm", "L2_Norm"),
  error_metric = c("mean", "max"),
  quant_method = c("kmeans", "kmedoids"),
  dim_reduction_method = c("sammon" , "tsne" , "umap")
  scale_summary = NA,
  diagnose = FALSE,
  hvt_validation = FALSE,
  train_validation_split_ratio = 0.8,
  projection.scale,
  tsne_perplexity,tsne_theta,tsne_verbose,
  tsne_eta,tsne_max_iter,
  umap_n_neighbors,umap_min_dist
)

Each of the parameters of trainHVT function have been explained below:

• data - A dataframe, with numeric columns (features) that will be used for training the model.

• min_compression_perc - An integer, indicating the minimum compression percentage to be achieved for the dataset. It indicates the desired level of reduction in dataset size compared to its original size.

• n_cells - An integer, indicating the number of cells per hierarchy (level). This parameter determines the granularity or level of detail in the hierarchical vector quantization.

• depth - An integer, indicating the number of levels. A depth of 1 means no hierarchy (single level), while higher values indicate multiple levels (hierarchy).

• quant.err - A number indicating the quantization error threshold. A cell will only breakdown into further cells if the quantization error of the cell is above the defined quantization error threshold.

• normalize - A logical value indicating if the dataset should be normalized. When set to TRUE, scales the values of all features to have a mean of 0 and a standard deviation of 1 (Z-score)

• distance_metric - The distance metric can be L1_Norm(Manhattan) or L2_Norm(Euclidean). L1_Norm is selected by default. The distance metric is used to calculate the distance between an n dimensional point and centroid.

• error_metric - The error metric can be mean or max. max is selected by default. max will return the max of m values and mean will take mean of m values where each value is a distance between a point and centroid of the cell.

• quant_method - The quantization method can be kmeans or kmedoids. Kmeans uses means (centroids) as cluster centers while Kmedoids uses actual data points (medoids) as cluster centers. kmeans is selected by default.

• projection.scale - A number indicating the scale factor for the tessellations to visualize the sub-tessellations well enough. It helps in adjusting the visual representation of the hierarchy to make the sub-tessellations more visible. Default is 10.

• dim_reduction_method - The dimensionality reduction method to be chosen. options are ‘tsne’ , ‘umap’ & ‘sammon’. Default is ‘sammon’.

• scale_summary - A list with user defined mean and standard deviation values for all the features in the dataset. Pass the scale summary when normalize is set to FALSE.

• diagnose - A logical value indicating whether user wants to perform diagnostics on the model. Default value is FALSE.

• hvt_validation - A logical value indicating whether user wants to holdout a validation set and find mean absolute deviation of the validation points from the centroid. Default value is FALSE.

• train_validation_split_ratio - A numeric value indicating train validation split ratio. This argument is only used when hvt_validation has been set to TRUE. Default value for the argument is 0.8.

• tsne_verbose - A logical value which indicates the t-SNE algorithm to print detailed information about its progress to the console.

• tsne_perplexity - A numeric, balances the attention t-SNE gives to local and global aspects of the data. Lower values focus more on local structure, while higher values consider more global structure. It is recommended to be between 5 and 50. Default value is 30.

• tsne_theta - A numeric, speed/accuracy trade-off parameter for Barnes-Hut approximation. If set to 0, exact t-SNE is performed, which is slower. If set to greater than 0, an approximation is used, which speeds up the process but may reduce accuracy. Default value is 0.5

• tsne_eta (learning_rate) - A numeric, learning rate for t-SNE optimization.Determines the step size during optimization. If too low, the algorithm might get stuck in local minima; if too high, the solution may become unstable. Default value is 200.

• tsne_max_iter - An integer, maximum number of iterations. Number of iterations for the optimization process. More iterations can improve results but increase computation time. Default value is 1000.

• umap_n_neighbors - An integer, the size of the local neighborhood (in terms of number of neighboring sample points) used for manifold approximation, controls the balance between local and global structure in the data, smaller values focus on local structure, while larger values capture more global structures. Default value is 15.

• umap_min_dist - A numeric, the minimum distance between points in the embedded space, controls how tightly UMAP packs points together, lower values result in a more clustered embedding. Default value is 0.1

The output of trainHVT function (list of 7 elements) have been explained below with an image attached for clear understanding.

NOTE: Here the attached image is the snapshot of output list generated from map A which can be referred later in this section

Figure 2: The Output list generated by trainHVT function.

• The ‘1st element’ is a list containing information related to plotting tessellations. This information might include coordinates, boundaries, or other details necessary for visualizing the tessellations

• The ‘2nd element’ is a list containing information related to Sammon’s projection coordinates of the data points in the reduced-dimensional space.

• The ‘3rd element’ is a list containing detailed information about the hierarchical vector quantized data along with a summary section containing no of points, Quantization Error and the centroids for each cell for 2D.

• The ‘4th element’ is a list that contains all the diagnostics information of the model when diagnose is set to TRUE. Otherwise NA.

• The ‘5th element’ is a list that contains all the information required to generates a Mean Absolute Deviation (MAD) plot, if hvt_validation is set to TRUE. Otherwise NA

• The ‘6th element’ is a list containing detailed information about the hierarchical vector quantized data along with a summary section containing no of points, Quantization Error and the centroids for each cell which is the output of hvq.

• The ‘7th element’ (model info) is a list that contains model generated timestamp, input parameters passed to the model, validation results and the dimensionality reduction evaluation metrics table.

We will use the trainHVT function to compress our data while preserving essential features of the dataset. Our goal is to achieve data compression upto atleast 80%. In situations where the compression ratio does not meet the desired target, we can explore adjusting the model parameters as a potential solution. This involves making modifications to parameters such as the quantization error threshold or increasing the number of cells and then rerunning the trainHVT function again.

As this is already done in HVT Vignette: please refer for more information.

Model Parameters

• Number of Cells at each Level = 500
• Maximum Depth = 1
• Quantization Error Threshold = 0.1
• Error Metric = Max
• Distance Metric = Euclidean
• Dimensionality Reduction method = Sammon

set.seed(240)
torus_mapA <- trainHVT(
  torus_train,
  n_cells = 500,
  depth = 1,
  quant.err = 0.1,
  normalize = FALSE,
  distance_metric = "L2_Norm",
  error_metric = "max",
  quant_method = "kmeans",
  dim_reduction_method = "sammon"
)

Let’s check the compression summary for torus.

displayTable(data = torus_mapA[[3]]$compression_summary,columnName = 'percentOfCellsBelowQuantizationErrorThreshold', value = 0.8, tableType = "compression")

We successfully compressed 90% of the data using n_cells parameter as 500, the next step involves performing data projection on the compressed data. In this step, the compressed data will be transformed and projected onto a lower-dimensional space to visualize and analyze the data in a more manageable form.

As per the manual, torus_mapA[[3]] gives us detailed information about the hierarchical vector quantized data. torus_mapA[[3]][['summary']] gives a nice tabular data containing no of points, Quantization Error and the codebook.

The datatable displayed below is the summary from torus_mapA showing Cell.ID, Centroids and Quantization Error for each of the 500 cells. For the sake of brevity, we are displaying only the first 100 rows.

displayTable(data =torus_mapA[[3]][['summary']], columnName= 'Quant.Error', value = 0.1, tableType = "summary", scroll = TRUE)

Now let us understand what each column in the above table means:

• Segment.Level - Level of the cell. In this case, we have performed Vector Quantization for depth 1. Hence Segment Level is 1.

• Segment.Parent - Parent segment of the cell.

• Segment.Child (Cell.Number) - The children of a particular cell. In this case, it is the total number of cells at which we achieved the defined compression percentage.

• n - No of points in each cell.

• Cell.ID - Cell_ID’s are generated for the multivariate data using 1-D Sammon’s Projection algorithm.

• Quant.Error - Quantization Error for each cell.

All the columns after this will contain centroids for each cell. They can also be called a codebook, which represents a collection of all centroids or codewords.

Now let’s try to understand plotHVT function. The parameters have been explained in detail below:

plotHVT <-(hvt.results, line.width, color.vec, pch1, centroid.size, 
           centroid.color,title, maxDepth, child.level, hmap.cols,
           quant.error.hmap, n_cells.hmap, label.size, 
           sepration_width, layer_opacity, cell_id,
           dim_size, plot.type = '2Dhvt')

• hvt.results - (1D/2Dproj/2Dhvt/2Dheatmap/surface_plot) A list obtained from the trainHVT function while performing hierarchical vector quantization on training dataset. This list provides an overview of the hierarchical vector quantized data, including diagnostics, tessellation details, Sammon’s projection coordinates, and model input information.

• line.width - (2Dhvt/2Dheatmap) A vector indicating the line widths of the tessellation boundaries for each layer.

• color.vec - (2Dhvt/2Dheatmap) A vector indicating the colors of the tessellations boundaries at each layer.

• pch1 - (2Dhvt/2Dheatmap) Symbol, It plots the centroids with a particular symbol such as (solid circle, bullet, filled square, filled diamond) in the tessellations.(default = 21 i.e filled circle).

• centroid.size - (2Dhvt/2Dheatmap) Vector of Size of centroids for each level of tessellations.

• centroid.color - (2Dhvt/2Dheatmap) Vector of color of centroids for each level of tessellations.

• title - (2Dhvt) Set a title for the plot (default = NULL).

• maxDepth - (2Dhvt) An integer indicating the number of levels.

• cell_id - (2Dhvt) Logical. To indicate whether the plot should have Cell IDs or not for the level 1. (default = FALSE)

• child.level - (2Dheatmap/surface_plot) A Number indicating the level for which the heat map is to be plotted.

• hmap.cols - (2Dheatmap/surface_plot) A Number or a Character which is the column number or column name from the dataset indicating the variables for which the heat map is to be plotted.

• label.size - (2Dheatmap) The size by which the tessellation labels should be scaled.(default = 0.5)

• quant.error.hmap - (2Dheatmap) A number indicating the quantization error threshold.

• sepration_width - (surface_plot) An integer indicating the width between two Levels.

• layer_opacity - (surface_plot) A vector indicating the opacity of each layer/ level.

• dim_size - (surface_plot) An integer indicating the dimension size used to create the matrix for the plot.

• plot.type - A Character indicating which type of plot should be generated. Accepted entries are ‘1D’,‘2Dproj’, ‘2Dhvt’,‘2Dheatmap’, ‘surface_plot’. Default value is ‘2Dhvt’.

Let’s plot the Voronoi tessellation for layer 1 (map A).

plotHVT(torus_mapA,
        line.width = c(0.4), 
        color.vec = c("navy blue"),
        centroid.size = 0.01,
        maxDepth = 1,
        plot.type = "2Dhvt") 

Figure 3: The Voronoi Tessellation for layer 1 (map A) shown for the 500 cells in the dataset ’torus’

4.1 Heatmaps

Now let’s plot the Voronoi Tessellation with the heatmap overlaid for all the features in the torus dataset for better visualization and interpretation of data patterns and distributions.

The heatmaps displayed below provides a visual representation of the spatial characteristics of the torus dataset, allowing us to observe patterns and trends in the distribution of each of the features (x,y,z). The sheer green shades highlight regions with higher values in each of the heatmaps, while the indigo shades indicate areas with the lowest values in each of the heatmaps. By analyzing these heatmaps, we can gain insights into the variations and relationships between each of these features within the torus dataset.

  plotHVT(
  torus_mapA,
  child.level = 1,
  hmap.cols = "x",
  line.width = c(0.2),
  color.vec = c("navy blue"),
  centroid.size = 0.1,
  plot.type = '2Dheatmap'
) 

Figure 4: The Voronoi Tessellation with the heat map overlaid for variable ’x’ in the ’torus’ dataset

  plotHVT(
  torus_mapA,
  child.level = 1,
  hmap.cols = "y",
  line.width = c(0.2),
  color.vec = c("navy blue"),
  centroid.size = 0.1,
  plot.type = '2Dheatmap'
) 

Figure 5: The Voronoi Tessellation with the heat map overlaid for variable ’y’ in the ’torus’ dataset

  plotHVT(
  torus_mapA,
  child.level = 1,
  hmap.cols = "z",
  line.width = c(0.2),
  color.vec = c("navy blue"),
  centroid.size = 0.1,
  plot.type = '2Dheatmap'
) 

Figure 6: The Voronoi Tessellation with the heat map overlaid for variable ’z’ in the ’torus’ dataset

5. Map B : Compressed Novelty Map

Let us try to visualize the Map B from the diagram below.

Figure 7: Data Segregation with highlighted bounding box in red around map B

In this section, we will manually figure out the novelty cells from the plotted torus_mapA and store it in identified_Novelty_cells variable.

Note: For manual selecting the novelty cells from map A, one can enhance its interactivity by adding plotly elements to the code. This will transform map A into an interactive plot, allowing users to actively engage with the data. By hovering over the centroids of the cells, a tag containing segment child information will be displayed. Users can explore the map by hovering over different cells and selectively choose the novelty cells they wish to consider. Added an image for reference.

Figure 8: Manually selecting novelty cells

The removeNovelty function removes the identified novelty cell(s) from the training dataset (containing 9600 datapoints) and stores those records separately.

It takes input as the cell number (Segment.Child) of the manually identified novelty cell(s) and the compressed HVT map (torus_mapA) with 500 cells. It returns a list of two items: data with novelty, and data without novelty.

NOTE: As we are using torus dataset here, the identified novelty cells given are for demo purpose.

identified_Novelty_cells <<- c(273,44,61,486,185,425)   #as a example
output_list <- removeNovelty(identified_Novelty_cells, torus_mapA)
data_with_novelty <- output_list[[1]]
data_without_novelty <- output_list[[2]]

Let’s have a look at the data with novelty(containing 115 records).

novelty_data <- data_with_novelty
novelty_data$Row.No <- row.names(novelty_data)
novelty_data <- novelty_data %>% dplyr::select("Row.No","Cell.ID","Cell.Number","x","y","z")
colnames(novelty_data) <- c("Row.No","Cell.ID","Segment.Child","x","y","z")
Table(novelty_data,scroll = TRUE)

5.1 Voronoi Tessellation with highlighted novelty cell

The plotNovelCells function is used to plot the Voronoi tessellation using the compressed HVT map (torus_mapA) containing 500 cells and highlights the identified novelty cell(s) i.e 6 cells (containing 115 records) in red on the map.

plotNovelCells(identified_Novelty_cells, torus_mapA,line.width = c(0.4),centroid.size = 0.01)

Figure 9: The Voronoi Tessellation constructed using the compressed HVT map (map A) with the novelty cell(s) highlighted in red

We pass the dataframe with novelty records (115 records) to trainHVT function along with other model parameters mentioned below to generate map B (layer2)

Model Parameters

• Number of Cells at each Level = 11
• Maximum Depth = 1
• Quantization Error Threshold = 0.1
• Error Metric = Max
• Distance Metric = Euclidean
• Dimensionality Reduction method = Sammon

colnames(data_with_novelty) <- c("Cell.ID","Segment.Child","x","y","z")
data_with_novelty <- data_with_novelty[,-1:-2]
mapA_scale_summary = torus_mapA[[3]]$scale_summary
torus_mapB <- trainHVT(data_with_novelty,
                  n_cells = 11,   
                  depth = 1,
                  quant.err = 0.1,
                  normalize = FALSE,
                  distance_metric = "L2_Norm",
                  error_metric = "max",
                  quant_method = "kmeans",
                  dim_reduction_method = "sammon")

The datatable displayed below is the summary from map B (layer 2) showing Cell.ID, Centroids and Quantization Error for each of the 11 cells.

displayTable(data =torus_mapB[[3]][['summary']], columnName= 'Quant.Error', value = 0.1, tableType = "summary")

Segment.Level	Segment.Parent	Segment.Child	n	Cell.ID	Quant.Error	x	y	z
1	1	1	6	6	0.05	-0.03	-2.99	0.13
1	1	2	7	2	0.07	-0.94	-2.84	0.00
1	1	3	9	10	0.1	0.46	2.94	0.20
1	1	4	7	11	0.06	0.46	2.96	-0.05
1	1	5	15	8	0.08	2.90	-0.68	0.18
1	1	6	21	9	0.1	-2.83	0.95	-0.07
1	1	7	11	5	0.05	-0.38	-2.96	0.12
1	1	8	16	7	0.08	2.81	-1.01	0.02
1	1	9	8	4	0.07	-0.25	-2.93	0.34
1	1	10	9	1	0.05	-0.98	-2.80	0.24
1	1	11	6	3	0.06	-0.73	-2.89	0.15

Now let’s check the compression summary for HVT (torus_mapB). The table below shows no of cells, no of cells having quantization error below threshold and percentage of cells having quantization error below threshold for each level.

displayTable(data = torus_mapB[[3]]$compression_summary,columnName = 'percentOfCellsBelowQuantizationErrorThreshold', value = 0.8, tableType = "compression")

segmentLevel	noOfCells	noOfCellsBelowQuantizationError	percentOfCellsBelowQuantizationErrorThreshold	parameters
1	11	10	0.91	n_cells: 11 quant.err: 0.1 distance_metric: L2_Norm error_metric: max quant_method: kmeans

As it can be seen from the table above, 91% of the cells have hit the quantization threshold error.Since we are successfully able to attain the desired compression percentage, so we will not further subdivide the cells

6. Map C : Compressed Map without Novelty

Let us try to visualize the compressed Map C from the diagram below.

Figure 10:Data Segregation with highlighted bounding box in red around compressed map C

torus_mapC <- trainHVT(dataset  = data_without_novelty,
                  n_cells = 10,
                  depth = 2,
                  quant.err = 0.1,
                  normalize = FALSE,
                  distance_metric = "L2_Norm",
                  error_metric = "max",
                  quant_method = "kmeans",
                  dim_reduction_method = "sammon")

displayTable(data = torus_mapC[[3]]$compression_summary,columnName = 'percentOfCellsBelowQuantizationErrorThreshold', value = 0.8, tableType = "compression")

6.2 Iteration 2

Since, we are yet to achive atleast 80% compression at depth 2. Let’s try to compress again using the below mentioned set of model parameters and the data without novelty (containing 9485 records).

Model Parameters

• Number of Cells at Level 1 = 46
• Number of Cells at Level 2 = 2116 (46 x 46)
• Maximum Depth = 2
• Quantization Error Threshold = 0.1
• Error Metric = Max
• Distance Metric = Euclidean
• Dimensionality Reduction method = Sammon

torus_mapC <- trainHVT(data_without_novelty,
                  n_cells = 46,    
                  depth = 2,
                  quant.err = 0.1,
                  normalize = FALSE,
                  distance_metric = "L2_Norm",
                  error_metric = "max",
                  quant_method = "kmeans",
                  dim_reduction_method = "sammon")

The datatable displayed below is the summary from map C (layer2). showing Cell.ID, Centroids and Quantization Error.

displayTable(data =torus_mapC[[3]][['summary']], columnName= 'Quant.Error', value = 0.1, tableType = "summary", scroll = T)

Segment.Level	Segment.Parent	Segment.Child	n	Cell.ID	Quant.Error	x	y	z
1	1	1	183	567	0.31	-1.57	2.40	0.00
1	1	2	236	355	0.31	-1.52	-1.28	-0.92
1	1	3	162	88	0.29	-1.79	-2.27	0.01
1	1	4	167	1865	0.29	2.51	-1.41	-0.20
1	1	5	183	874	0.3	0.46	-2.67	-0.51
1	1	6	251	1120	0.23	-0.16	1.00	-0.03
1	1	7	194	1576	0.25	1.39	-0.01	0.76
1	1	8	196	1208	0.32	-0.43	2.51	-0.70
1	1	9	189	2042	0.3	1.72	2.23	0.35
1	1	10	273	609	0.28	-1.19	-0.19	0.60
1	1	11	248	1320	0.28	0.24	1.46	0.81
1	1	12	257	1537	0.24	1.26	0.09	-0.63
1	1	13	187	602	0.3	-0.13	-2.43	0.79
1	1	14	207	331	0.32	-2.30	1.38	-0.56
1	1	15	154	2118	0.29	2.66	1.18	0.04
1	1	16	288	1465	0.28	0.59	1.27	-0.77
1	1	17	148	2003	0.29	2.76	-0.16	0.49
1	1	18	269	886	0.29	-0.92	1.40	-0.89
1	1	19	170	153	0.32	-2.53	-0.57	-0.71
1	1	20	243	1251	0.23	0.83	-0.67	0.34
1	1	21	189	1908	0.31	1.86	1.25	-0.88
1	1	22	265	880	0.26	-0.18	-1.08	0.39
1	1	23	265	467	0.31	-1.64	0.11	-0.89
1	1	24	258	807	0.23	-0.92	0.43	-0.16
1	1	25	184	1657	0.28	1.92	-1.17	0.88
1	1	26	151	352	0.32	-0.83	-2.43	-0.71
1	1	27	264	695	0.24	-0.83	-0.63	-0.25
1	1	28	166	1387	0.29	1.45	-2.41	0.40
1	1	29	288	1404	0.23	0.80	0.63	0.12
1	1	30	177	1852	0.28	0.94	2.46	-0.64
1	1	31	217	1101	0.28	0.70	-1.62	0.92
1	1	32	177	1362	0.27	1.38	-1.93	-0.83
1	1	33	182	287	0.27	-2.08	-0.47	0.93
1	1	34	238	815	0.28	-0.18	-1.40	-0.76
1	1	35	172	1983	0.28	2.55	0.13	-0.72
1	1	36	151	64	0.29	-2.46	-1.40	0.32
1	1	37	205	968	0.29	-0.83	2.08	0.89
1	1	38	238	1184	0.23	0.74	-0.96	-0.58
1	1	39	221	436	0.29	-1.18	-1.51	0.93
1	1	40	172	1607	0.32	0.24	2.65	0.62
1	1	41	143	125	0.31	-2.86	0.04	0.20
1	1	42	138	1891	0.24	2.15	0.58	0.92
1	1	43	229	412	0.3	-2.13	1.14	0.81
1	1	44	170	1646	0.24	1.81	-0.83	-0.94
1	1	45	190	1753	0.29	1.42	1.28	0.93
1	1	46	230	814	0.25	-1.06	0.93	0.78
2	1	1	3	845	0.03	-1.16	2.63	0.48
2	1	2	4	365	0.07	-2.10	2.11	0.23
2	1	3	3	409	0.09	-2.01	2.11	0.41
2	1	4	5	542	0.08	-1.60	2.41	-0.45
2	1	5	3	585	0.06	-1.52	2.14	-0.78
2	1	6	3	497	0.09	-1.76	2.38	0.26
2	1	7	2	604	0.03	-1.48	2.59	-0.20
2	1	8	5	434	0.13	-1.90	2.26	-0.31
2	1	9	2	766	0.02	-1.28	2.31	-0.77
2	1	10	4	957	0.1	-0.96	2.76	0.37
2	1	11	7	665	0.1	-1.42	2.60	0.27
2	1	12	4	878	0.07	-1.11	2.77	0.16
2	1	13	6	505	0.13	-1.70	2.45	-0.18
2	1	14	2	836	0.05	-1.16	2.75	-0.18
2	1	15	5	498	0.12	-1.74	2.06	-0.71
2	1	16	2	343	0.03	-2.20	1.95	0.33
2	1	17	8	790	0.14	-1.28	2.64	0.35
2	1	18	7	593	0.1	-1.51	2.29	-0.67
2	1	19	3	784	0.09	-1.27	2.71	0.10
2	1	20	5	644	0.05	-1.47	2.47	0.48
2	1	21	4	648	0.13	-1.43	2.55	-0.39
2	1	22	6	433	0.09	-1.91	2.11	-0.53
2	1	23	3	367	0.05	-2.08	2.15	0.13
2	1	24	6	479	0.12	-1.77	2.26	-0.49
2	1	25	3	341	0.08	-2.17	2.06	0.05
2	1	26	5	1004	0.08	-0.87	2.86	0.12
2	1	27	4	442	0.13	-1.89	2.32	0.09
2	1	28	3	627	0.06	-1.45	2.63	0.02
2	1	29	5	440	0.1	-1.93	2.22	0.34
2	1	30	4	405	0.07	-1.99	2.24	-0.08
2	1	31	3	420	0.07	-2.00	2.00	0.56
2	1	32	4	753	0.07	-1.33	2.68	-0.11
2	1	33	4	809	0.05	-1.20	2.58	-0.53
2	1	34	4	916	0.12	-1.03	2.81	-0.13
2	1	35	6	735	0.11	-1.34	2.48	-0.57
2	1	36	4	601	0.1	-1.53	2.35	0.60
2	1	37	5	910	0.1	-1.05	2.75	-0.31
2	1	38	7	553	0.13	-1.61	2.42	0.41
2	1	39	2	336	0.06	-2.18	2.05	-0.14
2	1	40	4	493	0.12	-1.82	2.06	0.66
2	1	41	2	379	0.04	-2.11	1.97	0.45
2	1	42	2	587	0.05	-1.52	2.59	0.01
2	1	43	3	555	0.09	-1.63	2.20	0.67
2	1	44	2	494	0.04	-1.80	2.22	0.51
2	1	45	3	615	0.03	-1.51	2.28	0.68
2	1	46	2	334	0.02	-2.22	2.00	0.17
2	2	1	6	297	0.07	-1.85	-0.84	-1.00
2	2	2	6	351	0.07	-1.33	-1.54	-1.00
2	2	3	4	386	0.07	-1.50	-1.04	-0.98
2	2	4	6	448	0.09	-1.35	-0.95	-0.94
2	2	5	7	346	0.11	-1.67	-0.90	-0.99
2	2	6	7	475	0.11	-0.93	-1.57	-0.98
2	2	7	5	178	0.07	-2.14	-1.22	-0.89
2	2	8	5	575	0.07	-0.87	-1.14	-0.83

Now let’s check the compression summary for HVT (torus_mapC). The table below shows no of cells, no of cells having quantization error below threshold and percentage of cells having quantization error below threshold for each level.

displayTable(data = torus_mapC[[3]]$compression_summary,columnName = 'percentOfCellsBelowQuantizationErrorThreshold', value = 0.8, tableType = "compression")

segmentLevel	noOfCells	noOfCellsBelowQuantizationError	percentOfCellsBelowQuantizationErrorThreshold	parameters
1	46	0	0	n_cells: 46 quant.err: 0.1 distance_metric: L2_Norm error_metric: max quant_method: kmeans
2	2116	1748	0.83	n_cells: 46 quant.err: 0.1 distance_metric: L2_Norm error_metric: max quant_method: kmeans

As it can be seen from the table above, 0% of the cells have hit the quantization threshold error in level 1 and 83% of the cells have hit the quantization threshold error in level 2

Let’s plot the Voronoi tessellation for layer 2 (map C)

plotHVT(torus_mapC,
        line.width = c(0.2,0.1), 
        color.vec = c("navyblue","steelblue"),
        centroid.size = 0.1,
        maxDepth = 2, 
        plot.type = '2Dhvt')

Figure 11: The Voronoi Tessellation for layer 2 (map C) shown for the 928 cells in the dataset ’torus’ at level 2

6.3 Heatmaps

Now let’s plot all the features for each cell at level two as a heatmap for better visualization.

The heatmaps displayed below provides a visual representation of the spatial characteristics of the torus dataset, allowing us to observe patterns and trends in the distribution of each of the features (x,y,z). The sheer green shades highlight regions with higher values in each of the heatmaps, while the indigo shades indicate areas with the lowest values in each of the heatmaps. By analyzing these heatmaps, we can gain insights into the variations and relationships between each of these features within the torus dataset.

  plotHVT(
  torus_mapC,
  child.level = 2,
  hmap.cols = "x",
  line.width = c(0.2,0.1),
  color.vec = c("navyblue","steelblue"),
  centroid.size = 0.1,
  plot.type = '2Dheatmap'
) 

Figure 12: The Voronoi Tessellation with the heat map overlaid for feature x in the ’torus’ dataset

  plotHVT(
  torus_mapC,
  child.level = 2,
  hmap.cols = "y",
  line.width = c(0.2,0.1),
  color.vec = c("navyblue","steelblue"),
  centroid.size = 0.1,
  plot.type = '2Dheatmap'
) 

Figure 13: The Voronoi Tessellation with the heat map overlaid for feature y in the ’torus’ dataset

  plotHVT(
  torus_mapC,
  child.level = 2,
  hmap.cols = "z",
  line.width = c(0.2,0.1),
  color.vec = c("navyblue","steelblue"),
  centroid.size = 0.1,
  plot.type = '2Dheatmap'
) 

Figure 14: The Voronoi Tessellation with the heat map overlaid for feature z in the ’torus’ dataset

We now have the set of maps (map A, map B & map C) which will be used to score, which map and cell each test record is assigned to.

7. Scoring

Now once we have built the model, let us try to score using our testing dataset (containing 2400 data points) which cell and which layer each point belongs to.

The scoreLayeredHVT function is used to score the testing dataset using the scored set of maps. This function takes an input - a testing dataset and a set of maps (map A, map B, map C).

Now, Let us understand the scoreLayeredHVT function.

scoreLayeredHVT(data,
                hvt_mapA,
                hvt_mapB,
                hvt_mapC,
                child.level = 1,
                mad.threshold = 0.2,
                normalize = TRUE,
                distance_metric="L1_Norm",
                error_metric="max",
                yVar)

Each of the parameters of scoreLayeredHVT function has been explained below:

Before that, the approach of scoreLayeredHVT function is to use scoreHVT function to score the test data against the given results of trainHVT which is referred as ‘map’ here. Hence the scoreLayeredHVT scores the test dataset against map A, B & C and further process and merge the final output. So the arguments used in scoreHVT is important here for smooth execution of function.

• data - A dataframe containing the test dataset. The dataframe should have all the variable(features) used for training.

• hvt_mapA - Result obtained from trainHVT function while performing hierarchical vector quantization on train data. This list containes information about the hierarchical vector quantized data along with a summary section.

• hvt_mapB - Result obtained from trainHVT function while performing hierarchical vector quantization on data with novelty data.It is a subset of the training data obtained as a result of removeNovelty function (1st element).

• hvt_mapC - Result obtained from trainHVT function while performing hierarchical vector quantization on data without novelty. It is a subset of the training data obtained as a result of removeNovelty function (2nd element).

• child.level - A number indicating the depth for which the heat map is to be plotted. Each depth represents a different level of clustering or partitioning of the data.

• mad.threshold - A numeric value indicating the permissible Mean Absolute Deviation which is obtained from Minimum Intra centroid plot(when diagnose is set to TRUE in trainHVT). mad.threshold value is important since it is used in anomaly detection. Default value is 0.2 NOTE: for a given datapoint, when the quantization error is above mad.threshold it is denoted as anomaly else not.

• normalize - A logical value indicating if the dataset should be normalized. When set to TRUE, the data (testing dataset) is standardized by mean and sd of the training dataset referred from the trainHVT(). When set to FALSE, the data is used as such without any changes.

• distance_metric - The distance metric can be L1_Norm(Manhattan) or L2_Norm(Euclidean). The metric is used when calculating distance between each datapoint(in test dataset) with the centroids obtained from results of trainHVT. Default is L1_Norm.

• error_metric - The error metric can be mean or max. max will return the max of m values and mean will take mean of m values where each value is a distance between the datapoint and centroid of the cell. This helps in calculating the scored quantization error. Default value is max.

• yVar - A character or a vector representing the name of the dependent variable(s)

When normalize is set to TRUE, the scoreHVT function has an inbuilt feature to standardize the testing dataset based on the mean and standard deviation of the training dataset from the trainHVT results.

• Steps involved in Normalizing the data:
• From the trainHVT results (here it is torus_mapA, torus_mapB & torus_mapC) the summary is accessed which is the third list object.
• From the summary list, the scale.summary object is taken which have two entries that stores mean_data and std_data of all the columns in training dataset.
• Those two entries are taken for the center and scale parameter in scale() which normalizes the testing dataset similar to training dataset in scoreHVT()
• Here in scoreLayeredHVT function the testing dataset is scored against all the maps (A, B & C) by using scoreHVT function and the results are merged and further processed.
• This approach ensures that the model that is evaluated on testing dataset scaled in the same way as the training dataset, maintaining consistency and improving the model’s ability to generalize to new, unseen data.

The function score based on the HVT maps - map A, map B and map C, constructed using trainHVT function. For each test record, the function will assign that record to Layer1 or Layer2. Layer1 contains the cell ids from map A and Layer 2 contains cell ids from map B (novelty map) and map C (map without novelty).

Scoring Algorithm

The Scoring algorithm recursively calculates the distance between each point in the testing dataset and the cell centroids for each level. The following steps explain the scoring method for a single point in the test dataset:

1. Calculate the distance between the point and the centroid of all the cells in the first level.
2. Find the cell whose centroid has minimum distance to the point.
3. Check if the cell drills down further to form more cells.
4. If it doesn’t, return the path. Or else repeat steps 1 to 4 till we reach a level at which the cell doesn’t drill down further.

Note : The Scoring algorithm will not work if some of the variables used to perform quantization are missing. In the testing dataset, we should not remove any features.

validation_data <- torus_test
new_score <- scoreLayeredHVT(
    data=validation_data,
    hvt_mapA = torus_mapA,
    hvt_mapB = torus_mapB,
    hvt_mapC = torus_mapC,
    normalize = FALSE
  )

Let’s see which cell and layer each point belongs to and check the Mean Absolute Difference for each of the 2400 records. For the sake of brevity, we are only displaying the first 100 rows.

act_pred <- new_score[["actual_predictedTable"]]
rownames(act_pred) <- NULL
act_pred %>% head(100) %>%as.data.frame() %>%Table(scroll = TRUE)

hist(act_pred$diff, breaks = 30, col = "blue", main = "Mean Absolute Difference", xlab = "Difference")

Figure 16: Mean Absolute Difference

8. Executive Summary

• We have considered torus dataset for creating a scored sequence of maps using scoreLayeredHVT() in this vignette.

• Our goal is to achieve data compression upto atleast 80%.

• We construct a compressed HVT map (torus_mapA) using the trainHVT() on the training dataset by setting n_cells to 500 and quant.error to 0.1 and we were able to attain a compression of 90%.

• Based on the output of the above step, we manually identify the novelty cell(s) from the plotted map A. For this dataset, we identify the 6 cells as the novelty cells. (since torus dataset does not have outliers we are using this for demo purpose.)

• We pass the identified novelty cell(s) as a parameter to the removeNovelty() along with HVT torus_mapA. The function removes that novelty cell(s) from the dataset and stores them separately. It also returns the data without novelty(s).

• The plotNovelCells() constructs hierarchical voronoi tessellations and highlights the identified novelty cell(s) in red.

• The data with novelty is then passed to the trainHVT() to construct another HVT map (torus_mapB). But here, we set the parameters n_cells = 10, depth = 2 etc. when constructing the map.

• The data without novelty is then passed to the trainHVT() to construct another HVT map (torus_mapC). But here, we set the parameters n_cells = 46, depth = 2 etc. when constructing the map.

• Finally, the set of maps - torus_mapA,torus_mapB,torus_mapC are passed to the scoreLayeredHVT() along with the test dataset to score which map and what cell each test record is assigned to.

• The output of scoreLayeredHVT is a dataset with two columns Layer1.Cell.ID and Layer2.Cell.ID. Layer1.Cell.ID contains cell ids from map A in the form A1,A2,A3…. and Layer2.Cell.ID contains cell ids from map B as B1,B2… depending on the identified novelties and map C as C1,C2,C3…..

9. References

1. Topology Preserving Maps

2. Vector Quantization

3. K-means

4. Sammon’s Projection

5. Voronoi Tessellations