In this project, we aim to figure out 3 potential hackers based on the information about the hacks. The data description is as follows

'Session_Connection_Time': How long the session lasted in minutes
'Bytes Transferred': Number of MB transferred during session
'Kali_Trace_Used': Indicates if the hacker was using Kali Linux
'Servers_Corrupted': Number of server corrupted during the attack
'Pages_Corrupted': Number of pages illegally accessed
'Location': Location attack came from (Probably useless because the hackers used VPNs)
'WPM_Typing_Speed': Their estimated typing speed based on session logs.

In [0]:

from pyspark.sql import SparkSession
from pyspark.ml.clustering import KMeans
from pyspark.ml.linalg import Vectors
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.feature import StandardScaler
from pyspark.ml.evaluation import ClusteringEvaluator

In [0]:

spark = SparkSession.builder.appName('hack_find').getOrCreate()

In [0]:

dataset = spark.read.csv("dbfs:/FileStore/shared_uploads/dizhen@hsph.harvard.edu/hack_data.csv",header=True,inferSchema=True)

In [0]:

dataset.head()

Out[3]: Row(Session_Connection_Time=8.0, Bytes Transferred=391.09, Kali_Trace_Used=1, Servers_Corrupted=2.96, Pages_Corrupted=7.0, Location='Slovenia', WPM_Typing_Speed=72.37)

In [0]:

dataset.describe().show()

+-------+-----------------------+------------------+------------------+-----------------+------------------+-----------+------------------+
|summary|Session_Connection_Time| Bytes Transferred|   Kali_Trace_Used|Servers_Corrupted|   Pages_Corrupted|   Location|  WPM_Typing_Speed|
+-------+-----------------------+------------------+------------------+-----------------+------------------+-----------+------------------+
|  count|                    334|               334|               334|              334|               334|        334|               334|
|   mean|     30.008982035928145| 607.2452694610777|0.5119760479041916|5.258502994011977|10.838323353293413|       null|57.342395209580864|
| stddev|     14.088200614636158|286.33593163576757|0.5006065264451406| 2.30190693339697|  3.06352633036022|       null| 13.41106336843464|
|    min|                    1.0|              10.0|                 0|              1.0|               6.0|Afghanistan|              40.0|
|    max|                   60.0|            1330.5|                 1|             10.0|              15.0|   Zimbabwe|              75.0|
+-------+-----------------------+------------------+------------------+-----------------+------------------+-----------+------------------+

In [0]:

dataset.columns

Out[6]: ['Session_Connection_Time',
 'Bytes Transferred',
 'Kali_Trace_Used',
 'Servers_Corrupted',
 'Pages_Corrupted',
 'Location',
 'WPM_Typing_Speed']

In [0]:

feat_cols = ['Session_Connection_Time', 'Bytes Transferred', 'Kali_Trace_Used',
             'Servers_Corrupted', 'Pages_Corrupted','WPM_Typing_Speed']

In [0]:

vec_assembler = VectorAssembler(inputCols = feat_cols, outputCol='features')

In [0]:

final_data = vec_assembler.transform(dataset)

In [0]:

scaler = StandardScaler(inputCol="features", outputCol="scaledFeatures", withStd=True, withMean=False)

In [0]:

scalerModel = scaler.fit(final_data)

In [0]:

cluster_final_data = scalerModel.transform(final_data)

In [0]:

kmeans3 = KMeans(featuresCol='scaledFeatures',k=3)
kmeans2 = KMeans(featuresCol='scaledFeatures',k=2)

In [0]:

model_k3 = kmeans3.fit(cluster_final_data)
model_k2 = kmeans2.fit(cluster_final_data)

In [0]:

wssse_k3 =  model_k3.transform(cluster_final_data)
wssse_k2 = model_k2.transform(cluster_final_data)

In [0]:

evaluator = ClusteringEvaluator()

In [0]:

silhouette3 = evaluator.evaluate(wssse_k3)
silhouette2 = evaluator.evaluate(wssse_k2)

In [0]:

print("With K=3")
print("Silhouette with squared euclidean distance = " + str(silhouette3))
print("With K=2")
print("Silhouette with squared euclidean distance = " + str(silhouette2))

With K=3
Silhouette with squared euclidean distance = 0.3068084951287429
With K=2
Silhouette with squared euclidean distance = 0.6683623593283755

In [0]:

wssse_k3.groupBy('prediction').count().show()
wssse_k2.groupBy('prediction').count().show()

+----------+-----+
|prediction|count|
+----------+-----+
|         1|   83|
|         2|   84|
|         0|  167|
+----------+-----+

+----------+-----+
|prediction|count|
+----------+-----+
|         1|  167|
|         0|  167|
+----------+-----+