# Unlike the other modules, we have been working so far, you have to download and install...
# To install this library in Jupyter notebook
import sys
!{sys.executable} -m pip install -q --upgrade pip
!{sys.executable} -m pip install -q statistics statsmodels scipy
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import statistics
import scipy.stats as st
import statsmodels as sm
Data can be sourced directly or indirectly.
Direct methods of data collection involve collecting new data for a specific study. This type of data is known as primary data.
Indirect methods of data collection involve sourcing and accessing existing data that were not originally collected for the purpose of the study. This type of data is known as secondary data.
A survey involves collecting information from every unit in the population (a census), or from a subset of units (a sample) from the population.
Population:
It is an entire pool of data from where a statistical sample is extracted. It can be visualized as a complete data set of items that are similar in nature.
Sample:
It is a subset of the population, i.e. it is an integral part of the population that has been collected for analysis.
Variable:
A value whose characteristics such as quantity can be measured, it can also be addressed as a data point, or a data item.
Types of Sampling:
- Descriptive statistics is used to describe, summarize and present different datasets through numerical calculations, tables and graphs.
- Inferential statistics is used to make inference and predictions for an entire population, based on sample data of that population.
Descriptive Statistics | Inferential Statistics |
---|---|
Describe the features of populations and/or samples | Use samples to make generalizations about larger populations |
Draw conclusions based on known data | Draw conclusions that go beyond the available data |
It helps in organizing, analyzing and to present data in a meaningful manner | It allows us to compare data, make hypothesis and predictions |
Present final results visually, using tables, charts, or graphs | Present final results in the form of probabilities |
Use measures like central tendency, distribution, and variance | Use techniques like hypothesis testing, confidence intervals, and regression and correlation analysis |
Arithmetic Mean
is the sum of the value of each observation in a dataset divided by the number of observations. The mean cannot be calculated for categorical data, as the values cannot be summed. Moreover, as mean includes every value in the distribution it is influenced by outliers and skewed distributions.Arithmetic Mean = (x1 + x2 + … + xN) / N
Geometric Mean
is calculated as the N-th root of the product of all values, where N is the number of values. The geometric mean is used if the data is comprised of different units of measure, e.g. some measure are height, some are dollars, some are miles, etc. In ML, geometric mean is used to calculate the G-Mean measure that is a model evaluation metric. The geometric mean does not accept negative or zero values.Geometric Mean = N-root(x1 * x2 * … * xN)
Harmonic Mean
is calculated as the number of values N divided by the sum of the reciprocal of the values. The harmonic mean is used if the data is comprised of rates, i.e., a ratio between two quantities with different measures, e.g. speed, acceleration, frequency, etc. In ML, harmonic mean is used to calculate the F1 measure that is a model evaluation metric. The harmonic mean does not accept negative or zero value.Harmonic Mean = N / (1/x1 + 1/x2 + … + 1/xN)
Median
Median is a statistical measure that determines the middle value of a dataset listed in ascending order.
Example 1:
import statistics
print(dir(statistics))
data1 = [14000, 19000, 16000, 20000, 15000, 16000, 15000, 15000, 22000]
print("mean(data1): %.2f" % statistics.mean(data1))
print("geometric_mean(data1): %.2f" % statistics.geometric_mean(data1))
print("harmonic_mean(data1): %.2f" % statistics.harmonic_mean(data1))
print("median(data1): ", statistics.median(data1))
print("mode(data1): ", statistics.mode(data1))
Example 2:
data2 = [14000, 19000, 16000, 20000, 15000, 16000, 15000, 15000, 22000, 900000]
print("mean(data2): %.2f" % statistics.mean(data2))
Range
is the spread of your data from the lowest to the highest value in the distribution,Example 1:
import numpy as np
data1 = [25, 10, 9, 6, 12, 11, 15]
range1 = np.max(data1) - np.min(data1)
range1
Example 2:
import numpy as np
data2 = [10, 9, 8, 11, 10, 9, 8, 11]
range2 = max(data2) - min(data2)
range2
Quantiles divedes a distribution and the most common are quartiles
, percentiles
, and deciles
.
The median, which divides a distribution in two at its midpoint, is the most well-known example of a quantile.
Quartiles:, as their name suggests, are quantiles that divide a distribution into quarters by splitting a rank-ordered dataset into four equal parts.
Inter Quartile Range is the difference between the third quartile(Q3) and the first Quartile (Q1)
The range gives us a measurement of how spread out the entirety of our data set is. The interquartile range, which tells us how far apart the first and third quartile are, indicates how spread out the middle 50% of our set of data is.
The primary advantage of using the interquartile range rather than the range for the measurement of the spread of a data set is that the interquartile range is not sensitive to outliers.
Due to its resistance to outliers, the interquartile range is useful in identifying when a value is an outlier.
Percentiles divide the distribution at any point out of one hundred. For example, if we'd like to identify the threshold for the top 5% of a distribution, we'd cut it at the 95th percentile. Or, for the top 1%, we'd cut at the 99th percentile.
Deciles: (from Latin decimus, meaning "tenth") divide a distribution into ten evenly-sized segments.
Example 1:
import statistics
data = [11,13,16,19,20,21,23,25,26,29,33,34,36,38,39,46,52,55,58]
q1,q2,q3 = statistics.quantiles(data, n=4)
print("Q1: ", q1)
print("Q2: ", q2)
print("Q3: ", q3)
print("IQR = Q3 - Q1: ", q3-q1)
Example 2:
data = [0, 0, 2, 5, 8, 8, 8, 9, 9, 10, 10, 10, 11, 12, 12, 12, 14, 15, 20, 25]
q1,q2,q3 = statistics.quantiles(data, n=4)
print("Q1: ", q1)
print("Q2: ", q2)
print("Q3: ", q3)
iqr = q3 - q1
print("IQR = Q3 - Q1: ", iqr)
# Any observations that are more than 1.5 times IQR above Q3 are considered outliers.
upperfence = q3 + iqr*1.5
upperfence
# Any observations that are less than 1.5 times IQR below Q1 re considered outliers.
lowerfence = q1 - iqr*1.5
lowerfence
There are 4 outliers: 0, 0, 20, and 25.
Variance
is a very simple statistic that gives you an extremely rough idea of how spread out a data set is. As a measure of spread, it’s actually pretty weak. A large variance of 22,000, for example, doesn’t tell you much about the spread of data — other than it’s big! The most important reason the variance exists is to give you a way to find the standard deviation which is the square root of variance.n-1
in case of sample variance is "the formula uses sample mean in the numerator and as a consequence may underestimate the true variance of the population. Dividing by n-1 instead of n corrects for that bias.Standard Deviation
is a measure of how spread out data is around center of the distribution (the mean). It also gives you an idea of where, percentage wise, a certain value falls. For example, let’s say you took a test and it was normally distributed (shaped like a bell). You score one standard deviation above the mean. That tells you your score puts you in the top 84% of test takers.
Example 1:
import statistics
data = [2, 4, 6]
print("Sample Mean: ", statistics.mean(data))
print("Sample Variance: ", statistics.variance(data))
print("Population Variance: ", statistics.pvariance(data))
print("Sample Standard Deviation: ", statistics.stdev(data))
print("Population Standard Deviation: ", statistics.pstdev(data))
data = [2, 4, 6]
np.std(data)
statistics.stdev(data)
Example 2: Machine Learning
teacher has compiled the result and wants to know whether most students are performing at the same level, or if there is a high standard deviation.
marks_ML = [75, 69, 80, 70, 60, 63, 64, 69, 71]
print("Mean of ML Marks: ", statistics.mean(marks_ML))
print("Standard Deviation of ML Marks: %.2f" % statistics.stdev(marks_ML))
Low standard deviation implies that most values are close to the mean.
Example 3: Artificial Intelligence
teacher has compiled the result and wants to know whether most students are performing at the same level, or if there is a high standard deviation.
marks_AI = [44, 95, 25, 60, 76, 81, 93, 84, 71, 33, 85, 81]
print("Mean of AI Marks: ", statistics.mean(marks_AI))
print("Standard Deviation of AI Marks: %.2f" % statistics.stdev(marks_AI))
Note: High standard deviation implies that the values are more broadly spread out.
Example 4: Suppose there are 10000 students in my Data Science class. You get 85% marks in your Data Science Exam. The mean of the overall result is 60% with a standard deviation of 10%. How many students are above you?
# Let us first generate the random marks of ten thousand students, with a mean of 60 and standard deviation of 10
mu = 60
sigma = 10
np.random.seed(54)
x = np.random.normal(mu, sigma, 10000)
x
# Let us verify, whether the mean and std dev of above distribution `x` is 60 and 10 respectively
print("np.mean(x): ", np.mean(x))
print("np.std(x): ", np.std(x))
- Let us calculate the number of students above you out of ten thousand and visualize this by drawing a graph.
a = len(np.where(x > 85)[0])
a
sns.displot(x, color='green')
plt.axvline(mu, color='orange')
for i in [-3, -2, -1, 1, 2, 3]:
plt.axvline(mu+i*sigma, color='red')
plt.axvline(85, color='purple')
plt.show()
Example 5: This is continuation of above example. Suppose your marks in Data Science are still 85%, but this time the mean of the overall result has increased to 90% with a standard deviation of 2. How many students are above you?
# Let us first generate the random marks of ten thousand students, with a mean of 90 and standard deviation of 2
mu = 90
sigma = 2
np.random.seed(54)
x = np.random.normal(mu, sigma, 10000)
x
# Let us verify, whether the mean and std dev of above distribution `x` is 90 and 2 respectively
print("np.mean(x): ", np.mean(x))
print("np.std(x): ", np.std(x))
- Let us calculate the number of students above you out of ten thousand and visualize this by drawing a graph.
a = len(np.where(x > 85)[0])
a
sns.displot(x, color='grey')
plt.axvline(mu, color='orange')
for i in [-3, -2, -1, 1, 2, 3]:
plt.axvline(mu+i*sigma, color='red')
plt.axvline(85, color='purple')
plt.show()
Skewness
: is the measure of how much the probability distribution of a random variable deviates from the normal distribution. The skewness for a normal distribution is zeroPositive Skewness / Right-Skewed Distribution: (median < mean):
If the given distribution is shifted to the left and with its tail on the right side, it is a positively skewed distribution. In this type, the majority of the observations are concentrated on the left, and the value of skewness is positive.Negative Skewness / Left-Skewed Distribution: (median > mean):
If the given distribution is shifted to the right and with its tail on the left side, it is a negatively skewed distribution. In this type, the majority of the observations are concentrated on the right, and the value of skewness is negative.Measuring Skewness
: Skewness can be measured using several methods; however, Pearson mode skewness
and Pearson median skewness
are the two frequently used methods.
Example 1:
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import statistics
import scipy.stats as st
import statsmodels as sm
import scipy
import statistics
import seaborn as sns
s = scipy.stats.skewnorm.rvs(20, size=10000)
print("Mean: ", statistics.mean(s))
print("Median: ", statistics.median(s))
print("Mode: ", statistics.mode(s))
print("Skew: ", scipy.stats.skew(s))
sns.displot(s, kde=True);
Mean: 0.7861795940638052 Median: 0.66541753627437 Mode: 0.870250440697566 Skew: 0.9917815445925732
from scipy import stats
import statistics
data1 = [30, 10, 15, 27, 15, 45, 31, 19, 54, 60, 95]
print("Mean: ", statistics.mean(data1))
print("Median: ", statistics.median(data1))
print("Mode: ", statistics.mode(data1))
print("Skew: ", stats.skew(data1))
import stats
import seaborn as sns
s = stats.skewnorm.rvs(20, size=10000)
print("Mean: ", statistics.mean(data))
print("Median: ", statistics.median(data))
print("Mode: ", statistics.mode(data))
print("Skew: ", stats.skew(data))
sns.displot(s, kde=True);
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) /var/folders/1t/g3ylw8h50cjdqmk5d6jh1qmm0000gn/T/ipykernel_61560/890888814.py in <module> 1 import stats 2 import seaborn as sns ----> 3 s = stats.skewnorm.rvs(20, size=10000) 4 print("Mean: ", statistics.mean(data)) 5 print("Median: ", statistics.median(data)) AttributeError: module 'stats' has no attribute 'skewnorm'
Example 2:
from scipy import stats
import statistics
data2 = [20, 33, 88, 91, 85, 89, 91, 95]
print("Mean: ", statistics.mean(data2))
print("Median: ", statistics.median(data2))
print("Mode: ", statistics.mode(data2))
print("Skew: ", stats.skew(data2))
s = stats.skewnorm.rvs(-20, size=10000)
print("Mean: ", statistics.mean(data))
print("Median: ", statistics.median(data))
print("Mode: ", statistics.mode(data))
print("Skew: ", stats.skew(data))
sns.displot(s, kde=True);
Example 3:
import numpy as np
from scipy import stats
import statistics
from matplotlib import pyplot as plt
mean = 25
stdev = 5
data3 = np.random.normal(loc=mean, scale=stdev,size=1000)
print("Mean: ", statistics.mean(data3))
print("Median: ", statistics.median(data3))
print("Mode: ", statistics.mode(data3))
print("Skew: ", stats.skew(data3))
plt.hist(data3, 50);
s = stats.skewnorm.rvs(0, size=10000)
print("Mean: ", statistics.mean(data))
print("Median: ", statistics.median(data))
print("Mode: ", statistics.mode(data))
print("Skew: ", stats.skew(data))
sns.displot(s, kde=True);
np.mean(data)
from scipy import stats
from matplotlib import pyplot as plt
import numpy as np
data = stats.skewnorm.rvs(0, size=1000) # first argument is "skewness"; 0 has no skew
print("Mean: ", statistics.mean(data))
print("Median: ", statistics.median(data))
print("Mode: ", statistics.mode(data))
print("Skew: ", stats.skew(data))
#fig, ax = plt.subplots()
#plt.hist(data, bins=50);
sns.displot(s, kde=True);
#plt.axvline(data = np.mean(data), color='black')
#plt.axvline(data = np.median(data), color='black')
plt.show();
Kurtosis
: is a statistical measure, to determine if the data is heavy-tailed or light-tailed in a normal distribution.
The main difference between skewness and kurtosis is that the skewness refers to the degree of symmetry, whereas the kurtosis refers to the degree of presence of outliers in the distribution.
Example 1:
#Mesokurtic: kurtosis==3
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
mean = 25
stdev = 5
data = np.random.normal(loc=mean, scale=stdev,size=1000)
print('Kurtosis:', stats.kurtosis(data, fisher=False))
print("Skew: ", stats.skew(data))
#plt.hist(data, 50);
sns.displot(data);
Kurtosis: 2.8626095865000307 Skew: -0.02740018788755584
Example 2:
#Leptokurtic: kurtosis > 3 (outliers)
import numpy as np
from scipy import stats
data = np.random.laplace(1.45, 15, 1000)
print('Kurtosis:', stats.kurtosis(data, fisher=False))
print("Skew: ", stats.skew(data))
plt.hist(data, 50);
Kurtosis: 5.425202860794532 Skew: -0.36819634158211506
Example 3:
#Platykurtic: kurtosis < 3 (less extreme outliers)
import numpy as np
from scipy import stats
data = np.random.rand(1000)
print('Kurtosis:', stats.kurtosis(data, fisher=False))
print("Skew: ", stats.skew(data))
plt.hist(data, 50);
Kurtosis: 1.7752860010898066 Skew: -0.0443904032575784
Variance
tells us how much a (single) quantity varies w.r.t. its mean. Its the spread of data around the mean value. You only know the magnitude here, as in how much the data is spread.Covariance
tells us direction in which two quantities vary with each other.Correlation
shows us both, the direction and magnitude of how two quantities vary with each other.
Positive covariance
: Indicates that two variables tend to move in the same direction.Negative covariance
: Reveals that two variables tend to move in inverse directions.Zero covarince
: Indicates that two variables have no relationship between each other.
Let us compute the variance and covariance for two variables
weight
andheight
of six persons
from matplotlib import pyplot as plt
import statistics
weight = [61,62,73,74,82,86]
height = [157,168,170,181,191,185]
print("mean(weight): ", statistics.mean(weight))
print("mean(height): ", statistics.mean(height))
print("var(weight): ", statistics.variance(weight))
print("var(height): ", statistics.variance(height))
print("cov(weight, height): ", statistics.covariance(weight,height))
print("Pearson Correlation Coefficient (r): ",statistics.correlation(weight, height))
plt.scatter(x=weight, y=height);
print("var(weight): ", statistics.variance(weight))
print("var(height): ", statistics.variance(height))
print("cov(weight, height): ", statistics.covariance(weight,height))
print("Covariance Matrix: \n", np.cov(weight,height))
print("Pearson Correlation Coefficient 'r': ",statistics.correlation(weight, height))
print("Correlation Matrix: \n", np.corrcoef(weight,height))
Let us compute the variance and covariance for two variables
time spent on yoga
andstress level
from matplotlib import pyplot as plt
import statistics
yoga = [1,2,3,4,5,6,7,8,9]
stress = [90,80,70,60,50,40,30,20,10]
print("mean(yoga): ", statistics.mean(yoga))
print("mean(stress): ", statistics.mean(stress))
print("var(yoga): ", statistics.variance(yoga))
print("var(stress): ", statistics.variance(stress))
print("cov(yoga, stress): ", statistics.covariance(yoga,stress))
print("Pearson Correlation Coefficient 'r': ",statistics.correlation(yoga, stress))
plt.scatter(x=yoga, y=stress);
print("var(yoga): ", statistics.variance(yoga))
print("var(stress): ", statistics.variance(stress))
print("cov(yoga, stress): ", statistics.covariance(yoga,stress))
print("Covariance Matrix: \n", np.cov(yoga,stress))
print("Pearson Correlation Coefficient 'r': ",statistics.correlation(yoga, stress))
print("Correlation Matrix: \n", np.corrcoef(yoga,stress))
# Correlation matrix of original data
weight = [86,82,74,73,62,61]
height = [157,168,170,181,191,185]
print("Correlation Matrix: \n", np.corrcoef(weight,height))
# Covariance matrix of standardized data
weight = [1.28, 0.89, 0.10, 0, -1.08, -1.18]
height = [-1.46, -0.58, -0.42, 0.45, 1.25, 0.77]
print("Covariance Matrix: \n", np.cov(weight,height))
Covariance and Correlation matrix will be identical in this case
import pandas as pd
df_titanic = pd.read_csv('datasets/titanic3.csv')
df_titanic
df_iris = pd.read_csv('datasets/iris.csv')
df_iris
df_tips = pd.read_csv('datasets/tips.csv')
df_tips
import seaborn as sns
import statistics
from scipy import stats
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
Example 1: Check the spread of data of sepal_width
column in IRIS
dataset using Scatter Plot
print("mean(sepal_width): %.2f" % statistics.mean(df_iris.loc[:, 'sepal_width']))
print("median(sepal_width): %.2f" % statistics.median(df_iris.loc[:, 'sepal_width']))
print("mode(sepal_width): %.2f" % statistics.mode(df_iris.loc[:, 'sepal_width']))
print("variance(sepal_width): %.2f" % statistics.variance(df_iris.loc[:, 'sepal_width']))
sns.relplot(x = df_iris.index, y = 'sepal_width', data = df_iris, kind='scatter');
Example 2: Check the measures of center for the age
column in TITANIC
dataset using Box Plot
#There are 264 NaN values under age column, which may disturb our calculations, so replace them with mean
df_titanic.loc[(df_titanic.age.isna()),'age'] = df_titanic.age.mean()
print("min(age): ", df_titanic.loc[:, 'age'].min())
print("max(age): ", df_titanic.loc[:, 'age'].max())
print("median(age): ", df_titanic.loc[:, 'age'].median())
print("q1, q2, q3: ", statistics.quantiles(df_titanic.loc[:, 'age'], n=4))
sns.catplot(y='age', kind='box', data = df_titanic);
Example 3: Check the distribution of data of sepal_width
column in IRIS
dataset using Histogram
# sepal_width Column of IRIS Dataset
print("skew(sepal_width): %.2f" % stats.skew(df_iris.loc[:, 'sepal_width']))
print('Kurtosis(sepal_width): %.2f' % stats.kurtosis(df_iris.loc[:, 'sepal_width'], fisher=False))
sns.displot(x= 'sepal_width', data=df_iris, kind='hist', kde=True);
Example 4: Check the distribution of data of age
column in TITANIC
dataset using Histogram
# age Column of TITANIC Dataset
print("skew(age): %.2f" % stats.skew(df_titanic.loc[:, 'age']))
print('Kurtosis(age): %.2f' % stats.kurtosis(df_titanic.loc[:, 'age'], fisher=False))
sns.displot(x= 'age', data=df_titanic, kind='hist', kde=True);
Example 5: Check the distribution of data of tip
column in TIPS
dataset using Histogram
# tip Column of TIPS Dataset
print("skew(tip): %.2f" % stats.skew(df_tips.loc[:, 'tip']))
print('Kurtosis(tip): %.2f' % stats.kurtosis(df_tips.loc[:, 'tip'], fisher=False))
sns.displot(x= 'tip', data=df_tips, kind='hist', kde=True);
Example 1: Compare the sex
and survived
columns of TITANIC
dataset using Bar Plot
sns.catplot(x ='sex', y ='survived',kind='bar', data = df_titanic);
Example 2: Check the survival. count of passengers bbased on sex
column of TITANIC
dataset using Count Plot and adding the hue argument
sns.catplot(x ='sex',kind='count', data = df_titanic, hue='survived');
Example 3: Check the survival count of passengers based on passenger class
column of TITANIC
dataset using Count Plot and adding the col argument
sns.catplot(x ='pclass',kind='count', data = df_titanic, col='survived');
Example 4: Check the survival rate of passengers based on passenger class
column of TITANIC
dataset using Bar Plot
sns.catplot(x ='pclass', y ='survived',kind='bar', data = df_titanic);
Example 1: Compare the sex
and age
columns of TITANIC
dataset using Box Plot
sns.catplot(x ='sex', y='age', kind='box', data = df_titanic);
- Have separate Boxplots for survived as not survived using
col
argument
sns.catplot(x ='sex', y='age', kind='box', data = df_titanic, col='survived');
Example 2: Compare the pclass
and age
columns of TITANIC
dataset using Box Plot
sns.catplot(x ='pclass', y='age', kind='box', data = df_titanic);
- Have separate Boxplots for survived as not survived using
col
argument
sns.catplot(x ='pclass', y='age', kind='box', data = df_titanic, col='survived');
Example 1: Compare the sepal_length
and sepal_width
columns of IRIS
dataset using Scatter Plot
print("corr(sepal_length,sepal_width): %.2f" %
statistics.correlation(df_iris['sepal_length'], df_iris['sepal_width']))
sns.relplot(x='sepal_length', y='sepal_width', data=df_iris, kind='scatter');
Example 2: Compare the petal_length
and petal_width
columns of IRIS
dataset using Scatter Plot
print("corr(petal_length,petal_width): %.2f" %
statistics.correlation(df_iris['petal_length'], df_iris['petal_width']))
sns.relplot(x='petal_length', y='petal_width', data=df_iris, kind='scatter');
Example 3: Compare the total_bill
and tip
columns of TIPS
dataset using Scatter Plot
print("Pearson Correlation Coefficient (r): %.2f" %
statistics.correlation(df_tips.loc[:, 'total_bill'],df_tips.loc[:, 'tip']))
sns.relplot(x='total_bill', y='tip', data=df_tips, kind='scatter');
Example 4: Check relationships using Correlation Matrix and Heat Map for the numberic columns of TIPS
dataset
df_tips.corr()
sns.heatmap(df_tips.corr(), annot=True);
Example 5: Check relationships using Correlation Matrix and Heat Map for the numberic columns of IRIS
dataset
df_iris.corr()
sns.heatmap(df_iris.corr(), annot=True);
Example 6: Check relationships using Correlation Matrix and Heat Map for the numeric columns of TITANIC
dataset
df_titanic.corr()
sns.heatmap(df_titanic.corr(), annot=True);