Open In Colab

Data Cleaning using Python with Pandas Library.

According to this article, data cleaning and organizing constitutes 57% of the total weight when it comes to the part of the data science.

alt text

The entire data cleaning process is divided into sub-tasks as shown below.

  1. Importing the required libraries.

  2. Getting the data-set from a different source (Kaggle) and displaying the dataset.

  3. Removing the unused or irrelevant columns.

  4. Renaming the column names as per our convenience.

  5. Replacing the value of the rows and make it more meaningful.

Even though this tutorial is small, but it’s a good way to start on small things and get our hands dirty later on. I will make sure that everyone with no prior experience in python programming or don’t know what is data science or data cleaning can easily understand this tutorial. I'm not very good at python in the first place, so even for me, this was a good place to start. One thing with python is that the code is self-explanatory, your focus should not be what the code does, because the code pretty much says what it does, rather you should tell why did you choose to do this, the “why” factor is important than the “what” factor.

Step 1: Importing the required libraries.

This step involves just importing the required libraries which are pandas, numpy. These are the necessary libraries when it comes to data science.

In [0]:
# Importing the necessary libraries.
import pandas as pd
import numpy as np

Step 2: Getting the data-set from a different source and displaying the data-set.

This step involves getting the data-set from a different source, and the link for the data-set is provided below.

Data-set Download

In [0]:
# Reading a CSV file
df = pd.read_csv("heart.csv")
df.head(5)
Out[0]:
age sex cp trestbps chol fbs restecg thalach exang oldpeak slope ca thal target
0 63 1 3 145 233 1 0 150 0 2.3 0 0 1 1
1 37 1 2 130 250 0 1 187 0 3.5 0 0 2 1
2 41 0 1 130 204 0 0 172 0 1.4 2 0 2 1
3 56 1 1 120 236 0 1 178 0 0.8 2 0 2 1
4 57 0 0 120 354 0 1 163 1 0.6 2 0 2 1

Note: If you are using Jupyter Notebook to practice this tutorial then there should be no problem to read the CSV file. But if you are a Google fan like me, then you ought to use Google Colab which is the best according to me, for practicing data science, then you must follow some steps in order to load or read the CSV file. So this article helps you to solve this issue (I was the one who wrote this article :0). I personally recommend everybody to go through this article.


Step 3: Removing the unused or irrelevant columns

This step involves removing irrelevant columns such as cp, fbs, thalach, and many more, and the code is pretty much self-explanatory.

In [0]:
# Dropping unused columns.
to_drop = ['cp',
          'fbs',
          'restecg',
          'thalach',
          'exang',
          'oldpeak',
          'slope',
          'thal',
           'target', 
          'ca']

df.drop(to_drop, inplace = True, axis = 1)
df.head(5)
Out[0]:
age sex trestbps chol
0 63 1 145 233
1 37 1 130 250
2 41 0 130 204
3 56 1 120 236
4 57 0 120 354

Step 4: Renaming the column names as per our convenience.

This step involves renaming the column names because many column names are kinda confusing and hard to understand.

In [0]:
# Renaming the column names
new_name = {'age': 'Age',
           'sex': 'Sex',
           'trestbps': 'Bps',
           'chol': 'Cholesterol'
            }

df.rename(columns = new_name, inplace = True)
df.head()
Out[0]:
Age Sex Bps Cholesterol
0 63 1 145 233
1 37 1 130 250
2 41 0 130 204
3 56 1 120 236
4 57 0 120 354

Step 5: Replacing the value of the rows if necessary.

This step involves replacing the incomplete values or making the values more readable, such as in here the Sex field consists of the values 1 and 0 being 1 as Male and 0 as Female, but it often seems ambiguous for the third person, so changing the value to an understandable one is a good idea.

In [0]:
# Replacing the values in the row
replace_values = {0: 'F', 1: 'M'}

df = df.replace({"Sex": replace_values})                                                                                             

df.head()
Out[0]:
Age Sex Bps Cholesterol
0 63 M 145 233
1 37 M 130 250
2 41 F 130 204
3 56 M 120 236
4 57 F 120 354

So the above is the overall simple data cleaning process obviously this is not the actual cleaning process at an industry level, but this is a good start, so let’s start from small and then go for huge data-sets which then involves more cleaning process. This was just to give an idea as to how the process of data cleaning looks like in a beginners perspective. Thank you guys for spending your time reading my article, stay tuned for more updates. Let me know what is your opinion about this tutorial in the comment section below. Also if you have any doubts regarding the code, comment section is all yours. Moreover I am an author in towardsdatascience.com and you can find this article there. Have a nice day.