Python practical for Data Science¶

Introduction¶

This guide explores best practices and fundamental knowledge that will allow you to perform data analysis using Python. In this guide, you will learn how to use Jupyter notebooks and Python libraries such as Pandas, Matplotlib or Numpy to easily and transparently explore and analyze your dataset.

What is Data Science?¶

Essentially, Data Science is about* extracting knowledge* from massive amounts of data to generate information. This is essentially done using several disciplines such as Mathematics and Computer Science such as statistics, probabilistic models, machine learning, data storage, computer programming, and so on.

For more information, We invite readers to take a look at this book.

Prerequisites¶

It is necessary to be initiated into Python programming before attending this guide. We will also refer to the basic notions of probability and statistics, as well as relational algebra.

Main purposes¶

After following this guide, readers should familiarize themselves with the following tasks:

• Creating and installing Python modules;
• Modification and use of Jupyter notebooks;
• Data visualization with Matplotlib and Seaborn;
• Handling tables with Numpy;
• Handling a dataset with Pandas.

Guide program¶

We will propose a hierarchical content according to the following plan:

1. Setup your work environment;
2. Get started with Python;
3. Manipulate data using Numpy and Matplotlib;
4. Deal with a large amount of data using Pandas library.

Let's get started.

1. Setup your work environment;¶

In order to begin analyzing data with Python, we need to have some background, as with all other relevant topics. Now, we will try to explain how to install Jupyter on your own machine.

In case you don't have Python yet, it is possible to install Python, including all the necessary libraries, and the Jupyter notebook directly, by using this solution. Once Python has been previously installed, you will need to install the Jupyter notebook, as well as the following libraries: Pandas , Matplotlib , Numpy and SciPy.

In practice, there are two possible solutions to install the Jupyter notebook and its necessary libraries:

• Full installation in conjunction with the Anaconda distribution;
• Installation of the Jupyter notebook exclusively (without Anaconda).

1.1. Full installation in conjunction with the Anaconda distribution

For those who have never installed Python, it is recommended to install the Anaconda distribution directly. This is what we will do in this section. Note that Anaconda is therefore a Python distribution, developed for Data Science.

Everything we need, Anaconda will install everything we need, certainly, but it can install a little too much (libraries that we won't use, and so on). You can download an improved version of Anaconda here: Download. However, libraries, especially Jupyter, will not be installed automatically for this purpose. As a result, this is a solution for experienced users.

On this page you will find the formal installation instructions in case these instructions are no longer effective.

In order to install Anaconda on Windows or MacOs, you must:

• (1) Download the setup file for Windows or MacOs, then launch it by double-clicking on the downloaded file;
• (2) Once the installation is finished, make sure that everything has been done by executing the Jupyter program

In case of any issues when installing Anaconda, please discuss them on this community.

** 1.2. Installation of the Jupyter notebook exclusively (without Anaconda)**

To avoid installing Anaconda, you can follow the following instructions after you have installed Python:

• Check that the pip program is installed on your machine. For that, you just type pip into a console. Usually, the pip program was started at the same time as Python. Next, input these codelines successively:

By typing the following command in your console, you can verify if the setup went well:

Now you can create a new notebook.

• Check that the pip program is installed on your machine. For that, you just type pip into a console. Then, type these commands into your console:

As an alternative, you can test Jupyter now, without installing anything via this link.

Let us do our first experiment in a Jupyter notebook:

To create a new notebook, click on New from the main Jupyter window and then on Python 3, like this:

In the directory from which you started Jupyter, a file named Untitled.ipynb has normally been created. All Python commands must be typed in the field beside the In [ ] label. To do this, simply type several instructions simultaneously. Functions can even be defined. All variables produced in each cell will be accessible in all cells of the notebook. When you are finished entering them, press Shift+Enter to execute them.

To test it, type for example 2 + 5 in the empty cell located in the center of the window. Then click on this button:

2. Get started with Python¶

In this section, we remember the basics of Python programming. In addition, we will not make a list of everything we need to master, but we will simulate a whole problem. Well, we first heard about the Monty Hall problem. The game involves a presenter against a competitor (the player). This player is positioned in front of three closed doors. There is a car behind one of them and a goat behind each of the others. First, it must indicate a door. Then the presenter must open a door that is neither the one chosen by the candidate nor the one hiding the car. The competitor then has the right to open the door or he initially chose, or to open the third door.

First of all, let's set up our workspace in a notebook:

Here we define a subclass of Enum that will contain the possible strategies. This code involves the use of object-oriented programming concepts, including classes and heritages.

It is possible to define our function. We will create a very simple function. It only represents a part of the game for a particular strategy.

Randint function returns a random integer between its two arguments. For example, randint(0,1) will return 0 or 1.

We will now test our function. Run the next line several times to make sure the result is random.

In this case, it will be necessary to define a function that will launch the game over and over again, and return the result of each game in a list.

In order to make calculations on these results, we will no longer keep them as Boolean variables (True or False) but according to the player's victory (1 if he has won, 0 if he has lost).

For a given number of games (10000), it is first necessary to know which strategy is the most effective for the player. We have a list containing up to 1 as the number of games won by the player. It is just necessary to calculate the sum of all the items in this list, with the sum function, to know the number of 1.

3. Manipulate data using Numpy and Matplotlib¶

3.1. Manipulate data using Numpy

This section will focus on how to efficiently load, store and manipulate data. They can be found in a wide variety of sources, but they can always be considered as arrays of numbers. We' re going to see a tool to manipulate these arrays: Numpy. NumPy (Numerical Python) provides an interface to store and process data. Numpy arrays are like Python lists, but Numpy allows to make things much more efficient, especially for larger arrays.

Let's start by importing Numpy:

Creating Numpy arrays

Numpy arrays, unlike Python lists, can only contain one type of members. There are several ways to create arrays in Numpy:

In case there are different types of data in the starting list, Numpy will try to convert them all to the most generic type. For example, integers int will be converted to float numbers:

As an alternative, it is also possible to manually set a type:

In many instances, it is more effective, particularly for large arrays, by creating them directly. Numpy has several functions for this task:

Array properties, Indexation and Slicing

Each Numpy array has some properties that are often useful.

In order to do this, we will often need to access one or more elements of a array.

In the same way that we can index elements using [], we can access to a set of elements by combining [] and :. There is a simple rule for syntax: x[start:end:step].

Two or more arrays can be concatenated:

So far, we have seen very basic things on Numpy arrays. Starting from here, we'll see what makes Numpy really essential. The benchmark implementation of Python, also called CPython, is very flexible, but this flexibility makes it unable to use all the possible optimizations.

We now provide an overview of such operations:

Alternatively, you can also call functions on Numpy arrays, and even on Python lists.

You can also execute Boolean operations on your arrays:

Numpy has functions to sum up these data on his arrays:

3.2. Manipulate data using Matplotlib

Matplotlib was created to generate graphics directly from Python. In this section, we focus on using Matplotlib as a visualization tool in Jupyter notebooks.

To begin the process, let's set up the workspace:

Let's start by examining a simple case-like drawing the curve of a sin function:

he fig variable refers to a container that contains all items (axes, labels, data, etc.). The axes match the square shown above, which will then contain the data from the graph.

In the case of discrete datasets (points), we frequently use error bars to represent, for each point, the incertitude as to its exact value:

4. Deal with a large amount of data using Pandas library¶

The Pandas library is one of the basic libraries for data science in Python. Pandas offers easy-to-use and robust data structures and the means to use them quickly. In this section, we will discuss what is of interest to the Pandas library, as well as the basic operations on the main object of this library, the Dataframe.

This pandas can be represented by a numpy array:

It is possible to do this:

Let's represent the family with Pandas:

The object that can be used to represent arrays is the DataFrame object

In fact, we can do even better, by specifying column names and row names:

Here are some of the little features of Dataframes. Firstly, to access to belly column of our array. There are two possible syntaxes, which return exactly the same result:

We will now see the whole family one by one, via the iterrows method by returning a tuple that has as its first element the index and the content of the line:

The Pandas library is the one that is pointed in the manipulation of arrays. It is therefore possible to read a CSV file with Pandas: it only takes one line to create a dataframe from a CSV:

• The data variable now contains a dataframe containing the data from the csv file;
• The values in our CSV file are separated by the symbol ; ;
• As a default, pd.read_csv expects values that are separated by a comma