Data Engineering of Matched FBref and TransferMarkt Data for Age Profiling Visualisation¶

Notebook to engineer matched data scraped from TransferMarkt using the Tyrone Mings web scraper by FCrSTATS and StatsBomb data provided by FBref using a Python webscraper by parth1902.¶

By Edd Webster ¶

Notebook first written: 13/09/2020
Notebook last updated: 30/12/2020

Click here to jump straight to the Exploratory Data Analysis section and skip the Task Brief, Data Sources, and Data Engineering sections. Or click here to jump straight to the Conclusion.

This notebook engineers previously scraped data from TransferMarkt using the FCrSTATS Tyrone Ming webscraper and manipulates this landed data as DataFrames using pandas and matplotlib for visualisation.

For more information about this notebook and the author, I'm available through all the following channels:

title

The accompanying GitHub repository for this notebook can be found here and a static version of this notebook can be found here.

1. Notebook Dependencies ¶

This notebook was written using Python 3 and requires the following libraries:

Jupyter notebooks for this notebook environment with which this project is presented;
NumPy for multidimensional array computing;
pandas for data analysis and manipulation;
Beautifulsoup for web scraping; and
matplotlib for data visualisations;

All packages used for this notebook except for BeautifulSoup can be obtained by downloading and installing the Conda distribution, available on all platforms (Windows, Linux and Mac OSX). Step-by-step guides on how to install Anaconda can be found for Windows here and Mac here, as well as in the Anaconda documentation itself here.

Import Libraries and Modules¶

In [4]:

# Python ≥3.5 (ideally)
import platform
import sys, getopt
assert sys.version_info >= (3, 5)
import csv

# Import Dependencies
%matplotlib inline

# Math Operations
import numpy as np
import math
from math import pi

# Datetime
import datetime
from datetime import date
import time

# Data Preprocessing
import pandas as pd
import os
import re
import random
from io import BytesIO
from pathlib import Path

# Reading directories
import glob
import os
from os.path import basename

# Flatten lists
from functools import reduce

# Working with JSON
import json
from pandas.io.json import json_normalize

# Web Scraping
import requests
from bs4 import BeautifulSoup
import re

# Currency Convert
from forex_python.converter import CurrencyRates

# APIs
from tyrone_mings import * 

# Fuzzy Matching - Record Linkage
import recordlinkage
import jellyfish
import numexpr as ne

# Data Visualisation
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
plt.style.use('seaborn-whitegrid')
import missingno as msno

# Progress Bar
from tqdm import tqdm

# Display in Jupyter
from IPython.display import Image, YouTubeVideo
from IPython.core.display import HTML

# Ignore Warnings
import warnings
warnings.filterwarnings(action="ignore", message="^internal gelsd")

print('Setup Complete')

Setup Complete

In [5]:

# Python / module versions used here for reference
print('Python: {}'.format(platform.python_version()))
print('NumPy: {}'.format(np.__version__))
print('pandas: {}'.format(pd.__version__))
print('matplotlib: {}'.format(mpl.__version__))
print('Seaborn: {}'.format(sns.__version__))

Python: 3.7.6
NumPy: 1.18.1
pandas: 1.0.1
matplotlib: 3.1.3
Seaborn: 0.10.0

Defined Variables¶

In [6]:

# Define today's date
today = datetime.datetime.now().strftime('%d/%m/%Y').replace('/', '')

Defined Filepaths¶

In [7]:

# Set up initial paths to subfolders
base_dir = os.path.join('..', '..', )
data_dir = os.path.join(base_dir, 'data')
data_dir_fbref_tm = os.path.join(base_dir, 'data', 'tm')
data_dir_fbref = os.path.join(base_dir, 'data', 'fbref')
data_dir_fbref_tm = os.path.join(base_dir, 'data', 'fbref-tm')
img_dir = os.path.join(base_dir, 'img')
fig_dir = os.path.join(base_dir, 'img', 'fig')
video_dir = os.path.join(base_dir, 'video')

Notebook Settings¶

In [8]:

pd.set_option('display.max_columns', None)

2. Project Brief ¶

This Jupyter notebook engineered data scraped from TransferMarkt and FBref using pandas for data manipulation through DataFrames.

The data of player values produced in this notebook is exported to CSV and used to create Age Profile visualisation in Tableau. See example here.

3. Data Sources ¶

Before conducting our EDA, the data needs to be imported as a DataFrame in the Data Sources section Section 3 and cleaned in the Data Engineering section Section 4.

We'll be using the pandas library to import our data to this workbook as a DataFrame.

3.2. Data Dictionaries ¶

The TransferMarkt dataset has six features (columns) with the following definitions and data types:

Feature	Data type
`position_number`	object
`position_description`	object
`name`	object
`dob`	object
`nationality`	object
`value`	object

3.3. Import CSV as pandas DataFrame ¶

In [9]:

# Import DataFrame as a CSV file
#df_fbref_tm = pd.read_csv(data_dir_fbref_tm + '/fbref_tm_big5_combined_latest.csv')
df_fbref_tm = pd.read_csv(data_dir + '/export/fbref_tm_big5_combined_latest.csv')

3.4. Preliminary Data Handling ¶

Let's quality of the dataset by looking first and last rows in pandas using the head() and tail() methods.

In [10]:

# Display the first 5 rows of the raw DataFrame, df_fbref_tm
df_fbref_tm.head()

Out[10]:

	tm_id	Season	player	player_name	age	age_tm	age_when_joining	dob	pob	cob	nationality	nationality_code	second_citizenship	club	position_code	position_fbref	position_TM	position_grouped	outfielder_goalkeeper	height	foot	market_value_euros	market_value_pounds	joined	years_since_joining	contract_expires	years_until_contract_expiry	contract_option	on_loan_from	on_loan_from_country	loan_contract_expiry	player_agent	games	games_starts	minutes	goals	assists	pens_made	pens_att	cards_yellow	cards_red	goals_per90	assists_per90	goals_assists_per90	goals_pens_per90	goals_assists_pens_per90	xg	npxg	xa	xg_per90	xa_per90	xg_xa_per90	npxg_per90	npxg_xa_per90	minutes_90s	shots_total	shots_on_target	shots_free_kicks	shots_on_target_pct	shots_total_per90	shots_on_target_per90	goals_per_shot	goals_per_shot_on_target	npxg_per_shot	xg_net	npxg_net	passes_completed	passes	passes_pct	passes_total_distance	passes_progressive_distance	passes_completed_short	passes_short	passes_pct_short	passes_completed_medium	passes_medium	passes_pct_medium	passes_completed_long	passes_long	passes_pct_long	xa_net	assisted_shots	passes_into_final_third	passes_into_penalty_area	crosses_into_penalty_area	progressive_passes	passes_live	passes_dead	passes_free_kicks	through_balls	passes_pressure	passes_switches	crosses	corner_kicks	corner_kicks_in	corner_kicks_out	passes_ground	passes_low	passes_high	passes_left_foot	passes_right_foot	passes_head	throw_ins	passes_other_body	passes_offsides	passes_oob	passes_intercepted	passes_blocked	sca	sca_per90	sca_passes_live	sca_passes_dead	sca_dribbles	sca_shots	sca_fouled	gca	gca_per90	gca_passes_live	gca_dribbles	gca_shots	gca_fouled	tackles	tackles_won	tackles_def_3rd	tackles_mid_3rd	tackles_att_3rd	dribble_tackles	dribbles_vs	dribble_tackles_pct	dribbled_past	pressures	pressure_regains	pressure_regain_pct	pressures_def_3rd	pressures_mid_3rd	pressures_att_3rd	blocks	blocked_shots	blocked_passes	interceptions	clearances	touches	touches_def_pen_area	touches_def_3rd	touches_mid_3rd	touches_att_3rd	touches_att_pen_area	touches_live_ball	dribbles_completed	dribbles	dribbles_completed_pct	players_dribbled_past	nutmegs	carries	carry_distance	carry_progressive_distance	pass_targets	passes_received	passes_received_pct	miscontrols	dispossessed	fouls	fouled	offsides	pens_won	ball_recoveries	aerials_won	aerials_lost	aerials_won_pct	games_gk	games_starts_gk	minutes_gk	goals_against_gk	goals_against_per90_gk	shots_on_target_against	saves	save_pct	wins_gk	draws_gk	losses_gk	clean_sheets	clean_sheets_pct	pens_att_gk	pens_allowed	pens_saved	pens_missed_gk	minutes_90s_gk	free_kick_goals_against_gk	corner_kick_goals_against_gk	own_goals_against_gk	psxg_gk	psnpxg_per_shot_on_target_against	psxg_net_gk	psxg_net_per90_gk	passes_completed_launched_gk	passes_launched_gk	passes_pct_launched_gk	passes_gk	passes_throws_gk	pct_passes_launched_gk	passes_length_avg_gk	goal_kicks	pct_goal_kicks_launched	goal_kick_length_avg	crosses_gk	crosses_stopped_gk	crosses_stopped_pct_gk	def_actions_outside_pen_area_gk	def_actions_outside_pen_area_per90_gk	avg_distance_def_actions_gk
0	342229.0	17/18	Kylian Mbappé	kylian mbappé	18.0	22.0	19.0	1998-12-20	Paris	France	France	FRA	NaN	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	178.0	right	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	Relatives	27.0	24.0	2095.0	13.0	9.0	0.0	0.0	2.0	0.0	0.56	0.39	0.95	0.56	0.95	14.5	14.5	7.8	0.62	0.33	0.96	0.62	0.96	23.3	81.0	32.0	0.0	39.5	3.48	1.37	0.16	0.41	0.18	-1.5	-1.5	558.0	701.0	79.6	8087.0	1794.0	324.0	373.0	86.9	174.0	223.0	78.0	35.0	54.0	64.8	1.2	51.0	36.0	33.0	9.0	52.0	674.0	27.0	2.0	1.0	144.0	12.0	62.0	4.0	0.0	0.0	578.0	38.0	85.0	108.0	573.0	8.0	11.0	0.0	3.0	9.0	18.0	30.0	99.0	4.25	66.0	0.0	14.0	5.0	13.0	14.0	0.60	8.0	2.0	3.0	1.0	8.0	4.0	2.0	3.0	3.0	3.0	10.0	30.0	7.0	190.0	67.0	35.3	28.0	59.0	103.0	6.0	2.0	4.0	1.0	0.0	1014.0	3.0	46.0	365.0	698.0	208.0	990.0	65.0	118.0	55.1	70.0	5.0	854.0	6615.0	4067.0	1089.0	840.0	77.1	60.0	61.0	9.0	37.0	23.0	1.0	98.0	1.0	4.0	20.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	342229.0	17/18	Kylian Mbappé	kylian mbappé	18.0	22.0	19.0	1998-12-20	Paris	France	France	FRA	NaN	Monaco	ST	FW,MF	attack - Centre-Forward	Forward	Outfielder	178.0	right	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	Relatives	1.0	1.0	74.0	0.0	0.0	0.0	0.0	0.0	0.0	0.00	0.00	0.00	0.00	0.00	0.2	0.2	0.0	0.20	0.00	0.20	0.20	0.20	0.8	3.0	2.0	0.0	66.7	3.65	2.43	0.00	0.00	0.06	-0.2	-0.2	19.0	31.0	61.3	306.0	44.0	9.0	13.0	69.2	9.0	11.0	81.8	1.0	4.0	25.0	0.0	0.0	1.0	0.0	0.0	0.0	31.0	0.0	0.0	0.0	0.0	0.0	5.0	0.0	0.0	0.0	22.0	5.0	4.0	4.0	26.0	1.0	0.0	0.0	0.0	1.0	4.0	1.0	3.0	3.60	2.0	0.0	0.0	0.0	1.0	1.0	1.20	1.0	0.0	0.0	0.0	1.0	1.0	0.0	0.0	1.0	0.0	0.0	0.0	0.0	4.0	1.0	25.0	0.0	3.0	1.0	0.0	0.0	0.0	1.0	0.0	43.0	0.0	1.0	17.0	28.0	11.0	43.0	4.0	7.0	57.1	6.0	0.0	44.0	342.0	256.0	52.0	38.0	73.1	2.0	2.0	0.0	1.0	2.0	0.0	3.0	0.0	0.0	0.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
2	342229.0	18/19	Kylian Mbappé	kylian mbappé	19.0	22.0	19.0	1998-12-20	Paris	France	France	FRA	NaN	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	178.0	right	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	Relatives	29.0	24.0	2343.0	33.0	7.0	1.0	2.0	5.0	1.0	1.27	0.27	1.54	1.23	1.50	27.2	25.7	5.5	1.05	0.21	1.26	0.99	1.20	26.0	121.0	65.0	1.0	53.7	4.65	2.50	0.26	0.49	0.21	5.8	6.3	637.0	823.0	77.4	9517.0	2275.0	393.0	478.0	82.2	182.0	241.0	75.5	48.0	65.0	73.8	1.5	44.0	47.0	48.0	8.0	76.0	783.0	40.0	3.0	12.0	176.0	13.0	48.0	6.0	0.0	0.0	658.0	100.0	65.0	141.0	668.0	5.0	5.0	3.0	4.0	11.0	24.0	34.0	111.0	4.26	70.0	1.0	16.0	10.0	13.0	22.0	0.85	12.0	3.0	3.0	4.0	7.0	4.0	2.0	2.0	3.0	1.0	9.0	11.1	8.0	281.0	58.0	20.6	17.0	120.0	144.0	6.0	3.0	3.0	5.0	2.0	1183.0	0.0	23.0	449.0	804.0	240.0	1142.0	73.0	132.0	55.3	82.0	4.0	969.0	6769.0	4291.0	1449.0	1022.0	70.5	59.0	49.0	19.0	44.0	30.0	4.0	80.0	2.0	5.0	28.6	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
3	342229.0	19/20	Kylian Mbappé	kylian mbappé	20.0	22.0	19.0	1998-12-20	Paris	France	France	FRA	NaN	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	178.0	right	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	Relatives	20.0	17.0	1513.0	18.0	5.0	0.0	0.0	0.0	0.0	1.07	0.30	1.37	1.07	1.37	17.1	17.1	8.5	1.02	0.51	1.52	1.02	1.52	16.8	88.0	49.0	0.0	55.7	5.23	2.91	0.20	0.37	0.20	0.9	0.9	515.0	671.0	76.8	7497.0	1831.0	327.0	396.0	82.6	139.0	185.0	75.1	35.0	50.0	70.0	-3.5	38.0	23.0	26.0	7.0	48.0	636.0	35.0	2.0	4.0	120.0	11.0	67.0	14.0	0.0	1.0	516.0	88.0	67.0	126.0	524.0	2.0	9.0	7.0	2.0	10.0	21.0	28.0	91.0	5.41	60.0	2.0	19.0	6.0	4.0	12.0	0.71	9.0	2.0	0.0	1.0	4.0	2.0	1.0	1.0	2.0	1.0	9.0	11.1	8.0	137.0	40.0	29.2	12.0	48.0	77.0	8.0	3.0	5.0	1.0	1.0	937.0	2.0	24.0	344.0	647.0	208.0	904.0	59.0	115.0	51.3	70.0	4.0	777.0	5887.0	4153.0	1040.0	813.0	78.2	44.0	43.0	11.0	22.0	18.0	1.0	69.0	4.0	4.0	50.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
4	342229.0	20/21	Kylian Mbappé	kylian mbappé	22.0	22.0	19.0	1998-12-20	Paris	France	France	FRA	NaN	Paris S-G	ST	FW,MF	attack - Centre-Forward	Forward	Outfielder	178.0	right	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	Relatives	13.0	9.0	815.0	12.0	4.0	4.0	4.0	1.0	0.0	1.33	0.44	1.77	0.88	1.33	11.6	8.6	2.2	1.28	0.24	1.53	0.95	1.19	9.1	38.0	20.0	1.0	52.6	4.20	2.21	0.21	0.40	0.23	0.4	-0.6	310.0	388.0	79.9	4139.0	695.0	193.0	230.0	83.9	91.0	106.0	85.8	9.0	18.0	50.0	1.8	15.0	12.0	10.0	3.0	19.0	366.0	22.0	4.0	2.0	54.0	7.0	23.0	6.0	2.0	0.0	327.0	45.0	16.0	80.0	299.0	4.0	4.0	1.0	1.0	3.0	6.0	21.0	43.0	4.75	25.0	0.0	8.0	4.0	6.0	12.0	1.33	4.0	4.0	1.0	3.0	4.0	1.0	1.0	0.0	3.0	1.0	4.0	25.0	3.0	48.0	15.0	31.3	2.0	11.0	35.0	4.0	0.0	4.0	0.0	1.0	532.0	0.0	13.0	181.0	380.0	113.0	509.0	40.0	78.0	51.3	44.0	5.0	400.0	3282.0	2336.0	579.0	455.0	78.6	19.0	20.0	8.0	16.0	7.0	3.0	36.0	3.0	3.0	50.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN

In [11]:

# Display the last 5 rows of the raw DataFrame, df_fbref_tm
df_fbref_tm.tail()

Out[11]:

	tm_id	Season	player	player_name	age	age_tm	age_when_joining	dob	pob	cob	nationality	nationality_code	second_citizenship	club	position_code	position_fbref	position_TM	position_grouped	outfielder_goalkeeper	height	foot	market_value_euros	market_value_pounds	joined	years_since_joining	contract_expires	years_until_contract_expiry	contract_option	on_loan_from	on_loan_from_country	loan_contract_expiry	player_agent	games	games_starts	minutes	goals	assists	pens_made	pens_att	cards_yellow	cards_red	goals_per90	assists_per90	goals_assists_per90	goals_pens_per90	goals_assists_pens_per90	xg	npxg	xa	xg_per90	xa_per90	xg_xa_per90	npxg_per90	npxg_xa_per90	minutes_90s	shots_total	shots_on_target	shots_free_kicks	shots_on_target_pct	shots_total_per90	shots_on_target_per90	goals_per_shot	goals_per_shot_on_target	npxg_per_shot	xg_net	npxg_net	passes_completed	passes	passes_pct	passes_total_distance	passes_progressive_distance	passes_completed_short	passes_short	passes_pct_short	passes_completed_medium	passes_medium	passes_pct_medium	passes_completed_long	passes_long	passes_pct_long	xa_net	assisted_shots	passes_into_final_third	passes_into_penalty_area	crosses_into_penalty_area	progressive_passes	passes_live	passes_dead	passes_free_kicks	through_balls	passes_pressure	passes_switches	crosses	corner_kicks	corner_kicks_in	corner_kicks_out	corner_kicks_straight	passes_ground	passes_low	passes_high	passes_left_foot	passes_right_foot	passes_head	throw_ins	passes_other_body	passes_offsides	passes_oob	passes_intercepted	passes_blocked	sca	sca_per90	sca_passes_live	sca_passes_dead	sca_dribbles	sca_shots	sca_fouled	gca	gca_per90	gca_passes_live	gca_passes_dead	gca_dribbles	gca_shots	gca_fouled	gca_og_for	tackles	tackles_won	tackles_def_3rd	tackles_mid_3rd	tackles_att_3rd	dribble_tackles	dribbles_vs	dribble_tackles_pct	dribbled_past	pressures	pressure_regains	pressure_regain_pct	pressures_def_3rd	pressures_mid_3rd	pressures_att_3rd	blocks	blocked_shots	blocked_shots_saves	blocked_passes	interceptions	clearances	errors	touches	touches_def_pen_area	touches_def_3rd	touches_mid_3rd	touches_att_3rd	touches_att_pen_area	touches_live_ball	dribbles_completed	dribbles	dribbles_completed_pct	players_dribbled_past	nutmegs	carries	carry_distance	carry_progressive_distance	pass_targets	passes_received	passes_received_pct	miscontrols	dispossessed	cards_yellow_red	fouls	fouled	offsides	pens_won	pens_conceded	own_goals	ball_recoveries	aerials_won	aerials_lost	aerials_won_pct	games_gk	games_starts_gk	minutes_gk	goals_against_gk	goals_against_per90_gk	shots_on_target_against	saves	save_pct	wins_gk	draws_gk	losses_gk	clean_sheets	clean_sheets_pct	pens_att_gk	pens_allowed	pens_saved	pens_missed_gk	minutes_90s_gk	free_kick_goals_against_gk	corner_kick_goals_against_gk	own_goals_against_gk	psxg_gk	psnpxg_per_shot_on_target_against	psxg_net_gk	psxg_net_per90_gk	passes_completed_launched_gk	passes_launched_gk	passes_pct_launched_gk	passes_gk	passes_throws_gk	pct_passes_launched_gk	passes_length_avg_gk	goal_kicks	pct_goal_kicks_launched	goal_kick_length_avg	crosses_gk	crosses_stopped_gk	crosses_stopped_pct_gk	def_actions_outside_pen_area_gk	def_actions_outside_pen_area_per90_gk	avg_distance_def_actions_gk
11300	NaN	18/19	Łukasz Skorupski	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Poland	POL	NaN	Bologna	NaN	GK	NaN	Goalkeeper	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	38.0	38.0	3420.0	56.0	1.47	173.0	121.0	0.699	11.0	11.0	16.0	10.0	26.3	4.0	4.0	0.0	0.0	38.0	1.0	5.0	1.0	50.4	0.28	-4.6	-0.12	265.0	607.0	43.7	783.0	221.0	48.5	40.8	343.0	66.2	51.3	406.0	41.0	10.1	9.0	0.24	12.3
11301	NaN	19/20	Łukasz Skorupski	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Poland	POL	NaN	Bologna	NaN	GK	NaN	Goalkeeper	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	37.0	37.0	3330.0	64.0	1.73	180.0	122.0	0.678	12.0	10.0	15.0	2.0	5.4	8.0	6.0	1.0	1.0	37.0	2.0	6.0	1.0	57.6	0.30	-5.4	-0.15	197.0	445.0	44.3	855.0	216.0	37.7	35.3	292.0	42.1	34.8	320.0	46.0	14.4	15.0	0.41	12.9
11302	NaN	20/21	Łukasz Skorupski	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Poland	POL	NaN	Bologna	NaN	GK	NaN	Goalkeeper	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	10.0	10.0	900.0	17.0	1.70	45.0	30.0	0.644	4.0	0.0	6.0	1.0	10.0	1.0	1.0	0.0	0.0	10.0	0.0	3.0	1.0	13.3	0.27	-2.7	-0.27	46.0	117.0	39.3	276.0	48.0	34.1	33.3	71.0	32.4	28.3	78.0	13.0	16.7	5.0	0.50	14.3
11303	NaN	18/19	Łukasz Teodorczyk	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Poland	POL	NaN	Udinese	NaN	FW,MF	NaN	Forward	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	16.0	2.0	400.0	1.0	0.0	0.0	1.0	1.0	0.0	0.23	0.00	0.23	0.23	0.23	1.9	1.0	0.0	0.44	0.01	0.45	0.23	0.24	4.4	12.0	2.0	0.0	16.7	2.70	0.45	0.08	0.5	0.08	-0.9	0.0	54.0	87.0	62.1	882.0	157.0	32.0	38.0	84.2	15.0	27.0	55.6	6.0	11.0	54.5	0.0	2.0	3.0	1.0	0.0	3.0	83.0	4.0	0.0	1.0	18.0	5.0	5.0	0.0	0.0	0.0	0.0	47.0	14.0	26.0	13.0	50.0	10.0	1.0	5.0	0.0	7.0	1.0	9.0	9.0	2.03	6.0	0.0	1.0	1.0	1.0	2.0	0.45	0.0	0.0	0.0	1.0	1.0	0.0	5.0	3.0	2.0	1.0	2.0	1.0	7.0	14.3	6.0	101.0	15.0	14.9	6.0	64.0	31.0	7.0	2.0	0.0	5.0	2.0	5.0	0.0	152.0	8.0	16.0	72.0	70.0	29.0	148.0	6.0	8.0	75.0	6.0	0.0	93.0	503.0	156.0	187.0	110.0	58.8	13.0	15.0	0.0	20.0	8.0	4.0	0.0	0.0	0.0	16.0	11.0	22.0	33.3	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
11304	NaN	19/20	Łukasz Teodorczyk	NaN	NaN	NaN	NaN	NaN	NaN	NaN	Poland	POL	NaN	Udinese	NaN	FW	NaN	Forward	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	14.0	1.0	222.0	0.0	1.0	0.0	0.0	1.0	0.0	0.00	0.41	0.41	0.00	0.41	0.4	0.4	0.8	0.18	0.33	0.51	0.18	0.51	2.5	4.0	2.0	0.0	50.0	1.62	0.81	0.00	0.0	0.11	-0.4	-0.4	55.0	74.0	74.3	884.0	184.0	28.0	31.0	90.3	19.0	29.0	65.5	3.0	8.0	37.5	0.2	5.0	6.0	1.0	0.0	7.0	70.0	4.0	0.0	1.0	26.0	1.0	1.0	0.0	0.0	0.0	0.0	49.0	10.0	15.0	19.0	39.0	6.0	1.0	0.0	0.0	1.0	0.0	1.0	6.0	2.43	6.0	0.0	0.0	0.0	0.0	2.0	0.81	2.0	0.0	0.0	0.0	0.0	0.0	8.0	5.0	2.0	4.0	2.0	2.0	6.0	33.3	4.0	79.0	18.0	22.8	19.0	28.0	32.0	1.0	0.0	0.0	1.0	0.0	2.0	0.0	109.0	1.0	15.0	63.0	43.0	10.0	105.0	4.0	7.0	57.1	4.0	0.0	68.0	558.0	348.0	114.0	74.0	64.9	3.0	10.0	0.0	11.0	3.0	3.0	0.0	0.0	0.0	18.0	9.0	11.0	45.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN

In [12]:

df_fbref_tm[df_fbref_tm['player'] == 'Raheem Sterling']

Out[12]:

	tm_id	Season	player	player_name	age	age_tm	age_when_joining	dob	pob	cob	nationality	nationality_code	second_citizenship	club	position_code	position_fbref	position_TM	position_grouped	outfielder_goalkeeper	height	foot	market_value_euros	market_value_pounds	joined	years_since_joining	contract_expires	years_until_contract_expiry	contract_option	on_loan_from	on_loan_from_country	loan_contract_expiry	player_agent	games	games_starts	minutes	goals	assists	pens_made	pens_att	cards_yellow	cards_red	goals_per90	assists_per90	goals_assists_per90	goals_pens_per90	goals_assists_pens_per90	xg	npxg	xa	xg_per90	xa_per90	xg_xa_per90	npxg_per90	npxg_xa_per90	minutes_90s	shots_total	shots_on_target	shots_free_kicks	shots_on_target_pct	shots_total_per90	shots_on_target_per90	goals_per_shot	goals_per_shot_on_target	npxg_per_shot	xg_net	npxg_net	passes_completed	passes	passes_pct	passes_total_distance	passes_progressive_distance	passes_completed_short	passes_short	passes_pct_short	passes_completed_medium	passes_medium	passes_pct_medium	passes_completed_long	passes_long	passes_pct_long	xa_net	assisted_shots	passes_into_final_third	passes_into_penalty_area	crosses_into_penalty_area	progressive_passes	passes_live	passes_dead	passes_free_kicks	through_balls	passes_pressure	passes_switches	crosses	corner_kicks	corner_kicks_in	passes_ground	passes_low	passes_high	passes_left_foot	passes_right_foot	passes_head	throw_ins	passes_other_body	passes_offsides	passes_oob	passes_intercepted	passes_blocked	sca	sca_per90	sca_passes_live	sca_passes_dead	sca_dribbles	sca_shots	sca_fouled	gca	gca_per90	gca_passes_live	gca_dribbles	gca_shots	gca_fouled	gca_og_for	tackles	tackles_won	tackles_def_3rd	tackles_mid_3rd	tackles_att_3rd	dribble_tackles	dribbles_vs	dribble_tackles_pct	dribbled_past	pressures	pressure_regains	pressure_regain_pct	pressures_def_3rd	pressures_mid_3rd	pressures_att_3rd	blocks	blocked_shots	blocked_passes	interceptions	clearances	touches	touches_def_pen_area	touches_def_3rd	touches_mid_3rd	touches_att_3rd	touches_att_pen_area	touches_live_ball	dribbles_completed	dribbles	dribbles_completed_pct	players_dribbled_past	nutmegs	carries	carry_distance	carry_progressive_distance	pass_targets	passes_received	passes_received_pct	miscontrols	dispossessed	cards_yellow_red	fouls	fouled	offsides	pens_won	pens_conceded	ball_recoveries	aerials_won	aerials_lost	aerials_won_pct	games_gk	games_starts_gk	minutes_gk	goals_against_gk	goals_against_per90_gk	shots_on_target_against	saves	save_pct	wins_gk	draws_gk	losses_gk	clean_sheets	clean_sheets_pct	pens_att_gk	pens_allowed	pens_saved	pens_missed_gk	minutes_90s_gk	free_kick_goals_against_gk	corner_kick_goals_against_gk	own_goals_against_gk	psxg_gk	psnpxg_per_shot_on_target_against	psxg_net_gk	psxg_net_per90_gk	passes_completed_launched_gk	passes_launched_gk	passes_pct_launched_gk	passes_gk	passes_throws_gk	pct_passes_launched_gk	passes_length_avg_gk	goal_kicks	pct_goal_kicks_launched	goal_kick_length_avg	crosses_gk	crosses_stopped_gk	crosses_stopped_pct_gk	def_actions_outside_pen_area_gk	def_actions_outside_pen_area_per90_gk	avg_distance_def_actions_gk
25	134425.0	17/18	Raheem Sterling	raheem sterling	22.0	26.0	20.0	1994-12-08	Kingston	Jamaica	England	ENG	Jamaica	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	170.0	right	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	Colossal Sports Management	33.0	29.0	2587.0	18.0	11.0	1.0	1.0	4.0	2.0	0.63	0.38	1.01	0.59	0.97	17.2	16.2	6.4	0.60	0.22	0.82	0.56	0.79	28.7	85.0	33.0	0.0	38.8	2.96	1.15	0.20	0.52	0.19	0.8	0.8	999.0	1205.0	82.9	13467.0	2885.0	618.0	703.0	87.9	313.0	371.0	84.4	29.0	50.0	58.0	4.6	53.0	32.0	60.0	6.0	88.0	1188.0	17.0	1.0	4.0	265.0	11.0	47.0	5.0	1.0	1036.0	67.0	102.0	171.0	994.0	24.0	6.0	7.0	0.0	11.0	26.0	60.0	134.0	4.65	94.0	1.0	16.0	5.0	18.0	27.0	0.94	20.0	1.0	0.0	6.0	0.0	22.0	17.0	8.0	8.0	6.0	5.0	14.0	35.7	9.0	300.0	99.0	33.0	59.0	107.0	134.0	31.0	1.0	30.0	14.0	2.0	1603.0	11.0	96.0	503.0	1116.0	247.0	1587.0	46.0	90.0	51.1	52.0	1.0	1307.0	8057.0	4812.0	1650.0	1312.0	79.5	70.0	64.0	1.0	39.0	58.0	19.0	6.0	1.0	161.0	4.0	8.0	33.3	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
26	134425.0	18/19	Raheem Sterling	raheem sterling	23.0	26.0	20.0	1994-12-08	Kingston	Jamaica	England	ENG	Jamaica	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	170.0	right	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	Colossal Sports Management	34.0	31.0	2771.0	17.0	9.0	0.0	0.0	3.0	0.0	0.55	0.29	0.84	0.55	0.84	13.1	13.1	9.2	0.43	0.30	0.73	0.43	0.73	30.8	76.0	37.0	1.0	48.7	2.47	1.20	0.22	0.46	0.17	3.9	3.9	1099.0	1337.0	82.2	15362.0	3204.0	694.0	764.0	90.8	356.0	434.0	82.0	32.0	78.0	41.0	-0.2	58.0	22.0	52.0	5.0	87.0	1300.0	37.0	4.0	6.0	322.0	19.0	90.0	19.0	0.0	1097.0	115.0	125.0	147.0	1139.0	27.0	14.0	5.0	3.0	14.0	35.0	47.0	130.0	4.22	89.0	0.0	22.0	8.0	9.0	27.0	0.88	18.0	4.0	0.0	4.0	1.0	29.0	17.0	8.0	15.0	6.0	9.0	20.0	45.0	11.0	475.0	135.0	28.4	81.0	211.0	183.0	36.0	1.0	35.0	21.0	6.0	1765.0	5.0	77.0	569.0	1237.0	294.0	1729.0	86.0	142.0	60.6	91.0	7.0	1583.0	9550.0	5948.0	1848.0	1453.0	78.6	60.0	105.0	0.0	41.0	44.0	13.0	2.0	0.0	161.0	8.0	26.0	23.5	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
27	134425.0	19/20	Raheem Sterling	raheem sterling	24.0	26.0	20.0	1994-12-08	Kingston	Jamaica	England	ENG	Jamaica	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	170.0	right	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	Colossal Sports Management	33.0	30.0	2651.0	20.0	1.0	0.0	2.0	5.0	0.0	0.68	0.03	0.71	0.68	0.71	17.1	15.1	7.0	0.58	0.24	0.82	0.51	0.75	29.5	96.0	36.0	1.0	37.5	3.26	1.22	0.21	0.56	0.16	2.9	4.9	983.0	1217.0	80.8	13850.0	2795.0	624.0	702.0	88.9	301.0	370.0	81.4	37.0	63.0	58.7	-6.0	47.0	31.0	54.0	6.0	87.0	1197.0	20.0	1.0	4.0	290.0	4.0	63.0	1.0	0.0	965.0	150.0	102.0	133.0	1027.0	19.0	13.0	14.0	1.0	13.0	33.0	55.0	117.0	3.97	82.0	1.0	10.0	12.0	11.0	11.0	0.37	5.0	0.0	3.0	3.0	0.0	41.0	29.0	8.0	19.0	14.0	10.0	33.0	30.3	23.0	412.0	132.0	32.0	67.0	178.0	167.0	25.0	3.0	22.0	10.0	8.0	1636.0	12.0	93.0	571.0	1087.0	296.0	1616.0	59.0	128.0	46.1	67.0	3.0	1328.0	8795.0	5850.0	1688.0	1343.0	79.6	51.0	71.0	0.0	30.0	40.0	12.0	3.0	0.0	160.0	20.0	35.0	36.4	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
28	134425.0	20/21	Raheem Sterling	raheem sterling	26.0	26.0	20.0	1994-12-08	Kingston	Jamaica	England	ENG	Jamaica	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	170.0	right	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	Colossal Sports Management	12.0	11.0	999.0	4.0	2.0	0.0	0.0	1.0	0.0	0.36	0.18	0.54	0.36	0.54	3.6	3.6	1.5	0.33	0.14	0.46	0.33	0.46	11.1	25.0	12.0	1.0	48.0	2.25	1.08	0.16	0.33	0.14	0.4	0.4	342.0	408.0	83.8	4461.0	935.0	217.0	242.0	89.7	104.0	117.0	88.9	6.0	13.0	46.2	0.5	13.0	23.0	7.0	1.0	25.0	396.0	12.0	1.0	0.0	85.0	1.0	12.0	2.0	0.0	353.0	33.0	22.0	46.0	341.0	13.0	6.0	2.0	1.0	2.0	10.0	18.0	39.0	3.51	26.0	0.0	6.0	2.0	5.0	5.0	0.45	2.0	1.0	1.0	1.0	0.0	11.0	11.0	1.0	7.0	3.0	2.0	8.0	25.0	6.0	134.0	40.0	29.9	16.0	64.0	54.0	9.0	0.0	9.0	2.0	1.0	551.0	3.0	33.0	232.0	324.0	90.0	538.0	26.0	48.0	54.2	28.0	0.0	467.0	3024.0	1908.0	630.0	467.0	74.1	20.0	42.0	0.0	14.0	16.0	6.0	1.0	0.0	41.0	1.0	9.0	10.0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN

In [13]:

# Print the shape of the raw DataFrame, df_fbref_tm
print(df_fbref_tm.shape)

(11305, 218)

In [14]:

# Print the column names of the raw DataFrame, df_fbref_tm
print(df_fbref_tm.columns)

Index(['tm_id', 'Season', 'player', 'player_name', 'age', 'age_tm',
       'age_when_joining', 'dob', 'pob', 'cob',
       ...
       'passes_length_avg_gk', 'goal_kicks', 'pct_goal_kicks_launched',
       'goal_kick_length_avg', 'crosses_gk', 'crosses_stopped_gk',
       'crosses_stopped_pct_gk', 'def_actions_outside_pen_area_gk',
       'def_actions_outside_pen_area_per90_gk', 'avg_distance_def_actions_gk'],
      dtype='object', length=218)

The dataset has thirteen features (columns). Full details of these attributes can be found in the Data Dictionary.

In [15]:

# Data types of the features of the raw DataFrame, df_fbref_tm
df_fbref_tm.dtypes

Out[15]:

tm_id                                    float64
Season                                    object
player                                    object
player_name                               object
age                                      float64
                                          ...   
crosses_stopped_gk                       float64
crosses_stopped_pct_gk                   float64
def_actions_outside_pen_area_gk          float64
def_actions_outside_pen_area_per90_gk    float64
avg_distance_def_actions_gk              float64
Length: 218, dtype: object

All thirteen of the columns have the object data type. Full details of these attributes and their data types can be found in the Data Dictionary.

In [16]:

# Info for the raw DataFrame, df_fbref_tm
df_fbref_tm.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 11305 entries, 0 to 11304
Columns: 218 entries, tm_id to avg_distance_def_actions_gk
dtypes: float64(195), object(23)
memory usage: 18.8+ MB

In [17]:

# Description of the raw DataFrame, df_fbref_tm, showing some summary statistics for each numberical column in the DataFrame
df_fbref_tm.describe()

Out[17]:

	tm_id	age	age_tm	age_when_joining	height	market_value_euros	market_value_pounds	years_since_joining	years_until_contract_expiry	games	games_starts	minutes	goals	assists	pens_made	pens_att	cards_yellow	cards_red	goals_per90	assists_per90	goals_assists_per90	goals_pens_per90	goals_assists_pens_per90	xg	npxg	xa	xg_per90	xa_per90	xg_xa_per90	npxg_per90	npxg_xa_per90	minutes_90s	shots_total	shots_on_target	shots_free_kicks	shots_on_target_pct	shots_total_per90	shots_on_target_per90	goals_per_shot	goals_per_shot_on_target	npxg_per_shot	xg_net	npxg_net	passes_completed	passes	passes_pct	passes_total_distance	passes_progressive_distance	passes_completed_short	passes_short	passes_pct_short	passes_completed_medium	passes_medium	passes_pct_medium	passes_completed_long	passes_long	passes_pct_long	xa_net	assisted_shots	passes_into_final_third	passes_into_penalty_area	crosses_into_penalty_area	progressive_passes	passes_live	passes_dead	passes_free_kicks	through_balls	passes_pressure	passes_switches	crosses	corner_kicks	corner_kicks_in	corner_kicks_out	corner_kicks_straight	passes_ground	passes_low	passes_high	passes_left_foot	passes_right_foot	passes_head	throw_ins	passes_other_body	passes_offsides	passes_oob	passes_intercepted	passes_blocked	sca	sca_per90	sca_passes_live	sca_passes_dead	sca_dribbles	sca_shots	sca_fouled	gca	gca_per90	gca_passes_live	gca_passes_dead	gca_dribbles	gca_shots	gca_fouled	gca_og_for	tackles	tackles_won	tackles_def_3rd	tackles_mid_3rd	tackles_att_3rd	dribble_tackles	dribbles_vs	dribble_tackles_pct	dribbled_past	pressures	pressure_regains	pressure_regain_pct	pressures_def_3rd	pressures_mid_3rd	pressures_att_3rd	blocks	blocked_shots	blocked_shots_saves	blocked_passes	interceptions	clearances	errors	touches	touches_def_pen_area	touches_def_3rd	touches_mid_3rd	touches_att_3rd	touches_att_pen_area	touches_live_ball	dribbles_completed	dribbles	dribbles_completed_pct	players_dribbled_past	nutmegs	carries	carry_distance	carry_progressive_distance	pass_targets	passes_received	passes_received_pct	miscontrols	dispossessed	cards_yellow_red	fouls	fouled	offsides	pens_won	pens_conceded	own_goals	ball_recoveries	aerials_won	aerials_lost	aerials_won_pct	games_gk	games_starts_gk	minutes_gk	goals_against_gk	goals_against_per90_gk	shots_on_target_against	saves	save_pct	wins_gk	draws_gk	losses_gk	clean_sheets	clean_sheets_pct	pens_att_gk	pens_allowed	pens_saved	pens_missed_gk	minutes_90s_gk	free_kick_goals_against_gk	corner_kick_goals_against_gk	own_goals_against_gk	psxg_gk	psnpxg_per_shot_on_target_against	psxg_net_gk	psxg_net_per90_gk	passes_completed_launched_gk	passes_launched_gk	passes_pct_launched_gk	passes_gk	passes_throws_gk	pct_passes_launched_gk	passes_length_avg_gk	goal_kicks	pct_goal_kicks_launched	goal_kick_length_avg	crosses_gk	crosses_stopped_gk	crosses_stopped_pct_gk	def_actions_outside_pen_area_gk	def_actions_outside_pen_area_per90_gk	avg_distance_def_actions_gk
count	8700.000000	8700.000000	8700.000000	8700.000000	8699.000000	8.700000e+03	8.700000e+03	8700.000000	8558.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.00000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.00000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	10536.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000	769.000000
mean	198745.178736	25.478621	27.154023	24.646322	183.010346	1.105018e+07	9.834664e+06	2.132989	1.617434	16.327733	12.751139	1144.773918	1.553341	1.064256	0.150342	0.192008	2.359814	0.126329	0.115119	0.080020	0.195135	0.105606	0.185611	1.555790	1.408922	1.036409	0.127524	0.083903	0.211465	0.118204	0.202193	12.719922	14.452164	4.814161	0.606872	24.469913	1.212727	0.386064	0.065485	0.178448	0.072630	-0.002544	-0.005923	445.634871	565.974943	76.122115	8827.256264	3038.852506	177.093869	204.182992	84.519514	192.297077	225.278284	81.062842	69.074222	117.065300	56.27574	0.027752	10.516705	34.107346	9.205771	2.548121	38.881929	511.013762	54.961181	14.751139	1.013667	90.14683	17.238421	14.329537	5.674544	1.200835	1.235383	0.329822	367.441819	76.063117	122.470008	160.832574	334.765850	26.652050	23.553910	7.622532	1.989655	11.138573	9.240509	14.304005	22.810744	1.847212	16.306758	2.071944	1.433276	1.150816	1.348994	2.601462	0.198038	1.743831	0.170463	0.185175	0.213269	0.208808	0.025816	19.616648	12.316344	9.839882	7.485668	2.291097	6.481682	19.180049	28.157375	12.698368	179.524582	49.865793	26.870852	59.812927	81.041572	38.670084	18.167711	3.866268	0.081340	14.301443	9.700550	25.451500	0.299829	699.534169	71.681188	227.078683	338.005695	179.133162	25.844818	646.053341	11.736427	19.259491	53.507878	12.706910	0.617502	453.841591	2660.550209	1449.656226	523.986238	445.633922	83.905942	14.367122	13.143793	0.053721	14.664389	14.064825	2.306473	0.146450	0.177582	0.046128	107.038724	16.218299	16.201879	40.133125	16.042913	15.866060	1426.869961	21.945384	1.528088	65.871261	46.286086	0.672059	5.931079	3.993498	5.931079	4.308192	24.626138	2.650195	2.058518	0.444733	0.146944	15.853706	0.503251	2.769831	0.633290	21.075553	0.295527	-0.185826	-0.053810	103.299090	258.067620	40.693238	377.946684	64.014304	46.858388	39.819376	114.182055	64.492458	49.987646	147.786736	11.448635	7.300650	9.986996	0.621027	14.262159
std	140916.420958	4.278513	4.287700	4.287661	6.693939	1.653892e+07	1.471964e+07	2.452002	1.178846	11.042952	10.778600	939.671219	3.091977	1.833079	0.685732	0.809798	2.620589	0.402514	0.359518	0.307404	0.486038	0.350339	0.477857	2.723957	2.367715	1.534730	0.209021	0.287236	0.369308	0.195932	0.360704	10.440818	19.770117	7.624307	2.129631	22.681837	1.580075	0.701303	0.112935	0.256188	0.065662	1.092530	1.074838	442.020033	534.511593	12.273815	9112.557105	3600.506319	180.555702	203.138924	14.462939	207.670695	232.470127	16.377774	84.450596	144.054658	20.81560	0.923221	14.325801	41.282225	12.876228	4.558782	44.320504	489.767835	82.794766	22.682831	2.053856	86.60123	21.773191	22.415632	17.073734	4.672015	4.871605	1.400619	373.104773	78.087450	137.478463	266.334692	393.887763	30.977225	55.994965	22.706910	2.570429	11.619288	10.063522	15.872323	28.181149	2.363747	19.475346	5.700910	2.850442	1.928414	2.381023	3.868968	0.446240	2.653250	0.634542	0.598981	0.554036	0.573635	0.163892	20.905584	13.448803	11.699808	8.678128	2.961647	7.640514	20.650102	22.288018	14.079527	179.511164	48.737734	12.875318	65.102729	89.400245	52.916640	18.536094	6.095666	0.327985	14.749407	11.428456	39.696326	0.686008	630.774586	164.082477	283.785218	347.334297	215.593757	37.305597	588.542610	16.071474	26.359860	30.966310	17.349937	1.274202	429.383972	2597.664857	1488.357245	478.293718	421.766470	16.470760	18.742053	16.593270	0.236570	14.419392	15.768498	4.866079	0.466131	0.462833	0.224207	102.282925	22.513176	20.034852	25.203098	13.162327	13.271661	1188.291334	18.635543	0.979380	56.838957	40.824974	0.156686	6.223208	3.854612	5.675623	4.462220	21.254558	2.648924	2.166376	0.758546	0.379136	13.203592	0.824623	2.947956	0.944135	18.063106	0.086241	3.401335	0.377456	97.184822	243.203325	11.361790	327.479748	57.849874	17.752818	8.553590	101.287149	24.308573	13.572026	129.255022	11.350605	5.172479	10.513500	0.491611	2.834688
min	2857.000000	14.000000	16.000000	15.000000	162.000000	0.000000e+00	0.000000e+00	0.000000	-1.000000	1.000000	0.000000	1.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	-0.280000	-0.280000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	-0.330000	-1.000000	0.000000	-7.700000	-6.900000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	-6.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.00000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	1.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	-15.000000	-2.180000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000
25%	79010.000000	22.000000	24.000000	21.000000	178.000000	1.600000e+06	1.424000e+06	0.000000	1.000000	7.000000	3.000000	319.750000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.100000	0.100000	0.000000	0.010000	0.010000	0.040000	0.010000	0.040000	3.575000	1.000000	0.000000	0.000000	0.000000	0.270000	0.000000	0.000000	0.000000	0.030000	-0.400000	-0.400000	96.000000	131.000000	70.800000	1687.500000	460.750000	38.000000	45.000000	81.400000	35.000000	45.000000	75.600000	9.000000	17.000000	46.60000	-0.300000	1.000000	5.000000	1.000000	0.000000	5.000000	117.000000	6.000000	1.000000	0.000000	20.00000	2.000000	1.000000	0.000000	0.000000	0.000000	0.000000	79.000000	16.000000	23.000000	19.000000	47.000000	4.000000	0.000000	0.000000	0.000000	2.000000	2.000000	2.000000	3.000000	0.540000	2.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	0.000000	3.000000	2.000000	1.000000	1.000000	0.000000	1.000000	3.000000	7.700000	2.000000	34.000000	9.000000	22.800000	8.000000	14.000000	3.000000	3.000000	0.000000	0.000000	2.000000	1.000000	2.000000	0.000000	175.000000	5.000000	31.000000	56.000000	20.000000	3.000000	162.000000	1.000000	2.000000	40.000000	1.000000	0.000000	110.000000	594.750000	301.000000	136.000000	111.000000	75.900000	2.000000	1.000000	0.000000	3.000000	2.000000	0.000000	0.000000	0.000000	0.000000	25.000000	2.000000	3.000000	23.800000	3.000000	3.000000	270.000000	5.000000	1.010000	13.000000	9.000000	0.632000	1.000000	1.000000	1.000000	1.000000	10.000000	0.000000	0.000000	0.000000	0.000000	3.000000	0.000000	0.000000	0.000000	4.400000	0.260000	-1.900000	-0.200000	17.000000	40.000000	35.100000	67.000000	11.000000	34.900000	34.200000	22.000000	47.800000	41.700000	27.000000	2.000000	4.500000	2.000000	0.320000	12.900000
50%	175169.000000	25.000000	27.000000	24.000000	183.000000	5.000000e+06	4.450000e+06	1.000000	2.000000	14.000000	10.000000	930.000000	0.000000	0.000000	0.000000	0.000000	2.000000	0.000000	0.000000	0.000000	0.080000	0.000000	0.080000	0.500000	0.500000	0.400000	0.050000	0.050000	0.120000	0.050000	0.120000	10.300000	7.000000	2.000000	0.000000	25.000000	0.800000	0.180000	0.000000	0.000000	0.060000	0.000000	0.000000	310.000000	410.000000	77.500000	5779.500000	1690.500000	122.000000	144.000000	86.900000	121.000000	149.000000	83.800000	36.500000	64.000000	57.80000	0.000000	5.000000	19.000000	4.000000	1.000000	24.000000	366.000000	22.000000	6.000000	0.000000	64.00000	9.000000	5.000000	0.000000	0.000000	0.000000	0.000000	252.000000	52.000000	75.000000	61.000000	178.000000	16.000000	2.000000	2.000000	1.000000	7.000000	6.000000	9.000000	12.000000	1.570000	9.000000	0.000000	0.000000	0.000000	0.000000	1.000000	0.100000	1.000000	0.000000	0.000000	0.000000	0.000000	0.000000	13.000000	8.000000	5.000000	5.000000	1.000000	4.000000	12.000000	28.600000	8.000000	126.000000	37.000000	27.600000	36.000000	51.000000	17.000000	12.500000	1.000000	0.000000	10.000000	6.000000	9.000000	0.000000	528.000000	20.000000	113.000000	231.500000	94.000000	12.000000	487.000000	6.000000	9.000000	59.100000	6.000000	0.000000	336.000000	1898.500000	991.000000	397.000000	331.000000	89.500000	7.000000	7.000000	0.000000	11.000000	9.000000	0.000000	0.000000	0.000000	0.000000	77.000000	8.000000	10.000000	43.200000	13.000000	13.000000	1170.000000	18.000000	1.420000	51.000000	35.000000	0.692000	4.000000	3.000000	4.000000	3.000000	23.800000	2.000000	1.000000	0.000000	0.000000	13.000000	0.000000	2.000000	0.000000	16.500000	0.290000	-0.200000	-0.030000	75.000000	188.000000	40.000000	313.000000	48.000000	46.700000	39.700000	89.000000	66.900000	51.000000	115.000000	8.000000	7.100000	6.000000	0.550000	14.300000
75%	289260.750000	28.000000	30.000000	27.000000	188.000000	1.300000e+07	1.157000e+07	3.000000	2.000000	26.000000	21.000000	1843.250000	2.000000	1.000000	0.000000	0.000000	4.000000	0.000000	0.140000	0.110000	0.280000	0.130000	0.260000	1.800000	1.700000	1.400000	0.170000	0.120000	0.300000	0.160000	0.290000	20.500000	19.000000	6.000000	0.000000	38.100000	1.800000	0.570000	0.110000	0.330000	0.100000	0.200000	0.200000	667.250000	865.250000	83.700000	13223.250000	4386.750000	259.250000	301.000000	91.300000	282.000000	334.000000	90.800000	97.000000	164.000000	69.20000	0.200000	15.000000	49.000000	13.000000	3.000000	59.000000	766.000000	62.000000	19.000000	1.000000	139.00000	24.000000	18.000000	2.000000	0.000000	0.000000	0.000000	539.000000	112.000000	175.000000	161.000000	498.250000	39.000000	11.000000	5.000000	3.000000	16.000000	14.000000	21.000000	32.000000	2.600000	23.000000	1.000000	2.000000	2.000000	2.000000	4.000000	0.290000	2.000000	0.000000	0.000000	0.000000	0.000000	0.000000	30.000000	19.000000	15.000000	11.000000	3.000000	10.000000	28.000000	42.300000	18.000000	269.250000	77.000000	32.400000	93.000000	116.000000	53.000000	28.000000	5.000000	0.000000	22.000000	15.000000	31.000000	0.000000	1083.000000	63.000000	315.250000	513.000000	264.000000	33.000000	977.250000	16.000000	26.000000	73.300000	17.000000	1.000000	680.000000	3939.000000	2110.000000	788.000000	667.000000	96.500000	20.000000	19.000000	0.000000	22.000000	20.000000	2.000000	0.000000	0.000000	0.000000	161.000000	21.000000	23.000000	57.800000	29.000000	29.000000	2610.000000	36.000000	1.770000	116.000000	81.000000	0.750000	9.000000	6.000000	10.000000	7.000000	33.300000	4.000000	3.000000	1.000000	0.000000	29.000000	1.000000	4.000000	1.000000	35.800000	0.330000	1.300000	0.140000	169.000000	417.000000	45.800000	657.000000	101.000000	57.500000	44.600000	198.000000	84.800000	60.500000	258.000000	19.000000	9.500000	15.000000	0.880000	15.700000
max	807128.000000	42.000000	42.000000	41.000000	202.000000	1.800000e+08	1.602000e+08	17.000000	7.000000	38.000000	38.000000	3420.000000	36.000000	21.000000	14.000000	15.000000	17.000000	5.000000	22.500000	18.000000	22.500000	22.500000	22.500000	29.700000	25.700000	18.400000	5.250000	25.400000	25.400000	5.250000	25.400000	38.000000	196.000000	92.000000	47.000000	100.000000	36.000000	30.000000	1.000000	1.000000	0.880000	12.700000	12.500000	2864.000000	3229.000000	100.000000	65376.000000	27017.000000	1638.000000	1745.000000	100.000000	1548.000000	1625.000000	100.000000	607.000000	1142.000000	100.00000	8.300000	131.000000	442.000000	141.000000	49.000000	321.000000	3153.000000	718.000000	232.000000	48.000000	527.00000	227.000000	223.000000	190.000000	85.000000	78.000000	25.000000	2505.000000	625.000000	1120.000000	2519.000000	2900.000000	237.000000	477.000000	234.000000	34.000000	98.000000	90.000000	122.000000	238.000000	90.000000	168.000000	68.000000	39.000000	27.000000	32.000000	43.000000	18.000000	27.000000	9.000000	11.000000	6.000000	7.000000	2.000000	160.000000	109.000000	98.000000	67.000000	28.000000	56.000000	141.000000	100.000000	98.000000	1056.000000	303.000000	100.000000	525.000000	604.000000	579.000000	125.000000	60.000000	4.000000	98.000000	89.000000	329.000000	7.000000	3520.000000	1441.000000	1683.000000	2395.000000	1746.000000	322.000000	3393.000000	184.000000	274.000000	100.000000	197.000000	14.000000	2617.000000	17993.000000	11141.000000	3096.000000	2833.000000	100.000000	138.000000	172.000000	2.000000	98.000000	167.000000	56.000000	6.000000	5.000000	4.000000	597.000000	266.000000	290.000000	100.000000	38.000000	38.000000	3420.000000	76.000000	15.000000	231.000000	167.000000	1.000000	32.000000	18.000000	25.000000	22.000000	100.000000	15.000000	13.000000	4.000000	2.000000	38.000000	5.000000	15.000000	5.000000	81.200000	0.790000	12.900000	1.720000	412.000000	1103.000000	100.000000	1362.000000	234.000000	100.000000	103.000000	451.000000	100.000000	79.300000	569.000000	53.000000	50.000000	59.000000	4.000000	27.500000

In [18]:

# Plot visualisation of the missing values for each feature of the raw DataFrame, df_fbref_tm
msno.matrix(df_fbref_tm, figsize = (30, 7))

Out[18]:

<matplotlib.axes._subplots.AxesSubplot at 0x10b4ead90>

In [19]:

# Counts of missing values
tm_bio_null_value_stats = df_fbref_tm.isnull().sum(axis=0)
tm_bio_null_value_stats[tm_bio_null_value_stats != 0]

Out[19]:

tm_id                                     2605
player_name                               2605
age                                       2605
age_tm                                    2605
age_when_joining                          2605
                                         ...  
crosses_stopped_gk                       10536
crosses_stopped_pct_gk                   10536
def_actions_outside_pen_area_gk          10536
def_actions_outside_pen_area_per90_gk    10536
avg_distance_def_actions_gk              10536
Length: 213, dtype: int64

The visualisation shows us very quickly that there a few missing values in most of the columns, but the dataset is ready for us.

4. Data Engineering ¶

Before we answer the questions in the brief through Exploratory Data Analysis (EDA), we'll first need to clean and wrangle the datasets to a form that meet our needs.

4.2. Select Columns of Interest ¶

In [35]:

# Select columns of interest
cols = ['player',
        'Season',
        'age',
        'age_tm',
        'age_when_joining',
        'dob',
        'cob',
        'nationality',
        'club',
        'position_code',
        'position_fbref',
        'position_TM',
        'position_grouped',
        'outfielder_goalkeeper',
        'market_value_euros',
        'market_value_pounds',
        'joined',
        'years_since_joining',
        'contract_expires',
        'years_until_contract_expiry',
        'contract_option',
        'on_loan_from',
        'on_loan_from_country',
        'loan_contract_expiry',
        'games',
        'games_starts',
        'minutes',
        'goals',
        'assists']

df_fbref_tm_select = df_fbref_tm[cols]

In [37]:

df_fbref_tm_select.head()

Out[37]:

	player	Season	age	age_tm	age_when_joining	dob	cob	nationality	club	position_code	position_fbref	position_TM	position_grouped	outfielder_goalkeeper	market_value_euros	market_value_pounds	joined	years_since_joining	contract_expires	years_until_contract_expiry	contract_option	on_loan_from	on_loan_from_country	loan_contract_expiry	games	games_starts	minutes	goals	assists
0	Kylian Mbappé	17/18	18.0	22.0	19.0	1998-12-20	France	France	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	27.0	24.0	2095.0	13.0	9.0
1	Kylian Mbappé	17/18	18.0	22.0	19.0	1998-12-20	France	France	Monaco	ST	FW,MF	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	1.0	1.0	74.0	0.0	0.0
2	Kylian Mbappé	18/19	19.0	22.0	19.0	1998-12-20	France	France	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	29.0	24.0	2343.0	33.0	7.0
3	Kylian Mbappé	19/20	20.0	22.0	19.0	1998-12-20	France	France	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	20.0	17.0	1513.0	18.0	5.0
4	Kylian Mbappé	20/21	22.0	22.0	19.0	1998-12-20	France	France	Paris S-G	ST	FW,MF	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	13.0	9.0	815.0	12.0	4.0

4.3. Pivot Dates ¶

In [41]:

df_fbref_tm_melt = df_fbref_tm_select.melt(id_vars=['player',
                                                    'Season',
                                                    'age',
                                                    'dob',
                                                    'cob',
                                                    'nationality',
                                                    'club',
                                                    'position_code',
                                                    'position_fbref',
                                                    'position_TM',
                                                    'position_grouped',
                                                    'outfielder_goalkeeper',
                                                    'market_value_euros',
                                                    'market_value_pounds',
                                                    'joined',
                                                    'years_since_joining',
                                                    'contract_expires',
                                                    'years_until_contract_expiry',
                                                    'contract_option',
                                                    'on_loan_from',
                                                    'on_loan_from_country',
                                                    'loan_contract_expiry',
                                                    'games',
                                                    'games_starts',
                                                    'minutes',
                                                    'goals',
                                                    'assists'],
                                            var_name='age_type', 
                                            value_name='age')

df_fbref_tm_melt = df_fbref_tm_melt.sort_values(['market_value_pounds', 'player', 'Season', 'club'], ascending=[False, True, True, True])

In [43]:

df_fbref_tm_melt.head(15)

Out[43]:

	player	Season	age	dob	cob	nationality	club	position_code	position_fbref	position_TM	position_grouped	outfielder_goalkeeper	market_value_euros	market_value_pounds	joined	years_since_joining	contract_expires	years_until_contract_expiry	contract_option	on_loan_from	on_loan_from_country	loan_contract_expiry	games	games_starts	minutes	goals	assists	age_type	age
1	Kylian Mbappé	17/18	22.0	1998-12-20	France	France	Monaco	ST	FW,MF	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	1.0	1.0	74.0	0.0	0.0	age_tm	22.0
11306	Kylian Mbappé	17/18	19.0	1998-12-20	France	France	Monaco	ST	FW,MF	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	1.0	1.0	74.0	0.0	0.0	age_when_joining	19.0
0	Kylian Mbappé	17/18	22.0	1998-12-20	France	France	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	27.0	24.0	2095.0	13.0	9.0	age_tm	22.0
11305	Kylian Mbappé	17/18	19.0	1998-12-20	France	France	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	27.0	24.0	2095.0	13.0	9.0	age_when_joining	19.0
2	Kylian Mbappé	18/19	22.0	1998-12-20	France	France	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	29.0	24.0	2343.0	33.0	7.0	age_tm	22.0
11307	Kylian Mbappé	18/19	19.0	1998-12-20	France	France	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	29.0	24.0	2343.0	33.0	7.0	age_when_joining	19.0
3	Kylian Mbappé	19/20	22.0	1998-12-20	France	France	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	20.0	17.0	1513.0	18.0	5.0	age_tm	22.0
11308	Kylian Mbappé	19/20	19.0	1998-12-20	France	France	Paris S-G	ST	FW	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	20.0	17.0	1513.0	18.0	5.0	age_when_joining	19.0
4	Kylian Mbappé	20/21	22.0	1998-12-20	France	France	Paris S-G	ST	FW,MF	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	13.0	9.0	815.0	12.0	4.0	age_tm	22.0
11309	Kylian Mbappé	20/21	19.0	1998-12-20	France	France	Paris S-G	ST	FW,MF	attack - Centre-Forward	Forward	Outfielder	180000000.0	160200000.0	2018-07-01	2.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	13.0	9.0	815.0	12.0	4.0	age_when_joining	19.0
5	Neymar	17/18	28.0	1992-02-05	Brazil	Brazil	Paris S-G	LW	FW,MF	attack - Left Winger	Forward	Outfielder	128000000.0	113920000.0	2017-08-03	3.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	20.0	20.0	1797.0	19.0	13.0	age_tm	28.0
11310	Neymar	17/18	25.0	1992-02-05	Brazil	Brazil	Paris S-G	LW	FW,MF	attack - Left Winger	Forward	Outfielder	128000000.0	113920000.0	2017-08-03	3.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	20.0	20.0	1797.0	19.0	13.0	age_when_joining	25.0
6	Neymar	18/19	28.0	1992-02-05	Brazil	Brazil	Paris S-G	LW	MF,FW	attack - Left Winger	Midfielder	Outfielder	128000000.0	113920000.0	2017-08-03	3.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	17.0	16.0	1439.0	15.0	7.0	age_tm	28.0
11311	Neymar	18/19	25.0	1992-02-05	Brazil	Brazil	Paris S-G	LW	MF,FW	attack - Left Winger	Midfielder	Outfielder	128000000.0	113920000.0	2017-08-03	3.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	17.0	16.0	1439.0	15.0	7.0	age_when_joining	25.0
7	Neymar	19/20	28.0	1992-02-05	Brazil	Brazil	Paris S-G	LW	MF,FW	attack - Left Winger	Midfielder	Outfielder	128000000.0	113920000.0	2017-08-03	3.0	2022-06-30	1.0	NaN	NaN	NaN	NaN	15.0	15.0	1318.0	13.0	6.0	age_tm	28.0

4.4. Exporting the Engineered DataFrames ¶

In [44]:

# Export DataFrame as a CSV file

## Export a copy to the 'archive' subfolder of the TM folder, including the date
df_fbref_tm_melt.to_csv(data_dir_fbref_tm + f'/archive/fbref_tm_big5_age_profile_last_updated_{today}.csv', index=None, header=True)

## Export another copy to the TM folder called 'latest' (can be overwritten)
df_fbref_tm_melt.to_csv(data_dir_fbref_tm + '/fbref_tm_big5_age_profile_latest.csv', index=None, header=True)

## Export a copy to the Export folder (can be overwritten)
df_fbref_tm_melt.to_csv(data_dir + '/export/fbref_tm_big5_age_profile_latest.csv', index=None, header=True)

Now we have created three pandas DataFrames and wrangled the data to meet our needs, we'll next conduct and Exploratory Data Analysis .

5. Exploratory Data Analysis ¶

...

In [45]:

df_fbref_tm_melt[df_fbref_tm_melt['player'] == 'Raheem Sterling']

Out[45]:

	player	Season	age	dob	cob	nationality	club	position_code	position_fbref	position_TM	position_grouped	outfielder_goalkeeper	market_value_euros	market_value_pounds	joined	years_since_joining	contract_expires	years_until_contract_expiry	contract_option	on_loan_from	on_loan_from_country	loan_contract_expiry	games	games_starts	minutes	goals	assists	age_type	age
25	Raheem Sterling	17/18	26.0	1994-12-08	Jamaica	England	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	33.0	29.0	2587.0	18.0	11.0	age_tm	26.0
11330	Raheem Sterling	17/18	20.0	1994-12-08	Jamaica	England	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	33.0	29.0	2587.0	18.0	11.0	age_when_joining	20.0
26	Raheem Sterling	18/19	26.0	1994-12-08	Jamaica	England	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	34.0	31.0	2771.0	17.0	9.0	age_tm	26.0
11331	Raheem Sterling	18/19	20.0	1994-12-08	Jamaica	England	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	34.0	31.0	2771.0	17.0	9.0	age_when_joining	20.0
27	Raheem Sterling	19/20	26.0	1994-12-08	Jamaica	England	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	33.0	30.0	2651.0	20.0	1.0	age_tm	26.0
11332	Raheem Sterling	19/20	20.0	1994-12-08	Jamaica	England	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	33.0	30.0	2651.0	20.0	1.0	age_when_joining	20.0
28	Raheem Sterling	20/21	26.0	1994-12-08	Jamaica	England	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	12.0	11.0	999.0	4.0	2.0	age_tm	26.0
11333	Raheem Sterling	20/21	20.0	1994-12-08	Jamaica	England	Manchester City	LW	FW	attack - Left Winger	Forward	Outfielder	110000000.0	97900000.0	2015-07-14	5.0	2023-06-30	2.0	NaN	NaN	NaN	NaN	12.0	11.0	999.0	4.0	2.0	age_when_joining	20.0

Data and Web Scraping¶

https://twitter.com/NdyStats/status/1343809317107986438
tyrone_mings GitHub repository by FCrSTATS
Python Package Index (PyPI) tyrone-mings library
Beyond crowd judgments: Data-driven estimation of market value in association football by Oliver Müllera, Alexander Simons, and Markus Weinmann.
06/04/2020: BBC - Premier League squads 'drop £1.6bn in value'.

*Visit my website EddWebster.com or my GitHub Repository for more projects. If you'd like to get in contact, my Twitter handle is @eddwebster and my email is: edd.j.webster@gmail.com.*

Back to the top

Data Engineering of Matched FBref and TransferMarkt Data for Age Profiling Visualisation¶

Notebook to engineer matched data scraped from TransferMarkt using the Tyrone Mings web scraper by FCrSTATS and StatsBomb data provided by FBref using a Python webscraper by parth1902.¶

By Edd Webster ¶

Introduction ¶

Notebook Contents ¶

1. Notebook Dependencies ¶

Import Libraries and Modules¶

Defined Variables¶

Defined Filepaths¶

Notebook Settings¶

2. Project Brief ¶

3. Data Sources ¶

3.2. Data Dictionaries ¶

3.3. Import CSV as pandas DataFrame ¶

3.4. Preliminary Data Handling ¶

4. Data Engineering ¶

4.2. Select Columns of Interest ¶

4.3. Pivot Dates ¶

4.4. Exporting the Engineered DataFrames ¶

5. Exploratory Data Analysis ¶

6. Summary ¶

7. Next Steps ¶

8. References ¶

Data and Web Scraping¶

Data Engineering of Matched FBref and TransferMarkt Data for Age Profiling Visualisation¶

Notebook to engineer matched data scraped from TransferMarkt using the Tyrone Mings web scraper by FCrSTATS and StatsBomb data provided by FBref using a Python webscraper by parth1902.¶

By Edd Webster¶

Import Libraries and Modules¶

Defined Variables¶

Defined Filepaths¶

Notebook Settings¶

Data and Web Scraping¶

By Edd Webster ¶