A Deep Dive into Geospatial Analysis

Many of the datasets that data scientists handle have some kind of geospatial component to them, and that information is oftentimes useful-to-critical for understanding the problem at hand. As such, an understanding of spatial data and how to work with it is a valuable skill for any data scientist to have. Even better, Python provides a rich toolset for working in this domain, and recent advances have greatly simplified and consolidated these.

In this tutorial we will take a deep dive into geospatial analysis in Python, using tools like geopandas, shapely, and pysal to analyze a dataset, provided by Kaggle (and originally from Inside AirBnB), of sample AirBnB locations in Boston, Massachusetts.

This tutorial is targeted at folks who know a thing or two about data but haven't used Python's geospatial data tools just yet. As such, it assumes a high level of familiarity with pandas. Some familiarity with scikit-learn, statsmodels, matplotlib, and seaborn is also helpful.

import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
pd.set_option("max_columns", None)

Table of Contents

• Plotting Points
• Plotting Geometries
• Upconverting DataFrame to GeoDataFrame Objects
• Plotting Geometries and Points
• Spatial Weights
• Spatial Lag
• Spatial Clustering
• Spatial Regression
• Conclusion
• Extra Credit

Plotting Points

First, let's boot up and examine our data. Since our data comes in a simple CSV file, we load it into a pandas DataFrame.

listings = pd.read_csv("../input/listings.csv")

listings.head()

	id	listing_url	scrape_id	last_scraped	name	summary	space	description	experiences_offered	neighborhood_overview	notes	transit	access	interaction	house_rules	thumbnail_url	medium_url	picture_url	xl_picture_url	host_id	host_url	host_name	host_since	host_location	host_about	host_response_time	host_response_rate	host_acceptance_rate	host_is_superhost	host_thumbnail_url	host_picture_url	host_neighbourhood	host_listings_count	host_total_listings_count	host_verifications	host_has_profile_pic	host_identity_verified	street	neighbourhood	neighbourhood_cleansed	neighbourhood_group_cleansed	city	state	zipcode	market	smart_location	country_code	country	latitude	longitude	is_location_exact	property_type	room_type	accommodates	bathrooms	bedrooms	beds	bed_type	amenities	square_feet	price	weekly_price	monthly_price	security_deposit	cleaning_fee	guests_included	extra_people	minimum_nights	maximum_nights	calendar_updated	has_availability	availability_30	availability_60	availability_90	availability_365	calendar_last_scraped	number_of_reviews	first_review	last_review	review_scores_rating	review_scores_accuracy	review_scores_cleanliness	review_scores_checkin	review_scores_communication	review_scores_location	review_scores_value	requires_license	license	jurisdiction_names	instant_bookable	cancellation_policy	require_guest_profile_picture	require_guest_phone_verification	calculated_host_listings_count	reviews_per_month
0	12147973	https://www.airbnb.com/rooms/12147973	20160906204935	2016-09-07	Sunny Bungalow in the City	Cozy, sunny, family home. Master bedroom high...	The house has an open and cozy feel at the sam...	Cozy, sunny, family home. Master bedroom high...	none	Roslindale is quiet, convenient and friendly. ...	NaN	The bus stop is 2 blocks away, and frequent. B...	You will have access to 2 bedrooms, a living r...	NaN	Clean up and treat the home the way you'd like...	https://a2.muscache.com/im/pictures/c0842db1-e...	https://a2.muscache.com/im/pictures/c0842db1-e...	https://a2.muscache.com/im/pictures/c0842db1-e...	https://a2.muscache.com/im/pictures/c0842db1-e...	31303940	https://www.airbnb.com/users/show/31303940	Virginia	2015-04-15	Boston, Massachusetts, United States	We are country and city connecting in our deck...	NaN	NaN	NaN	f	https://a2.muscache.com/im/pictures/5936fef0-b...	https://a2.muscache.com/im/pictures/5936fef0-b...	Roslindale	1	1	['email', 'phone', 'facebook', 'reviews']	t	f	Birch Street, Boston, MA 02131, United States	Roslindale	Roslindale	NaN	Boston	MA	02131	Boston	Boston, MA	US	United States	42.282619	-71.133068	t	House	Entire home/apt	4	1.5	2.0	3.0	Real Bed	{TV,"Wireless Internet",Kitchen,"Free Parking ...	NaN	$250.00	NaN	NaN	NaN	$35.00	1	$0.00	2	1125	2 weeks ago	NaN	0	0	0	0	2016-09-06	0	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	f	NaN	NaN	f	moderate	f	f	1	NaN
1	3075044	https://www.airbnb.com/rooms/3075044	20160906204935	2016-09-07	Charming room in pet friendly apt	Charming and quiet room in a second floor 1910...	Small but cozy and quite room with a full size...	Charming and quiet room in a second floor 1910...	none	The room is in Roslindale, a diverse and prima...	If you don't have a US cell phone, you can tex...	Plenty of safe street parking. Bus stops a few...	Apt has one more bedroom (which I use) and lar...	If I am at home, I am likely working in my hom...	Pet friendly but please confirm with me if the...	https://a1.muscache.com/im/pictures/39327812/d...	https://a1.muscache.com/im/pictures/39327812/d...	https://a1.muscache.com/im/pictures/39327812/d...	https://a1.muscache.com/im/pictures/39327812/d...	2572247	https://www.airbnb.com/users/show/2572247	Andrea	2012-06-07	Boston, Massachusetts, United States	I live in Boston and I like to travel and have...	within an hour	100%	100%	f	https://a2.muscache.com/im/users/2572247/profi...	https://a2.muscache.com/im/users/2572247/profi...	Roslindale	1	1	['email', 'phone', 'facebook', 'linkedin', 'am...	t	t	Pinehurst Street, Boston, MA 02131, United States	Roslindale	Roslindale	NaN	Boston	MA	02131	Boston	Boston, MA	US	United States	42.286241	-71.134374	t	Apartment	Private room	2	1.0	1.0	1.0	Real Bed	{TV,Internet,"Wireless Internet","Air Conditio...	NaN	$65.00	$400.00	NaN	$95.00	$10.00	0	$0.00	2	15	a week ago	NaN	26	54	84	359	2016-09-06	36	2014-06-01	2016-08-13	94.0	10.0	9.0	10.0	10.0	9.0	9.0	f	NaN	NaN	t	moderate	f	f	1	1.30
2	6976	https://www.airbnb.com/rooms/6976	20160906204935	2016-09-07	Mexican Folk Art Haven in Boston	Come stay with a friendly, middle-aged guy in ...	Come stay with a friendly, middle-aged guy in ...	Come stay with a friendly, middle-aged guy in ...	none	The LOCATION: Roslindale is a safe and diverse...	I am in a scenic part of Boston with a couple ...	PUBLIC TRANSPORTATION: From the house, quick p...	I am living in the apartment during your stay,...	ABOUT ME: I'm a laid-back, friendly, unmarried...	I encourage you to use my kitchen, cooking and...	https://a2.muscache.com/im/pictures/6ae8335d-9...	https://a2.muscache.com/im/pictures/6ae8335d-9...	https://a2.muscache.com/im/pictures/6ae8335d-9...	https://a2.muscache.com/im/pictures/6ae8335d-9...	16701	https://www.airbnb.com/users/show/16701	Phil	2009-05-11	Boston, Massachusetts, United States	I am a middle-aged, single male with a wide ra...	within a few hours	100%	88%	t	https://a2.muscache.com/im/users/16701/profile...	https://a2.muscache.com/im/users/16701/profile...	Roslindale	1	1	['email', 'phone', 'reviews', 'jumio']	t	t	Ardale St., Boston, MA 02131, United States	Roslindale	Roslindale	NaN	Boston	MA	02131	Boston	Boston, MA	US	United States	42.292438	-71.135765	t	Apartment	Private room	2	1.0	1.0	1.0	Real Bed	{TV,"Cable TV","Wireless Internet","Air Condit...	NaN	$65.00	$395.00	$1,350.00	NaN	NaN	1	$20.00	3	45	5 days ago	NaN	19	46	61	319	2016-09-06	41	2009-07-19	2016-08-05	98.0	10.0	9.0	10.0	10.0	9.0	10.0	f	NaN	NaN	f	moderate	t	f	1	0.47
3	1436513	https://www.airbnb.com/rooms/1436513	20160906204935	2016-09-07	Spacious Sunny Bedroom Suite in Historic Home	Come experience the comforts of home away from...	Most places you find in Boston are small howev...	Come experience the comforts of home away from...	none	Roslindale is a lovely little neighborhood loc...	Please be mindful of the property as it is old...	There are buses that stop right in front of th...	The basement has a washer dryer and gym area. ...	We do live in the house therefore might be som...	- The bathroom and house are shared so please ...	https://a2.muscache.com/im/pictures/39764190-1...	https://a2.muscache.com/im/pictures/39764190-1...	https://a2.muscache.com/im/pictures/39764190-1...	https://a2.muscache.com/im/pictures/39764190-1...	6031442	https://www.airbnb.com/users/show/6031442	Meghna	2013-04-21	Boston, Massachusetts, United States	My husband and I live on the property. He’s a...	within a few hours	100%	50%	f	https://a2.muscache.com/im/pictures/5d430cde-7...	https://a2.muscache.com/im/pictures/5d430cde-7...	NaN	1	1	['email', 'phone', 'reviews']	t	f	Boston, MA, United States	NaN	Roslindale	NaN	Boston	MA	NaN	Boston	Boston, MA	US	United States	42.281106	-71.121021	f	House	Private room	4	1.0	1.0	2.0	Real Bed	{TV,Internet,"Wireless Internet","Air Conditio...	NaN	$75.00	NaN	NaN	$100.00	$50.00	2	$25.00	1	1125	a week ago	NaN	6	16	26	98	2016-09-06	1	2016-08-28	2016-08-28	100.0	10.0	10.0	10.0	10.0	10.0	10.0	f	NaN	NaN	f	moderate	f	f	1	1.00
4	7651065	https://www.airbnb.com/rooms/7651065	20160906204935	2016-09-07	Come Home to Boston	My comfy, clean and relaxing home is one block...	Clean, attractive, private room, one block fro...	My comfy, clean and relaxing home is one block...	none	I love the proximity to downtown, the neighbor...	I have one roommate who lives on the lower lev...	From Logan Airport and South Station you have...	You will have access to the front and side por...	I love my city and really enjoy sharing it wit...	Please no smoking in the house, porch or on th...	https://a1.muscache.com/im/pictures/97154760/8...	https://a1.muscache.com/im/pictures/97154760/8...	https://a1.muscache.com/im/pictures/97154760/8...	https://a1.muscache.com/im/pictures/97154760/8...	15396970	https://www.airbnb.com/users/show/15396970	Linda	2014-05-11	Boston, Massachusetts, United States	I work full time for a public school district....	within an hour	100%	100%	t	https://a0.muscache.com/im/users/15396970/prof...	https://a0.muscache.com/im/users/15396970/prof...	Roslindale	1	1	['email', 'phone', 'reviews', 'kba']	t	t	Durnell Avenue, Boston, MA 02131, United States	Roslindale	Roslindale	NaN	Boston	MA	02131	Boston	Boston, MA	US	United States	42.284512	-71.136258	t	House	Private room	2	1.5	1.0	2.0	Real Bed	{Internet,"Wireless Internet","Air Conditionin...	NaN	$79.00	NaN	NaN	NaN	$15.00	1	$0.00	2	31	2 weeks ago	NaN	13	34	59	334	2016-09-06	29	2015-08-18	2016-09-01	99.0	10.0	10.0	10.0	10.0	9.0	10.0	f	NaN	NaN	f	flexible	f	f	1	2.25

len(listings)

3585

At the moment our listings have several geospatial variables, the most important of which are longitude and latitude, which give the exact coordinates of the BnB in question. This means that it's easy for us to, say, plot every BnB location on a map:

plt.scatter(listings['longitude'], listings['latitude'])

<matplotlib.collections.PathCollection at 0x34240470>

Chances are you've already done this before, and it's a perfectly adequate way to get started working with locations. In this plot we see...not much, really. If you're very intimately familiar with the layout of the city of Boston, you will probably be able to make sense of some of these clusters which are, to me, not being from the city, totally mysterious.

In other words, this plot is missing something important: geospatial context.

Additionally, this display is unprojected—it's displayed in terms of raw coordinates. The amount of distance contained in a coordinate degree varies greatly depending on where you are, so this naive plot potentially pretty badly distorts distances.

We'll come back to the projection issue later; for now, there's an easy fix for both these problems.

Enter mplleaflet. mplleaflet is a tool that automatically takes a coordinate matplotlib plot of any kind and places it on top of a leaflet slippy map. The best part is that it's just one additional line of code. Just throw mplleaflet.display() after generating your plot to drop it inline in your Jupyter notebook:

import mplleaflet

sample = listings.sample(1000)
plt.scatter(sample['longitude'], sample['latitude'])

mplleaflet.display()