A Deep Dive into Geospatial Analysis¶
Many of the datasets that data scientists handle have some kind of geospatial component to them, and that information is oftentimes useful-to-critical for understanding the problem at hand. As such, an understanding of spatial data and how to work with it is a valuable skill for any data scientist to have. Even better, Python provides a rich toolset for working in this domain, and recent advances have greatly simplified and consolidated these.
In this tutorial we will take a deep dive into geospatial analysis in Python, using tools like geopandas, shapely, and pysal to analyze a dataset, provided by Kaggle (and originally from Inside AirBnB), of sample AirBnB locations in Boston, Massachusetts.
This tutorial is targeted at folks who know a thing or two about data but haven't used Python's geospatial data tools just yet. As such, it assumes a high level of familiarity with pandas. Some familiarity with scikit-learn, statsmodels, matplotlib, and seaborn is also helpful.
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
pd.set_option("max_columns", None)
Plotting Points¶
First, let's boot up and examine our data. Since our data comes in a simple CSV file, we load it into a pandas DataFrame.
listings = pd.read_csv("../input/listings.csv")
listings.head()
| id | listing_url | scrape_id | last_scraped | name | summary | space | description | experiences_offered | neighborhood_overview | notes | transit | access | interaction | house_rules | thumbnail_url | medium_url | picture_url | xl_picture_url | host_id | host_url | host_name | host_since | host_location | host_about | host_response_time | host_response_rate | host_acceptance_rate | host_is_superhost | host_thumbnail_url | host_picture_url | host_neighbourhood | host_listings_count | host_total_listings_count | host_verifications | host_has_profile_pic | host_identity_verified | street | neighbourhood | neighbourhood_cleansed | neighbourhood_group_cleansed | city | state | zipcode | market | smart_location | country_code | country | latitude | longitude | is_location_exact | property_type | room_type | accommodates | bathrooms | bedrooms | beds | bed_type | amenities | square_feet | price | weekly_price | monthly_price | security_deposit | cleaning_fee | guests_included | extra_people | minimum_nights | maximum_nights | calendar_updated | has_availability | availability_30 | availability_60 | availability_90 | availability_365 | calendar_last_scraped | number_of_reviews | first_review | last_review | review_scores_rating | review_scores_accuracy | review_scores_cleanliness | review_scores_checkin | review_scores_communication | review_scores_location | review_scores_value | requires_license | license | jurisdiction_names | instant_bookable | cancellation_policy | require_guest_profile_picture | require_guest_phone_verification | calculated_host_listings_count | reviews_per_month | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 12147973 | https://www.airbnb.com/rooms/12147973 | 20160906204935 | 2016-09-07 | Sunny Bungalow in the City | Cozy, sunny, family home. Master bedroom high... | The house has an open and cozy feel at the sam... | Cozy, sunny, family home. Master bedroom high... | none | Roslindale is quiet, convenient and friendly. ... | NaN | The bus stop is 2 blocks away, and frequent. B... | You will have access to 2 bedrooms, a living r... | NaN | Clean up and treat the home the way you'd like... | https://a2.muscache.com/im/pictures/c0842db1-e... | https://a2.muscache.com/im/pictures/c0842db1-e... | https://a2.muscache.com/im/pictures/c0842db1-e... | https://a2.muscache.com/im/pictures/c0842db1-e... | 31303940 | https://www.airbnb.com/users/show/31303940 | Virginia | 2015-04-15 | Boston, Massachusetts, United States | We are country and city connecting in our deck... | NaN | NaN | NaN | f | https://a2.muscache.com/im/pictures/5936fef0-b... | https://a2.muscache.com/im/pictures/5936fef0-b... | Roslindale | 1 | 1 | ['email', 'phone', 'facebook', 'reviews'] | t | f | Birch Street, Boston, MA 02131, United States | Roslindale | Roslindale | NaN | Boston | MA | 02131 | Boston | Boston, MA | US | United States | 42.282619 | -71.133068 | t | House | Entire home/apt | 4 | 1.5 | 2.0 | 3.0 | Real Bed | {TV,"Wireless Internet",Kitchen,"Free Parking ... | NaN | $250.00 | NaN | NaN | NaN | $35.00 | 1 | $0.00 | 2 | 1125 | 2 weeks ago | NaN | 0 | 0 | 0 | 0 | 2016-09-06 | 0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | f | NaN | NaN | f | moderate | f | f | 1 | NaN |
| 1 | 3075044 | https://www.airbnb.com/rooms/3075044 | 20160906204935 | 2016-09-07 | Charming room in pet friendly apt | Charming and quiet room in a second floor 1910... | Small but cozy and quite room with a full size... | Charming and quiet room in a second floor 1910... | none | The room is in Roslindale, a diverse and prima... | If you don't have a US cell phone, you can tex... | Plenty of safe street parking. Bus stops a few... | Apt has one more bedroom (which I use) and lar... | If I am at home, I am likely working in my hom... | Pet friendly but please confirm with me if the... | https://a1.muscache.com/im/pictures/39327812/d... | https://a1.muscache.com/im/pictures/39327812/d... | https://a1.muscache.com/im/pictures/39327812/d... | https://a1.muscache.com/im/pictures/39327812/d... | 2572247 | https://www.airbnb.com/users/show/2572247 | Andrea | 2012-06-07 | Boston, Massachusetts, United States | I live in Boston and I like to travel and have... | within an hour | 100% | 100% | f | https://a2.muscache.com/im/users/2572247/profi... | https://a2.muscache.com/im/users/2572247/profi... | Roslindale | 1 | 1 | ['email', 'phone', 'facebook', 'linkedin', 'am... | t | t | Pinehurst Street, Boston, MA 02131, United States | Roslindale | Roslindale | NaN | Boston | MA | 02131 | Boston | Boston, MA | US | United States | 42.286241 | -71.134374 | t | Apartment | Private room | 2 | 1.0 | 1.0 | 1.0 | Real Bed | {TV,Internet,"Wireless Internet","Air Conditio... | NaN | $65.00 | $400.00 | NaN | $95.00 | $10.00 | 0 | $0.00 | 2 | 15 | a week ago | NaN | 26 | 54 | 84 | 359 | 2016-09-06 | 36 | 2014-06-01 | 2016-08-13 | 94.0 | 10.0 | 9.0 | 10.0 | 10.0 | 9.0 | 9.0 | f | NaN | NaN | t | moderate | f | f | 1 | 1.30 |
| 2 | 6976 | https://www.airbnb.com/rooms/6976 | 20160906204935 | 2016-09-07 | Mexican Folk Art Haven in Boston | Come stay with a friendly, middle-aged guy in ... | Come stay with a friendly, middle-aged guy in ... | Come stay with a friendly, middle-aged guy in ... | none | The LOCATION: Roslindale is a safe and diverse... | I am in a scenic part of Boston with a couple ... | PUBLIC TRANSPORTATION: From the house, quick p... | I am living in the apartment during your stay,... | ABOUT ME: I'm a laid-back, friendly, unmarried... | I encourage you to use my kitchen, cooking and... | https://a2.muscache.com/im/pictures/6ae8335d-9... | https://a2.muscache.com/im/pictures/6ae8335d-9... | https://a2.muscache.com/im/pictures/6ae8335d-9... | https://a2.muscache.com/im/pictures/6ae8335d-9... | 16701 | https://www.airbnb.com/users/show/16701 | Phil | 2009-05-11 | Boston, Massachusetts, United States | I am a middle-aged, single male with a wide ra... | within a few hours | 100% | 88% | t | https://a2.muscache.com/im/users/16701/profile... | https://a2.muscache.com/im/users/16701/profile... | Roslindale | 1 | 1 | ['email', 'phone', 'reviews', 'jumio'] | t | t | Ardale St., Boston, MA 02131, United States | Roslindale | Roslindale | NaN | Boston | MA | 02131 | Boston | Boston, MA | US | United States | 42.292438 | -71.135765 | t | Apartment | Private room | 2 | 1.0 | 1.0 | 1.0 | Real Bed | {TV,"Cable TV","Wireless Internet","Air Condit... | NaN | $65.00 | $395.00 | $1,350.00 | NaN | NaN | 1 | $20.00 | 3 | 45 | 5 days ago | NaN | 19 | 46 | 61 | 319 | 2016-09-06 | 41 | 2009-07-19 | 2016-08-05 | 98.0 | 10.0 | 9.0 | 10.0 | 10.0 | 9.0 | 10.0 | f | NaN | NaN | f | moderate | t | f | 1 | 0.47 |
| 3 | 1436513 | https://www.airbnb.com/rooms/1436513 | 20160906204935 | 2016-09-07 | Spacious Sunny Bedroom Suite in Historic Home | Come experience the comforts of home away from... | Most places you find in Boston are small howev... | Come experience the comforts of home away from... | none | Roslindale is a lovely little neighborhood loc... | Please be mindful of the property as it is old... | There are buses that stop right in front of th... | The basement has a washer dryer and gym area. ... | We do live in the house therefore might be som... | - The bathroom and house are shared so please ... | https://a2.muscache.com/im/pictures/39764190-1... | https://a2.muscache.com/im/pictures/39764190-1... | https://a2.muscache.com/im/pictures/39764190-1... | https://a2.muscache.com/im/pictures/39764190-1... | 6031442 | https://www.airbnb.com/users/show/6031442 | Meghna | 2013-04-21 | Boston, Massachusetts, United States | My husband and I live on the property. He’s a... | within a few hours | 100% | 50% | f | https://a2.muscache.com/im/pictures/5d430cde-7... | https://a2.muscache.com/im/pictures/5d430cde-7... | NaN | 1 | 1 | ['email', 'phone', 'reviews'] | t | f | Boston, MA, United States | NaN | Roslindale | NaN | Boston | MA | NaN | Boston | Boston, MA | US | United States | 42.281106 | -71.121021 | f | House | Private room | 4 | 1.0 | 1.0 | 2.0 | Real Bed | {TV,Internet,"Wireless Internet","Air Conditio... | NaN | $75.00 | NaN | NaN | $100.00 | $50.00 | 2 | $25.00 | 1 | 1125 | a week ago | NaN | 6 | 16 | 26 | 98 | 2016-09-06 | 1 | 2016-08-28 | 2016-08-28 | 100.0 | 10.0 | 10.0 | 10.0 | 10.0 | 10.0 | 10.0 | f | NaN | NaN | f | moderate | f | f | 1 | 1.00 |
| 4 | 7651065 | https://www.airbnb.com/rooms/7651065 | 20160906204935 | 2016-09-07 | Come Home to Boston | My comfy, clean and relaxing home is one block... | Clean, attractive, private room, one block fro... | My comfy, clean and relaxing home is one block... | none | I love the proximity to downtown, the neighbor... | I have one roommate who lives on the lower lev... | From Logan Airport and South Station you have... | You will have access to the front and side por... | I love my city and really enjoy sharing it wit... | Please no smoking in the house, porch or on th... | https://a1.muscache.com/im/pictures/97154760/8... | https://a1.muscache.com/im/pictures/97154760/8... | https://a1.muscache.com/im/pictures/97154760/8... | https://a1.muscache.com/im/pictures/97154760/8... | 15396970 | https://www.airbnb.com/users/show/15396970 | Linda | 2014-05-11 | Boston, Massachusetts, United States | I work full time for a public school district.... | within an hour | 100% | 100% | t | https://a0.muscache.com/im/users/15396970/prof... | https://a0.muscache.com/im/users/15396970/prof... | Roslindale | 1 | 1 | ['email', 'phone', 'reviews', 'kba'] | t | t | Durnell Avenue, Boston, MA 02131, United States | Roslindale | Roslindale | NaN | Boston | MA | 02131 | Boston | Boston, MA | US | United States | 42.284512 | -71.136258 | t | House | Private room | 2 | 1.5 | 1.0 | 2.0 | Real Bed | {Internet,"Wireless Internet","Air Conditionin... | NaN | $79.00 | NaN | NaN | NaN | $15.00 | 1 | $0.00 | 2 | 31 | 2 weeks ago | NaN | 13 | 34 | 59 | 334 | 2016-09-06 | 29 | 2015-08-18 | 2016-09-01 | 99.0 | 10.0 | 10.0 | 10.0 | 10.0 | 9.0 | 10.0 | f | NaN | NaN | f | flexible | f | f | 1 | 2.25 |
len(listings)
3585
At the moment our listings have several geospatial variables, the most important of which are longitude and latitude, which give the exact coordinates of the BnB in question. This means that it's easy for us to, say, plot every BnB location on a map:
plt.scatter(listings['longitude'], listings['latitude'])
<matplotlib.collections.PathCollection at 0x34240470>
Chances are you've already done this before, and it's a perfectly adequate way to get started working with locations. In this plot we see...not much, really. If you're very intimately familiar with the layout of the city of Boston, you will probably be able to make sense of some of these clusters which are, to me, not being from the city, totally mysterious.
In other words, this plot is missing something important: geospatial context.
Additionally, this display is unprojected—it's displayed in terms of raw coordinates. The amount of distance contained in a coordinate degree varies greatly depending on where you are, so this naive plot potentially pretty badly distorts distances.
We'll come back to the projection issue later; for now, there's an easy fix for both these problems.
Enter mplleaflet. mplleaflet is a tool that automatically takes a coordinate matplotlib plot of any kind and places it on top of a leaflet slippy map. The best part is that it's just one additional line of code. Just throw mplleaflet.display() after generating your plot to drop it inline in your Jupyter notebook:
import mplleaflet
sample = listings.sample(1000)
plt.scatter(sample['longitude'], sample['latitude'])
mplleaflet.display()