Sam Maurer // DCRP PhD Student // maurer@berkeley.edu // Oct. 7, 2015 // CP 255 demo
This notebook provides a demonstration of data-access APIs that operate over the web.
In Part 1, we'll load and parse data from an automated USGS feed of earthquakes. In Part 2, we'll add query parameters to the workflow, using the Google Maps Geolocation API as an example. In Part 3, we'll use authenticated APIs to access (public) Twitter data.
From the command line, install this Python package for connecting to Twitter APIs:
pip install TwitterAPI
There are several different package managers for Python. We're using "pip install" instead of "conda install" because Anaconda doesn't include this package in its index. The lab computers should already have TwitterAPI installed.
Make sure keys.py
is in the same directory as this notebook
keys.py
(provided separately) contains demo authentication keys for the Twitter APIs. For folks not taking the class, you can edit the keys-example.py
file, following the instructions in the next step to obtain your own credentials.
OPTIONAL: Sign up for your own Twitter API credentials (5 minutes)
Twitter limits the number of simultaneous connections from a single account, so if you're willing to sign up for your own credentials, it will help the in-class demo go smoother!
Log into Twitter or create an account: http://twitter.com
Register a new app development project: https://dev.twitter.com/apps/new
(The form is geared toward people making smartphone or web apps, but you still have to fill it out... You can call the app an in-class demo and give the URL of your own Twitter page, for example)
Submit the form, go to the "Keys and Access Tokens" tab, and click on "Create my access token" at the bottom of the page
Copy these four codes into the keys.py
file, replacing the demo credentials:
(a) consumer key, (b) consumer secret, (c) access token, (d) access token secret
All done! Try running the first few code blocks in Part 3 to see if it worked
This is an API for near-real-time data on earthquakes. Results are provided in JSON format over the web. No authentication is needed, and rather than accepting queries, the API has a separate endpoint for each permutation of the data that users might want.
API documentation:
http://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php
Sample API endpoint, for magnitude 4.5+ earthquakes in past day:
http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_day.geojson
%matplotlib inline
import pandas as pd
import urllib
import json
# use endpoint for magnitude 2.5+ quakes in past week
endpoint_url = "http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.geojson"
# open a connection to the URL
connection = urllib.urlopen(endpoint_url)
# download the results
results = connection.read()
print results[:500] # first 500 characters
print type(results)
{"type":"FeatureCollection","metadata":{"generated":1444158814000,"url":"http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/2.5_week.geojson","title":"USGS Magnitude 2.5+ Earthquakes, Past Week","status":200,"api":"1.0.17","count":255},"features":[{"type":"Feature","properties":{"mag":2.96,"place":"1km N of The Geysers, California","time":1444157633260,"updated":1444158221836,"tz":-420,"url":"http://earthquake.usgs.gov/earthquakes/eventpage/nc72536056","detail":"http://earthquake.usgs.gov/ <type 'str'>
# the results are a string with JSON-formatted data inside
# parse the string into a Python data structure
data = json.loads(results)
print data['features'][0] # first item from the array called 'features'
print type(data)
{u'geometry': {u'type': u'Point', u'coordinates': [-122.7580032, 38.7903328, 3.56]}, u'type': u'Feature', u'properties': {u'rms': 0.06, u'code': u'72536056', u'cdi': 2, u'sources': u',nc,', u'nst': 38, u'tz': -420, u'title': u'M 3.0 - 1km N of The Geysers, California', u'magType': u'md', u'detail': u'http://earthquake.usgs.gov/earthquakes/feed/v1.0/detail/nc72536056.geojson', u'sig': 135, u'net': u'nc', u'type': u'earthquake', u'status': u'automatic', u'updated': 1444158221836L, u'felt': 3, u'alert': None, u'dmin': 0.03405, u'mag': 2.96, u'gap': 63, u'types': u',dyfi,focal-mechanism,general-link,geoserve,nearby-cities,origin,phase-data,scitech-link,', u'url': u'http://earthquake.usgs.gov/earthquakes/eventpage/nc72536056', u'ids': u',nc72536056,', u'tsunami': 0, u'place': u'1km N of The Geysers, California', u'time': 1444157633260L, u'mmi': None}, u'id': u'nc72536056'} <type 'dict'>
# pull out the event descriptions
for quake in data['features']:
print quake['properties']['title']
M 3.0 - 1km N of The Geysers, California M 4.6 - 43km SE of Sary-Tash, Kyrgyzstan M 2.8 - 14km S of Volcano, Hawaii M 2.6 - 10km SE of Chester, California M 4.2 - 75km SW of Coquimbo, Chile M 3.5 - 6km WSW of Perry, Oklahoma M 2.7 - 46km W of Hawi, Hawaii M 5.1 - 39km WNW of Illapel, Chile M 2.7 - 3km SSE of Pole Ojea, Puerto Rico M 4.8 - 58km SE of Ofunato, Japan M 2.7 - 34km SSE of Redoubt Volcano, Alaska M 5.1 - 154km SSW of Puerto El Triunfo, El Salvador M 4.9 - 65km N of Sikabaluan, Indonesia M 2.8 - 32km SE of Tok, Alaska M 3.2 - 26km E of Honaunau-Napoopoo, Hawaii M 4.5 - 122km ENE of Ndoi Island, Fiji M 4.5 - 18km N of Taltal, Chile M 2.9 - 66km NNW of Charlotte Amalie, U.S. Virgin Islands M 4.7 - 89km WSW of Coquimbo, Chile M 2.9 - 44km NE of Road Town, British Virgin Islands M 2.6 - 8km WNW of Anza, California M 2.7 - 9km NNW of Shelbyville, Kentucky M 4.6 - 84km W of Illapel, Chile M 4.2 - 70km NW of Rapar, India M 3.9 - 135km NE of Chignik Lake, Alaska M 2.7 - 54km ESE of Lovelock, Nevada M 4.1 - 69km WNW of La Ligua, Chile M 4.2 - 71km WNW of La Ligua, Chile M 2.5 - 68km ESE of Lakeview, Oregon M 2.7 - 68km ESE of Lakeview, Oregon M 2.6 - 8km SE of Pole Ojea, Puerto Rico M 2.6 - 23km N of Upper Lake, California M 2.8 - 23km W of Perry, Oklahoma M 5.7 - 74km N of Amatignak Island, Alaska M 4.7 - 158km WSW of Ovalle, Chile M 5.9 - 43km SSW of Coquimbo, Chile M 2.9 - 69km ESE of Lakeview, Oregon M 2.6 - 42km SSW of Larsen Bay, Alaska M 3.0 - 95km WSW of Healy, Alaska M 3.4 - 85km N of Road Town, British Virgin Islands M 2.5 - 54km E of Old Iliamna, Alaska M 3.0 - 94km WSW of Healy, Alaska M 2.9 - 83km W of Cantwell, Alaska M 4.5 - 32km N of Ramechhap, Nepal M 3.5 - 20km E of Cherokee, Oklahoma M 2.9 - 81km NNW of Talkeetna, Alaska M 4.7 - 200km SE of L'Esperance Rock, New Zealand M 4.1 - 57km WSW of Illapel, Chile M 4.2 - 47km WSW of Amatignak Island, Alaska M 3.0 - 48km WSW of Stella, Puerto Rico M 5.0 - Off the east coast of the North Island of New Zealand M 4.3 - 102km ESE of Iwaki, Japan M 4.6 - 43km N of Palue, Indonesia M 3.8 - 26km SSE of Boca de Yuma, Dominican Republic M 2.8 - 67km SW of Anchor Point, Alaska M 4.7 - 124km SSW of Kotaagung, Indonesia M 4.4 - 20km SE of Jarm, Afghanistan M 3.0 - 30km SE of Redoubt Volcano, Alaska M 2.6 - 9km SW of Honoka'a, Hawaii M 2.8 - 7km NNE of Helena, Oklahoma M 4.8 - South of the Fiji Islands M 4.9 - 81km NNE of Obelobel, Indonesia M 2.6 - 9km W of Templeton, California M 4.9 - 93km W of Ovalle, Chile M 4.6 - South of the Fiji Islands M 4.8 - 146km S of Bambanglipuro, Indonesia M 3.3 - 24km NE of Fritz Creek, Alaska M 2.9 - 12km SSW of Alberto Oviedo Mota, B.C., MX M 4.8 - 91km ENE of Hihifo, Tonga M 5.4 - 33km SSW of Lluta, Peru M 3.0 - 70km ESE of Lakeview, Oregon M 4.4 - 58km NW of San Antonio de los Cobres, Argentina M 3.3 - 71km ESE of Lakeview, Oregon M 4.7 - 100km NE of Chichi-shima, Japan M 3.1 - 90km NNW of Chirikof Island, Alaska M 5.2 - 63km W of Ovalle, Chile M 4.6 - 64km WSW of Ovalle, Chile M 4.7 - 205km ESE of Antonio Enes, Mozambique M 3.0 - 90km NNW of Road Town, British Virgin Islands M 2.8 - 100km W of Willow, Alaska M 4.6 - South of the Fiji Islands M 2.6 - 71km N of Tierras Nuevas Poniente, Puerto Rico M 2.6 - 6km N of Edmond, Oklahoma M 2.9 - 45km W of Rincon, Puerto Rico M 4.7 - 20km NNE of Taniwel, Indonesia M 2.8 - 73km N of Tierras Nuevas Poniente, Puerto Rico M 3.3 - 8km NNE of Helena Valley Northwest, Montana M 5.3 - 85km WSW of Coquimbo, Chile M 2.9 - 8km NNE of Helena Valley Northwest, Montana M 4.5 - 207km WSW of Abepura, Indonesia M 2.6 - 50km NE of Road Town, British Virgin Islands M 2.6 - 14km SSE of Yosemite Valley, California M 2.7 - 21km NW of Ludlow, California M 3.9 - 172km W of Illapel, Chile M 4.1 - 30km SE of Al Marj, Libya M 2.8 - 6km N of Waikoloa, Hawaii M 2.7 - 19km SSE of Pahala, Hawaii M 2.7 - 6km N of Waikoloa, Hawaii M 4.6 - 86km SW of Canete, Chile M 4.3 - 53km N of La Serena, Chile M 4.4 - 62km ESE of Ishinomaki, Japan M 4.2 - 28km SW of Ovalle, Chile M 4.0 - 49km W of Ovalle, Chile M 2.9 - 7km S of Anthony, Kansas M 2.6 - 27km ENE of Road Town, British Virgin Islands M 2.6 - 19km SSE of Ridgemark, California M 4.9 - 29km SE of Nago, Japan M 2.5 - 49km NE of Road Town, British Virgin Islands M 2.5 - 27km SSW of Healy, Alaska M 4.5 - 43km WNW of Coquimbo, Chile M 4.9 - 198km NW of Tanahmerah, Indonesia M 2.6 - 7km NNE of Helena Valley Northwest, Montana M 2.8 - 64km SSW of Redoubt Volcano, Alaska M 2.9 - 6km ESE of Perry, Oklahoma M 4.9 - 43km WNW of Coquimbo, Chile M 4.9 - Indian Ocean Triple Junction M 4.7 - 126km W of Illapel, Chile M 5.4 - Indian Ocean Triple Junction M 2.8 - 45km NNE of Road Town, British Virgin Islands M 4.9 - 99km SSW of Biha, Indonesia M 5.0 - Northern Mid-Atlantic Ridge M 4.6 - 76km SW of Puerto Madero, Mexico M 5.8 - 45km SSW of Coquimbo, Chile M 5.6 - 17km N of Lebu, Chile M 2.5 - 30km NW of Rincon, Puerto Rico M 2.6 - 10km NE of Helena, Oklahoma M 2.7 - 11km SE of Esperanza, Puerto Rico M 4.3 - 22km NE of Agar, China M 4.4 - 111km E of Gualaquiza, Ecuador M 3.0 - 21km NW of Hawthorne, Nevada M 3.3 - 10km NW of Caldwell, Kansas M 4.4 - 59km WNW of Valparaiso, Chile M 5.1 - 188km SSE of L'Esperance Rock, New Zealand M 4.6 - 165km ENE of Olonkinbyen, Svalbard and Jan Mayen M 2.6 - 35km W of Kenai, Alaska M 4.4 - 183km ENE of Olonkinbyen, Svalbard and Jan Mayen M 4.3 - 61km SSE of Vanimo, Papua New Guinea M 4.3 - 38km SW of San Antonio, Chile M 3.0 - 3km SSE of Perry, Oklahoma M 2.5 - 125km WNW of Talkeetna, Alaska M 4.0 - 55km SSW of Ovalle, Chile M 3.1 - 5km SSE of Circle Hot Springs Station, Alaska M 4.5 - 100km W of Ambunti, Papua New Guinea M 3.3 - 144km NNE of Road Town, British Virgin Islands M 4.2 - 4km SSW of Izumi, Japan M 3.4 - 102km N of San Juan, Puerto Rico M 3.3 - 92km N of San Juan, Puerto Rico M 2.9 - 13km E of Pahala, Hawaii M 2.5 - 14km E of Pahala, Hawaii M 2.7 - 124km NW of Talkeetna, Alaska M 5.2 - 162km WSW of Ovalle, Chile M 5.1 - 73km W of Ovalle, Chile M 4.5 - 37km WSW of Ashkasham, Afghanistan M 5.5 - 15km ESE of Tondano, Indonesia M 4.5 - 94km WNW of La Ligua, Chile M 3.4 - 1km WNW of Campanilla, Puerto Rico M 3.2 - 66km WSW of Anchor Point, Alaska M 3.2 - 6km NNW of Stroud, Oklahoma M 2.7 - 9km NW of Ferndale, California M 4.6 - 33km SW of Hakha, Burma M 2.5 - 43km E of Cape Yakataga, Alaska M 4.3 - 25km SSW of El Rosario, El Salvador M 3.1 - 11km SE of Esperanza, Puerto Rico M 2.8 - 62km N of San Juan, Puerto Rico M 2.8 - 11km SE of Esperanza, Puerto Rico M 4.6 - 14km E of Coihueco, Chile M 2.9 - 9km SE of Esperanza, Puerto Rico M 2.8 - 9km SE of Esperanza, Puerto Rico M 2.9 - 7km ESE of Esperanza, Puerto Rico M 2.5 - 47km NNE of Vieques, Puerto Rico M 2.9 - 0km SW of Pawnee, Oklahoma M 2.9 - 33km ESE of Vieques, Puerto Rico M 2.8 - 34km SSE of Culebra, Puerto Rico M 2.6 - 0km ESE of Pawnee, Oklahoma M 3.8 - 13km ENE of Helena, Oklahoma M 2.7 - 18km E of Anchorage, Alaska M 4.2 - 18km SE of Esperanza, Puerto Rico M 4.6 - 81km N of Dobo, Indonesia M 5.4 - 31km S of Hihifo, Tonga M 4.6 - 12km S of Antsahe, Comoros M 2.7 - 0km E of Irving, Texas M 3.0 - 71km ESE of Lakeview, Oregon M 4.9 - 40km NW of Lluta, Peru M 4.5 - 60km ENE of Petite Riviere Salee, Martinique M 4.2 - 9km ENE of Quepos, Costa Rica M 2.5 - 27km W of Medford, Oklahoma M 4.1 - 50km WNW of Valparaiso, Chile M 2.8 - 12km SE of Alva, Oklahoma M 4.4 - 4km NE of Ishikawa, Japan M 2.9 - 39km NE of Luquillo, Puerto Rico M 3.1 - 55km NNE of Road Town, British Virgin Islands M 2.6 - 46km S of Redoubt Volcano, Alaska M 2.5 - 89km NW of Talkeetna, Alaska M 3.2 - 5km SE of Perry, Oklahoma M 2.7 - 6km WSW of Alberto Oviedo Mota, B.C., MX M 3.1 - 54km ESE of Road Town, British Virgin Islands M 2.7 - 61km NNE of Coldfoot, Alaska M 3.7 - 6km SE of Perry, Oklahoma M 2.9 - 69km ESE of Lakeview, Oregon M 2.7 - 21km ENE of Goldfield, Nevada M 4.4 - 103km WSW of Illapel, Chile M 5.6 - 139km SSW of Biha, Indonesia M 4.6 - 71km WSW of Coquimbo, Chile M 2.8 - 1km N of Fuig, Puerto Rico M 4.7 - 142km NE of San Pedro de Atacama, Chile M 4.5 - 116km W of Hihifo, Tonga M 5.3 - 86km SSW of Biha, Indonesia M 4.2 - 87km WNW of Ovalle, Chile M 4.4 - 35km WSW of Ovalle, Chile M 4.2 - 49km SSE of Jahrom, Iran M 3.3 - 98km S of Atka, Alaska M 4.4 - 94km W of Illapel, Chile M 2.9 - 24km WNW of Fairview, Oklahoma M 2.5 - 5km SE of Perry, Oklahoma M 4.2 - 50km W of Ovalle, Chile M 5.5 - 36km ESE of Tres Palos, Mexico M 2.6 - 4km NE of Langston, Oklahoma M 5.7 - 60km NNW of Visokoi Island, South Georgia and the South Sandwich Islands M 4.5 - 260km SE of Lambasa, Fiji M 2.8 - 4km NE of Langston, Oklahoma M 2.9 - 20km W of Perry, Oklahoma M 3.8 - 54km ESE of Boca de Yuma, Dominican Republic M 4.5 - Northern East Pacific Rise M 2.5 - 74km E of Chernabura Island, Alaska M 3.5 - 77km E of Chernabura Island, Alaska M 2.6 - 11km S of Alva, Oklahoma M 2.5 - 91km W of Ferndale, California M 3.3 - 120km N of Brenas, Puerto Rico M 2.8 - 36km SSE of Amukta Island, Alaska M 4.7 - 275km SE of Hachijo-jima, Japan M 2.6 - 16km W of Wellington, Kansas M 3.0 - 6km SSE of Cherokee, Oklahoma M 2.6 - 9km SSW of Amukta Island, Alaska M 3.0 - 75km SSW of Unalaska, Alaska M 2.5 - 23km E of Cherokee, Oklahoma M 2.8 - 89km E of Akutan, Alaska M 2.7 - 25km E of Cherokee, Oklahoma M 5.0 - 150km SW of Panguna, Papua New Guinea M 4.8 - 49km W of Ovalle, Chile M 4.8 - 227km SE of L'Esperance Rock, New Zealand M 3.8 - 49km W of Ovalle, Chile M 4.1 - 135km S of Atka, Alaska M 2.5 - 54km N of Amatignak Island, Alaska M 2.7 - 10km SW of Charleston, Missouri M 4.7 - 58km WNW of Illapel, Chile M 3.1 - 7km ENE of Greater Sudbury, Canada M 4.4 - 85km W of Kashi, China M 2.6 - 6km SE of Perry, Oklahoma M 5.4 - 42km W of Abepura, Indonesia M 3.3 - 10km NNE of Helena, Oklahoma M 2.5 - 10km NNE of Helena, Oklahoma M 2.7 - 18km NE of Fairview, Oklahoma M 2.6 - 17km SW of Amukta Island, Alaska M 5.2 - 181km SSW of Lorengau, Papua New Guinea M 4.2 - 5km WNW of Hualian, Taiwan
# pull out magnitudes and depths into a pandas dataframe
# first, set up a dictionary of empty arrays
d = {'magnitude': [], 'depth': []}
# loop through the earthquakes and pull out datapoints
for quake in data['features']:
d['magnitude'].append(quake['properties']['mag'])
d['depth'].append(quake['geometry']['coordinates'][2])
# then load it all into a dataframe
df = pd.DataFrame.from_dict(d)
print len(df)
255
print df.head()
depth magnitude 0 3.56 2.96 1 25.46 4.60 2 0.73 2.82 3 2.85 2.62 4 9.02 4.20
print df.describe()
depth magnitude count 255.000000 255.000000 mean 45.110569 3.671843 std 81.903339 0.973804 min 0.000000 2.500000 25% 9.010000 2.800000 50% 19.000000 3.300000 75% 46.015000 4.550000 max 551.360000 5.900000
# plot the depth vs. magnitude
df.plot(x='magnitude', y='depth', kind='scatter')
<matplotlib.axes._subplots.AxesSubplot at 0xbc04080>
# save dataframe to disk
df.to_csv('usgs_earthquake_data.csv')
print 'file saved'
file saved
# read it back later
new_df = pd.DataFrame.from_csv('usgs_earthquake_data.csv')
print new_df.head()
depth magnitude 0 3.56 2.96 1 25.46 4.60 2 0.73 2.82 3 2.85 2.62 4 9.02 4.20
Google Maps has several APIs for getting search results programmatically. This one looks up latitude-longidtude coordinates (and other place information) for street addresses, which is called geocoding.
It works similarly to the earthquakes example, with query parameters added to the URL endpoint.
API documentation:
https://developers.google.com/maps/documentation/geocoding/intro
API endpoint:
https://maps.googleapis.com/maps/api/geocode/json
API endpoint with query parameters:
https://maps.googleapis.com/maps/api/geocode/json?address=Wurster+Hall
# we have to encode the search query so that it can be passed as a URL,
# with spaces and other special characters removed
endpoint = 'https://maps.googleapis.com/maps/api/geocode/json'
params = {
'address': 'Wurster Hall, Berkeley, CA',
}
url = endpoint + '?' + urllib.urlencode(params)
print url
https://maps.googleapis.com/maps/api/geocode/json?address=Wurster+Hall%2C+Berkeley%2C+CA
# open a connection to the URL
connection = urllib.urlopen(url)
# download and parse the results
results = json.loads(connection.read())
print results
{u'status': u'OK', u'results': [{u'geometry': {u'location_type': u'ROOFTOP', u'bounds': {u'northeast': {u'lat': 37.871116, u'lng': -122.2541912}, u'southwest': {u'lat': 37.8700084, u'lng': -122.2552699}}, u'viewport': {u'northeast': {u'lat': 37.8719111802915, u'lng': -122.2533815697085}, u'southwest': {u'lat': 37.86921321970851, u'lng': -122.2560795302915}}, u'location': {u'lat': 37.8707352, u'lng': -122.2548935}}, u'address_components': [{u'long_name': u'Wurster Hall', u'types': [u'premise'], u'short_name': u'Wurster Hall'}, {u'long_name': u'Berkeley', u'types': [u'locality', u'political'], u'short_name': u'Berkeley'}, {u'long_name': u'Alameda County', u'types': [u'administrative_area_level_2', u'political'], u'short_name': u'Alameda County'}, {u'long_name': u'California', u'types': [u'administrative_area_level_1', u'political'], u'short_name': u'CA'}, {u'long_name': u'United States', u'types': [u'country', u'political'], u'short_name': u'US'}, {u'long_name': u'94720', u'types': [u'postal_code'], u'short_name': u'94720'}], u'place_id': u'ChIJnzKeUCV8hYARccYjWTGEJUg', u'formatted_address': u'Wurster Hall, Berkeley, CA 94720, USA', u'types': [u'premise']}]}
# pull out the formatted addresses
for item in results['results']:
print item['formatted_address']
Wurster Hall, Berkeley, CA 94720, USA
Twitter's APIs also operate over the web, but they require a back-and-forth authentication process at the beginning of a connection. It's easier to have a Python library handle this than to create the query URLs ourselves.
The REST APIs perform stand-alone operations: we submit a query and receive results, like in earlier examples. The Streaming API continues sending results in real time until we disconnect.
(REST is a set of principles describing how data transactions should work over the web, while the actual communication protocol is called HTTP. Web pages work through HTTP and REST too, but the browser steps in to interpret and display the content for you.)
API documentation:
https://dev.twitter.com/rest/public
https://dev.twitter.com/streaming/overview
Documentation for third-party Python "wrapper":
https://github.com/geduldig/TwitterAPI
from TwitterAPI import TwitterAPI
# import API credentials from keys.py file in the
# same directory as this notebook
from keys import *
# set up an API connection using credentials from the keys file
api = TwitterAPI(consumer_key, consumer_secret,
access_token, access_token_secret)
print "Connection is set up but not tested"
Connection is set up but not tested
# Most recent tweet from @GBoeing's timeline
endpoint = 'statuses/user_timeline'
params = {
'screen_name': 'gboeing',
'count': 1
}
r = api.request(endpoint, params)
for tweet in r.get_iterator():
print tweet['text']
Can the Bay Area house everyone who wants to live there without altering its character? Yes. https://t.co/jalpboUz9y http://t.co/rtQz8leoEC
# What other data is there?
print tweet.keys()
[u'contributors', u'truncated', u'text', u'is_quote_status', u'in_reply_to_status_id', u'id', u'favorite_count', u'source', u'retweeted', u'coordinates', u'entities', u'in_reply_to_screen_name', u'id_str', u'retweet_count', u'in_reply_to_user_id', u'favorited', u'user', u'geo', u'in_reply_to_user_id_str', u'possibly_sensitive', u'lang', u'created_at', u'in_reply_to_status_id_str', u'place', u'extended_entities']
# Contents of some additional fields...
# Here are the definitions: https://dev.twitter.com/overview/api/tweets
for tweet in r.get_iterator():
print "Tweet // ", tweet['text']
print "Timestamp // ", tweet['created_at']
print "Retweets // ", tweet['retweet_count']
print "Favorites // ", tweet['favorite_count']
print "Geotag // ", tweet['coordinates']
print "Language // ", tweet['lang']
print "User // ", tweet['user']['screen_name']
print "Followers // ", tweet['user']['followers_count']
print "Profile // ", tweet['user']['description']
Tweet // Can the Bay Area house everyone who wants to live there without altering its character? Yes. https://t.co/jalpboUz9y http://t.co/rtQz8leoEC Timestamp // Tue Oct 06 19:13:33 +0000 2015 Retweets // 0 Favorites // 0 Geotag // None Language // en User // gboeing Followers // 1676 Profile // PhD candidate in urban planning at @UCBerkeley. Cities, complexity, livability, urban design, data.
# Search for public tweets about #muni
endpoint = 'search/tweets'
params = {
'q': '#muni',
'count': 5
}
r = api.request(endpoint, params)
for tweet in r.get_iterator():
print tweet['text'] + '\n'
Dear #Muni: You always wreak of mothballs. I'm afraid to ask what the source may be... RT @sfexaminer: Governor signs bus-camera ticketing bill, #Muni to target double parkers http://t.co/fadMGLCos6 http://t.co/agz3xYFsAy Hey #Muni!! What did you do? Lol lol. The things us suburban gals get amused by. New Jim Colby video on #Fed’s September hike delay and Puerto Rico default: http://t.co/cyjhi9I8qX #muni #bonds http://t.co/SUrgQgR0hz RT @Scott_Wiener: Kicking off new campaign to put #art on #Muni buses. The 5 winning artists did a great job beautifying our #transit. http…
# Search for public tweets in Hindi
endpoint = 'search/tweets'
params = {
'q': '*',
'lang': 'hi',
'count': 5
}
r = api.request(endpoint, params)
for tweet in r.get_iterator():
print tweet['text'] + '\n'
Sheesh RT @sardanarohit: जब घर में हैं आज़म खान, तो क्यों चाहिए पाकिस्तान? https://t.co/JoZrUYGBDs Makkah Mina vipathil, Ilankayar oruvar maranam. गंगा माता बिलख रही थी लाचारी के घाटों पर, क्रूर लाठियां बरस रही थीं चन्दन लगे ललाटों पर, RT @rverma1080: सत्य परेशान हो सकता है लेकिन पराजित नहीं हो सकता।। http://t.co/dXjMyyIIFq
# Search for public tweets geotagged near the UC Berkeley campus
endpoint = 'search/tweets'
params = {
'q': '*',
'geocode': '37.873,-122.260,0.5km',
'count': 5
}
r = api.request(endpoint, params)
for tweet in r.get_iterator():
print tweet['text'] + '\n'
Classes! (@ South Hall - @cal in Berkeley, CA) https://t.co/1EMp7qMgU0 The new Lower Sproul Plaza and Eshleman Hall, by Moore Ruble Yudell Architects--already very popular… https://t.co/hooRcLhRA5 I'm at University of California, Berkeley - @cal in Berkeley, CA https://t.co/rQBd4E0woh Morning #meditation. And laundry. #dormlife #theearlybirdgetsthedryer #GradSchool #GottaGetItAllDone… https://t.co/w5q1Iyieh0 hey is this avant-garde @ UC Berkeley https://t.co/xqGnhy58s4
Here's the search documentation: https://dev.twitter.com/rest/reference/get/search/tweets
# Twitter limits simultaneous connections to the streaming API,
# so this part may not work using the demo API keys during class
endpoint = 'statuses/filter'
params = {'locations': '-180,-90,180,90'}
r = api.request(endpoint, params)
# 'enumerate' lets us count tweets as we receive them
for i, tweet in enumerate(r.get_iterator()):
print tweet['created_at']
print tweet['place']['full_name'] + ', ' + tweet['place']['country']
print tweet['text'] + '\n'
if (i > 20): break
r.close() # close streaming connection
Tue Oct 06 19:16:53 +0000 2015 Manaus, Amazonas, Brasil SHAJSJAKDKSKDK Tue Oct 06 19:16:53 +0000 2015 Yemen, Yemen @M8pnE @motalka_m7roma انا زبي قام وصل Tue Oct 06 19:16:53 +0000 2015 Quinta de Tilcoco, Chile, Chile @sacaloprendio he pecado, el ego es incontrolable: la estadística es sólo en Santiago.. http://t.co/xGrRhmBBcs Me flagelo Tue Oct 06 19:16:54 +0000 2015 Paris, Ile-de-France, France I can't tell you how beautiful this blue is. You just have to go see it. #YvesKlein @ Centre Pompidou https://t.co/imzJavALCg Tue Oct 06 19:16:54 +0000 2015 Italia, Italia Una que me baile Twerk Tue Oct 06 19:16:54 +0000 2015 Salwa, Kuwait, دولة الكويت اللهم اني اسالك ان اتخرج من الثانوي بكامل قواي العقليه . Tue Oct 06 19:16:53 +0000 2015 Norway, Norge @PatrickJD84 @CaribbeanChilly @Grummz and my talents http://t.co/i5WKRSOziR Tue Oct 06 19:16:54 +0000 2015 京都府 京都市 伏見区, 日本 きつねかわいい!!!(2243回目) Tue Oct 06 19:16:53 +0000 2015 Oaxaca de Juárez, Oaxaca, México En Semifinales el #TorneoIntramuros2015 de la @Admon_GobOax http://t.co/Hr93Ghls6x .@GabinoCue .@GobOax #Oaxaca http://t.co/IBdTaJA81j Tue Oct 06 19:16:54 +0000 2015 Poitiers, Poitou-Charentes, France @kfdoh Radiohead tambien esta en la lista. Tue Oct 06 19:16:53 +0000 2015 South East, England, United Kingdom @KTHopkins when didn't the police attend to a burglary? Ive had a couple and the police attended each time .. what did I miss? Tue Oct 06 19:16:54 +0000 2015 La Línea de la Concepción, Andalucía, España Pos na de tranquis cn la bal jaja Tue Oct 06 19:16:54 +0000 2015 Buenos Aires, Argentina, Argentina #EternamenteJuntosViciconte Gonza la 6G Te Espera | ochenta y uno Tue Oct 06 19:16:54 +0000 2015 Santa Monica, CA, United States @DrueMaggetti in studio w me and hrosmakeup 💄in #SantaMonica #dwphotograph #davidwalden.com #fresnel… https://t.co/EsirRHn9Xe Tue Oct 06 19:16:53 +0000 2015 Paisley, Scotland, United Kingdom I can't believe I see one direction live tomorrow Tue Oct 06 19:16:53 +0000 2015 Saen Suk, Chon Buri, ประเทศไทย @nutmeelia http://t.co/IoG3cWmkyy Tue Oct 06 19:16:53 +0000 2015 Viana de Cega, Castilla y León, España No podemos hacer nada por cambiar el rumbo. Tue Oct 06 19:16:54 +0000 2015 New York, NY, United States @NaughtyMabel <3 the tutu, Mabel! Tue Oct 06 19:16:53 +0000 2015 Villarrica, La Araucanía, Chile La Más Granosa xD Jaja Chicas Yo LAS BANCOO ♥ Seguro Que Al Choclo La Miras De Cerca Y Te Regala Granos xD 😂😂😝😝 https://t.co/TOjXqeHsKs Tue Oct 06 19:16:54 +0000 2015 Samarahan, Sarawak, Malaysia Bad molah lawak😒 https://t.co/rpXbV2J7Y1 Tue Oct 06 19:16:53 +0000 2015 الجيزة, مصر, مصر #اللهم ارزقني سجدة في بيت #نبيك احكي فيها عن كل ما في قلبي وينتهي بها كربي وارتاح بها من همي الشيخ د. #ماهر_المعيقلى http://t.co/X1n5WFmbJT Tue Oct 06 19:16:54 +0000 2015 Paddington, London, United Kingdom @Nhash40 @AlkwuaityQ هلا وغلا بنهاش فتى الصغل 😜
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-24-94fca7d55b81> in <module>() 14 if (i > 20): break 15 ---> 16 r.close() # close streaming connection AttributeError: 'TwitterResponse' object has no attribute 'close'
# first, save some tweets to an array instead of just printing them
r = api.request(endpoint, params)
tweets = []
for i, tweet in enumerate(r.get_iterator()):
if (i >= 500): break
tweets.append(tweet)
r.close()
print len(tweets)
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-26-f6b6e014ca6b> in <module>() 8 tweets.append(tweet) 9 ---> 10 r.close() 11 print len(tweets) AttributeError: 'TwitterResponse' object has no attribute 'close'
# the raw data is very messy though!
print tweets[0:5]
[{u'contributors': None, u'truncated': False, u'text': u'@thomasbernardes \xe9 sim', u'is_quote_status': False, u'in_reply_to_status_id': 651476149676212224L, u'id': 651476328290627584L, u'favorite_count': 0, u'source': u'<a href="http://twitter.com" rel="nofollow">Twitter Web Client</a>', u'retweeted': False, u'coordinates': None, u'timestamp_ms': u'1444159035166', u'entities': {u'user_mentions': [{u'id': 39062231, u'indices': [0, 16], u'id_str': u'39062231', u'screen_name': u'thomasbernardes', u'name': u'schaeffer'}], u'symbols': [], u'hashtags': [], u'urls': []}, u'in_reply_to_screen_name': u'thomasbernardes', u'id_str': u'651476328290627584', u'retweet_count': 0, u'in_reply_to_user_id': 39062231, u'favorited': False, u'user': {u'follow_request_sent': None, u'profile_use_background_image': False, u'default_profile_image': False, u'id': 3027528856L, u'verified': False, u'profile_image_url_https': u'https://pbs.twimg.com/profile_images/628396788584710144/E9w5oCAa_normal.jpg', u'profile_sidebar_fill_color': u'000000', u'profile_text_color': u'000000', u'followers_count': 129, u'profile_sidebar_border_color': u'000000', u'id_str': u'3027528856', u'profile_background_color': u'000000', u'listed_count': 4, u'profile_background_image_url_https': u'https://abs.twimg.com/images/themes/theme1/bg.png', u'utc_offset': -25200, u'statuses_count': 11833, u'description': u'semelhantes.', u'friends_count': 164, u'location': None, u'profile_link_color': u'DD2E44', u'profile_image_url': u'http://pbs.twimg.com/profile_images/628396788584710144/E9w5oCAa_normal.jpg', u'following': None, u'geo_enabled': True, u'profile_banner_url': u'https://pbs.twimg.com/profile_banners/3027528856/1442813970', u'profile_background_image_url': u'http://abs.twimg.com/images/themes/theme1/bg.png', u'name': u'thalos', u'lang': u'pt', u'profile_background_tile': False, u'favourites_count': 515, u'screen_name': u'kpthales', u'notifications': None, u'url': None, u'created_at': u'Tue Feb 10 01:41:02 +0000 2015', u'contributors_enabled': False, u'time_zone': u'Pacific Time (US & Canada)', u'protected': False, u'default_profile': False, u'is_translator': False}, u'geo': None, u'in_reply_to_user_id_str': u'39062231', u'lang': u'pt', u'created_at': u'Tue Oct 06 19:17:15 +0000 2015', u'filter_level': u'low', u'in_reply_to_status_id_str': u'651476149676212224', u'place': {u'country_code': u'BR', u'url': u'https://api.twitter.com/1.1/geo/id/80e53f07bd506475.json', u'country': u'Brasil', u'place_type': u'city', u'bounding_box': {u'type': u'Polygon', u'coordinates': [[[-54.702115, -31.653423], [-54.702115, -30.838278], [-53.429682, -30.838278], [-53.429682, -31.653423]]]}, u'full_name': u'Bag\xe9, Rio Grande do Sul', u'attributes': {}, u'id': u'80e53f07bd506475', u'name': u'Bag\xe9'}}, {u'contributors': None, u'truncated': False, u'text': u'@soz71 initial thinking out loud operation #helpsarah', u'is_quote_status': False, u'in_reply_to_status_id': 651476083821428737L, u'id': 651476327837601792L, u'favorite_count': 0, u'source': u'<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', u'retweeted': False, u'coordinates': None, u'timestamp_ms': u'1444159035058', u'entities': {u'user_mentions': [{u'id': 255517623, u'indices': [0, 6], u'id_str': u'255517623', u'screen_name': u'soz71', u'name': u'Sarah S'}], u'symbols': [], u'hashtags': [{u'indices': [43, 53], u'text': u'helpsarah'}], u'urls': []}, u'in_reply_to_screen_name': u'Shane2510', u'id_str': u'651476327837601792', u'retweet_count': 0, u'in_reply_to_user_id': 2334163613L, u'favorited': False, u'user': {u'follow_request_sent': None, u'profile_use_background_image': True, u'default_profile_image': False, u'id': 2334163613L, u'verified': False, u'profile_image_url_https': u'https://pbs.twimg.com/profile_images/647775215771123712/WXr58y5v_normal.jpg', u'profile_sidebar_fill_color': u'DDEEF6', u'profile_text_color': u'333333', u'followers_count': 11, u'profile_sidebar_border_color': u'C0DEED', u'id_str': u'2334163613', u'profile_background_color': u'C0DEED', u'listed_count': 4, u'profile_background_image_url_https': u'https://abs.twimg.com/images/themes/theme1/bg.png', u'utc_offset': None, u'statuses_count': 3114, u'description': None, u'friends_count': 50, u'location': u'Plymouth', u'profile_link_color': u'0084B4', u'profile_image_url': u'http://pbs.twimg.com/profile_images/647775215771123712/WXr58y5v_normal.jpg', u'following': None, u'geo_enabled': True, u'profile_banner_url': u'https://pbs.twimg.com/profile_banners/2334163613/1443276376', u'profile_background_image_url': u'http://abs.twimg.com/images/themes/theme1/bg.png', u'name': u'Shane Harris', u'lang': u'en', u'profile_background_tile': False, u'favourites_count': 2839, u'screen_name': u'Shane2510', u'notifications': None, u'url': None, u'created_at': u'Sun Feb 09 18:03:12 +0000 2014', u'contributors_enabled': False, u'time_zone': None, u'protected': False, u'default_profile': True, u'is_translator': False}, u'geo': None, u'in_reply_to_user_id_str': u'2334163613', u'lang': u'en', u'created_at': u'Tue Oct 06 19:17:15 +0000 2015', u'filter_level': u'low', u'in_reply_to_status_id_str': u'651476083821428737', u'place': {u'country_code': u'GB', u'url': u'https://api.twitter.com/1.1/geo/id/7aeb72d9e8a1c889.json', u'country': u'United Kingdom', u'place_type': u'city', u'bounding_box': {u'type': u'Polygon', u'coordinates': [[[-4.203701, 50.360251], [-4.203701, 50.444179], [-4.082159, 50.444179], [-4.082159, 50.360251]]]}, u'full_name': u'Plymouth, England', u'attributes': {}, u'id': u'7aeb72d9e8a1c889', u'name': u'Plymouth'}}, {u'contributors': None, u'truncated': False, u'text': u"I'm at Michell Som in Recife, PE https://t.co/CZba2ya91M", u'is_quote_status': False, u'in_reply_to_status_id': None, u'id': 651476327921524736L, u'favorite_count': 0, u'source': u'<a href="http://foursquare.com" rel="nofollow">Foursquare</a>', u'retweeted': False, u'coordinates': {u'type': u'Point', u'coordinates': [-34.94155121, -8.08021069]}, u'timestamp_ms': u'1444159035078', u'entities': {u'user_mentions': [], u'symbols': [], u'hashtags': [], u'urls': [{u'url': u'https://t.co/CZba2ya91M', u'indices': [33, 56], u'expanded_url': u'https://www.swarmapp.com/c/iaAzHNXlDc0', u'display_url': u'swarmapp.com/c/iaAzHNXlDc0'}]}, u'in_reply_to_screen_name': None, u'id_str': u'651476327921524736', u'retweet_count': 0, u'in_reply_to_user_id': None, u'favorited': False, u'user': {u'follow_request_sent': None, u'profile_use_background_image': True, u'default_profile_image': False, u'id': 1112894017, u'verified': False, u'profile_image_url_https': u'https://pbs.twimg.com/profile_images/551388293678387200/I8nLAs2l_normal.jpeg', u'profile_sidebar_fill_color': u'F6FFD1', u'profile_text_color': u'333333', u'followers_count': 1304, u'profile_sidebar_border_color': u'FFF8AD', u'id_str': u'1112894017', u'profile_background_color': u'FFF04D', u'listed_count': 1, u'profile_background_image_url_https': u'https://abs.twimg.com/images/themes/theme19/bg.gif', u'utc_offset': -10800, u'statuses_count': 9089, u'description': None, u'friends_count': 1307, u'location': u'recife', u'profile_link_color': u'F5ABB5', u'profile_image_url': u'http://pbs.twimg.com/profile_images/551388293678387200/I8nLAs2l_normal.jpeg', u'following': None, u'geo_enabled': True, u'profile_banner_url': u'https://pbs.twimg.com/profile_banners/1112894017/1419688227', u'profile_background_image_url': u'http://abs.twimg.com/images/themes/theme19/bg.gif', u'name': u'CineideBetalab2015', u'lang': u'pt', u'profile_background_tile': False, u'favourites_count': 10, u'screen_name': u'CineideBETA2015', u'notifications': None, u'url': None, u'created_at': u'Tue Jan 22 23:17:57 +0000 2013', u'contributors_enabled': False, u'time_zone': u'Brasilia', u'protected': False, u'default_profile': False, u'is_translator': False}, u'geo': {u'type': u'Point', u'coordinates': [-8.08021069, -34.94155121]}, u'in_reply_to_user_id_str': None, u'possibly_sensitive': True, u'lang': u'en', u'created_at': u'Tue Oct 06 19:17:15 +0000 2015', u'filter_level': u'low', u'in_reply_to_status_id_str': None, u'place': {u'country_code': u'BR', u'url': u'https://api.twitter.com/1.1/geo/id/f207b85be9f1513e.json', u'country': u'Brasil', u'place_type': u'city', u'bounding_box': {u'type': u'Polygon', u'coordinates': [[[-35.019805, -8.157554], [-35.019805, -7.929652], [-34.858893, -7.929652], [-34.858893, -8.157554]]]}, u'full_name': u'Recife, Pernambuco', u'attributes': {}, u'id': u'f207b85be9f1513e', u'name': u'Recife'}}, {u'contributors': None, u'truncated': False, u'text': u'Besyocunun hali bi ba\u015fka oluyor ya :-D :-D @ Kastamonu \xdcniversitesi Besyo https://t.co/9aiFGLyeBU', u'is_quote_status': False, u'in_reply_to_status_id': None, u'id': 651476328013799424L, u'favorite_count': 0, u'source': u'<a href="http://instagram.com" rel="nofollow">Instagram</a>', u'retweeted': False, u'coordinates': {u'type': u'Point', u'coordinates': [33.77157419, 41.43254392]}, u'timestamp_ms': u'1444159035100', u'entities': {u'user_mentions': [], u'symbols': [], u'hashtags': [], u'urls': [{u'url': u'https://t.co/9aiFGLyeBU', u'indices': [74, 97], u'expanded_url': u'https://instagram.com/p/8ge_6TBRwf/', u'display_url': u'instagram.com/p/8ge_6TBRwf/'}]}, u'in_reply_to_screen_name': None, u'id_str': u'651476328013799424', u'retweet_count': 0, u'in_reply_to_user_id': None, u'favorited': False, u'user': {u'follow_request_sent': None, u'profile_use_background_image': False, u'default_profile_image': False, u'id': 2983706218L, u'verified': False, u'profile_image_url_https': u'https://pbs.twimg.com/profile_images/643064819168509952/2Gl-yitB_normal.jpg', u'profile_sidebar_fill_color': u'000000', u'profile_text_color': u'000000', u'followers_count': 211, u'profile_sidebar_border_color': u'000000', u'id_str': u'2983706218', u'profile_background_color': u'000000', u'listed_count': 1, u'profile_background_image_url_https': u'https://abs.twimg.com/images/themes/theme1/bg.png', u'utc_offset': None, u'statuses_count': 609, u'description': u'Sahte kahramanlarla i\u015fim olmaz benim kahraman\u0131m Atat\xfcrk. Ya\u015fas\u0131n \xfclk\xfcc\xfc T\xfcrk gen\xe7li\u011fi #CcC#MHP', u'friends_count': 454, u'location': u'kayseri develi', u'profile_link_color': u'3B94D9', u'profile_image_url': u'http://pbs.twimg.com/profile_images/643064819168509952/2Gl-yitB_normal.jpg', u'following': None, u'geo_enabled': True, u'profile_banner_url': u'https://pbs.twimg.com/profile_banners/2983706218/1442057111', u'profile_background_image_url': u'http://abs.twimg.com/images/themes/theme1/bg.png', u'name': u'Erhan Mahmut', u'lang': u'tr', u'profile_background_tile': False, u'favourites_count': 584, u'screen_name': u'erhan_mahmut', u'notifications': None, u'url': None, u'created_at': u'Sun Jan 18 07:22:49 +0000 2015', u'contributors_enabled': False, u'time_zone': None, u'protected': False, u'default_profile': False, u'is_translator': False}, u'geo': {u'type': u'Point', u'coordinates': [41.43254392, 33.77157419]}, u'in_reply_to_user_id_str': None, u'possibly_sensitive': False, u'lang': u'tr', u'created_at': u'Tue Oct 06 19:17:15 +0000 2015', u'filter_level': u'low', u'in_reply_to_status_id_str': None, u'place': {u'country_code': u'TR', u'url': u'https://api.twitter.com/1.1/geo/id/6d7ab717d3a54d55.json', u'country': u'T\xfcrkiye', u'place_type': u'city', u'bounding_box': {u'type': u'Polygon', u'coordinates': [[[33.750449, 41.332337], [33.750449, 41.435605], [33.814936, 41.435605], [33.814936, 41.332337]]]}, u'full_name': u'Kastamonu, T\xfcrkiye', u'attributes': {}, u'id': u'6d7ab717d3a54d55', u'name': u'Kastamonu'}}, {u'contributors': None, u'truncated': False, u'text': u'Rabbim do\u011frular\u0131 g\xf6ren ve do\u011fru konu\u015fanlar\u0131n say\u0131s\u0131n\u0131 artt\u0131rs\u0131n... http://t.co/XsISXBOMqf', u'is_quote_status': False, u'in_reply_to_status_id': None, u'id': 651476327007170560L, u'favorite_count': 0, u'source': u'<a href="http://twitter.com/download/iphone" rel="nofollow">Twitter for iPhone</a>', u'retweeted': False, u'coordinates': None, u'timestamp_ms': u'1444159034860', u'entities': {u'user_mentions': [], u'symbols': [], u'hashtags': [], u'urls': [], u'media': [{u'expanded_url': u'http://twitter.com/Ayenurzderya/status/651476327007170560/photo/1', u'display_url': u'pic.twitter.com/XsISXBOMqf', u'url': u'http://t.co/XsISXBOMqf', u'media_url_https': u'https://pbs.twimg.com/media/CQqCQSJUsAAZPOF.jpg', u'id_str': u'651476312368918528', u'sizes': {u'large': {u'h': 606, u'resize': u'fit', u'w': 640}, u'small': {u'h': 321, u'resize': u'fit', u'w': 340}, u'medium': {u'h': 568, u'resize': u'fit', u'w': 600}, u'thumb': {u'h': 150, u'resize': u'crop', u'w': 150}}, u'indices': [67, 89], u'type': u'photo', u'id': 651476312368918528L, u'media_url': u'http://pbs.twimg.com/media/CQqCQSJUsAAZPOF.jpg'}]}, u'in_reply_to_screen_name': None, u'id_str': u'651476327007170560', u'retweet_count': 0, u'in_reply_to_user_id': None, u'favorited': False, u'user': {u'follow_request_sent': None, u'profile_use_background_image': True, u'default_profile_image': False, u'id': 408349043, u'verified': False, u'profile_image_url_https': u'https://pbs.twimg.com/profile_images/463768014542749696/vIA_8cnE_normal.jpeg', u'profile_sidebar_fill_color': u'DDEEF6', u'profile_text_color': u'333333', u'followers_count': 20, u'profile_sidebar_border_color': u'C0DEED', u'id_str': u'408349043', u'profile_background_color': u'C0DEED', u'listed_count': 0, u'profile_background_image_url_https': u'https://abs.twimg.com/images/themes/theme1/bg.png', u'utc_offset': None, u'statuses_count': 73, u'description': u'\u0130stanbul', u'friends_count': 152, u'location': None, u'profile_link_color': u'0084B4', u'profile_image_url': u'http://pbs.twimg.com/profile_images/463768014542749696/vIA_8cnE_normal.jpeg', u'following': None, u'geo_enabled': True, u'profile_banner_url': u'https://pbs.twimg.com/profile_banners/408349043/1436256720', u'profile_background_image_url': u'http://abs.twimg.com/images/themes/theme1/bg.png', u'name': u'Ay\u015fenur \xd6zderya', u'lang': u'tr', u'profile_background_tile': False, u'favourites_count': 11, u'screen_name': u'Ayenurzderya', u'notifications': None, u'url': None, u'created_at': u'Wed Nov 09 09:29:37 +0000 2011', u'contributors_enabled': False, u'time_zone': None, u'protected': False, u'default_profile': True, u'is_translator': False}, u'geo': None, u'in_reply_to_user_id_str': None, u'possibly_sensitive': False, u'lang': u'tr', u'created_at': u'Tue Oct 06 19:17:14 +0000 2015', u'filter_level': u'low', u'in_reply_to_status_id_str': None, u'place': {u'country_code': u'TR', u'url': u'https://api.twitter.com/1.1/geo/id/5e02a0f0d91c76d2.json', u'country': u'T\xfcrkiye', u'place_type': u'city', u'bounding_box': {u'type': u'Polygon', u'coordinates': [[[28.632104, 40.802734], [28.632104, 41.239907], [29.378341, 41.239907], [29.378341, 40.802734]]]}, u'full_name': u'\u0130stanbul, T\xfcrkiye', u'attributes': {}, u'id': u'5e02a0f0d91c76d2', u'name': u'\u0130stanbul'}, u'extended_entities': {u'media': [{u'expanded_url': u'http://twitter.com/Ayenurzderya/status/651476327007170560/photo/1', u'display_url': u'pic.twitter.com/XsISXBOMqf', u'url': u'http://t.co/XsISXBOMqf', u'media_url_https': u'https://pbs.twimg.com/media/CQqCQSJUsAAZPOF.jpg', u'id_str': u'651476312368918528', u'sizes': {u'large': {u'h': 606, u'resize': u'fit', u'w': 640}, u'small': {u'h': 321, u'resize': u'fit', u'w': 340}, u'medium': {u'h': 568, u'resize': u'fit', u'w': 600}, u'thumb': {u'h': 150, u'resize': u'crop', u'w': 150}}, u'indices': [67, 89], u'type': u'photo', u'id': 651476312368918528L, u'media_url': u'http://pbs.twimg.com/media/CQqCQSJUsAAZPOF.jpg'}]}}]
# we'll pull out some pieces into a dataframe
# first, set up a dictionary of empty arrays
d = {'place': [], 'latitude': [], 'longitude': []}
for t in tweets:
try:
# first check whether the fields we want exist
_test = t['coordinates']['coordinates']
# then pull out the data
d['place'].append(t['place']['name'])
d['latitude'].append(t['coordinates']['coordinates'][1])
d['longitude'].append(t['coordinates']['coordinates'][0])
except:
# if the test failed, continue to next tweet
continue
# load it into a dataframe
df = pd.DataFrame.from_dict(d)
print len(df)
80
print df.head()
latitude longitude place 0 -8.080211 -34.941551 Recife 1 41.432544 33.771574 Kastamonu 2 3.094285 103.085270 Bebar 3 39.091116 -94.415507 Independence 4 34.967096 135.772691 京都市 伏見区
print df.sort('place').head(12)
latitude longitude place 21 41.007700 39.617398 Akçaabat 64 32.930000 -111.580000 Arizona 45 29.274240 47.932065 Ashbeliah 17 39.330337 26.663693 Ayvalık 2 3.094285 103.085270 Bebar 34 42.347485 -71.076020 Boston 69 -15.783500 -47.899164 Brasília 26 51.454582 -2.598296 Bristol 9 51.505778 -0.099747 Camberwell 14 44.861972 -93.571570 Chanhassen 36 3.736233 -75.423942 Chaparral 16 36.810974 -119.713646 Clovis
df.plot(x='longitude', y='latitude', kind='scatter')
<matplotlib.axes._subplots.AxesSubplot at 0xebe3be0>
# Note that when working with text strings that include characters from
# other alphabets, you need to keep track of the text encoding.
# Some interesting related reading:
# - http://www.joelonsoftware.com/articles/Unicode.html
df.to_csv('saved_coords.csv', encoding='utf-8')
Choose one:
Using one of the APIs from this demo, save and graph a different aspect of the data.
Or, search the web for another API that provides data you're interested in. Can you figure out how to connect to it using Python code?
Some common terms for describing these APIs that operate over the web are "HTTP" and "REST". The most frequent data format they provide is JSON, but with some code modifications you can parse other formats as well.