Geospatial Data Science Applications: GEOG 4/590

Feb 14, 2022

Lecture 7: Data access

Johnny Ryan: jryan4@uoregon.edu

Content of this lecture

• Web2.0

* Standard data access using APIs

* What to do when an API is unavailable or insufficient?

* Background for this week's lab

from IPython.display import HTML
HTML('<iframe width="560" height="315" src="https://www.youtube.com/embed/BxV14h0kFs0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>')

/opt/anaconda3/envs/clouds/lib/python3.8/site-packages/IPython/core/display.py:724: UserWarning: Consider using IPython.display.IFrame instead
  warnings.warn("Consider using IPython.display.IFrame instead")

APIs

• Application programming interface

• A type of software that provides a standard set of protocols/functions so that our computer can communicate with other computers

• In contrast, a user interface is a type of software that connects a computer to a person

APIs

• Many organizations have great APIs because they want people to use their data

• We have used one...

• cenpy

# Install package
!pip install -U dataretrieval

Collecting dataretrieval
  Downloading dataretrieval-0.7-py3-none-any.whl (16 kB)
Requirement already satisfied: requests in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from dataretrieval) (2.27.1)
Requirement already satisfied: pandas in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from dataretrieval) (1.4.0)
Requirement already satisfied: python-dateutil>=2.8.1 in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from pandas->dataretrieval) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from pandas->dataretrieval) (2021.3)
Requirement already satisfied: numpy>=1.21.0 in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from pandas->dataretrieval) (1.22.1)
Requirement already satisfied: idna<4,>=2.5 in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from requests->dataretrieval) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from requests->dataretrieval) (2.0.11)
Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from requests->dataretrieval) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from requests->dataretrieval) (1.26.8)
Requirement already satisfied: six>=1.5 in /opt/anaconda3/envs/lab7/lib/python3.10/site-packages (from python-dateutil>=2.8.1->pandas->dataretrieval) (1.16.0)
Installing collected packages: dataretrieval
Successfully installed dataretrieval-0.7

# Import the functions for downloading data from NWIS
import dataretrieval.nwis as nwis

# Specify the USGS site code
site = '03339000'

# Get instantaneous values (iv)
df = nwis.get_record(sites=site, service='dv', start='2020-10-01', end='2021-09-30')
df

	00010_Mean	00010_Mean_cd	site_no	00060_Mean	00060_Mean_cd	00065_Mean	00065_Mean_cd	00095_Mean	00095_Mean_cd	00300_Minimum	...	63680_ysi_Mean	63680_ysi_Mean_cd	91049_Mean	91049_Mean_cd	99133_Maximum	99133_Maximum_cd	99133_Minimum	99133_Minimum_cd	99133_Mean	99133_Mean_cd
datetime
2020-10-01 00:00:00+00:00	14.9	A	03339000	75.7	A	2.04	A	672.0	A	8.9	...	3.7	A	1510.0	P	4.1	A	3.3	A	3.7	A
2020-10-02 00:00:00+00:00	14.5	A	03339000	66.0	A	1.99	A	678.0	A	9.2	...	3.6	A	1450.0	P	4.2	A	3.8	A	4.1	A
2020-10-03 00:00:00+00:00	14.2	A	03339000	60.2	A	1.96	A	678.0	A	9.4	...	3.2	A	1110.0	P	3.8	A	3.0	A	3.4	A
2020-10-04 00:00:00+00:00	14.4	A	03339000	68.8	A	2.01	A	705.0	A	9.1	...	3.3	A	1030.0	P	3.0	A	2.6	A	2.8	A
2020-10-05 00:00:00+00:00	13.4	A	03339000	66.8	A	1.99	A	718.0	A	9.6	...	3.1	A	1080.0	P	3.4	A	2.7	A	3.0	A
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
2021-09-26 00:00:00+00:00	18.7	A	03339000	174.0	A	2.52	A	571.0	A	NaN	...	12.6	A	2080.0	P	2.5	A	2.1	A	2.2	A
2021-09-27 00:00:00+00:00	20.2	A	03339000	155.0	A	2.46	A	578.0	A	NaN	...	11.7	A	2050.0	P	2.6	A	2.2	A	2.4	A
2021-09-28 00:00:00+00:00	21.6	A	03339000	132.0	P	2.37	P	587.0	A	NaN	...	10.7	A	1600.0	P	2.4	A	2.1	A	2.2	A
2021-09-29 00:00:00+00:00	22.3	A	03339000	117.0	P	2.31	P	614.0	A	NaN	...	9.8	A	1430.0	P	2.4	A	2.2	A	2.3	A
2021-09-30 00:00:00+00:00	22.6	A	03339000	111.0	P	2.28	P	628.0	A	NaN	...	8.6	A	1290.0	P	2.2	A	2.1	A	2.2	A

365 rows × 41 columns

# Simple plot
df['00060_Mean'].plot()

<AxesSubplot:xlabel='datetime'>

Tips for APIs

• Take a minute to make sure that package is used (e.g. lots of forks, stars) and up-to-date (e.g. last commit)

• Read the docs, demos, examples and hope we find what we're looking for (sometimes they are not that comprehensive)

• If you can't find what you're looking for, inspect the source code (.py files)

APIs are sometimes not available or have limitations

• These companies hoard data to secure market dominance

• Without access to their data it is difficult to tell whether they are in compliance

• By guarding data, they are also preventing it being used for good (maybe bad) causes

Facebook loophole that allowed third-party apps to access not only user profile data but profile data of all friends. Kogan copied someone else's idea and made "thisisyourdigitallife". Only 270K used it but collected data from all friends of users (50+ million FB profiles).

Web scraping

• Also known as crawling or harvesting is the practice of automatically gathering data from the internet without the use of an API

• Most commonly accomplished by writing a program that queries a web server, requests data (usually in the form of HTML), and parses that data to extract information

Suppose a friend wanted to do this?

• Some HTML basics

• requests: standard Python library for requesting data from the web

• BeautifulSoup: a library for pulling data out of HTML and XML files

• selenium: is a library for performing web browser automation

Geospatial Data Science Applications: GEOG 4/590

Feb 14, 2022

Lecture 7: Data access

Johnny Ryan: jryan4@uoregon.edu

requests

# Import packages
import requests 

# Open a webpage
html = requests.get('https://en.wikipedia.org/wiki/Climate_of_Oregon')

# HTML
html

<Response [200]>

BeautifulSoup4

• Now we could write a program to parse this HTML code (i.e. split into useful blocks)...

• ...or we could use another package called BeautifulSoup (also known as bs4) a Python library for parsing data out of HTML and XML files

Import packages

# Import package
from bs4 import BeautifulSoup, SoupStrainer

# Read HTML content as "soup object" and define default parser
soup = BeautifulSoup(html.text, 'html.parser')

Parse HTML using

The .find and .find_all are the most common methods we will use. They can be used to filter HTML code to find a list of tags or tags with specific attributes.

# Define heading tags
heading_tags = ["h1", "h2"]

# Find heading tags in HTML code
headings = soup.find_all(heading_tags)

# Loop over every heading and print text
for tags in headings:
    print(tags.name + ' -> ' + tags.text.strip())

h1 -> Climate of Oregon
h2 -> Contents
h2 -> Precipitation[edit]
h2 -> Temperature[edit]
h2 -> Selected climate charts[edit]
h2 -> See also[edit]
h2 -> References[edit]
h2 -> Navigation menu

# Find every hyperlink
links = soup.find_all('a')

# Loop over every link and print hyperlink
for link in links:
    print(link.get('href'))

None
#mw-head
#searchInput
/wiki/File:Oregon_K%C3%B6ppen.svg
/wiki/File:Oregon_K%C3%B6ppen.svg
/wiki/K%C3%B6ppen_climate_classification
/wiki/Oregon
/wiki/File:OR_trewartha.svg
/wiki/File:OR_trewartha.svg
/wiki/Trewartha_climate_classification
/wiki/K%C3%B6ppen_climate_classification
/wiki/Western_Oregon
/wiki/Warm-summer_mediterranean_climate
/wiki/Eastern_Oregon
/wiki/Cold_semi-arid_climate
/wiki/Cascade_Range
/wiki/Arctic
/wiki/Cold_waves
#cite_note-1
#Precipitation
#Snow
#Temperature
#Selected_climate_charts
#See_also
#References
#Works_cited
/w/index.php?title=Climate_of_Oregon&action=edit&section=1
/wiki/File:Oregon_Average_Annual_Precipitation_(1961-1990)_Map.png
/wiki/File:Oregon_Average_Annual_Precipitation_(1961-1990)_Map.png
/wiki/Precipitation_(meteorology)
/wiki/Alvord_Desert
/wiki/Rain_shadow
/wiki/Steens_Mountain
#cite_note-FOOTNOTEAllanBuckleyMeacham2001154–55-2
#cite_note-WRCC-3
/wiki/Oregon_Coast_Range
/wiki/Condensation
/wiki/Orographic_precipitation
#cite_note-FOOTNOTEAllanBuckleyMeacham2001154–55-2
/wiki/Willamette_Valley
/wiki/Cascade_Range
#cite_note-FOOTNOTEAllanBuckleyMeacham2001154–55-2
/wiki/Mount_Hood
#cite_note-FOOTNOTEAllanBuckleyMeacham2001130–31-4
#cite_note-FOOTNOTEAllanBuckleyMeacham2001154–55-2
/wiki/Semi-arid_climate
/wiki/Blue_Mountains_(Pacific_Northwest)
/wiki/Wallowa_Mountains
#cite_note-FOOTNOTEAllanBuckleyMeacham2001154–55-2
/wiki/Jet_stream
#cite_note-FOOTNOTEAllanBuckleyMeacham2001156–57-5
/w/index.php?title=Climate_of_Oregon&action=edit&section=2
/wiki/File:Wintersports_terrain_park_P1391.jpeg
/wiki/File:Wintersports_terrain_park_P1391.jpeg
/wiki/Cascade_Range
/wiki/Snowfall
#cite_note-WRCC-3
/wiki/Crater_Lake
#cite_note-FOOTNOTETaylorHannan1999xii-6
#cite_note-WRCC-3
/wiki/Glacier
#cite_note-WRCC-3
/wiki/Snake_River
#cite_note-WRCC-3
/w/index.php?title=Climate_of_Oregon&action=edit&section=3
/wiki/File:Oregon_High_Desert.jpg
/wiki/File:Oregon_High_Desert.jpg
/wiki/High_Desert_(Oregon)
/wiki/Columbia_Gorge
#cite_note-FOOTNOTEAllanBuckleyMeacham2001158–59-7
#cite_note-WRCC-3
/wiki/Hermiston,_Oregon
/wiki/Redmond,_Oregon
#cite_note-FOOTNOTETaylorHannan1999xii-6
/wiki/Ukiah,_Oregon
/wiki/Seneca,_Oregon
#cite_note-FOOTNOTETaylorHannan1999xii-6
/wiki/Willamette_Valley
#cite_note-8
/wiki/Jackson_County,_Oregon
#cite_note-9
/w/index.php?title=Climate_of_Oregon&action=edit&section=4
/wiki/File:USA_Oregon_relief_location_map.svg
/wiki/Astoria,_Oregon
/wiki/Bend,_Oregon
/wiki/Brookings,_Oregon
/wiki/Burns,_Oregon
/wiki/Eugene,_Oregon
/wiki/Medford,_Oregon
/wiki/Newport,_Oregon
/wiki/Ontario,_Oregon
/wiki/Pendleton,_Oregon
/wiki/Portland,_Oregon
/wiki/File:USA_Oregon_relief_location_map.svg
/wiki/Astoria,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/wiki/Bend,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/wiki/Brookings,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/wiki/Burns,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/wiki/Eugene,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/wiki/Medford,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/wiki/Newport,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/wiki/Ontario,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/wiki/Pendleton,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/wiki/Portland,_Oregon
/wiki/Template:Climate_chart/How_to_read_a_climate_chart
/w/index.php?title=Climate_of_Oregon&action=edit&section=5
/wiki/Climate_of_the_United_States
/wiki/Climate_change_in_Oregon
/w/index.php?title=Climate_of_Oregon&action=edit&section=6
#cite_ref-1
https://web.archive.org/web/20090531012223/http://www.geog.nau.edu/courses/alew/ggr346/text/chapters/ch10.html
/wiki/Northern_Arizona_University
http://www.geog.nau.edu/courses/alew/ggr346/text/chapters/ch10.html
#cite_ref-FOOTNOTEAllanBuckleyMeacham2001154–55_2-0
#cite_ref-FOOTNOTEAllanBuckleyMeacham2001154–55_2-1
#cite_ref-FOOTNOTEAllanBuckleyMeacham2001154–55_2-2
#cite_ref-FOOTNOTEAllanBuckleyMeacham2001154–55_2-3
#cite_ref-FOOTNOTEAllanBuckleyMeacham2001154–55_2-4
#CITEREFAllanBuckleyMeacham2001
#cite_ref-WRCC_3-0
#cite_ref-WRCC_3-1
#cite_ref-WRCC_3-2
#cite_ref-WRCC_3-3
#cite_ref-WRCC_3-4
#cite_ref-WRCC_3-5
http://www.wrcc.dri.edu/narratives/OREGON.htm
/wiki/Desert_Research_Institute
#cite_ref-FOOTNOTEAllanBuckleyMeacham2001130–31_4-0
#CITEREFAllanBuckleyMeacham2001
#cite_ref-FOOTNOTEAllanBuckleyMeacham2001156–57_5-0
#CITEREFAllanBuckleyMeacham2001
#cite_ref-FOOTNOTETaylorHannan1999xii_6-0
#cite_ref-FOOTNOTETaylorHannan1999xii_6-1
#cite_ref-FOOTNOTETaylorHannan1999xii_6-2
#CITEREFTaylorHannan1999
#cite_ref-FOOTNOTEAllanBuckleyMeacham2001158–59_7-0
#CITEREFAllanBuckleyMeacham2001
#cite_ref-8
https://web.archive.org/web/20100624153919/http://www.ocs.orst.edu/county_climate/Multnomah_files/Multnomah.html
/wiki/Oregon_State_University
http://www.ocs.orst.edu/county_climate/Multnomah_files/Multnomah.html
/wiki/Template:Cite_web
/wiki/Category:CS1_maint:_multiple_names:_authors_list
#cite_ref-9
https://web.archive.org/web/20100304143656/http://www.ocs.oregonstate.edu/county_climate/Jackson_files/Jackson.html
/wiki/Oregon_State_University
http://ocs.oregonstate.edu/county_climate/Jackson_files/Jackson.html
/wiki/Template:Cite_web
/wiki/Category:CS1_maint:_multiple_names:_authors_list
/w/index.php?title=Climate_of_Oregon&action=edit&section=7
/wiki/ISBN_(identifier)
/wiki/Special:BookSources/0-87114-101-9
/wiki/ISBN_(identifier)
/wiki/Special:BookSources/978-0-87071-468-9
/wiki/Template:Oregon
/wiki/Template_talk:Oregon
https://en.wikipedia.org/w/index.php?title=Template:Oregon&action=edit
/wiki/U.S._state
/wiki/Oregon
/wiki/Salem,_Oregon
/wiki/Lists_of_Oregon-related_topics
/wiki/Index_of_Oregon-related_articles
/wiki/Outline_of_Oregon
None
/wiki/Oregon#Geography
/wiki/Fauna_of_Oregon
/wiki/List_of_beaches_in_Oregon
/wiki/List_of_lakes_of_Oregon
/wiki/List_of_rivers_of_Oregon
/wiki/Government_of_Oregon
/wiki/Constitution_of_Oregon
/wiki/United_States_congressional_delegations_from_Oregon
/wiki/List_of_Oregon_ballot_measures
/wiki/Elections_in_Oregon
/wiki/Governor_of_Oregon
/wiki/List_of_governors_of_Oregon
/wiki/History_of_Oregon
/wiki/Bibliography_of_Oregon_history
/wiki/Oregon_Trail
/wiki/List_of_Oregon_state_parks
/wiki/List_of_people_from_Oregon
/wiki/Oregon_pioneer_history
/wiki/Oregon_State_Fair
/wiki/List_of_Oregon_state_symbols
/wiki/Culture_of_Oregon
/wiki/Crime_in_Oregon
/wiki/Oregon#Demographics
/wiki/Economy_of_Oregon
/wiki/Oregon#Education
/wiki/Gambling_in_Oregon
/wiki/Politics_of_Oregon
/wiki/List_of_regions_of_Oregon
/wiki/Western_Oregon
/wiki/Northwest_Oregon
/wiki/Oregon_Coast
/wiki/Portland_metropolitan_area,_Oregon
/wiki/Tualatin_Valley
/wiki/Willamette_Valley
/wiki/Eastern_Oregon
/wiki/Harney_Basin
/wiki/High_Desert_(Oregon)
/wiki/Palouse
/wiki/Treasure_Valley
/wiki/Central_Oregon
/wiki/Southern_Oregon
/wiki/Rogue_Valley
/wiki/Cascade_Range
/wiki/Columbia_River_Gorge
/wiki/Columbia_River
/wiki/Columbia_Plateau
/wiki/Great_Basin
/wiki/Mount_Hood_Corridor
/wiki/Trout_Creek_Mountains
/wiki/Oregon_statistical_areas
/wiki/Albany%E2%80%93Corvallis%E2%80%93Lebanon_combined_statistical_area
/wiki/Bend%E2%80%93Prineville_combined_statistical_area
/wiki/Eugene,_Oregon
/wiki/Springfield,_Oregon
/wiki/Medford,_Oregon
/wiki/Ashland,_Oregon
/wiki/Portland_metropolitan_area,_Oregon
/wiki/Salem_Metropolitan_Statistical_Area
/wiki/List_of_cities_in_Oregon
/wiki/Portland,_Oregon
/wiki/Salem,_Oregon
/wiki/Eugene,_Oregon
/wiki/Gresham,_Oregon
/wiki/Hillsboro,_Oregon
/wiki/Beaverton,_Oregon
/wiki/Bend,_Oregon
/wiki/Medford,_Oregon
/wiki/Springfield,_Oregon
/wiki/Corvallis,_Oregon
/wiki/Albany,_Oregon
/wiki/Tigard,_Oregon
/wiki/Lake_Oswego,_Oregon
/wiki/Keizer,_Oregon
/wiki/Grants_Pass,_Oregon
/wiki/Oregon_City,_Oregon
/wiki/McMinnville,_Oregon
/wiki/Redmond,_Oregon
/wiki/Tualatin,_Oregon
/wiki/West_Linn,_Oregon
/wiki/Woodburn,_Oregon
/wiki/Forest_Grove,_Oregon
/wiki/Newberg,_Oregon
/wiki/Wilsonville,_Oregon
/wiki/Roseburg,_Oregon
/wiki/Klamath_Falls,_Oregon
/wiki/Ashland,_Oregon
/wiki/Milwaukie,_Oregon
/wiki/Sherwood,_Oregon
/wiki/Happy_Valley,_Oregon
/wiki/Central_Point,_Oregon
/wiki/Canby,_Oregon
/wiki/Hermiston,_Oregon
/wiki/Pendleton,_Oregon
/wiki/List_of_counties_in_Oregon
/wiki/Baker_County,_Oregon
/wiki/Benton_County,_Oregon
/wiki/Clackamas_County,_Oregon
/wiki/Clatsop_County,_Oregon
/wiki/Columbia_County,_Oregon
/wiki/Coos_County,_Oregon
/wiki/Crook_County,_Oregon
/wiki/Curry_County,_Oregon
/wiki/Deschutes_County,_Oregon
/wiki/Douglas_County,_Oregon
/wiki/Gilliam_County,_Oregon
/wiki/Grant_County,_Oregon
/wiki/Harney_County,_Oregon
/wiki/Hood_River_County,_Oregon
/wiki/Jackson_County,_Oregon
/wiki/Jefferson_County,_Oregon
/wiki/Josephine_County,_Oregon
/wiki/Klamath_County,_Oregon
/wiki/Lake_County,_Oregon
/wiki/Lane_County,_Oregon
/wiki/Lincoln_County,_Oregon
/wiki/Linn_County,_Oregon
/wiki/Malheur_County,_Oregon
/wiki/Marion_County,_Oregon
/wiki/Morrow_County,_Oregon
/wiki/Multnomah_County,_Oregon
/wiki/Polk_County,_Oregon
/wiki/Sherman_County,_Oregon
/wiki/Tillamook_County,_Oregon
/wiki/Umatilla_County,_Oregon
/wiki/Union_County,_Oregon
/wiki/Wallowa_County,_Oregon
/wiki/Wasco_County,_Oregon
/wiki/Washington_County,_Oregon
/wiki/Wheeler_County,_Oregon
/wiki/Yamhill_County,_Oregon
/wiki/File:Flag_of_Oregon.svg
/wiki/Portal:Oregon
/wiki/File:Abies_lasiocarpa_0775.JPG
/wiki/Portal:Pacific_Northwest
/wiki/Template:ClimateUS
/wiki/Template_talk:ClimateUS
https://en.wikipedia.org/w/index.php?title=Template:ClimateUS&action=edit
/wiki/Climate_of_the_United_States
/wiki/Climate_of_Alabama
/wiki/Climate_of_Alaska
/wiki/Climate_of_Arizona
/wiki/Climate_of_Arkansas
/wiki/Climate_of_California
/wiki/Climate_of_Colorado
/wiki/Climate_of_Connecticut
/wiki/Climate_of_Delaware
/wiki/Climate_of_Florida
/wiki/Climate_of_Georgia_(U.S._state)
/wiki/Climate_of_Hawaii
/wiki/Climate_of_Idaho
/wiki/Climate_of_Illinois
/wiki/Climate_of_Indiana
/wiki/Climate_of_Iowa
/wiki/Climate_of_Kansas
/wiki/Climate_of_Kentucky
/wiki/Climate_of_Louisiana
/wiki/Climate_of_Maine
/wiki/Climate_of_Maryland
/wiki/Climate_of_Massachusetts
/wiki/Climate_of_Michigan
/wiki/Climate_of_Minnesota
/wiki/Climate_of_Mississippi
/wiki/Climate_of_Missouri
/wiki/Climate_of_Montana
/wiki/Climate_of_Nebraska
/wiki/Climate_of_Nevada
/wiki/Climate_of_New_Hampshire
/wiki/Climate_of_New_Jersey
/wiki/Climate_of_New_Mexico
/wiki/Climate_of_New_York
/wiki/Climate_of_North_Carolina
/wiki/Climate_of_North_Dakota
/wiki/Climate_of_Ohio
/wiki/Climate_of_Oklahoma
None
/wiki/Climate_of_Pennsylvania
/wiki/Climate_of_Rhode_Island
/wiki/Climate_of_South_Carolina
/wiki/Climate_of_South_Dakota
/wiki/Climate_of_Tennessee
/wiki/Climate_of_Texas
/wiki/Climate_of_Utah
/wiki/Climate_of_Vermont
/wiki/Climate_of_Virginia
/wiki/Climate_of_Washington_(state)
/wiki/Climate_of_West_Virginia
/wiki/Climate_of_Wisconsin
/wiki/Climate_of_Wyoming
/wiki/Federal_district
/wiki/Climate_of_Washington,_D.C.
/wiki/Insular_area
/wiki/Climate_of_American_Samoa
/wiki/Climate_of_Guam
/wiki/Climate_of_the_Northern_Mariana_Islands
/wiki/Climate_of_Puerto_Rico
/wiki/Climate_of_the_United_States_Virgin_Islands
/wiki/Climate_of_Allentown,_Pennsylvania
/wiki/Climate_of_Anchorage
/wiki/Climate_of_Beaumont,_Texas
/wiki/Climate_of_Bismarck,_North_Dakota
/wiki/Climate_of_Chicago
/wiki/Climate_of_Columbus,_Ohio
/wiki/Climate_of_Dallas
/wiki/Climate_of_Door_County,_Wisconsin
/wiki/Climate_of_Grand_Forks,_North_Dakota
/wiki/Climate_of_Houston
/wiki/Climate_of_Little_Rock,_Arkansas
/wiki/Climate_of_Los_Angeles
/wiki/Climate_of_New_England
/wiki/Climate_of_Miami
/wiki/Climate_of_Milwaukee
/wiki/Climate_of_Minneapolis%E2%80%93Saint_Paul
/wiki/Climate_of_New_York_City
/wiki/Climate_of_Oklahoma_City
/wiki/Climate_of_Salt_Lake_City
/wiki/Climate_of_San_Diego
/wiki/Climate_of_Seattle
/wiki/Climate_of_the_Tampa_Bay_area
/wiki/List_of_cities_by_sunshine_duration#North_America
/wiki/Climate_change_in_the_United_States
/wiki/List_of_derecho_events#North_America
/wiki/List_of_Storm_Prediction_Center_extremely_critical_days
/wiki/Drought_in_the_United_States
/wiki/Lists_of_floods_in_the_United_States
/wiki/List_of_Storm_Prediction_Center_high_risk_days
/wiki/List_of_costly_or_deadly_hailstorms#North_America
/wiki/List_of_United_States_hurricanes
/wiki/List_of_major_snow_and_ice_events_in_the_United_States
/wiki/Snow_in_Florida
/wiki/Snow_in_Louisiana
/wiki/List_of_snowiest_places_in_the_United_States_by_state
/wiki/Tornadoes_in_the_United_States
/wiki/Template:WeatherRecordsByState
/wiki/Weather-related_fatalities_in_the_United_States
https://en.wikipedia.org/w/index.php?title=Climate_of_Oregon&oldid=1030296200
/wiki/Help:Category
/wiki/Category:Climate_of_Oregon
/wiki/Category:Climate_of_the_United_States_by_state
/wiki/Category:CS1_maint:_multiple_names:_authors_list
/wiki/Special:MyTalk
/wiki/Special:MyContributions
/w/index.php?title=Special:CreateAccount&returnto=Climate+of+Oregon
/w/index.php?title=Special:UserLogin&returnto=Climate+of+Oregon
/wiki/Climate_of_Oregon
/wiki/Talk:Climate_of_Oregon
/wiki/Climate_of_Oregon
/w/index.php?title=Climate_of_Oregon&action=edit
/w/index.php?title=Climate_of_Oregon&action=history
/wiki/Main_Page
/wiki/Main_Page
/wiki/Wikipedia:Contents
/wiki/Portal:Current_events
/wiki/Special:Random
/wiki/Wikipedia:About
//en.wikipedia.org/wiki/Wikipedia:Contact_us
https://donate.wikimedia.org/wiki/Special:FundraiserRedirector?utm_source=donate&utm_medium=sidebar&utm_campaign=C13_en.wikipedia.org&uselang=en
/wiki/Help:Contents
/wiki/Help:Introduction
/wiki/Wikipedia:Community_portal
/wiki/Special:RecentChanges
/wiki/Wikipedia:File_Upload_Wizard
/wiki/Special:WhatLinksHere/Climate_of_Oregon
/wiki/Special:RecentChangesLinked/Climate_of_Oregon
/wiki/Wikipedia:File_Upload_Wizard
/wiki/Special:SpecialPages
/w/index.php?title=Climate_of_Oregon&oldid=1030296200
/w/index.php?title=Climate_of_Oregon&action=info
/w/index.php?title=Special:CiteThisPage&page=Climate_of_Oregon&id=1030296200&wpFormIdentifier=titleform
https://www.wikidata.org/wiki/Special:EntityPage/Q5133582
/w/index.php?title=Special:DownloadAsPdf&page=Climate_of_Oregon&action=show-download-screen
/w/index.php?title=Climate_of_Oregon&printable=yes
https://commons.wikimedia.org/wiki/Category:Climate_of_Oregon
https://www.wikidata.org/wiki/Special:EntityPage/Q5133582#sitelinks-wikipedia
//en.wikipedia.org/wiki/Wikipedia:Text_of_Creative_Commons_Attribution-ShareAlike_3.0_Unported_License
//creativecommons.org/licenses/by-sa/3.0/
//foundation.wikimedia.org/wiki/Terms_of_Use
//foundation.wikimedia.org/wiki/Privacy_policy
//www.wikimediafoundation.org/
https://foundation.wikimedia.org/wiki/Privacy_policy
/wiki/Wikipedia:About
/wiki/Wikipedia:General_disclaimer
//en.wikipedia.org/wiki/Wikipedia:Contact_us
//en.m.wikipedia.org/w/index.php?title=Climate_of_Oregon&mobileaction=toggle_view_mobile
https://www.mediawiki.org/wiki/Special:MyLanguage/How_to_contribute
https://stats.wikimedia.org/#/en.wikipedia.org
https://foundation.wikimedia.org/wiki/Cookie_statement
https://wikimediafoundation.org/
https://www.mediawiki.org/

# Find number of images on page
len(soup.find_all('img'))

23

# Print details of first image
print(soup.find_all('img')[0])

<img alt="" class="thumbimage" data-file-height="1359" data-file-width="1260" decoding="async" height="270" src="//upload.wikimedia.org/wikipedia/commons/thumb/c/c5/Oregon_K%C3%B6ppen.svg/250px-Oregon_K%C3%B6ppen.svg.png" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/c/c5/Oregon_K%C3%B6ppen.svg/375px-Oregon_K%C3%B6ppen.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/c/c5/Oregon_K%C3%B6ppen.svg/500px-Oregon_K%C3%B6ppen.svg.png 2x" width="250"/>

# Find attributes of first image
print(soup.find_all('img')[0].attrs['src'])

//upload.wikimedia.org/wikipedia/commons/thumb/c/c5/Oregon_K%C3%B6ppen.svg/250px-Oregon_K%C3%B6ppen.svg.png

# Download image
url = 'https://' + soup.find_all('img')[0].attrs['src'][2:]
response = requests.get(url)
if response.status_code == 200:
    with open("images/test_image.jpg", 'wb') as f:
        f.write(response.content)

# Import packages
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

# Read image
img = mpimg.imread('images/test_image.jpg')

# Plot image
plt.imshow(img)

<matplotlib.image.AxesImage at 0x1a1794d60>

Selenium

• Sometimes we want even more control...

• Selenium is a package for performing web browser automation

• We can use Selenium to enter text in search boxes, click buttons etc.

# Import packages
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager

# Install Chrome webdriver
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

# Open a web browser at the following page
driver.get("https://www.google.com/maps")

====== WebDriver manager ======
Current google-chrome version is 97.0.4692
Get LATEST chromedriver version for 97.0.4692 google-chrome
Driver [/Users/jryan4/.wdm/drivers/chromedriver/mac64/97.0.4692.71/chromedriver] found in cache

# Enter some text in the search box
inputElement = driver.find_element(By.ID, "searchboxinput")
inputElement.send_keys('South Sister Oregon')

# Click search button
element = driver.find_element(By.ID, "searchbox-searchbutton")
element.click()