This notebooks lets you harvest large amounts of data for Papers Past (via DigitalNZ) for further analysis. It saves the results as a CSV file that you can open in any spreadsheet program. It currently includes the OCRd text of all the newspaper articles, but I might make this optional in the future — thoughts?
You can edit this notebook to harvest other collections in DigitalNZ — see the notes below for pointers. However, this is currently only saving a small subset of the available metadata, so you'd probably want to adjust the fields as well. Add an issue on GitHub if you need help creating a custom harvester.
There's only two things you have to change — you need to enter your API key, and supply a search term. There are additional options for limiting your search results.
If you haven't used one of these notebooks before, they're basically web pages in which you can write, edit, and run live code. They're meant to encourage experimentation, so don't feel nervous. Just try running a few cells and see what happens!.
Some tips:
Go get yourself a DigitalNZ API key, then paste it between the quotes below. You need a key to make API requests, but they're free and quick to obtain.
# Past your API key between the quotes
# You might need to trim off any spaces at the beginning and end
API_KEY = '[YOUR API KEY]'
print('Your API key is: {}'.format(API_KEY))
Just run these cells to set up some things that we'll need later on.
# This cell just sets up some stuff that we'll need later
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import pandas as pd
from tqdm.auto import tqdm
import time
import re
from slugify import slugify
from time import strftime
from IPython.display import display, HTML
from pathlib import Path
s = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[ 502, 503, 504 ])
s.mount('https://', HTTPAdapter(max_retries=retries))
API_URL = 'http://api.digitalnz.org/v3/records.json'
This is where all the serious harvesting work gets done. You shouldn't need to change anything unless you want to harvest something other than Papers Past. Just run the cell.
def process_articles(results):
articles = []
for result in results:
# If you're harvesting something other than Papers Past, you'd probably
# want to change the way results are processed.
title = re.sub(r'(\([^)]*\))[^(]*$', '', result['title']).strip()
articles.append({
'id': result['id'],
'title': title,
'newspaper': result['publisher'][0],
'date': result['date'][0][:10],
'text': result['fulltext'],
'paperspast_url': result['landing_url'],
'source_url': result['source_url']
})
return articles
def get_total(params):
np = params.copy()
np['per_page'] = 0
data = get_records(np)
return data['search']['result_count']
def get_records(params):
response = requests.get(API_URL, params=params)
return response.json()
def harvest(params):
'''
Do the harvesting!
'''
more = True
articles = []
params['page'] = 1
total = get_total(params)
with tqdm(total=total) as pbar:
while more:
data = get_records(params)
results = data['search']['results']
if results:
articles += process_articles(data['search']['results'])
pbar.update(len(results))
params['page'] += 1
time.sleep(0.2)
else:
more = False
return articles
def start_harvest(query, start_year=None, end_year=None, **kwargs):
'''
Initiates a harvest.
If you've specified start and end years it'll loop over them getting results for each.
'''
params = {
'text': query,
'and[primary_collection][]': 'Papers Past',
'per_page': '100',
'api_key': API_KEY
}
for key, value in kwargs.items():
params[f'and[{key}][]'] = value
if start_year and end_year:
articles = []
for year in tqdm(range(start_year, end_year+1), desc='Years'):
current_year = year
params['and[year][]'] = year
articles += harvest(params)
else:
articles = harvest(params)
return articles
def save_as_csv(articles, query_name):
'''
Save the results as a CSV file.
Filename is constructed from the the supplied query_name and the current date/time.
Displays a download link when finished.
'''
Path('data').mkdir(exist_ok=True)
filename = f'{slugify(query_name)}-{strftime("%Y%m%d%H%M%S")}.csv'
df = pd.DataFrame(articles)
df.to_csv(Path('data', filename), index=False)
display(HTML(f'<a href="{Path("data", filename)}" download="{filename}">{filename}</a>'))
At the very least, the harvesting code is expecting you to supply a search query, like 'possum'. You just feed this query to the start_harvest()
function to kick things off.
articles = start_harvest('possum')
Now we can save the results as a CSV file.
save_as_csv(articles, 'possum')
There's no direct way to search for a range of years, but we can get around this by issuing a request for each year separately and then combining the results. Just give start_harvest()
a start_year
and end_year
value.
articles = start_harvest('possum', start_year=1905, end_year=1910)
save_as_csv(articles, 'possum-1905-1910')
You can add additional facets to the start_harvest()
function to limit the results further. For example, you can use the collection
facet to specify a particular newspaper:
collection='Evening Post'
Other facets you could use include:
century='1800'
decade='1850'
articles = start_harvest('possum', collection='Evening Post')
save_as_csv(articles, 'possum-evening-post')
Created by Tim Sherratt for the GLAM Workbench. Support this project by becoming a GitHub sponsor.