New to Jupyter notebooks? Try Using Jupyter notebooks for a quick introduction.
Most of the notebooks in this repository work with small slices of web archive data. In this notebook we'll scale things up a bit to try and find all of the subdomains that have existed in the gov.au
domain. As in other notebooks, we'll obtain the data by querying the Internet Archive's CDX API. The only real difference is that it will take some hours to harvest all the data.
All we're interested in this time are unique domain names, so to minimise the amount of data we'll be harvesting we can make use of the CDX API's collapse
parameter. By setting collapse=urlkey
we can tell the CDX API to drop records with duplicate urlkey
values – this should mean we only get one capture per page. However, this only works if the capture records are in adjacent rows, so there probably will still be some duplicates. We'll also use the fl
to limit the fields returned, and the filter
parameter to limit results by statuscode
and mimetype
. So the parameters we'll use are:
url=*.gov.au
– all of the pages in all of the subdomains under gov.au
collapse=urlkey
– as few captures per page as possiblefilter=statuscode:200,mimetype:text/html
– only successful captures of HTML pagesfl=urlkey,timestamp,original
– only these fieldsEven with these limits, the query will retrieve a LOT of data. To make the harvesting process easier to manage and more robust, I'm going to make use of the requests-cache
module. This will capture the results of all requests, so that if things get interrupted and we have to restart, we can retrieve already harvested requests from the cache without downloading them again. We'll also write the harvested results directly to disk rather than consuming all our computer's memory. The file format will be the NDJSON (Newline Delineated JSON) format – because each line is a separate JSON object we can just write it a line at a time as the data is received.
For a general approach to harvesting domain-level information from the IA CDX API see Harvesting data about a domain using the IA CDX API
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
from tqdm.auto import tqdm
import pandas as pd
import time
from requests_cache import CachedSession
import ndjson
from pathlib import Path
from slugify import slugify
import arrow
import json
import re
from newick import Node
import newick
from ete3 import Tree, TreeStyle
import ipywidgets as widgets
from IPython.display import display, HTML, FileLink
s = CachedSession()
retries = Retry(total=10, backoff_factor=1, status_forcelist=[ 502, 503, 504 ])
s.mount('https://', HTTPAdapter(max_retries=retries))
s.mount('http://', HTTPAdapter(max_retries=retries))
domain = 'gov.au'
def get_total_pages(params):
'''
Gets the total number of pages in a set of results.
'''
these_params = params.copy()
these_params['showNumPages'] = 'true'
response = s.get('http://web.archive.org/cdx/search/cdx', params=these_params, headers={'User-Agent': ''})
return int(response.text)
def prepare_params(url, **kwargs):
'''
Prepare the parameters for a CDX API requests.
Adds all supplied keyword arguments as parameters (changing from_ to from).
Adds in a few necessary parameters.
'''
params = kwargs
params['url'] = url
params['output'] = 'json'
# CDX accepts a 'from' parameter, but this is a reserved word in Python
# Use 'from_' to pass the value to the function & here we'll change it back to 'from'.
if 'from_' in params:
params['from'] = params['from_']
del(params['from_'])
return params
def get_cdx_data(params):
'''
Make a request to the CDX API using the supplied parameters.
Check the results for a resumption key, and return the key (if any) and the results.
'''
response = s.get('http://web.archive.org/cdx/search/cdx', params=params, headers={'User-Agent': ''})
response.raise_for_status()
results = response.json()
if not response.from_cache:
time.sleep(0.2)
return results
def convert_lists_to_dicts(results):
if results:
keys = results[0]
results_as_dicts = [dict(zip(keys, v)) for v in results[1:]]
else:
results_as_dicts = results
return results_as_dicts
def get_cdx_data_by_page(url, **kwargs):
page = 0
params = prepare_params(url, **kwargs)
total_pages = get_total_pages(params)
# We'll use a timestamp to distinguish between versions
timestamp = arrow.now().format('YYYYMMDDHHmmss')
file_path = Path(f'{slugify(domain)}-cdx-data-{timestamp}.ndjson')
# Remove any old versions of the data file
try:
file_path.unlink()
except FileNotFoundError:
pass
with tqdm(total=total_pages-page) as pbar1:
with tqdm() as pbar2:
while page < total_pages:
params['page'] = page
results = get_cdx_data(params)
with file_path.open('a') as f:
writer = ndjson.writer(f, ensure_ascii=False)
for result in convert_lists_to_dicts(results):
writer.writerow(result)
page += 1
pbar1.update(1)
pbar2.update(len(results) - 1)
# Note than harvesting a domain has the same number of pages (ie requests) no matter what filters are applied -- it's just that some pages will be empty.
# So repeating a domain harvest with different filters will mean less data, but the same number of requests.
# What's most efficient? I dunno.
get_cdx_data_by_page(f'*.{domain}', filter=['statuscode:200', 'mimetype:text/html'], collapse='urlkey', fl='urlkey,timestamp,original', pageSize=5)
After many hours, and many interruptions, the harvesting process finally finished. I ended up with a 65gb ndjson file. How many captures does it include?
%%time
count = 0
with open('gov-au-cdx-data.ndjson') as f:
for line in f:
count += 1
print(f'{count:,}')
189,639,944 CPU times: user 1min 4s, sys: 27 s, total: 1min 31s Wall time: 2min 26s
Now let's get extract a list of unique domains from all of those page captures. In the code below we extract domains from the urlkey
and add them to a list. After every 100,000 lines, we use set
to remove duplicates from the list. This is an attempt to find a reasonable balance between speed and memory consumption.
%%time
# This is slow, but will avoid eating up memory
domains = []
with open('gov-au-cdx-data.ndjson') as f:
count = 0
with tqdm() as pbar:
for line in f:
capture = json.loads(line)
# Split the urlkey on ) to separate domain from path
domain = capture['urlkey'].split(')')[0]
# Remove port numbers
domain = re.sub(r'\:\d+', '', domain)
domains.append(domain)
count += 1
# Remove duplicates after every 100,000 lines to conserve memory
if count > 100000:
domains = list(set(domains))
pbar.update(count)
count = 0
domains = list(set(domains))
HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))
CPU times: user 16min 7s, sys: 30.8 s, total: 16min 38s Wall time: 16min 40s
How many unique domains are there?
len(domains)
26233
df = pd.DataFrame(domains, columns=['urlkey'])
df.head()
Save the list of domains to a CSV file to save us having to extract them again.
df.to_csv('domains/gov-au-unique-domains.csv', index=False)
display(FileLink('domains/gov-au-unique-domains.csv'))
Reload the list of domains from the CSV if necessary.
domains = pd.read_csv('domains/gov-au-unique-domains.csv')['urlkey'].to_list()
Now that we have a list of unique domains we can use this to generate a count of unique urls per subdomain. This won't be exact. As noted previously, even with collapse
set to urlkey
there are likely to be duplicate urls. Getting rid of all the duplicates in such a large file would require a fair bit of processing, and I'm not sure it's worth it at this point. We really just want a sense of how subdomains are actually used.
# Create a dictionary with the domains as keys and the values set to zero
domain_counts = dict(zip(domains, [0] * len(domains)))
%%time
# FIND NUMBER OF URLS PER DOMAIN
# As above we'll go though the file line by line
# but this time we'll extract the domain and increment the corresponding value in the dict.
with open('gov-au-cdx-data.ndjson') as f:
count = 0
with tqdm() as pbar:
for line in f:
capture = json.loads(line)
# Split the urlkey on ) to separate domain from path
domain = capture['urlkey'].split(')')[0]
domain = re.sub(r'\:\d+', '', domain)
# Increment domain count
domain_counts[domain] += 1
count += 1
# This is just to update the progress bar
if count > 100000:
pbar.update(count)
count = 0
HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))
CPU times: user 16min 52s, sys: 34.9 s, total: 17min 27s Wall time: 17min 30s
We'll now convert the data to a dataframe and do a bit more processing.
# Reshape dict as a list of dicts
domain_counts_as_list = [{'urlkey': k, 'number_of_pages': v} for k, v in domain_counts.items()]
# Convert to dataframe
df_counts = pd.DataFrame(domain_counts_as_list)
df_counts.head()
urlkey | number_of_pages | |
---|---|---|
0 | au,gov,qld,justice,mogservices | 6 |
1 | au,gov,health,business | 9 |
2 | au,gov,qld,sasvrc | 173 |
3 | au,gov,qld,qfes,dmlms | 4 |
4 | au,gov,wa,kwinana,maps | 4 |
Now we're going to split the urlkey
into its separate subdomains.
# Split the urlkey on commas into separate columns -- this creates a new df
df_split = df_counts['urlkey'].str.split(',', expand=True)
# Merge the new df back with the original so we have both the urlkey and it's components
df_merged = pd.merge(df_counts, df_split, left_index=True, right_index=True)
Finally, we'll stich the subdomains back together in a traditional domain format just for readability.
def join_domain(x):
parts = x.split(',')
parts.reverse()
return '.'.join(parts)
df_merged['domain'] = df_merged['urlkey'].apply(join_domain)
df_merged.head()
urlkey | number_of_pages | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | domain | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | au,gov,qld,justice,mogservices | 6 | au | gov | qld | justice | mogservices | None | None | None | None | None | mogservices.justice.qld.gov.au |
1 | au,gov,health,business | 9 | au | gov | health | business | None | None | None | None | None | None | business.health.gov.au |
2 | au,gov,qld,sasvrc | 173 | au | gov | qld | sasvrc | None | None | None | None | None | None | sasvrc.qld.gov.au |
3 | au,gov,qld,qfes,dmlms | 4 | au | gov | qld | qfes | dmlms | None | None | None | None | None | dmlms.qfes.qld.gov.au |
4 | au,gov,wa,kwinana,maps | 4 | au | gov | wa | kwinana | maps | None | None | None | None | None | maps.kwinana.wa.gov.au |
df_merged.to_csv('domains/gov-au-domains-split.csv')
display(FileLink('domains/gov-au-domains-split.csv'))
How many third level domains are there?
len(pd.unique(df_merged[2]))
1720
Which third level domains have the most subdomains?
df_merged[2].value_counts()[:20]
nsw 7477 vic 3418 qld 2772 wa 2690 sa 1719 tas 957 nt 752 act 362 embassy 151 nla 138 govspace 111 deewr 77 ga 75 treasury 74 ato 73 health 73 dest 69 abs 61 govcms 60 bom 59 Name: 2, dtype: int64
Which domains have the most unique pages?
top_20 = df_merged[['domain', 'number_of_pages']].sort_values(by='number_of_pages', ascending=False)[:20]
top_20.style.format({'number_of_pages': '{:,}'})
domain | number_of_pages | |
---|---|---|
24311 | trove.nla.gov.au | 9,285,603 |
8551 | nla.gov.au | 2,592,182 |
17232 | collectionsearch.nma.gov.au | 2,422,514 |
4551 | passwordreset.parliament.qld.gov.au | 2,089,256 |
18817 | parlinfo.aph.gov.au | 1,882,646 |
2050 | aph.gov.au | 1,731,559 |
11539 | bmcc.nsw.gov.au | 1,414,711 |
18038 | jobsearch.gov.au | 1,293,760 |
4556 | arpansa.gov.au | 1,278,603 |
22182 | abs.gov.au | 961,526 |
1844 | libero.gtcc.nsw.gov.au | 959,490 |
24888 | canterbury.nsw.gov.au | 956,500 |
20982 | library.campbelltown.nsw.gov.au | 932,933 |
9451 | defencejobs.gov.au | 894,770 |
18377 | webopac.gosford.nsw.gov.au | 854,395 |
3162 | library.lachlan.nsw.gov.au | 838,972 |
6141 | library.shoalhaven.nsw.gov.au | 800,541 |
16750 | catalogue.nla.gov.au | 787,616 |
25461 | library.bankstown.nsw.gov.au | 767,550 |
14964 | myagedcare.gov.au | 759,384 |
Are there really domains made up of 10 levels?
df_merged.loc[df_merged[9].notnull()]['domain'].to_list()
['0-slwa.csiro.patron.eb20.com.henrietta.slwa.wa.gov.au', 'test-your-tired-self-prod.apps.p.dmp.aws.hosting.transport.nsw.gov.au', '0-www.library.eb.com.au.henrietta.slwa.wa.gov.au']
I thought it would be interesting to try and visualise all the subdomains as a circular dendrogram. After a bit of investigation I discovered the ETE Toolkit for the visualisation of phylogenetic trees – it seemed perfect. But to get data into ETE I first had to convert it into a Newick formatted string. Fortunately, there's a Python package for that.
Warning! While the code below will indeed generate circular dendrograms from a domain name hierarchy, if you have more than a few hundred domains you'll find that the image gets very big, very quickly. I successfully saved the whole of the gov.au
domain as a 32mb SVG file, which you can (very slowly) view in a web browser or graphics program. But any attempt to save into another image format at a size that would make the text readable consumed huge amounts of memory and forced me to pull the plug.
def make_domain_tree(domains):
'''
Converts a list of urlkeys into a Newick tree via nodes.
'''
d_tree = Node()
for domain in domains:
domain = re.sub(r'\:\d+', '', domain)
sds = domain.split(',')
for i, sd in enumerate(sds):
parent = '.'.join(reversed(sds[0:i])) if i > 0 else None
label = '.'.join(reversed(sds[:i+1]))
if not d_tree.get_node(label):
if parent:
d_tree.get_node(parent).add_descendant(Node(label))
else:
d_tree.add_descendant(Node(label))
return newick.dumps(d_tree)
# Convert domains to a Newick tree
full_tree = make_domain_tree(domains)
def save_dendrogram_to_file(tree, width, output_file):
t = Tree(tree, format=1)
circular_style = TreeStyle()
circular_style.mode = "c" # draw tree in circular mode
circular_style.optimal_scale_level = 'full'
circular_style.root_opening_factor = 0
circular_style.show_scale = False
t.render(output_file, w=width, tree_style=circular_style)
First let's play safe by creating a PNG with a fixed width.
# Saving a PNG with a fixed width will work, but you won't be able to read any text
save_dendrogram_to_file(full_tree, 1000, 'images/govau-all-1000.png')
Here's the result!
This will save a zoomable SVG version that allows you to read the labels, but it will be very slow to use, and difficult to convert into other formats.
# Here be dendrodragons!
# I don't think width does anything if you save to SVG
save_dendrogram_to_file(full_tree, 5000, 'govau-all.svg')
Let's try some third level domains.
def display_dendrogram(label, level=2, df=df_merged, width=300):
domains = df.loc[df[2] == label]['urlkey'].to_list()
tree = make_domain_tree(domains)
save_dendrogram_to_file(tree, width, f'images/{label}-domains-{width}.png')
return f'<div style="width: 300px; float: left; margin-right: 10px;"><img src="images/{label}-domains-{width}.png" style=""><p style="text-align: center;">{label.upper()}</p></div>'
# Create dendrograms for each state/territory
html = ''
for state in ['nsw', 'vic', 'qld', 'sa', 'wa', 'tas', 'nt', 'act']:
html += display_dendrogram(state)
display(HTML(html))
NSW
VIC
QLD
SA
WA
TAS
NT
ACT
If there are fewer domains you can see more detail.
act = display_dendrogram(state, width=8000)
display(HTML(act))
Created by Tim Sherratt for the GLAM Workbench.
Work on this notebook was supported by the IIPC Discretionary Funding Programme 2019-2020