This notebook examines data from a complete harvest of series publicly available through RecordSearch in May 2021. It also compares the results to an earlier harvest in May 2021. See this notebook for the harvesting method.
import pandas as pd
df = pd.read_csv(
"series_totals_April_2022.csv",
dtype={
"described_total": "Int64",
"digitised_total": "Int64",
"access_open_total": "Int64",
"access_owe_total": "Int64",
"access_closed_total": "Int64",
"access_nye_total": "Int64",
},
)
df_prev = pd.read_csv(
"series_totals_May_2021.csv",
dtype={
"described_total": "Int64",
"digitised_total": "Int64",
"access_open_total": "Int64",
"access_owe_total": "Int64",
"access_closed_total": "Int64",
"access_nye_total": "Int64",
},
)
df.head()
identifier | title | contents_date_str | contents_start_date | contents_end_date | quantity_total | described_note | described_total | digitised_total | access_open_total | access_owe_total | access_closed_total | access_nye_total | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | A1 | Correspondence files, annual single number ser... | 01 Jan 1890 - 31 Dec 1969 | 1890-01-01 | 1969-12-31 | 477.41 | All items from this series are entered on Reco... | 64444.0 | 56666.0 | 64445.0 | 3.0 | 0.0 | 8.0 |
1 | A2 | Correspondence files, annual single number series | 01 Jan 1895 - 31 Dec 1926 | 1895-01-01 | 1926-12-31 | 35.74 | All items from this series are entered on Reco... | 3409.0 | 365.0 | 3403.0 | 0.0 | 0.0 | 6.0 |
2 | A3 | Correspondence files, annual single number ser... | 01 Jan 1839 - 23 May 1963 | 1839-01-01 | 1963-05-23 | 26.48 | All items from this series are entered on Reco... | 1382.0 | 264.0 | 1374.0 | 8.0 | 0.0 | 0.0 |
3 | A4 | Correspondence files, single number series wit... | NaN | NaN | NaN | 0.36 | Click to see items listed on RecordSearch. Ple... | 45.0 | 2.0 | 45.0 | 0.0 | 0.0 | 0.0 |
4 | A5 | Correspondence files, annual single number ser... | 01 Jan 1923 - 31 Dec 1924 | 1923-01-01 | 1924-12-31 | 1.80 | Click to see items listed on RecordSearch. Ple... | 200.0 | 14.0 | 198.0 | 0.0 | 0.0 | 2.0 |
Note that these numbers might not be exact. To work around the 20,000 search result limit, some totals have been calculated by aggregating a series of searches. In most cases this will be accurate, but some items have multiple control symbols and may be duplicated in the results. I think any errors will be small.
The numbers in brackets indicate the change since the last harvest in May 2021.
print(f"{df.shape[0]:,} series ({df.shape[0] - df_prev.shape[0]:+,})")
65,747 series (+28)
print(
f'{round(df["quantity_total"].sum(), 2):,} metres of records ({round((df["quantity_total"].sum() - df_prev["quantity_total"].sum()), 2):+,})'
)
341,963.73 metres of records (+4,293.04)
print(
f'{df["described_total"].sum():,} items described ({df["described_total"].sum() - df_prev["described_total"].sum():+,})'
)
14,624,131 items described (+713,641)
print(
f'{df["digitised_total"].sum():,} items digitised ({df["digitised_total"].sum()- df_prev["digitised_total"].sum():+,})'
)
2,366,101 items digitised (+131,869)
prev_percent = df_prev["digitised_total"].sum() / df_prev["described_total"].sum()
current_percent = df["digitised_total"].sum() / df["described_total"].sum()
print(
f"{current_percent:0.2%} of described items are digitised ({current_percent - prev_percent:+0.2%})"
)
16.18% of described items are digitised (+0.12%)
access_totals = [
{
"access status": "Open",
"total": df["access_open_total"].sum(),
"change": df["access_open_total"].sum() - df_prev["access_open_total"].sum(),
},
{
"access status": "Open with exceptions",
"total": df["access_owe_total"].sum(),
"change": df["access_owe_total"].sum() - df_prev["access_owe_total"].sum(),
},
{
"access status": "Closed",
"total": df["access_closed_total"].sum(),
"change": df["access_closed_total"].sum()
- df_prev["access_closed_total"].sum(),
},
{
"access status": "Not yet examined",
"total": df["access_nye_total"].sum(),
"change": df["access_nye_total"].sum() - df_prev["access_nye_total"].sum(),
},
]
df_access = pd.DataFrame(access_totals)
df_access["percent"] = df_access["total"] / df_access["total"].sum()
df_access.style.format(
{"total": "{:,.0f}", "change": "{:+,}", "percent": "{:0.2%}"}
).hide()
access status | total | change | percent |
---|---|---|---|
Open | 8,026,169 | +243,348 | 54.06% |
Open with exceptions | 110,688 | +1,553 | 0.75% |
Closed | 10,917 | -153 | 0.07% |
Not yet examined | 6,698,686 | +512,033 | 45.12% |
There's no way of knowing this from the harvested data. However, the recently-released Tune Review says that 37% of the NAA's holdings are described. So as we know the number described, we should be able to calculate an approximate number of total items.
print(f'Approximately {int(df["described_total"].sum() / 0.37):,} items in total')
Approximately 39,524,678 items in total
To put that another way, this is the approximate number of items not listed on RecordSearch:
print(
f'Approximately {int(df["described_total"].sum() / 0.37) - df["described_total"].sum():,} items **are not** listed on RecordSearch'
)
Approximately 24,900,547 items **are not** listed on RecordSearch
That's something to keep in mind if you're just relying on item keyword searches to find relevant content.
The note that accompanies the number of items listed in RecordSearch indicates how much of the series has been described at item level. By looking at the frequency of each of the values for this note, we can get a sese of the level of description across the collection.
df_described = df["described_note"].value_counts().to_frame()
df_described.columns = ["total"]
df_described["percent"] = df_described["total"] / df_described["total"].sum()
df_described.style.format({"total": "{:,.0f}", "percent": "{:0.2%}"})
total | percent | |
---|---|---|
No items from the series are on RecordSearch. Please contact the National Reference Service if you need assistance. | 41,584 | 63.25% |
Click to see items listed on RecordSearch. Please contact the National Reference Service if you can't find the record you want as not all items from the series may be on RecordSearch. | 12,966 | 19.72% |
All items from this series are entered on RecordSearch. | 11,166 | 16.98% |
No items from the series are on RecordSearch. Please contact the Australian War Memorial if you need assistance. | 30 | 0.05% |
The numbers above might be a bit misleading because sometimes series are registered on RecordSearch before any items are actually transferred to the NAA. So the reason there are no items listed might be that there are no items currently in Archives custody. To try an get a more accurate picture, we can filter out series where the quantity held by the NAA is equal to zero metres.
df_described_held = (
df.loc[df["quantity_total"] != 0]["described_note"].value_counts().to_frame()
)
df_described_held.columns = ["total"]
df_described_held["percent"] = (
df_described_held["total"] / df_described_held["total"].sum()
)
df_described_held.style.format({"total": "{:,.0f}", "percent": "{:0.2%}"})
total | percent | |
---|---|---|
No items from the series are on RecordSearch. Please contact the National Reference Service if you need assistance. | 19,227 | 47.29% |
All items from this series are entered on RecordSearch. | 11,044 | 27.16% |
Click to see items listed on RecordSearch. Please contact the National Reference Service if you can't find the record you want as not all items from the series may be on RecordSearch. | 10,384 | 25.54% |
No items from the series are on RecordSearch. Please contact the Australian War Memorial if you need assistance. | 1 | 0.00% |
This brings down the 'undescribed' proportion, though strangely this seems to indicate that there are zero shelf metres of some series which are fully described.
df.loc[
(df["described_note"].str.startswith("All")) & (df["quantity_total"] == 0)
].shape[0]
122
For example:
df.loc[(df["described_note"].str.startswith("All")) & (df["quantity_total"] == 0)].head(
2
)
identifier | title | contents_date_str | contents_start_date | contents_end_date | quantity_total | described_note | described_total | digitised_total | access_open_total | access_owe_total | access_closed_total | access_nye_total | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
121 | A123 | Name index cards (Departments), 'G' series | NaN | NaN | NaN | 0.0 | All items from this series are entered on Reco... | 2.0 | 0.0 | 2.0 | 0.0 | 0.0 | 0.0 |
742 | A749 | Volume of Circulars of Public Service Commissi... | NaN | NaN | NaN | 0.0 | All items from this series are entered on Reco... | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 |
So perhaps in some cases locations and quantities are not reliably recorded on RecordSearch.
From the items described note it seems that 19,227 series held by the NAA or AWM have no item level descriptions. We can check that by simply looking for series where the described_total
value is zero.
print(
f'{df.loc[(df["quantity_total"] > 0) & (df["described_total"] == 0)].shape[0]:,} series held by NAA have no item descriptions'
)
19,228 series held by NAA have no item descriptions
Yay! That (almost) matches.
Boo! That's a pretty significant black hole. Let's look at the quantity of records that represents.
print(
f'{df.loc[(df["quantity_total"] > 0) & (df["described_total"] == 0)]["quantity_total"].sum():,} linear metres in series held by NAA with no item descriptions'
)
51,733.74 linear metres in series held by NAA with no item descriptions
Of course, this doesn't include the quantities of series that are partially described.
Created by Tim Sherratt for the GLAM Workbench. Support me by becoming a GitHub sponsor!