Exploring harvested series data, April 2022

This notebook examines data from a complete harvest of series publicly available through RecordSearch in May 2021. It also compares the results to an earlier harvest in May 2021. See this notebook for the harvesting method.

In [1]:
import pandas as pd
In [21]:
df = pd.read_csv(
    "series_totals_April_2022.csv",
    dtype={
        "described_total": "Int64",
        "digitised_total": "Int64",
        "access_open_total": "Int64",
        "access_owe_total": "Int64",
        "access_closed_total": "Int64",
        "access_nye_total": "Int64",
    },
)
In [22]:
df_prev = pd.read_csv(
    "series_totals_May_2021.csv",
    dtype={
        "described_total": "Int64",
        "digitised_total": "Int64",
        "access_open_total": "Int64",
        "access_owe_total": "Int64",
        "access_closed_total": "Int64",
        "access_nye_total": "Int64",
    },
)
In [3]:
df.head()
Out[3]:
identifier title contents_date_str contents_start_date contents_end_date quantity_total described_note described_total digitised_total access_open_total access_owe_total access_closed_total access_nye_total
0 A1 Correspondence files, annual single number ser... 01 Jan 1890 - 31 Dec 1969 1890-01-01 1969-12-31 477.41 All items from this series are entered on Reco... 64444.0 56666.0 64445.0 3.0 0.0 8.0
1 A2 Correspondence files, annual single number series 01 Jan 1895 - 31 Dec 1926 1895-01-01 1926-12-31 35.74 All items from this series are entered on Reco... 3409.0 365.0 3403.0 0.0 0.0 6.0
2 A3 Correspondence files, annual single number ser... 01 Jan 1839 - 23 May 1963 1839-01-01 1963-05-23 26.48 All items from this series are entered on Reco... 1382.0 264.0 1374.0 8.0 0.0 0.0
3 A4 Correspondence files, single number series wit... NaN NaN NaN 0.36 Click to see items listed on RecordSearch. Ple... 45.0 2.0 45.0 0.0 0.0 0.0
4 A5 Correspondence files, annual single number ser... 01 Jan 1923 - 31 Dec 1924 1923-01-01 1924-12-31 1.80 Click to see items listed on RecordSearch. Ple... 200.0 14.0 198.0 0.0 0.0 2.0

Some basic statistics

Note that these numbers might not be exact. To work around the 20,000 search result limit, some totals have been calculated by aggregating a series of searches. In most cases this will be accurate, but some items have multiple control symbols and may be duplicated in the results. I think any errors will be small.

The numbers in brackets indicate the change since the last harvest in May 2021.

Number of series

In [38]:
print(f"{df.shape[0]:,} series ({df.shape[0] - df_prev.shape[0]:+,})")
65,747 series (+28)

Quantity of records in linear metres

In [39]:
print(
    f'{round(df["quantity_total"].sum(), 2):,} metres of records ({round((df["quantity_total"].sum() - df_prev["quantity_total"].sum()), 2):+,})'
)
341,963.73 metres of records (+4,293.04)

Number of items described in RecordSearch

In [40]:
print(
    f'{df["described_total"].sum():,} items described ({df["described_total"].sum() - df_prev["described_total"].sum():+,})'
)
14,624,131 items described (+713,641)

Number of items digitised

In [41]:
print(
    f'{df["digitised_total"].sum():,} items digitised ({df["digitised_total"].sum()- df_prev["digitised_total"].sum():+,})'
)
2,366,101 items digitised (+131,869)
In [42]:
prev_percent = df_prev["digitised_total"].sum() / df_prev["described_total"].sum()
current_percent = df["digitised_total"].sum() / df["described_total"].sum()
print(
    f"{current_percent:0.2%} of described items are digitised ({current_percent - prev_percent:+0.2%})"
)
16.18% of described items are digitised (+0.12%)

Access status of items described

In [44]:
access_totals = [
    {
        "access status": "Open",
        "total": df["access_open_total"].sum(),
        "change": df["access_open_total"].sum() - df_prev["access_open_total"].sum(),
    },
    {
        "access status": "Open with exceptions",
        "total": df["access_owe_total"].sum(),
        "change": df["access_owe_total"].sum() - df_prev["access_owe_total"].sum(),
    },
    {
        "access status": "Closed",
        "total": df["access_closed_total"].sum(),
        "change": df["access_closed_total"].sum()
        - df_prev["access_closed_total"].sum(),
    },
    {
        "access status": "Not yet examined",
        "total": df["access_nye_total"].sum(),
        "change": df["access_nye_total"].sum() - df_prev["access_nye_total"].sum(),
    },
]

df_access = pd.DataFrame(access_totals)
df_access["percent"] = df_access["total"] / df_access["total"].sum()

df_access.style.format(
    {"total": "{:,.0f}", "change": "{:+,}", "percent": "{:0.2%}"}
).hide()
Out[44]:
access status total change percent
Open 8,026,169 +243,348 54.06%
Open with exceptions 110,688 +1,553 0.75%
Closed 10,917 -153 0.07%
Not yet examined 6,698,686 +512,033 45.12%

Digging deeper

How many items are there in total?

There's no way of knowing this from the harvested data. However, the recently-released Tune Review says that 37% of the NAA's holdings are described. So as we know the number described, we should be able to calculate an approximate number of total items.

In [51]:
print(f'Approximately {int(df["described_total"].sum() / 0.37):,} items in total')
Approximately 39,524,678 items in total

To put that another way, this is the approximate number of items not listed on RecordSearch:

In [52]:
print(
    f'Approximately {int(df["described_total"].sum() / 0.37) - df["described_total"].sum():,} items **are not** listed on RecordSearch'
)
Approximately 24,900,547 items **are not** listed on RecordSearch

That's something to keep in mind if you're just relying on item keyword searches to find relevant content.

How much of each series is described at item level?

The note that accompanies the number of items listed in RecordSearch indicates how much of the series has been described at item level. By looking at the frequency of each of the values for this note, we can get a sese of the level of description across the collection.

In [47]:
df_described = df["described_note"].value_counts().to_frame()
df_described.columns = ["total"]
df_described["percent"] = df_described["total"] / df_described["total"].sum()
df_described.style.format({"total": "{:,.0f}", "percent": "{:0.2%}"})
Out[47]:
  total percent
No items from the series are on RecordSearch. Please contact the National Reference Service if you need assistance. 41,584 63.25%
Click to see items listed on RecordSearch. Please contact the National Reference Service if you can't find the record you want as not all items from the series may be on RecordSearch. 12,966 19.72%
All items from this series are entered on RecordSearch. 11,166 16.98%
No items from the series are on RecordSearch. Please contact the Australian War Memorial if you need assistance. 30 0.05%

The numbers above might be a bit misleading because sometimes series are registered on RecordSearch before any items are actually transferred to the NAA. So the reason there are no items listed might be that there are no items currently in Archives custody. To try an get a more accurate picture, we can filter out series where the quantity held by the NAA is equal to zero metres.

In [48]:
df_described_held = (
    df.loc[df["quantity_total"] != 0]["described_note"].value_counts().to_frame()
)
df_described_held.columns = ["total"]
df_described_held["percent"] = (
    df_described_held["total"] / df_described_held["total"].sum()
)
df_described_held.style.format({"total": "{:,.0f}", "percent": "{:0.2%}"})
Out[48]:
  total percent
No items from the series are on RecordSearch. Please contact the National Reference Service if you need assistance. 19,227 47.29%
All items from this series are entered on RecordSearch. 11,044 27.16%
Click to see items listed on RecordSearch. Please contact the National Reference Service if you can't find the record you want as not all items from the series may be on RecordSearch. 10,384 25.54%
No items from the series are on RecordSearch. Please contact the Australian War Memorial if you need assistance. 1 0.00%

This brings down the 'undescribed' proportion, though strangely this seems to indicate that there are zero shelf metres of some series which are fully described.

In [14]:
df.loc[
    (df["described_note"].str.startswith("All")) & (df["quantity_total"] == 0)
].shape[0]
Out[14]:
122

For example:

In [15]:
df.loc[(df["described_note"].str.startswith("All")) & (df["quantity_total"] == 0)].head(
    2
)
Out[15]:
identifier title contents_date_str contents_start_date contents_end_date quantity_total described_note described_total digitised_total access_open_total access_owe_total access_closed_total access_nye_total
121 A123 Name index cards (Departments), 'G' series NaN NaN NaN 0.0 All items from this series are entered on Reco... 2.0 0.0 2.0 0.0 0.0 0.0
742 A749 Volume of Circulars of Public Service Commissi... NaN NaN NaN 0.0 All items from this series are entered on Reco... 1.0 0.0 0.0 0.0 0.0 1.0

So perhaps in some cases locations and quantities are not reliably recorded on RecordSearch.

Series with no item descriptions

From the items described note it seems that 19,227 series held by the NAA or AWM have no item level descriptions. We can check that by simply looking for series where the described_total value is zero.

In [49]:
print(
    f'{df.loc[(df["quantity_total"] > 0) & (df["described_total"] == 0)].shape[0]:,} series held by NAA have no item descriptions'
)
19,228 series held by NAA have no item descriptions

Yay! That (almost) matches.

Boo! That's a pretty significant black hole. Let's look at the quantity of records that represents.

In [50]:
print(
    f'{df.loc[(df["quantity_total"] > 0) & (df["described_total"] == 0)]["quantity_total"].sum():,} linear metres in series held by NAA with no item descriptions'
)
51,733.74 linear metres in series held by NAA with no item descriptions

Of course, this doesn't include the quantities of series that are partially described.


Created by Tim Sherratt for the GLAM Workbench. Support me by becoming a GitHub sponsor!