A (very brief) intro to exploring metadata on the Hugging Face Hub¶
How we can use the
huggingface_hublibrary to explore metadata on the Hugging Face Hub.
- toc: true
- badges: false
- comments: true
- categories: [huggingface, metadata]
- search_exclude: false
- badges: true
- image: https://repository-images.githubusercontent.com/323591830/9ba12f80-ca92-11eb-9da7-8bf28c3bacaf
The Hugging Face Hub has become the de facto place to share machine learning models and datasets.
As the number of models and datasets grow the challenge of finding the right model or dataset for your needs may become more challenging.
There are various ways in which we can try and make it easier for people to find relevant models and datasets.
One of these is by associating metadata with datasets and models.
This blog post will (very briefly) begin to explore metadata on the Hugging Face Hub.
Often you'll want to explore models and datasets via the Hub website but this isn't the only way to explore the Hub.
As part of the process of exploring metadata on the Hugging Face Hub we'll briefly look at how we can use the huggingface_hub library to programmatically interact with the Hub.
Library imports¶
For this post we'll need a few libraries, pandas, requests and matplotlib are likely old friends (or foes...). The huggingface_hub library might be new to you but will soon become a good friend too! The rich library is fantastically useful for quickly getting
familiar with a library (i.e. avoiding reading all the docs!) so we'll import that too.
In [1]:
import requests
from huggingface_hub import hf_api
import pandas as pd
import matplotlib.pyplot as plt
import rich
In [2]:
%matplotlib inline
plt.style.use("ggplot")
We'll instantiate an instance of the HfApi class.
In [3]:
api = hf_api.HfApi()
We can use rich inspect to get a better sense of what a function or class instance is all about. Let's see what methods the api has.
In [4]:
rich.inspect(api, methods=True)
╭──────────────────────────────────── <class 'huggingface_hub.hf_api.HfApi'> ─────────────────────────────────────╮ │ ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ │ │ <huggingface_hub.hf_api.HfApi object at 0x136a2ce80> │ │ │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │ │ │ │ endpoint = 'https://huggingface.co' │ │ token = None │ │ change_discussion_status = def change_discussion_status(repo_id: str, discussion_num: int, new_status: │ │ Literal['open', 'closed'], *, token: Optional[str] = None, comment: Optional[str] = │ │ None, repo_type: Optional[str] = None) -> │ │ huggingface_hub.community.DiscussionStatusChange: Closes or re-opens a Discussion or │ │ Pull Request. │ │ comment_discussion = def comment_discussion(repo_id: str, discussion_num: int, comment: str, *, token: │ │ Optional[str] = None, repo_type: Optional[str] = None) -> │ │ huggingface_hub.community.DiscussionComment: Creates a new comment on the given │ │ Discussion. │ │ create_branch = def create_branch(repo_id: str, *, branch: str, token: Optional[str] = None, │ │ repo_type: Optional[str] = None) -> None: Create a new branch from `main` on a repo │ │ on the Hub. │ │ create_commit = def create_commit(repo_id: str, operations: │ │ Iterable[Union[huggingface_hub._commit_api.CommitOperationAdd, │ │ huggingface_hub._commit_api.CommitOperationDelete]], *, commit_message: str, │ │ commit_description: Optional[str] = None, token: Optional[str] = None, repo_type: │ │ Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = │ │ None, num_threads: int = 5, parent_commit: Optional[str] = None) -> │ │ huggingface_hub.hf_api.CommitInfo: Creates a commit in the given repo, deleting & │ │ uploading files as needed. │ │ create_discussion = def create_discussion(repo_id: str, title: str, *, token: Optional[str] = None, │ │ description: Optional[str] = None, repo_type: Optional[str] = None, pull_request: │ │ bool = False) -> huggingface_hub.community.DiscussionWithDetails: Creates a │ │ Discussion or Pull Request. │ │ create_pull_request = def create_pull_request(repo_id: str, title: str, *, token: Optional[str] = None, │ │ description: Optional[str] = None, repo_type: Optional[str] = None) -> │ │ huggingface_hub.community.DiscussionWithDetails: Creates a Pull Request . Pull │ │ Requests created programmatically will be in `"draft"` status. │ │ create_repo = def create_repo(repo_id: str, *, token: Optional[str] = None, private: bool = False, │ │ repo_type: Optional[str] = None, exist_ok: bool = False, space_sdk: Optional[str] = │ │ None) -> str: Create an empty repo on the HuggingFace Hub. │ │ create_tag = def create_tag(repo_id: str, *, tag: str, tag_message: Optional[str] = None, │ │ revision: Optional[str] = None, token: Optional[str] = None, repo_type: │ │ Optional[str] = None) -> None: Tag a given commit of a repo on the Hub. │ │ dataset_info = def dataset_info(repo_id: str, *, revision: Optional[str] = None, timeout: │ │ Optional[float] = None, files_metadata: bool = False, token: Union[bool, str, │ │ NoneType] = None) -> huggingface_hub.hf_api.DatasetInfo: Get info on one specific │ │ dataset on huggingface.co. │ │ delete_branch = def delete_branch(repo_id: str, *, branch: str, token: Optional[str] = None, │ │ repo_type: Optional[str] = None) -> None: Delete a branch from a repo on the Hub. │ │ delete_file = def delete_file(path_in_repo: str, repo_id: str, *, token: Optional[str] = None, │ │ repo_type: Optional[str] = None, revision: Optional[str] = None, commit_message: │ │ Optional[str] = None, commit_description: Optional[str] = None, create_pr: │ │ Optional[bool] = None, parent_commit: Optional[str] = None) -> │ │ huggingface_hub.hf_api.CommitInfo: Deletes a file in the given repo. │ │ delete_folder = def delete_folder(path_in_repo: str, repo_id: str, *, token: Optional[str] = None, │ │ repo_type: Optional[str] = None, revision: Optional[str] = None, commit_message: │ │ Optional[str] = None, commit_description: Optional[str] = None, create_pr: │ │ Optional[bool] = None, parent_commit: Optional[str] = None) -> │ │ huggingface_hub.hf_api.CommitInfo: Deletes a folder in the given repo. │ │ delete_repo = def delete_repo(repo_id: str, *, token: Optional[str] = None, repo_type: │ │ Optional[str] = None): Delete a repo from the HuggingFace Hub. CAUTION: this is │ │ irreversible. │ │ delete_tag = def delete_tag(repo_id: str, *, tag: str, token: Optional[str] = None, repo_type: │ │ Optional[str] = None) -> None: Delete a tag from a repo on the Hub. │ │ edit_discussion_comment = def edit_discussion_comment(repo_id: str, discussion_num: int, comment_id: str, │ │ new_content: str, *, token: Optional[str] = None, repo_type: Optional[str] = None) │ │ -> huggingface_hub.community.DiscussionComment: Edits a comment on a Discussion / │ │ Pull Request. │ │ get_dataset_tags = def get_dataset_tags() -> huggingface_hub.utils.endpoint_helpers.DatasetTags: Gets │ │ all valid dataset tags as a nested namespace object. │ │ get_discussion_details = def get_discussion_details(repo_id: str, discussion_num: int, *, repo_type: │ │ Optional[str] = None, token: Optional[str] = None) -> │ │ huggingface_hub.community.DiscussionWithDetails: Fetches a Discussion's / Pull │ │ Request 's details from the Hub. │ │ get_full_repo_name = def get_full_repo_name(model_id: str, *, organization: Optional[str] = None, token: │ │ Union[bool, str, NoneType] = None): │ │ Returns the repository name for a given model ID and optional │ │ organization. │ │ get_model_tags = def get_model_tags() -> huggingface_hub.utils.endpoint_helpers.ModelTags: Gets all │ │ valid model tags as a nested namespace object │ │ get_repo_discussions = def get_repo_discussions(repo_id: str, *, repo_type: Optional[str] = None, token: │ │ Optional[str] = None) -> Iterator[huggingface_hub.community.Discussion]: Fetches │ │ Discussions and Pull Requests for the given repo. │ │ hide_discussion_comment = def hide_discussion_comment(repo_id: str, discussion_num: int, comment_id: str, *, │ │ token: Optional[str] = None, repo_type: Optional[str] = None) -> │ │ huggingface_hub.community.DiscussionComment: Hides a comment on a Discussion / Pull │ │ Request. │ │ list_datasets = def list_datasets(*, filter: │ │ Union[huggingface_hub.utils.endpoint_helpers.DatasetFilter, str, Iterable[str], │ │ NoneType] = None, author: Optional[str] = None, search: Optional[str] = None, sort: │ │ Union[Literal['lastModified'], str, NoneType] = None, direction: │ │ Optional[Literal[-1]] = None, limit: Optional[int] = None, cardData: Optional[bool] │ │ = None, full: Optional[bool] = None, token: Optional[str] = None) -> │ │ List[huggingface_hub.hf_api.DatasetInfo]: Get the list of all the datasets on │ │ huggingface.co │ │ list_metrics = def list_metrics() -> List[huggingface_hub.hf_api.MetricInfo]: Get the public list │ │ of all the metrics on huggingface.co │ │ list_models = def list_models(*, filter: Union[huggingface_hub.utils.endpoint_helpers.ModelFilter, │ │ str, Iterable[str], NoneType] = None, author: Optional[str] = None, search: │ │ Optional[str] = None, emissions_thresholds: Optional[Tuple[float, float]] = None, │ │ sort: Union[Literal['lastModified'], str, NoneType] = None, direction: │ │ Optional[Literal[-1]] = None, limit: Optional[int] = None, full: Optional[bool] = │ │ None, cardData: bool = False, fetch_config: bool = False, token: Union[bool, str, │ │ NoneType] = None) -> List[huggingface_hub.hf_api.ModelInfo]: Get the list of all the │ │ models on huggingface.co │ │ list_repo_files = def list_repo_files(repo_id: str, *, revision: Optional[str] = None, repo_type: │ │ Optional[str] = None, timeout: Optional[float] = None, token: Union[bool, str, │ │ NoneType] = None) -> List[str]: Get the list of files in a given repo. │ │ list_spaces = def list_spaces(*, filter: Union[str, Iterable[str], NoneType] = None, author: │ │ Optional[str] = None, search: Optional[str] = None, sort: │ │ Union[Literal['lastModified'], str, NoneType] = None, direction: │ │ Optional[Literal[-1]] = None, limit: Optional[int] = None, datasets: Union[str, │ │ Iterable[str], NoneType] = None, models: Union[str, Iterable[str], NoneType] = None, │ │ linked: bool = False, full: Optional[bool] = None, token: Optional[str] = None) -> │ │ List[huggingface_hub.hf_api.SpaceInfo]: Get the public list of all Spaces on │ │ huggingface.co │ │ merge_pull_request = def merge_pull_request(repo_id: str, discussion_num: int, *, token: Optional[str] = │ │ None, comment: Optional[str] = None, repo_type: Optional[str] = None): Merges a Pull │ │ Request. │ │ model_info = def model_info(repo_id: str, *, revision: Optional[str] = None, timeout: │ │ Optional[float] = None, securityStatus: Optional[bool] = None, files_metadata: bool │ │ = False, token: Union[bool, str, NoneType] = None) -> │ │ huggingface_hub.hf_api.ModelInfo: Get info on one specific model on huggingface.co │ │ move_repo = def move_repo(from_id: str, to_id: str, *, repo_type: Optional[str] = None, token: │ │ Optional[str] = None): Moving a repository from namespace1/repo_name1 to │ │ namespace2/repo_name2 │ │ rename_discussion = def rename_discussion(repo_id: str, discussion_num: int, new_title: str, *, token: │ │ Optional[str] = None, repo_type: Optional[str] = None) -> │ │ huggingface_hub.community.DiscussionTitleChange: Renames a Discussion. │ │ repo_info = def repo_info(repo_id: str, *, revision: Optional[str] = None, repo_type: │ │ Optional[str] = None, timeout: Optional[float] = None, files_metadata: bool = False, │ │ token: Union[bool, str, NoneType] = None) -> Union[huggingface_hub.hf_api.ModelInfo, │ │ huggingface_hub.hf_api.DatasetInfo, huggingface_hub.hf_api.SpaceInfo]: Get the info │ │ object for a given repo of a given type. │ │ set_access_token = def set_access_token(access_token: str): │ │ Saves the passed access token so git can correctly authenticate the │ │ user. │ │ space_info = def space_info(repo_id: str, *, revision: Optional[str] = None, timeout: │ │ Optional[float] = None, files_metadata: bool = False, token: Union[bool, str, │ │ NoneType] = None) -> huggingface_hub.hf_api.SpaceInfo: Get info on one specific │ │ Space on huggingface.co. │ │ unset_access_token = def unset_access_token(): Resets the user's access token. │ │ update_repo_visibility = def update_repo_visibility(repo_id: str, private: bool = False, *, token: │ │ Optional[str] = None, organization: Optional[str] = None, repo_type: Optional[str] = │ │ None, name: Optional[str] = None) -> Dict[str, bool]: Update the visibility setting │ │ of a repository. │ │ upload_file = def upload_file(*, path_or_fileobj: Union[str, bytes, BinaryIO], path_in_repo: str, │ │ repo_id: str, token: Optional[str] = None, repo_type: Optional[str] = None, │ │ revision: Optional[str] = None, commit_message: Optional[str] = None, │ │ commit_description: Optional[str] = None, create_pr: Optional[bool] = None, │ │ parent_commit: Optional[str] = None) -> str: │ │ Upload a local file (up to 50 GB) to the given repo. The upload is done │ │ through a HTTP post request, and doesn't require git or git-lfs to be │ │ installed. │ │ upload_folder = def upload_folder(*, repo_id: str, folder_path: Union[str, pathlib.Path], │ │ path_in_repo: Optional[str] = None, commit_message: Optional[str] = None, │ │ commit_description: Optional[str] = None, token: Optional[str] = None, repo_type: │ │ Optional[str] = None, revision: Optional[str] = None, create_pr: Optional[bool] = │ │ None, parent_commit: Optional[str] = None, allow_patterns: Union[List[str], str, │ │ NoneType] = None, ignore_patterns: Union[List[str], str, NoneType] = None): │ │ Upload a local folder to the given repo. The upload is done │ │ through a HTTP requests, and doesn't require git or git-lfs to be │ │ installed. │ │ whoami = def whoami(token: Optional[str] = None) -> Dict: Call HF API to know "whoami". │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
You'll see from looking through this there is a bunch of different things we can now do programmatically via the hub. For this post we're interested in the list_datasets and list_models methods. If we look at one of these we can see it has a bunch of different options we can use when listing datasets or models.
In [5]:
rich.inspect(api.list_models)
╭─────────── <bound method HfApi.list_models of <huggingface_hub.hf_api.HfApi object at 0x136a2ce80>> ────────────╮ │ def HfApi.list_models(*, filter: Union[huggingface_hub.utils.endpoint_helpers.ModelFilter, str, Iterable[str], │ │ NoneType] = None, author: Optional[str] = None, search: Optional[str] = None, emissions_thresholds: │ │ Optional[Tuple[float, float]] = None, sort: Union[Literal['lastModified'], str, NoneType] = None, direction: │ │ Optional[Literal[-1]] = None, limit: Optional[int] = None, full: Optional[bool] = None, cardData: bool = False, │ │ fetch_config: bool = False, token: Union[bool, str, NoneType] = None) -> │ │ List[huggingface_hub.hf_api.ModelInfo]: │ │ │ │ Get the list of all the models on huggingface.co │ │ │ │ 28 attribute(s) not shown. Run inspect(inspect) for options. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
For our use case we want everything, so we set limit=None, we don't want any filters so we set this to None (this is the default behaviour, but we set them explicitly here to make it clearer for our future selves). We also set full=True so we get back more verbose information about our dataset and models. We also wrap the result in iter and list since the behaviour of these methods will change in future versions to support paging.
In [6]:
hub_datasets = list(iter(api.list_datasets(limit=None, filter=None, full=True)))
In [7]:
hub_models = list(iter(api.list_models(limit=None, filter=None, full=True)))
Let's peek at an example of what we get back
In [8]:
hub_models[0]
Out[8]:
ModelInfo: {
modelId: albert-base-v1
sha: aeffd769076a5c4f83b2546aea99ca45a15a5da4
lastModified: 2021-01-13T15:08:24.000Z
tags: ['pytorch', 'tf', 'albert', 'fill-mask', 'en', 'dataset:bookcorpus', 'dataset:wikipedia', 'arxiv:1909.11942', 'transformers', 'exbert', 'license:apache-2.0', 'autotrain_compatible', 'has_space']
pipeline_tag: fill-mask
siblings: [RepoFile(rfilename='.gitattributes', size='None', blob_id='None', lfs='None'), RepoFile(rfilename='README.md', size='None', blob_id='None', lfs='None'), RepoFile(rfilename='config.json', size='None', blob_id='None', lfs='None'), RepoFile(rfilename='pytorch_model.bin', size='None', blob_id='None', lfs='None'), RepoFile(rfilename='spiece.model', size='None', blob_id='None', lfs='None'), RepoFile(rfilename='tf_model.h5', size='None', blob_id='None', lfs='None'), RepoFile(rfilename='tokenizer.json', size='None', blob_id='None', lfs='None'), RepoFile(rfilename='with-prefix-tf_model.h5', size='None', blob_id='None', lfs='None')]
private: False
author: None
config: None
securityStatus: None
_id: 621ffdc036468d709f174328
id: albert-base-v1
gitalyUid: 4f35551ea371da7a8762caab54319a54ade836044f0ca7690d21e86b159867eb
likes: 1
downloads: 75182
library_name: transformers
}
In [9]:
hub_datasets[0]
Out[9]:
DatasetInfo: {
id: acronym_identification
sha: 173af1516c409eb4596bc63a69626bdb5584c40c
lastModified: 2022-11-18T17:25:49.000Z
tags: ['task_categories:token-classification', 'annotations_creators:expert-generated', 'language_creators:found', 'multilinguality:monolingual', 'size_categories:10K<n<100K', 'source_datasets:original', 'language:en', 'license:mit', 'acronym-identification', 'arxiv:2010.14678']
private: False
author: None
description: Acronym identification training and development sets for the acronym identification task at SDU@AAAI-21.
citation: @inproceedings{veyseh-et-al-2020-what,
title={{What Does This Acronym Mean? Introducing a New Dataset for Acronym Identification and Disambiguation}},
author={Amir Pouran Ben Veyseh and Franck Dernoncourt and Quan Hung Tran and Thien Huu Nguyen},
year={2020},
booktitle={Proceedings of COLING},
link={https://arxiv.org/pdf/2010.14678v1.pdf}
}
cardData: {'annotations_creators': ['expert-generated'], 'language_creators': ['found'], 'language': ['en'], 'license': ['mit'], 'multilinguality': ['monolingual'], 'size_categories': ['10K<n<100K'], 'source_datasets': ['original'], 'task_categories': ['token-classification'], 'task_ids': [], 'paperswithcode_id': 'acronym-identification', 'pretty_name': 'Acronym Identification Dataset', 'train-eval-index': [{'config': 'default', 'task': 'token-classification', 'task_id': 'entity_extraction', 'splits': {'eval_split': 'test'}, 'col_mapping': {'tokens': 'tokens', 'labels': 'tags'}}], 'tags': ['acronym-identification'], 'dataset_info': {'features': [{'name': 'id', 'dtype': 'string'}, {'name': 'tokens', 'sequence': 'string'}, {'name': 'labels', 'sequence': {'class_label': {'names': {'0': 'B-long', '1': 'B-short', '2': 'I-long', '3': 'I-short', '4': 'O'}}}}], 'splits': [{'name': 'train', 'num_bytes': 7792803, 'num_examples': 14006}, {'name': 'validation', 'num_bytes': 952705, 'num_examples': 1717}, {'name': 'test', 'num_bytes': 987728, 'num_examples': 1750}], 'download_size': 8556464, 'dataset_size': 9733236}}
siblings: []
_id: 621ffdd236468d709f181d58
disabled: False
gated: False
gitalyUid: 6570517623fa521aa189178e7c7e73d9d88c01b295204edef97f389a15eae144
likes: 9
downloads: 6074
paperswithcode_id: acronym-identification
}
Since we want both models and datasets we create a dictionary which stores the types of item i.e. is it a dataset or a model.
In [10]:
hub_data = {"model": hub_models, "dataset": hub_datasets}
We'll be putting our data inside a pandas DataFrame, so we'll grab the .__dict__ attribute for each hub item, so it's more pandas friendly.
In [11]:
hub_item_dict = []
for hub_type, hub_item in hub_data.items():
for item in hub_item:
data = item.__dict__
data["type"] = hub_type
hub_item_dict.append(data)
In [12]:
df = pd.DataFrame.from_dict(hub_item_dict)
How many hub items do we have?
In [13]:
len(df)
Out[13]:
151343
What info do we have?
In [14]:
df.columns
Out[14]:
Index(['modelId', 'sha', 'lastModified', 'tags', 'pipeline_tag', 'siblings',
'private', 'author', 'config', 'securityStatus', '_id', 'id',
'gitalyUid', 'likes', 'downloads', 'library_name', 'type',
'description', 'citation', 'cardData', 'disabled', 'gated',
'paperswithcode_id'],
dtype='object')
Tags¶
Models and datasets have a bunch of metadata i.e. last modified and number of downloads. We'll focus on tags here. Let's start by looking at a single example.
In [15]:
df.loc[30, "tags"]
Out[15]:
['pytorch', 'tf', 'rust', 'safetensors', 'distilbert', 'text-classification', 'en', 'dataset:sst2', 'dataset:glue', 'doi:10.57967/hf/0181', 'transformers', 'license:apache-2.0', 'model-index', 'has_space']
We can see that tags capture can relate to tasks i.e. text-classification, libraries supported i.e. tf, or the licence associated with a model or dataset. As a starting point for exploring tags we can take a look at how many tags models and datasets have. We'll add a new column to capture this number.
In [16]:
def calculate_number_of_tags(tags: [str]) -> int:
return len(tags)
In [17]:
df["number_of_tags"] = df["tags"].apply(lambda x: calculate_number_of_tags(x))
We can now use describe to see the breakdown of this number.
In [113]:
df.number_of_tags.describe()
Out[113]:
count 151343.000000 mean 3.855566 std 6.878613 min 0.000000 25% 0.000000 50% 4.000000 75% 6.000000 max 650.000000 Name: number_of_tags, dtype: float64
We can see that we have quite a range of tag numbers ranging from 0 to 650! If your brain works anything like mine you probably want to know what this high value is about!
In [19]:
df[df.number_of_tags > 640][["id", "tags"]]
Out[19]:
| id | tags | |
|---|---|---|
| 136372 | bible-nlp/biblenlp-corpus | [task_categories:translation, annotations_crea... |
In [20]:
df[df.number_of_tags > 640]["tags"].tolist()
Out[20]:
[['task_categories:translation', 'annotations_creators:no-annotation', 'language_creators:expert-generated', 'multilinguality:translation', 'multilinguality:multilingual', 'size_categories:1M<n<10M', 'source_datasets:original', 'language:aau', 'language:aaz', 'language:abx', 'language:aby', 'language:acf', 'language:acu', 'language:adz', 'language:aey', 'language:agd', 'language:agg', 'language:agm', 'language:agn', 'language:agr', 'language:agu', 'language:aia', 'language:ake', 'language:alp', 'language:alq', 'language:als', 'language:aly', 'language:ame', 'language:amk', 'language:amp', 'language:amr', 'language:amu', 'language:anh', 'language:anv', 'language:aoi', 'language:aoj', 'language:apb', 'language:apn', 'language:apu', 'language:apy', 'language:arb', 'language:arl', 'language:arn', 'language:arp', 'language:aso', 'language:ata', 'language:atb', 'language:atd', 'language:atg', 'language:auc', 'language:aui', 'language:auy', 'language:avt', 'language:awb', 'language:awk', 'language:awx', 'language:azg', 'language:azz', 'language:bao', 'language:bbb', 'language:bbr', 'language:bch', 'language:bco', 'language:bdd', 'language:bea', 'language:bel', 'language:bgs', 'language:bgt', 'language:bhg', 'language:bhl', 'language:big', 'language:bjr', 'language:bjv', 'language:bkd', 'language:bki', 'language:bkq', 'language:bkx', 'language:bla', 'language:blw', 'language:blz', 'language:bmh', 'language:bmk', 'language:bmr', 'language:bnp', 'language:boa', 'language:boj', 'language:bon', 'language:box', 'language:bqc', 'language:bre', 'language:bsn', 'language:bsp', 'language:bss', 'language:buk', 'language:bus', 'language:bvr', 'language:bxh', 'language:byx', 'language:bzd', 'language:bzj', 'language:cab', 'language:caf', 'language:cao', 'language:cap', 'language:car', 'language:cav', 'language:cax', 'language:cbc', 'language:cbi', 'language:cbk', 'language:cbr', 'language:cbs', 'language:cbt', 'language:cbu', 'language:cbv', 'language:cco', 'language:ces', 'language:cgc', 'language:cha', 'language:chd', 'language:chf', 'language:chk', 'language:chq', 'language:chz', 'language:cjo', 'language:cjv', 'language:cle', 'language:clu', 'language:cme', 'language:cmn', 'language:cni', 'language:cnl', 'language:cnt', 'language:cof', 'language:con', 'language:cop', 'language:cot', 'language:cpa', 'language:cpb', 'language:cpc', 'language:cpu', 'language:crn', 'language:crx', 'language:cso', 'language:cta', 'language:ctp', 'language:ctu', 'language:cub', 'language:cuc', 'language:cui', 'language:cut', 'language:cux', 'language:cwe', 'language:daa', 'language:dad', 'language:dah', 'language:ded', 'language:deu', 'language:dgr', 'language:dgz', 'language:dif', 'language:dik', 'language:dji', 'language:djk', 'language:dob', 'language:dwr', 'language:dww', 'language:dwy', 'language:eko', 'language:emi', 'language:emp', 'language:eng', 'language:epo', 'language:eri', 'language:ese', 'language:etr', 'language:faa', 'language:fai', 'language:far', 'language:for', 'language:fra', 'language:fuf', 'language:gai', 'language:gam', 'language:gaw', 'language:gdn', 'language:gdr', 'language:geb', 'language:gfk', 'language:ghs', 'language:gia', 'language:glk', 'language:gmv', 'language:gng', 'language:gnn', 'language:gnw', 'language:gof', 'language:grc', 'language:gub', 'language:guh', 'language:gui', 'language:gul', 'language:gum', 'language:guo', 'language:gvc', 'language:gvf', 'language:gwi', 'language:gym', 'language:gyr', 'language:hat', 'language:haw', 'language:hbo', 'language:hch', 'language:heb', 'language:heg', 'language:hix', 'language:hla', 'language:hlt', 'language:hns', 'language:hop', 'language:hrv', 'language:hub', 'language:hui', 'language:hus', 'language:huu', 'language:huv', 'language:hvn', 'language:ign', 'language:ikk', 'language:ikw', 'language:imo', 'language:inb', 'language:ind', 'language:ino', 'language:iou', 'language:ipi', 'language:ita', 'language:jac', 'language:jao', 'language:jic', 'language:jiv', 'language:jpn', 'language:jvn', 'language:kaq', 'language:kbc', 'language:kbh', 'language:kbm', 'language:kdc', 'language:kde', 'language:kdl', 'language:kek', 'language:ken', 'language:kew', 'language:kgk', 'language:kgp', 'language:khs', 'language:kje', 'language:kjs', 'language:kkc', 'language:kky', 'language:klt', 'language:klv', 'language:kms', 'language:kmu', 'language:kne', 'language:knf', 'language:knj', 'language:kos', 'language:kpf', 'language:kpg', 'language:kpj', 'language:kpw', 'language:kqa', 'language:kqc', 'language:kqf', 'language:kql', 'language:kqw', 'language:ksj', 'language:ksr', 'language:ktm', 'language:kto', 'language:kud', 'language:kue', 'language:kup', 'language:kvn', 'language:kwd', 'language:kwf', 'language:kwi', 'language:kwj', 'language:kyf', 'language:kyg', 'language:kyq', 'language:kyz', 'language:kze', 'language:lac', 'language:lat', 'language:lbb', 'language:leu', 'language:lex', 'language:lgl', 'language:lid', 'language:lif', 'language:lww', 'language:maa', 'language:maj', 'language:maq', 'language:mau', 'language:mav', 'language:maz', 'language:mbb', 'language:mbc', 'language:mbh', 'language:mbl', 'language:mbt', 'language:mca', 'language:mcb', 'language:mcd', 'language:mcf', 'language:mcp', 'language:mdy', 'language:med', 'language:mee', 'language:mek', 'language:meq', 'language:met', 'language:meu', 'language:mgh', 'language:mgw', 'language:mhl', 'language:mib', 'language:mic', 'language:mie', 'language:mig', 'language:mih', 'language:mil', 'language:mio', 'language:mir', 'language:mit', 'language:miz', 'language:mjc', 'language:mkn', 'language:mks', 'language:mlh', 'language:mlp', 'language:mmx', 'language:mna', 'language:mop', 'language:mox', 'language:mph', 'language:mpj', 'language:mpm', 'language:mpp', 'language:mps', 'language:mpx', 'language:mqb', 'language:mqj', 'language:msb', 'language:msc', 'language:msk', 'language:msm', 'language:msy', 'language:mti', 'language:muy', 'language:mva', 'language:mvn', 'language:mwc', 'language:mxb', 'language:mxp', 'language:mxq', 'language:mxt', 'language:myu', 'language:myw', 'language:myy', 'language:mzz', 'language:nab', 'language:naf', 'language:nak', 'language:nay', 'language:nbq', 'language:nca', 'language:nch', 'language:ncj', 'language:ncl', 'language:ncu', 'language:ndj', 'language:nfa', 'language:ngp', 'language:ngu', 'language:nhg', 'language:nhi', 'language:nho', 'language:nhr', 'language:nhu', 'language:nhw', 'language:nhy', 'language:nif', 'language:nin', 'language:nko', 'language:nld', 'language:nlg', 'language:nna', 'language:nnq', 'language:not', 'language:nou', 'language:npl', 'language:nsn', 'language:nss', 'language:ntj', 'language:ntp', 'language:nwi', 'language:nyu', 'language:obo', 'language:ong', 'language:ons', 'language:ood', 'language:opm', 'language:ote', 'language:otm', 'language:otn', 'language:otq', 'language:ots', 'language:pab', 'language:pad', 'language:pah', 'language:pao', 'language:pes', 'language:pib', 'language:pio', 'language:pir', 'language:pjt', 'language:plu', 'language:pma', 'language:poe', 'language:poi', 'language:pon', 'language:poy', 'language:ppo', 'language:prf', 'language:pri', 'language:ptp', 'language:ptu', 'language:pwg', 'language:quc', 'language:quf', 'language:quh', 'language:qul', 'language:qup', 'language:qvc', 'language:qve', 'language:qvh', 'language:qvm', 'language:qvn', 'language:qvs', 'language:qvw', 'language:qvz', 'language:qwh', 'language:qxh', 'language:qxn', 'language:qxo', 'language:rai', 'language:rkb', 'language:rmc', 'language:roo', 'language:rop', 'language:rro', 'language:ruf', 'language:rug', 'language:rus', 'language:sab', 'language:san', 'language:sbe', 'language:seh', 'language:sey', 'language:sgz', 'language:shj', 'language:shp', 'language:sim', 'language:sja', 'language:sll', 'language:smk', 'language:snc', 'language:snn', 'language:sny', 'language:som', 'language:soq', 'language:spa', 'language:spl', 'language:spm', 'language:sps', 'language:spy', 'language:sri', 'language:srm', 'language:srn', 'language:srp', 'language:srq', 'language:ssd', 'language:ssg', 'language:ssx', 'language:stp', 'language:sua', 'language:sue', 'language:sus', 'language:suz', 'language:swe', 'language:swh', 'language:swp', 'language:sxb', 'language:tac', 'language:tav', 'language:tbc', 'language:tbl', 'language:tbo', 'language:tbz', 'language:tca', 'language:tee', 'language:ter', 'language:tew', 'language:tfr', 'language:tgp', 'language:tif', 'language:tim', 'language:tiy', 'language:tke', 'language:tku', 'language:tna', 'language:tnc', 'language:tnn', 'language:tnp', 'language:toc', 'language:tod', 'language:toj', 'language:ton', 'language:too', 'language:top', 'language:tos', 'language:tpt', 'language:trc', 'language:tsw', 'language:ttc', 'language:tue', 'language:tuo', 'language:txu', 'language:ubr', 'language:udu', 'language:ukr', 'language:uli', 'language:ura', 'language:urb', 'language:usa', 'language:usp', 'language:uvl', 'language:vid', 'language:vie', 'language:viv', 'language:vmy', 'language:waj', 'language:wal', 'language:wap', 'language:wat', 'language:wbp', 'language:wed', 'language:wer', 'language:wim', 'language:wmt', 'language:wmw', 'language:wnc', 'language:wnu', 'language:wos', 'language:wrk', 'language:wro', 'language:wsk', 'language:wuv', 'language:xav', 'language:xed', 'language:xla', 'language:xnn', 'language:xon', 'language:xsi', 'language:xtd', 'language:xtm', 'language:yaa', 'language:yad', 'language:yal', 'language:yap', 'language:yaq', 'language:yby', 'language:ycn', 'language:yka', 'language:yml', 'language:yre', 'language:yuj', 'language:yut', 'language:yuw', 'language:yva', 'language:zaa', 'language:zab', 'language:zac', 'language:zad', 'language:zai', 'language:zaj', 'language:zam', 'language:zao', 'language:zar', 'language:zas', 'language:zat', 'language:zav', 'language:zaw', 'language:zca', 'language:zia', 'language:ziw', 'language:zos', 'language:zpc', 'language:zpl', 'language:zpo', 'language:zpq', 'language:zpu', 'language:zpv', 'language:zpz', 'language:zsr', 'language:ztq', 'language:zty', 'language:zyp', 'language:be', 'language:br', 'language:cs', 'language:ch', 'language:zh', 'language:de', 'language:en', 'language:eo', 'language:fr', 'language:ht', 'language:he', 'language:hr', 'language:id', 'language:it', 'language:ja', 'language:la', 'language:nl', 'language:ru', 'language:sa', 'language:so', 'language:es', 'language:sr', 'language:sv', 'language:to', 'language:uk', 'language:vi', 'license:cc-by-4.0', 'license:other']]
We can see that in this case many of the tags relate to language. Since the dataset is bible related and the bible has been heavily translated this might not be as surprising.
Although these high-level stats are somewhat interesting we probably want to break these numbers down. At a high level we can groupby datasets vs models.
In [21]:
df.groupby("type")["number_of_tags"].describe()
Out[21]:
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| type | ||||||||
| dataset | 19576.0 | 2.46935 | 13.137220 | 0.0 | 0.0 | 0.0 | 2.0 | 650.0 |
| model | 131767.0 | 4.06151 | 5.327066 | 0.0 | 0.0 | 4.0 | 6.0 | 413.0 |
We can see that the mean number of tags for models is higher than datasets. We can also see at the 75% percentile models also have more tags compared to datasets. The possible reasons for this (and whether this is a problem or not) is something we may wish to explore further...
Since the hub hosts models from different libraries we may want to also breakdown by library. First let's grab only the model part of our DataFrame.
In [22]:
models_df = df[df["type"] == "model"]
The library_name column contains info about the library. Let's see how many unique libraries we have.
In [23]:
models_df.library_name.unique().shape
Out[23]:
(63,)
This is quite a few! We can do a groupby on this column
In [24]:
models_df.groupby("library_name")["number_of_tags"].describe()
Out[24]:
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| library_name | ||||||||
| BERT | 1.0 | 7.0 | NaN | 7.0 | 7.0 | 7.0 | 7.0 | 7.0 |
| Doc-UFCN | 2.0 | 4.0 | 0.000000 | 4.0 | 4.0 | 4.0 | 4.0 | 4.0 |
| EveryDream | 2.0 | 7.0 | 0.000000 | 7.0 | 7.0 | 7.0 | 7.0 | 7.0 |
| FastAI | 1.0 | 1.0 | NaN | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 |
| JoeyNMT | 1.0 | 4.0 | NaN | 4.0 | 4.0 | 4.0 | 4.0 | 4.0 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| ultralytics | 4.0 | 10.0 | 1.414214 | 8.0 | 9.5 | 10.5 | 11.0 | 11.0 |
| ultralyticsplus | 1.0 | 9.0 | NaN | 9.0 | 9.0 | 9.0 | 9.0 | 9.0 |
| yolor | 2.0 | 9.0 | 0.000000 | 9.0 | 9.0 | 9.0 | 9.0 | 9.0 |
| yolov5 | 36.0 | 9.0 | 0.000000 | 9.0 | 9.0 | 9.0 | 9.0 | 9.0 |
| yolov6detect | 1.0 | 10.0 | NaN | 10.0 | 10.0 | 10.0 | 10.0 | 10.0 |
62 rows × 8 columns
We might find this a bit tricky to look at. We may want to only include the top n libraries since some of these libraries may be less well used.
In [25]:
models_df.library_name.value_counts()[:15]
Out[25]:
transformers 63754 stable-baselines3 3183 diffusers 2802 sentence-transformers 1273 ml-agents 763 keras 470 timm 383 espnet 381 spacy 296 sample-factory 273 adapter-transformers 201 sklearn 113 nemo 103 fastai 99 speechbrain 94 Name: library_name, dtype: int64
In [26]:
top_libraries = models_df.library_name.value_counts()[:9].index.to_list()
In [27]:
top_libraries_df = models_df[models_df.library_name.isin(top_libraries)]
In [28]:
top_libraries_df.groupby("library_name")["number_of_tags"].describe()
Out[28]:
| count | mean | std | min | 25% | 50% | 75% | max | |
|---|---|---|---|---|---|---|---|---|
| library_name | ||||||||
| diffusers | 2802.0 | 4.374732 | 2.171226 | 1.0 | 3.0 | 4.0 | 5.0 | 18.0 |
| espnet | 381.0 | 6.965879 | 0.595060 | 3.0 | 7.0 | 7.0 | 7.0 | 9.0 |
| keras | 470.0 | 3.842553 | 14.422674 | 1.0 | 1.0 | 2.0 | 5.0 | 311.0 |
| ml-agents | 763.0 | 6.965924 | 0.273775 | 2.0 | 7.0 | 7.0 | 7.0 | 7.0 |
| sentence-transformers | 1273.0 | 6.984289 | 3.221840 | 2.0 | 6.0 | 6.0 | 7.0 | 36.0 |
| spacy | 296.0 | 4.611486 | 0.985180 | 2.0 | 4.0 | 5.0 | 5.0 | 10.0 |
| stable-baselines3 | 3183.0 | 4.997801 | 0.163426 | 3.0 | 5.0 | 5.0 | 5.0 | 8.0 |
| timm | 383.0 | 3.548303 | 1.315291 | 2.0 | 3.0 | 3.0 | 3.0 | 13.0 |
| transformers | 63754.0 | 6.912037 | 5.262633 | 1.0 | 5.0 | 6.0 | 8.0 | 240.0 |
Let's take a quick look at some examples from the library with the highest and lowest number or tags.
In [29]:
top_libraries_df[top_libraries_df.library_name == "sentence-transformers"].sample(15)[
"tags"
]
Out[29]:
6123 [pytorch, gpt_neo, arxiv:2202.08904, sentence-... 2488 [pytorch, distilbert, sentence-transformers, f... 37669 [pytorch, distilbert, sentence-transformers, f... 71483 [pytorch, bert, sentence-transformers, feature... 20710 [pytorch, tf, roberta, ko, sentence-transforme... 27073 [pytorch, tf, jax, roberta, arxiv:1908.10084, ... 92037 [pytorch, mpnet, sentence-transformers, featur... 90320 [pytorch, mpnet, sentence-transformers, featur... 63555 [pytorch, bert, sentence-transformers, feature... 87707 [pytorch, mpnet, sentence-transformers, featur... 80570 [pytorch, bert, sentence-transformers, feature... 111407 [pytorch, bert, sentence-transformers, feature... 82690 [pytorch, mpnet, sentence-transformers, featur... 36217 [pytorch, bert, pl, dataset:Wikipedia, arxiv:1... 100086 [pytorch, roberta, sentence-transformers, feat... Name: tags, dtype: object
In [30]:
top_libraries_df[top_libraries_df.library_name == "timm"].sample(15)["tags"]
Out[30]:
104432 [pytorch, timm, image-classification] 110296 [pytorch, arxiv:2301.00808, timm, image-classi... 24158 [pytorch, timm, image-classification] 26471 [pytorch, timm, image-classification] 104437 [pytorch, timm, image-classification] 61630 [pytorch, dataset:beans, timm, image-classific... 110298 [pytorch, arxiv:2301.00808, timm, image-classi... 104015 [pytorch, timm, image-classification] 101124 [pytorch, timm, image-classification] 57882 [coreml, onnx, en, dataset:imagenet-1k, arxiv:... 83459 [pytorch, timm, image-classification, vision, ... 99461 [pytorch, timm, image-classification] 104029 [pytorch, timm, image-classification] 84402 [pytorch, timm, image-classification, vision, ... 104428 [pytorch, timm, image-classification] Name: tags, dtype: object
We can see here that some tags for sentence-transformers are very closely tied to that libraries purpose e.g. the sentence-similarity tag. This tag migth be useful when a user is looking for models to do sentence-similarity but might be less useful if you are trying to choose between models for this task i.e. trying to find the setence-transformer model that will be useful for you. We should be careful, therefore, in treating number of tags as a proxy for quality.
Grouping by pipeline tags¶
We have a column in our dataframe pipeline tag, which refers to the type of task a model is for. We should be careful relying too much on this but we can have a quick look at how often these are used.
In [31]:
models_df["pipeline_tag"].value_counts()
Out[31]:
text-classification 14479 text2text-generation 8102 text-generation 7602 reinforcement-learning 6885 token-classification 6386 automatic-speech-recognition 6238 fill-mask 5447 question-answering 3147 feature-extraction 2661 translation 1837 conversational 1770 image-classification 1760 text-to-image 1604 sentence-similarity 1248 summarization 735 unconditional-image-generation 428 text-to-speech 244 audio-classification 234 multiple-choice 169 object-detection 158 image-segmentation 134 audio-to-audio 130 tabular-classification 97 zero-shot-classification 97 image-to-text 76 zero-shot-image-classification 56 video-classification 50 table-question-answering 47 tabular-regression 44 image-to-image 43 depth-estimation 37 document-question-answering 18 visual-question-answering 13 voice-activity-detection 6 other 4 time-series-forecasting 1 Name: pipeline_tag, dtype: int64
We may also want to see if there are some type of task that have more tags.
In [32]:
models_df.groupby("pipeline_tag")["number_of_tags"].mean().sort_values().plot.barh()
Out[32]:
<AxesSubplot: ylabel='pipeline_tag'>
We can also look at the breakdown for a particular task
In [98]:
text_classification_df = models_df[models_df["pipeline_tag"] == "text-classification"]
In [99]:
text_classification_df["number_of_tags"].describe()
Out[99]:
count 14479.000000 mean 5.948822 std 3.718800 min 1.000000 25% 4.000000 50% 5.000000 75% 7.000000 max 240.000000 Name: number_of_tags, dtype: float64
Again, we have some extreme outliers
In [100]:
text_classification_df[text_classification_df.number_of_tags > 230][["tags", "modelId"]]
Out[100]:
| tags | modelId | |
|---|---|---|
| 22457 | [pytorch, tf, roberta, text-classification, mu... | m3hrdadfi/zabanshenas-roberta-base-mix |
| 101628 | [pytorch, canine, text-classification, ace, af... | SebOchs/canine-c-lang-id |
We see that these mostly seem to relate to language. Let's remove these outliers and look at the distribution in the number of tags without these.
In [114]:
text_classification_df_no_outliers = text_classification_df[
text_classification_df["number_of_tags"]
<= text_classification_df["number_of_tags"].quantile(0.95)
]
text_classification_df_no_outliers["number_of_tags"].plot.hist(bins=9)
Out[114]:
<AxesSubplot: ylabel='Frequency'>
Why counting tags might not make sense¶
I'e already hinted at why looking at raw number of tags might not be a good idea. Let's close this blog by briefly digging into at least one reason why. We'll use the toolz library for some of this analysis.
In [102]:
from toolz import concat
First we grab all the tags and put them in a single list.
In [103]:
all_tags = list(concat(df.tags.tolist()))
If we look at some examples, we'll see some tags are in the form of something:somethingelse.
In [104]:
all_tags[:10]
Out[104]:
['pytorch', 'tf', 'albert', 'fill-mask', 'en', 'dataset:bookcorpus', 'dataset:wikipedia', 'arxiv:1909.11942', 'transformers', 'exbert']
for example dataset:wikipedia, we should therefore avoid treating all tags as the same since tags can have a particular purpose. i.e. indicating a dataset is associated with a model.
In [105]:
def is_special_tag(tag: str):
return ":" in tag
In [106]:
from toolz import countby, valmap
In [107]:
special_tag_vs_normal = countby(is_special_tag, all_tags)
In [108]:
special_tag_vs_normal
Out[108]:
{False: 467758, True: 115755}
In [109]:
total = sum(special_tag_vs_normal.values())
valmap(lambda x: x / total, special_tag_vs_normal)
Out[109]:
{False: 0.8016239569641121, True: 0.1983760430358878}
We can see that a good chunk of tags are 'special' tags. i.e. they have a 'type' associated with them. If we want to explore tags on the hub more carefully we'll need to take this into account...