Analyzing Hugging Face Datasets¶

This notebook shows how you can use fastdup to analyze any dataset from Hugging Face Datasets.

We will analyze an image classification dataset for:

Duplicates / near-duplicates
Outliers
Wrong labels

Installation¶

In [ ]:

!pip install -Uq fastdup datasets

Now, test the installation. If there's no error message, we are ready to go.

In [16]:

import fastdup
fastdup.__version__

Out[16]:

'1.33'

Load Dataset¶

In this example we load the Tiny ImageNet dataset from Hugging Face Datasets..

Tiny ImageNet contains 100,000 images of 200 classes (500 for each class) downsized to 64×64 colored images. Each class has 500 training images, 50 validation images, and 50 test images.

Let's load the dataset into our local directory.

In [17]:

import datasets
dataset = datasets.load_dataset("zh-plus/tiny-imagenet", split="train", cache_dir='images_dir')

We can inspect the dataset object.

In [18]:

dataset

Out[18]:

Dataset({
    features: ['image', 'label'],
    num_rows: 100000
})

In [19]:

dataset[0]

Out[19]:

{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=64x64 at 0x7F01B3C34AF0>,
 'label': 0}

In [20]:

dataset[0]['image']

Out[20]:

In [21]:

dataset[0]['label']

Out[21]:

Get Labels Mapping¶

Tiny ImageNet follows the original ImageNet class names. Let's download the class mappings classes.py.

In [22]:

!wget -q https://huggingface.co/datasets/zh-plus/tiny-imagenet/raw/main/classes.py

Here's the top 10 lines of classes.py.

In [23]:

!head -n 10 classes.py

i2d = {
  "n00001740": "entity",
  "n00001930": "physical entity",
  "n00002137": "abstraction, abstract entity",
  "n00002452": "thing",
  "n00002684": "object, physical object",
  "n00003553": "whole, unit",
  "n00003993": "congener",
  "n00004258": "living thing, animate thing",
  "n00004475": "organism, being",

Now we can get the class names by providing the class ID. For example:

In [24]:

from classes import i2d

In [25]:

i2d["n00001740"]

Out[25]:

'entity'

Save Images to Disk¶

The images are downloaded in a parquet format. Let's save them into the local disk.

In [36]:

import os
import re
from tqdm.auto import tqdm

for idx, item in tqdm(enumerate(dataset), total=len(dataset)):        
    # extract the image and label
    image = item['image']
    label = i2d[dataset.features['label'].int2str(item['label'])]

    # remove apostrophes
    label = label.replace("'", "")

    # replace commas with underscores
    label = label.replace(", ", "_")

    # replace spaces with dashes
    label = label.replace(" ", "-")

    # create a directory for the class if it doesn't exist
    label_dir = os.path.join('images_dir/images', str(label))
    os.makedirs(label_dir, exist_ok=True)

    # save the image to the appropriate directory
    image.save(os.path.join(label_dir, f'{idx}.jpg'))

  0%|          | 0/100000 [00:00<?, ?it/s]

Load Annotations¶

To load the image labels into fastdup we need to prepare a DataFrame with the following column:

filename
label
split

In [37]:

import os
import pandas as pd

path = 'images_dir/images'

filenames = []
labels = []

for label in os.listdir(path):
    label_path = os.path.join(path, label)
    if os.path.isdir(label_path):
        for filename in os.listdir(label_path):
            filenames.append(os.path.join(label_path, filename))
            labels.append(label)

df = pd.DataFrame({
    'filename': filenames,
    'label': labels
})

df["split"] = "train"

df

Out[37]:

	filename	label	split
0	images_dir/images/swimming-trunks_bathing-trunks/73748.jpg	swimming-trunks_bathing-trunks	train
1	images_dir/images/swimming-trunks_bathing-trunks/73509.jpg	swimming-trunks_bathing-trunks	train
2	images_dir/images/swimming-trunks_bathing-trunks/73946.jpg	swimming-trunks_bathing-trunks	train
3	images_dir/images/swimming-trunks_bathing-trunks/73669.jpg	swimming-trunks_bathing-trunks	train
4	images_dir/images/swimming-trunks_bathing-trunks/73733.jpg	swimming-trunks_bathing-trunks	train
...	...	...	...
99995	images_dir/images/Yorkshire-terrier/11304.jpg	Yorkshire-terrier	train
99996	images_dir/images/Yorkshire-terrier/11244.jpg	Yorkshire-terrier	train
99997	images_dir/images/Yorkshire-terrier/11439.jpg	Yorkshire-terrier	train
99998	images_dir/images/Yorkshire-terrier/11290.jpg	Yorkshire-terrier	train
99999	images_dir/images/Yorkshire-terrier/11490.jpg	Yorkshire-terrier	train

100000 rows × 3 columns

Run fastdup¶

In [38]:

fd = fastdup.create(input_dir='images_dir/')
fd.run(annotations=df, overwrite=True)

Warning: fastdup create() without work_dir argument, output is stored in a folder named work_dir in your current working path.
FastDup Software, (C) copyright 2022 Dr. Amir Alush and Dr. Danny Bickson.
2023-08-07 20:18:12 [INFO] Going to loop over dir /tmp/tmphta0o4og.csv
2023-08-07 20:18:12 [INFO] Found total 100000 images to run on, 100000 train, 0 test, name list 100000, counter 100000 
2023-08-07 20:21:51 [INFO] Found total 100000 images to run onmated: 0 Minutes
Finished histogram 24.316
Finished bucket sort 24.500
2023-08-07 20:22:11 [INFO] 20350) Finished write_index() NN model
2023-08-07 20:22:11 [INFO] Stored nn model index file work_dir/nnf.index
2023-08-07 20:22:22 [INFO] Total time took 249230 ms
2023-08-07 20:22:22 [INFO] Found a total of 40 fully identical images (d>0.990), which are 0.02 %
2023-08-07 20:22:22 [INFO] Found a total of 0 nearly identical images(d>0.980), which are 0.00 %
2023-08-07 20:22:22 [INFO] Found a total of 10829 above threshold images (d>0.900), which are 5.41 %
2023-08-07 20:22:22 [INFO] Found a total of 10001 outlier images         (d<0.050), which are 5.00 %
2023-08-07 20:22:22 [INFO] Min distance found 0.594 max distance 1.000
2023-08-07 20:22:22 [INFO] Running connected components for ccthreshold 0.960000 
.0
 ########################################################################################

Dataset Analysis Summary: 

    Dataset contains 100000 images
    Valid images are 88.00% (100,000) of the data, invalid are 0.00% (0) of the data
    Similarity:  0.04% (46) belong to 1 similarity clusters (components).
    87.96% (99,954) images do not belong to any similarity cluster.
    Largest cluster has 4 (0.00%) images.
    For a detailed analysis, use `.connected_components()`
(similarity threshold used is 0.9, connected component threshold used is 0.96).

    Outliers: 5.58% (6,345) of images are possible outliers, and fall in the bottom 5.00% of similarity values.
    For a detailed list of outliers, use `.outliers()`.

########################################################################################
Would you like to see awesome visualizations for some of the most popular academic datasets?
Click here to see and learn more: https://app.visual-layer.com/vl-datasets?utm_source=fastdup
########################################################################################

Out[38]:

Inspect Issues¶

There are several methods we can use to inspect the issues found:

fd.vis.duplicates_gallery()    # create a visual gallery of duplicates
fd.vis.outliers_gallery()      # create a visual gallery of anomalies
fd.vis.component_gallery()     # create a visualization of connected components
fd.vis.stats_gallery()         # create a visualization of images statistics (e.g. blur)
fd.vis.similarity_gallery()    # create a gallery of similar images

In [39]:

fd.vis.duplicates_gallery()

/home/dnth/anaconda3/envs/datapull/lib/python3.10/site-packages/fastdup/galleries.py:106: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df[out_col] = df[in_col].apply(lambda x: get_label_func.get(x, MISSING_LABEL))
/home/dnth/anaconda3/envs/datapull/lib/python3.10/site-packages/fastdup/galleries.py:106: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  df[out_col] = df[in_col].apply(lambda x: get_label_func.get(x, MISSING_LABEL))
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 360.31it/s]

Stored similarity visual view in  work_dir/galleries/duplicates.html
########################################################################################
Would you like to see awesome visualizations for some of the most popular academic datasets?
Click here to see and learn more: https://app.visual-layer.com/vl-datasets?utm_source=fastdup
########################################################################################

Duplicates Report

Info
Distance	1.0
From	/images/iPod/51355.jpg
To	/images/sock/69225.jpg
From_Label	iPod
To_Label	sock

Info
Distance	1.0
From	/images/pop-bottle_soda-bottle/62815.jpg
To	/images/beer-bottle/33973.jpg
From_Label	pop-bottle_soda-bottle
To_Label	beer-bottle

Info
Distance	1.0
From	/images/black-widow_Latrodectus-mactans/4204.jpg
To	/images/spider-web_spiders-web/70895.jpg
From_Label	black-widow_Latrodectus-mactans
To_Label	spider-web_spiders-web

Info
Distance	1.0
From	/images/banana/89072.jpg
To	/images/lemon/88830.jpg
From_Label	banana
To_Label	lemon

Info
Distance	1.0
From	/images/walking-stick_walkingstick_stick-insect/17643.jpg
To	/images/mantis_mantid/18797.jpg
From_Label	walking-stick_walkingstick_stick-insect
To_Label	mantis_mantid

Info
Distance	1.0
From	/images/meat-loaf_meatloaf/90376.jpg
To	/images/mashed-potato/87457.jpg
From_Label	meat-loaf_meatloaf
To_Label	mashed-potato

Info
Distance	1.0
From	/images/coral-reef/95198.jpg
To	/images/brain-coral/6866.jpg
From_Label	coral-reef
To_Label	brain-coral

Info
Distance	1.0
From	/images/lemon/88852.jpg
To	/images/banana/89002.jpg
From_Label	lemon
To_Label	banana

Info
Distance	1.0
From	/images/banana/89263.jpg
To	/images/orange/99657.jpg
From_Label	banana
To_Label	orange

Info
Distance	1.0
From	/images/orange/99746.jpg
To	/images/lemon/88551.jpg
From_Label	orange
To_Label	lemon

Info
Distance	1.0
From	/images/snail/7463.jpg
To	/images/slug/99073.jpg
From_Label	snail
To_Label	slug

Info
Distance	1.0
From	/images/brain-coral/6631.jpg
To	/images/coral-reef/95277.jpg
From_Label	brain-coral
To_Label	coral-reef

Out[39]:

In [40]:

fd.vis.outliers_gallery()

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 29006.25it/s]

Stored outliers visual view in  work_dir/galleries/outliers.html
########################################################################################
Would you like to see awesome visualizations for some of the most popular academic datasets?
Click here to see and learn more: https://app.visual-layer.com/vl-datasets?utm_source=fastdup
########################################################################################

Outliers Report

Showing image outliers, one per row

Info
Distance	0.59868
Path	/images/slug/99254.jpg
label	slug

Info
Distance	0.630931
Path	/images/fountain/47232.jpg
label	fountain

Info
Distance	0.640042
Path	/images/jellyfish/6152.jpg
label	jellyfish

Info
Distance	0.654982
Path	/images/walking-stick_walkingstick_stick-insect/17626.jpg
label	walking-stick_walkingstick_stick-insect

Info
Distance	0.658417
Path	/images/remote-control_remote/65026.jpg
label	remote-control_remote

Info
Distance	0.665128
Path	/images/pretzel/86745.jpg
label	pretzel

Info
Distance	0.665253
Path	/images/centipede/5052.jpg
label	centipede

Info
Distance	0.665985
Path	/images/centipede/5240.jpg
label	centipede

Info
Distance	0.666785
Path	/images/chain/98818.jpg
label	chain

Info
Distance	0.67015
Path	/images/nail/98207.jpg
label	nail

Info
Distance	0.6707
Path	/images/abacus/27129.jpg
label	abacus

Info
Distance	0.674768
Path	/images/volleyball/80147.jpg
label	volleyball

Info
Distance	0.675084
Path	/images/abacus/27267.jpg
label	abacus

Info
Distance	0.67606
Path	/images/chain/98911.jpg
label	chain

Info
Distance	0.676764
Path	/images/nail/98021.jpg
label	nail

Info
Distance	0.677318
Path	/images/crane/43785.jpg
label	crane

Info
Distance	0.677565
Path	/images/fountain/47006.jpg
label	fountain

Info
Distance	0.678366
Path	/images/space-heater/70208.jpg
label	space-heater

Info
Distance	0.679209
Path	/images/nail/98027.jpg
label	nail

Info
Distance	0.680928
Path	/images/barbershop/29697.jpg
label	barbershop

Out[40]:

In [42]:

fd.vis.similarity_gallery(slice='diff')

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7146/7146 [00:00<00:00, 24530.28it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 457.24it/s]

Stored similar images visual view in  work_dir/galleries/similarity.html
########################################################################################
Would you like to see awesome visualizations for some of the most popular academic datasets?
Click here to see and learn more: https://app.visual-layer.com/vl-datasets?utm_source=fastdup
########################################################################################

Similarity Report, label_score

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26003.jpg

Info To
0.905542	/images/boa-constrictor_Constrictor-constrictor/2954.jpg	boa-constrictor_Constrictor-constrictor
0.900586	/images/German-shepherd_German-shepherd-dog_German-police-dog_alsatian/12798.jpg	German-shepherd_German-shepherd-dog_German-police-dog_alsatian

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26046.jpg

Info To
0.90894	/images/military-uniform/55737.jpg	military-uniform
0.906594	/images/cliff_drop_drop-off/94750.jpg	cliff_drop_drop-off

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26049.jpg

Info To
0.906849	/images/boa-constrictor_Constrictor-constrictor/2593.jpg	boa-constrictor_Constrictor-constrictor
0.905895	/images/cliff-dwelling/41561.jpg	cliff-dwelling

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26083.jpg

Info To
0.911342	/images/trilobite/3492.jpg	trilobite
0.904064	/images/cliff_drop_drop-off/94863.jpg	cliff_drop_drop-off

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26114.jpg

Info To
0.929913	/images/mushroom/97992.jpg	mushroom
0.92827	/images/suspension-bridge/73142.jpg	suspension-bridge

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26164.jpg

Info To
0.910503	/images/bison/22886.jpg	bison
0.901101	/images/cliff_drop_drop-off/94749.jpg	cliff_drop_drop-off

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26203.jpg

Info To
0.919515	/images/spider-web_spiders-web/70711.jpg	spider-web_spiders-web
0.906872	/images/American-alligator_Alligator-mississipiensis/2277.jpg	American-alligator_Alligator-mississipiensis

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26226.jpg

Info To
0.912121	/images/koala_koala-bear_kangaroo-bear_native-bear_Phascolarctos-cinereus/5984.jpg	koala_koala-bear_kangaroo-bear_native-bear_Phascolarctos-cinereus
0.907932	/images/Arabian-camel_dromedary_Camelus-dromedarius/24191.jpg	Arabian-camel_dromedary_Camelus-dromedarius

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26260.jpg

Info To
0.919056	/images/cliff-dwelling/41511.jpg	cliff-dwelling
0.910586	/images/bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis/23409.jpg	bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26303.jpg

Info To
0.903905	/images/cliff-dwelling/41936.jpg	cliff-dwelling
0.903871	/images/cougar_puma_catamount_mountain-lion_painter_panther_Felis-concolor/15392.jpg	cougar_puma_catamount_mountain-lion_painter_panther_Felis-concolor

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26308.jpg

Info To
0.92572	/images/bison/22662.jpg	bison
0.908881	/images/baboon/25895.jpg	baboon

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26344.jpg

Info To
0.936067	/images/American-alligator_Alligator-mississipiensis/2277.jpg	American-alligator_Alligator-mississipiensis
0.932738	/images/cliff-dwelling/41788.jpg	cliff-dwelling

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26363.jpg

Info To
0.904164	/images/cliff-dwelling/41534.jpg	cliff-dwelling
0.90095	/images/ox/22268.jpg	ox

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26398.jpg

Info To
0.918615	/images/bison/22767.jpg	bison
0.911312	/images/bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis/23192.jpg	bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26410.jpg

Info To
0.912708	/images/koala_koala-bear_kangaroo-bear_native-bear_Phascolarctos-cinereus/5690.jpg	koala_koala-bear_kangaroo-bear_native-bear_Phascolarctos-cinereus
0.911665	/images/American-alligator_Alligator-mississipiensis/2468.jpg	American-alligator_Alligator-mississipiensis

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26412.jpg

Info To
0.90663	/images/cougar_puma_catamount_mountain-lion_painter_panther_Felis-concolor/15355.jpg	cougar_puma_catamount_mountain-lion_painter_panther_Felis-concolor
0.903551	/images/bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis/23465.jpg	bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26420.jpg

Info To
0.947837	/images/cliff_drop_drop-off/94962.jpg	cliff_drop_drop-off
0.941967	/images/bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis/23069.jpg	bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis

Query Image

Similar

Info From
label	African-elephant_Loxodonta-africana
from	/images/African-elephant_Loxodonta-africana/26498.jpg

Info To
0.901463	/images/baboon/25566.jpg	baboon
0.900453	/images/cliff-dwelling/41861.jpg	cliff-dwelling

Query Image

Similar

Info From
label	American-alligator_Alligator-mississipiensis
from	/images/American-alligator_Alligator-mississipiensis/2044.jpg

Info To
0.932808	/images/bullfrog_Rana-catesbeiana/1027.jpg	bullfrog_Rana-catesbeiana
0.924776	/images/European-fire-salamander_Salamandra-salamandra/697.jpg	European-fire-salamander_Salamandra-salamandra

Query Image

Similar

Info From
label	American-alligator_Alligator-mississipiensis
from	/images/American-alligator_Alligator-mississipiensis/2068.jpg

Info To
0.929694	/images/bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis/23144.jpg	bighorn_bighorn-sheep_cimarron_Rocky-Mountain-bighorn_Rocky-Mountain-sheep_Ovis-canadensis
0.924442	/images/spider-web_spiders-web/70818.jpg	spider-web_spiders-web

Query Image

Similar

Out[42]:

	from	to	label	label2	distance	score	length
1	images_dir/images/African-elephant_Loxodonta-africana/26003.jpg	[images_dir/images/German-shepherd_German-shepherd-dog_German-police-dog_alsatian/12798.jpg, images_dir/images/boa-constrictor_Constrictor-constrictor/2954.jpg]	[African-elephant_Loxodonta-africana, African-elephant_Loxodonta-africana]	[German-shepherd_German-shepherd-dog_German-police-dog_alsatian, boa-constrictor_Constrictor-constrictor]	[0.900586, 0.905542]	0.0	2
5	images_dir/images/African-elephant_Loxodonta-africana/26046.jpg	[images_dir/images/cliff_drop_drop-off/94750.jpg, images_dir/images/military-uniform/55737.jpg]	[African-elephant_Loxodonta-africana, African-elephant_Loxodonta-africana]	[cliff_drop_drop-off, military-uniform]	[0.906594, 0.90894]	0.0	2
6	images_dir/images/African-elephant_Loxodonta-africana/26049.jpg	[images_dir/images/cliff-dwelling/41561.jpg, images_dir/images/boa-constrictor_Constrictor-constrictor/2593.jpg]	[African-elephant_Loxodonta-africana, African-elephant_Loxodonta-africana]	[cliff-dwelling, boa-constrictor_Constrictor-constrictor]	[0.905895, 0.906849]	0.0	2
7	images_dir/images/African-elephant_Loxodonta-africana/26083.jpg	[images_dir/images/cliff_drop_drop-off/94863.jpg, images_dir/images/trilobite/3492.jpg]	[African-elephant_Loxodonta-africana, African-elephant_Loxodonta-africana]	[cliff_drop_drop-off, trilobite]	[0.904064, 0.911342]	0.0	2
13	images_dir/images/African-elephant_Loxodonta-africana/26114.jpg	[images_dir/images/suspension-bridge/73142.jpg, images_dir/images/mushroom/97992.jpg]	[African-elephant_Loxodonta-africana, African-elephant_Loxodonta-africana]	[suspension-bridge, mushroom]	[0.92827, 0.929913]	0.0	2
...	...	...	...	...	...	...	...
7097	images_dir/images/water-tower/81086.jpg	[images_dir/images/water-tower/81095.jpg, images_dir/images/water-tower/81128.jpg]	[water-tower, water-tower]	[water-tower, water-tower]	[0.90285, 0.905127]	100.0	2
7099	images_dir/images/water-tower/81102.jpg	[images_dir/images/water-tower/81331.jpg, images_dir/images/water-tower/81308.jpg]	[water-tower, water-tower]	[water-tower, water-tower]	[0.903888, 0.91205]	100.0	2
7100	images_dir/images/water-tower/81128.jpg	[images_dir/images/water-tower/81208.jpg, images_dir/images/water-tower/81086.jpg]	[water-tower, water-tower]	[water-tower, water-tower]	[0.900657, 0.905127]	100.0	2
7106	images_dir/images/water-tower/81308.jpg	[images_dir/images/water-tower/81082.jpg, images_dir/images/water-tower/81102.jpg]	[water-tower, water-tower]	[water-tower, water-tower]	[0.904132, 0.91205]	100.0	2
7112	images_dir/images/water-tower/81429.jpg	[images_dir/images/water-tower/81316.jpg, images_dir/images/water-tower/81441.jpg]	[water-tower, water-tower]	[water-tower, water-tower]	[0.900633, 0.911426]	100.0	2

3683 rows × 7 columns

Wrap Up¶

That's a wrap! In this notebook we showed how you can run fastdup on a Hugging Face Dataset. You can use similar methods to run on other similar datasets on Huggging Face Datasets.

Try it out and let us know what issues you find.

Next, feel free to check out other tutorials -

⚡ Quickstart: Learn how to install fastdup, load a dataset and analyze it for potential issues such as duplicates/near-duplicates, broken images, outliers, dark/bright/blurry images, and view visually similar image clusters. If you're new, start here!
🧹 Clean Image Folder: Learn how to analyze and clean a folder of images from potential issues and export a list of problematic files for further action. If you have an unorganized folder of images, this is a good place to start.
🖼 Analyze Image Classification Dataset: Learn how to load a labeled image classification dataset and analyze for potential issues. If you have labeled ImageNet-style folder structure, have a go!
🎁 Analyze Object Detection Dataset: Learn how to load bounding box annotations for object detection and analyze for potential issues. If you have a COCO-style labeled object detection dataset, give this example a try.

VL Profiler¶

If you prefer a no-code platform to inspect and visualize your dataset, try our free cloud product VL Profiler - VL Profiler is our first no-code commercial product that lets you visualize and inspect your dataset in your browser.

Sign up now, it's free.

As usual, feedback is welcome!

Questions? Drop by our Slack channel or open an issue on GitHub.