#!/usr/bin/env python
# coding: utf-8

# [![image](https://raw.githubusercontent.com/visual-layer/visuallayer/main/imgs/vl_horizontal_logo.png)](https://www.visual-layer.com)

# # Quick Dataset Analysis
# 
# [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/visual-layer/fastdup/blob/main/examples/quick-dataset-analysis.ipynb)
# [![Open in Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/visual-layer/fastdup/blob/main/examples/quick-dataset-analysis.ipynb)
# 
# This notebook shows how to quickly analyze an image dataset for potential issues using fastdup. We'll take you on a high level tour showcasing the core functions of fastdup in the shortest time.
# 
# By the end of this notebook you will learn how to find out if your dataset has issues such as:
# 
# + Broken images.
# + Duplicates/near-duplicates.
# + Outliers.
# + Dark/bright/blurry images.
# 
# We'll also visualize clusters of visually similar looking images to let you have a birds eye view on your dataset.

# ## Installation
# 
# If you're new, we encourage you to run the notebook in [Google Colab](https://colab.research.google.com/github/visual-layer/fastdup/blob/main/examples/quick-dataset-analysis.ipynb) or [Kaggle](https://kaggle.com/kernels/welcome?src=https://github.com/visual-layer/fastdup/blob/main/quick-dataset-analysis.ipynb) for the best experience. If you'd like to just view and skim through the notebook, we recommend viewing using [nbviewer](https://nbviewer.org/github/visual-layer/fastdup/blob/main/examples/quick-dataset-analysis.ipynb).  
# 
# Let's start with the installation:

# In[1]:


get_ipython().system('pip install fastdup -Uq')


# Now, test the installation by printing out the version. If there's no error message, we are ready to go!

# In[2]:


import fastdup
fastdup.__version__


# ## Download Dataset
# 
# For demonstration, we will use a widely available and well curated dataset [Oxford IIIT Pet dataset](https://www.robots.ox.ac.uk/~vgg/data/pets/). For that reason we might not find a lot of issues here but feel free to swap this dataset with your own.
# 
# The dataset consists of images and annotations for 37 category pet with roughly 200 images for each class. For now, we are only interested in finding issues in the images and not the annotations. More on annotations in the upcoming [notebook](./analyzing-image-classification-dataset.ipynb).
# 
# Let's download only from the dataset and extract them into the local directory:

# In[ ]:


get_ipython().system('wget https://thor.robots.ox.ac.uk/~vgg/data/pets/images.tar.gz -O images.tar.gz')
get_ipython().system('tar xf images.tar.gz')


# ## Run fastdup
# 
# Once the extraction completes, we can run fastdup on the images.
# 
# For that let's create a `fastdup` object and specify the input directory which points to the folder of images.

# In[3]:


fd = fastdup.create(input_dir="images/")


# The `.create` method also has an optional `work_dir` parameter which specifies the directory to store artifacts from the run.
# 
# You can optionally run `fastdup.create(work_dir="my_work_dir/", input_dir="images/")` if you'd like to store the artifacts in a specific working directory.

# In[4]:


fd.run()


# ## View Run Summary

# In[5]:


fd.summary()


# ## Invalid Images
# From the logs printed above, we see there are a few invalid images. These are broken images that cannot be read.
# 
# You can get a list of broken images with:

# In[6]:


fd.invalid_instances()


# ## Duplicate/Near-duplicates
# 
# One of the lowest hanging fruits in cleaning a dataset is finding and eliminating duplicates.
# 
# fastdup provides a handy way of visualizing duplicates/near-duplicates using the `duplicates_gallery` method. The `Distance` value indicates how visually similar are the image pairs in the gallery. A `Distance` of `1.0` indicates an exact copy and vice-versa.

# In[7]:


fd.vis.duplicates_gallery()


# ## Outliers
# 
# Similar to duplicate pairs, you can visualize potential outliers in your dataset with:

# In[8]:


fd.vis.outliers_gallery() 


# ## Dark, Bright and Blurry Images
# 
# fastdup also lets you visualize images from your dataset using statistical metrics.
# 
# For example, with `metric='dark'` we can visualize the darkest images from the dataset.

# In[9]:


fd.vis.stats_gallery(metric='dark')


# In[10]:


fd.vis.stats_gallery(metric='bright')


# In[11]:


fd.vis.stats_gallery(metric='blur')


# ## Image Clusters
# 
# One of fastdup's coolest feature is visualizing image clusters. In the previous section we saw how to visualize similar image pairs. In this section we group similar looking image (or even duplicates) as a cluster and visualize them in gallery.
# 
# To do so, simply run:
# 
# 

# > **Note**: fastdup uses default parameter values when creating image clusters. Depending on your data and use case, the best value may vary. Read more [here](https://visual-layer.readme.io/docs/dataset-cleanup) on how to change parameter values to cluster images.

# In[12]:


fd.vis.component_gallery()


# ## Wrap Up
# 
# That's a wrap! In this notebook we showed how you can run fastdup on a dataset or any folder of images. 
# 
# We've seen how to use fastdup to find:
# 
# + Broken images.
# + Duplicate/near-duplicates.
# + Outliers.
# + Dark, bright and blurry images.
# + Image clusters.
# 
# Next, feel free to check out other tutorials -
# 
# + ⚡ [**Quickstart**](https://nbviewer.org/github/visual-layer/fastdup/blob/main/examples/quick-dataset-analysis.ipynb): Learn how to install fastdup, load a dataset and analyze it for potential issues such as duplicates/near-duplicates, broken images, outliers, dark/bright/blurry images, and view visually similar image clusters. If you're new, start here!
# + 🧹 [**Clean Image Folder**](https://nbviewer.org/github/visual-layer/fastdup/blob/main/examples/cleaning-image-dataset.ipynb): Learn how to analyze and clean a folder of images from potential issues and export a list of problematic files for further action. If you have an unorganized folder of images, this is a good place to start.
# + 🖼 [**Analyze Image Classification Dataset**](https://nbviewer.org/github/visual-layer/fastdup/blob/main/examples/analyzing-image-classification-dataset.ipynb): Learn how to load a labeled image classification dataset and analyze for potential issues. If you have labeled ImageNet-style folder structure, have a go!
# + 🎁 [**Analyze Object Detection Dataset**](https://nbviewer.org/github/visual-layer/fastdup/blob/main/examples/analyzing-object-detection-dataset.ipynb): Learn how to load bounding box annotations for object detection and analyze for potential issues. If you have a COCO-style labeled object detection dataset, give this example a try. 
# 

# 
# ## VL Profiler
# If you prefer a no-code platform to inspect and visualize your dataset, [**try our free cloud product VL Profiler**](https://app.visual-layer.com) - VL Profiler is our first no-code commercial product that lets you visualize and inspect your dataset in your browser. 
# 
# [Sign up](https://app.visual-layer.com) now, it's free.
# 
# [![image](https://raw.githubusercontent.com/visual-layer/fastdup/main/gallery/vl_profiler_promo.svg)](https://app.visual-layer.com)
# 
# As usual, feedback is welcome! 
# 
# Questions? Drop by our [Slack channel](https://visualdatabase.slack.com/join/shared_invite/zt-19jaydbjn-lNDEDkgvSI1QwbTXSY6dlA#/shared-invite/email) or open an issue on [GitHub](https://github.com/visual-layer/fastdup/issues).

#