Double Trouble: Eliminate Image Duplicates with FiftyOne
Tags: computervision, machinelearning, ai, datascience
Published: 2024-03-13 19:34:43

_Author: [Jacob Marks](https://www.linkedin.com/in/jacob-marks/) (Machine Learning Engineer at [Voxel51](https://voxel51.com/))_

## Find Exact and Approximate Duplicate Images with This Plugin

Welcome to week four of _Ten Weeks of Plugins_. During these ten weeks, we will be building a FiftyOne Plugin (or multiple!) each week and sharing the lessons learned!

If you’re new to them, FiftyOne Plugins provide a flexible mechanism for anyone to extend the functionality of their FiftyOne App. You may find the following resources helpful:

- [FiftyOne Plugins Repo](https://github.com/voxel51/fiftyone-plugins)
- [FiftyOne Plugin Docs](https://docs.voxel51.com/plugins/index.html#downloading-plugins)
- [Plugins Channel](https://fiftyone-users.slack.com/archives/plugins) in the [FiftyOne Community Slack](https://slack.voxel51.com/)

What we’ve built so far:

- Week 0: [Image Quality Issues](https://github.com/jacobmarks/image-quality-issues) & [Concept Interpolation](https://github.com/jacobmarks/concept-interpolation)
- Week 1: [AI Art Gallery](https://github.com/jacobmarks/ai-art-gallery) & [Twilio Automation](https://github.com/jacobmarks/twilio-automation-plugin)
- Week 2: [Visual Question Answering](https://github.com/jacobmarks/vqa-plugin)
- Week 3: [YouTube Player Panel](https://github.com/jacobmarks/fiftyone-youtube-panel-plugin)

Ok, let’s dive into this week’s FiftyOne Plugin — [Image Deduplication](https://github.com/jacobmarks/image-deduplication-plugin)!

## Image Deduplication 🖼️🪞🧹

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pbuln361godkd8rfg80f.gif)

The biggest challenge in training machine learning models is curating a high quality dataset. Duplicate (or very similar) data is a major roadblock to building such a dataset. Multiple copies of the same (or approximately the same) samples can lead to longer training times, higher training costs, and lower overall performance. On the flip side, you likely want a diverse dataset with good coverage over the data domain.

Duplicates come in two flavors:

1. **Exact duplicates**: pixel-perfect matches, where one image is literally a down-to-the-bit copy of another
2. **Approximate duplicates**: images (or other data) that are highly similar — typically evaluated by computing the closeness between samples with some similarity metric — and setting a threshold for similarity using this metric.

_Deduplication_ is the task of removing these exact and approximate duplicates from a dataset. 

Typically, deduplication involves writing a lot of code to find, visualize, and remove all of the duplicates in your dataset. With this FiftyOne plugin, that all changes. Now you can deduplicate your entire dataset from within the FiftyOne App, without writing a single line of code!

## Plugin Overview & Functionality

For the fourth week of _10 Weeks of Plugins_, I built an Image Deduplication Plugin. This plugin allows you to:

- Find both exact and approximate duplicate images in your dataset
- Visualize these groups of duplicates
- Delete all duplicates OR Keep a representative from each set of duplicates

The plugin has eight (!) [operators](https://docs.voxel51.com/plugins/index.html#operators) (a powerful feature in FiftyOne that allow plugin developers to define custom operations that can be executed by users of the FiftyOne App), but don’t get overwhelmed — really it’s just two sets of analogous operators, for exact and approximate deduplication workflows.

After you install the plugin, when you open the operators list (pressing " `` ` `` " in the FiftyOne App) you should see these operators. Search for “dedup” to narrow down the list!

### 🔍 Finding Duplicates

The first pair of operators helps you to find duplicate images in your dataset.

- `find_approximate_duplicate_images`: uses a [similarity index](https://docs.voxel51.com/user_guide/brain.html#similarity) to find approximate duplicates.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w87a0ti2s67ed6rb2yrt.gif)
 
You can specify either a distance threshold (how close the images need to be according to the similarity metric to be considered near duplicates) or a fraction of the dataset to mark as near duplicates. 

If you haven’t computed a similarity index on your dataset, you can do so by running:

```
import fiftyone.brain as fob
fob.compute_similarity(dataset, brain_key = "sim", metric="cosine")
```

For a large dataset, you may want to use a vector database. In this case, check out our native integrations with [Pinecone](https://docs.voxel51.com/integrations/pinecone.html), [Qdrant](https://docs.voxel51.com/integrations/qdrant.html), [Milvus](https://docs.voxel51.com/integrations/milvus.html), and [LanceDB](https://docs.voxel51.com/integrations/lancedb.html)!

When the operation finishes, it will have created two [saved views](https://docs.voxel51.com/user_guide/using_views.html#saving-views): `approx_dup_view`, and `approx_dup_groups_view`. You can access these by clicking on the saved views selector in the FiftyOne App, or programmatically via Python:

```
approx_dup_view = dataset.load_saved_view("approx_dup_view")
approx_dup_groups_view = dataset.load_saved_view("approx_dup_groups_view")
```

- `find_exact_duplicates`: uses file hashes to find exact duplicates


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h40blynezy58g6lsgc9a.gif)

Essentially, the file hash computes a short signature for each sample based on the binary data stored in the image. The operator then checks if there are duplicate values of these signatures and marks these samples as duplicates. 

The operator adds a `filehash` field to each sample, and creates a saved view `exact_dup_view`, which contains just the images with duplicate filehashes.

## 🪟Viewing Duplicates

Once you have found exact and/or approximate duplicates in your dataset, you may want to view these duplicates. For approximate duplicates, for instance, you may want to verify that the distance threshold you set was rigorous enough.

The Image Deduplication plugin makes it easy to do this with the `display_approximate_duplicate_groups` and `display_exact_duplicate_groups` operators. The names are pretty self-explanatory, but the former loads the `approx_dup_groups_view` view we saved earlier, and the latter displays the samples in `exact_dup_view`, grouped by `filehash`.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hnsa8axd8c6a8xzzhg13.gif)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6bmtxrajagywccwiacjd.gif)

### 🗑️Removing Duplicates

Once you have viewed your identified duplicates, it is time to clean your dataset. At this point, you have two options:

1. Remove ALL duplicates: delete all samples marked as an exact or approximate duplicate
2. Keep a representative: remove all but one duplicate from each set of exact or approximate duplicates

As always, there are sister operators for working with approximate and exact duplicates:

- `remove_all_approximate_duplicates`: removes all near-duplicate images from a dataset
- `remove_all_exact_duplicates`: removes all exact duplicate images from a dataset
- `deduplicate_approximate_duplicates`: removes near-duplicate images from a dataset, keeping a _representative image_ from each duplicate set
- `deduplicate_exact_duplicates`: removes exact duplicate images from a dataset, _keeping a representative image_ from each duplicate set

Here’s an example of each:


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k6ce0na9dce9om4yemj5.gif)


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zc29klnd6vw9n5trks0h.gif)

## Installing the Plugin

If you haven’t already done so, install FiftyOne:

```
pip install fiftyone
```
Then you can download this plugin from the command line with:

```
fiftyone plugins download https://github.com/jacobmarks/image-dedup-plugin
```
Refresh the FiftyOne App, and you should see the eight operators in your operators list when you press the " `` ` `` " key.

## Lessons Learned

The Image Deduplication plugin is a Python Plugin with the usual structure (an `__init__.py`, `fiftyone.yml`, and `REAMDE.md` files). Additionally, it has the following:

- An assets folder for storing icons
- A Python file `exact_dups.py` for handling the logic and computations involved for exact duplicates
- A Python file `approx_dups.py` for handling the logic and computations involved for approximate duplicates

### Splitting Code into Submodules

It’s typically good practice in software development to make code modular, splitting self-contained pieces of logic into separate functions or files. This is known as [separation of concerns](https://en.wikipedia.org/wiki/Separation_of_concerns). 

The Image Deduplication plugin was a good exercise in applying this principle to FiftyOne’s plugin system. To utilize functions or variables you define in another file in the FiftyOne plugin’s directory, you need to add the path to that file to your system path.

Here’s an example where we import `find_exact_duplicates` from the `exact_dups` file: 

```
from fiftyone.core.utils import add_sys_path
with add_sys_path(os.path.dirname(os.path.abspath(__file__))):
            # pylint: disable=no-name-in-module,import-error
            from exact_dups import find_exact_duplicates
```

Starting from the innermost part of this expression:

- `__file__` is a variable containing the path to the current module — in this case the `__init__.py` file.
- `os.path.abspath` gets the absolute path for this file
- `os.path.dirname` extracts the directory name of this absolute path
- `add_sys_path` is a FiftyOne utility function that adds this to our system path

The second to last line, # `pylint: disable=no-name-in-module,import-error` tells our linter not to throw an error when [linting](https://en.wikipedia.org/wiki/Lint_(software)) the file.

### Loading a View

When executed, the `display_approximate_duplicate_groups` and `display_exact_duplicate_groups` operators each trigger the loading of specific views. Doing this is pretty straightforward, but it is worth noting that the data passed into params in the `ctx.trigger()` call needs to be serialized. In fact, _all_ data passed into parameter dictionaries for FiftyOne operators needs to be serialized.

Fortunately, FiftyOne `DatasetView` objects are easy to serialize!

```
import json
from bson import json_util

def serialize_view(view):
    return json.loads(json_util.dumps(view._serialize()))
```

### Icons for Each Operator

The last tip is a simple but fun one: By utilizing the icon argument in the operator config, you can specify a unique icon for each operator. This is the icon that will then show up in the operators list with you hit " `` ` `` ".

For example, here’s the start of the operator definition for `FindExactDuplicates`:

```
class FindExactDuplicates(foo.Operator):
    @property
    def config(self):
        return foo.OperatorConfig(
            name="find_exact_duplicate_images",
            label="Dedup: Find exact duplicates",
            description="Find exact duplicates in the dataset",
	     icon="/assets/exact_duplicates.svg",
            dynamic=True,
        )
```

I like to put all of the SVGs I use as icons in an `assets` folder to stay organized 📁.

## Conclusion

Building a high quality dataset doesn’t have to be a hassle. With our [Image Quality Issues Plugin](https://github.com/jacobmarks/image-quality-issues) from week 0, you can find a variety of common issues potentially plaguing images in your dataset, from peculiar aspect ratios to oversaturation. Now with the Image Deduplication Plugin (this post) you can also find and eliminate duplicates from your dataset in mere minutes!

Stay tuned over the remaining weeks in the _Ten Weeks of FiftyOne Plugins_ while we continue to pump out a killer lineup of plugins! You can track our journey in our [ten-weeks-of-plugins repo](https://github.com/jacobmarks/ten-weeks-of-plugins) — and I encourage you to fork the repo and join me on this journey!