Handling Array Duplicates Can Be Tricky
Tags: javascript, webdev, discuss
Published: 2019-05-11 20:45:51

---
title: Handling Array Duplicates Can Be Tricky
published: true
description: Handling Array Duplicates Can Be Tricky
tags: #javascript, #webdev, #discuss
cover_image: https://images.unsplash.com/photo-1461685265823-f8d5d0b08b9b?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=500&q=60
---

> Originally posted on [my-blog](https://www.devinduct.com/blogpost/17/handling-array-duplicates-can-be-tricky). Check it out for more up to date content

Let's begin by defining a simple array:

```js
const cars = [
    'Mazda', 
    'Ford', 
    'Renault', 
    'Opel', 
    'Mazda'
]
```

As you can see, the first and the last item are the same. Finding this duplicate is straightforward considering that we have an array of items that are a primitive type. To achieve this, we can simply use the method `filter` along with the `indexOf` method within the provided callback.

```js
const unique = cars.filter((car, idx) => cars.indexOf(car) === idx);
console.log(unique); // outputs ['Mazda', 'Ford', 'Renault', 'Opel']
```

Note that the `indexOf` method will return the first occurrence of an item within the array. This is why we can compare the index returned by the `indexOf` method with the current index in each iteration to see if the current item is a duplicate.

## Finding Object Duplicates

This is the tricky part. Objects are compared via reference rather than the value or structure. This means that if we compare two objects that are exactly the same they won't match. We cannot simply do something like `obj1 === obj2` because of the way they are compared.

```js
const obj1 = {
   name: 'John',
   surname: 'Doe'
}

const obj2 = {
   name: 'John',
   surname: 'Doe'
}

const match = obj1 === obj2;
console.log(match) // outputs false
```

Now, what if we have an array of object duplicates? How are we going to filter out those? Considering what we've just read, it's not possible to use something simple like `indexOf`.

Example array:

```js
const names = [{
   name: 'John',
   surname: 'Doe'
}, {
   name: 'Muhamed',
   surname: 'Ali'
}, {
   name: 'Mike',
   surname: 'Tyson'
}, {
   name: 'John',
   surname: 'Doe'
}, {
   name: 'John',
   surname: 'Doe'
}, {
   name: 'Mike',
   surname: 'Tyson'
}, {
   name: 'Mike',
   surname: 'Tyson'
}];
```
As you can see, we have a couple of duplicate items. Let's implement the function that will find the duplicates.

## The Longer Version

In this approach, we will manually loop through the source array (`forEach` method) and check if each item exists in the resulting array by using the `find` method.
Considering that we have an array of objects, we must compare each property of the current object in order to make sure that the items are the same. Broken down into steps the process looks like this:

1. Get the object properties
2. Define the resulting arrays (`unique` and `duplicates`)
3. Loop through the source array
4. Try to locate the current item within the `unique` array
5. If the item is found, push it into the `duplicates` otherwise into the `unique` array

```js
const findDuplicates = (source) => {
    const keys = Object.keys(source[0]);
    let unique = [], duplicates = [];

    source.forEach((item, idx) => {
        
        if(idx == 0) {
            unique.push(item);
            return;
        };

        const resultItem = unique.find(resultItem => {
            let notFound = true;

            keys.forEach(key => {
                notFound = notFound && 
                    item[key] != resultItem[key];
            });

            return !notFound;
        });

        (!resultItem ? unique : duplicates).push(item);

    });

    return { unique: unique, duplicates: duplicates };
};

const result = findDuplicates(names);
console.log(result.unique, result.duplicates);

// expected output

// unique items

// 0: {name: "John", surname: "Doe"}
// 1: {name: "Muhamed", surname: "Ali"}
// 2: {name: "Mike", surname: "Tyson"}

// duplicate items

// 0: {name: "John", surname: "Doe"}
// 1: {name: "John", surname: "Doe"}
// 2: {name: "Mike", surname: "Tyson"}
// 3: {name: "Mike", surname: "Tyson"}
```
## A Bit Shorter Version

We could use the `reduce` method in order to achieve the same thing. This is a very powerful method and it can be used to transform the array into the desired result. It accepts a callback as a parameter which is executed for each item in the array. The return value of the callback is the given accumulator modified within each iteration. Considering that this is not an article about the `reduce` method, check out the official [MDN documentation](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/reduce)

Ok, back to our code. The modified version of the `findDuplicates` method looks like this:

```js
const findDuplicates = (source) => {
    const keys = Object.keys(source[0]);
    return source.reduce((acc, item) => {
        const resultItem = acc.unique.find(x => {
            let notFound = true;

            keys.forEach(key => {
                notFound = notFound && 
                    item[key] != x[key];
            });

            return !notFound;
        });

        (!resultItem ? acc.unique : acc.duplicates).push(item);
        return acc;
    }, {
        unique: [],
        duplicates: []
    })
};
```
The modified version should return the same resulting arrays as it did before.

```js
// unique items

// 0: {name: "John", surname: "Doe"}
// 1: {name: "Muhamed", surname: "Ali"}
// 2: {name: "Mike", surname: "Tyson"}

// duplicate items

// 0: {name: "John", surname: "Doe"}
// 1: {name: "John", surname: "Doe"}
// 2: {name: "Mike", surname: "Tyson"}
// 3: {name: "Mike", surname: "Tyson"}
```

That's it. Thank you for reading and see you in the next article.

# Further Reading

See [this cheat sheet](https://devinduct.com/cheatsheet/8/array-operations) that will guide you through the most common use cases when it comes to the array manipulation.