How to Remove Duplicates From a Python List
Published: 2024-03-28 11:23:08

[Removing duplicates from a list in Python](https://pythonkb.com/how-to-remove-duplicates-from-a-python-list/) can be accomplished in various ways, each suited to different situations based on the order preservation requirement, the type of elements in the list, and performance considerations. Let's explore five examples, detailing each approach's mechanism and showcasing its output.

### 1. Using a Set to Remove Duplicates

**Code:**

```python
original_list = [1, 2, 2, 3, 3, 3]
unique_list = list(set(original_list))
print(unique_list)
```

**Explanation:**

- A set is created from `original_list`. Sets automatically remove duplicate elements since a set in [Python](https://pythonkb.com/category/python/) cannot contain duplicates.
- The set is then converted back into a list.
- This method does not preserve the original order of elements.

**Output:**

```
[1, 2, 3]
```

### 2. Using a Dictionary (Preserving Order)

**Code:**

```python
original_list = ['a', 'b', 'a', 'c', 'b']
unique_list = list(dict.fromkeys(original_list))
print(unique_list)
```

**Explanation:**

- A dictionary is created from `original_list` using `fromkeys()`. This method removes duplicates because dictionary keys are unique.
- The order of elements is preserved because dictionaries in Python 3.7+ are ordered.
- The dictionary keys are then converted back into a list.

**Output:**

```
['a', 'b', 'c']
```

### 3. Using List Comprehension (Preserving Order)

**Code:**

```python
original_list = [1, 2, 2, 3, 4, 4, 5]
unique_list = []
[unique_list.append(x) for x in original_list if x not in unique_list]
print(unique_list)
```

**Explanation:**

- An empty list `unique_list` is initialized.
- A list comprehension iterates over `original_list`. For each element, it checks if the element is not already in `unique_list`. If it's not, the element is appended to `unique_list`.
- This method preserves the original order.

**Output:**

```
[1, 2, 3, 4, 5]
```

### 4. Using the `collections.OrderedDict` (For Older Python Versions)

**Code:**

```python
from collections import OrderedDict
original_list = ['apple', 'banana', 'apple', 'pear', 'banana']
unique_list = list(OrderedDict.fromkeys(original_list))
print(unique_list)
```

**Explanation:**

- `OrderedDict.fromkeys()` is used to create an ordered dictionary where all duplicate `original_list` items are removed.
- The order of elements is preserved.
- This method is useful for versions of Python older than 3.7, where regular dictionaries do not guarantee order preservation.

**Output:**

```
['apple', 'banana', 'pear']
```

### 5. Using a Loop (Preserving Order)

**Code:**

```python
original_list = [10, 20, 20, 30, 40, 40, 50]
unique_list = []
for item in original_list:
    if item not in unique_list:
        unique_list.append(item)
print(unique_list)
```

**Explanation:**

- `unique_list` is initialized as an empty list.
- The code iterates through `original_list`. For each element, it checks if the element is not already in `unique_list`. If it's not, the element is appended.
- This method preserves the original order and is straightforward but may not be the most efficient for large lists due to the `if item not in unique_list` check.

**Output:**

```
[10, 20, 30, 40, 50]
```

Each of these methods provides a way to remove duplicates from a list, with differences in order preservation and performance considerations.