How to Remove Duplicates from a List in Python

Python gives you a fast one-liner for this, but it comes with a tradeoff most people hit the first time they use it: it doesn't preserve order. Here's the standard approach, the order-preserving fix, and what to do when your items aren't simple strings or numbers.

The fast way: set()

items = ['apple', 'banana', 'apple', 'cherry', 'banana']

unique = list(set(items))
print(unique)  # order not guaranteed, e.g. ['banana', 'cherry', 'apple']

Converting to a set and back to a list is O(n) and about as fast as deduplication gets in Python. The cost is that sets have no defined order, so the output order has no relationship to the input order.

The order-preserving way: dict.fromkeys()

items = ['apple', 'banana', 'apple', 'cherry', 'banana']

unique = list(dict.fromkeys(items))
print(unique)  # ['apple', 'banana', 'cherry'] - first-seen order preserved

dict.fromkeys(items) builds a dictionary with each item as a key - duplicate keys are automatically overwritten rather than duplicated, and since Python 3.7 insertion order is guaranteed. This is the version to reach for whenever the order of your list is meaningful, which is most of the time.

Deduplicating a list of dictionaries by key

users = [
    {'id': 1, 'name': 'Alice'},
    {'id': 2, 'name': 'Bob'},
    {'id': 1, 'name': 'Alice'},
]

seen_ids = set()
unique_users = []
for user in users:
    if user['id'] not in seen_ids:
        seen_ids.add(user['id'])
        unique_users.append(user)

print(unique_users)  # keeps id 1 once, id 2 once

Dicts (and lists) aren't hashable, so you can't put them directly into a set or use them as dict keys. Instead, track the key you're deduplicating on - here, id - in a separate set, and only keep items whose key hasn't been seen yet.

Case-insensitive deduplication

items = ['Apple', 'apple', 'Banana', 'APPLE']

seen = set()
unique = []
for x in items:
    key = x.lower()
    if key not in seen:
        seen.add(key)
        unique.append(x)

print(unique)  # ['Apple', 'Banana'] - first-seen casing kept

set() and dict.fromkeys() both compare items exactly, so 'Apple' and 'apple' are treated as distinct. To dedupe case-insensitively while keeping the original casing in the output, compare on a lowercased copy but store the original value.

Performance note

Both set(items) and dict.fromkeys(items) run in roughly O(n) - each item is hashed once and checked against the existing keys in constant time. Avoid a nested-loop approach ([x for i, x in enumerate(items) if x not in items[:i]]) for anything but tiny lists - it's O(n squared) and gets slow past a few thousand items. The same hashing is what makes comparing two lists fast: set(a) - set(b) finds what's in one list but not the other, as shown in difference between two lists in Python.

Working with pandas instead of a plain list

If your data is already a pandas DataFrame rather than a plain Python list, df.drop_duplicates() is the equivalent - it works on whole rows, with subset and keep parameters for finer control. Full pandas guide →