How to Remove Duplicates from a List in Every Programming Language

"Remove duplicates" means the same thing in every language: keep only one copy of each distinct value. The fast way to do it is also nearly identical everywhere - lean on a hash-based structure instead of comparing every item to every other one. This page is a quick reference across the languages people search for most; jump to yours, or read the general no-code guide if you'd rather not write anything.

Python

items = ['apple', 'banana', 'apple', 'cherry', 'banana']

# Order not preserved
unique = list(set(items))

# Order preserved (Python 3.7+, dicts keep insertion order)
unique_ordered = list(dict.fromkeys(items))
print(unique_ordered)  # ['apple', 'banana', 'cherry']

dict.fromkeys() is the idiomatic order-preserving dedup in modern Python - it's faster and more readable than a manual loop with a seen-set. Full Python guide →

JavaScript

const items = ['apple', 'banana', 'apple', 'cherry', 'banana'];

const unique = [...new Set(items)];
console.log(unique); // ['apple', 'banana', 'cherry']

Unlike Python's plain set, JavaScript's Set preserves insertion order, so the spread-into-array pattern above is both fast and order-preserving in one step. Full JavaScript guide →

SQL

SELECT DISTINCT customer_id
FROM orders;

DISTINCT removes duplicate rows from a query result. Deleting duplicate rows that already exist in a table is a different problem, usually solved with a ROW_NUMBER() window function. Full SQL guide →

Java

List<String> items = new ArrayList<>(List.of("apple", "banana", "apple", "cherry", "banana"));

List<String> unique = new ArrayList<>(new LinkedHashSet<>(items));
System.out.println(unique); // [apple, banana, cherry]

LinkedHashSet is the order-preserving equivalent of HashSet - it dedupes exactly the same way but keeps insertion order, which a plain HashSet does not guarantee.

Bash

# Order not preserved, requires sorting
sort -u file.txt

# Order preserved
awk '!seen[$0]++' file.txt

sort -u is the fastest one-liner but reorders your data alphabetically as a side effect. awk '!seen[$0]++' tracks every line it has already printed and skips repeats, preserving the original order. Full Bash guide →

PowerShell

$items = @('apple', 'banana', 'apple', 'cherry', 'banana')

$unique = $items | Select-Object -Unique
$unique # apple, banana, cherry

Select-Object -Unique preserves order and is case-insensitive by default - worth knowing if your list contains values that differ only by case. Full PowerShell guide →

pandas (Python DataFrames)

df_unique = df.drop_duplicates()

# Only consider certain columns, keep the last occurrence
df_unique = df.drop_duplicates(subset=['email'], keep='last')

drop_duplicates() operates on whole rows by default, with subset and keep parameters to control which columns count and which occurrence survives. Full pandas guide →

Excel and Google Sheets

Excel's Data ribbon has a built-in Remove Duplicates command, and Google Sheets has both a UNIQUE() formula and a Data Cleanup menu option. Both tools compare stored values, not visual appearance, so rows that look identical can survive if they differ in an invisible way - trailing spaces, non-breaking spaces, or numbers stored as text are the usual culprits. Why Excel's Remove Duplicates sometimes fails → · Full Google Sheets guide →