How to Remove Duplicates from a List in Bash

Q: What is the fastest way to remove duplicate lines in Bash?

sort -u file.txt is the standard one-liner - it sorts the file and removes duplicates in the same pass. The tradeoff is that your output ends up alphabetically sorted, which changes the original line order.

Q: Why does uniq not remove all my duplicates?

uniq only removes consecutive duplicate lines - it compares each line to the one immediately before it, not the whole file. If your duplicates aren't already adjacent, you need to sort first: sort file.txt | uniq , which is equivalent to sort -u file.txt .

Q: How do I remove duplicates without changing the line order?

Use awk's seen-tracking idiom: awk '!seen[$0]++' file.txt . It keeps a running record of every line already printed and only prints a line the first time it appears, leaving the rest of the order untouched.

Q: How does awk '!seen[$0]++' actually work?

seen[$0] is an associative array keyed by the current line's full text. The first time a line appears, seen[$0] is 0 (falsy), so !seen[$0] is true and the line prints; the ++ then increments that entry to 1. Every later occurrence of the same line has a truthy seen[$0] , so !seen[$0] is false and it's skipped.

Q: How do I remove duplicates based on only one column or field, not the whole line?

Change the awk key to the field you care about: awk '!seen[$1]++' file.txt deduplicates based on the first whitespace-separated field only, keeping the first full line seen for each distinct first field. Use -F',' to set a different field separator for CSV-like input.

The classic gotcha here is uniq - it looks like the obvious tool for the job, but it only catches duplicates that are already next to each other. Here's why that trips people up, and the order-preserving alternative when sorting isn't an option.

Quick answer

sort -u file.txt is the standard one-liner but reorders your data alphabetically as a side effect. If you need to preserve the original line order, use awk '!seen[$0]++' file.txt instead - it tracks lines already printed and skips repeats without sorting anything.

Need the same thing in another language?

See the full reference for Python, JavaScript, SQL, PowerShell, and more.

See every language →

The standard way: sort -u

sort -u file.txt

sort -u sorts the file and removes duplicate lines in the same pass - it's the fastest built-in option and works well when you don't care about preserving the original order, which is most of the time for things like sorted ID lists.

The uniq gotcha

# This does NOT remove all duplicates unless they're adjacent:
uniq file.txt

# This does, because sort makes duplicates adjacent first:
sort file.txt | uniq

uniq only compares each line to the one directly before it - it has no memory of lines further back. If "apple" appears on line 1 and again on line 50 with different lines in between, plain uniq keeps both copies. This is why uniq is almost always paired with sort first - and why sort -u exists as a shortcut for exactly that pairing.

Order-preserving: the awk idiom

awk '!seen[$0]++' file.txt

This is the standard way to dedupe without sorting. seen is an associative array keyed by the full line ($0). The first time a line is seen, seen[$0] is unset (treated as 0), so !seen[$0] is true and awk prints the line by default; the trailing ++ then bumps that entry to 1. Every subsequent occurrence of the same line finds a truthy seen[$0], so it's skipped - and the original line order is completely untouched.

Deduplicating by one field instead of the whole line

# Dedupe by first whitespace-separated field
awk '!seen[$1]++' file.txt

# Dedupe by first comma-separated field
awk -F',' '!seen[$1]++' file.csv

Swapping $0 for $1 keys the seen-tracker on just the first field rather than the entire line - useful for CSV-like data where you want to keep the first full row for each distinct ID or key, even if the rest of the row differs.

Which is faster on large files?

sort -u is generally faster for very large files - GNU sort can spill to disk and merge-sort data too large to fit in memory. The awk approach keeps every unique line in memory in the seen array, which can become a limiting factor on files with millions of distinct lines, though it's rarely an issue at typical list sizes.

Just need the answer, not the command line? Paste your list.

Open Remove Duplicate Lines →

Doing the same thing in PowerShell instead?

See the PowerShell equivalent - and its default case-insensitivity gotcha.

Read the PowerShell guide →

Frequently Asked Questions

What is the fastest way to remove duplicate lines in Bash?

sort -u file.txt is the standard one-liner - it sorts the file and removes duplicates in the same pass. The tradeoff is that your output ends up alphabetically sorted, which changes the original line order.

Why does uniq not remove all my duplicates?

uniq only removes consecutive duplicate lines - it compares each line to the one immediately before it, not the whole file. If your duplicates aren't already adjacent, you need to sort first: sort file.txt | uniq, which is equivalent to sort -u file.txt.

How do I remove duplicates without changing the line order?

Use awk's seen-tracking idiom: awk '!seen[$0]++' file.txt. It keeps a running record of every line already printed and only prints a line the first time it appears, leaving the rest of the order untouched.

How does awk '!seen[$0]++' actually work?

seen[$0] is an associative array keyed by the current line's full text. The first time a line appears, seen[$0] is 0 (falsy), so !seen[$0] is true and the line prints; the ++ then increments that entry to 1. Every later occurrence of the same line has a truthy seen[$0], so !seen[$0] is false and it's skipped.

How do I remove duplicates based on only one column or field, not the whole line?

Change the awk key to the field you care about: awk '!seen[$1]++' file.txt deduplicates based on the first whitespace-separated field only, keeping the first full line seen for each distinct first field. Use -F',' to set a different field separator for CSV-like input.

Is sort -u or the awk approach faster on very large files?

sort -u is typically faster for very large files since sort implementations are highly optimized and can use external merge sort for data larger than memory. The awk approach keeps every unique line in memory in the seen array, which can matter on files with millions of unique lines.

What if I don't want to write any shell commands at all?

Paste your list into the Remove Duplicate Lines tool and it dedupes instantly in the browser while preserving order, with a case-sensitivity option built in.