How to Remove Duplicate Rows in Pandas

pandas' drop_duplicates() handles whole-row deduplication out of the box, but the two parameters that make it flexible - subset and keep - are worth understanding properly before you run it on real data.

The default: whole-row deduplication

import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Alice'],
    'email': ['alice@x.com', 'bob@x.com', 'alice@x.com'],
})

df_unique = df.drop_duplicates()
print(df_unique)
# Only the first Alice row remains - both columns matched exactly

Without a subset, drop_duplicates() considers every column - two rows only count as duplicates if all of their values match exactly. keep='first' is the default, so the first occurrence of each duplicate group survives.

Deduplicating on specific columns: subset

df_unique = df.drop_duplicates(subset=['email'])

Passing subset narrows the comparison to just the listed columns - two rows with the same email but a different name are still treated as duplicates and one is dropped, since only email is being compared.

Choosing which row survives: keep

df.drop_duplicates(subset=['email'], keep='last')   # keep the last occurrence
df.drop_duplicates(subset=['email'], keep=False)    # keep NEITHER - drop all involved

keep='last' is handy when your data is chronologically ordered and you want the most recent record per key. keep=False is different from the other two - instead of keeping one row per duplicate group, it removes every row that has any duplicate at all, leaving only genuinely unique rows behind.

Inspecting duplicates before deleting

dupes = df[df.duplicated(subset=['email'], keep=False)]
print(dupes)  # every row involved in a duplicate group, for review

duplicated() returns a boolean mask instead of dropping anything, using the same subset/keep logic as drop_duplicates(). Filtering the DataFrame with this mask first lets you see exactly what would be removed before committing to the deletion.

Ignoring case or whitespace when comparing

df['email_norm'] = df['email'].str.strip().str.lower()
df_unique = df.drop_duplicates(subset=['email_norm']).drop(columns=['email_norm'])

drop_duplicates() compares exact string values, so "Alice@X.com" and " alice@x.com " would not be treated as duplicates without normalizing first. Building a normalized helper column, deduping on that, then dropping the helper column keeps the original casing and whitespace in your final output.

Counting occurrences instead of removing them

print(df['email'].value_counts())

Sometimes you want to understand the duplication before acting on it - value_counts() returns how many times each distinct value appears, sorted from most to least frequent, which is useful for spotting which values are duplicated the most before deciding on a subset/keep strategy.