How to Find the Difference Between Two Pandas DataFrames

Q: How do I get rows that differ between the two DataFrames (symmetric difference)?

Concatenate both frames and drop rows that appear in both: pd.concat([df1, df2]).drop_duplicates(keep=False) . keep=False removes every row that has any duplicate at all, leaving only rows unique to one DataFrame or the other.

Q: What does DataFrame.compare() do differently?

df1.compare(df2) (pandas 1.1+) is built for aligned, same-shape DataFrames - it shows only the cells that differ, side by side, and requires both DataFrames to have the same columns and index. It answers 'what changed in this row' rather than 'which rows exist in one but not the other'.

Q: Does pandas care about DataFrame index alignment when comparing?

Yes - df1.compare(df2) and direct arithmetic/equality comparisons align on the index first, so two DataFrames with the same data but different index values or order won't line up correctly. merge and concat don't have this problem since they work off column values, not index position - reset the index with .reset_index(drop=True) if you need index-based comparison to behave predictably.

Q: How do I compare just one column across two DataFrames, like a list difference?

Use isin() : only_in_a = df1[~df1['id'].isin(df2['id'])] returns rows from df1 whose id doesn't appear anywhere in df2['id'] . This is the pandas equivalent of a plain Python set difference, scoped to a single column.

"Difference" between two DataFrames can mean rows missing from one, rows that changed, or just a single column compared like a plain list - pandas has a different tool built for each. Here are the four approaches worth knowing, and which question each one actually answers.

Quick answer

For 'which rows exist in df1 but not df2', use df.merge(df2, indicator=True, how='left').query('_merge == "left_only"'). For symmetric difference (rows unique to either frame), use pd.concat([df1, df2]).drop_duplicates(keep=False). For comparing just one column like a list difference, use df1[~df1['id'].isin(df2['id'])]. For seeing exactly which cells changed between two same-shape, aligned frames, use df1.compare(df2).

Need this for a plain list instead of a DataFrame?

See the full reference for Python, JavaScript, SQL, Java, and more.

See every language →

merge with indicator=True: the standard approach

import pandas as pd

df1 = pd.DataFrame({'id': [1, 2, 3], 'name': ['a', 'b', 'c']})
df2 = pd.DataFrame({'id': [2, 3, 4], 'name': ['b', 'c', 'd']})

merged = df1.merge(df2, on='id', how='outer', indicator=True)
only_in_df1 = merged.query('_merge == "left_only"')
print(only_in_df1)  # row with id=1

indicator=True adds a _merge column with values left_only, right_only, or both, telling you exactly where each row came from after an outer join. Filtering on left_only gives you rows unique to df1; right_only gives you rows unique to df2.

concat + drop_duplicates: symmetric difference

combined = pd.concat([df1, df2])
sym_diff = combined.drop_duplicates(keep=False)
print(sym_diff)  # rows that appear in exactly one of the two frames

Stacking both frames and dropping every row that has any duplicate (keep=False) leaves only rows unique to one side or the other - equivalent to a symmetric difference. This compares whole rows, so it requires both frames to have identical columns in the same order; mismatched dtypes on a shared column will prevent rows that look the same from being recognized as duplicates.

isin(): comparing a single column like a list

only_in_df1 = df1[~df1['id'].isin(df2['id'])]
print(only_in_df1)  # rows where id doesn't appear anywhere in df2['id']

This is the closest pandas equivalent to a plain Python set(a) - set(b), scoped to one column instead of comparing full rows. isin() builds a hash lookup internally, so it stays fast even on large columns - the ~ negates the boolean mask to keep only non-matches.

DataFrame.compare(): what changed, cell by cell

df1 = pd.DataFrame({'id': [1, 2, 3], 'price': [10, 20, 30]})
df2 = pd.DataFrame({'id': [1, 2, 3], 'price': [10, 25, 30]})

diff = df1.compare(df2)
print(diff)
#   price
#   self  other
# 1   20     25

compare() (pandas 1.1+) assumes both DataFrames share the same shape, columns, and index, and shows only the cells that differ - the self/other column pair makes it easy to see the old and new value side by side. It's the right tool for "did anything change between these two snapshots of the same table", not for finding rows that exist in one frame but not the other - mismatched indexes will raise an error or produce misleading results, so reset the index first if it isn't already aligned.

Just need the answer, not the code? Paste both lists.

Open Compare 2 Lists →

Need the same thing without pandas?

See the plain-Python guide, or the full reference for every other language.

See the Python guide →

Frequently Asked Questions

What is the standard way to find rows in one DataFrame but not another in pandas?

df.merge(df2, indicator=True, how='left').query('_merge == "left_only"') is the standard approach. The indicator=True flag adds a _merge column showing whether each row came from the left DataFrame only, the right only, or both.

How do I get rows that differ between the two DataFrames (symmetric difference)?

Concatenate both frames and drop rows that appear in both: pd.concat([df1, df2]).drop_duplicates(keep=False). keep=False removes every row that has any duplicate at all, leaving only rows unique to one DataFrame or the other.

What does DataFrame.compare() do differently?

df1.compare(df2) (pandas 1.1+) is built for aligned, same-shape DataFrames - it shows only the cells that differ, side by side, and requires both DataFrames to have the same columns and index. It answers 'what changed in this row' rather than 'which rows exist in one but not the other'.

Why does merge with indicator give unexpected results?

The most common cause is dtype mismatch on the join key - a column that's an integer in one DataFrame and a string in the other never matches, even for the same logical value, and pandas raises no error. Check df1[key].dtype and df2[key].dtype match before merging, and cast one side with .astype() if they don't.

Does pandas care about DataFrame index alignment when comparing?

Yes - df1.compare(df2) and direct arithmetic/equality comparisons align on the index first, so two DataFrames with the same data but different index values or order won't line up correctly. merge and concat don't have this problem since they work off column values, not index position - reset the index with .reset_index(drop=True) if you need index-based comparison to behave predictably.

How do NaN values affect DataFrame comparison?

NaN is never equal to NaN by IEEE floating-point rules, which pandas follows - so two rows that are otherwise identical but both contain NaN in the same column can appear as 'different' under naive equality checks. DataFrame.equals() treats NaNs in the same position as equal (unlike ==), which is usually the behavior you actually want.

How do I compare just one column across two DataFrames, like a list difference?

Use isin(): only_in_a = df1[~df1['id'].isin(df2['id'])] returns rows from df1 whose id doesn't appear anywhere in df2['id']. This is the pandas equivalent of a plain Python set difference, scoped to a single column.