Comparing
DataFrames with assert_frame_equal in Python
In data
analysis and manipulation, working with DataFrames is a core task. As you work
with large datasets, there are times when you may need to compare two
DataFrames for equality. In Python, the pandas library provides a powerful
utility function called assert_frame_equal to help you compare two DataFrames
efficiently.
In this blog
post, we’ll explore how to use assert_frame_equal to compare DataFrames and
discuss common issues that might arise while using it.
What is assert_frame_equal?
python
CopyEdit
import
pandas as pd
from
pandas.testing import assert_frame_equal
Syntax of
assert_frame_equal
The basic
syntax of assert_frame_equal is as follows:
python
CopyEdit
assert_frame_equal(left,
right, check_dtype=True, check_index_type='equiv', check_column_type='equiv',
check_like=False, check_exact=False, check_datetimelike_compat=False,
check_less_precise=False, rtol=1e-5, atol=1e-8, obj='DataFrame')
Key
Parameters:
- left, right: The two DataFrames to compare.
- check_dtype: Whether to check the data type
of each column. Default is True.
- check_index_type: Whether to check the index
types. Default is 'equiv', which means it checks for the same structure
but allows different types of indexes (like RangeIndex vs Index).
- check_column_type: Whether to check the column
types. Default is 'equiv'.
- check_like: If True, the comparison is
done by ignoring the order of the rows and columns.
- check_exact: If True, checks for exact
equality (no floating-point tolerance).
- rtol and atol: These are
relative and absolute tolerance for comparing floating point values
(useful for numerical comparisons).
- obj: The name for the comparison
object (defaults to 'DataFrame').
Example:
Basic Comparison
Let's look
at an example where we compare two DataFrames using assert_frame_equal.
python
CopyEdit
import
pandas as pd
from
pandas.testing import assert_frame_equal
# Create two
identical DataFrames
df1 =
pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 6]
})
df2 =
pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 6]
})
# Compare
the DataFrames
assert_frame_equal(df1,
df2)
print("The
DataFrames are equal!")
In this
case, df1 and df2 are identical, so the assert_frame_equal function will not
raise an error, and it will print "The DataFrames are equal!".
Example:
Handling Differences Between DataFrames
If the two
DataFrames are not identical, assert_frame_equal will raise an error and
provide useful information about the differences.
python
CopyEdit
df1 =
pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 6]
})
df2 =
pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 7] # Different value in the last element of
column 'B'
})
try:
assert_frame_equal(df1, df2)
except
AssertionError as e:
print(f"Error: {e}")
Output:
less
CopyEdit
Error:
DataFrame.iloc[:, 1] are different
DataFrame.shape:
(3, 2) vs (3, 2)
DataFrame.columns:
Index(['A', 'B'], dtype='object') vs Index(['A', 'B'], dtype='object')
DataFrame.iloc[:,
1] values are different (100.0 %)
[ 4 5 6]
vs [4 5 7]
Here, the
difference between df1 and df2 is in the last element of column 'B'. assert_frame_equal
clearly shows the discrepancy, making it easy to troubleshoot.
Useful
Parameters to Handle Different Scenarios
1. Allowing
Small Numerical Differences:
If your
DataFrames contain floating-point numbers and you want to account for small
numerical errors (due to rounding), you can use the check_exact=False parameter
or set tolerances with rtol and atol.
python
CopyEdit
df1 =
pd.DataFrame({
'A': [1.0001, 2.0001, 3.0001],
'B': [4.0001, 5.0001, 6.0001]
})
df2 =
pd.DataFrame({
'A': [1.0002, 2.0002, 3.0002],
'B': [4.0002, 5.0002, 6.0002]
})
# Allow
small numerical differences
assert_frame_equal(df1,
df2, check_exact=False, rtol=1e-4, atol=1e-6)
print("The
DataFrames are approximately equal!")
In this
case, the two DataFrames are very close in value, and assert_frame_equal will
pass because the relative and absolute tolerance is specified.
2. Ignoring
Row and Column Order:
If you want
to compare two DataFrames but do not care about the order of the rows or
columns (i.e., a row-wise or column-wise comparison), you can use the check_like=True
parameter.
python
CopyEdit
df1 =
pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 6]
})
df2 =
pd.DataFrame({
'B': [4, 5, 6],
'A': [1, 2, 3] # Columns are swapped
})
# Compare
ignoring column order
assert_frame_equal(df1,
df2, check_like=True)
print("The
DataFrames are equal (ignoring column order)!")
This will
pass even though the columns in df1 and df2 are in different orders.
3. Ignoring
Index Type:
If the
DataFrames have the same data but different types of indexes (e.g., RangeIndex
vs Index), you can use the check_index_type parameter to allow for index-type
differences.
python
CopyEdit
df1 =
pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 6]
})
df2 =
pd.DataFrame({
'A': [1, 2, 3],
'B': [4, 5, 6]
}, index=['x',
'y', 'z'])
# Compare
while ignoring the index type
assert_frame_equal(df1,
df2, check_index_type='ignore')
print("The
DataFrames are equal (ignoring index type)!")
Here, the
DataFrames have the same data, but df2 uses a custom index, and assert_frame_equal
will pass by ignoring the index type.
Common
Issues
- Data Types Mismatch: If the data types of columns
are different between two DataFrames (e.g., one column is int64 and the
other is float64), assert_frame_equal will raise an error. Use the check_dtype
parameter to handle or ignore this.
- Index Issues: If the indexes of two
DataFrames differ, assert_frame_equal will highlight the difference unless
check_index_type='ignore' is used.
- Floating Point Precision Issues: When comparing floating-point
values, minor differences can arise due to precision. Use check_exact=False,
or adjust the rtol and atol parameters to allow small discrepancies.
Conclusion
assert_frame_equal
is an excellent utility in the pandas library that makes it easy to compare two
DataFrames for equality. Whether you’re working with data for testing purposes
or validating transformations, this function can help you pinpoint differences
quickly.
By
leveraging parameters like check_like, check_dtype, and tolerance settings, you
can make your DataFrame comparisons more flexible and robust.
If you are
dealing with data discrepancies, understanding how to use assert_frame_equal
can save you a lot of time and frustration in debugging and ensuring data
integrity.
Happy
coding!
