pandas

Imports

import pandas as pd
from pandas import DataFrame, Series, Index
from pandas import read_csv, read_excel, read_json, read_sql, read_parquet, read_pickle
from pandas import Timestamp, Timedelta, Period
from pandas import get_option, set_option, option_context
from pandas import NA, NaT
from pandas import concat, merge, pivot_table, melt
from pandas import cut, qcut, get_dummies
from pandas import to_datetime, to_timedelta, to_numeric
from pandas import isna, isnull, notna, notnull
from pandas import date_range, timedelta_range, period_range, interval_range

Core Patterns

Creating DataFrames ✅ Current

import pandas as pd

# From dictionary
df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35],
    'city': ['NYC', 'SF', 'LA']
})

# From CSV file
df = pd.read_csv('data.csv', index_col=0)

# From Excel file
df = pd.read_excel('data.xlsx', sheet_name='Sheet1')

# From JSON
df = pd.read_json('data.json', orient='records')

# From Parquet
df = pd.read_parquet('data.parquet')

# From pickle
df = pd.read_pickle('data.pkl')

• DataFrames are two-dimensional labeled data structures with columns of potentially different types
• Use read_* functions for loading data from various file formats
• Specify index_col to set which column becomes the row index
• copy parameter default changed in v3.0 - see Migration section

Creating Series ✅ Current

import pandas as pd

# From list with index
s = pd.Series([10, 20, 30], index=['a', 'b', 'c'], name='values')

# From dictionary
s = pd.Series({'a': 10, 'b': 20, 'c': 30})

# Extracting from DataFrame
df = pd.DataFrame({'col1': [1, 2, 3], 'col2': [4, 5, 6]})
s = df['col1']  # Returns Series

• Series are one-dimensional labeled arrays capable of holding any data type
• Index provides labels for fast lookup
• Series maintain their name attribute for identification
• copy parameter default changed in v3.0 - see Migration section

Data Selection and Filtering ✅ Current

import pandas as pd

df = pd.DataFrame({
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35],
    'score': [85, 90, 88]
})

# Select single column (returns Series)
ages = df['age']

# Select multiple columns (returns DataFrame)
subset = df[['name', 'age']]

# Boolean filtering
adults = df[df['age'] >= 30]

# Multiple conditions
high_scorers = df[(df['age'] >= 25) & (df['score'] >= 85)]

# Using .loc for label-based indexing
row = df.loc[0]  # First row by index label
value = df.loc[0, 'name']  # Specific cell

# Using .iloc for position-based indexing
row = df.iloc[0]  # First row by position
value = df.iloc[0, 1]  # First row, second column

• Use bracket notation for column selection
• Boolean indexing filters rows based on conditions
• .loc[] uses labels, .iloc[] uses integer positions
• Combine conditions with & (and), | (or), ~ (not) - wrap each condition in parentheses
• Note: Copy-on-Write behavior changed in v3.0 - selections always return copies

Working with Time Series ✅ Current

import pandas as pd

# Create Timestamp
ts = pd.Timestamp('2024-01-15 14:30:00')
ts = pd.Timestamp(year=2024, month=1, day=15, hour=14, minute=30)

# Create DatetimeIndex
dates = pd.date_range('2024-01-01', periods=10, freq='D')
df = pd.DataFrame({'value': range(10)}, index=dates)

# Create Timedelta
td = pd.Timedelta('2 days')
td = pd.Timedelta(days=2, hours=3)

# Create TimedeltaIndex
deltas = pd.timedelta_range(start='1 day', periods=5, freq='D')

# Create Period
p = pd.Period('2024-01', freq='M')

# Create PeriodIndex
periods = pd.period_range('2024-01', periods=12, freq='M')

# Convert to datetime
df['date'] = pd.to_datetime(df['date_string'])

# Convert to timedelta
df['duration'] = pd.to_timedelta(df['duration_string'])

• Timestamp replaces Python's datetime.datetime with nanosecond precision
• DatetimeIndex enables time-based indexing and slicing
• Timedelta represents duration between two dates or times
• Period represents a span of time at a particular frequency
• Use to_datetime() and to_timedelta() for conversions

Missing Values ✅ Current

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'A': [1, np.nan, 3],
    'B': [4, 5, pd.NA],
    'C': [7, pd.NaT, 9]
})

# Detect missing values
has_nulls = df.isna()  # or df.isnull()
has_values = df.notna()  # or df.notnull()

# Check for any nulls
any_nulls = df['A'].isna().any()

# Drop rows with any null values
df_clean = df.dropna()

# Fill null values
df_filled = df.fillna(0)
df_filled = df.fillna(method='ffill')  # Forward fill

# Use pandas NA for missing values
value = pd.NA  # Scalable missing value indicator
nat = pd.NaT  # Not-a-Time for datetime/timedelta

• Use .isna() or .isnull() to detect missing values (they are aliases)
• Use .notna() or .notnull() to detect non-missing values
• pd.NA is the recommended missing value indicator for nullable dtypes
• pd.NaT is used specifically for datetime/timedelta missing values
• Never compare to NaN directly with == - always use .isna()

Combining DataFrames ✅ Current

import pandas as pd

df1 = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})
df2 = pd.DataFrame({'A': [5, 6], 'B': [7, 8]})

# Concatenate along rows (vertical stack)
result = pd.concat([df1, df2], axis=0)

# Concatenate along columns (horizontal stack)
result = pd.concat([df1, df2], axis=1)

# Merge (join) DataFrames
left = pd.DataFrame({'key': ['A', 'B'], 'value': [1, 2]})
right = pd.DataFrame({'key': ['A', 'B'], 'value': [3, 4]})

# Inner join
merged = pd.merge(left, right, on='key', how='inner')

# Left join
merged = pd.merge(left, right, on='key', how='left')

# Outer join
merged = pd.merge(left, right, on='key', how='outer')

• concat() stacks DataFrames along an axis
• merge() performs database-style joins
• Use how parameter to specify join type: 'inner', 'left', 'right', 'outer'
• Use on parameter to specify join key(s)

Reshaping Data ✅ Current

import pandas as pd

# Pivot table
df = pd.DataFrame({
    'date': ['2024-01', '2024-01', '2024-02', '2024-02'],
    'product': ['A', 'B', 'A', 'B'],
    'sales': [100, 150, 120, 180]
})

pivot = pd.pivot_table(df, values='sales', index='date', columns='product')

# Melt (unpivot) from wide to long format
melted = pd.melt(df, id_vars=['date'], value_vars=['product'], 
                 var_name='category', value_name='value')

# Create dummy variables (one-hot encoding)
df = pd.DataFrame({'color': ['red', 'blue', 'red', 'green']})
dummies = pd.get_dummies(df['color'], prefix='color')

• pivot_table() creates spreadsheet-style pivot tables
• melt() transforms wide format to long format
• get_dummies() creates dummy/indicator variables for categorical data

Binning and Discretization ✅ Current

import pandas as pd

data = pd.Series([1, 7, 5, 4, 6, 3, 9, 2, 8])

# Cut into equal-width bins
bins = pd.cut(data, bins=3, labels=['low', 'medium', 'high'])

# Cut with custom bin edges
bins = pd.cut(data, bins=[0, 3, 7, 10], labels=['low', 'medium', 'high'])

# Quantile-based discretization (equal-sized bins)
quantiles = pd.qcut(data, q=3, labels=['low', 'medium', 'high'])

• cut() bins values into discrete intervals with equal width
• qcut() bins values based on quantiles (equal frequency)
• Use labels parameter to assign custom category names

Type Conversion ✅ Current

import pandas as pd

df = pd.DataFrame({
    'numbers': ['1', '2', '3'],
    'dates': ['2024-01-01', '2024-01-02', '2024-01-03']
})

# Convert to numeric
df['numbers'] = pd.to_numeric(df['numbers'])

# Convert to datetime
df['dates'] = pd.to_datetime(df['dates'])

# Convert column dtype
df['numbers'] = df['numbers'].astype('int64')

# Convert to categorical
df['category'] = df['category'].astype('category')

• to_numeric() converts to numeric types with error handling
• to_datetime() converts to datetime with flexible parsing
• astype() explicitly converts dtypes
• Use errors='coerce' parameter to handle conversion failures

Configuring Display Options ✅ Current

import pandas as pd

# Get current option value
max_rows = pd.get_option('display.max_rows')

# Set option value
pd.set_option('display.max_rows', 100)
pd.set_option('display.max_columns', 50)
pd.set_option('display.precision', 2)

# Temporarily set options with context manager
with pd.option_context('display.max_rows', 10, 'display.max_columns', 5):
    print(df)  # Uses temporary settings

# Outside context, original settings restored

# Reset option to default
pd.reset_option('display.max_rows')

# Describe available options
pd.describe_option('display')  # All display options
pd.describe_option('display.max_rows')  # Specific option

• Use get_option() and set_option() for global configuration changes
• option_context() provides temporary settings that restore automatically
• Common options: display.max_rows, display.max_columns, display.precision, display.width

Configuration

Display Settings

# Default values
pd.get_option('display.max_rows')  # 60
pd.get_option('display.max_columns')  # 20
pd.get_option('display.width')  # 80
pd.get_option('display.precision')  # 6

# Common customizations
pd.set_option('display.max_rows', None)  # Show all rows
pd.set_option('display.max_columns', None)  # Show all columns
pd.set_option('display.float_format', '{:.2f}'.format)  # Format floats

File Reading Options

# CSV reading with common parameters
df = pd.read_csv(
    'data.csv',
    sep=',',  # Delimiter (default: ',')
    header=0,  # Row to use as column names (default: 'infer')
    index_col=0,  # Column to use as row index
    usecols=['col1', 'col2'],  # Columns to read
    dtype={'col1': int, 'col2': str},  # Column data types
    parse_dates=['date_col'],  # Parse as datetime
    na_values=['NA', 'null'],  # Additional NA values
    encoding='utf-8',  # File encoding
    nrows=1000,  # Number of rows to read
    skiprows=5  # Rows to skip at start
)

Index and Data Types

# Creating typed indexes
idx = pd.Index([1, 2, 3], dtype='int64', name='id')
cat_idx = pd.CategoricalIndex(['A', 'B', 'C'], name='category')
range_idx = pd.RangeIndex(start=0, stop=10, step=2)
multi_idx = pd.MultiIndex.from_tuples([('A', 1), ('A', 2), ('B', 1)])

# Creating categoricals
cat = pd.Categorical(['A', 'B', 'A', 'C'], categories=['A', 'B', 'C'], ordered=True)

# Creating intervals
interval = pd.Interval(left=0, right=5, closed='right')
interval_idx = pd.IntervalIndex.from_breaks([0, 1, 2, 3])

Pitfalls

Wrong: Using chained assignment

import pandas as pd
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# This may not work as expected and raises SettingWithCopyWarning
df[df['A'] > 1]['B'] = 99

Why: Chained indexing creates intermediate copies, so assignment may not affect the original DataFrame.

Right: Use .loc for assignment

import pandas as pd
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Correctly modifies the original DataFrame
df.loc[df['A'] > 1, 'B'] = 99

Wrong: Iterating over DataFrame rows with loops

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Very slow for large DataFrames
results = []
for i in range(len(df)):
    results.append(df.iloc[i]['A'] + df.iloc[i]['B'])

Why: Row-by-row iteration is extremely slow and defeats pandas' vectorization.

Right: Use vectorized operations

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Much faster - operates on entire columns at once
df['result'] = df['A'] + df['B']

Wrong: Not specifying dtype when creating structures

import pandas as pd

# Mixed types cause object dtype (slow operations)
df = pd.DataFrame({'id': ['1', '2', '3'], 'value': [10, 20, 30]})
# df['id'].dtype is 'object', not efficient

Why: Object dtype prevents optimized operations and uses more memory.

Right: Specify dtypes explicitly or convert after creation

import pandas as pd

# Specify dtype at creation
df = pd.DataFrame({
    'id': pd.Series([1, 2, 3], dtype='int64'),
    'value': [10, 20, 30]
})

# Or convert after creation
df = pd.DataFrame({'id': ['1', '2', '3'], 'value': [10, 20, 30]})
df['id'] = df['id'].astype('int64')

Wrong: Using inplace=True for method chaining

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Cannot chain - inplace returns None
result = df.drop(columns=['B'], inplace=True).reset_index()  # Error!

Why: Methods with inplace=True return None, breaking method chains.

Right: Avoid inplace, assign results

import pandas as pd

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Chain operations naturally
result = df.drop(columns=['B']).reset_index(drop=True)

# Or assign back if needed
df = df.drop(columns=['B'])

References

• homepage
• documentation
• repository
• changelog
• PyPI

Migration from v2.x

Copy Semantics Changed

v2.x behavior:

df = pd.DataFrame({'A': [1, 2, 3]})
df2 = df[['A']]  # Creates view in many cases
df2.iloc[0, 0] = 99  # May modify original df

v3.0 behavior:

df = pd.DataFrame({'A': [1, 2, 3]})
df2 = df[['A']]  # Always creates copy (Copy-on-Write enforced)
df2.iloc[0, 0] = 99  # Never modifies original df

Migration: Copy-on-Write (CoW) is now the default and only mode in pandas 3.0. All indexing operations that return a subset of data will return a new copy. If you need to modify the original DataFrame, use direct assignment with .loc[] or .iloc[] rather than chaining operations.

Constructor Parameter Changes

The copy parameter behavior has changed in DataFrame and Series constructors:

v2.x:

df = pd.DataFrame(data, copy=True)  # Explicitly copy data

v3.0:

df = pd.DataFrame(data, copy=None)  # Default changed to None
# copy=None respects Copy-on-Write semantics
# copy=True still available but rarely needed with CoW

Migration: The default copy=None is sufficient for most cases under CoW. Only use copy=True if you need to ensure immediate physical copying of data.

Deprecated Parameters Removed

Several long-deprecated parameters have been removed in v3.0:

• infer_datetime_format in read_csv() and similar functions (datetime format inference is now automatic)
• Various convert_* parameters in IO functions

Migration: Remove these parameters from your code. Datetime format inference happens automatically in v3.0.

Index Constructor Changes

The tupleize_cols parameter default behavior may affect MultiIndex creation:

# May need explicit handling for tuple columns
idx = pd.Index(data, tupleize_cols=True)  # Explicit if needed

API Breaking Changes

Refer to the full changelog for comprehensive breaking changes: https://pandas.pydata.org/pandas-docs/stable/whatsnew/v3.0.0.html

Key areas to review:

• Copy-on-Write is now mandatory (no opt-out)
• Index and MultiIndex behavior changes
• DataFrame/Series constructor parameter defaults changed
• IO function parameter updates
• Deprecated method removals

API Reference

Core Data Structures

• DataFrame(data=None, index=None, columns=None, dtype=None, copy=None) - Two-dimensional labeled data structure with columns of potentially different types
• Series(data=None, index=None, dtype=None, name=None, copy=None) - One-dimensional labeled array capable of holding any data type

Index Types

• Index(data=None, dtype=None, copy=False, name=None, tupleize_cols=True) - Immutable sequence used for indexing and alignment
• RangeIndex(start=None, stop=None, step=None, dtype=None, copy=False, name=None) - Memory-efficient index for monotonic integer ranges
• MultiIndex(levels=None, codes=None, sortorder=None, names=None, dtype=None, copy=False, name=None, verify_integrity=True) - Multi-level or hierarchical index object
• DatetimeIndex(data=None, freq=None, tz=None, normalize=False, closed=None, ambiguous='raise', dayfirst=False, yearfirst=False, dtype=None, copy=False, name=None) - Immutable ndarray of datetime64 data
• TimedeltaIndex(data=None, unit=None, freq=None, closed=None, dtype=None, copy=False, name=None) - Immutable ndarray of timedelta64 data
• PeriodIndex(data=None, ordinal=None, freq=None, dtype=None, copy=False, name=None) - Immutable ndarray holding ordinal values indicating regular periods in time
• CategoricalIndex(data=None, categories=None, ordered=None, dtype=None, copy=False, name=None) - Index based on categorical data
• IntervalIndex(data, closed=None, dtype=None, copy=False, name=None, verify_integrity=True) - Index of intervals closed on the same side

Scalars

• Timestamp(ts_input=None, freq=None, tz=None, unit=None, year=None, month=None, day=None, hour=None, minute=None, second=None, microsecond=None, nanosecond=None, tzinfo=None, fold=None) - Pandas replacement for datetime.datetime with nanosecond precision
• **Timedelta(value=None, unit=None, kwargs) - Duration representing difference between two dates or times
• Period(value=None, freq=None, ordinal=None, year=None, month=None, quarter=None, day=None, hour=None, minute=None, second=None) - Represents a time period at a particular frequency
• Interval(left, right, closed='right') - Immutable object representing an interval
• NA - Scalar missing value indicator
• NaT - Pandas Not-A-Time, used to represent null dates/times

Data Types

• Categorical(values, categories=None, ordered=None, dtype=None, copy=True) - Represents categorical variable for memory efficiency and operations

IO Functions

• read_csv(filepath_or_buffer, sep=',', delimiter=None, header='infer', names=None, index_col=None, usecols=None, dtype=None, ...) - Read CSV file into DataFrame
• read_excel(io, sheet_name=0, header=0, names=None, index_col=None, usecols=None, dtype=None, ...) - Read Excel file into DataFrame
• read_json(path_or_buf, orient=None, typ='frame', dtype=None, ...) - Convert JSON to DataFrame or Series
• read_sql(sql, con, index_col=None, coerce_float=True, params=None, parse_dates=None, columns=None, chunksize=None, dtype_backend=None, dtype=None) - Read SQL query or table into DataFrame
• **read_parquet(path, engine='auto', columns=None, storage_options=None, use_nullable_dtypes=False, dtype_backend=None, filesystem=None, filters=None, kwargs) - Load parquet object into DataFrame
• read_pickle(filepath_or_buffer, compression='infer', storage_options=None) - Load pickled pandas object from file
• to_pickle(obj, filepath_or_buffer, compression='infer', protocol=5, storage_options=None) - Pickle (serialize) object to file

Data Manipulation

• concat(objs, axis=0, join='outer', ignore_index=False, keys=None, levels=None, names=None, verify_integrity=False, sort=False, copy=True) - Concatenate pandas objects along a particular axis
• merge(left, right, how='inner', on=None, left_on=None, right_on=None, left_index=False, right_index=False, sort=False, suffixes=('_x', '_y'), copy=True, indicator=False, validate=None) - Merge DataFrame or named Series objects with database-style join
• pivot_table(data, values=None, index=None, columns=None, aggfunc='mean', fill_value=None, margins=False, dropna=True, margins_name='All', observed=False, sort=True) - Create spreadsheet-style pivot table as DataFrame
• melt(frame, id_vars=None, value_vars=None, var_name=None, value_name='value', col_level=None, ignore_index=True) - Unpivot DataFrame from wide to long format
• cut(x, bins, right=True, labels=None, retbins=False, precision=3, include_lowest=False, duplicates='raise', ordered=True) - Bin values into discrete intervals
• qcut(x, q, labels=None, retbins=False, precision=3, duplicates='raise') - Quantile-based discretization function
• get_dummies(data, prefix=None, prefix_sep='_', dummy_na=False, columns=None, sparse=False, drop_first=False, dtype=None) - Convert categorical variable into dummy/indicator variables

Type Conversion

• to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False, utc=None, format=None, exact=True, unit=None, infer_datetime_format=False, origin='unix', cache=True) - Convert argument to datetime
• to_timedelta(arg, unit=None, errors='raise') - Convert argument to timedelta
• to_numeric(arg, errors='raise', downcast=None, dtype_backend=None) - Convert argument to numeric type

Missing Value Detection

• isna(obj) - Detect missing values for an array-like object
• isnull(obj) - Detect missing values (alias of isna)
• notna(obj) - Detect non-missing values for an array-like object
• notnull(obj) - Detect non-missing values (alias of notna)

Index Generation

• **date_range(start=None, end=None, periods=None, freq=None, tz=None, normalize=False, name=None, inclusive='both', kwargs) - Return fixed frequency DatetimeIndex
• timedelta_range(start=None, end=None, periods=None, freq=None, name=None, closed=None) - Return fixed frequency TimedeltaIndex
• period_range(start=None, end=None, periods=None, freq=None, name=None) - Return fixed frequency PeriodIndex
• interval_range(start=None, end=None, periods=None, freq=None, name=None, closed='right') - Return fixed frequency IntervalIndex

Other Functions

• array(data, dtype=None, copy=True) - Create an ExtensionArray
• factorize(values, sort=False, use_na_sentinel=True, size_hint=None) - Encode the object as an enumerated type or categorical variable

Configuration

• get_option(pat: str) - Get value of single configuration option
• set_option(pat: str, value: Any) - Set value of single configuration option
• reset_option(pat: str) - Reset option to default value
• *option_context(args) - Context manager for temporary option changes
• describe_option(pat: str = '', _print_desc: bool = True) - Get description of configuration option
• options - Configuration options accessor (property)