NN_descriptive_topic_name.ipynb

Examples:
00_setup_introduction.ipynb
01_data_loading_basics.ipynb
02_filtering_and_selection.ipynb
10_final_project.ipynb

## Loading Data

Let's load our sales data using pandas. Pandas is a powerful library for working
with tabular data (think of it like Excel, but with code!).

```python
# Import the pandas library and give it the short name 'pd'
# This is standard practice - everyone uses 'pd' for pandas
import pandas as pd

# Load the CSV file from our data folder
# CSV = "Comma Separated Values" - a simple data format
sales_data = pd.read_csv('data/sample/sales.csv')

# Let's see what we loaded!
print(f"Loaded {len(sales_data)} rows of sales data")
print(f"Columns: {list(sales_data.columns)}")

# Display the first few rows to get a sense of the data
sales_data.head()

**Characteristics**:
- Nearly every line has a comment
- Explains WHY, not just WHAT
- Shows intermediate results
- Uses encouraging language
- Breaks operations into smallest steps

### Intermediate Level (⭐⭐): "Fill-in-the-Blanks" Mode

Provide structure with student completion:

```markdown
## Exercise: Filter High-Value Sales

You've learned about boolean indexing. Now apply it to find high-value sales.

**Your task**: Complete the code below to find all sales over $1000.

```python
# TODO: Create a boolean mask for sales over $1000
# Hint: Use the 'amount' column and the > operator
high_value_mask = sales_data[___] ___ ___

# TODO: Apply the mask to filter the data
high_value_sales = sales_data[___]

# Check your work
print(f"Found {len(high_value_sales)} high-value sales")
assert len(high_value_sales) > 0, "Check your filter condition"

high_value_mask = sales_data['amount'] > 1000
high_value_sales = sales_data[high_value_mask]

**Characteristics**:
- Clear TODO markers
- Hints about approach
- Validation cells
- Solutions available (in separate file or later section)
- Assumes basic concept understanding

### Advanced Level (⭐⭐⭐): "Now-You-Try" Mode

Provide problem statement with minimal scaffolding:

```markdown
## Challenge: Optimize Query Performance

**Problem**: The current data filtering approach using pandas boolean indexing works
well for our sample dataset (10K rows), but production data has 10M+ rows.

**Your task**: Implement an optimized filtering strategy that:
1. Handles datasets too large for memory
2. Achieves <2 second query time for typical filters
3. Supports multiple filter conditions efficiently

**Constraints**:
- Data is stored in SQLite database (see `data/production.db`)
- Cannot load entire dataset into memory
- Must support these filter operations: >, <, ==, BETWEEN, IN

**Approach considerations**:
- SQL queries with appropriate indexes?
- Chunked processing with pandas?
- Dask or Vaex for out-of-core computing?
- Query optimization vs. data structure optimization?

**Deliverable**:
- Implementation with performance benchmarks
- Comparison of approaches tried
- Justification of final choice

**Evaluation**: Performance on provided test queries in `tests/query_benchmark.py`

## {Concept Name}

### What is it?
{Clear, simple definition}

### Why do we need it?
{Practical motivation - what problem does it solve?}

### How does it work?
{Explanation with visual aid if possible}

### Example: {Concrete scenario}
{Code demonstration with real data}

### Common Mistakes
{Pitfalls to avoid with examples}

### Practice
{Exercise applying the concept}

# 1. Show the problem/motivation
# 2. Demonstrate the solution
# 3. Explain what happened
# 4. Show the results

# Problem: Our dates are stored as strings, not datetime objects
print(type(sales_data['date'][0]))  # Shows: <class 'str'>
print(sales_data['date'].head())

# Solution: Convert to datetime
sales_data['date'] = pd.to_datetime(sales_data['date'])

# Verify the conversion worked
print(type(sales_data['date'][0]))  # Now shows: <class 'Timestamp'>

# Result: Now we can do date operations!
sales_data['month'] = sales_data['date'].dt.month
sales_data['year'] = sales_data['date'].dt.year
sales_data[['date', 'month', 'year']].head()

markdown_chars = sum(len(''.join(cell['source']))
                    for cell in notebook['cells']
                    if cell['cell_type'] == 'markdown')
code_chars = sum(len(''.join(cell['source']))
                for cell in notebook['cells']
                if cell['cell_type'] == 'code')
ratio = markdown_chars / (markdown_chars + code_chars)
# ratio should be >= 0.30

import shutil
shutil.copy('.claude/skills/notebook-creator/templates/beginner_template.ipynb',
            'notebooks/new_notebook.ipynb')

project/
├── data/
│   ├── raw/           # Original, immutable data
│   ├── processed/     # Cleaned, ready for analysis
│   ├── sample/        # Small datasets for examples (< 10 MB)
│   └── external/      # Third-party data with README attribution

# 1. Load
data = pd.read_csv('data/sample/sales.csv')

# 2. Validate
print(f"Loaded {len(data)} rows, {len(data.columns)} columns")
print(f"Columns: {list(data.columns)}")
assert len(data) > 0, "Data file is empty!"

# 3. Display
data.head()

# 4. Check types
data.dtypes

# Unexplained magic numbers
data[data['age'] > 18]

# No intermediate outputs
result = data.groupby('category')['amount'].sum().sort_values(ascending=False)

# Meaningless variable names
df2 = df1[df1['status'] == 'active']
df3 = df2.groupby('type').mean()

# Assuming code is self-explanatory (it rarely is for learners)
x = np.log1p(y)

# Explain the threshold
ADULT_AGE_THRESHOLD = 18  # Legal adult age
adults = data[data['age'] > ADULT_AGE_THRESHOLD]

# Show intermediate steps
category_totals = data.groupby('category')['amount'].sum()
sorted_totals = category_totals.sort_values(ascending=False)
print("Top categories by sales:")
sorted_totals.head()

# Use descriptive names
active_customers = all_customers[all_customers['status'] == 'active']
average_by_type = active_customers.groupby('customer_type').mean()

# Explain transformations
# Apply log transformation to handle right-skewed distribution
# Adding 1 before log prevents issues with zero values
log_transformed_amounts = np.log1p(sales_amounts)