ReddRadar
Agent skill
l0
L0 regularization for neural network sparsification and intelligent sampling - used in survey calibration. Triggers: "L0", "sparsification", "sample selection", "hard concrete", "sparse weights", "household selection", "gate", "survey sparsity", "l0-python"
Install this agent skill to your Project
npx add-skill https://github.com/PolicyEngine/policyengine-claude/tree/main/skills/data-science/l0-skill
SKILL.md
L0 Regularization
L0 is a PyTorch implementation of L0 regularization for neural network sparsification and intelligent sampling, used in PolicyEngine's survey calibration pipeline.
For Users
What is L0?
L0 regularization helps PolicyEngine create more efficient survey datasets by intelligently selecting which households to include in calculations.
Impact you see:
- Faster population impact calculations
- Smaller dataset sizes
- Maintained accuracy with fewer samples
Behind the scenes: When PolicyEngine shows population-wide impacts, L0 helps select representative households from the full survey, reducing computation time while maintaining accuracy.
For Analysts
What L0 Does
L0 provides intelligent sampling gates for:
- Household selection - Choose representative samples from CPS
- Feature selection - Identify important variables
- Sparse weighting - Create compact, efficient datasets
Used in PolicyEngine for:
- Survey calibration (via microcalibrate)
- Dataset sparsification in policyengine-us-data
- Efficient microsimulation
Installation
uv pip install l0-python
Quick Example: Sample Selection
from l0 import SampleGate
# Select 1,000 households from 10,000
gate = SampleGate(n_samples=10000, target_samples=1000)
selected_data, indices = gate.select_samples(data)
# Gates learn which samples are most informative
Integration with microcalibrate
from l0 import HardConcrete
from microcalibrate import Calibration
# L0 gates for household selection
gates = HardConcrete(
len(household_weights),
temperature=0.25,
init_mean=0.999 # Start with most households
)
# Use in calibration
# microcalibrate applies gates during weight optimization
For Contributors
Repository
Location: PolicyEngine/L0
Clone:
git clone https://github.com/PolicyEngine/L0
cd L0
Current Implementation
To see structure:
tree l0/
# Key modules:
ls l0/
# - hard_concrete.py - Core L0 distribution
# - layers.py - L0Linear, L0Conv2d
# - gates.py - Sample/feature gates
# - penalties.py - L0/L2 penalty computation
# - temperature.py - Temperature scheduling
To see specific implementations:
# Hard Concrete distribution (core algorithm)
cat l0/hard_concrete.py
# Sample gates (used in calibration)
cat l0/gates.py
# Neural network layers
cat l0/layers.py
Key Concepts
Hard Concrete Distribution:
- Differentiable approximation of L0 norm
- Allows gradient-based optimization
- Temperature controls sparsity level
To see implementation:
cat l0/hard_concrete.py
Sample Gates:
- Binary gates for sample selection
- Learn which samples are most informative
- Used in microcalibrate for household selection
Feature Gates:
- Select important features/variables
- Reduce dimensionality
- Maintain prediction accuracy
Usage in PolicyEngine
In microcalibrate (survey calibration):
from l0 import HardConcrete
# Create gates for household selection
gates = HardConcrete(
n_items=len(households),
temperature=0.25,
init_mean=0.999 # Start with almost all households
)
# Gates produce probabilities (0 to 1)
probs = gates()
# Apply to weights during calibration
masked_weights = weights * probs
In policyengine-us-data:
# See usage in data pipeline
grep -r "from l0 import" ../policyengine-us-data/
Temperature Scheduling
Controls sparsity over training:
from l0 import TemperatureScheduler, update_temperatures
scheduler = TemperatureScheduler(
initial_temp=1.0, # Start relaxed
final_temp=0.1, # End sparse
total_epochs=100
)
for epoch in range(100):
temp = scheduler.get_temperature(epoch)
update_temperatures(model, temp)
# ... training ...
To see implementation:
cat l0/temperature.py
L0L2 Combined Penalty
Prevents overfitting:
from l0 import compute_l0l2_penalty
# Combine L0 (sparsity) with L2 (regularization)
penalty = compute_l0l2_penalty(
model,
l0_lambda=1e-3, # Sparsity strength
l2_lambda=1e-4 # Weight regularization
)
loss = task_loss + penalty
Testing
Run tests:
make test
# Or
pytest tests/ -v --cov=l0
To see test patterns:
cat tests/test_hard_concrete.py
cat tests/test_gates.py
Advanced Usage
Hybrid Gates (L0 + Random)
from l0 import HybridGate
# Combine L0 selection with random sampling
hybrid = HybridGate(
n_items=10000,
l0_fraction=0.25, # 25% from L0
random_fraction=0.75, # 75% random
target_items=1000
)
selected, indices, types = hybrid.select(data)
Feature Selection
from l0 import FeatureGate
# Select top features
gate = FeatureGate(n_features=1000, max_features=50)
selected_data, feature_indices = gate.select_features(data)
# Get feature importance
importance = gate.get_feature_importance()
Mathematical Background
L0 norm:
- Counts non-zero elements
- Non-differentiable (discontinuous)
- Hard to optimize directly
Hard Concrete relaxation:
- Continuous, differentiable approximation
- Enables gradient descent
- "Stretches" binary distribution to allow gradients
Paper: Louizos, Welling, & Kingma (2017): "Learning Sparse Neural Networks through L0 Regularization" https://arxiv.org/abs/1712.01312
Related Packages
Uses L0:
- microcalibrate (survey weight calibration)
- policyengine-us-data (household selection)
See also:
- microcalibrate-skill - Survey calibration using L0
- policyengine-us-data-skill - Data pipeline integration
Resources
Repository: https://github.com/PolicyEngine/L0 Documentation: https://policyengine.github.io/L0/ Paper: https://arxiv.org/abs/1712.01312 PyPI: https://pypi.org/project/l0-python/
Didn't find tool you were looking for?