Agent skill
multi-format-config-management
Implement configuration management supporting TOML/YAML/JSON formats with inheritance and runtime overrides
Stars
163
Forks
31
Install this agent skill to your Project
npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/data/multi-format-config-management
Metadata
Additional technical details for this skill
- tags
- config,parsing,yaml,toml,json,inheritance,overrides
- version
- 1
- category
- infrastructure
- complexity
- medium
- created at
- 2026-01-29T22:56:19.276Z
- updated at
- 2026-01-29T22:56:19.276Z
SKILL.md
Purpose
Implement comprehensive configuration management that supports multiple file formats (TOML, YAML, JSON), environment inheritance, and runtime CLI overrides.
When To Use
- User requests configuration management with multi-format support
- User needs config inheritance between environments
- User requires runtime configuration overrides
- Ticket specifies "config file formats" as a requirement
Architecture
Format Detection
- Search for config files in preference order: TOML → YAML → JSON
- Detect format from file extension (.toml, .yaml, .yml, .json)
- Map extensions to appropriate parsers: tomllib, yaml.safe_load, json.load
- Provide helpful error messages indicating which format failed
Configuration Loading
python
from pathlib import Path
import tomllib
import json
try:
import yaml
from yaml import YAMLError
YAML_AVAILABLE = True
except ImportError:
YAML_AVAILABLE = False
YAMLError = type("YAMLError")
def _load_file_data(path: Path) -> dict[str, Any]:
suffix = path.suffix.lower()
if suffix == ".toml":
with path.open("rb") as f:
return tomllib.load(f)
elif suffix in (".yaml", ".yml"):
if not YAML_AVAILABLE:
raise ValueError("YAML support requires PyYAML")
with path.open("r", encoding="utf-8") as f:
return yaml.safe_load(f)
elif suffix == ".json":
with path.open("r", encoding="utf-8") as f:
return json.load(f)
else:
raise ValueError(f"Unsupported format: {suffix}")
def find_config_file(search_path: Path | None = None) -> Path | None:
search_path = search_path or Path.cwd()
config_names = [
"vibepiper.toml",
"vibepiper.yaml",
"vibepiper.yml",
"vibepiper.json",
]
for path in [search_path, *search_path.parents]:
for config_name in config_names:
config_file = path / config_name
if config_file.exists():
return config_file
return None
Exception Handling for Strict MyPy
- Catch general Exception, then use isinstance() to distinguish specific errors:
python
try:
data = _load_file_data(path)
except Exception as e:
if isinstance(e, (tomllib.TOMLDecodeError, json.JSONDecodeError, YAMLError)):
msg = f"Invalid {_get_format_name(path)} syntax"
else:
msg = "Failed to read configuration file"
raise ConfigLoadError(msg, path=path, cause=e) from e
Environment Inheritance
Two-Pass Parsing
- First pass: Parse all environment configs into dictionary
- Extract
inheritsfield from each environment - Second pass: Merge child with parent using inheritance chain
Merge Logic
- Child values override parent values
- Unspecified fields inherit from parent
- Use
_merge_additional_config()for custom fields
python
def _merge_additional_config(parent: dict[str, Any], child: dict[str, Any]) -> dict[str, Any]:
merged = parent.copy()
merged.update(child)
return merged
# Inheritance parsing
environments_data = data.get("environments", {})
# First pass: parse all
for env_name, env_data in environments_data.items():
environments[env_name] = EnvironmentConfig(...)
# Second pass: apply inheritance
for env_name, env_data in environments_data.items():
inherits_from = env_data.get("inherits")
if inherits_from:
if inherits_from not in environments:
raise ValueError(f"Environment '{env_name}' inherits from '{inherits_from}' which doesn't exist")
parent_env = environments[inherits_from]
child_env = environments[env_name]
environments[env_name] = EnvironmentConfig(
io_manager=child_env.io_manager or parent_env.io_manager,
log_level=child_env.log_level or parent_env.log_level,
parallelism=child_env.parallelism or parent_env.parallelism,
bucket=child_env.bucket or parent_env.bucket,
region=child_env.region or parent_env.region,
endpoint=child_env.endpoint or parent_env.endpoint,
credentials_path=child_env.credentials_path or parent_env.credentials_path,
additional_config=_merge_additional_config(
parent_env.additional_config,
child_env.additional_config
),
)
Runtime Overrides
Architecture
- Store CLI overrides separately in Config object
- Add
apply_overridesparameter toget_environment()method - Create
_apply_overrides()method that returns new merged config
Merge Implementation
python
def get_environment(self, env_name: str, apply_overrides: bool = True) -> EnvironmentConfig:
env_config = self.environments[env_name]
if apply_overrides and self.cli_overrides:
return self._apply_overrides(env_config)
return env_config
def _apply_overrides(self, env_config: EnvironmentConfig) -> EnvironmentConfig:
overrides = self.cli_overrides
known_fields = {
"io_manager", "log_level", "parallelism",
"bucket", "region", "endpoint", "credentials_path",
}
additional_config = env_config.additional_config.copy()
additional_config.update(
{k: v for k, v in overrides.items() if k not in known_fields}
)
return EnvironmentConfig(
io_manager=overrides.get("io_manager", env_config.io_manager),
log_level=overrides.get("log_level", env_config.log_level),
parallelism=overrides.get("parallelism", env_config.parallelism),
bucket=overrides.get("bucket", env_config.bucket),
region=overrides.get("region", env_config.region),
endpoint=overrides.get("endpoint", env_config.endpoint),
credentials_path=overrides.get("credentials_path", env_config.credentials_path),
additional_config=additional_config,
)
Validation
Cloud Storage Validation
python
def _validate_environment_config(config: Config, env_name: str, env_config: Any) -> None:
# Validate cloud storage configuration
if env_config.io_manager in ("s3", "gcs", "azure") and not env_config.bucket:
msg = f"Bucket is required for io_manager '{env_config.io_manager}'"
raise ConfigValidationError(msg, field=f"environments.{env_name}.bucket")
Dependencies
- Add
pyyaml>=6.0.0to runtime dependencies - Add
types-PyYAML>=6.0.0to dev dependencies for MyPy
Testing
Test Structure
- Multi-format tests: Test loading TOML, YAML, JSON configs with valid/invalid syntax
- Inheritance tests: Test simple inheritance, inheritance with additional config, inheritance across formats
- Override tests: Test overriding all config fields, multiple overrides, overrides with/without apply_overrides
- Test validation errors (missing bucket, invalid log level, etc.)
Examples
toml
# vibepiper.toml - Inheritance example
[project]
name = "my-pipeline"
version = "0.1.0"
[environments.base]
io_manager = "s3"
bucket = "base-bucket"
region = "us-east-1"
[environments.prod]
inherits = "base"
log_level = "warning"
yaml
# vibepiper.yaml - Same in YAML
project:
name: "my-pipeline"
version: "0.1.0"
environments:
base:
io_manager: s3
bucket: "base-bucket"
region: "us-east-1"
prod:
inherits: base
log_level: warning
python
# Runtime overrides example
from vibe_piper.config import load_config
config = load_config(
"vibepiper.toml",
cli_overrides={"io_manager": "s3", "bucket": "override-bucket"}
)
# Get environment with overrides applied
prod_env = config.get_environment("prod", apply_overrides=True)
print(f"IO Manager: {prod_env.io_manager}")
print(f"Bucket: {prod_env.bucket}") # Uses override
Acceptance Criteria
- Support TOML, YAML, JSON configuration formats
- Auto-detect format from file extension
- Search for config files in preference order
- Implement environment inheritance with 'inherits' field
- Support multi-level inheritance (grandparent → parent → child)
- Merge child values with parent values correctly
- Implement runtime CLI overrides
- Add apply_overrides parameter to get_environment()
- Preserve environment-specific values not being overridden
- Add comprehensive tests for all features
- Pass MyPy strict mode with proper type hints
- Update module documentation with examples
- Add PyYAML and types-PyYAML dependencies
Gotchas
- YAML parsing requires pyyaml to be installed - provide clear error message
- MyPy requires types-PyYAML for YAML imports - add to dev dependencies
- Inheritance chains should not contain cycles - validation needed
- Runtime overrides should not modify stored config - create new merged objects
- File extension check should be case-insensitive (.TOML == .toml)
- JSON.load() returns Any type - use type: ignore[no-any-return] for strict MyPy
Related Patterns
- See author-agents-md-uv-python for project setup conventions
- See loom-manager-workflow for ticket management patterns
Manual notes
This section is preserved when the skill is updated. Put human notes, caveats, and exceptions here.
Didn't find tool you were looking for?