Agent skill
exps-performance
Documentation for the main LLM performance experiments (exps_performance)
Install this agent skill to your Project
npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/other/exps-performance-terrytong-git-cohere
SKILL.md
exps_performance - Main Performance Experiments
Purpose
Run LLM inference on algorithmic problems, collect CoT rationales and evaluate correctness.
Key Files
| File | Purpose |
|---|---|
src/exps_performance/main.py |
Main runner |
src/exps_performance/dataset.py |
Dataset definitions |
src/exps_performance/arms.py |
Evaluation arms (NL, code_sim, code_exec) |
src/exps_performance/llm.py |
LLM interface |
src/exps_performance/analysis.py |
Analysis and plotting |
src/exps_performance/noise.py |
Noise injection experiments |
Results Structure
src/exps_performance/results/
{model}_seed{seed}/
tb/
run_{timestamp}/
res.jsonl # Main results file
res.jsonl Schema
| Field | Description |
|---|---|
kind |
Problem type (e.g., "knap", "bellman_ford") |
digit |
Problem size/digits |
nl_correct |
NL evaluation result |
code_correct |
Code execution result |
sim_correct |
Code similarity result |
model |
Model name |
seed |
Random seed |
Data Sources
- Performance results:
src/exps_performance/results/{model}_seed{seed}/tb/run_*/res.jsonl - Excludes
gsm8kfrom accuracy calculations in logistic analysis
Running Commands
# Run main experiment
uv run python src/exps_performance/main.py --model <model> --seed <seed>
# Run analysis
uv run python src/exps_performance/analysis.py
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
majiayu000/claude-skill-registry
agent-ops-spec
Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.
majiayu000/claude-skill-registry
agent-ops-state
Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.
majiayu000/claude-skill-registry
agent-ops-spec
Manage specification documents in .agent/specs/. Use when user provides requirements, acceptance criteria, or feature descriptions that need to be tracked and validated against implementation.
majiayu000/claude-skill-registry
agent-ops-testing
Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.
majiayu000/claude-skill-registry
agent-ops-testing
Test strategy, execution, and coverage analysis. Use when designing tests, running test suites, or analyzing test results beyond baseline checks.
majiayu000/claude-skill-registry
agent-ops-state
Maintain .agent state files. Use at session start, after meaningful steps, and before concluding: read/update constitution/memory/focus/issues/baseline consistently.
Didn't find tool you were looking for?