Agent skill

synth-api

Use the Synth AI API end-to-end (SDK + HTTP) for eval + GEPA

View SKILL.md on GitHub Repository
Stars 1
Forks 0

Install this agent skill to your Project

npx add-skill https://github.com/synth-laboratories/skills/tree/main/skills/synth-api

SKILL.md

Synth API (SDK + HTTP, end-to-end)

This skill explains how to run Synth end-to-end with:

  • a Local API task app exposed via Cloudflare tunnel
  • GEPA prompt optimization
  • Eval jobs on held-out seeds

Reference demo: demos/gepa_banking77/gepa_banking77_prompt_optimization.ipynb

Required env

  • SYNTH_API_KEY: your API key (or mint a demo key if using a demo workflow)
  • SYNTH_BACKEND_URL (optional): backend base URL, default https://api.usesynth.ai

Auth + API keys

Synth uses two keys:

  • SYNTH_API_KEY authenticates your SDK/CLI calls to the Synth backend.
  • ENVIRONMENT_API_KEY authenticates backend-to-task-app requests (sent as X-API-Key or Authorization: Bearer ...).

Mint a demo Synth API key (optional)

Demo keys are short‑lived (default 4 hours) and are great for notebooks or quick starts.

python
import os

from synth_ai.core.utils.env import mint_demo_api_key

SYNTH_API_BASE = os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai")
SYNTH_API_KEY = os.environ.get("SYNTH_API_KEY") or mint_demo_api_key(SYNTH_API_BASE)
os.environ["SYNTH_API_KEY"] = SYNTH_API_KEY

Mint + upload an Environment API key

Your task app should use the same ENVIRONMENT_API_KEY that the backend stores for your org. The helper below generates a key locally and uploads it to the backend using your SYNTH_API_KEY.

python
import os

from synth_ai.sdk.localapi.auth import mint_environment_api_key, setup_environment_api_key

SYNTH_API_BASE = os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai")
SYNTH_API_KEY = os.environ["SYNTH_API_KEY"]

ENVIRONMENT_API_KEY = mint_environment_api_key()
os.environ["ENVIRONMENT_API_KEY"] = ENVIRONMENT_API_KEY

setup_environment_api_key(SYNTH_API_BASE, SYNTH_API_KEY, token=ENVIRONMENT_API_KEY)

Core concepts

  • Local API: Your task app runs locally and exposes /rollout + /task_info.
  • Tunnel: Cloudflare Quick Tunnel makes the local app reachable by Synth.
  • GEPA: Prompt optimizer that mutates prompts to maximize reward.
  • Eval jobs: Formal evaluation on held‑out seeds after optimization.

Quick SDK health check

python
import os
import asyncio

from synth_ai.sdk.jobs import JobsClient


async def main() -> None:
    async with JobsClient(
        base_url=os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai"),
        api_key=os.environ["SYNTH_API_KEY"],
    ) as client:
        files = await client.files.list(limit=5)
        print(files)


if __name__ == "__main__":
    asyncio.run(main())

1) Define a Local API task app

Minimum Local API shape:

  • provide_taskset_description()
  • provide_task_instances(seeds)
  • rollout(request) -> RolloutResponse
python
from synth_ai.sdk.localapi import LocalAPIConfig, create_local_api
from synth_ai.sdk.localapi._impl.contracts import RolloutMetrics, RolloutRequest, RolloutResponse, TaskInfo


def create_banking77_local_api(system_prompt: str):
    async def run_rollout(request: RolloutRequest, fastapi_request) -> RolloutResponse:
        # Use your own task logic here; return a reward in [0, 1].
        reward = 1.0
        return RolloutResponse(
            trace_correlation_id=request.trace_correlation_id,
            reward_info=RolloutMetrics(outcome_reward=reward),
            trace=None,
        )

    def provide_taskset_description():
        return {"splits": ["train", "test"], "sizes": {"train": 1000, "test": 1000}}

    def provide_task_instances(seeds):
        for seed in seeds:
            yield TaskInfo(
                task={"id": "banking77", "name": "Banking77 Intent Classification"},
                dataset={"id": "banking77", "split": "train", "index": seed},
                inference={"tool": "banking77_classify"},
                limits={"max_turns": 1},
                task_metadata={"seed": seed},
            )

    return create_local_api(
        LocalAPIConfig(
            app_id="banking77",
            name="Banking77 Intent Classification",
            description="Classify customer queries into intents.",
            provide_taskset_description=provide_taskset_description,
            provide_task_instances=provide_task_instances,
            rollout=run_rollout,
            cors_origins=["*"],
        )
    )

2) Expose the Local API with a Cloudflare tunnel

Use the built‑in tunnel helper to auto‑start the server and provision a URL. The helper spins up your local server, creates a public trycloudflare.com URL, and forwards requests from the public URL to your local port. Keep the process running while Synth calls your task app.

python
from synth_ai.core.tunnels import TunnelBackend, TunneledLocalAPI

app = create_banking77_local_api("baseline prompt")
baseline_tunnel = await TunneledLocalAPI.create_for_app(
    app=app,
    local_port=None,  # auto-select
    backend=TunnelBackend.CloudflareQuickTunnel,
    progress=True,
)
LOCAL_API_URL = baseline_tunnel.url
print("Local API URL:", LOCAL_API_URL)

3) Run GEPA (prompt optimization)

GEPA mutates prompt candidates and evaluates them via rollouts. Use a GEPA config body or a config file. Example config body:

python
from synth_ai.sdk.optimization.internal.prompt_learning import PromptLearningJob

config_body = {
    "prompt_learning": {
        "algorithm": "gepa",
        "task_app_url": LOCAL_API_URL,
        "env_name": "banking77",
        "initial_prompt": {
            "messages": [
                {"role": "system", "order": 0, "pattern": "Baseline system prompt"},
                {"role": "user", "order": 1, "pattern": "Customer Query: {query}\n\nAvailable Intents:\n{available_intents}"},
            ],
            "wildcards": {"query": "REQUIRED", "available_intents": "OPTIONAL"},
        },
        "policy": {
            "model": "gpt-4.1-nano",
            "provider": "openai",
            "inference_mode": "synth_hosted",
            "temperature": 0.0,
            "max_completion_tokens": 256,
        },
        "gepa": {
            "env_name": "banking77",
            "evaluation": {
                "seeds": list(range(50)),
                "validation_seeds": list(range(50, 60)),
            },
            "rollout": {"budget": 80, "max_concurrent": 8, "minibatch_size": 8},
            "mutation": {"rate": 0.3},
            "population": {"initial_size": 4, "num_generations": 3, "children_per_generation": 3},
            "archive": {"size": 5, "pareto_set_size": 10},
        },
    },
}

job = PromptLearningJob.from_dict(config_dict=config_body, skip_health_check=True)
job_id = job.submit()
result = job.poll_until_complete(timeout=3600.0, interval=3.0, progress=True)
print(result.status.value)

4) Run Eval jobs (held‑out seeds)

Eval jobs score a fixed set of held‑out seeds for a final report once optimization finishes.

python
from synth_ai.sdk.eval.job import EvalJob, EvalJobConfig

config = EvalJobConfig(
    local_api_url=LOCAL_API_URL,
    backend_url=os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai"),
    api_key=os.environ["SYNTH_API_KEY"],
    env_name="banking77",
    seeds=list(range(100, 150)),
    policy_config={"model": "gpt-4.1-nano", "provider": "openai"},
    env_config={"split": "test"},
    concurrency=10,
)
job = EvalJob(config)
job.submit()
result = job.poll_until_complete(timeout=600.0, interval=2.0, progress=True)
print(result.status)

5) Retrieve optimized prompts

python
from synth_ai.sdk.optimization.internal.learning.prompt_learning_client import PromptLearningClient

client = PromptLearningClient()
prompt_results = await client.get_prompts(job_id)
best_score = prompt_results.best_score
print("Best score:", best_score)

HTTP example (raw)

python
import os
import requests

base = os.environ.get("SYNTH_BACKEND_URL", "https://api.usesynth.ai")
resp = requests.get(
    f"{base}/api/health",
    headers={"Authorization": f"Bearer {os.environ['SYNTH_API_KEY']}"},
    timeout=30,
)
resp.raise_for_status()
print(resp.json())

Troubleshooting checklist

  • Cloudflare tunnel: Make sure TunneledLocalAPI returns a reachable URL; expect a trycloudflare.com URL.
  • Inference URL: If using hosted inference, your model requests should point to https://api.usesynth.ai/api/inference/v1.
  • Auth: Confirm SYNTH_API_KEY is set and valid.
  • Task app shape: Ensure /task_info and /rollout return valid RolloutResponse.
  • Polling errors: If a poll fails, re‑query /api/prompt-learning/online/jobs/{job_id} to confirm job status.

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

mattpocock/skills

migrate-to-shoehorn

Migrate test files from `as` type assertions to @total-typescript/shoehorn. Use when user mentions shoehorn, wants to replace `as` in tests, or needs partial test data.

111,310 9,758
Explore
mattpocock/skills

setup-pre-commit

Set up Husky pre-commit hooks with lint-staged (Prettier), type checking, and tests in the current repo. Use when user wants to add pre-commit hooks, set up Husky, configure lint-staged, or add commit-time formatting/typechecking/testing.

111,310 9,758
Explore
mattpocock/skills

obsidian-vault

Search, create, and manage notes in the Obsidian vault with wikilinks and index notes. Use when user wants to find, create, or organize notes in Obsidian.

111,310 9,758
Explore
mattpocock/skills

edit-article

Edit and improve articles by restructuring sections, improving clarity, and tightening prose. Use when user wants to edit, revise, or improve an article draft.

111,310 9,758
Explore
mattpocock/skills

git-guardrails-claude-code

Set up Claude Code hooks to block dangerous git commands (push, reset --hard, clean, branch -D, etc.) before they execute. Use when user wants to prevent destructive git operations, add git safety hooks, or block git push/reset in Claude Code.

111,310 9,758
Explore
mattpocock/skills

scaffold-exercises

Create exercise directory structures with sections, problems, solutions, and explainers that pass linting. Use when user wants to scaffold exercises, create exercise stubs, or set up a new course section.

111,310 9,758
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results