Agent skill
gemini-batch
This skill should be used when the user asks to "use Gemini Batch API", "process documents at scale", "submit a batch job", "upload files to Gemini", or needs large-scale LLM processing. Includes production gotchas and best practices.
Stars
163
Forks
31
Install this agent skill to your Project
npx add-skill https://github.com/majiayu000/claude-skill-registry/tree/main/skills/product/gemini-batch
SKILL.md
Gemini Batch API Skill
Large-scale asynchronous document processing using Google's Gemini models.
When to Use
- Process thousands of documents with the same prompt
- Cost-effective bulk extraction (50% cheaper than synchronous API)
- Jobs that can tolerate 24-hour completion windows
Quick Start
python
from examples.batch_processor import GeminiBatchProcessor
processor = GeminiBatchProcessor(
bucket_name="my-batch-bucket", # Must be in us-central1
model="gemini-2.0-flash-lite"
)
results = processor.run_pipeline(
input_dir="./documents",
prompt="Extract as JSON: {title, date, summary}",
output_dir="./results"
)
Core Workflow
- Upload files to GCS bucket (us-central1 region required)
- Create JSONL request file with document URIs and prompts
- Submit batch job via
genai.batches.create() - Poll for completion (jobs expire after 24 hours)
- Download and parse results from output URI
- Handle failures gracefully (partial failures are common)
Key Gotchas
| Issue | Solution |
|---|---|
| Region mismatch | Use us-central1 bucket only |
| Wrong URI format | Use gs:// not https:// |
| Invalid JSONL | Use scripts/validate_jsonl.py |
| Duplicate IDs | Hash file content + prompt for unique IDs |
| Large PDFs fail | Split at 50 pages / 50MB max |
| JSON parsing fails | Use robust extraction (see gotchas.md) |
| Output not found | Output URI is prefix, not file path |
See references/gotchas.md for detailed solutions.
Rate Limits
| Limit | Value |
|---|---|
| Max requests per JSONL | 10,000 |
| Max concurrent jobs | 10 |
| Max job size | 100MB |
| Job expiration | 24 hours |
Recommended Models
| Model | Use Case | Cost |
|---|---|---|
gemini-2.0-flash-lite |
Most batch jobs | Lowest |
gemini-2.0-flash |
Complex extraction | Medium |
gemini-1.5-pro |
Highest accuracy | Highest |
Additional Resources
References
references/gotchas.md- 9 critical production gotchas with solutionsreferences/best-practices.md- Idempotent IDs, state tracking, validationreferences/troubleshooting.md- Common errors and debuggingreferences/vertex-ai.md- Enterprise alternative with comparisonreferences/cli-reference.md- gsutil and gcloud commands
Examples
examples/batch_processor.py- Complete GeminiBatchProcessor classexamples/pipeline_template.py- Customizable pipeline template
Scripts
scripts/validate_jsonl.py- Validate JSONL before submissionscripts/test_single.py- Test single request before batch
External Documentation
Didn't find tool you were looking for?