CRAB - Cross-environment Agent Benchmark for Multimodal Language Model Agents

CRAB Cross-environment Agent Benchmark for Multimodal Language Model Agents

What is CRAB?

CRAB is a comprehensive framework designed to facilitate the development, operation, and evaluation of Multimodal Language Model (MLM) agents. It features cross-environment support, a graph evaluator for detailed performance analysis, and automated task generation to simulate real-world scenarios.

The framework stands out by supporting multiple environments, allowing agents to adapt across different interfaces. CRAB offers fine-grained evaluation with graph evaluator, and uses a graph-based method for task generation which combines multiple sub-tasks. The system's architecture ensures ease of use, enabling the addition of new environments with minimal Python coding, and experiment reproducibility through a declarative programming paradigm.

Features

Cross-environments: Supports multiple environments, ensuring agents adapt across different interfaces.
Graph evaluator: Provides fine-grained evaluation, and detailed analysis of agent performance.
Task Generation: Automates task creation using a graph-based method.
Easy-to-use: Adding a new environment requires only a few lines of Python code.

Use Cases

Evaluating the performance of Multimodal Language Models.
Developing and testing agents in diverse operating environments (Ubuntu and Android).
Creating dynamic tasks that mimic real-world scenarios for agent training.
Analyzing agent strengths and weaknesses through detailed performance metrics.
Reproducing experimental environments for consistent benchmarking.