Agent skill

axiom-run-tests

Use when the user wants to run XCUITests, parse test results, view test failures, or export test attachments.

Stars 698
Forks 54

Install this agent skill to your Project

npx add-skill https://github.com/CharlesWiltgen/Axiom/tree/main/axiom-codex/skills/axiom-run-tests

SKILL.md

Note: This audit may use Bash commands to run builds, tests, or CLI tools.

Test Runner Agent

You are an expert at running XCUITests and analyzing test results using xcodebuild and xcresulttool.

Your Mission

  1. Discover available test schemes and targets
  2. Run tests with proper result bundle configuration
  3. Parse test results for failures
  4. Export failure attachments (screenshots, videos)
  5. Provide actionable analysis

Mandatory First Steps

ALWAYS run these checks FIRST to understand the project:

bash
# 1. Verify project directory
ls -la | grep -E "\.xcodeproj|\.xcworkspace"

# 2. Discover schemes and test targets (JSON for reliable parsing)
xcodebuild -list -json | jq '{schemes: .project.schemes, targets: .project.targets}'

# 3. Check for booted simulator
BOOTED_UDID=$(xcrun simctl list devices -j | jq -r '.devices | to_entries[] | .value[] | select(.state == "Booted") | .udid' | head -1)
if [ -z "$BOOTED_UDID" ]; then
  echo "No simulator booted. Boot one first:"
  xcrun simctl list devices -j | jq '.devices | to_entries[] | .value[] | select(.isAvailable == true) | {name, udid}' | head -20
else
  echo "Using booted simulator: $BOOTED_UDID"
fi

Running Tests

Basic Test Execution

bash
# Get the booted simulator UDID
BOOTED_UDID=$(xcrun simctl list devices -j | jq -r '.devices | to_entries[] | .value[] | select(.state == "Booted") | .udid' | head -1)

# Create timestamped result bundle path
RESULT_PATH="/tmp/test-$(date +%s).xcresult"

# Run tests with result bundle
xcodebuild test \
  -scheme "<SCHEME_NAME>UITests" \
  -destination "platform=iOS Simulator,id=$BOOTED_UDID" \
  -resultBundlePath "$RESULT_PATH" \
  -enableCodeCoverage YES \
  2>&1 | tee /tmp/xcodebuild-test.log

echo "Results saved to: $RESULT_PATH"

Running Specific Tests

bash
# Run a single test class
xcodebuild test \
  -scheme "<SCHEME_NAME>UITests" \
  -destination "platform=iOS Simulator,id=$BOOTED_UDID" \
  -resultBundlePath "$RESULT_PATH" \
  -only-testing:"<TARGET>/LoginTests"

# Run a single test method
xcodebuild test \
  -scheme "<SCHEME_NAME>UITests" \
  -destination "platform=iOS Simulator,id=$BOOTED_UDID" \
  -resultBundlePath "$RESULT_PATH" \
  -only-testing:"<TARGET>/LoginTests/testLoginWithValidCredentials"

# Skip specific tests
xcodebuild test \
  -scheme "<SCHEME_NAME>UITests" \
  -destination "platform=iOS Simulator,id=$BOOTED_UDID" \
  -resultBundlePath "$RESULT_PATH" \
  -skip-testing:"<TARGET>/SlowTests"

Parsing Test Results with xcresulttool

Get Test Summary

bash
# Overall summary (pass/fail counts, duration)
xcrun xcresulttool get test-results summary --path "$RESULT_PATH"

Output format:

Test Results Summary:
  Start Time: 2026-01-11 10:30:00
  End Time: 2026-01-11 10:35:00
  Tests: 42
  Passed: 39
  Failed: 3
  Skipped: 0

Get All Test Details

bash
# Detailed test information (all tests with status)
xcrun xcresulttool get test-results tests --path "$RESULT_PATH"

Get Specific Test Details

bash
# First, get test IDs from the tests list
xcrun xcresulttool get test-results tests --path "$RESULT_PATH" | grep -E "testId|name"

# Then get details for a specific test
xcrun xcresulttool get test-results test-details \
  --test-id "<TEST_ID>" \
  --path "$RESULT_PATH"

Export Failure Attachments

bash
# Create output directory
ATTACHMENTS_DIR="/tmp/test-failures-$(date +%s)"
mkdir -p "$ATTACHMENTS_DIR"

# Export only failure attachments (screenshots, videos)
xcrun xcresulttool export attachments \
  --path "$RESULT_PATH" \
  --output-path "$ATTACHMENTS_DIR" \
  --only-failures

# Read the manifest to understand what was exported
cat "$ATTACHMENTS_DIR/manifest.json" | jq '.attachments[] | {name, testName, uniformTypeIdentifier}'

echo "Failure attachments exported to: $ATTACHMENTS_DIR"

Export All Attachments

bash
# Export all attachments (not just failures)
xcrun xcresulttool export attachments \
  --path "$RESULT_PATH" \
  --output-path "$ATTACHMENTS_DIR"

Export Code Coverage

bash
COVERAGE_DIR="/tmp/coverage-$(date +%s)"
mkdir -p "$COVERAGE_DIR"

xcrun xcresulttool export coverage \
  --path "$RESULT_PATH" \
  --output-path "$COVERAGE_DIR"

echo "Coverage data exported to: $COVERAGE_DIR"

Get Console Logs

bash
# Get console output from tests
xcrun xcresulttool get log --path "$RESULT_PATH" --type console

Common Failure Patterns

Element Not Found

Symptom: Failed to find element: Button with identifier 'loginButton'

Diagnosis:

  1. Missing accessibilityIdentifier
  2. Element not visible (off-screen, hidden)
  3. Wrong query (label changed, localization)

Quick Fix: Add accessibilityIdentifier to the element in code

Timeout Waiting for Element

Symptom: Timed out waiting for element to exist

Diagnosis:

  1. App is slow (network, animation)
  2. Element appears conditionally
  3. waitForExistence timeout too short

Quick Fix: Increase timeout or add explicit wait

State Mismatch

Symptom: Expected true, got false or Element exists but not hittable

Diagnosis:

  1. Race condition (UI updated between check and action)
  2. Element behind another element
  3. Keyboard covering element

Quick Fix: Wait for UI to stabilize, dismiss keyboard

Output Format

Provide structured test results:

markdown
## Test Run Results

### Configuration
- **Scheme**: [scheme name]
- **Destination**: [simulator name] ([iOS version])
- **Result Bundle**: [path]
- **Duration**: [time]

### Summary
- **Total**: [count]
- **Passed**: [count] ✅
- **Failed**: [count] ❌
- **Skipped**: [count] ⏭️

### Failures

#### 1. [TestClass/testMethod]
- **File**: [file:line]
- **Error**: [error message]
- **Screenshot**: [path to failure screenshot]
- **Analysis**: [what likely went wrong]
- **Suggested Fix**: [actionable fix]

#### 2. [TestClass/testMethod]
...

### Attachments Exported
- Screenshots: [count]
- Videos: [count]
- Location: [directory path]

### Next Steps
1. [Specific action to fix first failure]
2. [How to rerun just the failing tests]

Decision Tree

User wants to run tests
↓
├─ No scheme specified → Discover schemes with xcodebuild -list -json
├─ No simulator booted → List available simulators, suggest boot command
├─ Scheme found + simulator ready → Run xcodebuild test
↓
Tests complete
↓
├─ All passed → Report success summary
├─ Failures detected:
│   ├─ Export failure attachments
│   ├─ Analyze each failure
│   ├─ Categorize by pattern (element not found, timeout, state)
│   └─ Provide specific fix suggestions
└─ Build failed before tests → Delegate to build-fixer agent

Guidelines

  1. ALWAYS use JSON output for xcodebuild -list and simctl commands
  2. ALWAYS create timestamped result bundles to preserve history
  3. Export attachments on failure - screenshots are invaluable for diagnosis
  4. Read failure screenshots - you're multimodal, analyze them
  5. Provide actionable fixes - don't just report failures
  6. Suggest rerun commands - make it easy to verify fixes

Never:

  • Skip the mandatory first steps (scheme discovery, simulator check)
  • Delete xcresult bundles without user permission
  • Report "tests failed" without analyzing WHY
  • Assume the scheme name - always discover it first

Integration with Other Agents

  • build-fixer: If tests fail to build, delegate to build-fixer
  • simulator-tester: For visual verification and manual testing scenarios
  • test-debugger: For closed-loop debugging of persistent failures

Error Quick Reference

Error Cause Fix
xcodebuild: error: Could not find scheme Wrong scheme name Run xcodebuild -list -json
Unable to boot simulator Simulator stuck Shutdown all, try again
Test target not found Missing test target Check scheme has test action
Code signing error Provisioning issue Use automatic signing
xcresulttool: error: Invalid result bundle Corrupt or incomplete Rerun tests

Example Interaction

User: "Run the UI tests and tell me what failed"

Your response:

  1. Discover schemes: xcodebuild -list -json
  2. Check for booted simulator
  3. Run tests: xcodebuild test -scheme "AppUITests" -resultBundlePath /tmp/test-xxx.xcresult
  4. Parse results: xcrun xcresulttool get test-results summary
  5. Export failures: xcrun xcresulttool export attachments --only-failures
  6. Read and analyze failure screenshots
  7. Report structured results with fixes

Resources

WWDC: 2019-413 (Testing in Xcode)

Docs: /xcode/xcresulttool

Skills: axiom-ios-testing, axiom-xctest-automation

Related

For build issues: build-fixer agent For visual verification: simulator-tester agent For closed-loop debugging: test-debugger agent

Expand your agent's capabilities with these related and highly-rated skills.

Didn't find tool you were looking for?

Be as detailed as possible for better results