Agent skill
axiom-analyze-test-failures
Use when the user mentions flaky tests, tests that pass locally but fail in CI, race conditions in tests, or needs to diagnose WHY a specific test fails.
Install this agent skill to your Project
npx add-skill https://github.com/CharlesWiltgen/Axiom/tree/main/axiom-codex/skills/axiom-analyze-test-failures
SKILL.md
Test Failure Analyzer Agent
You are an expert at diagnosing WHY tests fail, especially intermittent/flaky failures in Swift Testing.
Your Mission
Analyze the codebase to find patterns that cause flaky tests, focusing on:
- Swift Testing async patterns (missing
confirmation, wrong waits) - Swift 6 concurrency issues (
@MainActormissing) - Parallel execution races (shared state, missing
.serialized) - Timing-dependent assertions
Report findings with:
- File:line references
- Severity ratings (CRITICAL/HIGH/MEDIUM/LOW)
- Root cause explanation
- Fix with code example
Files to Scan
Include: *Tests.swift, *Test.swift, **/*Tests/*.swift
Skip: */Pods/*, */Carthage/*, */.build/*, */DerivedData/*, */scratch/*, */docs/*, */.claude/*, */.claude-plugin/*
Flaky Test Patterns (iOS 18+ / Swift Testing Focus)
Pattern 1: Missing await confirmation (CRITICAL)
Issue: Async work without proper waiting
Why flaky: Test completes before async callback fires
Detection: Closures/callbacks without confirmation {}
// ❌ FLAKY - Test may complete before callback
@Test func fetchData() async {
var result: Data?
service.fetch { data in
result = data // May not run before assertion
}
#expect(result != nil) // FAILS intermittently
}
// ✅ CORRECT - Waits for callback
@Test func fetchData() async {
await confirmation { confirm in
service.fetch { data in
#expect(data != nil)
confirm()
}
}
}
Pattern 2: @MainActor Missing on UI Tests (CRITICAL)
Issue: Swift 6 requires explicit actor isolation Why flaky: Data races when accessing @MainActor types Detection: Tests accessing UI types without @MainActor
// ❌ FLAKY - Data race accessing MainActor ViewModel
@Test func viewModelUpdates() async {
let vm = ContentViewModel() // @MainActor type
vm.load() // Data race!
}
// ✅ CORRECT - Proper isolation
@Test @MainActor func viewModelUpdates() async {
let vm = ContentViewModel()
await vm.load()
}
Pattern 3: Shared Mutable State in @Suite (HIGH)
Issue: Static/class vars shared across parallel tests
Why flaky: Tests pass individually, fail together
Detection: static var in test suites
// ❌ FLAKY - Parallel tests mutate shared state
@Suite struct CacheTests {
static var sharedCache: [String: Data] = [:] // Shared!
@Test func storeItem() {
Self.sharedCache["key"] = Data() // Race condition
}
}
// ✅ CORRECT - Instance property, fresh per test
@Suite struct CacheTests {
var cache: [String: Data] = [:] // Fresh per test
@Test func storeItem() {
cache["key"] = Data()
}
}
Pattern 4: Task.sleep in Assertions (MEDIUM)
Issue: Arbitrary waits for async completion
Why flaky: CI has variable timing
Detection: Task.sleep or try await Task.sleep in tests
// ❌ FLAKY - Timing-dependent
@Test func loadData() async throws {
viewModel.startLoading()
try await Task.sleep(for: .seconds(2)) // May not be enough
#expect(viewModel.isLoaded)
}
// ✅ CORRECT - Condition-based waiting
@Test func loadData() async {
await confirmation { confirm in
viewModel.$isLoaded
.filter { $0 }
.sink { _ in confirm() }
.store(in: &cancellables)
viewModel.startLoading()
}
}
Pattern 5: Missing .serialized Trait (MEDIUM)
Issue: Tests with shared resources run in parallel
Why flaky: Order-dependent or resource-contention failures
Detection: Tests accessing singletons/files without .serialized
// ❌ FLAKY - Parallel tests compete for singleton
@Suite struct DatabaseTests {
@Test func writeData() { Database.shared.write("a") }
@Test func readData() { _ = Database.shared.read() }
}
// ✅ CORRECT - Force serial execution
@Suite(.serialized) struct DatabaseTests {
@Test func writeData() { Database.shared.write("a") }
@Test func readData() { _ = Database.shared.read() }
}
Pattern 6: #expect with Date Comparisons (LOW)
Issue: Date assertions drift across timezones/DST
Why flaky: Passes in one timezone, fails in CI (UTC)
Detection: #expect with Date() or date comparisons
// ❌ FLAKY - Timezone-dependent
@Test func expirationDate() {
let item = CacheItem()
#expect(item.expiresAt > Date()) // May fail near midnight
}
// ✅ CORRECT - Use fixed dates or tolerances
@Test func expirationDate() {
let now = Date()
let item = CacheItem(createdAt: now)
#expect(item.expiresAt.timeIntervalSince(now) > 3600)
}
Audit Process
Step 1: Find All Test Files
Use Glob: **/*Tests.swift, **/*Test.swift
Step 2: Search for Flaky Patterns
Pattern 1 - Missing confirmation:
Grep: \.sink\s*\{|completion\s*:|\.fetch\s*\{
# Then verify no surrounding confirmation {}
Pattern 2 - Missing @MainActor:
Grep: @Test\s+func|@Test\s+@MainActor
# Check tests that access @MainActor types
Pattern 3 - Shared mutable state:
Grep: static var.*=|class var.*=
# In files matching *Tests.swift
Pattern 4 - Task.sleep in tests:
Grep: Task\.sleep|try await Task\.sleep
Pattern 5 - Missing .serialized:
Grep: @Suite\s+struct|@Suite\s*\(
# Check for Database, FileManager, UserDefaults access
Pattern 6 - Date assertions:
Grep: #expect.*Date\(\)|#expect.*\.date
Step 3: Read Context and Verify
For each match:
- Read surrounding context (20 lines)
- Verify it's a real issue (not false positive)
- Check if fix is already present
Output Format
# Test Failure Analysis Results
## Summary
- **CRITICAL Issues**: [count] (Will cause intermittent failures)
- **HIGH Issues**: [count] (Likely flaky in parallel execution)
- **MEDIUM Issues**: [count] (May cause timing issues)
- **LOW Issues**: [count] (Edge case failures)
## Flakiness Risk Score: HIGH / MEDIUM / LOW
## CRITICAL Issues
### Missing `await confirmation`
- `Tests/NetworkTests.swift:45`
```swift
@Test func fetchUser() async {
var user: User?
api.fetchUser { user = $0 }
#expect(user != nil) // FLAKY!
}
- Root cause: Test completes before async callback
- Fix:
@Test func fetchUser() async {
await confirmation { confirm in
api.fetchUser { user in
#expect(user != nil)
confirm()
}
}
}
Missing @MainActor
Tests/ViewModelTests.swift:23swift@Test func updateUI() async { let vm = MainActorViewModel() // Data race }- Root cause: Accessing @MainActor type without isolation
- Fix: Add
@MainActorto test function
HIGH Issues
Shared Mutable State
Tests/CacheTests.swift:12-static var testCache- Root cause: Parallel tests mutate same collection
- Fix: Use instance property instead of static
MEDIUM Issues
Missing .serialized Trait
Tests/DatabaseTests.swift- Suite accesses shared database- Root cause: Parallel writes cause constraint violations
- Fix: Add
.serializedtrait to@Suite
Verification Steps
After fixes, verify with:
# Run tests multiple times to detect flakiness
swift test --parallel --num-workers 8
# Run specific test repeatedly
swift test --filter "TestName" --iterations 100
# Xcode: Edit Scheme → Test → Options → "Repeat Until Failure"
Swift Testing Best Practices
| Pattern | Use When |
|---|---|
confirmation {} |
Any callback/closure-based async |
@MainActor |
Test accesses UI types |
.serialized |
Tests share singleton/file/database |
| Instance properties | Any test data that changes |
## Severity Definitions
**CRITICAL**: Will definitely cause intermittent failures
- Missing `confirmation` for async callbacks
- Missing `@MainActor` for UI tests
**HIGH**: Likely to cause parallel execution failures
- Shared mutable state (`static var`)
- Order-dependent tests
**MEDIUM**: May cause timing-related failures
- `Task.sleep` for waiting
- Missing `.serialized` for shared resources
**LOW**: Edge case failures
- Date/timezone assertions
- Locale-dependent comparisons
## False Positives to Avoid
**Not issues**:
- `static let` constants (immutable is fine)
- `confirmation` already present
- Tests marked with `.serialized`
- `@MainActor` already present
- One-time setup in `static var` that's read-only
**Verify before reporting**:
- Read surrounding context
- Check for `confirmation {}` wrapper
- Check for trait annotations
## XCTest Flaky Patterns (Legacy)
For XCTest code, also check:
### XCTestExpectation Issues
```swift
// ❌ FLAKY - Timeout too short for CI
wait(for: [expectation], timeout: 1.0)
// ✅ BETTER - Generous timeout
wait(for: [expectation], timeout: 10.0)
Missing waitForExistence
// ❌ FLAKY - Element may not exist yet
XCTAssertTrue(app.buttons["Submit"].exists)
// ✅ CORRECT - Wait for element
XCTAssertTrue(app.buttons["Submit"].waitForExistence(timeout: 5))
When No Issues Found
Report:
# Test Failure Analysis Results
## Summary
No flaky test patterns detected.
## Verified
- ✅ Async tests use `confirmation` properly
- ✅ UI tests have `@MainActor` isolation
- ✅ No shared mutable state in suites
- ✅ No timing-dependent assertions
## Recommendations
- Run tests with `--iterations 100` to verify stability
- Enable parallel testing to expose hidden races
- Use Xcode's "Repeat Until Failure" for suspect tests
Recommended Agent Skills
Expand your agent's capabilities with these related and highly-rated skills.
axiom-mapkit-diag
MapKit troubleshooting — annotations not appearing, region jumping, clustering not working, search failures, overlay rendering issues, user location problems
axiom-core-location
Use for Core Location implementation patterns - authorization strategy, monitoring strategy, accuracy selection, background location
axiom-energy
Use when app drains battery, device gets hot, users report energy issues, or auditing power consumption - systematic Power Profiler diagnosis, subsystem identification (CPU/GPU/Network/Location/Display), anti-pattern fixes for iOS/iPadOS
axiom-audit-storage
Use when the user mentions file storage issues, data loss, backup bloat, or asks to audit storage usage.
axiom-app-store-connect-ref
Use when navigating App Store Connect to find crash data, read TestFlight feedback, interpret metrics dashboards, or export diagnostic logs. Covers crash-free rates, dSYM symbolication, termination types, MetricKit.
axiom-testflight-triage
Use when ANY beta tester reports a crash, ANY crash appears in Organizer or App Store Connect, crash logs need symbolication, app was killed without crash report, or you need to triage TestFlight feedback
Didn't find tool you were looking for?