Agent skill

log-analysis

This skill should be used when analyzing Breenix kernel logs for debugging, testing verification, or understanding kernel behavior. Use for searching timestamped logs, finding checkpoint signals, tracing execution flow, identifying errors or panics, and extracting diagnostic information.

View SKILL.md on GitHub Repository
Stars 7
Forks 0

Install this agent skill to your Project

npx add-skill https://github.com/ryanbreen/breenix/tree/main/breenix-log-analysis

SKILL.md

Kernel Log Analysis for Breenix

Search, analyze, and extract information from Breenix kernel logs for debugging and testing.

Purpose

Breenix logs all kernel runs to logs/breenix_YYYYMMDD_HHMMSS.log. This skill provides patterns for searching these logs efficiently, finding checkpoint signals, tracing execution, and diagnosing issues.

When to Use

  • Finding test signals: Locate checkpoint markers like 🎯 KERNEL_POST_TESTS_COMPLETE 🎯
  • Tracing execution: Follow kernel boot sequence or specific subsystem initialization
  • Debugging failures: Find panics, faults, or error messages
  • Verifying behavior: Confirm expected operations occurred
  • Performance analysis: Check timing of operations via log timestamps

Log Location and Format

bash
# Logs stored in
logs/breenix_YYYYMMDD_HHMMSS.log

# View latest log
ls -t logs/*.log | head -1 | xargs less

# View specific log
less logs/breenix_20250120_143022.log

Log Format

[ INFO] kernel::memory: Physical memory: 94 MiB usable
[DEBUG] kernel::memory: Frame allocator initialized
[ WARN] kernel::process: No processes ready

Levels: TRACE, DEBUG, INFO, WARN, ERROR

Search Using find-in-logs Script

The scripts/find-in-logs tool searches recent logs:

bash
# Create search query (avoids approval prompts)
echo '-A50 "Creating user process"' > /tmp/log-query.txt
./scripts/find-in-logs

# The script reads from /tmp/log-query.txt and searches logs

Common Search Patterns

bash
# Find panics
echo '-i "panic"' > /tmp/log-query.txt
./scripts/find-in-logs

# Find page faults
echo '-i "page fault"' > /tmp/log-query.txt
./scripts/find-in-logs

# Find context around checkpoint
echo '-A20 -B10 "KERNEL_POST_TESTS_COMPLETE"' > /tmp/log-query.txt
./scripts/find-in-logs

# Find process creation
echo '"Creating user process"' > /tmp/log-query.txt
./scripts/find-in-logs

Direct grep Usage

bash
# Find specific error
grep -n "ERROR" logs/breenix_20250120_*.log

# Find with context
grep -A10 -B5 "Double Fault" logs/breenix_20250120_*.log

# Case-insensitive search
grep -i "memory" logs/breenix_20250120_*.log

# Multiple patterns
grep -E "panic|fault|error" logs/breenix_20250120_*.log

# Count occurrences
grep -c "Timer interrupt" logs/breenix_20250120_*.log

Common Checkpoint Signals

bash
# Test completion
grep "🎯 KERNEL_POST_TESTS_COMPLETE 🎯" logs/*.log

# Userspace execution
grep "USERSPACE OUTPUT:" logs/*.log
grep "Hello from userspace" logs/*.log

# System calls
grep "🎉 USERSPACE SYSCALL" logs/*.log

# Initialization checkpoints
grep "initialized\|INITIALIZED" logs/*.log

# Process creation
grep "Process created: PID" logs/*.log

Execution Flow Tracing

Boot Sequence

bash
# Full boot trace
grep -E "Boot|GDT|IDT|PIC|Memory|Heap|Timer|Keyboard" logs/latest.log

# Memory subsystem only
grep "memory\|page table\|frame allocator" logs/latest.log

# Process subsystem
grep "process\|fork\|exec\|PID" logs/latest.log

Subsystem Analysis

bash
# Timer subsystem
grep -n "timer\|RTC\|tick" logs/latest.log

# Interrupt handling
grep -n "interrupt\|IRQ\|IDT" logs/latest.log

# System calls
grep -n "syscall\|sys_\|INT 0x80" logs/latest.log

Error and Fault Analysis

Finding Faults

bash
# Double faults
grep -A20 "DOUBLE FAULT" logs/*.log

# Page faults
grep -A10 "PAGE FAULT" logs/*.log

# General panics
grep -B10 -A20 "PANIC" logs/*.log

Error Context

bash
# Find errors with context
grep -A15 -B5 "ERROR" logs/latest.log

# Find warnings that might indicate problems
grep -A5 "WARN" logs/latest.log

Log Analysis Patterns

Timeline Analysis

bash
# Extract just log levels and messages for overview
grep -E "\[(INFO|WARN|ERROR|DEBUG)\]" logs/latest.log | less

# Filter to specific subsystem
grep "\[.*\] kernel::process:" logs/latest.log

Success/Failure Detection

bash
# Check if test completed
if grep -q "KERNEL_POST_TESTS_COMPLETE" logs/latest.log; then
  echo "Test completed"
else
  echo "Test did not complete"
  # Find last successful checkpoint
  grep "SUCCESS\|initialized\|completed" logs/latest.log | tail -10
fi

Performance Markers

bash
# Find timing information
grep -E "took|elapsed|ms|seconds" logs/latest.log

# Specific operations
grep "Context switch\|schedule\|preempt" logs/latest.log

Integration with Other Skills

With kernel-debug-loop

bash
# Run quick test
kernel-debug-loop/scripts/quick_debug.py --signal "TARGET_CHECKPOINT"

# Then analyze its output
grep "TARGET_CHECKPOINT" logs/latest.log

With ci-failure-analysis

bash
# Analyze CI logs
ci-failure-analysis/scripts/analyze_ci_failure.py target/xtask_*_output.txt

# Then search for specific patterns found
grep "PATTERN" target/xtask_*_output.txt

Best Practices

  1. Use specific patterns: Narrow searches to relevant subsystems
  2. Add context: Use -A (after) and -B (before) flags
  3. Check latest first: ls -t logs/*.log | head -1
  4. Save search queries: Use /tmp/log-query.txt for complex patterns
  5. Look for first error: Often followed by cascading failures
  6. Check initialization: Ensure subsystems initialized before use
  7. Verify checkpoints: Confirm expected signals appear

Summary

Effective log analysis requires:

  • Knowing checkpoint signals
  • Using grep with context flags
  • Understanding log levels and formats
  • Tracing execution flow
  • Finding first failures
  • Verifying expected behavior

Logs are the primary window into kernel behavior - use them liberally during development and debugging.

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

ryanbreen/breenix

legacy-migration

This skill should be used when migrating features from src.legacy/ to the new kernel implementation or removing legacy code after reaching feature parity. Use for systematic legacy code removal, updating FEATURE_COMPARISON.md, verifying feature equivalence, and ensuring safe code retirement.

7 0
Explore
ryanbreen/breenix

kernel-debug-loop

This skill should be used when performing fast iterative kernel debugging, running time-bound kernel sessions to detect specific log signals or test kernel behavior. Use for rapid feedback cycles during kernel development, boot sequence analysis, or feature verification.

7 0
Explore
ryanbreen/breenix

gdb-chat

Conversational GDB debugging for Breenix kernel. Use for interactive debugging sessions - set breakpoints, inspect registers, examine memory, step through code, investigate crashes.

7 0
Explore
ryanbreen/breenix

os-research

OS kernel research specialist. Use when researching operating system best practices, comparing implementations to Linux/FreeBSD/XNU, evaluating kernel design patterns, or ensuring Breenix follows production kernel conventions.

7 0
Explore
ryanbreen/breenix

register-watch

Use when debugging register corruption issues - registers have unexpected values after context switches, userspace processes crash with corrupted state, stack pointer corruption, syscall return values corrupted, or timer interrupt handlers corrupting register state.

7 0
Explore
ryanbreen/breenix

memory-debugging

This skill should be used when debugging memory-related issues in the Breenix kernel including page faults, double faults, frame allocation problems, page table issues, heap allocation failures, stack overflows, and virtual memory mapping errors.

7 0
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results