Notebook Evaluator

1. Purpose

Use this skill to execute Jupyter notebooks (.ipynb) safely without modifying the original file. It evaluates notebooks using their configured kernel and returns structured JSON output with execution results, captured outputs, and any errors—perfect for LLM consumption and automated testing.

Important: Run the script directly (./scripts/eval_notebook.sh). Do not prefix with bash — the script requires zsh and will fail under bash.

2. Usage Scenarios

Run before:

• Validating notebook changes in a pull request
• Testing notebooks in CI/CD pipelines
• Debugging notebook execution errors
• Verifying notebook reproducibility

3. Helper Scripts

The wrapper script enforces a global execution timeout via gtimeout (default: 15m). Pass --timeout DURATION to override it. The duration format follows GNU coreutils (e.g. 30s, 5m, 1h). This is separate from the per-notebook --timeout SECONDS option handled by the Python evaluator.

Arguments

• Required: One or more paths to .ipynb notebook files
• Optional: See CLI options below

4. CLI Options

Warning: Notebook cells can produce huge output, e.g., when producing diagrams. Make sure to alway choose sane outputs for individual cells.

5. Examples

Basic Evaluation

./scripts/eval_notebook.sh analysis.ipynb --pretty

Executes the notebook and returns pretty-printed JSON with results.

Multiple Notebooks

./scripts/eval_notebook.sh notebook1.ipynb notebook2.ipynb

Returns an array of result objects, one per notebook.

Strict Evaluation

./scripts/eval_notebook.sh analysis.ipynb --fail-fast --timeout 120

Stops immediately on any error with a 2-minute timeout.

6. Output Format

Single Notebook Result

{
  "notebook": "/path/to/notebook.ipynb",
  "cwd": "/path/to",
  "kernelspec": "python3",
  "status": "ok",
  "duration_ms": 1234,
  "exec_exception": null,
  "error_count": 0,
  "errors": [],
  "cells": [
    {
      "index": 0,
      "execution_count": 1,
      "source_preview": "print('hello')",
      "outputs": [
        {
          "type": "stream",
          "name": "stdout",
          "text": "hello\n"
        }
      ],
      "output_count": 1
    }
  ]
}

Cell Output Types

7. Exit Codes

Note: Cell execution errors are reported in the JSON output; the script itself succeeds if it can evaluate the notebook.

8. Your Task

When processing evaluation results:

1. If status=ok: Provide a concise summary of key outputs and execution time.

2. If status=error:

   • List each error by cell_index with ename, evalue, and relevant traceback lines
   • Identify the most likely root cause
   • Propose the fastest verification step
   • If code changes are needed, describe them precisely

3. Never overwrite the original notebook file—this skill is read-only by design.