Building GitHub Index

Create markdown indexes of GitHub repositories optimized for Claude project knowledge. Indexes enable retrieval of source content (.md, .qmd, .ipynb) via GitHub API without parsing rendered HTML/CSS/JS noise.

Quick Start

python scripts/github_index.py owner/repo -o index.md
python scripts/github_index.py owner/repo1 owner/repo2 -o combined.md
python scripts/github_index.py owner/repo --include-patterns "docs/**" "blog/**"

Script Options

Default exclusions: node_modules, .git, __pycache__, venv, dist, build, _site, .next, plus underscore-prefixed files.

Rate limits: 60/hr unauthenticated, 5000/hr with token.

Output Structure

# {Repo Name} - Content Index

**Repository:** {url}
**Branch:** `{branch}`

## Retrieval Method
{API curl commands}

---

## {Category}

| Description | Path |
|-------------|------|
| {What this covers} | `{path/to/file.md}` |

## Key Themes
{Concepts that reduce need to fetch}

Lead rows with description (relevance matching), path second (retrieval key).

API Access

Fetch tree (enumerate files):

curl -sL "https://api.github.com/repos/OWNER/REPO/git/trees/BRANCH?recursive=1"

Fetch file content:

curl -s "https://api.github.com/repos/OWNER/REPO/contents/PATH?ref=BRANCH" \
  -H "Accept: application/vnd.github+json" | \
  python3 -c "import sys,json,base64; print(base64.b64decode(json.load(sys.stdin)['content']).decode())"

Include retrieval instructions in generated indexes.

Manual Index Creation

When script doesn't fit or for curated indexes:

1. Get tree, filter to content files (.md, .qmd, .ipynb)
2. Fetch files, extract frontmatter (title/description from YAML between --- markers)
3. Group by directory structure or theme
4. Write tables: Description | Path
5. Add Key Themes section for frequent queries

Quality Standards

Descriptions: Lead with problem/question addressed. Include searchable terms. Under 100 chars.

Categories: Semantic groupings, not flat lists. Subcategories help navigation.

Themes: Extract after reviewing content. Summarize frequently-asked concepts.

Network Requirements

Project allowlist: api.github.com (required), raw.githubusercontent.com (optional).

Related Skills

• accessing-github-repos - Private repos, PAT setup
• mapping-codebases - Code structure (functions/classes) vs documentation