Agent skill

dedupe-rank

Dedupe and rank a raw paper set (`papers/papers_raw.jsonl`) to produce `papers/papers_dedup.jsonl` and `papers/core_set.csv`. **Trigger**: dedupe, rank, core set, 去重, 排序, 精选论文, 核心集合. **Use when**: 检索后需要把广覆盖集合收敛成可管理的 core set（用于 taxonomy/outline/mapping）。 **Skip if**: 已经有人手工整理了稳定的 `papers/core_set.csv`（无需再次 churn）。 **Network**: none. **Guardrail**: 偏 deterministic；输出应可重复（稳定 paper_id、字段规范）。

View SKILL.md on GitHub Repository
Stars 377
Forks 25

Install this agent skill to your Project

npx add-skill https://github.com/WILLOSCAR/research-units-pipeline-skills/tree/main/.codex/skills/dedupe-rank

SKILL.md

Dedupe + Rank

Turn a broad retrieved set into a smaller core set for taxonomy/outline building.

This is a deterministic “curation” step: it should be stable and repeatable.

Load Order

Always read:

  • references/domain_pack_overview.md — how domain packs drive topic-specific behavior

Domain packs (loaded by topic match):

  • assets/domain_packs/llm_agents.json — pinned classics, survey detection, ranking signals for LLM agent topics

Script Boundary

Use scripts/run.py only for:

  • title normalization and deduplication logic
  • relevance scoring from query tokens
  • core set CSV generation with stable paper_id values

Do not treat run.py as the place for:

  • hardcoded pinned paper IDs (use domain packs)
  • hardcoded survey detection rules (use domain packs)
  • domain-specific topic detection logic (use domain packs)

Input

  • papers/papers_raw.jsonl

Outputs

  • papers/papers_dedup.jsonl
  • papers/core_set.csv

Workflow (high level)

  1. Dedupe by normalized (title, year) and keep the richest metadata per duplicate cluster.
  2. Rank by relevance/recency signals (and optionally pin known classics for certain topics). For LLM-agent topics, also ensure a small quota of prior surveys/reviews is present to support a paper-like Related Work section.
  3. Write papers/core_set.csv with stable paper_id values and useful metadata columns (arxiv_id, pdf_url, categories).

Quality checklist

  • papers/papers_dedup.jsonl exists and is valid JSONL.
  • papers/core_set.csv exists and has a header row.

Script

Quick Start

  • python .codex/skills/dedupe-rank/scripts/run.py --help
  • python .codex/skills/dedupe-rank/scripts/run.py --workspace <workspace_dir> --core-size 300

All Options

  • --core-size <n>: target size for papers/core_set.csv
  • queries.md also supports core_size / core_set_size / dedupe_core_size (overrides default when present)

Examples

  • Smaller core set for fast iteration (non-A150++):
    • python .codex/skills/dedupe-rank/scripts/run.py --workspace <ws> --core-size 25

Notes

  • This step may annotate papers/core_set.csv:reason with tags such as pinned_classic and prior_survey (deterministic, topic-aware guards for survey writing).
  • Systematic-review default: if the active pipeline is systematic-review and core_size is not specified, the script keeps the full deduped pool in papers/core_set.csv (so screening does not silently drop candidates).
  • This step is deterministic; reruns should be stable for the same inputs.

Troubleshooting

Common Issues

Issue: papers/core_set.csv is too small / empty

Symptom:

  • Core set has very few rows.

Causes:

  • Input papers/papers_raw.jsonl is small, or many rows are missing required fields.

Solutions:

  • Broaden retrieval (or provide a richer offline export) and rerun.
  • Lower --core-size only if you intentionally want a small core set.

Issue: Duplicates still appear after dedupe

Symptom:

  • Near-identical titles remain.

Causes:

  • Title normalization is defeated by noisy exports.

Solutions:

  • Clean title fields in the export (strip prefixes/suffixes, fix encoding) and rerun.

Recovery Checklist

  • papers/papers_raw.jsonl lines contain title/year/url.
  • papers/core_set.csv has stable paper_id values.

Recommended Agent Skills

Expand your agent's capabilities with these related and highly-rated skills.

WILLOSCAR/research-units-pipeline-skills

thesis-compile-review

对中文毕业论文进行编译、warning 分级、模板模式检查、数据与引用复查，并把问题回写成可继续迭代的 review checklist。 **Trigger**: 毕业论文编译检查, thesis compile review, warning 分级, 终稿复查, main.pdf 检查. **Use when**: 论文已经回写到 TeX 交付层，需要确认是否真正达到“可提交”的质量，而不是只做到能编译。 **Skip if**: 还处于中间层重构阶段，`chapters/*.tex` 尚未形成稳定交付稿。 **Network**: none. **Guardrail**: 不在这里重构章节主线；如果发现结构问题，明确回退到上游修复。

377 25
Explore
WILLOSCAR/research-units-pipeline-skills

front-matter-writer

Write the survey's front matter files (Abstract, Introduction, Related Work, Discussion, Conclusion) in paper voice, with high citation density and a single evidence-policy paragraph. **Trigger**: front matter writer, introduction writer, related work writer, abstract writer, discussion writer, conclusion writer, 引言, 相关工作, 摘要, 讨论, 结论. **Use when**: you are in C5 (prose allowed) and need the paper-like shell to stop the draft reading like stitched subsections. **Skip if**: `Approve C2` is missing in `DECISIONS.md`, or `citations/ref.bib` is missing. **Network**: none. **Guardrail**: no invented facts/citations; no pipeline jargon in final prose; no repeated evidence disclaimers; only use keys present in `citations/ref.bib`.

377 25
Explore
WILLOSCAR/research-units-pipeline-skills

thesis-question-list

维护中文毕业论文的 `codex_md/question_list.md`：把本轮问题、边界、优先级、协作方案和验收口径结构化，作为整条 thesis pipeline 的控制面。 **Trigger**: 毕业论文问题清单, thesis question list, 论文修改清单, 本轮目标, 结构问题梳理, review问题整理. **Use when**: 你已经有一批材料或上一轮 review 结果，需要明确这一轮到底修什么、不修什么，并给后续重构与编译复查提供统一入口。 **Skip if**: 当前只是在做一次性局部措辞修改，且没有形成新一轮结构/证据/编译问题。 **Network**: none. **Guardrail**: 不在这里写正文；不把问题单写成长篇散文；每条问题必须可执行、可验收。

377 25
Explore
WILLOSCAR/research-units-pipeline-skills

novelty-matrix

Create a novelty/prior-work matrix comparing the submission’s contributions against related work (overlaps vs deltas). **Trigger**: novelty matrix, prior-work matrix, overlap/delta, 相关工作对比, 新颖性矩阵. **Use when**: peer review 中评估 novelty/positioning，需要把贡献与相关工作逐项对齐并写出差异点证据。 **Skip if**: 缺少 claims（先跑 `claims-extractor`）或你不打算做新颖性定位分析。 **Network**: none (retrieval of additional related work is out-of-scope unless provided). **Guardrail**: 明确 overlap 与 delta；尽量给出可追溯证据来源（来自稿件/引用/作者陈述）。

377 25
Explore
WILLOSCAR/research-units-pipeline-skills

protocol-writer

Write a systematic review protocol into `output/PROTOCOL.md` (databases, queries, inclusion/exclusion, time window, extraction fields). **Trigger**: protocol, PRISMA, systematic review, inclusion/exclusion, 检索式, 纳入排除. **Use when**: systematic review pipeline 的起点（C1），需要先锁定 protocol 再开始 screening/extraction。 **Skip if**: 不是做 systematic review（或 protocol 已经锁定且不允许修改）。 **Network**: none. **Guardrail**: protocol 必须包含可执行的检索与筛选规则；需要 HUMAN 签字后才能进入 screening。

377 25
Explore
WILLOSCAR/research-units-pipeline-skills

rubric-writer

Write a rubric-based peer review report (`output/REVIEW.md`) using extracted claims and evidence gaps (novelty/soundness/clarity/impact). **Trigger**: rubric review, referee report, peer review write-up, 审稿报告, REVIEW.md. **Use when**: peer-review pipeline 的最后阶段（C3），已有 `output/CLAIMS.md` + `output/MISSING_EVIDENCE.md`（以及可选 novelty matrix）。 **Skip if**: 上游产物未就绪（claims/evidence gaps 缺失）或你不打算输出完整审稿报告。 **Network**: none. **Guardrail**: 给可执行建议（actionable feedback），并覆盖 novelty/soundness/clarity/impact；避免泛泛而谈。

377 25
Explore

Didn't find tool you were looking for?

Be as detailed as possible for better results