UniHIR is a unified multimodal large language model for end-to-end Historical Inscription Restoration (HIR), accepted at ACL 2026, that localizes damaged regions, predicts illegible characters, and restores full-page rubbing appearance through draft-guided localization and hierarchical self-refinement. Prior pipelines separated OCR-style text prediction from diffusion-based inpainting, accumulating irreversible errors and producing patchy typography. Zhang et al. from South China University of Technology show a single BAGEL-initialized model can iterate like human epigraphers: sketch damage, verify against context, refine boxes and glyphs, then synthesize consistent stone texture. Digital humanities teams following AI research on cultural heritage or building on AI research infrastructure for museum digitization should treat UniHIR as a reference architecture for page-level stele restoration rather than character-only OCR.
The ACL 2026 contribution matters because epigraphy sits at the intersection of NLP, computer vision, and conservation ethics: a wrong character is both a transcription error and a visual forgery on cultural heritage. UniHIR's explicit draft and verify stages mirror scholarly practice more closely than single-shot inpainting models marketed for generic document cleanup.
The HIR Problem: Fractures and Erosion
Historical Inscription Restoration aims to recover both readable text and visual appearance of damaged stone rubbing images under the principle of restoring the old as the old: repair only eroded regions while preserving original glyph style and background texture. Centuries of war, water, oxidation, and manual rubbing wear leave fractures, missing strokes, and illegible clusters on steles that anchor linguistic and cultural archives. Manual restoration by epigraphers takes days or months per monument and does not scale to thousands of endangered sites across China, Korea, Japan, and other inscription traditions.
Formally, given damaged rubbing image Id, a model outputs restored inscription Ir and character set C = {(ci, bi)} where each character content ci pairs with bounding box bi. Tasks demand joint visual and textual reasoning: a missing radical changes both transcription and how ink should be painted back onto stone texture.
Existing document restoration methods transfer imperfectly. Historical document AI often predicts text or cleans parchment patches without stele-specific layout rules (right-to-left, top-to-bottom columns) or lithic texture statistics. Patch-based diffusion restores local tiles but breaks global stroke width and spacing continuity across a full page.
Draft-Guided Localization
Draft-Guided Localization (DGL) first generates a visual damage-location draft: a scratchpad image with blue masks highlighting eroded regions, then conditions dense character prediction on both the original rubbing and the draft. Dense stele damage overwhelms vanilla detectors; even Qwen3-VL hallucinates shifted or overlapping boxes on hundreds of micro-fractures. UniHIR inverts the usual MLLM flow: generation guides understanding. A flow-matching branch synthesizes draft D from VAE latents of Id; an autoregressive branch then outputs serialized tuples (character, bounding box) in conventional inscription reading order.
Experiments in the ACL paper report that explicit spatial drafts improve content prediction versus understanding-first pipelines. UniHIR also drops redundant VAE features after draft generation during inference, cutting latency and memory versus running full dual encoders throughout.
The draft metaphor matches expert workflow: epigraphers mark suspect zones lightly before committing ink reconstructions. Automating that mark-up as a trainable intermediate supervision signal is UniHIR's central localization insight.
Hierarchical Self-Refinement Loops
Hierarchical Self-Refinement (HSR) iterates macro-level semantic grounding and micro-level spatial-semantic correction before final appearance synthesis. Macro refinement infers global attributes A: source text, date, engraver identity. Micro refinement performs spatial edits (add, remove, adjust boxes for false positives and missed cracks), semantic coherence checks using surrounding five-character context windows, and draft-mediated feedback where regenerated drafts validate alignment between predicted boxes and visual damage masks.
Each iteration updates predictions C(t) and attributes A(t) until verification stabilizes. This draft-verify-restore loop addresses single-pass hallucination: early wrong characters get corrected before pixels are painted, unlike cascaded pipelines where OCR errors poison inpainting unconditionally.
On severely damaged steles where baseline OCR accuracy sits near 55.68 percent, UniHIR reports boosting transcription accuracy to 86.65 percent in paper benchmarks, alongside improved appearance metrics for stroke consistency. Independent museum replication should confirm on held-out collections not in HIRBench.
| Stage | Function | Output |
|---|---|---|
| Draft-Guided Localization | Mark damage, predict characters | Draft mask + initial C |
| Macro HSR | Source, date, engraver context | Global attributes A |
| Micro HSR | Box and character verification | Refined C and draft |
| Appearance Restoration | Full-page texture synthesis | Restored rubbing Ir |
HIRBench and UHIRFactory Training
Authors built HIRBench with step-aware annotations for intermediate drafts and refinements, and UHIRFactory for memory-efficient step-isolated instruction tuning on high-resolution long sequences. Full-page rubbings exceed typical MLLM context budgets. Step-wise training isolates optimization per pipeline stage, sampling data adaptively to fit GPU memory while preserving supervision on draft quality, box edits, and final pixels. Public code and weights are released at github.com/ZZXF11/UniHIR with ACL Anthology DOI 10.18653/v1/2026.acl-long.1254.
HIRBench enables ablations proving each component (DGL alone, HSR alone, unified training) contributes. Comparable benchmarks let future papers (EpiAgent, diffusion-only baselines) report on shared splits instead of incomparable private stele photos.
UniHIR initializes from BAGEL, a unified MLLM combining SigLIP2 vision encoding, FLUX VAE latents, and Qwen2.5 Mixture-of-Transformer-Experts for alternating understanding and rectified-flow generation. Heritage tasks thus ride general unified-model progress rather than bespoke CNN-only inpainting stacks.
ACL 2026 Publication Details
UniHIR appears in ACL 2026 Volume 1 Long Papers, pages 27216-27231, presented in San Diego with DOI 10.18653/v1/2026.acl-long.1254. Authors Yuyi Zhang, Junle Liu, Peirong Zhang, Jianliang Liu, Zhenhua Yang, and Lianwen Jin from South China University of Technology's DLVC Lab released weights and inference scripts in July 2026. The title "Draft, Verify, Restore" encodes the operational loop museums would audit step by step rather than trusting a single forward pass.
Peer reviewers emphasized page-level consistency metrics absent in prior HIR leaderboards. HIRBench step-aware labels let future work ablate draft quality separately from final pixel loss, improving scientific comparability across restoration papers in computational linguistics and computer vision venues.
Museum Digitization Impact
Museums and archives could use UniHIR-style assistants to prioritize which steles need human conservator hours, generate scholarly drafts for peer review, and produce public-facing legible facsimiles without touching physical stone. Workflow integration might look like: high-resolution scan ingestion, automated draft transcript with confidence heatmaps, epigrapher correction UI, locked approved version for publication, separate watermark for speculative fills. None of this removes expert sign-off for authoritative editions; it compresses first-pass labor on heavily damaged corpora.
Cross-institution sharing demands metadata standards linking restored characters to CIDOC-CRM entities and recording which strokes were inferred versus measured. Provenance blocks forgeries: models must not invent politically convenient text; HSR's verify step is a technical mirror of scholarly skepticism.
Colonial collection ethics apply when restoring inscriptions removed from original sites. Community stakeholders should govern whether AI fills appear in visitor apps for diaspora languages still under revival efforts.
Raster scan resolutions above 600 dpi on full stele faces stress GPU memory; UHIRFactory's step-isolated training exists precisely because heritage institutions will not downsample culturally significant rubbings just to fit consumer hardware. Cloud batch jobs with conservator review queues are the realistic near-term deployment shape.
Appearance Restoration and Typography
After HSR converges, UniHIR's appearance restoration branch synthesizes pixel-level rubbing texture so inferred characters match stroke width, ink density, and background weathering of intact neighbors. Patch-based diffusion predecessors often produced sharper, anachronistically clean glyphs beside genuinely aged stone noise. Page-level unified generation propagates style vectors across the full rubbing, preserving column alignment and engraver hand consistency. The ACL paper positions this as a key advantage over three-stage pipelines that OCR first, then inpaint locally without re-verifying text against global layout.
Rectified-flow generation in BAGEL's architecture alternates understanding tokens and VAE latents, letting UniHIR fluidly switch between interpreting damage and painting repairs. Heritage conservators still reject fully automated publication without human sign-off on each character, especially for legal or religious texts where erroneous restoration carries cultural harm.
Comparison With Prior HIR Pipelines
Earlier systems such as Aeneas combined vision backbones with Transformers for text, geographic origin, and date prediction but remained text-centric without full appearance repair. Zhang et al. (2025d) and Zhu et al. (2024) used decoupled LLM plus diffusion stages: language models guess missing characters, diffusion models paint patches. Error accumulation is structural: a wrong OCR hypothesis becomes a confident visual forgery. UniHIR's verify loops explicitly target that failure mode with draft regeneration and contextual five-character windows borrowed from human epigraphic practice.
Crowdsourcing frameworks like Han et al. (2024) blended human and machine labor; UniHIR automates first pass but implies human-in-the-loop review UI for museum deployment. Digital epigraphy conferences will likely benchmark UniHIR against EpiAgent and other agentic restorers on HIRBench splits as code propagates.
Open rubbings in HIRBench also enable negative controls: models must learn when damage is too severe for confident restoration and flag uncertainty rather than fabricating plausible but false historical narratives attractive to sensational media coverage.
Frequently Asked Questions
Which languages and scripts does UniHIR target?
The ACL paper focuses on Chinese stone inscription rubbings with classical layout conventions. Transfer to Korean, Japanese kanbun, or Latin epigraphy requires new benchmarks and fine-tuning; architecture is script-agnostic but data is not yet.
Can UniHIR fabricate plausible fake inscriptions?
Any generative restorer can hallucinate if verification is skipped. Museums should require human approval and publish uncertainty maps. HSR reduces but does not eliminate semantic invention on highly degraded panels.
How does UniHIR compare to agentic systems like EpiAgent?
UniHIR unifies steps in one MLLM with internal refinement loops. Agent frameworks orchestrate external tools with explicit Observe-Conceive-Execute cycles. Choice depends on maintainability and whether institutions want modular toolchains versus single-model deployment.
Is UniHIR production-ready for museums?
Research code exists; production needs GPU serving, conservator UI, audit logs, and legal review. Treat 2026 release as research prototype, not certified conservation instrument.
When is OCR-only enough?
Clear rubbings with minor gaps may need transcription help only. UniHIR targets cases where visual appearance restoration matters for publication-quality facsimiles and stroke-level scholarship.
What hardware runs inference?
Unified MLLMs with high-resolution images demand modern GPUs; paper emphasizes memory-adaptive training implying substantial VRAM at inference. Edge deployment on museum tablets is unlikely without cloud offload.
How should researchers cite UniHIR?
Use the ACL 2026 proceedings citation (Zhang et al., pages 27216-27231) and specify checkpoint version from the GitHub repository when comparing reproduction results.
Does UniHIR input rubbings or field photos?
The paper trains on rubbing images (ink on paper transfers of stone inscriptions), the standard epigraphic archive format. Field photos with glare require domain adaptation not reported in the main benchmark.
What should human experts still do?
Validate every proposed character against paleographic corpora, reject culturally sensitive inferences, and approve public release. UniHIR accelerates first drafts; scholarly editions remain human-governed.
Why build on BAGEL?
BAGEL natively alternates visual understanding and image-to-image generation inside one MoT backbone, matching HIR needs better than bolting a diffusion inpainter onto a text-only LLM without shared parameters.
Can museums commercialize UniHIR outputs?
Check GitHub license and institutional agreements with SCUT authors. Scholarly and public-education uses are the stated intent; merchandise or NFT-style resale of restored rubbings needs separate legal review.
