Text Splitter Tool

Functionality

Split text into specified-sized chunks while maintaining semantic integrity, facilitating subsequent batch processing and analysis.

Usage Scenarios

• Process extra-long text by splitting it into agent-processable segments
• Perform text preprocessing to provide standardized input for subsequent content analysis, summarization, or evaluation tasks
• Optimize text processing workflows to ensure each processing unit is controllable in size and semantically complete

Core Capabilities

• Semantic Integrity Priority: When splitting, avoid truncating sentences or paragraphs; prioritize splitting at natural boundaries (periods, line breaks) to ensure semantic integrity of each chunk
• Preserve Original Format: During splitting, preserve the original format and structure of text, such as Markdown, HTML tags, etc.
• Precise Chunk Size Control: Strictly split according to specified size limits, ensuring each text chunk does not exceed the preset maximum length
• Multiple Splitting Strategies: Support various splitting strategies based on character count, token count, paragraph count, or specific delimiters

Input Requirements

• Text Content: Original text to be split
• Chunk Size Limit: Maximum length of each text chunk (recommend providing character count or token count)
• Splitting Strategy (optional): Specify preferred splitting strategy, such as split by sentence, split by paragraph, split by specific delimiter, etc.

Output Format

[Text Splitting Report]

- Original text length: [integer] words/tokens
- Target chunk size: [integer] words/tokens
- Actual number of chunks: [integer] chunks

### Splitting Results
- Chunk 1 (length: [integer] words/tokens): "[content preview...]"
- Chunk 2 (length: [integer] words/tokens): "[content preview...]"
- Chunk 3 (length: [integer] words/tokens): "[content preview...]"
...

Constraints

• Splitting results must strictly comply with specified chunk size limits
• Ensure maximum preservation of text semantic integrity during splitting
• Output format must be structured, clearly displaying content and length of each text chunk
• Avoid introducing any extra information or explanations in output; only provide splitting results

Examples

See {baseDir}/references/examples.md for more detailed examples:

• examples.md - Contains splitting examples for different lengths, different splitting strategies, and complex text structures

Detailed Documentation

See {baseDir}/references/examples.md for detailed guidance and cases on the text splitter tool.

Version History