Chunkr

What is Chunkr?

Chunkr is an open-source document processing API from Lumina AI, designed specifically for RAG (Retrieval Augmentation) and knowledge base scenarios. Chunkr can convert complex documents (such as PDFs, PPTs, Word documents, and images) into structured data, supporting intelligent parsing across multiple formats. Chunkr’s core features include high-precision OCR, semantic chunking, multi-format output (HTML, Markdown, JSON, plain text), and seamless integration with various LLMs (such as OpenAI, Claude, and Ollama). Users can quickly get started via cloud services or deploy locally using Docker. Chunkr performs exceptionally well in document question answering, enterprise knowledge bases, OCR scenarios, and RAG systems, making it a powerful tool for document processing.

Chunkr’s main functions

• Multi-format document parsing : Supports multiple formats such as PDF, PPT, Word, and images, and can convert complex documents into structured data.
• High-precision OCR : Extracts text while preserving the spatial relationships and positional information of the text, and supports OCR with bounding boxes.
• Semantic chunking : Automatically segments the document into contextual chunks suitable for RAG and LLM, facilitating subsequent processing.
• Multi-format output : Supports output results in multiple formats such as HTML, Markdown, JSON, and plain text.
• Python SDK : Provides a Python SDK for easy integration into Python applications or backend services.
• LLM support : Supports various local or remote LLMs (such as OpenAI, Claude, Ollam, etc.) and can be flexibly configured.

Chunkr’s technical principles

• Visual Language Model (VLM) : Chunkr uses a Visual Language Model (VLM) to understand the layout and content of documents. Combining computer vision and natural language processing techniques, VLM can identify elements such as text, images, and tables in a document and understand spatial relationships. Based on VLM, Chunkr can achieve high-precision OCR and semantic segmentation, ensuring accurate extraction and reasonable segmentation of document content.
• Document Layout Analysis : Chunkr analyzes the document layout, identifying the position and structure of elements such as headings, paragraphs, tables, and charts. Based on this layout analysis, the document content is divided into blocks according to its logical structure, generating contextual blocks suitable for RAG and LLM processing.
• OCR Technology : Chunkr uses advanced OCR technology to extract text content from documents while preserving the text’s location information and spatial relationships. The text and location information extracted by OCR are used in subsequent semantic segmentation and structured processing.
• Semantic chunking : Chunkr uses natural language processing techniques to perform semantic analysis on the extracted text, dividing the document content into logically independent chunks. Each chunk contains relevant contextual information, making it suitable for direct use as input for RAG or LLM.

Chunkr’s project address

• Project website : https://chunkr.ai/
• GitHub repository : https://github.com/lumina-ai-inc/chunkr

Chunkr’s application scenarios

• Document Question Answering System : Converts complex documents into structured data, generates a high-quality corpus, and provides accurate contextual information for the question answering system.
• Enterprise knowledge base construction : Quickly convert internal enterprise documents into structured data, efficiently build a knowledge base, and improve knowledge management efficiency.
• OCR Scenarios : Provides high-precision OCR and text location information, supporting accurate recognition of complex documents (such as tables and mixed text and images).
• RAG System : Outputs structured data (such as JSON, Markdown) suitable for the RAG system, improving retrieval efficiency and generation quality.
• Intelligent document processing : Using semantic chunking and LLM support, it enables intelligent processing functions such as document summarization, classification, and automatic annotation.