Manticore Search adds chunking to improve vector search on long documents
vector-db
| Source: HN | Original article
Manticore Search adds a chunking feature that automatically splits long documents at insert time, improving vector‑search efficiency for extensive texts.
Manticore Search has rolled out a major upgrade that tackles one of the thorniest problems in AI‑driven retrieval: searching long, unstructured texts. Version 29.9.0 introduces a built‑in chunking mechanism that automatically splits documents at INSERT time. Users simply add a chunk_strategy parameter to a vector column and choose from five predefined strategies; the engine handles the rest, eliminating the need for external splitter libraries or custom ingest pipelines.
The change is more than a convenience feature. In internal benchmarks the company reports that recall@5 for deep‑content queries jumped from 55 % to 83 % once chunking was enabled. The release also adds multi‑vector auto‑embeddings, a MAX_INPUT_TOKENS limit to protect embedding models, and default memory‑mapped access to columnar attributes, alongside a suite of fixes for hybrid and KNN search.
Why it matters is twofold. First, developers building semantic search applications can now index full‑length articles, reports or manuals without hand‑crafting preprocessing steps, accelerating time‑to‑value. Second, the boost in recall signals that native chunking can close the performance gap that has traditionally required heavyweight pipelines, potentially reshaping the competitive landscape among open‑source and commercial vector search solutions.
The community will be watching how quickly the new capabilities are adopted in production environments, whether other search platforms introduce comparable native chunking, and what further refinements Manticore may deliver to address scaling, multilingual support and real‑time updates. The upgrade positions the fork of Sphinxsearch as a more complete answer for enterprises seeking high‑quality, AI‑enhanced retrieval without the overhead of bespoke data preparation.
Sources
Back to AIPULSEN