Factorized MLLM Embeddings Enhance Composed Image Search
embeddings multimodal
| Source: HF Papers | Original article
A new method uses factorized MLLM embeddings to boost composed image retrieval, enhancing how reference images and modification text are combined.
A new paper titled **FactEmbed: Factorized MLLM Embeddings for Composed Image Retrieval** proposes a fresh way to tackle composed image retrieval (CIR), a task where a system must find a target image based on a reference picture and a textual modification. The authors argue that existing approaches, which feed the combined query straight into a multimodal large language model (MLLM) and extract a single contrastive embedding, leave the model’s compositional reasoning underexploited.
FactEmbed separates query understanding from embedding construction. It parses the reference‑image + text pair into a set of factorized latent tokens that simulate the visual change described by the text. These tokens are supervised through a lightweight decoder that reconstructs the target image, while a dual InfoNCE and global contrastive loss aligns them with the correct retrieval embedding. The result is a “factor‑level” contrastive objective that propagates the decomposed representations deeper into the MLLM’s attention layers.
The method sets new state‑of‑the‑art results on two standard CIR benchmarks. On Refined‑CIRR it reaches 77.82 % recall@1, and on the original CIRR test set it records 82.97 % recall@1. On FashionIQ validation it achieves an average recall@10 of 91.17 %, surpassing all previously reported numbers.
Why it matters: By explicitly modelling the visual modification before retrieval, FactEmbed extracts more nuanced semantics from MLLMs, narrowing the gap between language‑driven intent and visual search. This could sharpen e‑commerce visual search, content recommendation and any application that relies on fine‑grained image manipulation queries.
As we noted in our October 30 coverage of MLLM retrieval heads, the retrieval layer is a bottleneck for many multimodal systems. FactEmbed shows that rethinking the embedding pipeline can yield sizable gains. The next steps to watch include scaling the factorized approach to larger, more diverse datasets, integrating it with real‑time search engines, and exploring whether similar factorization can improve video‑based retrieval or cross‑modal generation tasks.
Sources
Back to AIPULSEN