Token Pooling-Aware Training for Multi-Vector Embedding Models

Multi-vector embedding models store separate vectors for each token in a document, enabling precise retrieval but requiring substantial storage. Pooling merges similar token vectors to reduce storage, though aggressive compression risks losing useful information. Pooling-aware training adjusts the encoder to retain critical details during pooling, improving retrieval performance within the same storage budget. The method is demonstrated using Sentence Transformers with a pool_factor of 4, showing how training can maintain or even enhance retrieval quality.

Token pooling reduces document vectors by grouping similar embeddings using hierarchical clustering, such as Ward’s method, which merges vectors with minimal increase in within-group variance. The process produces a hierarchy of groups, allowing selection of a compression level that fits storage constraints. With pool_factor=4, documents retain roughly one-quarter of their original vectors, with the first token protected from merging to preserve query relevance.

Pooling-aware training combines retrieval objectives for both pooled and unpooled representations, teaching the model to optimize embeddings for compressed retrieval. The loss function balances performance with and without pooling, enabling the encoder to learn from errors in pooled representations. Gradient caching supports training with small mini-batches while incorporating a larger set of in-batch negatives, improving efficiency and effectiveness.

Adaptive pooling introduces a distance threshold to dynamically adjust vector counts based on document content, merging aggressively where embeddings are similar and preserving more vectors where distinctions matter. Calibration against a representative corpus sample determines the threshold, allowing reuse during indexing. The approach is compatible with any multi-vector model, including those trained without pooling-aware objectives, and can be applied to models like ColBERT.