Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers

Hugging Face’s Sentence Transformers library now supports MultiVectorEncoder models in version 6.0, expanding its embedding capabilities beyond dense and sparse approaches to include ColBERT-style late interaction retrieval. The update introduces a complete training workflow for these models, allowing users to fine-tune existing multi-vector models or train new ones from base transformers using a single GPU. Installation is handled via pip install -U "sentence-transformers[train]".

Fine-tuning multi-vector models involves configuring components such as datasets, loss functions, training arguments, evaluators, and the trainer class. The process preserves the model’s architecture—including query and document marker tokens and projection heads—while adapting to domain-specific data. Practical examples demonstrate how to adjust document length limits and apply token-level optimizations like punctuation skiplists to improve retrieval performance and reduce index size.

Training multi-vector models yields significant gains in retrieval quality for long documents, where general-purpose models often truncate content before scoring. In a medical retrieval evaluation using 941-token passages, a fine-tuned model outperformed all general-purpose retrievers—dense, sparse, lexical, and multi-vector—after 14.5 hours on a single RTX 3090. The approach also addresses domain-specific challenges, as queries and relevance criteria vary widely across fields like medicine, law, or code search.

The training pipeline supports datasets from the Hugging Face Hub or local sources in formats such as CSV, JSON, or Parquet. For domain adaptation, starting with pre-supervised checkpoints often yields better results than fully fine-tuned models, which may regress during further training. The MultiVectorEncoderTrainer accommodates standard retrieval datasets, including MIRIAD’s 4.4 million medical question-passage pairs, demonstrating that simple (query, relevant passage) pairs are sufficient for effective fine-tuning.