How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code
Hugging Face details how Papers with Code uses Jobs, Buckets, and Inference Endpoints to power hybrid search combining keyword and vector retrieval for AI research papers.
Papers with Code, an open platform for AI research artifacts, introduced a hybrid search system to improve discovery of papers, models, and benchmarks. The system combines PostgreSQL’s full-text search with pgvector for semantic embeddings, using reciprocal rank fusion to merge results. This approach balances exact matches with fuzzy semantic similarity, addressing challenges like incomplete titles or typos in queries. The architecture separates throughput-heavy embedding generation from latency-sensitive online search to optimize performance and reliability.
Embedding generation is handled by Jobs, which process database snapshots into vector corpora using a pinned Qwen/Qwen3-Embedding-0.6B model. The process exports papers from PostgreSQL, streams them as JSONL shards, and stores outputs in a private Storage Bucket. Jobs run on NVIDIA L4 GPUs, with completed shards tracked to allow resumable work. The system supports multiple embedding dimensions (e.g., 256, 512, 1024) to evaluate trade-offs between storage and recall without additional inference costs.
Storage Buckets serve as mutable, S3-like object storage on the Hugging Face Hub, acting as the boundary between batch processing and production systems. Each artifact is validated against checksums and manifests before loading into PostgreSQL. Vector indexes are built using HNSW, with new generations activated only after full validation. The Bucket’s immutability is enforced at the application level, ensuring consistency across the pipeline from export to retrieval.
Online search relies on an authenticated Inference Endpoint running Text Embeddings Inference, which embeds queries at request time. The endpoint scales to zero when idle to control costs but must handle cold starts gracefully. If the endpoint fails or is unavailable, the system falls back to full-text search, ensuring users always receive results. In tests on a 5,000-paper corpus, the 256-dimensional index achieved high recall with low latency, demonstrating the system’s efficiency and resilience.