The Definitive Fine-Tuning Guide for 3D Generation Models

Hugging Face introduces scripts and containers to fine-tune TRELLIS.2, a 3D generation model, allowing users to adapt it for specific tasks such as accurate furniture geometry or detailed shoe soles. The process supports both local execution and Hugging Face Jobs, simplifying deployment. Users can leverage preconfigured scripts or customize them via a provided repository, streamlining the fine-tuning workflow for 3D assets.

TRELLIS.2 uses a unified latent representation to standardize 3D model outputs across formats like GLB and PLY, addressing inconsistencies in 3D generation. The model’s pipeline includes stages for background removal, image encoding, coarse structure prediction, detailed geometry generation, and material creation. LoRA adapters are added to specific DiT components to refine shape and geometry while preserving texture capabilities, ensuring targeted improvements without disrupting existing strengths.

The fine-tuning process targets the sparse-structure flow DiT and shape flow DiT components, which control object geometry, using LoRA adapters with ranks 8 and 16 respectively. The texture flow DiT remains frozen to avoid catastrophic forgetting, as the dataset lacks separate material maps. The configuration trains approximately 9.3 million adapter parameters, balancing efficiency and performance for small datasets like the 188-footwear subset from Google Scanned Objects.

Hugging Face provides two Docker images—one for preprocessing and one for fine-tuning—to simplify setup, compatible with Hugging Face Jobs or local GPU environments. The scripts include detailed explanations of settings and their impact on training, enabling users to optimize the process for their specific 3D generation needs while minimizing computational overhead.