How Flo Health built safer, more useful open models with Databricks AI Runtime
Flo Health improved AI medical safety and usefulness scores by over 25 percentage points each using a privacy-first, judge-driven pipeline on Databricks, reducing inference costs and gaining control over model behavior.
Flo Health, a women’s health app with 81 million monthly active users, faced challenges balancing medical safety and usefulness in its generative AI systems. Traditional ML approaches limited flexibility and scalability, prompting Flo to explore fine-tuned open models. The company sought a compound AI system capable of selecting the right model for each task while maintaining consistent safety standards across applications.
To operationalize its safety and usefulness goals, Flo developed an evaluation framework with 26 AI judges, each designed to measure specific medical risks and response quality. The judges were validated against synthetic and anonymized real-user cases, with clinical reviewers resolving disagreements to establish ground truth. This system enabled scalable assessment without requiring manual labeling of every training example.
Flo then used these judges as reward signals for training, generating over 15,000 synthetic questions under privacy-first principles to create training pairs. The team adopted a Reinforcement Fine-Tuning approach, fine-tuning models to encode quality in their weights rather than relying on costly best-of-N search during production. This process improved both medical safety and usefulness scores significantly.
The fine-tuning workflow combined Databricks AI Runtime, LLM Foundry, and Compose-RL, leveraging 32 NVIDIA H100 GPUs for distributed training. The resulting model achieved a medical safety score of approximately 61%, usefulness of 76%, and a combined score of 47%, nearly tripling the initial combined benchmark. Flo plans to apply this system to newer open-weight models while preserving its validated safety and usefulness criteria.