How to evaluate LLMs before production

Language models may excel in controlled benchmarks yet fail in real-world scenarios due to ambiguous inputs, inconsistent labels, and edge cases absent from curated datasets. Benchmarks are useful for initial prototyping but often do not reflect production conditions, where noise, missing context, and distribution shifts can degrade performance. GitHub encountered these challenges while developing an LLM-based system to reduce false positives in secret scanning, where alerts resembling but not containing real credentials waste developer time. The team found that offline metrics did not reliably predict production behavior, necessitating a shift toward evaluation methods aligned with actual workflow requirements.

The evaluation process required defining clear decision criteria before adjusting technical components like prompts or models. GitHub prioritized reducing false positives while treating recall as a non-negotiable safety constraint, as suppressing real credentials could have severe consequences. Metrics were not treated as interchangeable; a change improving precision but harming recall was rejected, as was one that compromised speed, cost, or integration feasibility. Success was measured by whether the system improved developer experience without violating predefined guardrails, ensuring alignment with product goals rather than isolated metric improvements.

Evaluation was treated as an ongoing process rather than a one-time event, with GitHub rerunning tests whenever prompts, models, or input construction changed. This approach mirrored end-to-end integration testing, ensuring that any regression or unexpected behavior was detected promptly. The team emphasized repeatability by recording prompt versions, model choices, dataset iterations, and system configurations for each run, enabling accurate comparisons against baselines.

The lessons apply broadly to LLM-powered systems in code analysis, security, and developer tools, where production performance depends on more than benchmark scores. GitHub’s approach underscores the importance of defining user-centric success criteria and guardrails upfront, designing experiments to isolate causes of results, and maintaining disciplined evaluation practices throughout iterative development cycles.