GitHub’s model for secret detection: Finding more secrets with less noise

Secret scanning balances between missing real credentials and overwhelming developers with false positives. Generic secrets complicate this balance because the same string may appear in documentation or code without being a credential. GitHub’s new classifier distinguishes real secrets from lookalikes by analyzing surrounding code and text, reducing median post-push scan execution time by 86% compared to the previous LLM-based pipeline.

The pipeline uses regex patterns to flag potential credentials, then passes candidates to a fine-tuned ModernBERT classifier. This classifier evaluates each candidate alongside file path, variable names, and surrounding context to estimate the likelihood of a real secret. High-confidence findings become alerts, while likely false positives like placeholders or test fixtures are filtered out before reaching developers.

ModernBERT was selected for its compact size, efficiency, and suitability for classification tasks over code-adjacent text. Unlike general-purpose LLMs, it processes short candidate windows with high throughput and low latency, making it ideal for secret scanning. The model is trained directly on GitHub’s decision boundary and evaluated using recall and precision metrics relevant to secret scanning quality.

Training data included synthetic rows, real secrets, realistic false positives, and hard negatives that frequently confuse regex-only approaches. Each input combined the candidate value, file path, secret type, and a bounded amount of surrounding context. Thresholds were selected through offline evaluation and validated on production-like traffic before broader rollout.