GitHub’s model for secret detection: Finding more secrets with less noise

GitHub replaced its LLM-based generic secret detection pipeline with a two-stage system: broad regex candidate generation followed by a fine-tuned ModernBERT classifier that reads each candidate's surrounding context (file path, variable names, nearby code) to judge whether it's a real credential or a lookalike. GitHub reports this cut median post-push scan time by 86%, serving cost by over 98%, and improved owner-weighted alert acceptance by 11.5% versus the prior LLM pipeline, and the speed gain let them extend detection to pre-push protection.

Why it matters

This is a concrete, demonstrated case of replacing a general-purpose LLM call with a small fine-tuned encoder for a narrow classification task, achieving large cost and latency reductions while improving precision — a pattern directly relevant to anyone running LLM-based classification at scale. It also strengthens a security control (secret scanning) used across a huge number of repositories.

What is verified

GitHub replaced an LLM-based secret detection pipeline with a fine-tuned ModernBERT classifier using a two-stage pipeline (regex candidate generation plus context-aware classification). [1]

GitHub reports 11.5% higher owner-weighted alert acceptance, 86% lower median post-push scan execution time, and more than 98% lower serving cost compared to the previous LLM pipeline. [1]

The efficiency gain allowed GitHub to extend generic secret detection to push protection, so secrets can be flagged before a push is accepted, not only after commit. [1]

The model is exported to ONNX format with 8-bit quantization for fast inference. [1]

What is not

• No independent benchmark or third-party reproduction of GitHub's reported improvement numbers is provided.
• The customer A/B evaluation methodology, sample size, and repository mix are not detailed beyond summary figures.
• No information on rollout timeline or whether this applies to all GitHub plans (free, Enterprise) or only specific tiers.

Sources

1. 1GitHub Changeloggithub.blog · Primary source

Original source published 13 October 2026 at 19:56.

An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.