How precise is secret detection? We measured it

The numbers

The false-positive count is the headline; recall on the tested formats is 100% (111 of 111), held against regression, and the gap list is published in full. On a tool you run against your own work every day, a quiet, honest scanner beats a loud, overreaching one.

Why a false-positive number is the right number

A precision claim is only worth as much as the corpus behind it. The credible bar on the secrets surface was set by published benchmarks with a named dataset, a stated method, and exact precision and recall figures. We follow that bar: a defined corpus, a method published in full, and a number that is the output of one command - reproducible and falsifiable, not a marketing figure.

The false-positive rate is the number that matters for a tool you run against your own work every day. The hard part of secret detection is not matching AKIA...; it is staying silent on the forty other things that look like AKIA.... A 40-character hex string is a git SHA, a content checksum, a request ID, or an AWS secret key, depending only on context. Skarn's value is that it tells them apart.

Skarn is not the only scanner that reads this surface locally. GitGuardian's Developer Endpoint Protection, announced 2026-06-16, scans AI tool directories and log files on the endpoint, and states that only structured metadata leaves it - so the honest axis of comparison is not local versus cloud. It is what gets correlated. The documented scope of that product is a credential inventory, honeytokens, and forwarding to a SIEM; as of that announcement it documents no attack-chain, behavioral, or MITRE ATLAS correlation across the session. A credential inventory answers "is there a key here". Skarn measures precision on that question and on the harder one next to it - did this session behave like an attack - and publishes the false-positive rate for each, with the command that produces it.

How the corpus is built - real scenarios, synthetic secrets

Two rules govern every fixture.

Real scenarios

Every credential format is grounded in its authoritative source - an official provider doc, a published detector definition, or a format spec. Almost every benign value is grounded in the spec or the documented scanner false-positive that establishes it as a non-secret: the UUID and JWT RFCs, the ULID, nanoid, and KSUID specifications, the W3C trace-context header, the Subresource Integrity spec, npm and Go module checksum formats, and real scanner false-positive reports. 158 of the 161 secret-track fixtures carry that citation in their label metadata. The other 3 are grounded in something narrower but no weaker: they are regression guards for false positives this benchmark caught in Skarn itself, which we then fixed.

Synthetic secrets

Not one real credential appears anywhere in the corpus. Every token is synthetic, format-valid, and non-live - it matches a detector's shape, and its checksum where one applies, such as a GitHub token's CRC, but unlocks nothing. The two values that look most real are deliberately the universal PUBLIC documentation constants - the example JWTs printed in the spec and on jwt.io, and a payment provider's shared sample test key - included as decoys precisely to prove the scanner stays quiet on documentation. An automated invariant extracts every secret-shaped token from the corpus and proves none of them ever appears, unredacted, in any output Skarn produces; 189 synthetic secrets are pushed through that gate and it passes.

The benign families we test against

These are the values that fool a naive secret scanner. Skarn stays silent on all 40, across 5 families.

• Identifiers (12) - UUIDs, ULIDs, nanoids, KSUIDs, distributed-tracing IDs, error-tracker event IDs, and the object and request IDs that payment and cloud APIs return on every call. Deliberately public.
• Content hashes (8) - git SHA-1 and SHA-256 object names, Docker image digests, Cargo and Go module checksums, Subresource Integrity attributes, npm lockfile integrity, and the value of a field whose own name says it holds a digest. Fingerprints of public artifacts, not keys.
• Example and placeholder keys (8) - documentation sample keys, publishable (client-side) keys, masked scanner output, .env.example placeholders, angle-bracket template placeholders, and a licence token quoted in prose or carrying a checksum that does not validate.
• Code and config identifiers (7) - dotted symbol references, .NET assembly public-key tokens, environment-variable NAMES referenced in deploy scripts, public-key config fields, a public analytics beacon token embedded in page HTML, and version pins that hold a commit SHA.
• Encoded data (5) - inline base64 images and fonts, sealed and encrypted ciphertext blobs, and the canonical example JWTs from specs and docs.

The corpus is built to be hard, not easy. It includes the boundary cases - a publishable key versus a secret key, a variable's NAME versus its value, a 40-hex commit SHA versus a 40-hex secret - precisely the places a sloppy scanner turns a config file into a false alarm. Each one trips a different suppression gate and stays silent, and each is a regression guard that fails the build if it ever starts firing.

The formats we detect

The same corpus proves the scanner did not go quiet to win on precision. 111 positives confirm current-format detection across 73 services: OpenAI (including the project, service-account, and admin key shapes), Anthropic, Groq, DeepSeek, xAI, Perplexity, Replicate, Hugging Face, Cohere, Mistral, Together, Fireworks, NVIDIA, OpenRouter, Voyage AI, Jina, Langfuse, Mem0, LlamaCloud, Cursor, Composio, RunPod, MiniMax, Friendli, Zhipu, Dify, Baseten, Ollama, and Deepgram; AWS (including both Bedrock key lifetimes), Azure, DigitalOcean, Fly.io, and Render; GitHub and GitLab in all their token variants, Bitbucket, npm, PyPI, RubyGems, Docker Hub, Buildkite, CircleCI, Vercel, Netlify, Jenkins, AppVeyor, and TeamCity; Stripe, Twilio, SendGrid, Mailgun, Slack, Discord, Microsoft Teams, Telegram, Postmark, Square, and PayPal; Supabase, Neon, MongoDB, Redis, Vault, Doppler, Dynatrace, Grafana, 1Password, connection strings, JWTs, SSH and PEM private keys, and Skarn's own licence tokens. Each positive is generated to match the live rule, so a miss is a real recall regression that fails the gate.

What we do not yet catch - and why we tell you

10 current formats have no dedicated rule yet: a handful of newer AI providers, including one whose newest key shape is still rolling out; a cloud storage shared-access-signature token; several unprefixed cloud and VPS provider tokens whose shape is too generic to match safely; a package-registry token; a managed-Redis token; and one managed-database password prefix. Some are still caught by generic high-entropy detection, with less precision and no service attribution. We track them as a published gap list rather than add aggressive rules that would regress precision elsewhere - because on a tool you run daily, a quiet, honest scanner beats a loud, overreaching one. The list shrinks over time; the honesty does not. A scanner's candour about what it misses is part of its trustworthiness, so we publish the gap register next to the precision number rather than behind it.

Reproduce it

The number is not a slide. It is the output of one command - so here is the command, the corpus it runs over, and the scorecard it prints.

The benchmark and the labeled corpus ship with the tool. One command scores the whole corpus and gates the build: it fails the moment one new benign value trips a detector, one current-format positive stops firing, or a number published on this page stops matching the corpus. A second command proves the corpus holds no live credential. skarn taxonomies prints the same figures from the binary itself.

What the first command scores, on the secret track: 161 labeled fixtures, in the four label kinds the corpus uses.

• 111 positives - current credential formats across 73 services, each a format-valid synthetic token generated to match the live rule. Each must fire its expected rule; a miss is a recall regression and fails the build.
• 40 decoys - benign, credential-shaped values across the 5 false-positive families: 12 identifiers, 8 content hashes, 8 example and placeholder keys, 7 code and config identifiers, 5 encoded data. Each must produce no finding at all; one finding is a false positive and fails the build.
• 10 known gaps - current formats with no dedicated rule yet. Tracked, published on this page, and deliberately gate-neutral: they are the register, not a failure.
• 0 known bugs - the fourth kind, for a rule that exists but is defective. That register is currently empty.

158 of the 161 carry a cited public source in their label metadata; the other 3 are regression guards for false positives this benchmark caught in Skarn itself. The wider corpus these sit in holds 261 fixtures across the behavioral, supply-chain, lockfile, and insecure-code tracks as well.

Every token in it is synthetic and non-live, and that is not an assurance but a second gate: zig build -Dtest-mode redaction-check extracts all 189 secret-shaped tokens from the corpus and asserts none of them appears unredacted in any output Skarn produces - every format, the baseline file, the audit log, the export, the local scan API, and the guard verdicts.

The scorecard the gate prints:

The raw corpus is not published as a download, and the reason is the same one that governs the rest of this page. The fixture bodies, the generator, and the scoring rule map together spell out the exact shape each detector keys on and the precise boundary of every suppression - a tuning guide for evading the scanner. What we miss is not the withheld part: the gap register is published in full, above. What is withheld is the map of how each rule decides. The manifest, the command, and the expected output are the falsifiable part - they state exactly what is measured and what result would contradict it.

Honest limitations

Stated plainly so the number survives scrutiny rather than inviting it.

It is precision, not prevalence

The number is a false-positive measurement over a curated corpus, not a field study. It says Skarn does not false-fire on these 40 real benign values - not that the false-positive rate in the wild is exactly zero. A corpus sampled from real traffic would strengthen it further; this one is built adversarially from the documented false-positive families, which is the right first bar.

Recall is 100% on the tested set, not a completeness claim

All 111 tested formats fire - 100% recall on the tested set, held against regression - but that is coverage of the formats we test, not a claim of completeness over every credential format in existence. The 10-entry gap register is the explicit incompleteness statement; several of those gaps are still caught generically, so the true miss set is smaller than 10 and the register is a conservative upper bound. The false-positive rate stays the number we lead with.

Test keys are still flagged on purpose

A test-mode key and the canonical AWS docs example key are intentionally still flagged as secrets; only the universally-shared sample constants - a payment provider's shared sample test key and the canonical example JWTs - are suppressed. A test-mode key is a real, if low-blast-radius, secret, so environment classification down-ranks it rather than hiding it.

References and sources

The credential formats are grounded in scanner detector definitions and official specs; the benign families cite the same specs and the documented scanner false-positives that establish each value as a non-secret. Each source was fetched and verified.

• gitleaks default configuration - the detector definitions the credential formats are grounded against: github.com/gitleaks/gitleaks.
• trufflehog detectors - provider key formats and shapes: github.com/trufflesecurity/trufflehog.
• gitleaks #1578 - generic-api-key false positives on public_key= and similar names: github.com/gitleaks/gitleaks/issues/1578.
• gitleaks #1775 - generic-api-key false positives on git commit hashes in build config: github.com/gitleaks/gitleaks/issues/1775.
• gitleaks #1728 - Kubernetes SealedSecret ciphertext detected as a generic-api-key: github.com/gitleaks/gitleaks/issues/1728.
• gitleaks #1830 - entropy detection firing on plaintext variable names and placeholders: github.com/gitleaks/gitleaks/issues/1830.
• trufflehog #4631 - detector false positives on checksums, encrypted data, and service names: github.com/trufflesecurity/trufflehog/issues/4631.
• detect-secrets #384 - base64 images in notebooks flagged as high-entropy secrets: github.com/Yelp/detect-secrets/issues/384.
• RFC 7519 - JSON Web Token (the header.payload.signature structure and example tokens): datatracker.ietf.org/doc/html/rfc7519.
• RFC 4122 - Universally Unique Identifier (UUID) format: datatracker.ietf.org/doc/html/rfc4122.
• W3C Trace Context - the traceparent header format: w3.org/TR/trace-context.
• W3C Subresource Integrity - the integrity attribute hash format: w3.org/TR/SRI.
• GitGuardian, extending our mission with Developer Endpoint Protection (2026-06-16) - the source for the scope described above: local endpoint scanning of AI tool directories and log files, metadata-only egress, credential inventory, honeytokens, and SIEM forwarding: blog.gitguardian.com.

Identifier and checksum specs grounding the benign families, not linked individually above: the ULID, nanoid, and KSUID specifications; the Go module go.sum checksum format and the npm package-lock integrity field; and the .NET strong-name public-key-token documentation. Provider key formats for the 73 covered services are grounded in the gitleaks and trufflehog detector definitions above and the respective official provider docs. Sources re-verified 2026-07-12; corpus counts re-measured against the live gate the same day.