BPE Tokenization for Secret Detection: Higher Recall Than Entropy-Only

Secret detection has traditionally relied on two approaches: regular expression patterns that match known credential formats, and Shannon entropy analysis that flags high-randomness strings as potential secrets. The first approach is precise but limited to known formats. The second catches unknown formats but generates excessive false positives. At Netallion AI Assurance, we developed a third approach based on byte-pair encoding (BPE) tokenization — the same subword segmentation LLMs use — which lets the engine isolate the high-entropy subsequences and structural fingerprints that regex cannot express and that raw entropy scoring cannot separate from UUIDs, hashes, and base64. The result is substantially higher recall on generic secrets than entropy-only methods, without drowning security teams in false positives.

The Problem with Entropy Analysis

Shannon entropy measures the randomness of a string by calculating the distribution of characters. A string like "password123" has low entropy because the character distribution is predictable. A string like "aK7mN2pQ9rS4tV6wX8yZ0bD3eF5gH" has high entropy because the characters are well-distributed. Entropy analysis uses a threshold to classify high-entropy strings as potential secrets.

The problem is that many legitimate strings also have high entropy. UUIDs ("550e8400-e29b-41d4-a716-446655440000"), content hashes, base64-encoded non-secret data, and even some natural language produce entropy values above typical thresholds. Entropy-only analysis with a threshold tuned for reasonable recall flags a large fraction of all strings in a typical codebase as potential secrets. This makes entropy analysis impractical as a primary detection mechanism without extensive suppression tuning.

Lowering the entropy threshold reduces false positives but also reduces recall. Holding entropy-only analysis to a low false positive rate cuts its recall on generic secrets sharply, and a substantial share of real secrets is missed.

What BPE Tokenization Brings to the Table

Byte-pair encoding is a subword tokenization algorithm originally developed for text compression and later adopted by large language models for vocabulary construction. BPE iteratively merges the most frequent pairs of bytes (or characters) in a corpus to build a vocabulary of subword tokens. The key insight for secret detection is that the tokenization pattern of a string reveals its structural properties in ways that entropy alone cannot capture.

When BPE processes a real secret like an API key, the tokenization tends to produce a characteristic pattern: relatively few merges, many single-character tokens, and low token-to-character ratio. This is because secrets are designed to be random and do not contain the repeated subword patterns that BPE learns from natural text or structured data.

When BPE processes a non-secret high-entropy string like a UUID or content hash, the tokenization is different. UUIDs have consistent structural patterns (groups of hexadecimal characters separated by hyphens) that BPE recognizes and merges efficiently. Base64-encoded data contains recurring character pairs that map to common byte patterns. These structural regularities produce higher merge rates and different token distribution profiles than true random secrets.

The Netallion AI Assurance Detection Pipeline

Netallion AI Assurance uses a three-stage detection pipeline that combines regex patterns, BPE tokenization analysis, and live verification.

In the first stage, 467 regular expression patterns scan for known secret formats. This catches the majority of secrets with specific, well-defined structures: AWS access keys, GitHub personal access tokens, Stripe API keys, and hundreds more. Regex matching is fast, precise, and produces minimal false positives for well-defined formats.

In the second stage, strings that do not match any known pattern are analyzed using BPE tokenization. This is where generic secrets are caught: custom tokens, internal credential schemes, and split or obfuscated credentials that have no regex to match and no entropy signature clean enough to isolate on its own. The BPE model was evaluated on a labeled corpus of confirmed secrets and confirmed non-secrets, learning the tokenization fingerprints that separate true random credentials from structured high-entropy noise. It produces a confidence score for each candidate string, and strings above the threshold are escalated as potential generic secrets.

In the third stage, high-confidence findings from both stages are validated against 20 live verifiers. The verifiers make API calls to the respective service providers (AWS STS, GitHub API, Stripe API, and others) to determine whether the detected credential is still active. Verified-active findings are escalated to critical severity. Verified-inactive findings are downgraded. Unverifiable findings retain their confidence-based severity.

Why BPE Wins the Comparison

We evaluated the Netallion AI Assurance detection engine against a labeled corpus that mixes confirmed secrets (both known-format and generic), high-entropy non-secrets (UUIDs, hashes, base64), medium-entropy code strings, and low-entropy text strings. The structure of that corpus is exactly where the mechanism advantage shows up: it is deliberately loaded with the high-entropy non-secrets that trip entropy-only scanners, and with generic secrets that regex has no pattern for.

The mechanism is why the gap exists, not a tuning artifact. Entropy-only detection is stuck on a single dial: raise the threshold and it misses generic secrets that use restricted character sets or short lengths; lower it and it drowns in UUIDs, content hashes, and base64 payloads that are just as random-looking but perfectly legitimate. BPE sidesteps the dial entirely. A true random credential fragments into many rare subword tokens, while a UUID or hash carries the recurring byte pairs BPE learned to merge — so the two land in different token-distribution profiles even when their raw entropy is identical. That is a distinction regex cannot express and pure entropy cannot isolate, which is exactly why BPE-augmented detection recalls substantially more generic secrets than entropy-only analysis at the same false positive budget.

Implications for Enterprise Security

The recall gap between BPE tokenization and entropy-only analysis translates directly to missed secrets in production environments. For an organization with many generic secrets exposed across their infrastructure, entropy-only analysis would leave a meaningful fraction undetected, while BPE tokenization catches nearly all of them. Those additional detected secrets represent real credentials that an attacker could exploit.

The precision improvement is equally important for enterprise adoption. A tool that generates excessive false positives is a tool that security teams learn to ignore. Netallion AI Assurance's high precision means the overwhelming majority of findings are real secrets, making every alert actionable. Combined with live verification that brings the effective false positive rate for verified types to near zero, the detection engine maintains the trust of security teams handling hundreds of findings per week.

What Comes Next

We are continuing to invest in the BPE tokenization approach. Our current roadmap includes expanding the training corpus with additional secret types, tuning the model for specific data sources (log entries have different characteristics than code files), and developing domain-specific BPE vocabularies for cloud provider credential formats. We believe that statistical structural analysis will become the standard approach for generic secret detection over the next two to three years, complementing rather than replacing regex pattern matching for known formats.

If you want to see the BPE detection engine in action on your own data, Netallion AI Assurance offers a 14-day Business trial. Connect your Azure Monitor workspaces or GitHub repositories and see the generic secrets that regex- and entropy-only scanners miss — each one verified active before it reaches your queue.