Pangram’s Max Spero on why AI detection is harder than ‘Real or Fake’
The internet has a trust problem, and it's not just because social media feeds are filling up with AI slop. AI-generated text and images are now making their way into job applications, product reviews, and even insurance claims, leaving platforms and users alike scrambling to figure out what's real. A handful of startups have cropped up in the past couple of years promising to solve this with a simple binary verdict: human or machine. But Max Spero, co-founder and CEO of Pangram Labs, argues that approach is fundamentally flawed. In a recent conversation, he explained why detection is an evolving arms race, why "accuracy" is the wrong metric to chase, and what small businesses actually need to know before they trust a detector with their reputation.
The Binary Trap: Why Real vs. Fake Misses the Point
Most public discourse treats AI detection like a metal detector at an airport: you walk through, it beeps or it doesn't. Spero says that mental model is dangerous. "The 'Real or Fake' framing implies a ground truth that often doesn't exist," he said. "A cover letter written by a human but heavily edited by ChatGPT, a product description drafted by an AI and polished by a marketer, a student essay outlined by a human but written sentence-by-sentence with a copilot — these are the norm, not the exception. Forcing a binary label on hybrid content creates false positives that hurt real people and false negatives that let bad actors slide."
Pangram's approach, Spero explained, is to move away from a single probability score toward a more granular analysis. Instead of asking "Is this AI?", their models attempt to answer "Which parts look synthetic, and why?" This distinction matters immensely for platforms handling high volumes of user-generated content. A job board that auto-rejects every application flagged as "AI" risks discarding qualified candidates who used a writing assistant for grammar. An insurance company that auto-approves claims passing a binary check risks paying out on fabricated narratives. The nuance isn't academic; it's operational risk.
The Arms Race: Training Data, Adversarial Attacks, and the "Humanizer" Market
Detection models are only as good as the data they train on, and that data is shifting weekly. Spero described a three-front war. First, foundation models (GPT-4, Claude, Llama, Gemini) improve constantly, making their output statistically closer to human distributions. Second, a cottage industry of "humanizer" tools — paraphrasers designed explicitly to evade detectors — has exploded. Third, adversarial actors fine-tune open-weight models on specific domains (legal briefs, medical notes, SEO blogs) to mimic the idiosyncrasies of those niches.
"We see new evasion techniques almost daily," Spero noted. "Some are simple: insert random typos, vary sentence length aggressively, sprinkle in personal anecdotes. Others are sophisticated: prompt chains that force the model to reason in a specific style, then rewrite its own output three times." Pangram counters this by continuously retraining on fresh synthetic data generated by the latest models and humanizers, but Spero admitted it's a Red Queen's race — running as fast as possible just to stay in place. For small businesses relying on off-the-shelf detectors, this means a tool that worked last quarter may be blind to this quarter's AI output. Vendor transparency about update cadence and benchmark performance on recent data is not optional; it's a procurement requirement.
False Positives: The Hidden Cost for Small Business
Enterprise platforms can absorb some false-positive rate; they have appeal processes, human reviewers, and legal buffers. Small businesses rarely do. Spero highlighted a scenario: a local marketing agency uses a detector to vet freelance writers. A false positive flags a loyal contractor's original work as AI. The agency either loses the contractor or wastes hours investigating. Multiply that across client deliverables, and the detector becomes a liability.
"We optimize heavily for precision at the cost of recall," Spero said. "We'd rather miss some AI content than wrongly accuse a human. But every detector makes different trade-offs. A small business owner needs to ask: What is this tool's false-positive rate on *my* domain — legal contracts, real estate listings, code comments? Generic benchmarks on Wikipedia or news articles don't translate." He recommended that any team adopting detection run a blind evaluation on 100-200 samples of their own historical content — known human, known AI, known hybrid — before deploying. "If the vendor won't give you API access for a weekend test, walk away."
Beyond Detection: Provenance, Watermarking, and the Platform Layer
Spero is blunt about detection's long-term ceiling: "Text detection will never be 100% reliable. The information-theoretic limits are real. Short texts, highly constrained formats (tweets, commit messages, form fields), and heavily edited hybrids are fundamentally ambiguous." He sees the durable solution shifting toward provenance — cryptographic signatures, embedded watermarks, and platform-level attestation. "If Google Docs, Notion, and Microsoft Word all adopt C2PA-style content credentials, the detector becomes a fallback, not the primary gate."
For small businesses, this implies a two-track strategy. Short term: deploy a best-in-class detector with a human-in-the-loop review for borderline cases, and publish a clear policy so contractors and customers know the rules. Long term: pressure the platforms you depend on — your ATS, your review site, your CMS — to integrate provenance standards. "Don't build your trust stack on a detector alone," Spero warned. "Build it on verifiable workflows."
What This Means for Operators Right Now
The trust problem isn't abstract. It shows up in a fake review tanking a local restaurant's rating, a fabricated insurance claim raising premiums for everyone, a bot-written grant application diverting funds from a legitimate nonprofit. Detection is a necessary but insufficient tool. Spero's pragmatism — acknowledge the limits, measure on your own data, plan for provenance — is the most useful framework I've heard for operators who can't wait for perfect technology. The startups that survive this wave won't be the ones claiming 99% accuracy on a press release. They'll be the ones telling you exactly where their model fails, how often, and what you should do about it.
Source: Original Article