Inside the Race to Measure Frontier Intelligence
a16z and Rayan Krishnan discuss why independent AI evaluation has become critical infrastructure, covering how enterprises pick models, measure ROI, and verify safety claims.
The gist
-
Self-reported benchmarks are unreliable: Meta's Llama 4 scored well publicly but underperformed on independent private tests, so third-party evaluation matters.
-
One Fortune 10 company spent 10x more on tokens than employee salaries in a single month, making ROI measurement urgent.
-
Krishnan recommends building evaluations against your own work output, since cheaper models often beat expensive ones for specific tasks.
-
Governments lack the tools to test risky AI behaviors themselves, so independent evaluators should verify compliance without slowing release cycles.
What they recommend · 1 pick
Devin
AI tools
"The Cognition Devin tool is actually very token efficient, and so that's a place we've chosen to adopt more."
Also mentioned
7:08
Gartner
· "When we're talking about enterprise software, there's Gartner rating on like 70 different metrics. You can..."
16:56
Claude Code
· "The way that they've adopted Claude Code has been with a roughly $100 a day budget for their engineers. And..."
◎
Related episodes
Episodes covering similar topics
Anthropic's $30T Assumption & OpenAI Confirms IPO | Why Customer Service & Robotics are Overinflated
enterprise ai adoption
Enterprises Fear Frontier Models | Sam Altman Offers Trump 5% of OpenAI | DeepSeek Builds Own Chips
enterprise ai adoption
Nikesh Arora on The Future of Token Costs | Memory Becoming the Moat & Why Enterprise AI Isn't Ready
enterprise ai adoption
More from a16z