Why Inference Matters 12x More Than Training | YC Paper Club
At a YC Paper Club night, Dan Fu of Together AI, Jeffrey Morgan of Ollama, Joe Fioti of Luminal and Liangsheng Yin of SGLang explain why serving models, not training them, is where the cost and hard engineering lie.
The gist
-
Using DeepSeek's figures, the host estimates inference uses about 6x the compute of training per year, or 12x over a two-year model life.
-
Dan Fu says Together's training-free quantization recipe let MiniMax's model run in NVFP4 while matching original quality, checked with benchmarks and a Flappy Bird test.
-
Morgan says the quality gap between the worst and best providers of the same open model can exceed the gap between open and closed models.
-
Fioti's Luminal breaks PyTorch models into about 14 or 15 basic operations, then uses egglog to search options and times them on real hardware.
-
Yin says SGLang's radix tree reuses shared prefixes, and its tiered cache across GPU, host memory and storage cut time to first token by 56%.
"We use a lot of tools that PyTorch provides where they're really nice about tagging the high level ops, and then we kind of track that metadata as it flows through our compiler. We use a lot of the standard PyTorch profiler to go and work out bottlenecks and issues."
▶ Play at 1:25:31"Specifically, how we use the egglog engine, which powers our equality saturation."
▶ Play at 1:06:55"Then we live inside of the great inference engines of vLLM and SGLang. And so, the great solutions they already provide for speculative decoding, we're able to just go and directly piggyback on them."
▶ Play at 1:29:47What they recommend · 6 picks
"If you want to learn more, I think the two best lectures I've seen on this: Percy gives an amazing lecture, his lecture 10 on CS 336, LLMs from scratch."
"And then the Reiner Pope and Dwarkesh one I thought was just super, super cool and super good."
"Specifically, how we use the egglog engine, which powers our equality saturation."
Y, Riley, Greg and Alex agree on 20 things
See what they all use, in their own words. Paid ads are left out.
See what they all swear by Or pick your own peopleJoe Fioti
4 picks · guest
"Specifically, how we use the egglog engine, which powers our equality saturation."
"We use a lot of tools that PyTorch provides where they're really nice about tagging the high level ops, and then we kind of track that metadata as it flows through our compiler...."
"Then we live inside of the great inference engines of vLLM and SGLang. And so, the great solutions they already provide for speculative decoding, we're able to just go and direc..."
Y Combinator
2 picks · host
"If you want to learn more, I think the two best lectures I've seen on this: Percy gives an amazing lecture, his lecture 10 on CS 336, LLMs from scratch."
"And then the Reiner Pope and Dwarkesh one I thought was just super, super cool and super good."
Made by someone on this episode
Also mentioned
Related episodes
Episodes covering similar topics
More from Y Combinator