← All creators
V
Volo Kuleshov
0 recommended products
Everything Volo personally uses, recommends, or has created — plus things they don't recommend — sourced from their own show and appearances on other podcasts.
Warned against
Taxbench
negative
"Right now, perhaps the most widely used and representative benchmark for running evals is tau-bench. tau-bench has a few problems. First of all, tau-bench doesn't really capture all the range of complexity that you see in real-world data. It probably has an order of magnitude fewer specifications than you would have in a real-world setting where a business would be doing real-world customer support. And it's extremely benchmark-maxed — most models are achieving scores in the 90s, but if you run these same models on real-world production logs, you get much lower scores."
Recent episodes
All →Showing 2 of 2 products
Clear filtersMentioned in 1 episode
"Here for example we have a 120 billion parameter model running on Cerebras, and a diffusion Mercury model can achieve both higher quality and run faster. Cerebras are specialized chips designed and built to make autoregressive models run really fast. If you have diffusion, you can achieve similar speeds on GPUs by having more intelligent software."
Mentioned in 1 episode
"Right now, perhaps the most widely used and representative benchmark for running evals is tau-bench. tau-bench has a few problems. First of all, tau-bench doesn't really capture all the range of complexity that you see in real-world data. It probably has an order of magnitude fewer specifications than you would have in a real-world setting where a business would be doing real-world customer support. And it's extremely benchmark-maxed — most models are achieving scores in the 90s, but if you run these same models on real-world production logs, you get much lower scores."
0 recommended · 0 sponsored · 1 warned against · 0 episodes