Volo Kuleshov

0 things Volo uses or recommends, from 0 episodes.

What Volo warns against

Things Volo says to avoid or be careful with. Other experts don't always agree.

"Right now, perhaps the most widely used and representative benchmark for running evals is tau-bench. tau-bench has a few problems. First of all, tau-bench doesn't really capture all the range of complexity that you see in real-world data. It probably has an order of magnitude fewer specifications than you would have in a real-world setting where a business would be doing real-world customer support. And it's extremely benchmark-maxed — most models are achieving scores in the 90s, but if you run these same models on real-world production logs, you get much lower scores."

Going In Deep On Data | YC Paper Club

All 0 of Volo's picks

Taxbench Warns against "Right now, perhaps the most widely used and representative benchmark for running evals is tau-bench. tau-bench has a few problems. First of all, tau-bench doesn't really capture all the range of complexity that you see in real-world data. It probably has an order of magnitude fewer specifications than you would have in a real-world setting where a business would be doing real-world customer support. And it's extremely benchmark-maxed — most models are achieving scores in the 90s, but if you run these same models on real-world production logs, you get much lower scores." No one recommends it
Nothing matches.

Also mentioned in passing (1)