Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)
Tulsee Doshi, head of product for Google's frontier models including Gemini, tells Lenny Rachitsky why PMs matter more now that building is easy, how to write evals, and how far ahead of models to build.
The gist
-
Doshi says when anyone can build, taste and turning chaos into order matter, so she hires PMs who use models daily and can build evals.
-
For a first eval, she suggests starting with five prompts and growing to about 20 that show clear differences between model A and model B.
-
Her 60/20/20 rule: build so 60% works now, 20% works well enough, and 20% is where you want models to reach.
-
In responsible AI her team replaced the six-tone Fitzpatrick scale with Professor Monk's 10-tone scale, then used it to improve Pixel camera and image generation.
-
Doshi says burnout comes from churn, chaos, and grunt work, and she wants AI to handle data in docs while people supply the insight.
"For me a big thing is start with even just five prompts, right? You don't have to start large... even just a mini evaluation where you get to like 20 prompts that can really stress test a set of capabilities where you can see signal between your model and another model. I thin..."
▶ Play at 23:32"He had a scale of 10 skin tones that he had built through a number of user interviews and research that he had been doing over years. And we pulled that set of skin tones into our evaluations and actually redid a number of the ways that we evaluated our products and our models..."
▶ Play at 27:57What they recommend · 10 picks
AI tools
4 picks · Tulsee Doshi and Lenny Rachitsky
"For me a big thing is start with even just five prompts, right? You don't have to start large... even just a mini evaluation where you get to like 20 prompts that can really str..."
"He had a scale of 10 skin tones that he had built through a number of user interviews and research that he had been doing over years. And we pulled that set of skin tones into o..."
"Wispr Flow. I don't know if you're using it, but I have wrist problems, so I have trouble typing for long stretches. And so Wispr Flow has been game-changing for me, actually."
Books
3 picks · Tulsee Doshi ·Tulsee's books →
"Another that's maybe more standard in PM circles is Creativity, Inc., which I love. It's the Pixar story. But I think it actually speaks a lot to how do you create good culture...."
"I also just have been really into, you know, the ACOTAR series recently... for anyone who's just looking for something fun to read, you should read Throne of Glass or A Court of..."
"That's romantic fantasy. So for anyone who's just looking for something fun to read, you should read Throne of Glass or A Court of Thorns and Roses."
Other picks
3 picks · Tulsee Doshi and Lenny Rachitsky
"But I did really enjoy Severance on Apple TV. Incredibly well done, both stylistically and content-wise. So that's probably top of the list."
"If you are a Pixel user on Gboard, try using our Rambler feature. It's awesome."
"Google products definitely make my life better. It's just like amazing how much of your life is within a Google ecosystem. Like not even the AI stuff, but just Google Calendar, ..."
Tulsee Doshi
8 picks · guest
"For me a big thing is start with even just five prompts, right? You don't have to start large... even just a mini evaluation where you get to like 20 prompts that can really str..."
"He had a scale of 10 skin tones that he had built through a number of user interviews and research that he had been doing over years. And we pulled that set of skin tones into o..."
"Another that's maybe more standard in PM circles is Creativity, Inc., which I love. It's the Pixar story. But I think it actually speaks a lot to how do you create good culture...."
Lenny Rachitsky
2 picks
"No, I love Wispr Flow. Use it all the time. Press function, just start talking, and it just magically transcribes everything."
"Google products definitely make my life better. It's just like amazing how much of your life is within a Google ecosystem. Like not even the AI stuff, but just Google Calendar, ..."
Also mentioned
Ads in this episode
WorkOS, DX. They're paid, so we keep them out of the picks above.
See what Lenny's Podcast recommends without being paid →Related episodes
Episodes covering similar topics
More from Lenny's Podcast