
Lenny's Podcast
Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)
In one sentence
Gemini's head of product explains why models stay unpredictable, how to judge them with small evals and good vibes, and why PMs with taste matter again.
Lenny Rachitsky talks with Tulsee Doshi, head of product for Google's frontier models. They cover how models are shaped but never fully predicted, Google's full-stack strategy, safety through monitoring, the model's personality as its UX, what she looks for in PMs, how to start writing evals, the 60/20/20 rule for building ahead of the models, and how her team learns new tools without burning out.
Who's talking
- Lenny RachitskyHost · Host of Lenny's Podcast
- Tulsee DoshiGuest · Head of product, Google's frontier models (Gemini)
Speakers are inferred from the conversation.
Key points
- Model development has goals and evals, but the result always holds surprises; a model PM studies what comes out and builds on where it shines.
- Google's strategy is full stack, from hardware to products with over a billion users, with models across the cost-quality curve.
- The model has a UX of its own: how transparent it is, how it communicates and what it asks back; a behavior spec becomes rubrics, data and reward models.
- When building is cheap, taste and the ability to turn chaos into order are what make a PM valuable.
- Good evals start tiny, five to twenty prompts with yourself as the rater, and the judge, the serving stack and filters all affect the result.
- Build ahead of the models: most of the product works today, some almost works, and some waits for the next model.
Main claims
Google has 14 products with more than a billion users (she said she should check the number).
The six-tone Fitzpatrick scale was built for healthcare and skews light, so it cannot represent a global population.
In his survey of tech workers, more hours did not mean more burnout.
Chain-of-thought monitors that flag misalignment give her more confidence in deploying models.
Quotes
The model isn't predictable.
— Tulsee Doshi Taste is now really important.
— Tulsee Doshi Start with even just five prompts.
— Tulsee Doshi What you should be providing is the insight. What the model should be providing is the data.
— Tulsee Doshi
Takeaways
- Write a mini eval this week: five prompts you care about, rated by you, run on two models.
- Before doing a task twice, ask whether a model can do it, and share what works as a reusable skill.
- Retest your harness with each new model; old workarounds can hold a better model back.
- In AI-written docs, keep the data from the model and the insight from yourself.
- Gemini
- Product management
- Evals
- AI safety
- AI agents
- Burnout