Hamel Husain’s insights for developing and releasing a large language model based product1:
- Build a tool that allows you to view and annotate conversations with the LLM for real users. Getting close to the data in this way will allow you to see real trends and blockers. Focus on improve the failures and measure the improvement. Prefer these metrics over squishier metrics.
- Get domain experts involved in prompt writing, but tools in their hands to do it.
- When data is not available, use LLMs to generate synthetic.
- Focus on binary outcomes for LLM success (Pass/Fail) rather than a gradients or scales. This gives you clear outcomes
- You can scale by keeping humans closer in the loop and gradually leaning on AI to decide pass/fail. Finally, you can move to a statistical sample approach to maintain on going calibration
- AI Projects and Project
Management do not look
like typical product management where features are discrete and
knowable. AI projects are much more experimental and iterative and
so framing this up for stakeholders is important.
- Update stakeholders on a regular cadence with how experiments are progressing.
1. Husain, H. A Field Guide to Rapidly Improving AI Products – Hamel’s Blog. https://hamel.dev/blog/posts/field-guide/ (2025).