The Good, The Bad and the Eval
Product Managers struggle with defining clear success metrics for AI based features. Here's a story:
I once met Ron Huldai, who told me he wanted to spend 100$K on an AI based feature tat would benefit Tel-Aviv. I told him I can build him an amazing AI model that would predict each day, if there's going to be a devastating earthquake in the following day in tel-aviv. He asked me: "what will the accuracy of the model be?" and I said: "99.9%" . My model was very simple, giving "no" as an answer every day.
This story shows, for example, that "accuracy" is a metric that s not the right one to use in this situation.
In our talk I will explain how to measure how "good" are your AI Features, in order to understand how they improve over time, and create a shared language with customers and R&D.