I lead AI product work at Poe / Quora. This is a place for the questions I work through by building, testing, and talking them through.
My work has included APIs at Twilio and Foursquare, speech and applied ML at Zoom, and now AI products. The personal experiments make room to follow a question further.
Watch: Evals for AI Agents ↗In this session, three imperfect signals contribute to a reward function. Select a signal to see how the experiment represents it.
Used as a proxy for helpfulness in this demo. Positive sentiment is not the same thing as actually helping.
Hear the proxy choices · 1:01:13 ↗A conceptual guide to the talk, not a validated evaluation system. The inputs are normalized and weighted; no measured scores are shown here.