The problem
Agents that claim to do product management are judged on demos. There is no standard way to test whether an agent can synthesise messy research, prioritise under constraints, write an unambiguous spec or read an experiment correctly.
Why it matters
Product work is a large share of how software gets decided. If agents are going to help with it, teams need to know where they are reliable and where they make things up.
Research question
Can an agent do the actual work of a product manager, and where does it fail?
Approach
- 336 tasks across 14 categories, from research synthesis and prioritisation to experiment analysis, executive communication and regulated domains.
- Three difficulty levels, roughly junior, mid-level and senior product manager.
- Each task is a YAML definition with source documents, a rubric and ground truth.
- An LLM judge scores five dimensions from 0 to 20. An anti-hallucination rule caps the accuracy score when an answer fabricates.
Methodology
- Tasks and rubrics are versioned in the repository.
- Judging runs with more than one model family to reduce self-preference.
Findings so far
No results yet. The task set and harness are released, but no model or agent has been scored. The leaderboard holds only a placeholder entry.
Limitations
- LLM judges have known biases, including favouring answers from their own model family.
- How closely the rubrics agree with experienced product managers has not been measured.
Artifacts
- Repository · MIT