Five Measures of Product Manager Performance

Joy posted a really good article last week at Seilevel’s requirements defined blog, Measuring product manager performance on internal system products. Her post is a followup to an extensive and heated debate that happened last fall on the Austin PMM forum. It’s a great forum to subscribe to - a lot of experienced people with strong opinions and steamer trunks full of anecdotal data - and they don’t all agree. You get to see the coin from all three sides* with this group - it’s awesome.

She puts together a summary of five reasonable, measurable metrics for Product Managers

1. Track scope creep beyond a baseline of the requirements by counting new requirements added (and outside the original project scope).
2. Track errors of omission by counting new requirements added after the baseline (but within the original project scope).
3. Track errors of ambiguity/incorrectness by counting changes to requirements after they have been baselined.
4. Measure user adoption or satisfaction, presumably relative to an initial goal.
5. Measure the “missed ROI“ of the delivered project, relative to the initial plan.

There is good detail behind the sources of the measurements and methods of measurement. Joy also recognizes that they are imperfect metrics. I love the discussion that she has started, and hope it extends beyond this post as well. If you have a blog, post another followup to this one, and link back to both of our posts.

People perform to the metric
Measuring performance quantitatively is extremely hard. It can be very easy to measure, but it is very hard to know what to measure. The only thing worse than not having any clear objectives for an employee is having the wrong metrics. At the end of a review period (or during a project debrief), we want to be able to tell someone more than “exceeds/meets/fails-to-meet expectations.”

The problem then becomes defining a metric for measuring performance. Even worse than not having a metric is reaching the end of a review period and finding that someone excelled against the metrics, but failed against the objectives of the company. This is a classical organizational design problem, not just a “how do I measure product managers” problem. We will explore the larger problem in general, but use the PdM in specific for examples.

We contend that people’s performance is at best influenced by, and at worst strictly designed to meet metrics. Our philosophy and approach below is built upon that premise.

Looking at the bigger picture
On The West Wing, Leo is coached before a press briefing to respond with “I don’t accept the premise of your question.” when asked a lose-lose question. We should take a step back and explore the premise, instead of blindly critiquing the actionable ideas Joy presents.

- - -

Thinking outside the box
Since we talk about requirements definition a lot here, let’s approach the definition of a performance measurement system as a requirements elicitation exercise.

Why do we measure someone’s performance?
As an analog to the market requirements, we identify the problem/opportunity of increasing the success of the company. We believe that people impact the success of our company through their performance.

• As a company, we have a goal. This could be philanthropic, profit motivated, or any other company vision. Assume for the sake of this exercise that the company is profit motivated. Our company also has a set of implicit or explicit values - Google includes “do no evil” in their list of values. Think of these as constraints on what we do and how we do it.
• People’s performance in their role either advances or hinders our company in reaching its goal. We therefore want to put systems in place that maximize people’s aligned performance with our goal. We will also dedicate internal resources (a cost) to create and maintain this system.
• We want to maximize the long term, positive, impact that a person’s performance has on our company goals.

What are the elements of our company’s goals that we want to impact?
This is the analog to our product requirements - what are those things that we want to achieve as a company, which we want to achieve through individual performance? For our exercise, let’s assume that the company is profit motivated, and believes that profits are best achieved through long term successful product management relationships with our clients.

Based on that, and focusing on product managers, what are the traits or actions we want to encourage to support our goals? Just as converting an MRD to a PRD is an ideation process, determining those parts of our company objectives that are best addressed through individual performance is an ideation process.

• We want our projects to succeed. We believe that successful projects will occasionally lead to repeat business, and failed projects will cause us to lose long term clients by weakening our relationships. We also believe this will differentiate us from our competitors.
• We want to improve project ROI for our customers. We believe that providing our customers with more cost-effective projects, we will gain additional business, and also make existing business more profitable for us.
• We want relationships with decision makers at our clients. A bad reputation will stop the best sales cycle on a dime. A good reputation can open doors, uncover opportunities, and serve as a tie-breaker in competitive bids. We believe having a good reputation will help us close more deals, and close them more profitably.

What are the measurements we should use to assess performance?

This is the analog of software design. We are crafting a system of measurement, designed to achieve the subset of goals that we believe are best accomplished through individual performance. As such, we should review some design considerations.

• We need to be aware of the danger of false precision. Here’s an example - in a 3 month period, one product manager wrote 100 requirements that did not change after the baseline. Another product manager wrote 50. Who is the better product manager? What about developers - is the developer who writes more lines of code the better developer? What about the one who closes the most issues? These metrics do not give us useful information without context. They are easy to measure, but hard to show the relevance of the measurement to the stated objectives.
• We need to avoid destructive (but instinctive) human behavior. Another example - we penalize product managers for changing requirements after the baseline - specifically based on %changed data. Our developers find the spec to be ambiguous, so they ask questions to get clarification. The product manager answers the questions instead of updating the spec. Our product manager ends up wasting a lot of time explaining, but avoids getting dinged on an easily measurable metric.
• We need to avoid performance metrics that can not be controlled by the employee. It can be demotivating to an employee to know that their efforts to affect their performance review are futile. In the worst case, the employee will give up. In the best case, the employee will do what is right in spite of the metric - and we will fail to equitably reward the employee. There is an entire industry that defines compensation systems using quantified metrics. The stock market uses the same approach.
• We want to give feedback regularly. We believe that presenting an employee with feedback and suggestions for improvement will result both in more job satisfaction and in ever-increasing impact from the employee. This is not best done during a performance review (although it should happen then). Feedback should be given repeatedly during the performance period - to give the employee an opportunity to course-correct along the way. This would be agile management. Waiting until the review to surprise someone with good or bad news is the waterfall management method.

OK, sounds impossible - what do we do?

We believe that a fomulaic appraisal system should not be used.

It is inefficient because the metrics can mislead us about who is doing a great job and who isn’t. We also believe that metrics will stimulate sub-optimal behavior in employees, ultimately subverting our company objectives to some degree.
The right way to assess performance is to qualitatively assess the results of the work done by the employee. We would talk to our contacts at the client to assess the impact the employee has had on the relationship. We would have the employee submit periodic status reports with both anecdotes and progress updates. We would review the success of the project, and identify the context - were the results in spite of or because of the efforts of the employee? We would ask the consumers of his artifacts (requirements documents) for their opinions of the quality of the documents - and we would develop an understanding of their capabilities to judge. Through the course of the review period, we would interview the employee and ask questions about the content of the status report. By understanding what’s working and what isn’t, what’s easy and what’s hard, and what’s on time and what’s late; we can apply our own expert opinion about the employee’s performance - and share it with the employee right then. This also then gives us the ability to assess the employee’s improvement relative to the feedback he has received.

And another thing

Another way to assess performance is to identify goals at the beginning of a review period, and track progress towards those goals. These high level abstractions are easy to align with company objectives, and can be easy to measure (did we sell an extra million dollars in services to your account this quarter?). This approach can be very effective when evaluating adaptive high-performing employees with the latitude to operate in a loose organizational structure to “do whatever it takes” to get the job done. This can be done in conjunction with the other qualitative assessments outlined above.

Our suggestion - align bonuses with goal-based measurement of final results (not interim steps), and manage salary via qualitative assessments. Note that this approach is only effective when reviewing empowered, typically professional, employees.
*heads, tails, and the edge make up the three sides of a coin

This article was published on Monday, January 30th, 2006 at 11:10 pm and is filed under Consulting, Lists, People management, Requirements.
You can leave a comment on this post.
Trackback URL