Five Measures of Product Manager Performance

Joy posted a really good article last week at Seilevel’s requirements defined blog, Measuring product manager performance on internal system products. Her post is a followup to an extensive and heated debate that happened last fall on the Austin PMM forum. It’s a great forum to subscribe to – a lot of experienced people with strong opinions and steamer trunks full of anecdotal data – and they don’t all agree. You get to see the coin from all three sides* with this group – it’s awesome.

Five Reasonable Metrics for Product Managers

She puts together a summary of five reasonable, measurable metrics for Product Managers

1. Track scope creep beyond a baseline of the requirements by counting new requirements added (and outside the original project scope).
2. Track errors of omission by counting new requirements added after the baseline (but within the original project scope).
3. Track errors of ambiguity/incorrectness by counting changes to requirements after they have been baselined.
4. Measure user adoption or satisfaction, presumably relative to an initial goal.
5. Measure the “missed ROI“ of the delivered project, relative to the initial plan.

There is good detail behind the sources of the measurements and methods of measurement. Joy also recognizes that they are imperfect metrics. I love the discussion that she has started, and hope it extends beyond this post as well. If you have a blog, post another followup to this one, and link back to both of our posts.

People perform to the metric

Measuring performance quantitatively is extremely hard. It can be very easy to make measurements, but it is very hard to know what to measure. The only thing worse than not having any clear objectives for an employee is having the wrong metrics. At the end of a review period (or during a project debrief), you want to be able to tell someone more than “exceeds/meets/fails-to-meet expectations.”

The problem then becomes defining a metric for measuring performance. Even worse than not having a metric is reaching the end of a review period and finding that someone excelled against the metrics, but failed against the objectives of the company. This is a classical organizational design problem, not just a “how do I measure product managers” problem. We will explore the larger problem in general, but use the PdM in specific for examples.

I contend that people’s performance is at best influenced by, and at worst strictly designed for meeting metrics. The philosophy and approach below is built upon that premise.

Looking at the bigger picture

On The West Wing, Leo is coached before a press briefing to respond with “I don’t accept the premise of your question.” when asked a lose-lose question. We should take a step back and explore the premise, instead of blindly critiquing the actionable ideas Joy presents.

Thinking outside the box

Since we talk about requirements definition a lot here, let’s approach the definition of a performance measurement system as a requirements elicitation exercise.

Why do you measure someone’s performance?

First identify the problem/opportunity of increasing the success of the company. It is reasonable to believe that people impact the success of your company through their performance.

• As a company, you have a goal. This could be philanthropic, profit motivated, or be based on any other company vision. Assume for the sake of this exercise that your company is profit motivated. Your company also has a set of implicit and explicit values – Google includes “do no evil” in their list of values. Think of these as constraints on what your employees do and how they do it.
• People’s performance in their role either advances or hinders your company in reaching its goal. You therefore want to put systems in place that maximize people’s aligned performance with your company’s goals. You will also dedicate internal resources (a cost) to create and maintain this system.
• You want to maximize the long term, positive impact that a person’s performance has on your company goals.

What are the elements of your company’s goals that you want to impact?

What are those things that you want to achieve as a company, which you want to achieve through individual performance? For this exercise, let’s assume that you believe that profits are best achieved through long term successful product management relationships with your clients.

Based on that, and focusing on product managers, what are the traits or actions you want to encourage to support your goals? Determining those aspects of your company objectives that are best addressed through individual performance is an ideation process.

• You want your projects to succeed. Successful projects will occasionally lead to repeat business, and failed projects will cause you to lose long term clients by weakening your relationships. Successful projects will differentiate you from your competitors.
• You want to improve project ROI for your customers. By providing your customers with more cost-effective projects, you will gain additional business, and also make existing business more profitable.
• You want relationships with decision makers. A bad reputation will stop the best sales cycle on a dime. A good reputation can open doors, uncover opportunities, and serve as a tie-breaker in competitive bids. Having a good reputation will help you close more deals, and close them more profitably.

What are the measurements you should use to assess performance?

You are crafting a system of measurement, designed to achieve the subset of goals that you believe are best accomplished through individual performance. As such, you should review some design considerations.

• You need to be aware of the danger of false precision. Here’s an example – in a 3 month period, one product manager wrote 100 requirements that did not change after the baseline. Another product manager wrote 50. Who is the better product manager? What about developers – is the developer who writes more lines of code the better developer? What about the one who closes the most issues? These metrics do not give you useful information without context. They are easy to measure, but hard to show the relevance of the measurement to the stated objectives. In fact, the best thing that can be said about them is that they are easy to measure.
• You need to avoid destructive (but instinctive) human behavior. Another example – you penalize product managers for changing requirements after the baseline – specifically based on %changed data. Your developers find the spec to be ambiguous, so they ask questions to get clarification. The product manager answers the questions instead of updating the spec. Your product manager ends up wasting a lot of time explaining, but avoids getting dinged on an easily measurable metric. By aligning behavior with the metric, this product manager becomes less efficient (by repeating the same explanation over and over), while simultaneously improving his “score.”
• You need to avoid performance metrics that can not be controlled by the employee. It can be demotivating to employees to know that their efforts to affect their performance reviews are futile. In the worst case, the employee will give up. In the best case, the employee will do what is right in spite of the metric – and you will fail to equitably reward the employee. There is an entire industry that defines compensation systems using quantified metrics. The stock market uses the same approach.
• You want to give feedback regularly. Presenting an employee with feedback and suggestions for improvement will result both in more job satisfaction and in ever-increasing impact from the employee. This is not best done during a performance review (although it should happen then). Feedback should be given repeatedly during the performance period – to give the employee an opportunity to course-correct along the way. This would be agile performance-management. Waiting until the performance review to surprise someone with good or bad news is the waterfall performance management method.

OK, sounds impossible – what do I do?

A fomulaic appraisal system should not be used.

It is inefficient because the metrics can mislead you about who is doing a great job and who isn’t. Metrics will stimulate sub-optimal behavior in employees, ultimately subverting your company objectives to some degree.

The right way to assess performance is to qualitatively assess the results of the work done by the employee. Talk to your contacts at the client to assess the impact the employee has had on the relationship. Have the employee submit periodic status reports with both anecdotes and progress updates. Review the success of the project, and identify the context – were the results in spite of or because of the efforts of the employee? Ask the consumers of his artifacts (requirements documents) for their opinions on the quality of the documents – and develop an understanding of their capabilities to judge. Through the course of the review period, interview the employee and ask questions about the content of the status report. By understanding what’s working and what isn’t, what’s easy and what’s hard, and what’s on time and what’s late; you can apply your own expert opinion about an employee’s performance – and share it with the employee right then. This also then gives you the ability to assess the employee’s improvement relative to the feedback he has received.

And another thing

Another way to assess performance is to identify goals at the beginning of a review period, and track progress towards those goals. These high level abstractions are easy to align with company objectives, and can be easy to measure (did you sell an extra million dollars in services to your account this quarter?). This approach can be very effective when evaluating adaptive high-performing employees with the latitude to operate in a loose organizational structure to “do whatever it takes” to get the job done. This can be done in conjunction with the other qualitative assessments outlined above.

Align bonuses with goal-based measurement of final results (not interim steps), and manage salary via qualitative assessments. Note that this approach is only effective when reviewing empowered, typically professional, employees.

[Update: 2009.08.31: Edited the above article for style, and struck through the final piece of advice. Watch Dan Pink’s fantastic TED talk on the science of motivation. Based on his research, you should not reward product managers for performance. Honestly, I don’t know what the right answer is – I’m conflicted, as Dan’s ideas (and facts) resonate with me, but I also can’t shake my belief in a reward system.]
*heads, tails, and the edge make up the three sides of a coin