What They Don't Tell You about Calibrations and Ratings

A behind-the-scenes look at an opaque process that greatly impacts your career

Photo by Graphic Node on Unsplash

I recently had a great conversation with Shreyas Doshi, where we discussed topics like communication, building a learning mindset, and managing your career as a Product Manager. Inspired by our chat, I thought I would give you a behind-the-scenes look at how performance assessments, calibrations, and ratings work. 

I have been a manager for nearly two decades, and I previously helped design a large PM calibration system. There is a lot going on behind the scenes that is invisible to most of the people who get rated, and non-managers usually never see what happens in these closed rooms. 

Each of these systems is imperfect in its own way, and there is no single “optimal” review process. They are designed to accomplish different things at different times, evolving from the early days of a company to the growth phase to scaling. They are set up to help align the workforce toward making an impact and delivering against business goals. 

You have more power in this process than you think, so knowing how to optimize it for yourself is critical. In today’s post, I will give you the inside scoop on what goes into these ratings behind closed doors and show you what you can do to affect the outcome. 

No system can be perfect, because they are optimized for scale and implementability across an organization 

Calibrations and ratings are blunt instruments. They are used as tools to accomplish something. They are designed to help you grow, give you feedback, and gauge the distance between where you are now and the next level. They are used to lead a workforce, encode values, and deliver productivity for the company. They are also used to ensure ratings are consistent across different organizations and leaders. 

These goals are all embedded into any ratings system. Those who design them want to make them as fair as possible for as many people as possible, but these systems are imperfect in various ways—sometimes unpredictable ones. They are meant to accomplish something beyond individual outcomes, which is why they sometimes feel fraught and unsettling. 

The reality is that no performance process is ever 100% objective, because any set of objective criteria is ultimately being evaluated by human beings. This leaves lots of room for subjectivity to creep in, whether we like it or not. 

There was a period of two and half years when I built really impactful products at Meta, but never got more than a “meets all” rating. Was that fair? I’m not sure. I created a lot of value for the company, but I also constantly chose to work on new things—and many new things fail or take longer to succeed than expected. I knew the risks of the choices I made, but I also had a chance to build transformative products, so I have no regrets. That said, the ratings system did not take that type of path into account. I accepted it as the cost of the way I worked. 

Humans are fallible, which means that any review system will also be fallible. This can create challenges, but the important thing to remember is that you are more than your rating or the pace at which you are promoted. 

There is unintentional bias that’s hard to eradicate 

There has been much written about the biases surrounding gender and race in performance evaluations. I won’t retread that ground in this article, because those biases are well-known and documented. Instead, I want to discuss a few lesser-known biases that can creep up in ratings and reviews, ones that can be difficult to overcome. 

• Bias toward (or against) your manager: The dynamics of an organization can creep into even the most objectively designed review processes. Someone on my team once wanted to give the highest rating to a PM who deserved it. He fought for it but was shot down. I signaled in the final calibration that I wanted to change the rating, and I was able to push it through. My colleague was seen as partisan for fighting for one of his few direct reports, but I ran the larger organization. Thus, my asking the same group for a single person’s rating to be raised was seen as more fair. 

• Affinity bias: People tend to support others that are most similar to them. Human nature is susceptible to fundamental attribution error (e.g. we blame others people for missing goals, but assume when we miss, it is due to outside factors) as well as the desire to support those who remind us of ourselves. I have seen this creep into calibrations. For example, people who work on Ads defend other monetization teams because they know how hard it is to do the work, but they trivialize the work they don’t understand. And those who value design and user experience will dismiss the challenge of back-end work. 

• Speaking order: When it comes to evaluations, there is also the potential for a huge bias involving who speaks first. If the first person to speak on behalf of someone says something positive, the result may be completely different than if they’d opened with something negative. The way the discussion starts can set the tone for the whole conversation. Similarly, one small example, comment, or issue can color or sidetrack the whole discussion. The next thing you know, your manager is suddenly on the defensive just to keep your rating on track.