Home » Metrics » What should go into player ratings?

What should go into player ratings?

Soccer analysts – and sports analysts in general – put a lot of effort into evaluating players. Whether it’s for recruiting, team selection, scouting the opposition, planning tactics, or predicting results, measuring players’ ability and performance is crucial. Yet there is surprising little written about the content of player ratings.

That’s too bad, since a club probably won’t want to use an analyst’s ratings if it can’t understand the underlying components, or indeed why the analyst chose them. I like to begin this discussion is by building from the ground up in the most general way.

Let’s start by talking about what goes into a player’s performance. Here’s a schematic that helps me to think about what goes into any kind of personal achievement:

perfomanceschematic

What’s going on here? First, internal inputs are combining with external inputs to create outputs. Then, modifiers are acting on those outputs to transform them into outcomes. It’s a bit like a factory; you put in raw materials (inputs), you get products (outputs), and then, if you can sell them at a good price under the current conditions in the market (modifiers), you earn a profit (outcome).

In terms of soccer players, it might go like this:

Internal inputs are the factors specific to the player. Some of these, like height, are innate. Others, like fitness levels and work rate, can be controlled – at least in part – by the player, coaches, and staff.

External inputs are the factors outside the player that will still affect his output. Among these might be weather, the strength of the opposing team, and aspects of his personal life.

Outputs include the actions the player takes on the field: passes, shots, tackles, etc. Depending on how performance is measured, outputs might also encompass actions the player takes off the field: charity work, binge drinking, etc.

Outcomes are the endpoint – the things a club wants at the end of the day. For many clubs, the main outcome that matters is the final position in the table. Some clubs may reduce this outcome to “qualify for the Champions League” or add another outcome like “play good football“. For clubs that have to worry about money – and Financial Fair Play is trying to ensure that most big European teams do – then another important outcome might be profitability. Either way, as I’ve written before, the outcome used in most player ratings should be a contribution to the club’s desired outcomes at the team level.

Modifiers determine how outputs turn into outcomes. A player who joins a club when it’s already been relegated may offer outstanding output without improving any outcomes, since the latter have already been determined.

Though it might seem surprising at first, any of these five types of factors might be useful as part of a player rating. The right way to use them depends in large part on your objectives. The following case study will show what I mean.

Case study: Recruiting

Let’s say a club is trying to find a new striker. For each player being considered, the team wants to predict how he will affect its ability to win and its financial bottom line.

We can start by asking how the new striker might help the club to climb the league table (outcome) by scoring more goals (output). Goals scored are strongly related to league positions, though issues of timing (modifiers) make it a less-than-perfect relationship. Still, these issues will affect any striker signed by the club.

To predict how many goals the striker will score, we might start by asking how many shots he’s likely to take, and how accurate those shots are likely to be. To take a lot of shots, the striker will need to stay healthy enough to play often, find open spaces where he can receive passes, and out-run and out-jump opposing defenders. The accuracy of his shots will depend on his proximity and angle to the goal, as well as his ability to place the ball where the keeper can’t reach it (or where he ain’t, as Willie Keeler might have said).

Some of these factors may be innate, and others, though not innate, may still be under the player’s control (internal inputs). Other factors may be determined by the club or the rest of the teams in the league (external inputs). Innate inputs will change little from season to season, except when linked to age. All the other inputs, and any measure of output that relies on them, may vary even from game to game. “Did the player keep himself fit?” “Did the coach play him in his best position?” A club will only get the most out of a player if it can ensure the answer to these questions is “Yes.”

Now let’s consider how well the striker will help the club to achieve profitability (outcome). If buying a particular striker – say one who always avoided injury and suspension – would allow the club to sell another player (modifier), then that may count in the striker’s favor. The same might be true if a large group of the club’s supporters have the same nationality as the new striker (modifier); more of them might show up for matches and buy merchandise with his name on it.

So far, we’ve identified the contributors to performance for potential new strikers. Now, how do we rate the strikers numerically? The answer depends in large part on how we assess risk.

Let’s start from outcomes again and work our way backwards. As I said, goals will always translate into points and positions imperfectly. We have to measure the resulting risk to gauge the likely effect of a new signing on outcomes, but it’s not necessary for comparisons between the candidates. For the latter, we can focus on outputs.

Here the output is goals scored, so we want to figure out the importance of each input to scoring. This is what expected goals models do. But we also need to know how much these inputs might vary from season to season.

By definition, innate inputs have no variation or risk. A player’s height usually won’t change during his career. But several of the other internal and external inputs do carry risks that will be specific to a player. For each one, the club will want to ask, “What is the potential variation in this input, and how much can we control it if we sign this player?” The answers to these questions will affect how the ideal rating is constructed.

For example, if a club believes it can control a player’s fitness perfectly, then it will only care what the maximum value for that input has been over the player’s career. The club will help the player to attain that value, and any variation will be reduced to zero. For instance, a striker may have played between 1,000 and 3,000 minutes in the past four seasons; the club might be confident of ensuring he plays 3,000 minutes in all the seasons to come. But if the club has no control over a player’s fitness, then it could be exposed to all the variation the player has shown so far; his output will be anybody’s guess.

By breaking down each player’s expected output into its component inputs, the club can begin to construct its ideal ratings and make meaningful comparisons. One player might have a high level of innate inputs, but his other inputs might be extremely variable. Another might have a lower level of innate inputs, but his other inputs might be fairly constant.

NYA has developed several kinds of ratings to capture different aspects of performance. The “Pep” rating, so named because of a quote from Pep Guardiola’s recent biography, only looks at players’ ability to maintain possession. The adjusted contribution to expected goals, explained here, tries to control for the effects of a player’s teammates. The total contribution can be compared to the adjusted contribution to see if players are fulfilling their potential. And the Shapley value, in its simplest form, can measure how pivotal players are in achieving final results – goals, points, positions, or whatever outcome is selected. Here is how they stack up:

ratingstable

As we move away from innate inputs, two things happen: we introduce more variation in a given player’s ratings from season to season, and we strengthen the relationship between his ratings and outcomes for his team. In other words, we add risk related to inputs and outputs while subtracting risk related to modifiers and outcomes. It’s a tradeoff.

For a club involved in recruiting, its prioritization of these ratings will depend on its ability to control different risks and the role envisioned for its new player. Of course, numerical ratings alone aren’t sufficient to make a final decision, especially since so many risks stem from aspects of a player’s personality and playing style that are hard to quantify. But as I often say, these ratings can be a useful first cut at the beginning of the recruiting process – and also a worthwhile check at the end.

(Note: The schematic above borrows from one I created during a consulting project for the World Bank some years ago.)