Success in managing soccer teams isn’t all about merit – just ask Roberto di Matteo, who took control of Chelsea for just a few months and came away with a European Cup winner’s medal. There are managers whose teams can’t catch a break, and there are managers whose teams always get the goals just when they need them. Alan Pardew, who is apparently leaving Newcastle after four years for Crystal Palace, belongs definitively to the latter category. Yet this doesn’t necessarily mean he was lucky.

One of the big topics in soccer analytics is distinguishing luck (or other unknown factors) from skill among players. The same question is just as important for managers. I’m a Newcastle supporter, and I’ve followed Alan Pardew’s time at St. James’s Park with equal measures of bemused delight and bitter dismay. Upon his departure, I thought I’d try to answer the old luck-versus-skill question for him, using some simple metrics.

To start, I looked at all the games played in the Premier League between 9 December 2010, when Pardew took the reins at Newcastle, and the last match day of 2014, which was also his last in charge of the team. I added up all the points won by the 30 teams that played in the league during this time, and I also added up their goal difference. Then I divided both by the number of games each team had played. Finally, I regressed points per game on goal difference per game and then ranked the teams by the residuals:

Tottenham Hotspur took 0.19 more points per game than goal difference would have predicted, but none of their managers lasted more than a year or two; maybe their chairman, Daniel Levy, surmised they were just lucky (or maybe he had too little patience for the occasional blowout loss). The next teams that flattered most to deceive were Birmingham City and Cardiff City, both of which were relegated while Pardew held his job. Even if they overachieved, they didn’t overachieve enough.

In fourth place was Newcastle United, with 0.14 points per game more than predicted over 155 league matches. Pardew’s men definitely scored when it counted – indeed, perhaps only when it counted. But how should we interpret this result in terms of Pardew’s abilities?

I have two hypotheses. The first is that Pardew was lucky, and his players just happened to score goals at the right times. Put another way, they won by much narrower margins than they lost. You can’t choose exactly when to score, so a manager can’t take credit when goals arrive exactly when his team needs them. (You shouldn’t be choosing when to concede, either, right Austin Ejide?) Unlike Levy, Newcastle’s chairman Mike Ashley just never caught on.

In my second hypothesis, Pardew got the most out of his men that anyone could have hoped for, helping them to beat expectations time and again. His team talks fired up players to turn around losing causes. A tactical mastermind, he made substitutions at exactly the right times and gave his teams the best chances to take home points.

A while back, I might have been pretty sure that the first hypothesis was correct. But lately, two things have started to change my mind. First of all, Pardew does seem to understand that subbing in a quicksilver striker fairly early in the second half, just as defenders are starting to slow down, can change a game. This season, he’s put on the mercurial Papiss Cissé for a half or less on eight occasions – yet only once for fewer than 20 minutes – and used him longer only five times.

Second, Manchester United attained 0.13 points per game more than expected while their legendary coach Sir Alex Ferguson overlapped with Pardew, and by popular acclaim Sir Alex is a genius. The Scot was well-known in his last few seasons for grabbing points in the dying minutes of matches. He might have called those minutes “squeaky bum time” for the way anxious fans fidgeted in their seats; others called them “Fergie time“, especially when the referee added on quite a few, seemingly at Ferguson’s request.

Because those clutch goals came for Ferguson’s Red Devils, few commentators were bold enough to call them lucky. So should Pardew be afforded the same respect? It’s always hard to gain the same aura when you haven’t won a single trophy as a manager, and even those who overcome that handicap – such as David Moyes, Ferguson’s ill-fated successor – may find the aura can quickly disappear.

My own calculations suggest that Pardew’s players performed slightly better than their underlying levels of ability might have predicted, but not by as great a margin as the overwhelmingly tactical squads at West Ham or Stoke City. Of course, these measures of player performance evaluate players individually; they have nothing to do with team selection. On that count, the evidence is more mixed.

For a simple measure of how well managers used their players, I looked at the correlations between playing time and NYA’s measure of innate ability for Pardew’s two complete seasons, 2011-12 and 2012-13. I limited the analysis to teams that were in the top flight for both seasons and players who had at least 1,000 minutes on the field – that is, those who offered enough data for a reasonable evaluation of their talent:

Injuries and suspensions may have gotten in the way, but most of the top teams had a correlation of at least 0.30 between innate ability and playing time. (Yes, it’s time to ask questions of André Villas-Boas again.) Pardew only managed 0.23, a hint that he didn’t always go with his best players. I always had misgivings about his use of the talent on hand, and this result confirms them somewhat.

Based on his time at Newcastle, Pardew is by no means a bad manager, but he’s probably not an outstanding one, either. If he’s the luckiest manager in the league, then it’s by virtue of netting two lucrative contracts in a space of five years without being sacked, despite his ordinariness. Luck on the field may well have contributed to Pardew’s personal good fortune. But luck, as even Sir Alex can tell you, never hurts.