George Akerlof shared the 2001 Nobel prize in economics for his work on asymmetric information, which describes the situation when buyers don’t know as much about a product as sellers do. His most famous example of this situation is the market for used cars. Sellers know a lot about their used cars, but potential buyers don’t. So it’s hard for buyers to tell a good car from a bad one, popularly known as a “lemon”. As long as there’s some chance that a used car will turn out to be a lemon, a buyer will never pay full value for it. In fact, the market can collapse altogether, if buyers’ offers are always lower than the value to sellers of their used cars.
This is soccer analytics today in a nutshell. The buyers are mainly clubs but also media, data providers, league offices and national teams. The sellers are people like me. Some sellers are purveying good cars – soccer analytics based on scientific principles and robust statistical methods, with measurable connections to what happens on the field of play. Others are selling lemons.
The problem is that a lot of buyers can’t easily distinguish good analytics from lemons. It’s not their fault; plenty of extremely intelligent people work for clubs and other potential buyers of analytics, but very few of them have statistical training. And even if they bring in analysts on a trial basis, the results of implementing analytics within any short period of time may depend largely on luck. After all, analytics don’t provide certainties, only probabilities.
Making matters worse, the sellers of lemons may be just as eloquent in touting their wares as the sellers of good analytics. Because of the gaps in knowledge between buyers and sellers, a good communicator selling bad analytics may be more successful than a bad communicator selling good analytics. But unlike slimy used-car dealers, quite a few of the people selling lemons are doing it earnestly and in good faith. That’s because many of them don’t have statistical training, either; they don’t even know they’re pushing lemons.
With a market for lemons in full swing, it’s not surprising that potential buyers are reluctant to take a chance on analytics, or that those who do are paying low rates for tools that could save them millions. The question is, how can the asymmetry of information between buyers and sellers – and sellers and sellers – be corrected?
In economics, there are several classic solutions to the problem. One is reputation – an analyst builds a track record of quality and reliability. This solution can be a bit of a catch-22, however, since it’s hard to build a reputation if you can’t find any clients in the first place. Some analysts try to get around the catch-22 by publishing their work online and then reminding the public of their correct predictions. Unfortunately, they don’t always own up to the ones that turn out to be mistaken.
Another solution is to offer a guarantee, so clients pay nothing if the work turns out to be useless. Yet a client who pays thousands for a flawed analysis that ends up costing millions won’t find much consolation in a refund. It’s still not worth the risk.
A third solution is to have some sort of certification by an impartial authority. No such thing exists in analytics, except for the sort of highly technical work that tends to appear in peer-reviewed journals. In any case, professional analysts are loath to make their most powerful tools public. As a result, what passes for certification is a sort of acclamation in public forums.
Pointing out flawed analytics in such forums is usually a thankless task. Online audiences – not always experts in statistics, unsurprisingly – may side with the analyst who has the biggest following rather than the one whose work is the most robust. Despite the occasional trenchant critique, the flow of analytics using outdated and unscientific methods continues unabated. Just as in the used car market, the barriers to entry in the analytics market are low.
That’s no reason to give up, though. I’ve been lucky to work with some great clients – yes, luck does play a role, especially in the market for lemons – and the number of buyers who understand statistics is growing. Next year, I’m planning to co-teach a sports analytics seminar at New York University’s Stern School of Business. Hopefully, we’ll create a few more good analysts to drive out the lemons.