In my last post, I talked about the tendency to (accidentally) abuse statistics in business. Again, this abuse is generally not a conscious act. We often end up measuring things that can be too easily gamed and abusing them by, intentionally or not, aligning incentives with a particular metric that ends up being an end in and of itself when it wasn’t created with that intent.
In this post I’ll talk about one industry that is really doing a pretty good job of creating metrics that are meaningful and very hard to abuse in this way: professional sports. I only really know the NBA well so I’ll stick with that for this post. (And yes, this is basically an excuse to talk about basketball at work). While I was sort of tempted to do some NBA stats analysis for this post, folks at places like ESPN and 538 have put in enough work on that front already.
When playing in a rec basketball league in Cambridge, I observed just how pervasive stats have become in sports. First off, we actually even track player stats online — yes, that’s right, a bunch of former high school benchwarmers created a more-sophisticated-than-called-for system for doing this. We even have a Player Rating formula. One of my teammates likes to yell “ASSIST!” if one of us scores off of his pass — to make sure somebody captures his numbers. He’s (sort of) just doing it to be funny, but it reminded me of the same way stats get abused and have been abused even in sports. An assist was just something that was manufactured to measure a player’s impact on the court beyond the most obvious measure of points scored. But that kind of stat can get a bit corrupted because it’s so easy to exploit — a player can run around trying to be the league leader in assists, even if maybe the right play was to shoot or to just hold the ball when a pass is likely to be a turnover. Any Given Sunday has a good side narrative where a player’s bonus was contingent on getting 1 more sack and 3 more tackles … and he gets his bonus despite his health. Of course the sacks, like assists, are just one of many possible statistics you could track for a player, that may or may not really reflect a player’s value in the bigger picture.
In the last few years, the NBA has begun creating some valuable stats that are much harder to fake, game or exploit. For instance, John Hollinger created the Player Efficiency Rating (PER), a measure of per minute productivity, and Value Added (VA), the estimated number of points a player adds to a team’s season total above what a ‘replacement player’ would have given the team. Unlike more “raw” statistics — points, rebounds, assists, blocked shots — these are harder for players to game but generally more valuable for forecasting value of players. One really cool analysis from June 2015 can be found here. It implies that stats like PER do correlate strongly with wins, but that there is still room for a lot of improvement, reproduced below.
Another interesting new NBA stat is the Real Plus Minus (RPM), which estimates net point differential per 100 offensive and defensive possessions. Some analysts still have complaints about this one. (And I do wonder myself, even as a Celtics fan: this year, did Jae Crowder really contribute to more wins this year than his teammate Isaiah Thomas? Not immediately intuitive — Isaiah is averaging about 20 points per game and will probably be an All Star this year.)
Pro sports have developed arguably the most sophisticated and well-meaning approaches for measuring performance. Maybe most importantly there is an ongoing, vibrant conversation and real effort to measure things well, compared to many industries. Pro sports have an advantage over most industries though: namely, wins and losses represent the ultimate, indisputable measure of success. (Unless you’re Philly, which has an interesting take on things). That means in theory, any statistical model or metric you build can be trained or at least examined by wins and losses, and nobody should really argue with what you built if it correlates well.
My long-winded point is this: pro sports generally have managed to create statistics that do a better job reflecting value than most other industries. The takeaways that could be (maybe not as easily) applied to other industries are as follows:
- Multivariate statistics — those that capture multiple dimensions of something desirable are going to be harder to “fake” or build faulty incentives around. In the NBA, PER looks at a lot of offensive and defensive statistics in an attempt to get a more well-rounded stat, despite its shortcomings.
- Ground truth — this isn’t always easy, but as best as possible, organizations should think about an ultimate measure of “ground truth”, or a holy grail that everything should ultimately drive toward. With consumer mobile app usage data, this probably should not just be revenue, or just be daily active users, or just day 7 retention — it might need to combine all these things somehow. We can learn something from ideas like PER and RPM, which tried to combine multiple things at once to map to ground truth. But this will require more thought than running regressions on wins and losses — they don’t exist.
- Multiple measurements — no one metric can or will capture everything. Having a few different measurements offers different windows into data. If different, high-quality measurements are giving the same insight, there’s a good chance that it’s a reliable one.
- Conversation — nobody in the NBA is taking any of these metrics for granted. To the contrary, there are a lot of people intently focused on getting them right and highlighting the shortcomings of even the best metrics. This is something every industry focused on measurement should be doing actively and in specific terms — not just vague terms like “Engagement”.