Mountains Out of Molehills and a Known SportVU Issue

By now you’ve probably heard the news. SportVU is broken:

"The obvious flaw is that contesting shots requires more than just positioning. The defender has to actually get a hand up prior to the shot. But, hey, we’re talking about NBA players, so maybe we can assume that if these guys are within 3.5 feet, they’re going to get a hand up. And, sure, while we’re talking about a sport with some of the tallest athletes in the world, maybe anybody outside 3.5 feet can’t really contest the shot. Nobody closes out on shooters in the NBA, right?Crunching the numbers on the publicly available optical data (using the within-3.5-feet definition), SportVU classified only 12.8 percent of three-pointers as contested during the 2014-2015 regular season. I don’t know about you, but when I watch an NBA game, I don’t have the sense that virtually all three-pointers are open shots."

Marketed[1. I use the term extremely intentionally.] as an expose of the flaws of the SportVU system, this is nothing more than sensationalist repetition of a well known issue, at least for people paying attention. It takes literal minutes of research on the publicly available SportVU data set see that setting the contested/uncontested line at a defender 3.5 feet from the shooter is incorrect with respect to three pointers. There is a reason when looking at the identities of the most open shooters and publishing shooting breakdowns here on Nylon Calculus, I’ve broken threes into bins where the defender is 4 feet or closer, 4-6 feet away, 6-10 feet and 10 or more feet from the shooter[2. Measured body-to-body, as are all SportVU distances between players.]. Based on now two seasons of data, those are roughly where the breakpoints exist between various degrees of defensive pressure. In 2014/15, the 3FG% in each bucket were on aggregate 30.8%, 34.7%, 38.3%, and 41%.

Considering league average 3FG% is usually right around 35%, a little higher when you remove end of quarter heaves[2. In 2014/15 the league as a whole shot 35.5% counting only threes taken from closer than 30 feet.] and end of shot clock launches from near midcourt, I’ve taken to calling the latter two buckets “open” and “wide open.” Those labels don’t really do anything other than convert the data into a more colloquial understanding. The proportion of threes which, according to SportVU, fall into the open and wide open categories? 37.8%. Meaning 62.2% were either tightly or moderately[5. Again, my terminology.] contested. The magic bullet answer offered by Vantage? 66.3%. All of a sudden the “holy crap!” discrepancy becomes simple nibbling around the margins.

This isn’t to say SportVU is perfect. And, by the way, when we are talking about “SportVU,” we are almost always talking about “the data made available publicly having been washed through certain algorithms.” The challenge is pulling out the basketball concepts from the sea of moving dots which is the raw SportVU data and translating them into our understanding as viewers and analysts of the game. But the algorithms feeding the public data miss things. I’ve noticed a few dozen examples from last season of the those processes simply missing quick give-and-go passes[5. Leading to situations where according to the public shot logs, a point guard might dribble 15 times and then make a shot for which a teammate is awarded an assist. What really happened in every case I looked at was quick ball movement between too players being recognized by the algorithm as another dribble, while a human scorekeeper correctly identified it as a pass, hence the assist.]. At other times shot distances, times and even the identity of the shooter don’t align between the SportVU data and the play-by-play. But of the more than 200,000 shots taken last year, our informal estimate is at most 1-2% have some sort of discrepancy.

The neat thing is, by improving the algorithm, those errors can be reduced and vanished. In fact, merely by relabeling and slightly refining how threes were categorized as contested or uncontested, the problem, purportedly invalidating of SportVU in toto, goes away.[2. And it should be noted that in many other areas where SportVU data is presented on NBA.com, more accurate labeling is and categorizing of shots is available, whether in the form of raw shot logs – linked to video so you can judge for yourself! – individual player and team dashboards as well as across the NBA as a whole. The contested/uncontested fields of the player tracking box score are a design decision prioritizing simplicity over total accuracy. Possibly not the decision I would have made, but that choice, more aesthetic than analytics can and should not be taken as invalidating the whole endeavor.] Manual tracking can do many things well, but going backwards to correct errors en masse isn’t one of them.

Of course, manual “eyes on film” breakdowns are important. Nobody worth listening to has claimed otherwise. And there are things that by the nature of how SportVU tracks movement simply aren’t captured yet. Hand position, body orientation. Where a player is looking. Subtleties of footwork and balance. These are important things that can and should be tracked to further aid in the understanding of how the 11 actors[11. Ten players and the ball.] interact with each other to create the outcomes we see. But it’s a yes/and not and either/or thing, and reducing it to a schoolyard debate about my tracking system can beat up your tracking system is about as productive as the tired “Basketball PhD’s vs. Numbers Geeks” debates we’re maybe, finally, moving past.