Nylon Questions: How can we best measure individual defensive impact?

Sports analytics is a constantly evolving field and keeping up can be a challenge, especially with so much work being divided between the public and private spheres. As we head into the 2018-19 season, Nylon Calculus wanted to take stock of where we are and of what’s coming next.

This project is a throwback to work Keith Woolner did for Baseball Prospectus nearly two decades ago, and an update to Kevin Pelton’s basketball-specific version from five years ago. Our staff compiled 10 questions whose answering will likely guide the next few years of public analytic work. Not knowing what has been accomplished in private by NBA teams and consulting firms, we focused on questions that could be worked on in the public sphere, wouldn’t have to be answered with existing datasets (we don’t want to imagine the data we have now is all we’ll ever have) and things that would theoretically have an effect on how teams operate on and off the court.

Hopefully, these questions will help spark, refocus, and recalibrate conversations and lead to collaborative progress here at Nylon and everywhere else sports analytic work is being done.

4. How can we best measure individual defensive impact?

As the advent of modern sports analytics occurred in MLB, baseball has had a great deal of influence in shaping how we think about basketball analytics. We can think of points scored and allowed in basketball similarly to runs scored and allowed in baseball. We rate how well basketball teams are on each end similarly to how well baseball teams rate when hitting and pitching.

However, one advantage that baseball has over basketball is that it is relatively easy to split up baseball events into discrete events, where doing so in basketball is comparatively more difficult. In baseball, the pitcher throws the ball with a certain velocity to a part of the strike zone, which the batter then hits (or not) to a certain part of the field, that the defender either makes a play on or doesn’t. We can measure how hard a pitcher throws and how many runs he gives up, how often and how hard a batter hits the ball, and how far a defender can range to make a certain percentage of plays. It is distinctly more complicated to evaluate how well a player can fight through a screen to defend a 3, given that good defense may not necessarily result in a block or even a miss.

While on the offensive end in basketball we have a decent number of tools to measure an individual’s performance, there are less available for the defensive end. The number of times a player turns the ball over and how often they make their shots tell us a decent amount about that player; it tells us less about the player defending him.

For example, consider a play where a good shooter takes a wide open 3 and misses it. It is certainly fair to penalize the player for missing that wide open shot, but it seems less fair to credit the defender for leaving a good shooter wide open. While evaluating a player’s defense by the amount and percentage of field goals he allows isn’t a terrible metric, it fails to totally take into account the quality of looks that the player is giving up, which while related, isn’t exactly the same thing as field goals given up.

The player may be guarding disproportionally bad or good shooters, and those shooters may be having good or bad nights independent of the defense being played against them. It is the purpose of analytics to understand the root of the numbers that are surfacing, and in terms of individual defense, it appears that basketball analytics has not quite cracked the case yet. For example, consider the following chart of steal and block percentage by defensive points per possession of teams last year.

While there is clearly some correlation between the two categories, as evidenced by the downward slope of the trend line, there is a lot of variation we are not accounting for. Why is Boston one of the best defensive teams in the league while the Bucks are below average given the obvious advantage in counting stats that the Milwaukee has over the Celtics? Our eyes can tell us that the Celtics’ players are clearly superior defenders when watching a game, but we struggle to convert the defensive dominance of teams to the defensive dominance of individuals through counting stats like blocks and steals.

Stats like defensive points per possession offer up a fairly definitive picture of a team’s defensive, and the amount that these numbers change when a player is on or off the court gives us a fairly effective proxy of a player’s impact on his team defense. In his aptly titled Basketball Analytics, Stephen Shea discusses what he calls top down and bottom up metrics, wherein a bottom up metric uses individual performances to build up to a team level and a top down metric uses team performances to drill down to understand individual player’s impact. Real Plus-Minus, which is “the player’s estimated impact in terms of net point differential per 100 offensive and defensive possessions”, is a textbook example of a top down metric and is an example of analytics’ current best attempt at understanding individual impact.

The reason for the tendency towards top down explanations of individual defense is largely one of practicality in that it is no great secret that current defensive box score stats do not effectively capture the quality of defense being played, which the chart near the beginning of the article can attest to. There is simply much more to defense beyond blocks, steals and rebounds, and thus these statistics like Real Plus-Minus try avoid them (although using them as a prior to create a baseline).

As the goal of a stat like Real +/- is to calculate an individual’s impact on a team, it is logical that it tries to isolate the individual’s performance from the quality of that player’s team. Nevertheless, it is entirely debatable whether or not isolating a player’s performance from his team is entirely possible. If a player is moved from the starting lineup to the bench, it is probably that the player’s calculated “impact” on the team would improve given the lesser competition of matching up with opposing teams’ benches, despite nothing about the play of the actual player changing. Furthermore and more fundamentally, in taking this approach we are getting away from the original purpose of our analysis, to understand the root of the numbers we are seeing, as these top down metrics by design do not make any attempt at explaining why players are effective defenders.

While the limitation of the bottom up metrics that have the potential to explain the “why” of players being effective defenders or not is ultimately the lack of sufficiently comprehensive statistics into the quality of defense being played, it does not mean that those statistics will never exist. Tracking data, for example, appears to have great potential to be able to understand how effective a player is defensively by giving insight into the amount of space that a defender is giving a shooter and what effect that will have on the the shooter’s accuracy. Ultimately, it is the continued challenge of of NBA analysts to find these individual statistics that paint a comprehensive picture of a player’s ability to prevent scoring.