In a previous article of mine, I looked at which teams were due for regression in their defense based on the %’s allowed on open shots. But a number of people on reddit, social media and in the comments section pointed out that good defenses will force the ball where they want it to go and allow “bad shooters” to take those open shots.[1. The tricky part is that non-shooters aren’t actually likely to take those shots and if they’re so wide open that they’re forced to take it, they’ll often think about it enough so that the defense can recover.] This is a plausible theory that I wasn’t completely convinced by. So I decided to look into it. Before I go on though, I want to make one thing clear: I’m not saying that defenses don’t leave bad shooters open. Clearly, defenses are going to allow Rajon Rondo to shoot threes[2. Whether he takes them is a different story and this is where this study is going to be incomplete- we can’t account for the shots not taken.]. My hypothesis is that there’s no difference between the great defenses and everyone else. I don’t think you build a great defense around leaving shooters open. And I don’t think great defenses are going to be any different from bad defenses in wanting to leave non-shooters like Rondo open.
Anyways, on to the study. I took all the three-point shooters top-five defenses left open and all the three-point shooters everyone else left open and compared their overall Open 3PT% against all teams[3. Basically a measure of how good of a shooter they are.] to gauge WHO was shooting and what the difference was between top five defenses and everyone else. Let’s look at a few different graphs to gauge the range and distribution:
As we can see with the graphs, the ranges and distributions are almost identical. And we can test the differences in variance by conducting Levene’s test for Equality of Variance:
The test finds that there is an equality of variance between the two groups of defenses. This matches what we saw with the graphs earlier. The next step would be to test the difference in means. If the mean for top five defenses is less than the mean for everyone else, this would suggest that top five defenses are leaving “worse shooters” open. We can test this by running a two sample t-test for the difference in means:
So the difference in means is statistically significant[4. In statistical terms: the p-value is less than 0.05, which means we reject the null hypothesis of there being no difference in means.]. So top five defenses do let worse shooters take open shots right? Not so fast. This is where we have to be careful with statistics and interpreting them. There’s a difference between statistical significance and practical significance. Looking at the two means, there doesn’t seem to be a large difference: just 0.2%. I think you’ll be hard pressed to find any basketball analyst who’ll tell you a difference of 0.2% is significant. But if that’s the case, why were the results of the t-test statistically significant? Sample size. When you have a large sample size, which this was[5. You can tell from the degrees of freedom, which is n-2, n being the sample size.], everything is going to be statistically significant. But there is a way we can test the difference in means while accounting for the large sample size. I sampled data from each group[6. a thousand observations from each group] and compared the difference in means of the samples by running the t-test again. I did this 10,000 times and found the p-values were significant in 11.3% of the t-tests at the 5% significance level and 19.1% at the 10% significance level. So, we can say that in ~ 90% of the samples, there was not enough evidence to suggest the means were different. This confirms what we were talking about earlier with regards to practical significance.
Finally, I decided to test out a few more comparisons between top five defenses and everyone else. Do top five defenses allow less open shots compared to everyone else? What if we break this down by location? The answers might seem obvious but maybe the results were less so.
Note: Open Rate is simply (Open FGA + Wide Open FGA) / Total FGA. The definitions of these can be found on SportVu’s NBA.stats.com page.
Some interesting results[7. And in case anyone is wondering, the sample sizes here are not nearly large enough to be affecting the results. We’re talking about hundreds of observations here, not 10s of thousands like we were in the previous example.]. Overall Open rate is on the edge of statistical significance at the 10% level. But ultimately, it’s not. So we can’t say with any certainty that the means for Open Rate are different for top five defenses and everyone else. That’s surprising. We’d expect top five defenses to allow less open shots than everyone else but the difference in the rate is really small.
However, when we look at the locations, we can see why that might be the case. For shots in the Restricted Area, the difference in means of Open rate isn’t statistically significant. This is also a bit surprising but consider how rare it is for teams to get open shots at the basket and it makes a bit more sense. Simply put, most defenses are good at not allowing open shots at the rim. And so this isn’t really what makes the difference between a great defense and every other defense. The occurrence is simply too rare for it to matter.
If we look at threes, the difference in means for Open Rate is statistically significant. This isn’t surprising considering how the three has become a vital part of the game today. And again, consider how many Threes are open (over 80%!) and we can understand why it’s so important to not allow them.
And finally, the difference in means for Open Midrange Rate isn’t statistically significant. But notice the means. The one for top five defenses is higher. This suggests great defenses are encouraging that shot at the expense of preventing threes and shots near the basket. It’s no secret that the midrange shot is the most inefficient in the game and even open midrange shots aren’t always preferable to other shot types so it appears great defenses are ok giving teams open midrange shots if it means preventing threes and shots near the basket.
Perhaps more interesting is what effect this has on the offense. It’s long been known that great defenses try to prevent threes and shots near the basket while encouraging midrange shots[8. This article may be one of the few to quantify that notion by looking at defense type (if not, link me!). Seth did look at this with respect to the Rockets awhile back.] And so the effect that has on an offense- where the open midrange shot might be the best shot it’ll get vs. a great defense- is important because teams who don’t have players who are willing or used to taking that shot, may struggle more vs. the top defenses. That effect is certainly something worth exploring in greater detail and with the SportVu data, we certainly have the tools to dig into that more.