Freelance Friday: Contextualizing DRE

Freelance Friday is a semi-regular series on Nylon Calculus where we solicit and publish analysis from readers and other members of the analytics community. In this edition, Jake Garbarino provides some important reminders on the positives and negatives of statistical plus minus models such are our own Daily RAPM Estimate (or DRE). Jake is a Celtics fan and chess expert who coached a high school basketball team to a 2-10 record. More important than wins and losses was the ability to shout “Kobe!” every time they took a long contested 2. Freelance Friday submissions and pitches are welcome at TheNylonCalculus at gmail dot com.

Kevin Ferrigan’s DRE uses a simple regression to obtain box score weights from Jeremias Engelmann’s RAPM. Let us assume that RAPM is a strong estimate of a player’s value. There is good reason to believe this to be the case, as RAPM based prediction models have generally performed extremely well in prediction contests. Ferrigan’s initial multivariable regression assigned offensive rebounds an insignificant value, he states that he (correctly) “believe(s) they are important and have value” and thus used total rebounds instead of orebs and drebs separately. This is reasonable. It should also begin to shed doubt that the weights it assigns will be explanatorily useful.

In baseball we’ve seen regressions assign triples negative value, assign triples equal value to home runs and assign getting caught stealing as a neutral event. These models can be effective at prediction, though they normally are overfit to their sample and perform worse out of sample. Much like triples, we know offensive rebounds have positive value through the logic of the game. Which leads to an important point:

"If you have data that suggests offensive rebounds have no value you are likely seeing a bias that isn’t otherwise accounted for[1. Ed note: to put it another way, often times, counter-intuitive results should lead one to question one’s analytical model rather than immediately jumping to conclusions about the incorrectness of conventional wisdom. As an example, metrics suggesting Nick Young is an excellent defender should be viewed with a great deal of skepticism. Even without statistical proof, this skepticism is warranted because our prior, that as basketball fans we have strong reason to believe he is not, is well-informed based on years of observation of Nick Young the actual basketball player rather than him as the sum of statistical accumulation in a given measurement system.]."

It is possible that offensive rebounds correlate highly with negative effects. Perhaps teams that get more offensive rebounds have worse transition defense or allow more transition opportunities. Each offensive rebound must, necessarily, have positive value, but teams that get offensive rebounds may be likely to do poorly at other things. Transition defense is not included in box score stats, transition opportunities conceded are not included. They are, however, included in RAPM. The disadvantage of RAPM is it gives us a total value that includes factors like transition defense, but cannot tell us how much that factor is worth. It gives an evaluation but no explanation[2.  a point made repeatedly and effectively by various Nylon authors.].

Regression is a wonderful tool and in this case DRE gives a useful estimate of RAPM that can be applied to individual games or smaller samples than RAPM is publicly available for (or could be calculable from without enormous noise and margins of error). This is great. However, there is not convincing reason to believe its component parts – the weights it gives the individual box score stats – are accurate weights for those stats individually. Like RAPM it only gives its audience a useful aggregate. DRE can say a player performed badly, and looking at DRE we can see it is because of excessive turnovers, but once we have isolated turnovers the weight we are seeing is the amount turnovers predicts the error of the other variables[3. Multivariable regression weights each variable by observing how much of the error in the other variables’ best fit line it explains. If two variables are correlated then it will only offer additional information to the extent that it is not correlated and will be weighted as such. When not all relevant predictors are included – and they aren’t, the box score is very incomplete for these purposes – the weights we see are not going to be the actual values of the variables but which correlate best with variables that have not been included.]. This is not the same as a turnover’s “value”.

Stolen bases in baseball correlate with being fast, being a good fielder, and many other positive characteristics. Simple regressions can thus assign them extraordinary value. 538’s widely distributed article on steals in basketball similarly assigned steals an enormous, ridiculously, implausibly large value (9 points). How could a steal conceivably be worth this much? Change of possession isn’t worth close to that amount even if the offensive team was otherwise guaranteed to score and now the defense is guaranteed to score. But it might be that steals correlate with all those valuable things the box score is misses, like wide swaths of defense. A steal is not worth 9 times a point, but through correlation with other missing variables steals could indicate 9 times as much about a player’s actual value[4. In this particular case, it probably doesn’t as the analysis in question uses so few useful variables, ignoring shooting %s entirely for instance, the main takeaway was that points themselves are a fairly poor indicator of impact more than the importance of any other input.].

DRE is indicators. It does not tell us a player is bad because turnovers have x negative value and they had a lot of turnovers. It suggest to us that a player is probably bad because turnovers are bad or correlate highly with other bad things that other box score stats do not correlate with. It does not tell us what the actual cost of a turnover is. It does not inform the audience much better about a how a player causes their impact than RAPM[2. Ed note: If I may counter respond to Jake’s response, my preference for statistical plus/minus models such as DRE over APM-style models is because those things which the metrics doesn’t tell me are far less opaque. I know it’s probably wrong, and I know how and why the errors are likely to creep in, which allows me to better weigh and contextualize the number, something which an APM model can make extremely difficult. But knowing what I don’t know is not quite the same thing as actually knowing things in the first place, so from that standpoint Jake’s critique is well-taken.].