PT-PM, RPM and the Curious Case of Ersan Ilyasova and Khris Middleton
Brian Spurlock-USA TODAY Sports
My beta “Yet Another Plus Minus model” utilizing SportVU data along with traditional box score data and play by play data—officially named Player Tracking Plus Minus (PT-PM) is three-quarters through its first out of sample season. Therefore it is the first real test of the metric[1. PT-PM in a blend with RAPM has lead the APBR prediction contest for most of the year, though that contest always contains a significant amount of luck. It is not as though I predicted Kevin Durant’s injury, for example, or this year’s crazy trade deadline] and a month before I completely re-evaluate it.
I decided to compare the PT-PM scores thus far to two of the better known plus minus metrics, ESPN’s Real Plus Minus (RPM) and Basketball Reference Box Plus Minus (BPM), to see how closely they track to together and where they differ. There’s really no right answer, given that all of these metrics are mere approximations, but I would also probably rather not see my metric completely on its own island.
In addition to the overall correlations I was able to break down the offensive and defensive splits, which are in the table below for all players with 600 or more minutes:
The table shows that though the overall relationship between PT-PM and the other two metrics is virtually identical, the splits are quite different. The relationship with the box score derived BPM is split, offensive PT-PM and offensive BPM have the strongest relationship, while the defensive side has the weakest relationship. On the other hand the relationship with RPM is completely symmetrical with the exact same R^2 value on offense and defense.
The defensive side of the court is in some ways the more interesting, partially due to the relative difficulty of modeling it and the greater impact the SportVU and play-by-play data has on the PT-PM defensive model. Given that there is a decent relationship between defensive PT-PM and RPM we can also look at the outliers to get a better feel for the differences in the models’ design.
Below is the scatter plot for defensive PT-PM and defensive RPM, with the biggest outlier highlighted:
The Buck’s Ersan Ilyasova comes out as the largest discrepancy between the two metrics, a player that comes out as an above-average defender in PT-PM and well below average in RPM.
His teammate Khris Middleton, an RPM star, is also a bit of an outlier, though both metrics rate him as an above-average defender. Middleton is similar to a number of the other outliers in that they are rated positively, or negatively, by both metrics with PT-PM giving them a much more conservative estimate, closer to average. That’s in part a reflection of the PT-PM model being conservatively designed using one year of SportVU data, as well as the lesser power of defensive statistics to capture individual player value[2. To me, the fact that we can not as reliably explain individual defense makes a more conservative estimation of value on defense compared to offense a virtue in a metric, giving us a more realistic estimate of our actual knowledge, even if it doesn’t ‘explain’ wins as well in that design].
It is pretty clear that Middleton has a better defensive reputation than Ilyasova, and I am comfortable saying that PT-PM may over-estimate Ilyasova’s contributions this year[3. Gasp!]. But, looking at how how PT-PM applies values to each player is instructive about areas for improvement in the model, the many holes in what we can measure, especially on defense, and maybe items where they have been more or less successful than their overall reputation gives them credit.
As far as PT-PM goes, what the model attempts to measure is that, in a small sample size, Ilyasova has provided above-average rim protection[4. As the result of this study I am testing new methods to stabilize the rim protection measure], drawn offensive fouls at an above-average rate, committed shooting fouls at a low rate, and has been approximately average in defensive rebounds, two-point shot defense, steals, and the team’s defense while he is on the floor (though below the team’s average). For Middleton, his steal rate is above average, while his other measures are near average with the exception of the team’s defensive efficiency, which has been stellar.
RPM is a combination of a ridge regression adjusted plus-minus and box score stats. It is therefore a bit of a black box. But, undoubtedly RPM is, in part, picking up on that team-best defensive efficiency when Middleton has played. It is impossible to tell precisely how much credit Middleton deserves, there are always issues of multicollinearity in adjusted plus-minus models and, especially in a single season, simple luck. It is important to note that the RPM used on ESPN is a single season version, and not what its creators call the predictive version. That makes sense for a media site that wants to generate narratives, but it is a big difference in interpreting the statistic, one that seemed to be lost on commentators at the Sloan Sports conference. I see no reason ESPN could not present both ratings appropriately labelled[5. Note to self, include stabilized version of PT-PM when ever presenting it].
Neither rating has been particularly stable for these two players over the last two years, granted the team is playing much different on defense under Jason Kidd, with Middleton’s rating swinging the most under both. I also have the average weighted by minutes played in the last two years.
In terms of performance this year and the two player’s differing on/off numbers, I looked at some of the details of the Ilyasova/Middleton on/off splits to see why and how the Bucks have been better on defense with the latter player, via Evan Zamir’s site NBAwowy.com[6. NBA With Or Without You].
There’s a lot going on in this table, but I want to highlight a couple of pieces that I think are interesting, if not dispositive. The Bucks defense this year has indeed given up more points when Ilyasova is on the court and Middleton is not (PPP in the table above, higher is worse), though the line-ups with both of them on have not had the same inefficiency.
The best defenses try to limit their opponents shot opportunities and funnel them to the least efficient locations and, of course, contest their shots. The types of shots opposing teams are getting are nearly the same according to Wowy’s data, with the three-point rate and free throw rate about the same, with actually a higher percent of shots at the rim coming when Middleton plays without Ilyasova on the floor. On the other hand, the Ilyasova-only defenses gave opponents more opportunities to score per possession.
But the most questionable difference is perhaps the opponent three-point percentage allowed, which tends to be a fairly noisy number[7. Ken Pomeroy did a nice study in college basketball that indicated that rate of three point shots allowed was a better indicator of team defense than opponent three point percentage.]. Though without on/off contested shot numbers we can not tell if there is any difference in defensive attention in these lineups. In any case, the difference in three-pointers made accounts for approximately 3.7 points per possession (PPP), or 30% of the efficiency differential between Middleton-only and Ilyasova-only lineups, and that difference disappears with both on the court.
Overall, I am relatively happy with how defensive PT-PM stacked up; it is not out there completely on its own island, and the outliers were a mixture of conservative estimates (probably a good thing) and things I now have to look into further. But, measuring defense is incredibly tricky for a number of reasons including the lack of data, underlying team defense being noisier than offense, and dynamic team nature of defense. And predicting it is even harder[7. Though that won’t stop me from taking the Partnow Challenge to test PT-PM as defensive predictive measure equal to PER’s offensive prediction].