Breaking Down Scouting Factors in the NBA Draft

Intuitively, I think, most NBA draft observers realize that the prospect evaluation is most likely to be successful when it includes both analytical analysis and in-depth scouting. Choosing one at the expense of the other is perpetuating a false dichotomy.

Layne Vashro came up with a clever and simple way to capture this with what he called his Humble Model that included the player’s draft position scaled to emphasize the highest picks. I have also incorporated Vashro’s methodology in some of my models, as well as using the high school consensus prospect rating known as the RSCI rating in a similar top-heavy rating.[1. Both draft and high school prospect ratings tend to perform in a way best captured by a log-linear fashion, weighted heavily to the top and declining more gradually later.]

Both the draft pick and RSCI rating are basically blunt objects that gives us a rough estimate of overall draft evaluation but do not give us an idea what qualities scouts and front offices are picking up that is not captured by the stats. There are three broad categories that strike me as the most plausible:

• Defense: Box score statistics are notoriously weak in measuring defense, relying primarily on blocks, steals and defensive rebounds. It is reasonable that scouts should pick up more information on that end that is predictive of future performance.

• Athleticism: One of the most pertinent questions is how to weigh the observed athletic potential of a prospect against on court performance. Here we get into “ceilings” and “floors” of players that may not be evident in the current performance.

• Character: Last year I did a sort of meta study of how character traits have been used to project future performance in sports and academics, finding only a modest success in the literature. But it still stands out as one of the more interesting potential areas for scouting value. Undoubtedly hard work, mental toughness, leadership and more all play a roll in future NBA success, but well can outside observers predict those characteristics in 19 and 20 year old?

The site NBADraft.net has been tracking twelve different scouting measures for draft prospects since 2006 measured on a ten point scale. I decided to do an exploratory study to see if any of these measures correlated to subsequent NBA success when examined alongside the other measures used to make my base draft model. I want to emphasize that this is simply an exploratory analysis, NBADraft.net is simply one public site that evaluates draft picks (Precisely because there are so few it is difficult to gauge how closely their measures align with a more general consensus on these measures), the overlap between my draft data and the NBADraft.net ratings is incomplete leading to a relatively small pool for analysis, and throwing these measures together in an analysis creates a multiple comparison issue.

There are two measures that roughly fall under character measures; Intangibles and Leadership. In the data to the two measures are strongly correlated at .662. Both are somewhat correlated to the player’s assist figures and a slight inverse relationship to offensive rebounds[2. Below are the complete set of correlations between the ratings, box score measures and NBA performance as measured by Max AWS

In addition, there are two measures the would fall under a rating of athleticism; Athleticism and Quickness. Again the two ratings are highly correlated at .627. The primary difference is that ‘quickness’ is a trait attributed to guards more often evidenced by the positive correlation to assists, while athleticism didn’t appear to have any positional bias in its assignment.

Defense is included as a separate stand alone rating. It’s most closely related to the Athleticism rating with a .313 correlation. In terms of stats it is most closely associated with blocks but not steals.

As I noted the two character related measures were closely related, making any comparison between the two in a trained draft model noisy, both ratings were consistently associated with better NBA performance but there was no clear favorite.  Below is a plot of the two ratings[3. I would say it tends to look like the overall consensus, but note that Adam Morrison is given a ten on both measures, so it doesn’t appear to have been retroactively cleaned up either]:

Eventually I combined the two as Intangible-Leadership with a simple average of the two ratings. This was highly stable as a positive predictor in nearly all cross validations,including bagging and boosting models. In the entire sample Intangible-Leadership explained approximately 11% of the variance in NBA performance when combined with other measures like statistical performance and age, and ranged between 8% and 12% in sub-samples. I would also note that the partial probability plots looked like the effect was slightly non-linear with their being more of a penalty for being rated low in Intangible-Leadership than a boost for being high.

If we are going to hypothesize after the results are known, that would indicate that perhaps it is easier to spot potential problem players than extraordinary leaders, but again it’s important not to read too much into this, these are the ratings of one NBA draft site on six years of draftees.

Athleticism and Quickness were also highly related as stated above. But in this case the identifiable differences in application of ratings lead to a consistent preference in the modeling for the more guard heavy Quickness rating in terms of association with the NBA performance.

Below is the plot of Athleticism vs Quickness, relatively higher Quickness in the lower right.

In cross validation the inclusion and effect size of the Quickness rating was somewhat more variable depending on the sub-sample and other variables selected, ranging from 3% to 10% of variation. Again the effect in partial probability plots had a slightly non-linear appearance, though this time there was more of a boost for very high Quickness ratings rather than penalties for lower ratings[4. Post Skills was the only other rating that had some traction, but then primarily as a counter point to Quickness in step wise selection].

Lastly, Defense. The Defense rating was very rarely included in a cross validation model. For this sample and the target variables relying on box score production and playing time the Defense rating did not indicate an independent effect. Again I don’t want to claim too much other than that, actual NBA scouting would likely have better results[5. Though it’s not always clear all teams employ or utilitize ‘actual NBA scouting.’] and the box score-based training variable is far from ideal as a test of NBA defense.

This exercise is intended more as food for thought than anything. Teams judge prospects a number of different aspects through the draft process, implicitly even if they are not writing down separate scores for each. But I’d argue writing those down and then going back after the fact to measure their impact is an important step improving that process.