Psychological testing is the last frontier of NBA Draft analysis

For years, student-athletes in nearly every major sport at the University of Michigan went through a preseason ritual of sorts. Apart from all their workouts, playbook study and team activities, players underwent an assessment from a university-wide expert, one who then worked with their coaches and physicians on individual breakdowns. The top shot-callers across different sports at the university may not have had much in common when it came to day-to-day responsibilities, but nearly every single one of them spent time with this specialist before a new season got started.

This guru, if you will, one whose fingerprints were all over most successful Michigan sports teams for years — he’s not a world-renowned athletic trainer or anything of the sort. His name is Scott Goldman, and he’s a performance psychologist.

The assessment had a real effect, and not only on players. After just a year or two using it, Goldman started to notice many of the coaches he worked with — the same types who in years past would have been downright allergic to the idea of “mental health coaches” messing with their players–– changing their terminology to match his own.

“’Scott, I went on a recruiting trip and saw an athlete I know would have great spatial/visual processing.’ That’s how they started talking to me,” Goldman tells The Step Back today.

Read More:

An end to exceptionalism is only a Kawhi Leonard deal away

Gone were old-timey platitudes like “grit” and “smarts,” replaced by terms like visual awareness, coachability and decision-making skills. These coaches and trainers quickly realized that these areas had a demonstrable effect on their athletes’ performance.

Goldman is no longer with Michigan, but he remains an employee of an NFL team and an important name in the field of sports psychology, holding several active consultation contracts with teams across all four major North American pro sports plus the MLS. He’s also the founder of Athletics Intelligence Measures, LLC, a company that created and now implements a mental analysis tool called the AIQ (Athletic Intelligence Quotient) test — the same one he used to give to Michigan athletes.

Goldman is far from the only member of his field making waves in the sports world, however.

There’s long been a faint, blurred public connection between sports and psychology. Maybe the most well-known sports-related psychological test is the NFL’s Wonderlic, which becomes a point of debate around the draft every few years. Some NBA teams may even have their own version of the Wonderlic, though its application and use has been sporadic at best.

Over recent years, general understanding of the connections between the brain and physical performance has increased exponentially. In turn, there’s been increasing demand for experts with Goldman’s kind of skill set: The ability to not only identify and connect various psychological variables to performance, but to help standardize these principles across an entire organization or even an entire sport.

It’s an area some NBA teams have put an enormous number of resources into — and, curiously, one where some on the other end of the spectrum are almost entirely absent. But as tests like Goldman’s become commonplace, particularly during pre-draft assessment periods, teams ignoring this realm are at the risk of falling behind.

The Step Back took a look inside the growing world of psychological testing in the NBA, the clear effects it’s already had on title-winning teams, and the people working to bring it to the forefront of pre-draft analysis.

Goldman and his partner Jim Bowman, a school psychologist in New York, began working on the AIQ in 1998. They spent nearly 15 years refining it before taking it to market in 2012 under the Athletics Intelligence Measures, LLC umbrella. They currently have contracts with six NBA teams, with roughly 30 total across all sports, Esports and Olympic training facilities.

A big draw of the AIQ test is the control it gives teams over the entire assessment process. When a new franchise contracts the Athletics Intelligence Measures, LLC team, the very first step taken is identifying the proper “touch points” within that franchise — the in-house psychologists or scouting staff who will be giving the test. These people are all given specialized training by Goldman and his team, showing them how to administer the test using tablets loaned to the organization.

This means there’s no requirement for an AIQ team member to be in the building to administer the test. Teams can do it themselves, with all the information then sent to AIQ headquarters when the test is complete. From here, teams have a variety of options when it comes to the processing and analyzing of the data: Some teams want very little beyond the standardized test score for big data analysis; others request a one- or seven-page analysis from the AIQ team; still others will actually fly Goldman or Bowman out to their facility to provide in-depth analysis in person.

The AIQ test itself is relatively simple, with four broad cognitive abilities tested for: Visual Spatial Processing, Reaction Time, Decision Making, and Learning Efficiency. Every section has its own set of subtests based on 10 narrow cognitive abilities, each of which take two to three minutes to complete. The test is meant to be dynamic and interactive — like playing a game of Tetris — and should take 35 to 40 minutes on average. A final score is determined, then analysis and recommendations are given based on the team’s desired level of insight.

Over 50 potential NBA draft prospects have already taken the AIQ so far in 2018, and Goldman expects another 10 or 20 to complete it before the big day. There are still some benchmarks to reach when it comes to sample size and linking on-court performance with the test in a statistically significant way — benchmarks leagues like the NFL and MLB were able to cross much faster due to far larger drafting pools. All the trendlines look the same as they did in those sports at the same period in their sampling, though.

“It was panning out exactly how we’re seeing in the NBA,” Goldman says. “The trend is definitely consistent with what we saw and what we later confirmed with [NFL and MLB].”

Six NBA teams have already bought in, in this case primarily for the descriptive quality of the test — its ability to provide real insight into how an athlete thinks. That number will only grow as the sample becomes more robust, allowing specific links to on-court performance to become stronger.

The importance of this data is becoming clearer throughout the sports landscape. “Teams that use us, they use us every year, and they have a tendency to buy more,” Goldman says. Athletic Intelligence Measures, LLC’s client retention rate is 93 percent; clients virtually never go back once they’ve started the program.

“We have been honored to work with three or four Super Bowl winners, three World Series winners, one NBA championship team, one Final Four runner-up, a number-one ranked basketball team, some BCS bowl winners,” Goldman says. “It’s been an amazing experience watching these leaders of their leagues use the AIQ data to their advantage. They have been kind enough to circle back to us and report how useful the information was on their successful season.”

At the same time, he’s realistic about the effect it has. There’s no definitive public proof at the team level regarding the level of impact these kinds of assessments have had on organizational success, though growing samples suggest this could be possible in the near future. It’s important to keep perspective, though, and Goldman does.

“I don’t want to mislead and say the AIQ made them successful, but the AIQ is folded into their processes,” Goldman says. “The AIQ is just one valuable piece.”

So, too, are Goldman and his team just one valuable piece of the growing psychological emphasis around the league. Nearly every NBA team has at least one full-time employee in this realm now, with some pouring huge resources into it and outsourcing multiple areas while others basically pay the field lip service and nothing more. Getting detailed information in these areas is virtually impossible, even off the record; teams protect this stuff as closely as nuclear launch codes.

Some of Goldman’s own team clients are even going further, combining variables from the AIQ with other physical variables like height, weight, wingspan and more — and finding even more statistically significant results. He chuckles at the fact that some of these clients won’t even share their new revelations with him when they find them.

All of which brings us conveniently to the other big part of this conversation: Publicizing and standardizing psychological variables across the NBA and the very sport of basketball.

When team analytics staffs create statistical draft models, they’re essentially assigning weights to objective pockets of data — stats — that are standardized throughout the basketball world. A rebound in college is the same as a rebound in the NBA from a counting standpoint; the goal of these models is to determine how repeatable and predictable various stats or skills are from lower levels up to the NBA.

What if we wanted to incorporate psychological factors into that same approach?

See, not every team with an interest in psychological evaluations for their draft prospects works with Goldman and the AIQ group and gets their analysis as part of the package. Some have in-house teams, others may outsource in various ways, and still others use hybrid forms. For draft evaluators, this means there’s no standardized set of objective variables to work from like there would be for a statistical draft model.

Not every GM can convince their owner to spend big for a test like AIQ that comes with analysis and recommendations at a certain price point. As a result, many staffs are left relying on mostly qualitative psychological evaluations done by one or two in-house psychologists — evaluations that, to be clear, are enormously valuable coming from specialists within the organization. At the same time, though, these evaluations lack the kind of historical perspective that actively connects personality and situation in a quantifiable way, meaning they can be very tough to translate into terms other non-psychologists in the organization are familiar with.

That’s where Eric Weiss, founder of SA LLC and creator of DraftExpress Chemistry, comes in.

For nearly 15 years, Weiss has been working with a number of NBA teams on achieving this kind of standardization. His DX CHEM tool includes a network for collecting player information and correlating it directly to team environment and performance data. After teams have administered their first-person assessments, such as the AIQ test or others, they can integrate that data seamlessly in order to baseline it against on-court results. And just like a draft model in many ways, the tool assigns weights and context to several psychological variables that have shown predictive power historically.

This kind of integration holds potentially massive value for team evaluators. This is particularly true in promoting collaboration between analytics staffers, who view things primarily through a quantifiable lens, and team performance psychologists, who are helping to support the person behind the production.

“He brings the ease of analysis for those analytics professionals,” Goldman says of Weiss’s innovation. “His device is the scalpel, and they’re the surgeons.”

Weiss’s tool, similar to the AIQ, looks to put things in terms basketball people will understand. Areas tracked include chemistry, role/responsibility, historical comparisons to others with similar character traits, goal orientation and many more. The goal is to inform and empower people on both sides of the evaluation — the evaluators and the players themselves — to understand what these variables truly mean for their future performance potential.

“There’s a stigma on how data is used to qualify players,” Weiss tells The Step Back “In the absence of quality research, most of it has been used in a pass/fail manner.”

What Weiss means is this: Teams were simply looking for a yes or no answer when it came to psychological tests — instantly steering clear of guys with even minor red flags rather than assessing the full context.

Weiss is bringing that research and helping change that stigma. With his tool, teams and players can get a great idea of several important fit areas: How they might function in a given NBA role after having a very different one in college, for instance, or how they’ll respond to various coaching styles. A huge amount of attention is paid to how a situation might augment or hinder the success of a player given his personality traits.

Teams can break down the contributions they’ve received from various players, measured by Win Shares, based on their personality type. This shows the value of a centralized database that collects more and more data to increase a robust sample every year.

Franchises can look at which prospects are most likely to fit with their team, given a number of team and individual traits. They can also go deeper here, breaking down exactly how and why a prospect matches with their team’s overall makeup:

For a more individual look, teams also have access to detailed core dynamic comparisons between prospects and current players:

This doesn’t stop at the draft, either. Weiss and his team are able to work with franchises on player personnel areas years into the future as well.

“The book isn’t written on a player once he’s drafted,” Weiss says. “Situations can have a huge effect.” Weiss and his team have helped multiple franchises obtain players who were in bad situations for their personality, then turn them around into productive NBA contributors.

“Player personnel scouting is the biggest market inefficiency, in my opinion, because teams tend to over-attribute situational failure to confirmation of the player’s previously known flaws. It’s human nature to assign more credit or blame to the person and discount situation, which is a mistake.”

In an optimal future, Weiss sees his toolset as self-guided and managed fully by teams themselves. Only limited consultations would be needed– teams would have a fully centralized database of psychological variables to plug their assessments into.

Let’s be clear: The NBA as a whole isn’t there yet, and probably isn’t close. There’s still major skepticism about this stuff in some circles, particularly when it comes to a central database allowing access to multiple teams. Certain front office types would sooner see this data eliminated from NBA circles altogether than share their assessments with other franchises. Some in this league are absolutely that petty and that protective of their secrets – even if, as Weiss suggests, it might be hurting them in the long run.

“Teams that think they are gaining an advantage by keeping their information in a silo are missing the point,” Weiss says. “Every team has the ability and access to a viable personality assessment tool, so there is no real advantage gained by using different assessments and keeping small data samples proprietary.”

Weiss’s argument, essentially, is that pooling this data together league-wide benefits everyone. He compares this area to several prominent Fortune 500 companies who, roughly a decade ago, realized the symbiotic value of standardizing quantitative evaluations across the industry: The information is available to everyone, and the real arms race is hiring the best people to find the crux points in that information before your competitors do.

The NBA isn’t the financial world, of course. It’s more of a zero-sum game. There truly might be some teams who view no progress in this area as a positive — they might feel they already hold distinct drafting edges over other front offices, and these kinds of league-wide developments might diminish those edges in their eyes.

Next: The next seven days in the NBA will be wild, but which domino will fall first?

Which side is right here — and which ultimately wins out, perhaps more importantly — is for another time.

Broadly, though, it’s impossible to deny the growing relevance of psychological assessments in the NBA, whether it’s during the draft process or elsewhere. NBA teams are always looking for the next league-wide inefficiency to exploit, and this could be it.

Experts like Goldman, Weiss and others are starting to provide a whole lot of answers here – including to questions talent evaluators never knew they had. Don’t be surprised if the next major NBA evaluation frontier isn’t about what guys do on the court, but who they are away from it.