Nylon Calculus Discusses Replacement Level

Sometimes, the best stuff the Nylon Calculus staff comes up with is in our e-mail discussion threads. We previously took a long and occasionally contentious look at the utility of “one-number” metrics and measures of player value. Prior to the draft and free agency, we discussed the concept of “Replacement Value” as applied to the NBA. As draft picks and free agent signings are often the best way to improve a team’s roster knowing how much a given pick or signing projects to improve a team all but requires a comparison of the minutes, touches and shots being replaced by the new signing. So, without further ado:

Seth Partnow (@SethPartnow): With the NBA Draft coming up, it seemed like a good time to start to discuss the concept of “replacement value” as applied to the NBA. After all, that’s the whole purpose of the draft, to allow a team to get better by replacing existing players with better ones, not to mention the similar effect of free agency.

How can you evaluate how much to offer a free agent, or how high to draft a player if you can’t define the value he will bring to your team?

How can you evaluate how much to offer a free agent, or how high to draft a player if you can’t define the value he will bring to your team? And not just his individual “value” (however we define that nebulous concept) but his marginal value to the team. If the Cavaliers were somehow able to add DeMarre Carroll to their roster this offseason, he’d add great value if replacing the aggregate minutes and shots they got from Shawn Marion, Dion Waiters, James Jones and Mike Miller. Plug him in for LeBron James, and suddenly they don’t make negative numbers large enough.

That, I think is the key difference from the baseball concept of replacement value. it’s a fairly well (if still somewhat contentious) established concept, supplying a baseline against which to measure a player’s contributions. And it works because for the most part you simply are swapping out one fungible part for another, with few (in economic terms) externalities affecting the rest of the lineup. This isn’t at all the case in basketball.

There was a stat going around Twitter immediately after the Finals indicating Tristan Thompson had more offensive win shares in that series than did James. While this is pretty silly and looks bad for WS as a stat, that’s somewhat unfair.[1. Andrew Johnson points out WS does not directly use replacement value as a parameter, however, there is a certain level of shooting efficiency below which additional shots are presumed to subtract value. As a general rule, this is fine, but as discussed last week, the next best shot for the Cavs if LeBron didn’t shoot or otherwise create was going to be pretty damn ugly.]  It better illustrates the need for properly defining replacement values. The formula in place works pretty well for the majority of cases, but the extreme nature of the Cavs’ offense in the Finals is illustrative of the general inexactitude of one universal baseline. Sure, James had some ugly shooting numbers in the Finals to be sure, but who takes those shots if he doesn’t, with the rest of the team struggling to reach 30% eFG on shots not directly resulting from James’ own actions? It’s essentially the opposite situation as Russell Westbrook supposedly stealing shots from the much more efficient Kevin Durant.

How do we reconcile the need for a baseline measure of performance with the fact that applying any universal standard is intrinsically problematic in basketball?

The other thing that occurs to me is that we have a hard enough time accounting for role and context in determining overall value at the top of the scale, it’s really not that surprising that all the same challenges exist at the bottom end as well. This is probably just a quieter analytic issues because the stakes are so much lower for drawing this line as compared to identifying the best player in the league or something similar. It also strikes me as something of a fool’s errand to try and measure the value of a player at the top of scale by his distance from an abstractly defined line near the bottom of the scale that is just as hard to measure as the actual value at the top.

Also, get off my lawn you damn kids.

Andrew Johnson (@CountingBaskets) I always think of replacement level pretty much the same way Ian describes. When I was first introduced to the idea of wins above replacement I thought the construct was a little weird. Especially since, in that instance adjusting to replacement level had no effect on the order or distribution of player ratings compared to using the Wins Compared to Average, other than to make everyone’s score a little higher. But I have since been subsumed by the analytics Borg and gotten used to it.

[T]o the extent that replacement level is merely conceptual it is only valuable to the degree that it helps analysts and their audience understand value, and it is not useful to the degree that it is jargon that delineates insiders and outsiders.

I would say to the extent that replacement level is merely conceptual it is only valuable to the degree that it helps analysts and their audience understand value, and it is not useful to the degree that it is jargon that delineates insiders and outsiders. However, there are instances where using a replacement, or fringe, level is helpful in making inferences and predictions.

For example, analytic types discuss mean reversion all the time, but for a player like Jorge Gutierez who only played 180 minutes over the year it is a better assumption that they are a fringe NBA player than an average one. So I use a replacement reversion in my season prediction and draft models rather than a mean reversion, There are similar dynamics in any aspect where there is a selection bias for opportunities at work, like three point shooting or rim protection where we know that players given few opportunities are likely to be less adept.

Krishna Narsu (@Knarsu3): I’m glad you referenced your post Ian. I was about to bring it up: I like that you divided replacement level into positions, which itself is sort of subjective in nature. But it’s still important because different positions have different levels of depth. For example, a replacement level player at PG may be different than a replacement level player at C because of the positional depth of PG. But even that is a bit broad in that within each “position”, there are different skills. So someone who has “replacement level” skills at pretty much everything except shooting may not be valuable to the upper echelon teams and the teams that have shooting but perhaps that one skill is valuable enough to be rosterable for one team. And so I think that sort of brings up an interesting question- should replacement value be defined in terms of aggregate skill (like it is in most all in one metrics) or should it be defined in terms of each separate skill. I think this is where basketball differs from baseball in that it’s easier to quantify total value in baseball but in basketball, team fit is so important.

I guess my whole point is that one player who is considered replacement level to one team might not be considered replacement level to another team because of the skills he has. And I think we see this a lot in basketball in that there are “replacement level” players who have good stretches of above replacement level play, maybe even average play because they landed in the perfect situation.

And I think because of that, I find it’s such a hard term to grasp in basketball. And yet, I totally understand the purpose of it.

‘Our superstar wing player is on a short sabbatical in Miami. We need a halfway capable imitation of a 3 and D player to replace some of his minutes, while other players take a up more of his usage rate.’ Sounds to me more like what we are looking for.

So, somehow I would intuitively define replacement big as a mediocre rim protector with not much more (Brendan Haywood right now), replacement wing as someone who shoots 33% on open three’s on offense and is not a complete minus on defense (Shawn Marion right now) and a replacement point as someone who is similar to the wing, but capable of bringing the ball over mid-court without losing it every time (Matthew Dellavedova right now?)

Andrew: Actually, I think Johannes is on target here. I don’t think it’s useful to be too literal about the ‘replacement’ aspect of replacement level. It’s most useful in setting up your priors whether looking at overall production, a filling a position or examining a particular skill. It would probably be useful to look at which skills are most lacking in fringe players separately for bigs, wings and points to see if there is a pattern. That would help identify the most and least ‘replaceable’ skills, and which skills might be most valuable- I nominate Johannes for this by the way.

Ian: Seems like we’re really dealing with two issues. The first is the conceptual framework of replacement level as a hypothetical benchmark of overall performance. The second is the functional reality of replacement, which bangs up against the real world context of skills being unevenly distributed across players or even within a single player. Seems like they each sort of chip away at each other, but they can’t totally coexist separately.

Johannes: I’ll ignore Andrew’s attempt to force labor on me for now. But yeah, the skill and role question is so much more complicated in basketball than in baseball.  On offense, you basically have two levels of replacement. The lower level is ‘Can this player stand on a basketball court without getting hurt?’. Those players offensive tasks would be either ‘set screens for non-ballhandlers and go for offensive rebounds’ for bigs;  ‘stand in the corner and hope you produce gravity’ for wings; or, ‘get the ball to half-court and then give it our top banana’ for PGs.

The higher level is ‘Can this player be a main part of our offense, with a usage rate above 20%?’. In this regard, Luol Deng or Rudy Gay could probably be considered as replacement players. One big underlying question for me is, how much offensive burden can a specific player handle before he ‘breaks’?

Those studies exist as a bunch approach (at least as far as I remember), but that somehow doesn’t completely make sense to me. Maybe if we calculate scores for each player and each game by usage and some metric of achievement[1. Perhaps DRE?], we see different ‘breaking points’ for different players / player types.

(I’ll now try to think about Andrew’s question instead of piling up more work.)

Nick Restifo (@itsastat): I’m in on this idea that the concept of replacement level is flexible, mutable, and unique to each skill set. There are undoubtedly skill sets for which there is a smaller professional supply, skill sets with a wider range between the best and worst players, and certain skill sets that are inherently more valuable in professional basketball.

There are undoubtedly skill sets for which there is a smaller professional supply, skill sets with a wider range between the best and worst players, and certain skill sets that are inherently more valuable in professional basketball.

For example, shooting off the dribble may be more important than traditional post play ability. What does this mean for the replacement level criteria of these types of players? All this opens up a wide range of questions. What are the replacement level thresholds in each skill? How many replacement level thresholds does a player need to pass to make it in the NBA? Which skill sets are most important? Given a player’s skill set and position, the theoretical concept of “replacement level” should be completely different from player to player. A player may have one elite skill but be “below-replacement” in all others. And how does a player’s skill set impact the players he plays with? Steve Novak might be at or below replacement level in a lot of areas, but his one elite skill (3PT Shooting) can make him look totally worth minutes given the right players on the floor. How much do a player’s teammates move the “replacement level” line?

So to define “replacement level” appropriately, you would have to consider all these factors. Players would have to be grouped not by traditional positions, but by the group of players they play the most like.Then you’d have to define skill sets and not only how valuable they are to that group of players but to the player’s team and teammates as well. This is why it’s much harder to define replacement level in basketball. Because of it’s synergistic nature, there is a mess of co-varying factors.

Seth: Everyone is making cogent points about the factors which go into discerning replacement level. But so many factors have been identified it makes me wonder if the entire process of tying value to “replacement level” is simply impractical? Especially since there is no real consensus as to the replacement value for individual skillsets. Different models use very different baselines for replacement level shooting value (though the ridiculousness of equating replacement level shooting accuracy with league averages is a whole other debate which no one needs rehashed again), and if we can’t really set a standard in the area of the game where there has been the most study on the best and most thorough data, what chance do we have for playmaking, or help defense, or even something as ephemeral as chemistry? Or am I letting the perfect become the enemy of the better?

Justin Willard (@acrossthecourt): Many concepts are borrowed from other sports and applied to the NBA. This is a smart approach, and this happens in science too. Some campuses, in fact, encourage inter-discipline mingling to sprout new ideas and create new relationships. Replacement level is one such concept basketball has used after it was popularized in baseball, but there’s a specific difference between the two sports: baseball’s positions are more defined while basketball’s positions are more fluid.

Replacement level is the quality level of a player you could acquire at a minimum price. This is supposedly useful when a player gets injured and you need someone to fill the void in minutes. But think about what actually happens when a starter gets injured: a good bench player is usually chosen to play more often, even if that player doesn’t play the same position. For instance, Tristan Thompson filled in for Kevin Love and he’s definitely not a replacement level player, and Iman Shumpert saw his minutes increase when Irving got injured even though nominally he’s a shooting guard and Irving a point guard.

Consequently, replacement level is misleading in a general sense. When a team loses a player, the production you use to fill in the gaps depends on the players you have and other players available on the market. There is a rough level of value for a player you could pick up on a minimum contract, but it’ll depend on other factors like if it’s the summer free agency period or during the season right before the trade deadline. Also, if you want precise analysis, these players usually have some sort of background — college, the Spanish ACB league, previous NBA experience, etc. — and you don’t have to blindly use some general number.

There’s another issue here too. When NBA metrics are posted and adjusted for playing time, replacement level is used to translate it to total wins or some other value measure. The problem is that replacement level is so low that it awards high minute players significantly. When we think about what makes greatness, it is not playing huge minutes at an average level — it’s about the peaks you reach and how long you sustain them. Magic Johnson and Larry Bird have roughly the same amount of Win Shares, a basketball-reference measure, but so does Robert Parish, who’s high on the list because he played at a decent level for a very long time.

Replacement level might be useful for the first step of analysis, but it’s not nuanced enough and there’s better information out there.

Andrew: I go back again to the concept of replacement level simply being a tool to help organize and understand basketball statistics, as well as, sometimes to inform our initial estimates (ie priors). Again, sometimes it can just be analytic jargon, which is probably harmful.

But, I don’t think it’s necessary to completely and categorically define replacement level for it to be useful necessarily.  For example, if we turn this around and try to define “Star” players I think we get most of the same definition issues, but pretty much everyone agrees that great teams need stars and should try to draft stars. So is Jimmy Butler a star? Is Klay Thompson? Sometimes going through the process of defining and thinking about a concept is useful analytically, as long as we don’t try to get too complicated in the definition.

Johannes: It should be somehow possible to at least get an idea what a player need to stay in the league. I think there is some kind of hidden ‘fitness value’ (combination of skills) that you need to at least have to remain in the league.

I think there is some kind of hidden ‘fitness value’ (combination of skills) that you need to at least have to remain in the league.

So basically, we would need to see what’s the difference between those players that are out of the league with 26 and those that get 5 to 10 minutes per game until they are 30.[1. Like Walter Herrmann and Louis Amundson. Same weight, same height, same hair. But Walter was out of the league with 29 and Louis still started games for the Knicks with 32 this season. ( I realize that my example is pretty bad, I just got carried away)] Regarding replacement level shooting, I see one huge problem with shooting stats and that is survival bias. I could pretty well imagine that a lot of fringe players shot badly on their first 10 shots – and never got a chance again.

Krishna: I really like your idea Johannes- I think that’d be a great study to look at. But as you mentioned, it’d be difficult to look at replacement level for shooting and that’s a very important skill. If a theoretical replacement level player is replacement level at basically every skill but shooting, where maybe he is elite, is he a usable player in the NBA? We don’t know the answer to that and given the selection bias Johannes has mentioned, we may never really know the answer to this. I think maybe the best way to try and answer this would be to use the d-league stats (where theoretical replacement players are all over the place) and attempt to have an adjusted shooting metric that translates to what those players might do in the NBA. Then compare these players with the fringe players in the NBA who get 5-10 mins for numerous years (maybe up to 30 like Johannes suggested). You’d need a sample of players who’ve bounced back and forth between the d-league and NBA at an early age but have significant mins in both leagues. Not sure if this is really feasible.

And regarding the idea of looking at the combination of skills required to remain in the NBA would a very important study I would imagine. But as I mentioned earlier, there are going to be issues in that I think replacement level would/could be different for every team given the fit/context of the other players on the team. One player might be replacement level for the Celtics while he might actually be usable for the Kings. And this works even when looking at the salary a team might be willing to give the player- to the Celtics, he might be the minimum player who is easily replaceable but for the Kings, that player might be worth slightly more than the minimum because he brings a certain skill above replacement level. So this is where even the whole salary level of a replacement level player- the league minimum player- is even more complicated than baseball. I don’t know for sure but I don’t think you have that issue in baseball. Player A is generally worth the same salary on Team A and Team B. In basketball, that’s not necessarily the case.

Andrew: Johannes- I am fairly confident the selection bias works in the opposite direction on shooting threes. Coaches have many opportunities to observe a player’s shooting form and ability outside of games, and are applying that knowledge in setting plays and influencing shot selection. That’s why that information improves out of sample predictions. To be clear this isn’t directly related to being an overall replacement level player it’s just an improved prior for reversion

Seth: Once we posit replacement level is different for different teams, haven’t we sort of lost the purpose behind the concept though? Or are we just being inexact with terminology?

A more accurate way to phrase it is to say the Knicks will be improved by any addition of above replacement minutes, but Celtics need (hypothetical and made up) “replacement +.5” level play to improve their overall point guard situation, and whether that comes from Marcus Smart improving or a new guy coming in, that isn’t a replacement level discussion so much as a marginal value calculation?

Positive Residual (@presidual): To Seth’s point, it’s precisely the unwieldy nature of the replacement-level concept that leads many critics to wonder whether it’s worth our time. In baseball, the definition (articulated by Tom Tango) is very specific and contains multiple parts:

“The talent level for which you would pay the minimum salary on the open market, or for which you can obtain at minimal cost in a trade.”

Right away, we’re confronted with three criteria, each of which comes with its own set of assumptions that are hard to define:

1. Player productivity

2. Player compensation

3. Market conditions

The group has already unpacked #1, so here are my two cents: even if we can come up with a useful definition of player productivity, we would still need to connect it with player compensation. And, as Dave Berri has noted, while there’s some evidence that production and pay are closely related in baseball, it’s hard to say the same for other sports, including the NBA.

More broadly, the idea of opportunity cost (something Seth referenced in his opening remarks) is hard to incorporate. Tango’s definition highlights minimum salaries as well as minimum transaction costs if a trade is pursued as an alternative. But they’re not the only costs. To paraphrase JC Bradbury, even “playing the player isn’t free,” because at a minimum the team forgoes the option of selling his services to another franchise. Overall, I’m not ready to give up on the replacement-level concept (in baseball, I use Wins Above Replacement fairly regularly). But I do think it’s very difficult to execute and, well, trying to overcome these difficulties has opportunity costs of its own.

Christopher Long (@octonion): Replacement level is pointless, as everything must be set relative to some baseline. A league average baseline is by far the easiest to understand and calculate. Player value is not an absolute, it’s always relative to some baseline, and for the purpose of comparing players, the baseline is irrelevant.

Player value is not an absolute, it’s always relative to some baseline, and for the purpose of comparing players, the baseline is irrelevant.

Proof – (A+C)-(B+C) = (A+D)-(B+D)

When I discuss this topic I feel like I’ve dropped in from the Bizarro universe. What’s the attraction here?

Johannes: How would you define a league average player?

Chris: The same way as in any sport – the median by any metric of your choosing. In baseball the players naturally segregate by fielding position or pitching usage, and you can do something similar (or even more fluid) in basketball.

Andrew: Chris’s proof is the one I worked out when I first encountered replacement level. But, when I built my league win projections using Player Tracking Plus Minus I needed a base point to regress player prior year numbers against. All my research indicated that for players with little playing time they are likely to revert to a number below the league average and that minutes played was a positive (log-linear) indicator for all players, And it helped my out of sample predictions.

The class of minimum salary veterans is also sizable portion of NBA rosters and team building due to the structure of the NBA salary cap and star driven . So projects like Johannes upcoming positional analysis of that class is pretty useful and interesting.

Seth: Despite my practical concerns with drawing the replacement level line, I would go quite as far as Chris. Average is good for a lot of things, but the vast majority of teams would be helped by the addition of below-average, but only slightly, minutes. In plus/minus models, isn’t it typical that the cutoff for useful/helpful to an NBA team is around -2/100? In a salary-capped league, you’re just not going to fill your 8-man rotation with above-average players, so the threshold for what might colloquially be called “playability” is something worth knowing, right?

Chris:  In baseball average salary is, say, $2.5m. Call a win $6m. All you need are the production levels for average (0, by definition), and the production level and cost of the replacement. Then subtract. Or do each positions separately; by the rules, you need exactly one player at each position. One shortstop comes in, one goes out.

In basketball, you have 5 players on the court. One comes in, one goes out. It’s all relative differences. The very notion of the “value” of a player as it’s used is perverse and artificial. Players have different values to different teams. At best replacement level is confusing; at worst it goes a long way towards glossing over details absolutely critical to anyone actually running a team.

Seth: So, without diving back too far into what appears a sore subject, you’re not a fan of the concept even in the baseball environment where it makes the most sense?

Chris: It makes as little sense in baseball for exactly the same reasons. When you buy a TV, do you use a “replacement level” TV as your guiding baseline to decide if you’re getting a good deal? All you need is an intercept point with attached cost and utility.

Johannes: If we are talking about YAPMs[2. Johannes’ wonderful acronym for “Yet Another Plus-minus Model.], then take 0 or whatever, call it an average player and be done with it. I do not actually care.

I do think though that it is easier to define a replacement level player than an average player. Because “In basketball, you have 5 players on the court. One comes in, one goes out. It’s all relative differences.” is not true. Basketball is not a simple formula of addition and subtraction. As Christopher says in the next sentence “Players have different values to different teams”. But maybe there is a universal zero level that doesn’t need any algebra.

Of course, you can use “the median by any metric of your choosing”. But that sounds very artificial to me. The type of players that have an average YAPM have completely different minutes and usages on the team. It seems to me that’s much harder to define an average player (what would be average in any case? Someone you accept as a 7th man on a 5th seed?) than it is to define a replacement level player (“A player without development upside that at least one team is willing to hire at minimum salary”).

The other question is, is it useful to spent time thinking about players that have so little impact on your team?

The other question is, is it useful to spent time thinking about players that have so little impact on your team? Is knowing the lower boundary that helpful for the rest of your team?

Krishna: I think knowing the lower boundaries of your team and the league is important for injury replacements. And injuries happen. Even as technology gets better, I’m not sure how much we’re going to be able to prevent injuries. If one player goes down on your team, that player has a certain skillset that really maximizes the effectiveness of the lineup, then it’d be in the best interest of the team to try to find a “replacement player” who has that skillset and can maybe 50% replicate what the injured player was doing vs. finding joe schmo replacement player who has random skills that you don’t really need.

Seth:  In economic terms once we are being team-specific we’re talking about marginal value rather than some more universal WARP/VORP stat.

Here’s a question: in the NBA would we expect a team starting 5 average players to win 41 games? I think it would be less. Of course, I could just be continuing my long fight against the concept of universal, single-metric value.

Andrew: Seth, Obviously the bench would all have to be average as well for them to be .500. But I think the Celtics gave us the answer last year, they would go 40-42.

Chris: I’ve done a poor job explaining why replacement level in unnecessary, and in fact, has some negative features. For some reason it’s intuitively obvious to me but not to most people, possibly because I’m a mathematician and I’ve spent a decade working with teams. It’s like the Monty Hall problem: https://en.wikipedia.org/wiki/Monty_Hall_problem

PR: My understanding is that the theoretical underpinning of replacement level is essentially non-existent. There’s no such concept in labor economics. It presumes that there’s “freely available talent” when, in fact, there is always a cost.

I don’t know if I’m missing anything in the literature, but if you can expound on the theory — whether economic, mathematical, or any other discipline — I think it would help.

I get the general sense that, even if the theoretical foundation is weak, many of us feel there’s an empirical need for it. And hence we strain to make it work, despite our own misgivings — almost like a necessary evil. It’s as though we’re settling for it because we can’t identify a better alternative. So, Chris, I think our understanding would advance if you can show that there is something better, and we don’t need to rely on a concept that’s on a weak foundation, is hard to define, and is even more unwieldy to explain.

Johannes: I would play for the minimum NBA salary, so I guess there is some kind of “freely available talent”

Jokes aside, I have the impression the biggest disagreement in this replacement level discussion is that we have different starting points. I thought we would just try to look which ‘minimal skill set combination’ players require to have a chance in the NBA – and that’s more or less the end of it. That might be unnecessary, but I cannot see any negative features in it.

PR: Johannes, I’d settle for a fraction of the league minimum. Does that make me “irreplacement” level?

Chris: The notion of “freely available” or “easily available” is generally taken to mean “best easily available”. Assuming you carry a full roster, all sports have a league minimum salary, so “free” in this context is relative to the sunk cost of league minimum.

Seth: I think the “freely available” part is problematic here too. Between college and Europe, how many of the “best non-NBA players” really are freely available for a D-league call up and minimum salary? The fact that fringe NBA rotation guys tend to massacre the D-league when sent down is instructive.

Justin: A better alternative for things like all-time rankings might be one based on expected titles odds. This rewards super high-end players more, like Jordan or Wade.

Seth: Thanks guys, lots of food for thought even if we didn’t really come to much consensus here!