Are the NFL Stat Geeks Actually Arrogant?

Football's Confirmation Bias Problem

In response to my column on quarterback rankings and NFL culture largely being anti data, our friend Freddie deBoer left a comment:

As I’ve said for a million billion years, sports statheads desperately need to take some graduate philosophy of science classes. It’s crazy that actual research physicists are more humble about their truth claims than, like, Seth Walder. Every quantitative indicator is subject to its own corruptions, and how you choose which metrics to elevate is always a value-laden process. Which isn’t an argument against statistical analysis, but it is an argument in favor of epistemological modesty, which I don’t see in someone like Walder or Bill Barnwell.

And Kacsmar is a funny reference here because he’s notoriously subject to the dictates of his own resentments; he has insisted for years that Josh Allen is not a good quarterback, and has only been forced to by events in the last couple of years. Why? Because he was a very loud Allen skeptic in his draft process, and he’s mad that Allen proved him wrong. Lots and lots of stat guys are subject to those corrupting influences.

For all I know Freddie is right about Kacsmar. I don’t know much about him beyond agreeing with the gist of a tweet about quarterbacks. I’m better positioned to discuss Freddie’s first point on the insufficiency of statistical analysis in sports and the alleged arrogance of statheads. When it comes to NFL specifically, I’m more so Team Geek and less so Team Vibes. I hate admitting that about myself. It would be cooler to be on the Vibes side. Here’s my thinking on the topic…

Who could disagree with the point that, in this messy human world of games,
”every quantitative indicator is subject to its own corruptions”? It’s an underrated paradox of sports that they’re a man-made construct men can’t ever fully understand. We created the rules, conditions, and packed the test subjects into a Skinner box we call an arena. At this point, every iota of the action is filmed from multiple angles and there’s tracking information on the speed of participants at all times. Yet, somehow, we still don’t completely know why one team won and one team lost. We understand that the aforementioned Josh Allen is an upper end quarterback, and possibly the best, but we’ve no clear, objective way of determining how much better he is than the 5th-most capable guy at his position. Space programs have landed many crafts on Mars, but none of our scientists can prove exactly why Patrick Mahomes has statistically declined in recent years.

So yes, it’s good to approach data-based value judgments in this area with a degree of humility. Covering the Golden State Warriors, initially a horribly-run team that I could reflexively sneer at for making anti-math choices, turned into a humbling experience. At the event level I learned what little I actually knew. I grew to understand that coaches make a lot of sensible, deeply-considered choices that might look insane to the savvy outsider.

I can recall how frustrated Warriors coach Steve Kerr would get about his emails from the Bay Area’s many intelligent fans. At one point Kerr was getting besieged with demands for more JaVale McGee playing time, because McGee’s rim-running was such an effective weapon on a Steph Curry-led pick and roll. The stats loudly called for more McGee. What the fans weren’t considering, as Steve lamented, was that JaVale had asthma. Right or wrong, Kerr wanted to avoid falling off an invisible productivity cliff, and the minutes were designed to maintain sustained potency. The coach’s choice made sense, but he didn’t feel comfortable openly discussing a player’s limitations on the basis of a medical condition.

So yes, there’s a big risk of overinterpretation when analyzing sports data. This is especially true in the NFL, our most violently chaotic sport. Sample sizes are small, schemes vary greatly, and a large amount of players are performing while injured. Who’s to know what’s actually happening?

I think my issue with NFL culture, team and media, is that the obvious confounds have led to a reflective rejection of information. Or as I said in the comments:

What I’d say is that the difficulty of finding signal through all the statistical football noise has led to a situation where the sport’s culture doesn’t even bother looking.

Maybe NFL culture seems bizarre to me because of my aforementioned NBA experience. This annual quarterbacks list situation doesn’t really happen in basketball. Nikola Jokić, Shai Gilgeous-Alexander and Victor Wembanyama finish at the top of rankings in accordance with their obvious recent statistical dominance. There’s no conflicting vibes-based assessment. There’s no vague “film watcher” reinterpretation.

So why do I believe that chaotic, violent football shouldn’t be so different? Why do I assume that the most obsessive sports culture almost happens to run on unfalsifiable and thus untrustworthy confirmation bias?

My defense, in a word, is “baseball.” The Moneyball era revealed that the baseball professionals, for all their granular knowledge, had an inability to separate statistical signal from noise. The dumbed down version of this story is that baseball men assumed batting average meant more than it did and that on base percentage meant less than it did. Scouts believed in the ineffable quality of a “five-tool player” versus prioritizing what the athletes actually produced. So why would the NFL be any different?

The rebuttal is that static baseball gives us trustworthy outputs whereas garbled football is rife with confounding variables. We can’t simply trust the stats in the NFL. That makes sense, for reasons I’ve noted, but we’re still left with a situation where the data-averse sport probably misunderstands itself, and arguably more so. If you’re culturally invested in the idea that advanced numbers are confounded, you’re not going to allow for an objective correction of preconceived notions.

Football’s chaos means we often can’t know the “why,” but I’d argue that this understanding bleeds into a rejection of the “what.” Thanks in part to decades of work from Aaron Schatz and Football Outsiders, we’ve got a lot of “what” to work with.

We might not be able to exactly determine who gets credit, but numbers demonstrate which team’s passing attack actually performed best on average. Those numbers are largely correct, even if they contradict old standbys like “TD/INT ratio.” We can demonstrate that sacks kill drives, for obvious reasons, even if NFL media rarely incorporates this aspect when discussing a quarterback’s standard statistical profile. We can understand that, even if the yardage is equal, a running back who gets consistent gains outperforms a “boom-and-bust” runner who often gets tackled for a loss.

The nerds grinded and made these discoveries while many professionals were relying on surface-level information. What’s funny about football is the surface-level stuff still gets broadly cited. Long ago, NBA media moved away from “points per game” as a proxy for team defense, due to pace as a confound, but NFL media continues to embrace the old flawed stuff. We can’t explain this tendency with “football is complicated, so you can’t trust stats.” At a certain point, it’s just prolonged sports Ludditism.

Some of these nerds got invited to the broader football media conversation, but subjectively, I see their, “here’s what’s objective” influence waning. I have a theory as to why, which could be the basis for another article. As we trend towards post-literacy, I believe that shift carries with it a post-data externality. One of the reasons John Hollinger had such a big influence on 2010s NBA discourse is that he’d write detailed, evidence-based columns on a large platform. Many readers, probably numbering in the hundreds of thousands, sat down and actually digested that information.

Nowadays, and this is especially true for our most popular sport, the bulk of fan consumption happens via video and audio. The medium might not be the message, but medium has an impact on which messages become popular. There are certainly statheads doing deep dives on YouTube, but I’m skeptical that the visual medium lends itself to popularizing objective analysis.

At some point, I was supposed to address the alleged “arrogance” of NFL statheads. I’ve not read everything by Bill Barnwell, but based on what I’ve consumed, let’s just say I can understand the charge. I’d also counter by saying the geeks might have good reason to scoff at football’s conventional wisdom. This sport is rife with blindspots. The failure rate on quarterbacks drafted in the first round is over 60 percent.

On the media side, for two years, I was loudly told by NFL pundits that the 49ers had an imminently crippling QB contract bomb. When the deal was finally signed I learned that, thanks to a forseeable NFL CBA quirk, the team’s salary cap would be impacted on a mere single-digit percentage level. A couple nerds saw this coming, but the near-entirety of that sport’s media apparatus was blind to a bit of math. No offense, but these sorts of oversights don’t happen in the league I used to cover. It’s a football thing, arguably because in the NFL, there’s a lot of, “everybody is saying this, so it must be true.”

What I’m saying is, our most complicated sport could stand to have more geek influence on the discourse. The nerds might be arrogant, but “let’s not look into whether we’re wrong” is the most arrogant position of all.