Great discussion. I do have a viewpoint on the intersection of football strategy with analytics to try and explain the gap between the two camps. Specifically, when I hear a guest say that something is "clearly the right analytical decision" because the EPA for that situation is slightly positive, they are making two errors in my view.
First, large averages of many data points - such as League averages like EPA - are not necessarily predictive of your specific outcome. The reason is simple...your team is not the average team and is usually facing several other exogenous factors (e.g., strength of your opponent, home versus road, weather, injuries, day of the week, day versus night, grass versus turf etc.) that can be different than those that comprise the average. Put another way, you are facing a unique single use case in front of you with your specific team where the average that combines a lot of factually different outcomes might not be helpful to the specifics for you in that situation.
It's why the Eagles ignore the League averages and go for it on 4th down far more than other teams. Their team is not the average team in that situation. They smartly don't care what the average is.
In my view it is incorrect to apply the average of a group of data points to a single point and automatically presume it to be true. Even with a bell curve with a normal distribution of outcomes, the probability that any one data point will be the same as the average is approximately 50% the same as a random guess. In short, averages are interesting but are too often misleading and not predictive of the present situation facing your team. Too many people in sports overvalue the probative quality of large averages as applied to specific decisions.
Ethan takes the right approach with his kickers example and does not solely rely on the League average. Instead, he also looks at the other relevant factor for THAT game - elevation, the long kicking abilities of the specific kickers etc. This is the right approach and one that the guest acknowledges he didn't think of. I suspect he didn’t think of it because he was too busy looking at averages.
So, I am immediately skeptical when someone claims to know the "analytically right answer" only to learn they are talking about broad averages.
Second, there is a sample size problem. Even if you assume an average is predictive in a given situation, that average requires a larger sample size of outcomes and longer time horizon to be proven correct. Hedge fund managers like the guest are accustomed to using averages because their outcomes involve enormous numbers of moving data points over a long-time horizon to evaluate their performance. They can afford to wait for the average to arrive and prove them right.
Football coaches cannot. Their sample size is small. Their time horizon is remarkably short.
To them, the fact that a particular play is slightly EPA positive at a high level "on average" is not persuasive since each outcome will be scrutinized and inform how the front office and fans evaluate their performance. Football coaches are operating in an environment vastly different from that of hedge fund managers, and their behaviors are quite different as a result.
I would argue they are not being foolish as suggested, but instead quite rational. This is like people who purchase homeowners’ insurance even though their premiums - on average - will far exceed their payouts over their time in the home.
Yet, knowing this does not deter people from purchasing insurance because should the average not bear out for their specific situation, the severity of their loss is extraordinarily high. Thus, they take steps that might at first blush appear to be wrong "on average" but are in fact quite prudent and rational relative to the risks they potentially face for their specific outcome.
Reduction of outsized personal risks - whether for homeowners or football coaches - is not foolish. It's quite smart.
Great discussion. I do have a viewpoint on the intersection of football strategy with analytics to try and explain the gap between the two camps. Specifically, when I hear a guest say that something is "clearly the right analytical decision" because the EPA for that situation is slightly positive, they are making two errors in my view.
First, large averages of many data points - such as League averages like EPA - are not necessarily predictive of your specific outcome. The reason is simple...your team is not the average team and is usually facing several other exogenous factors (e.g., strength of your opponent, home versus road, weather, injuries, day of the week, day versus night, grass versus turf etc.) that can be different than those that comprise the average. Put another way, you are facing a unique single use case in front of you with your specific team where the average that combines a lot of factually different outcomes might not be helpful to the specifics for you in that situation.
It's why the Eagles ignore the League averages and go for it on 4th down far more than other teams. Their team is not the average team in that situation. They smartly don't care what the average is.
In my view it is incorrect to apply the average of a group of data points to a single point and automatically presume it to be true. Even with a bell curve with a normal distribution of outcomes, the probability that any one data point will be the same as the average is approximately 50% the same as a random guess. In short, averages are interesting but are too often misleading and not predictive of the present situation facing your team. Too many people in sports overvalue the probative quality of large averages as applied to specific decisions.
Ethan takes the right approach with his kickers example and does not solely rely on the League average. Instead, he also looks at the other relevant factor for THAT game - elevation, the long kicking abilities of the specific kickers etc. This is the right approach and one that the guest acknowledges he didn't think of. I suspect he didn’t think of it because he was too busy looking at averages.
So, I am immediately skeptical when someone claims to know the "analytically right answer" only to learn they are talking about broad averages.
Second, there is a sample size problem. Even if you assume an average is predictive in a given situation, that average requires a larger sample size of outcomes and longer time horizon to be proven correct. Hedge fund managers like the guest are accustomed to using averages because their outcomes involve enormous numbers of moving data points over a long-time horizon to evaluate their performance. They can afford to wait for the average to arrive and prove them right.
Football coaches cannot. Their sample size is small. Their time horizon is remarkably short.
To them, the fact that a particular play is slightly EPA positive at a high level "on average" is not persuasive since each outcome will be scrutinized and inform how the front office and fans evaluate their performance. Football coaches are operating in an environment vastly different from that of hedge fund managers, and their behaviors are quite different as a result.
I would argue they are not being foolish as suggested, but instead quite rational. This is like people who purchase homeowners’ insurance even though their premiums - on average - will far exceed their payouts over their time in the home.
Yet, knowing this does not deter people from purchasing insurance because should the average not bear out for their specific situation, the severity of their loss is extraordinarily high. Thus, they take steps that might at first blush appear to be wrong "on average" but are in fact quite prudent and rational relative to the risks they potentially face for their specific outcome.
Reduction of outsized personal risks - whether for homeowners or football coaches - is not foolish. It's quite smart.
Why do owners not pay someone like Steve $500k to just give them some of these nuances to play with ideas. Seems super easy and cheap.