It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert).
A calibrated forecast means that if you say there is a 20% chance of rain, then it actually rains 20% of the time.
It’s a desired feature, but not the only one (e.g. you could be calibrated by stating: Chick-fil-A is open every day except Monday, but your forecast will always be wrong on Sunday and Monday).
So if
1. you are Bayesian (you state your beliefs)
2. and coherent (the laws of probability apply, so e.g. if P(A) = 0.4, then P(not A) cannot be anything other than 0.6):
and you are predicting something (e.g rain tomorrow), then
if you believe it will rain with probability 0.7 but you are 80% sure of your belief, you won’t say 0.7; you will say something else: 0.8 × 0.7 + 0.2 × something_you_believe = 0.58.
Coherence forces you to collapse your uncertainty into your probability at each forecast.
This theorem shows that, over many forecasts, in your belief system you are certain to be producing a calibrated forecast: your current beliefs assign probability 1 to the proposition that your future forecasts will be calibrated.
But that can’t be, which is the paradox. So Bayesianism is too strong compared to how scientists reason, because scientists always think their model can have an error.
I didn't see a real Bayesian point of view in that article.
A Bayesian does not give you a probability estimate they give you a probability distribution for the probability!
Like in Star Trek Spock is always saying something like "Captain, we have a 15.31% chance of surviving this mission" which is a ridiculous example of precision without accuracy. [1]
If you observe a coin flipped 100 times and it came up heads 65 times it is not a crazy point estimate to say it has a 65% chance of coming up heads but this is just one sample and if you did it another time maybe it comes up 61 or 68 times. You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36) since that has the "error bars" built in, can be updated if you get more samples, etc.
[1] ... and you know he underestimates survival probabilities the same way Scotty overestimates how long it will take to fix the engines
We've got a coin here! Let's ask a frequentist and a bayesian what they think about the probability that flipping it would make it land heads up?
Frequentist: How would I know? I haven't seen it flipped once, nor do I know how you've selected it from all the other coins that exist.
Bayesian: It's 50%!
Then we flip the coin 10,000 times and observes that it landed up heads exactly 5,000 times.
Frequentist: Huh, that's weird. You see, if the probability was 1/2, the expected deviation from the mean in this case would've been 50, so I'd expected to see either about 4'950, or 5'050 heads... still, MLE provides the answer of 1/2 bu-u-ut...
Bayesian: It's 50%!
This two-strawmen thought experiment clearly demonstrates the superiority of the Bayesian approach in learning useful information from the real-world observations.
It's really kinda shame that both personal certainties and physical probabilities follow the same algebraical rules while having entirely different nature; most of the time, you are not very interested in how much is someone is certain of some outcome, you're much more interested in the actual outcome or at least the actual probability of that outcome. Granted, most of the time you can only readily access someone's certainty of an outcome, but this is just a proxy for the quantity you're actually interested in knowing.
> You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36)
s/probability/your personal certainty/g. The probability of the coin landing heads up is what it is, and it usually doesn't depend on any of your knowledge.
The \pi_i in the paper is not the estimate of a latent parameter. It is the predictive probability of the event, which is a single number by necessity in a binary challenge. It's the integration of a distribution function which can contains very complex distributions: in my example something_you_believe can be a probability distribution.
So everything in the paper is distribution and when you forecast for a binary event, you give a number which is the expectation of that distribution. This is a probabilistic forecast.
If you were to give a probabilistic forecast for a continuous quantity, then yes you would give in a distribution, as in section 4.2
Spock had not realized that James Kirk emitted a psionic reality distortion field through higher dimensional "luck", if he had he would have been a bit more relaxed.
> if you believe it will rain with probability 0.7 but you are 80% sure of your belief
What does this even mean...?
If I believe it will rain with probability 0.7, that 0.7 figure should already be taking into account the sum total of all of my uncertainty over all of my beliefs: my trust in the weather forecast, my past experience with the local area in this season, my certainty that the earth will continue to exist tomorrow.
Bayesians of course accept that their models can have errors, and if they're doing a good job they'll factor all of the most influential ones into the probability calculation itself.
You are making the same point that I did, if you read the rest. 0.7 at 80% is an intermediate step in formulating a forecast, and the point is to show that you cannot stop there and have to include everything in your number, like you have said. In your words, the intermediate steps are what you list (trust in weather forecasts, past experience,…).
No, what I'm saying is that 0.7 is the ending point of the calculation, and there is no 80% associated with it. The 0.7 number already factors in my uncertainty in all of my knowledge about whether it will rain tomorrow. Saying that you're 80% sure of your 0.7 probability forecast throws a TypeErrorException.
Something Bayesian. Despite my best effort I just do not get Bayesian probability, it more or less just does not make sense to me. Can you convince me otherwise? What is your best example of something with a probability that can not be analyzed in terms of frequencies or other proportions? And your Bayesian account of it must make sense, I am 90 % certain that P != NP and that is why I would take bets based on those odds does not cut it.
Someone walks out of a magic store holding a coin.
They propose a bet. If they flip it 100 times and the proportion of heads is within [0.4, 0.6], you win $100. If it's not, you pay $100. Do you take that bet?
Explanation: absent the magic store scenario, a `rational' person would take the bet. Your prior belief is that most coins are roughly unbiased. Given that they walked out of a magic store, you now have additional information. Maybe the coin is a trick coin. In that case, your belief that the coin is unbiased should be weaker, even if you don't know which direction the coin is biased in.
This illustrates two things: one, additional information (magic store) can update your beliefs. Two, a strong prior and a weak prior, in this case about the coin's bias, can lead to materially different decisions.
The bet does not really matter, the central question is whether they have a fair coin or if they are trying to me in some way. Even without the magic store, I would be very suspicious of anyone approaching random people with an offer like that.
So I would certainly consider it likely, that they are trying to trick me. But the probability I would assign to this, would still be rooted in some frequency, somewhere under the hood I would try to estimate the possible situations leading to such an offer and in which fraction of them I will be tricked.
If I am doing a good job with that, then repeatedly being in this situation should result in me getting tricked with the probability I cooked up. If I am bad at figuring out the possible states and their probabilities, then I they will not match.
One could look at the likelihood of spontaneous coal fires and their effects, gather evidence about the activities surrounding the event, essentially trying to narrow down the set of possible states that would lead to the incident and then see what proportion of those states cause the sinking by fire, military action, or something else. But then attaching a number to that seems a dauting task.
Attaching a number to it isn't so daunting, if you view that number as a reflection of your own subjective certainty as to the truth of the proposition. There's no other "correct" number to shoot for; in reality it was either an accident or it wasn't; Nature knows which proposition is true but we don't. Our mission is only to find the figure that minimizes the logarithm of our error, if the truth were at some point to be revealed to us. The more evidence you gather and the more work you put in will get you a better minimum, but there's no reason to be afraid of putting even a rough quantitative number on it before doing that legwork; you'd just be doing it from a state of higher uncertainty, and your stated probability would have to be farther from either confident extreme (0% or 100%) accordingly.
Any one off event is an example. But I assume you know that, so can you clarify what you mean by "a probability that can not be analyzed in terms of frequencies or other proportions"?
Let us start with coin flips. You repeatedly flip a coin and the number of heads will come out to be about half the number of trials.
Where does that come from? It is not some intrinsic property of the coin, it comes from varying initial conditions. If you had enough precision when controlling your hand movements, you could in principle force an outcome with high probability.
But assuming you can not or at least do not do that, there is a certain set of initial states, some will lead to heads, some to tails, and each toss will start from a randomly selected initial state. So given my ignorance of the exact initial state, the coin will land heads with a probability equal to the number of initial states leading to heads divided by the number of initial states compatible with my observations of the initial state. [1]
Repeatedly tossing a coin will sample the set of initial states and the result will match the proportion of the number of states. At least as long as I am not wrong about the set of initial states.
The same applies to something like an election. I have imperfect knowledge about the state of the world but there is a set of states compatible with my knowledge about the world and certain subsets of them will lead to certain candidates to win.
[1] Maybe adjusted by some probability distribution over the initial states if they are not equally likely to be picked.
Must ensure that this quote is present whenever Cromwell's Rule is mentioned - "I beseech you, in the bowels of Christ, think it possible that you may be mistaken."
It took me a while, but here’s what I gather about this (I’m pretty sure it’s correct, but I’m not an expert).
A calibrated forecast means that if you say there is a 20% chance of rain, then it actually rains 20% of the time. It’s a desired feature, but not the only one (e.g. you could be calibrated by stating: Chick-fil-A is open every day except Monday, but your forecast will always be wrong on Sunday and Monday).
So if
1. you are Bayesian (you state your beliefs)
2. and coherent (the laws of probability apply, so e.g. if P(A) = 0.4, then P(not A) cannot be anything other than 0.6):
and you are predicting something (e.g rain tomorrow), then if you believe it will rain with probability 0.7 but you are 80% sure of your belief, you won’t say 0.7; you will say something else: 0.8 × 0.7 + 0.2 × something_you_believe = 0.58. Coherence forces you to collapse your uncertainty into your probability at each forecast.
This theorem shows that, over many forecasts, in your belief system you are certain to be producing a calibrated forecast: your current beliefs assign probability 1 to the proposition that your future forecasts will be calibrated.
But that can’t be, which is the paradox. So Bayesianism is too strong compared to how scientists reason, because scientists always think their model can have an error.
I didn't see a real Bayesian point of view in that article.
A Bayesian does not give you a probability estimate they give you a probability distribution for the probability!
Like in Star Trek Spock is always saying something like "Captain, we have a 15.31% chance of surviving this mission" which is a ridiculous example of precision without accuracy. [1]
If you observe a coin flipped 100 times and it came up heads 65 times it is not a crazy point estimate to say it has a 65% chance of coming up heads but this is just one sample and if you did it another time maybe it comes up 61 or 68 times. You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36) since that has the "error bars" built in, can be updated if you get more samples, etc.
[1] ... and you know he underestimates survival probabilities the same way Scotty overestimates how long it will take to fix the engines
We've got a coin here! Let's ask a frequentist and a bayesian what they think about the probability that flipping it would make it land heads up?
Frequentist: How would I know? I haven't seen it flipped once, nor do I know how you've selected it from all the other coins that exist.
Bayesian: It's 50%!
Then we flip the coin 10,000 times and observes that it landed up heads exactly 5,000 times.
Frequentist: Huh, that's weird. You see, if the probability was 1/2, the expected deviation from the mean in this case would've been 50, so I'd expected to see either about 4'950, or 5'050 heads... still, MLE provides the answer of 1/2 bu-u-ut...
Bayesian: It's 50%!
This two-strawmen thought experiment clearly demonstrates the superiority of the Bayesian approach in learning useful information from the real-world observations.
It's really kinda shame that both personal certainties and physical probabilities follow the same algebraical rules while having entirely different nature; most of the time, you are not very interested in how much is someone is certain of some outcome, you're much more interested in the actual outcome or at least the actual probability of that outcome. Granted, most of the time you can only readily access someone's certainty of an outcome, but this is just a proxy for the quantity you're actually interested in knowing.
> You are better saying that the probability distribution of the probability is β(65,35) or maybe β(65.5,35.5) or β(66,36)
s/probability/your personal certainty/g. The probability of the coin landing heads up is what it is, and it usually doesn't depend on any of your knowledge.
The \pi_i in the paper is not the estimate of a latent parameter. It is the predictive probability of the event, which is a single number by necessity in a binary challenge. It's the integration of a distribution function which can contains very complex distributions: in my example something_you_believe can be a probability distribution.
So everything in the paper is distribution and when you forecast for a binary event, you give a number which is the expectation of that distribution. This is a probabilistic forecast.
If you were to give a probabilistic forecast for a continuous quantity, then yes you would give in a distribution, as in section 4.2
>and you know he underestimates survival probabilities the same way Scotty overestimates how long it will take to fix the engines
NOT SPOCK!!!
Was Spock poorly calibrated?
If he was well calibrated there is no way they would have made it through 79 episodes!
Spock had not realized that James Kirk emitted a psionic reality distortion field through higher dimensional "luck", if he had he would have been a bit more relaxed.
> if you believe it will rain with probability 0.7 but you are 80% sure of your belief
What does this even mean...?
If I believe it will rain with probability 0.7, that 0.7 figure should already be taking into account the sum total of all of my uncertainty over all of my beliefs: my trust in the weather forecast, my past experience with the local area in this season, my certainty that the earth will continue to exist tomorrow.
Bayesians of course accept that their models can have errors, and if they're doing a good job they'll factor all of the most influential ones into the probability calculation itself.
> What does this even mean...?
You are making the same point that I did, if you read the rest. 0.7 at 80% is an intermediate step in formulating a forecast, and the point is to show that you cannot stop there and have to include everything in your number, like you have said. In your words, the intermediate steps are what you list (trust in weather forecasts, past experience,…).
No, what I'm saying is that 0.7 is the ending point of the calculation, and there is no 80% associated with it. The 0.7 number already factors in my uncertainty in all of my knowledge about whether it will rain tomorrow. Saying that you're 80% sure of your 0.7 probability forecast throws a TypeErrorException.
Something Bayesian. Despite my best effort I just do not get Bayesian probability, it more or less just does not make sense to me. Can you convince me otherwise? What is your best example of something with a probability that can not be analyzed in terms of frequencies or other proportions? And your Bayesian account of it must make sense, I am 90 % certain that P != NP and that is why I would take bets based on those odds does not cut it.
Someone walks out of a magic store holding a coin.
They propose a bet. If they flip it 100 times and the proportion of heads is within [0.4, 0.6], you win $100. If it's not, you pay $100. Do you take that bet?
Explanation: absent the magic store scenario, a `rational' person would take the bet. Your prior belief is that most coins are roughly unbiased. Given that they walked out of a magic store, you now have additional information. Maybe the coin is a trick coin. In that case, your belief that the coin is unbiased should be weaker, even if you don't know which direction the coin is biased in.
This illustrates two things: one, additional information (magic store) can update your beliefs. Two, a strong prior and a weak prior, in this case about the coin's bias, can lead to materially different decisions.
The bet does not really matter, the central question is whether they have a fair coin or if they are trying to me in some way. Even without the magic store, I would be very suspicious of anyone approaching random people with an offer like that.
So I would certainly consider it likely, that they are trying to trick me. But the probability I would assign to this, would still be rooted in some frequency, somewhere under the hood I would try to estimate the possible situations leading to such an offer and in which fraction of them I will be tricked.
If I am doing a good job with that, then repeatedly being in this situation should result in me getting tricked with the probability I cooked up. If I am bad at figuring out the possible states and their probabilities, then I they will not match.
What's the probability that the sinking of the USS Maine in 1898 was accidental?
One could look at the likelihood of spontaneous coal fires and their effects, gather evidence about the activities surrounding the event, essentially trying to narrow down the set of possible states that would lead to the incident and then see what proportion of those states cause the sinking by fire, military action, or something else. But then attaching a number to that seems a dauting task.
Attaching a number to it isn't so daunting, if you view that number as a reflection of your own subjective certainty as to the truth of the proposition. There's no other "correct" number to shoot for; in reality it was either an accident or it wasn't; Nature knows which proposition is true but we don't. Our mission is only to find the figure that minimizes the logarithm of our error, if the truth were at some point to be revealed to us. The more evidence you gather and the more work you put in will get you a better minimum, but there's no reason to be afraid of putting even a rough quantitative number on it before doing that legwork; you'd just be doing it from a state of higher uncertainty, and your stated probability would have to be farther from either confident extreme (0% or 100%) accordingly.
Any one off event is an example. But I assume you know that, so can you clarify what you mean by "a probability that can not be analyzed in terms of frequencies or other proportions"?
Let us start with coin flips. You repeatedly flip a coin and the number of heads will come out to be about half the number of trials.
Where does that come from? It is not some intrinsic property of the coin, it comes from varying initial conditions. If you had enough precision when controlling your hand movements, you could in principle force an outcome with high probability.
But assuming you can not or at least do not do that, there is a certain set of initial states, some will lead to heads, some to tails, and each toss will start from a randomly selected initial state. So given my ignorance of the exact initial state, the coin will land heads with a probability equal to the number of initial states leading to heads divided by the number of initial states compatible with my observations of the initial state. [1]
Repeatedly tossing a coin will sample the set of initial states and the result will match the proportion of the number of states. At least as long as I am not wrong about the set of initial states.
The same applies to something like an election. I have imperfect knowledge about the state of the world but there is a set of states compatible with my knowledge about the world and certain subsets of them will lead to certain candidates to win.
[1] Maybe adjusted by some probability distribution over the initial states if they are not equally likely to be picked.
Title a reference to The Well-Tempered Clavier?
(1982)
Must ensure that this quote is present whenever Cromwell's Rule is mentioned - "I beseech you, in the bowels of Christ, think it possible that you may be mistaken."
This is a classic!