How Accurate Are Your Predictions? A Calibration Guide

What Is a Good Forecast?

Most people think of forecasting as "either you get it right or you don't." But professional forecasters think differently. What matters to them isn't the outcome of a single forecast, but how consistent you are across hundreds of forecasts.

This is called "calibration."

What Does Calibration Mean?

Simply put: when you say "I'd give this a 70% probability," does it actually come true about 70% of the time?

A perfectly calibrated forecaster looks like this:

• 90% of the things they say "I'm 90% sure about" turn out to be correct
• 60% of the things they say "I'm 60% sure about" turn out to be correct
• 30% of the things they say "I'm 30% sure about" turn out to be correct

The problem with most people is overconfidence. When they say "I'm 90% sure," only about 70% of those things actually turn out to be true. That's poor calibration.

Why Does It Matter?

A well-calibrated forecaster measures uncertainty accurately. This skill comes in handy everywhere, from professional life to personal decisions.

If a manager says "we have a 90% chance of finishing this project on time" and they can actually deliver at that 90% rate, they can plan resources properly. But if they're overconfident, they'll constantly face delayed projects and crises.

The research Philip Tetlock describes in his book "Superforecasting" makes this clear: the best forecasters aren't the smartest ones, they're the best-calibrated ones.

How to View Your Calibration on Pusulam

Premium members have a "Statistics" tab on their profile page. Your calibration chart is displayed there. The dashed line on the chart represents "perfect calibration." The closer your dots are to that line, the better calibrated you are.

You'll also see your Brier score. This is a number between 0 and 1. The closer to 0, the better. Generally, 0.1-0.2 is considered "very good" and 0.2-0.3 is considered "good."

5 Tips for Better Calibration

1. Learn base rates. Before estimating the probability of something, research how often similar events have happened in the past. For the question "will a civil war break out in this country?", look at how many countries have experienced civil wars in the last 50 years.

2. Watch out for overconfidence. Before saying "I'm 95% sure," stop and think: is it really 19 out of 20? Most people who say 95% are actually closer to 75%.

3. Consider the opposite. If you believe something will happen, ask yourself what conditions would need to be true for it not to happen. This exercise gives you perspective.

4. Update. As new information comes in, update your forecast. Don't cling to your first instinct. Bayesian thinking requires this: every new piece of evidence updates the probability.

5. Keep a record. On Pusulam, every forecast is already saved for you. But in everyday life too, make predictions like "I think there's an X% chance of Y happening" and then check the outcome. Over time, you'll develop an instinct for calibration.

The Community Effect

On Pusulam, you're not forecasting alone. You're comparing yourself to the community's forecast. This is an incredibly valuable feedback mechanism.

If the community says 30% and you say 80%, either you know something the community doesn't (great, that's the advantage of getting in early), or you've fallen into your own bias. That awareness is what improves calibration.

Check your calibration