Brier Score for Traders: Test Whether Your Probabilities Deserve Trust

Learning objective

By the end of this guide, you will be able to turn a trading opinion such as “this breakout has a 70% chance of succeeding” into a testable forecast, calculate its Brier score, compare it with a simple baseline and identify whether your confidence is calibrated.

Most trading journals record entry, exit and profit. They rarely test whether the trader's stated confidence was accurate. That leaves a hidden problem: a trader may be right often but badly overconfident, or cautious but more reliable than the result alone suggests. Probability scoring separates the quality of the forecast from the profit or loss produced by one trade.

The Brier score was introduced by Glenn W. Brier in 1950 for probability forecasts. It is used far beyond markets, but its logic is useful for any trading process that assigns a probability to a clearly defined yes-or-no outcome.

What the Brier score measures

For each forecast, record:

p: your probability from 0 to 1;
y: the outcome, where 1 means the event occurred and 0 means it did not.

Then calculate:

Brier score = average of (p − y)²

A perfect score is 0. For a binary event, a lower score is better. The squared error means a confident wrong call receives a much larger penalty than a cautious wrong call.

Suppose you forecast a breakout success with 90% probability:

if it succeeds, the error is (0.90 − 1)² = 0.01;
if it fails, the error is (0.90 − 0)² = 0.81.

Now compare a 60% forecast:

success produces an error of 0.16;
failure produces an error of 0.36.

The score therefore rewards useful confidence and punishes confidence that is not supported by outcomes.

Step 1: define the event before seeing the result

A probability is meaningless unless the event has an exact deadline and observable outcome. “Gold looks bullish” cannot be scored. A valid event might be:

“December gold will close above today's high within the next five trading sessions.”

Write the instrument, contract or ticker, data source, observation time, condition and deadline. Decide in advance how holidays, missing data and contract rolls will be handled. Do not change the definition after the market moves.

The event also needs to match the decision. A five-day forecast cannot validate a 30-minute setup. If your trade thesis depends on the daily timeframe, score a daily-timeframe event.

Step 2: record a probability, not a confidence adjective

Replace words such as “likely”, “strong” and “high conviction” with a number. A practical journal might limit forecasts to 10% increments until enough evidence exists to justify finer precision.

Do not use 50% as a hiding place. If every forecast is 50%, the score may avoid extreme errors but provides no useful separation between strong and weak setups. Likewise, avoid 0% and 100% unless the event is logically certain; markets can gap and unexpected information can arrive.

Record the probability before entry or before the forecast cutoff. A value added after the move begins contains hindsight.

Step 3: resolve the event consistently

At the deadline, set the outcome to 1 if the event occurred and 0 if it did not. Use the same data source specified in the journal. Do not count “almost” as half a success unless the event was designed from the beginning as a multi-category forecast with a different scoring method.

Keep voided events separate. A data outage, delisting or contract-definition error should not be silently converted into a loss or removed only when the forecast was wrong.

Worked example: five breakout forecasts

Assume a trader makes five forecasts using the same event definition and horizon:

Forecast	Probability p	Outcome y	(p − y)²
1	0.70	1	0.09
2	0.60	0	0.36
3	0.80	1	0.04
4	0.40	0	0.16
5	0.30	1	0.49
Total / mean			1.14 / 5 = 0.228

The Brier score is 0.228.

Three of the five events occurred, so the sample base rate was 60%. A naive baseline that predicted 60% every time would score:

[(0.6−1)² + (0.6−0)² + (0.6−1)² + (0.6−0)² + (0.6−1)²] / 5 = 0.240

The trader's 0.228 score is slightly better than the 0.240 constant baseline. The Brier skill score expresses this as:

Brier skill score = 1 − (forecast score / baseline score)

1 − (0.228 / 0.240) = 0.05, or 5% skill over the baseline.

That is a small advantage, not proof of a durable forecasting edge. Five observations are far too few for a strong conclusion. The example shows the calculation, not a recommended threshold.

Step 4: check calibration

Calibration asks whether stated probabilities match observed frequencies over many comparable forecasts. If 100 forecasts were labelled 70%, a well-calibrated process would produce the event about 70 times. It will not produce exactly 70 in every sample, especially when the sample is small.

Create probability buckets such as 20–29%, 30–39% and so on. For each bucket, compare:

average forecast probability;
actual event frequency;
number of observations.

If your 70% bucket succeeds only 48% of the time across a sufficiently large and stable sample, you are overconfident. If it succeeds 82%, you are underconfident or your probabilities are compressed towards 50%.

Calibration alone is not enough. A trader who predicts the same base rate every time can be calibrated but provide no useful discrimination. A useful model also gives higher probabilities to events that occur and lower probabilities to events that do not. Forecasting literature calls this ability resolution.

Step 5: compare against a fair baseline

A Brier score has no universal “good” level because event frequency matters. Predicting rare events and common events produces different natural scores. Compare like with like.

Possible baselines include:

the historical event frequency measured without future information;
a simple rule already used by the desk;
the market-implied probability, when the instrument genuinely provides a comparable probability;
an earlier model evaluated on the same dates and event definitions.

Do not compare scores from different instruments, regimes or event definitions without adjustment. The Japan Meteorological Agency's forecast-verification guide notes that Brier scores depend on event frequency and recommends a skill score when comparing with a reference forecast.

How to connect probability quality with a trading decision

The Brier score tests probability accuracy. It does not test whether the trade made money. A 40% event can be attractive if the potential gain is much larger than the loss; an 80% event can be unattractive if the price already reflects the outcome or the downside is severe.

Keep two linked records:

Forecast layer: event, probability, outcome, Brier error and calibration bucket.
Decision layer: entry, stop or invalidation, target or exit rule, position size, costs, slippage and realised return.

This separation helps diagnose the process. Accurate probabilities with poor returns may indicate bad payoff selection or execution. Poorly calibrated probabilities with good short-run returns may indicate luck or a payoff structure that temporarily masked forecast errors.

Common mistakes

Scoring vague predictions

“The market will rally” lacks a threshold and deadline. It can be reinterpreted after the event and should not enter the dataset.

Using the trade's profit as the outcome

The scored event must be the event you forecast. A breakout may occur and the trade may still lose because of execution, sizing or exit rules.

Changing the horizon after entry

Extending a five-day forecast because the move has not happened creates look-ahead bias. Resolve it at the original deadline.

Pooling unrelated setups

Combining earnings gaps, daily trend trades and intraday mean reversion can hide where calibration is weak. Analyse comparable forecast families separately.

Trusting a tiny sample

One confident mistake can dominate a short record. Report the observation count beside every score and bucket.

Optimising probabilities on the same history

If you repeatedly adjust probability rules until the historical Brier score looks best, you can overfit. Reserve unseen data or use a walk-forward process.

Ignoring trading costs and payoff

A better Brier score does not automatically mean a better strategy. Costs, payoff asymmetry, liquidity and risk limits still determine whether a forecast is tradable.

Practical journal template

Field	Example
Forecast timestamp	Before the daily close
Instrument	Exact ticker or futures contract
Event	Close above a pre-defined level within five sessions
Probability	0.70
Deadline	Date and market close
Outcome	1 or 0
Brier error	(0.70 − outcome)²
Setup family	Breakout, trend, mean reversion, event risk
Decision record	Entry, size, invalidation, exit, costs

Checklist

Define a binary event with an exact deadline.
Record the probability before the result begins to unfold.
Use one consistent data source to resolve outcomes.
Calculate (probability − outcome)² for every forecast.
Average the errors only across comparable events.
Compare the score with a base-rate or simple-model baseline.
Plot calibration buckets and include their sample sizes.
Keep probability accuracy separate from payoff and execution.
Test changes on unseen data before trusting an improvement.

Sources

The scoring rule comes from Glenn W. Brier's original 1950 paper, Verification of Forecasts Expressed in Terms of Probability. The formula, interpretation, baseline dependence and Brier skill score are also documented in the Japan Meteorological Agency's 2026 forecast-verification appendix.

This tutorial was prepared with AI-assisted research and writing under The Money Man editorial workflow. It is educational information, not personalised investment advice. Examples are hypothetical, and no score or forecasting method guarantees profitable trades.

Brier Score for Traders: Test Whether Your Probabilities Deserve Trust

Executive Summary