Reliability is one of the measures of the quality of a forecast. “Reliability” measures how well the forecast probability and the observed frequency for an event or outcome that is being predicted match.
A reliable probabilistic forecast is one in which the forecasted event is observed at the frequency indicated by the probability of the forecast. In other words, when averaging across every forecast that predicts colder than normal conditions three weeks from now with a 60 percent probability, a colder-than-normal week should occur 60 percent of the time. Basically, “reliability” relates to whether the user can rely on the forecast when making decisions.
Calibration plots that assess the agreement between forecast and actual observed probabilities, such as the reliability diagram in Figure 1, are available by variable/region/season/lead time by selecting a grid point from the Skill tab. As shown below, dynamical model ensembles from GEFS and ECMWF typically don't fully represent the variability of true conditions and have other biases that lead to overconfidence.
Figure 1. The above calibration plot shows that the Salient forecast model is better calibrated and more reliable than the NOAA forecast, which is under confident at low probabilities and over confident at high probabilities.
We use our tercile forecast probabilities to build the diagrams, assessing and plotting the reliability of forecast probabilities of being in either the Above Normal or Below Normal category using the Salient climatology.
The diagram is composed of two parts; the top axis plots the forecast probability against the observed frequency. In a perfect reliability diagram, the forecasted probabilities on the x-axis align with the observed frequency on the y-axis along the 45° diagonal, indicated by the dotted line. An overconfident forecast will fall below it.
The bottom axis shows a histogram of the relative number of forecasts at a given forecast probability. A 'climatological' forecast for any given tercile, by definition, is 33% - this is marked on the plot with the dashed grey line. One consequence of calibration is that more forecasts look like a climatological forecast - this is because calibration reduces overconfidence! While it may look less exciting, it will enable much more confident decision making.
<aside> 💡 Note: The reliability diagram is not available for the hourly or daily timescale.
</aside>
https://api.salientpredictions.com/v2/documentation/api/#/Validation/reliability
