**5. Understanding prediction utility: Simple decision models**

In order to begin to understand potential uses of seasonal forecasts it is instructive to study simple cost-loss decision models. Such simple models provide a framework to begin to quantify the potential value of forecasts. Before proceeding, we note that real-world decisions are typically made with far more parameters and subject to greater uncertainty regarding potential costs and payoffs than the simple models studied here.

We first consider a simple binary event, binary decision model in which there are two possible outcomes – the occurrence or non-occurrence of an event - and the user makes a decision to protect, or not protect, against the event. Protection has a cost; failure to protect incurs a loss. The classic example is the decision to carry an umbrella to protect against the possibility of rain. A seasonal timescale example is the decision to apply fertiliser to a crop based on the likelihood of future rainfall over a season. An early study of these issues was made by Anders Angstrom as documented by [24].

A failure to protect with cost C results in a loss L. In this framework it only makes sense to take action given the probability of the event P if P > CL. If it is not, then the expected loss is less than the cost of taking protective action. The combination of the joint distribution of forecasts and observations and the decision-makers cost function determines the potential economic value of the forecasts. (Table 5.)

### **5.1. Adjusting model output: Introducing calibration**

Decisions about the use of GCMs for seasonal climate forecasting are usually based upon measures of model performance over a hind-cast (retrospective forecast) period. A natural

and popular extension of this idea is that GCM-based forecasts should be adjusted by this skill assessment. This motivates 'Model Output Statistics' methods[25] and 'model calibration' and has been widely adopted in medium range weather forecasting [26] and seasonal forecasting [27] [28] [29].

Managing Climate Risk with Seasonal Forecasts 571

Outcome if

Missed profit Minimal loss

Normal profit Moderate loss

Bumper crop Greatest loss

Insufficient Rainfall

**5.2. Extending the binary cost-loss model** 

rationality of forecast users.

potential yield in the event of good rains.

0-20% No fertilizer

20-70% Normal fertilizer

70-100% Maximum fertilizer

Forecast Probability Action Outcome if Sufficient

amount, or a maximum amount to take advantage of expected seasonal rainfall.

application

application

application

skill they do not have value to this particular decision.

In the simple cost-loss model the cost of taking protective action is the same whether the event occurs or does not occur. While this may be true for many economic decisions, when social and political dimensions are considered there is a clear penalty, in terms of confidence in the forecasting system and reduced possibility of action in the future, for false alarms. The binary cost-loss model can be developed further to include such a false alarm or 'cry wolf' effect. Such an extension is effectively an adjustment for the deviation from perfect

The above model can also be extended to more sophisticated decisions based on event probability thresholds, with different actions to be taken at different probability thresholds, depending on the users attitude to risk. We present a hypothetical example of an agriculturalist making a decision about whether to apply additional fertilizer, at a cost, with a potential payoff depending on the probability of expected rainfall being above median. In this example 20% rainfall probability is the threshold at which the cost of applying fertilizer is less than the expected payoff (Table 6). The decision thresholds in Table 6 provide a way of mapping from a given forecast to an action, again in relation to a binary yes/no event. Such tables are dependent on the details of individual enterprises and must be determined with regard to their operating costs and potential losses. The premise for Table 6 is the decision by wheat farmers to apply top-dressed fertiliser in order to benefit from expected rainfall[12], however the numbers selected are arbitrary and shown for illustration. Another management decision that could be studied using this methodology is choice of cultivar, for example to decide whether to plant a drought tolerant strain of wheat or one with a higher

Rainfall

**Table 6.** Example probability thresholds for a decision about whether to apply no fertilizer, a normal

Using the true positive ratio we calculated for our sample rainfall forecasts in Table 3, the farmer would find that the calibrated 'low probability' forecasts from POAMA are not sufficient to justify the 'no fertilizer' action, because the observed frequency of above median rainfall events is above the 20% threshold. In other words, while the forecast have

In order to make rational decisions based on quantifiable costs, losses and probabilities the end user needs the calibrated forecast probabilities, and needs to know what their costs and losses are for each contingency. Given the calibrated forecast probabilities, with reliable confidence intervals, they are in a position to use these probabilities to determine the optimum course of action to follow for their unique cost function. Given information about climatology, a model and its verification, the calibrated model probability p**(**E**|**F**)** is this best estimate, subject to the assumptions made in determining the calibrated probability.

As a simple example of calibration, consider the true positive ratio calculated above. While crude and subject to sampling error, this represents the conditional probability of the event given the model forecast category. The true positive ratio, proposed as the best estimate of the event probability from the POAMA MDB seasonal outlooks discussed above is a conditioning of probabilistic forecasts derived from the GCM ensemble upon probabilities obtained from comparison of the hindcast set with observations. These conditional probabilities are needed for users to make optimal decisions [30]. Skill for coupled models is commonly presented as correlation plots, mean error plots and sometimes more esoteric scores for probabilistic forecasts. While these scores are useful for model diagnostics, and can quantify potential forecast value, it is not obvious how users who need to make decisions based on forecasts should convert these measures into new estimates of probability. We note that some effort has been spent into developing verification measures that do have a direct relationship to economic value such the ROC (Receiver Operating Characteristic) score and the logarithmic score based on the information content of a forecast.

Resolution can be degraded by calibration and it is expected that the application of calibration techniques will involve some trade-off in which resolution is traded for reliability. It is also the case that cross-validation methods used on the application of calibration in order to avoid 'artificial skill' can also result in artificial reduction in skill scores, and thus in the assessment of such methods it can be difficult to disentangle cross validation artefacts from true reduction of model skill due to calibration.

This simple calibration framework can be extended: similar methods can be applied to parametric probability density functions [28]. Below we discuss different calibration methods, but first we turn to more sophisticated decision models.


**Table 5.** Simple binary cost-loss model.
