Guidance Document on Model Quality Objectives and Benchmarking

5.2.Model performance criteria (MPC) and formulation of the model quality objective (MQO)

Although statistical performance indicators provide insight on model performance in general they do not tell whether model results have reached a sufficient level of quality for a given application, e.g. for policy support. This is the reason why Model Performance Criteria (MPC), defined as the minimum level of quality to be achieved by a model for policy use, are also needed.

To derive performance criteria for the selected statistical indicators we take into account the observation uncertainty. We define as the quadratic mean of the measurement uncertainty:

(5)

where denotes the uncertainty for the i-th observed concentration level, .

With the simple principle of allowing the same margin of tolerance to both model and observations we can define the Model Quality Objective () as:

(6)

With this formulation for the the error between observed and modelled values (numerator) is compared to the absolute measured uncertainty (denominator). Three cases can then be distinguished:

1. 
   ≤ 0.5: the model results are within the range of observation uncertainty (U) and it is not possible to assess whether further improvements to the model are closer to the true value;
   
2. 
   0.5 < ≤ 1: is larger than , but model results could still be closer to the true value than the observation
   
3. 
   1 < : the observation and model uncertainty ranges do not overlap and model and observation are more than 2U apart. Observation is closer to the true value than the model value in this case.
   

This is illustrated in Figure in which examples of these three different cases occur respectively on day 3, day 13 and day 10.

Figure Example PM₁₀ time series (measured and modelled concentrations) for a single station, together with a coloured area representative of the model and observed uncertainty ranges. (from Thunis et al., 2012)

The proposed has the advantage that it allows for introducing more detailed information on observation uncertainty when this becomes available.

For annual average values, the simplifies to:

(7)

5.3. Additional model performance criteria (MPC) for Bias, R and standard deviation

A drawback of the proposed is that errors in either BIAS, σ_M and R are condensed into a single number. These three different statistics are related as follows:

(8)

By considering the ideal cases where , and separate MPC can be derived from (8) for each of these three statistics:

Statistic	Model Performance Criterion
BIAS (,)		(9)
R (,)		(10)
Standard deviation (,)		(11)

One of the main advantages of this approach for deriving separate MPC is that it provides a selection of statistical indicators with a consistent set of performance criteria based on one single input: the observation uncertainty U. The , the main MPC, is based on the RMSE indicator and provides a general overview of the model performance. The associated MPC for correlation, standard deviation and BIAS can then be used to highlight which of the model performance aspects need to be improved. It is important to note that the performance criteria for BIAS, R, and standard deviation represent necessary but not sufficient conditions to ensure that the is fulfilled.

If one of the terms in equation 8 is larger than 0.5 the error type (BIAS, standard deviation or R) associated with this term will be predominant. This allows us to distinguish the following three cases:

Statistic	Model Performance Criterion
BIAS		(12)
R		(13)
Standard deviation		(14)

Finally, the MPC can also be derived for the individual statistics based on (8) for the case where the

MQO ≤ 0.5 and thus the error between modelled and observed values lies within the measurement uncertainty range:

Statistic	Model Performance Criterion
BIAS		(15)
R		(16)
Standard deviation		(17)