Guidance Document on Model Quality Objectives and Benchmarking

6.Reporting model performance

6.1.The proposed template

In the reporting composite diagrams (e.g. Taylor, Target,…) are favoured. Benchmarking reports are currently available for the hourly NO₂, the 8h daily maximum O₃ and daily PM₁₀ and PM_2.5. There are different reports for the evaluation of hourly and yearly average model results. Below we present details for these two reports.

10. Hourly

The report consists of a Target diagram followed by a summary table.

Target Diagram (Figure )

The MQO as described by equation (6) is used as main indicator. In the normalised Target diagram, the MQO represents the distance between the origin and a given station point. The performance criterion for the target indicator is set to unity regardless of spatial scale and pollutant and it is expected to be fulfilled by at least 90% of the available stations.

In the Target diagram the X and Y axis correspond to the BIAS and and are normalized by the observation uncertainty, U. The is defined as:

(30)

and is related to RMSE and BIAS as follows:

(31)

and to the standard deviation, σ and correlation, R :

C (32)

For each point representing one station on the diagram the abscissa is then bias/2U, the ordinate is /2U and the radius is proportional to RMSE_U. The green area on the Target plot identifies the area of fulfilment of the MQO.

Because is always positive only the right hand side of the diagram would be needed in the Target plot, the negative X axis section can then be used to provide additional information. This information is obtained through relation (32) which is used to further investigate the related error and see whether it is dominated by or by σ. The ratio of two , one obtained assuming a perfect correlation (, numerator), the other assuming a perfect standard deviation
(, denominator) is calculated and serves as basis to decide on which side of the Target diagram the point will be located:

(33)

For ratios larger than 1 the σ error dominates and the station is represented on the right, whereas the reverse applies for values smaller than 1.

The percentage of stations fulfilling the target criterion is indicated in the upper left corner and is meant to be used as the main indicator in the benchmarking procedure. As mentioned above, values higher than 90% must be reached. The uncertainty parameters () used to produce the diagram are listed on the top right-hand side.

In addition to the information mentioned above the proposed Target diagram also provides the following information:

• 
  A distinction between stations according to whether their error is dominated by bias (either negative or positive), by correlation or standard deviation. The sectors where each of these dominates are delineated on the Target diagram by the diagonals in Figure .
  
• 
  Identification of performances for single stations or group of stations by the use of different symbols and colours.
  

Figure Target diagram to visualize the main aspects of model performance

Summary Report (Figure )

The summary statistics table provides additional information on model performances. It is meant as a complementary source of information to the MQO (Target diagram) to identify model strengths and weaknesses. The summary report is structured as follows:

• 
  ROWS 1-2 provide the measured observed yearly means calculated from the hourly values and the number of exceedances for the selected stations. In benchmarking mode, the threshold values for calculating the exceedances are set automatically to 50, 120 and 200 µg/m³ for the daily PM₁₀, the hourly NO₂ and the 8h daily O₃ maximum, respectively. For other variables (PM_2.5, WS…) for which no threshold exists, the value is set to 1000 so that no exceedance will be shown.
  
• 
  ROWS 3-6 provide an overview of the temporal statistics for bias (row 3), correlation (row 4) and standard deviation (row 5) as well as information on the ability of the model to capture the highest range of concentration values (row 6). Each point represents a specific station. Values for these four parameters are estimated via equations (9), (10), (11) and (29) respectively. The points for stations for which the model performance criterion is fulfilled lie within the green and the orange shaded areas. If a point falls within the orange shaded area the error associated with the particular statistical indicator is dominant. Note again that fulfilment of the bias, correlation, standard deviation and high percentile related indicators does not guarantee that the overall MQO based on RMSE is fulfilled.
  
• 
  ROWS 7-8 provide an overview of spatial statistics for correlation and standard deviation. Average values over the selected time period are first calculated for each station and these values are then used to compute the averaged spatial correlation and standard deviation. Fulfilment of the performance criteria (8) and (9) is then checked for these values. As a result only one point representing the spatial correlation of all selected stations is plotted. Colour shading follows the same rules as for rows 3-5.
  

Note that for indicators in rows 3 to 8, values beyond the proposed scale will be represented by the station symbol being plotted in the middle of the dashed zone on the right/left side of the proposed scale

Figure Summary table for statistics

For all indicators, the second column with the coloured circle provides information on the number of stations fulfilling the performance criteria: the circle is coloured green if more than 90% of the stations fulfil the criterion and red if the number of stations is lower than 90%.

11. Yearly average

For the evaluation and reporting of yearly averaged model results a Scatter diagram is used to represent the MQO instead of the Target plot because the CRMSE is zero for yearly averaged results so that the RMSE is equal to the BIAS in this case. The report then consists of a Scatter Diagram followed by the Summary Statistics (Figure )

Scatter Diagram

For yearly averaged results the MQO based on the BIAS (equation 7) is used as the main indicator. In the scatter plot, it is used to represent the distance from the 1:1 line. The MQO is expected to be fulfilled by at least 90% of the available stations. The uncertainty parameters () used to produce the diagram are listed on the top right-hand side

The Scatter diagram also provides information on the performance for single stations or group of stations by presenting these with different symbols and colours.

Summary Report

The summary statistics table provides additional information on the model performance. It is meant as a complementary source of information to the bias-based MQO to identify model strengths and weaknesses. It is structured as follows:

• 
  ROW 1 provides the measured observed means for the selected stations.
  
• 
  ROW 2 provides information on the fulfilment of the bias-based MQO for each selected stations. Note that this information is redundant as it is already available from the scatter diagram but this was kept so that the summary report can be used independently of the scatter diagram.
  
• 
  ROWS 3-4 provide an overview of spatial statistics for correlation and standard deviation. Annual values are used to calculate the spatial correlation and standard deviation. Equations (10) and (11) are used to check fulfilment of the performance criteria. Points that are within the green and the orange shaded area represent those stations where the model performance criterion is fulfilled. For the points that are in the orange shaded area the error associated to the particular statistical indicator is dominant.
  

Note that for the indicators in rows 2 to 4, values beyond the proposed scale will be represented by plotting the station symbol in the middle of the dashed zone on the right/left side of the proposed scale.

The second column with the coloured circle provides information on the number of stations fulfilling the performance criteria: a green circle indicates that more than 90% of the stations fulfil the performance criterion while a red circle is used when this is less than 90% of the stations.

Figure Example of a scatterplot and summary report based on yearly averaged model results.