Guidance Document on Model Quality Objectives and Benchmarking



Yüklə 189,05 Kb.
səhifə1/10
tarix23.11.2017
ölçüsü189,05 Kb.
#12168
  1   2   3   4   5   6   7   8   9   10

Guidance Document on Model Quality Objectives and Benchmarking

Peter Viaene, Stijn Janssen, Philippe Thunis, Elke Trimpeneers, Joost Wesseling, Alexandra Montero,, Ana Miranda, Jenny Stocker, Helge Rørdam Olesen, Cristina Guerreiro, Gabriela Sousa Santo, Keith Vincent, Claudio Carnevale, Michele Stortini, , Giovanni Bonafè, Enrico Minguzzi and Marco Deserti


January, 2015



Table of contents


1.Introduction 3

2.Benchmarking: a word of caution 4

3.Scope and focus 5

4.Overview of existing literature 6

4.1.Introduction 6

4.2.Literature on how these model performance criteria and model quality objectives are defined. 6

4.3.Literature on the implementation and use of the Delta tool 8

5.Model Quality Objective (MQO) 10

5.1.Statistical performance indicators 10

5.2.Model performance criteria (MPC) and formulation of the model quality objective (MQO) 11

5.3. Additional model performance criteria (MPC) for Bias, R and standard deviation 12

5.4. Observation uncertainty 14

1.1.1.General expression 14

1.1.2.Derivation of parameters for the uncertainty 15

5.5.Open issues 17

1.1.3.Data assimilation 17

1.1.4.Station representativeness 17

1.1.5.Handling changes in observation data uncertainty 17

1.1.6.Performance criteria for high percentile values 17

1.1.7.Data availability 18

1.1.9.Application of the procedure to other parameters 19

6.Reporting model performance 21

6.1.The proposed template 21

1.1.10.Hourly 21



1.1.11.Yearly average 24

6.2.Open issues 25

7.Examples of good practice 27

7.1.CERC experience 27

7.2.Applying the DELTA tool v4.0 to NINFA Air Quality System 31

7.3.JOAQUIN Model comparison PM10 NW Europe 35

7.4.UAVR experience with DELTA 39

7.5.TCAM evaluation with DELTA tool 42

7.6.UK feedback Ricardo AEA 44

8.References 46

8.1.Peer reviewed articles: 46

8.2.Reports/ working documents / user manuals: 46

8.3.Other documents/ e-mail: 47


1.Introduction


The objective of this guidance document is twofold:

  1. to summarize the contents of different documents that have been produced in the context of FAIRMODE with the aim to define a methodology to evaluate air quality model performance for policy applications, especially related to the Ambient Air Quality Directive 2008/50/EC (AQD). Air quality models can have various applications (forecast, assessment, scenario analysis, …). The focus of this document is only on the use of air quality models for the assessment of air quality.

  2. to present user feedback based on a number of examples in which this methodology has been applied.



2.Benchmarking: a word of caution


UNESCO1 defines benchmarking as follows:

  • a standardized method for collecting and reporting model outputs in a way that enables relevant comparisons, with a view to establishing good practice, diagnosing problems in performance, and identifying areas of strength;

  • a self-improvement system allowing model validation and model intercomparison regarding some aspects of performance, with a view to finding ways to improve current performance;

  • a diagnostic mechanism for the evaluation of model results that can aid the judgment of models quality and promote good practices.

When we talk about benchmarking, it is normally implicitly assumed that the best model is one which produces results within the observation uncertainty of monitoring results. In many cases, this is a reasonable assumption. However, it is important to recognize that this is not always the case, so you should proceed with caution when you interpret benchmarking results. Here are two examples in which blind faith in benchmarking statistics would be misplaced:

  • Emission inventories are seldom perfect. If not all emission sources are included in the inventory used by the model then a perfect model should not match the observations, but have a bias. In that case seemingly good results would be the result of compensating errors.

  • If the geographical pattern of concentrations is very patchy – such as in urban hot spots – monitoring stations are only representative of a very limited area. It can be a major challenge – and possibly an unreasonable challenge – for a model to be asked to reproduce such monitoring results.

In general, in the EU member states there are different situations which pose different challenges to modelling including among others the availability of input data, emission patterns and the complexity of atmospheric flows due to topography.

The implication of all the above remarks is that if you wish to avoid drawing unwarranted conclusions from benchmarking results, then it is not sufficient to inspect benchmarking results. You should acquire some background information on the underlying data and consider the challenges they represent.



Good benchmarking results are therefore not a guarantee that everything is perfect. Poor benchmarking results should be followed by a closer analysis of their causes. This should include examination of the underlying data and some exploratory data analysis.

3.Scope and focus


The focus of this Guidance Document and the work performed within FAIRMODE is on producing a model quality objective (MQO) and model performance criteria (MPC) for different statistical indicators related to a given air quality model application for air quality assessment in the frame of the AQD. These statistical indicators are produced by comparing air quality model results and measurements at monitoring sites. This has the following consequences:

  1. data availability

A minimum data availability is required for statistics to be produced at a given station. Presently the requested percentage of available data over the selected period is 75%. Statistics for a single station are only produced when data availability of paired modelled and observed data is at least of 75% for the time period considered. When time averaging operations are performed the same availability criteria of 75% applies. For example, daily averages will be performed only if data for 18 hours are available. Similarly, an 8 hour average value for calculating the O3 daily maximum 8-hour means is only calculated for the 8 hour periods in which 6 hourly values are available. In open issues (Error: Reference source not found) the choice of the data availability criterion is further elaborated.


  1. Model performance criteria

The model performance criteria (MPC) are in this document only defined for pollutants and temporal scales that are relevant to the AQD. Currently only O3, NO2, PM10 and PM2.5 data covering an entire calendar year are considered.


  1. MPC fulfilment criteria

According to the Data Quality Objectives in Annex I of the AQD the uncertainty for modelling is defined as the maximum deviation of the measured and calculated concentration levels for 90 % of individual monitoring points, over the period considered, by the limit value (or target value in the case of ozone), without taking into account the timing of the events. For benchmarking we also need to select a minimum value for the number of stations in which the model performance criterion has to be fulfilled and propose to also set this number to
90 %. This means that the model performance criteria must be fulfilled for at least 90% of the available stations. As the number of stations is an integer value this means that sometimes more than 90% of the available stations will need to fulfil the criteria and for example in the specific case that there are less than 10 observation stations, all stations will need to fulfil the criteria. In the open issues (Error: Reference source not found) an alternative interpretation of the fulfilment criterion is presented.


Yüklə 189,05 Kb.

Dostları ilə paylaş:
  1   2   3   4   5   6   7   8   9   10




Verilənlər bazası müəlliflik hüququ ilə müdafiə olunur ©genderi.org 2024
rəhbərliyinə müraciət

    Ana səhifə