Presentation Outline Data Centre Status

Presentation Outline

• Data Centre Status

• Tools

• Local end-user tools
• WWW search tools

• Commissioning Phase Activities

Data flow

• Signed protocol required for access to tools, docs and files

• ECMWF data requires additional protocol

Data Centre Status (1/3): Hardware

• The machines have been swapped between production and spare servers

• The hardware has not been changed

Data Centre Status (2/3): Software

• Web-server (nadir) runs Linux Redhat 7.2 with up2date ~weekly (6.2 at 2nd rehearsal)

• File server (zardoz) runs Solaris 7, as before

• Nusphere MySQL

• Apache 1.3.22 (1.3.14)
• MySQL 3.23.44 (3.23.32)
• PHP 4.0.6 (4.0.4pl1)

• IDL 5.4, as before

• HDF 4.1r3, as before

• No re-boot on nadir since April 2002

• Uptime on zardoz since April 2002 is approximately 99,98% (2 half-hour maintenance + re-boot)

Data Centre Status (3/3): Metadata

• Metadata Guidelines updated by metadata board on 24 April 2002

• During commissioning phase, updates of various metadata table entries, without affecting structure:

• Project-related information
• Actual descriptors

• Guidelines still applicable, but more clarifications to be added.

New user tools

• Major re-write of the ASC2HDF file generation and checking tool around launch

• 6 smaller upgrades of the ASC2HDF package after launch

• Entire ASC2HDF source code (FORTRAN) published to users. HDF output routines may be combined with user’s own input and processing modules

• IDL-tool to generate HDF data files (without error checking): IDL2HDF

• IDL tool for reading and plotting HDF data files (HDF2IDL), to be downloaded and used locally

• IDL subroutine to read HDF Cal/Val data, for integration into user-developed software (nilu_hdf_read_user.pro)

New web functionality - ECMWF

New web functionality - search and plot

• Use of defaults

• Facility to save and retrieve a search

• Provision for downloading multiple files in tarball via web interface

• Improved on-line plotting (also for multidimensional data)

• Corrected display of comments to variables

File searching (1/4)

• Defaults are marked with ***

• Page initially opens without defaults

• Page may be reset or set to saved defaults bys selecting a previously saved query

• Files may be filtered by a large number of metadata parameters

File searching (2/4)

• The default criteria page lists all criteria sets that this user has previously saved

• Entries may easily be removed

• The main search page will be loaded with the selected defaults when launched from this page

• Defaults may include 4D space-time parameters and variable name wild-cards

File searching (3/4)

• The NUMERICAL SIMULATION entry is now marked with *** since we selected it as default through the saved query page

• Variable name / mode / descriptor have wildcard entries automatically added to the drop down lists.

• Variable name and variable mode accept multiple selections

• 4D space-time criteria may be entered as a single point, or as a max/min box

• All files that include or touch the 4D criteria are listed as matches

File searching (4/4)

• Files may be selected by posting date

• Files updated after a given date may be selected

• The file list may be sorted by file name, by submitter or PI, by posting date, or by AOID

• If no criteria are selected, the entire file list will be requested

File List

• The file list displays the current selection criteria

• A single file may be downloaded immediately

• Files may be marked and downloaded in a compressed tarball

• Search criteria may be saved for later re-use (type a meaningful description before saving)

• The Variables link brings up a variable list and other documentation contained in the file

Variable List

• The variable list page displays all parameters contained in a file

• Variable name, mode and descriptor are displayed

• The measurement unit and corresponding SI unit are displayed

• All documentation fields attached to the file are displayed

• Additional documentation fields attached to a single variable may be displayed through the Comments link

• Variables may be selected to generate an on-line plot (next slide)

• Observe the dependencies listed for each variable – not all variable combinations will produce meaningful plots

Variable plotting (on-line browse)

Administrative tools

• Web tools to report uploads/downloads

• Web tools to archive/retrieve outdated data files

• Web tools to check and maintain index database consistency and integrity

User activities

• Users have sent approximately 200 emails with requests for accounts or updates to metadata, or with questions to HDF file generation

• After launch focus has slowly shifted to requests for help with HDF file formatting issues and removal of outdated files

• From May to November 2002 users have submitted 5880 HDF data files (570 of these were rejected due to formatting problems, and 238 have later been archived).

• From May to November 2002 users have downloaded 4000 HDF data files (totalling 215 MB) (ftp downloads via zardoz not included)

• There are currently 4741 ACVT files open to users (22 active AOIDs)

• There are currently 462 MAVT files open to users (5 active AOIDs)

User activities: Up and Downloads

Nadir-team activities

• 300+ interactions with users (of which 50% during Commissioning Phase)

• 250+ interactions with ESA and the Metadata Board (10% during Commissioning Phase)

• One major re-write and 6 updates of the ASC2HDF and HDF2ASC programs

• Maintained and improved shell scripts for automatic file checking and storage

• Created additional IDL and FORTRAN tools

• Performed extensive web program development

• Performed user administration updates

• Added metadata entries and updated the table.dat file 23 times since launch

• Developed and maintained scripts and data flow to provide pre-processed ECMWF data extractions and plots on the web

• Monitored the automatic file reception system (now containing nearly 5000 files totalling 275 MB + index database 20 MB)

Metadata issues

• Although many quality checks are performed on the data upon submission to the database:

• The metadata definition is currently not complete enough to perform automatic checks of the validity of reported measurement units

• Some components and units may still be inconsistently named or mis-used

• Several files have indeed been submitted containing such errors due to misinterpretation of the guidelines. The data submittors will be contacted to correct these files.