Eu dataGrid Data Management Workpackage : wp2 Status and Plans Peter z kunszt

EU DataGrid Data Management Workpackage : WP2 Status and Plans

• Peter Z Kunszt

• IT/DB

• http://cern.ch/grid-data-management/

Outline

• WP2 Mandate, Tasks and Structure

• Achievements to date

• Task status

• Replication
• Metadata
• Optimization
• Security

• Plans

• Issues

DataGrid workpackages

• WP1: Workload Management

• WP2: Grid Data Management

• WP3: Grid Monitoring Services

• WP4: Fabric management

• WP5: Mass Storage Management

• WP6: Integration Testbed – Production quality International Infrastructure

• WP7: Network Services

• WP8: High-Energy Physics Applications

• WP9: Earth Observation Science Applications

• WP10: Biology Science Applications

• WP11: Information Dissemination and Exploitation

• WP12: Project Management

Mandate: Data Management

• The goal of this work package is to

• specify, develop, integrate and test

• tools and middleware infrastructure to

• coherently manage and share petabyte-scale information volumes in high-throughput production-quality grid environments.

• The work package will

• develop a

• general-purpose information sharing solution with unprecedented automation, ease of use, scalability, uniformity, transparency and heterogeneity.

Mandate: Data Management

• It will allow to securely access

• massive amounts of data in a universal global name space, to

• move and replicate data at high speed from one geographical site to another,

• and to manage synchronisation of remote replicas.

• Novel software for

• automated wide-area data caching and distribution will act according to dynamic usage patterns.

Data Management Tasks

• Data Transfer Efficient, secure and reliable transfer of data between sites

• Data Replication Replicate data consistently across sites

• Data Access Optimization Optimize data access using replication and remote open

• Data Access Control Authentication, ownership, access rights on data

• Metadata Storage Grid-wide persistent metadata store for all kinds of Grid information

Current Constituency

• CERN Peter Kunszt, Heinz Stockinger, Leanne Guy, Diana Bosio, Akos Frohner, Wolfgang Hoschek, Kurt Stockinger

• INFN Flavia Donno, Andrea Domenici, Livio Salconi, Giuseppe Andronico, Federico Di Carlo, Marco Serra

• PPARC Gavin McCance, William Bell, David Cameron, Paul Millar

• Trento Cultural Institute Ruben Carvajal Schiaffino, Luciano Serafini, Floriano Zini

• U.Helsinki/CSC Mika Silander, Joni Hahkala, Ville Nenonen, Niklas Karlsson

• KDC Stockholm Olle Mulmo, Gian Luca Volpato

Additional Resources, collaborators

• LCG:Erwin Laure, Itzhak Ben-Akiva, James Casey

• PPDG/Condor: Andy Hanushevsky, Aleksandr Konstantinov, Alan Roy

• Globus/ISI: Ann Chervenak, Bob Schwarzkopf

Outline

• WP2 Mandate, Tasks and Structure

• Achievements to date

• Task status

• Replication
• Metadata
• Optimization
• Security

• Plans

• Issues

Deliverables

Communications

• Quarterly reports on WP2

• Regular meetings and workshops

• 2 DataGrid conferences per year
• 2 WP2 dedicated workshops per quarter
• Weekly management meetings
• Weekly phone conferences depending on task

• 8 dedicated WP2 mailinglists

• Long list of publications in all of our areas – see www. 26 in total, 12 in journals/proceedings.

Outline

• WP2 Mandate, Tasks and Structure

• Achievements to date

• Task status

• Replication
• Metadata
• Optimization
• Security

• Plans

• Issues

File Management

Architecture

Current Components

• File Transfer: Use GridFTP – deployed

• Close collaboration with Globus
• NetLogger (Brian Tierney and John Bresnahan)

• Replication: GDMP – deployed

• Wrapper around Globus ReplicaCatalog
• All functionality in one integrated package
• Using Globus 2
• Uses GridFTP for transferring file

• Replication: edg-replica-manager – deployed

• Replication: Replica Location Service Giggle – in testing

• Distributed Replica Catalog

• Replication: Replica Manager Reptor – in testing

• Optimization: Replica Selection OptorSim – in simulation

• Metadata Storage: SQL Database Service Spitfire – deployed

• Servlets on HTTP(S) with XML (XSQL)
• GSI enabled access + extensions

• GSI interface to CASTOR – delivered

Current Status

• GDMP version 3.0.5

• Support for multiple VOs
• New security mechanism for server and clients
• uses Globus 2.0 beta 21 on EDG testbed
• Linux RedHat 6.2 (6.1.1) partly already RedHat 7.2 (some manual changes required)
• GDMP is part of VDT (Virtual Data Toolkit)
• Alain Roy (Condor team) does support for US sites

• Replica Manager

• Wrapper around existing Globus replica manager edg-replica-manager
• Core API as defined in Replica Manager Design Document is implemented Reptor
• Java Servlet, alpha release in internal testing

Current Status

• Replica Catalog

• GUI and command line tool for existing RC
• Available to the production testbed
• Introduced a security patch for RC
• Collaboration with Globus and PPDG on new, distributed Replica Catalog Framework (Giggle)
• joint design with Globus, G:coding, EDG:testing
• Internal alpha release, integrated with Reptor

• Working prototype of Simulator

File mappings

Replica location problem

• Replica Location Service:

• maintains information on the physical location of files
• maintains mapping between the logical identifier of data and all its physical instances
• provides access to this information

RLS Requirements

• Versioning & read only data

• distinct versions of files can be uniquely identified
• data published to the community are immutable

• Size

• scale to hundreds of replica sites, 50 x 108 LFNs, 500 x 108 PFNs

• Performance

• 200 updates/second, average response time < 10ms

• Security

• knowledge and existence of private data must be protected
• storage system protects integrity of data content

• Consistency

• view of all available PFNs not necessarily consistent

• Reliability

• no single point of failure,
• local and global state decoupled,
• failure of remote component does not hinder access to local component

Giggle Framework

• Giggle: A Framework for Constructing Scalable Replica Location Services

• Joint collaboration between WP2 and Globus
• Paper submitted to SC2002

• Independent local state maintained in Local Replica Catalogues : LRCs

• Unreliable collective state maintained in Replica Location Indices : RLIs

• Soft state maintenance of RLI state

• relaxed consistency in the RLI, full state information in LRC

• Compression of soft states

• compress LFN information based on knowledge of logical collections

• Membership and partitioning information maintenance

• RLS components change over time : failure, new components added
• Service discovery and system policies

Local Replica Catalogue (LRC)

• Maintains replica information at a single site

• Complete locally consistent record
• Queries across multiple sites not supported

• Maintains mappings between LFNs and PFNs on associated storage systems

• Coordinates its contents with those of the storage system

• Responds to the following queries:

• Given an LFN, find the set of PFNS associated with that LFN
• Given a PFN, find the set of LFNS associated with that PFN

• Supports authentication and authorisation when processing remote requests

• Periodically sends information about its state to the RLIs

Replica Location Index (RLI)

• Index structure needed to support queries across multiple sites

• One or more RLIs to map LFNs to LRCs

• Structure w.r.t LRCs can be freely defined
• redundancy, performance, scalability

• Geographical partitioning – all PFNs of a set of LRCs are indexed

• Namespace partitioning 1 – for load balancing purposes

• Namespace partitioning 2 – only LFNs adhering to a specified pattern are indexed for all LRCs

• possibly not good for load balancing

• Many identical RLIs may be set up for load balancing

RLS Architecture (1)

RLS Architecture (2)

Metadata Management and Security

• Project Spitfire

• 'Simple' Grid Persistency

• Grid Metadata
• Application Metadata
• Unified Grid enabled front end to relational databases.

• Metadata Replication and Consistency

• Publish information on the metadata service

• Secure Grid Services

• Grid authentication, authorization and access control mechanisms enabled in Spitfire

• Modular design, reusable by other Grid Services

Spitfire Architecture

• Atomic RDBMS is always consistent

• No local replication of data

• Role-based authorization

Security Mechanism

Status

• Integrated XSQL-Spitfire including security mechanisms

• XSQL version of a Replica MetaData Catalog

• Schema is given
• Roles are fixed
• Queries are predefined
• No SOAP, just XML over http

• Installation decoupled from Tomcat & MySQL

• Work on SOAP interface using axis

• Security mechanisms planned

Outline

• WP2 Mandate, Tasks and Structure

• Achievements to date

• Task status

• Replication
• Metadata
• Optimization
• Security

• Plans

• Issues

WP2 Replication Services

Plans

• Service-type Replica Manager integrated with Giggle and Replica Metadata Catalog

• Optimization module integrated

• Automated replication module designed and prototyped

Plans

• All functionality available

• Security fully integrated

• Installation works with more than just Tomcat and MySQL

• Test-suite completely implemented

• Working Replication Metadata Catalog using this Beta, integration with Replica Manager

Summary of Plans

• GDMP v3.0 will be the LAST release of GDMP as we know it.

• GDMP in the future will rely on the Replica Manager and provide the subscription based mirroring functionality. It will be implemented as a web service.

• The Replica Catalog will be replaced with the Giggle framework, jointly developed with Globus.

• The Replica Manager will take over most of GDMPs current functionality and will be a web service. We provide client APIs. All user interaction should go through the RM.

• Spitfire will have both an XML over http access method for static queries and a SOAP interface.

• Security infrastructure for all services in place as presented.

Outline

• WP2 Mandate, Tasks and Structure

• Achievements to date

• Task status

• Replication
• Metadata
• Optimization
• Security

• Plans

• Issues

Issues

• Getting everybody started, coordination with partners – one very late deliverable

• Technology challenges – evolving field, keeping on the top needs a lot of training effort of everybody

• Coordination within WP2 and US projects – not enough funds for travel @ CERN

• Very late arrival of and changes in manpower – additional training and supervision

• A lot of young members are working / have worked on their PhD

• Different agenda of user community delays development: continuous requests for support and enhancements of components that we want to phase out (GDMP) takes out development manpower, documentation suffers most.

Outlook

• Very ambitious programme, can we make it?

• Current workforce is very well motivated and has a very high level of interaction – see our mailinglist archives

• A lots of enhancements in inter-task coordination was done, further effort is necessary – workshops

• THANK YOU FOR YOUR ATTENTION