High Throughput Computing 10 years ago we had “The Grid”

High Throughput Computing

10 years ago we had “The Grid”

“ … We claim that these mechanisms, although originally developed in the context of a cluster of workstations, are also applicable to computational grids. In addition to the required flexibility of services in these grids, a very important concern is that the system be robust enough to run in “production mode” continuously even in the face of component failures. … “

• “ … We claim that these mechanisms, although originally developed in the context of a cluster of workstations, are also applicable to computational grids. In addition to the required flexibility of services in these grids, a very important concern is that the system be robust enough to run in “production mode” continuously even in the face of component failures. … “

In the words of the CIO of Hartford Life

• Resource: What do you expect to gain from grid computing? What are your main goals?

• Severino: Well number one was scalability. …

• Second, we obviously wanted scalability with stability. As we brought more servers and desktops onto the grid we didn’t make it any less stable by having a bigger environment. 

• The third goal was cost savings. One of the most …

2,000 years ago we had the words of Koheleth son of David king in Jerusalem

35 years ago we had The ALOHA network

One of the early computer networking designs, the ALOHA network was created at the University of Hawaii in 1970 under the leadership of Norman Abramson. Like the ARPANET group, the ALOHA network was built with DARPA funding. Similar to the ARPANET group, the ALOHA network was built to allow people in different locations to access the main computer systems. But while the ARPANET used leased phone lines, the ALOHA network used packet radio.

• One of the early computer networking designs, the ALOHA network was created at the University of Hawaii in 1970 under the leadership of Norman Abramson. Like the ARPANET group, the ALOHA network was built with DARPA funding. Similar to the ARPANET group, the ALOHA network was built to allow people in different locations to access the main computer systems. But while the ARPANET used leased phone lines, the ALOHA network used packet radio.

• ALOHA was important because it used a shared medium for transmission. This revealed the need for more modern contention management schemes such as CSMA/CD, used by Ethernet. Unlike the ARPANET where each node could only talk to a node on the other end, in ALOHA everyone was using the same frequency. This meant that some sort of system was needed to control who could talk at what time. ALOHA's situation was similar to issues faced by modern Ethernet (non-switched) and Wi-Fi networks.

• This shared transmission medium system generated interest by others. ALOHA's scheme was very simple. Because data was sent via a teletype the data rate usually did not go beyond 80 characters per second. When two stations tried to talk at the same time, both transmissions were garbled. Then data had to be manually resent. ALOHA did not solve this problem, but it sparked interest in others, most significantly Bob Metcalfe and other researchers working at Xerox PARC. This team went on to create the Ethernet protocol.

30 years ago we had Distributed Processing Systems

Claims for “benefits” provided by Distributed Processing Systems

• High Availability and Reliability
• High System Performance
• Ease of Modular and Incremental Growth
• Automatic Load and Resource Sharing
• Good Response to Temporary Overloads
• Easy Expansion in Capacity and/or Function

Definitional Criteria for a Distributed Processing System

• Multiplicity of resources
• Component interconnection
• Unity of control
• System transparency
• Component autonomy

Multiplicity of resources

• The system should provide a number of assignable resources for any type of service demand. The greater the degree of replication of resources, the better the ability of the system to maintain high reliability and performance

Component interconnection

• A Distributed System should include a communication subnet which interconnects the elements of the system. The transfer of information via the subnet should be controlled by a two-party, cooperative protocol (loose coupling).

Unity of Control

• All the component of the system should be unified in their desire to achieve a common goal. This goal will determine the rules according to which each of these elements will be controlled.

System transparency

• From the users point of view the set of resources that constitutes the Distributed Processing System acts like a “single virtual machine”. When requesting a service the user should not require to be aware of the physical location or the instantaneous load of the various resources

Component autonomy

• The components of the system, both the logical and physical, should be autonomous and are thus afforded the ability to refuse a request of service made by another element. However, in order to achieve the system’s goals they have to interact in a cooperative manner and thus adhere to a common set of policies. These policies should be carried out by the control schemes of each element.

Challenges

• Name spaces …

• Distributed ownership …

• Heterogeneity …

• Object addressing …

• Data caching …

• Object Identity …

• Trouble shooting …

• Circuit breakers …

24 years ago I wrote my Ph.D. thesis – “Study of Load Balancing Algorithms for Decentralized Distributed Processing Systems”

BASICS OF A M/M/1 SYSTEM

BASICS OF TWO M/M/1 SYSTEMS

Wait while Idle (WwI) in m*M/M/1

“ … Since the early days of mankind the primary motivation for the establishment of communities has been the idea that by being part of an organized group the capabilities of an individual are improved. The great progress in the area of inter-computer communication led to the development of means by which stand-alone processing sub-systems can be integrated into multi-computer ‘communities’. … “

• “ … Since the early days of mankind the primary motivation for the establishment of communities has been the idea that by being part of an organized group the capabilities of an individual are improved. The great progress in the area of inter-computer communication led to the development of means by which stand-alone processing sub-systems can be integrated into multi-computer ‘communities’. … “

20 years ago we had “Condor”

1986-2006 Celebrating 20 years since we first installed Condor in our department

The Condor Project (Established ‘85)

• Distributed Computing research performed by a team of ~40 faculty, full time staff and students who

• face software/middleware engineering challenges in a UNIX/Linux/Windows/OS X environment,
• involved in national and international collaborations,
• interact with users in academia and industry,
• maintain and support a distributed production environment (more than 3800 CPUs at UW),
• and educate and train students.

• Funding – DOE, NIH, NSF, INTEL,

• Micron, Microsoft and the UW Graduate School

Main Threads of Activities

• Distributed Computing Research – develop and evaluate new concepts, frameworks and technologies

• The Open Science Grid (OSG) – build and operate a national distributed computing and storage infrastructure

• Keep Condor “flight worthy” and support our users

• The NSF Middleware Initiative (NMI) – develop, build and operate a national Build and Test facility

• The Grid Laboratory Of Wisconsin (GLOW) – build, maintain and operate a distributed computing and storage infrastructure on the UW campus

The past year:

• Two Ph.D students graduated:

• Tevfik Kosar went to LSU
• Sonny (Sechang) Son went to NetApp

• Three staff members left to start graduate studies

• Released Condor 6.6.9-.11

• Released Condor 6.7.6-.18

• Contributed to the formation of the Open Science Grid (OSG) consortium and the OSG Facility

• Interfaced Condor with BOINC

• Started the NSF funded CondorDB project

• Released Virtual Data Toolkit (VDT) 1.3.3-.10

• Distributed five instances of the NSF Middleware Initiative (NMI) Build and Test facility

The search for SUSY

• Sanjay Padhi is a UW Chancellor Fellow who is working at the group of Prof. Sau Lan Wu at CERN

• Using Condor Technologies he established a “grid access point” in his office at CERN

• Through this access-point he managed to harness in 3 month (12/05-2/06) more that 500 CPU years from the LHC Computing Grid (LCG) the Open Science Grid (OSG) and UW Condor resources

High Throughput Computing

• We first introduced the distinction between High Performance Computing (HPC) and High Throughput Computing (HTC) in a seminar at the NASA Goddard Flight Center in July of 1996 and a month later at the European Laboratory for Particle Physics (CERN). In June of 1997 HPCWire published an interview on High Throughput Computing.

Why HTC?

• For many experimental scientists, scientific progress and quality of research are strongly linked to computing throughput. In other words, they are less concerned about instantaneous computing power. Instead, what matters to them is the amount of computing they can harness over a month or a year --- they measure computing power in units of scenarios per day, wind patterns per week, instructions sets per month, or crystal configurations per year.

High Throughput Computing is a 24-7-365 activity

Obstacles to HTC

• Ownership Distribution

• Customer Awareness

• Size and Uncertainties

• Technology Evolution

• Physical Distribution

Focus on the problems that are unique to HTC not the latest/greatest technology

HTC on the Internet

• Retrieval of atmospheric temperature and humidity profiles from 18 years of data from the TOVS sensor system.

• 200,000 images
• ~5 minutes per image

10 years ago we had “The Grid”

What makes an “O” a “VO”?

What is new beneath the sun?

• Distributed ownership – who defines the “system’s common goal”? No more one system.

• Many administrative domains – authentication, authorization and trust.

• Demand is real – many have computing needs that can not be addressed by centralized locally owned systems

• Expectations are high – Regardless of the question, distributed technology is “the” answer.

• Distributed computing is once again “in”.

Benefits to Science

• Democratization of Computing – “you do not have to be a SUPER person to do SUPER computing.” (accessibility)

• Speculative Science – “Since the resources are there, lets run it and see what we get.” (unbounded computing power)

• Function shipping – “Find the image that has a red car in this 3 TB collection.” (computational mobility)

The NUG30 Quadratic Assignment Problem (QAP)

NUG30 Personal Grid …

• Managed by one Linux box at Wisconsin

• Flocking: -- the main Condor pool at Wisconsin (500 processors)

• -- the Condor pool at Georgia Tech (284 Linux boxes)

• -- the Condor pool at UNM (40 processors)

• -- the Condor pool at Columbia (16 processors)

• -- the Condor pool at Northwestern (12 processors)

• -- the Condor pool at NCSA (65 processors)

• -- the Condor pool at INFN Italy (54 processors)

• Glide-in: -- Origin 2000 (through LSF ) at NCSA. (512 processors)

• -- Origin 2000 (through LSF) at Argonne (96 processors)

• Hobble-in: -- Chiba City Linux cluster (through PBS) at Argonne

• (414 processors).

Solution Characteristics.

The NUG30 Workforce

“ … Grid computing is a partnership between clients and servers. Grid clients have more responsibilities than traditional clients, and must be equipped with powerful mechanisms for dealing with and recovering from failures, whether they occur in the context of remote execution, work management, or data output. When clients are powerful, servers must accommodate them by using careful protocols.… “

• “ … Grid computing is a partnership between clients and servers. Grid clients have more responsibilities than traditional clients, and must be equipped with powerful mechanisms for dealing with and recovering from failures, whether they occur in the context of remote execution, work management, or data output. When clients are powerful, servers must accommodate them by using careful protocols.… “

Being a Master

• Customer “delegates” task(s) to the master who is responsible for:

• Obtaining allocation of resources
• Deploying and managing workers on allocated resources
• Delegating work unites to deployed workers
• Receiving and processing results
• Delivering results to customer

Master must be …

• Persistent – work and results must be safely recorded on non-volatile media

• Resourceful – delegates “DAGs” of work to other masters

• Speculative – takes chances and knows how to recover from failure

• Self aware – knows its own capabilities and limitations

• Obedience – manages work according to plan

• Reliable – can mange “large” numbers of work items and resource providers

• Portable – can be deployed “on the fly” to act as a “sub master”

Master should not do …

• Predictions …

• Optimal scheduling …

• Data mining …

• Bidding …

• Forecasting …

The Ethernet Protocol

• IEEE 802.3 CSMA/CD - A truly distributed (and very effective) access control protocol to a shared service.

• Client responsible for access control
• Client responsible for error detection
• Client responsible for fairness

Never assume that what you know is still true and that what you ordered did actually happen.

Every Community can benefit from the services of Matchmakers!

Why? Because ...

• .. someone has to bring together community members who have requests for goods and services with members who offer them.

• Both sides are looking for each other
• Both sides have constraints
• Both sides have preferences

Being a Matchmaker

• Symmetric treatment of all parties

• Schema “neutral”

• Matching policies defined by parties

• “Just in time” decisions

• Acts as an “advisor” not “enforcer”

• Can be used for “resource allocation” and “job delegation”

Bringing it all Together

From Condor to Condor-G to Condor-C

The Layers of Condor

Be matched, claim (+maintain), and then delegate

Job Submission Options

• leave_in_queue =

• on_exit_remove =

• on_exit_hold =

• periodic_remove =

• periodic_hold =

• periodic_release =

• noop_job =

How can we accommodate an unbounded need for computing with an unbounded amount of resources?

Close by storage is small and fast faraway storage is big and slow

Many data challenges …

• Managing data is a hard problem. Doing it in a distributed environment does not make it easier or simpler:

• Catalogs and metadata
• Access control
• Consistency and coherency
• Revocation and auditing
• Replication/cashing management
• Planning (optimization?)

Almost everything we do requires a dependable data placement mechanism

We are making progress …

• The Storage Resource Management (SRM) protocol – management of file copies and support for space reservations

• The Reliable File Transfer (RFT) service – management of large numbers of GridFTP requests

• The File Transfer Service (FTS) – manages file transfer requests and supports the concept of “channels”

• The Planning for Execution in Grids (Pegasus) planner – supports data placement steps in the workflow

High capacity networks are deployed all over the world and almost everyone is concerned about how to allocated their bandwidth. However, is bandwidth the real issue?

ESnet4 Target Configuration

Main trend

• The ratio between the size of the organization and the volume (and complexity) of the data/information/knowledge the organization owns/manages/depends on will continue to dramatically increase

• Ownership cost of managed storage capacity goes down
• Data/information/knowledge generated and consumed goes up
• Network capacity goes up
• Distributed computing technology matures and is more widely adopted

Managed Object Placement

• Management of storage space and bulk data transfers plays a key role in the end-to-end effectiveness of many scientific applications:

• Object Placement operations must be treated as “first class” tasks and explicitly expressed in the work flow
• Fabric must provide services to manage storage space
• Object Placement schedulers and matchmakers are needed
• Object Placement and computing must be coordinated
• Smooth transition of Compute/Placement interleaving across software layers and granularity of compute tasks and object size
• Error handling and garbage collection

Customer requests: Place y = F(x) at L! System delivers.

Simple plan for y=F(x)L

• Allocate (size(x)+size(y)+size(F)) at SEi

• Place x from SEj at SEi

• Place F on CEk

• Compute F(x) at CEk

• Move y from SEi at L

• Release allocated space at SEi

The Basic Approach*

Stork-A possible solution

• A portable, flexible and extensible Object Placement Scheduler.

• Uses ClassAds to capture jobs and policies (just like Condor)
• Supports matchmaking (just like Condor)
• Provides a suite of data transfer jobs that interface with a broad collection of storage systems and protocols and provide end-to-end reliability
• Supports storage allocate/release jobs

Customer requests: Place y@S at L! System delivers.

Basic step for y@SL

• Allocate size(y) at L,

• Allocate resources (disk bandwidth, memory, CPU, outgoing network bandwidth) on S

• Allocate resources (disk bandwidth, memory, CPU, incoming network bandwidth) on L

• Match S and L

Or in other words, it takes two (or more) to Tango (or to place an object)!

When the “source” plays “nice” it “asks” in advance for permission to place an object at the “destination”

The SC’05 effort Joint with the Globus GridFTP team

A Master Worker view of the same effort

When the “source” does not play “nice”, destination must protect itself

NeST

• Manages storage space and connections for a GridFTP server with commands like:

• ADD_NEST_USER
• ADD_USER_TO_LOT
• ATTACH_LOT_TO_FILE
• TERMINATE_LOT

How can we accommodate an unbounded amount of data with an unbounded amount of storage and network bandwidth?