Reprocessing CMS events with Bosco

Prior to the LHC long shutdown, the CMS experiment increased the trigger rate of the detector, therefore increasing the data coming off the detector.  The Tier-0 was unable to process all of the events coming off of the detector, therefore the events where only stored and not processed.  After the run, the experiment wanted to process the backlog of events, but didn't have the computing power available to do it.  So they turned to opportunistic computing and Bosco.

The CMS collaborators at UCSD worked with the San Diego Supercomputing Resource to run the processing on the Gordon supercomputer.  Gordon is an XSEDE resource and does not include a traditional OSG Globus Gatekeeper.  Also, we did not have root access to the cluster to install a gatekeeper.  Therefore, Bosco was used to submit and manage the GlidienWMS Condor glideins to the resource.

Running jobs at Gordon, the SDSC supercomputer

As you can see from the graph, we reached nearly 4,000 CMS processing jobs on Gordon.  4k cores is larger than most CMS Tier 2's, and as big as a European Tier-1.  With Bosco, overnight, Gordon became one of the largest CMS clusters in the world.

Full details will be written in a submitted paper to CHEP '13 in Amsterdam, and Bosco will be presented in a poster (and paper) as well.  I hope to see you there!

(If I got any details wrong about the CMS side of this run, please let me know.  I have intimate knowledge of the Gordon side, but not so much the CMS side).

Running Quantum Espresso on the OSG

While running Quantum Espresso on the Open Science Grid, we found a number of issues:

• OpenMPI needs to have an rsh binary.  Even if you are using shared memory for openmpi, and openmpi does not use rsh, it still looks for the binary and fails if it cannot find it.
• Chroots (used on HCC machines for grid jobs) do not support pty's.  OpenMPI has a compile option to turn off pty support.

Once these issues where fixed, we were able to submit QE jobs to the OSG using Condor's partitionable slots on 8 cores.

Preparing Submission

Before submitting our first QE job, we had to compile OpenMPI and QE.  Since we are an HPC center, we had OpenMPI compiled for our Infiniband, therefore it would always fail on the OSG where there is no Infiniband (let alone our brand and drivers).

After compiling, we created compressed files that contained the required files to run QE:

• bin.tar.gz - Only includes the cp.x file, specific to our run.  It could have well included much more common pw.x.
• lib.tar.gz - Includes the Intel math libraries and libgfortran.
• openmpi.tar.gz - Includes the entire openmpi install directory (make install)

Additionally, we wrote a wrapper script, run_espresso_grid.sh, that unpacks the required files and sets the environment.

#!/bin/bash
tar xzf bin.tar.gz
tar xzf lib.tar.gz
tar xzf pseudo.tar.gz
tar xzf openmpi.tar.gz
mkdir tmp
 
export PATH=$PWD/bin:$PWD/openmpi/bin:$PATH
export LD_LIBRARY_PATH=$PWD/lib:$PWD/openmpi/lib:$LD_LIBRARY_PATH
export OPAL_PREFIX=$PWD/openmpi
 
mpirun --mca orte_rsh_agent `pwd`/rsh -np 8 cp.x < h2o-64-grid.in > h2o-64-grid.out

Submission

We used GlideinWMS to submit to the OSG, below is our HTCondor submit file.

universe = vanilla
output = condor.out.$(CLUSTER).$(PROCESS)
error = condor.err.$(CLUSTER).$(PROCESS)
log            = condor.log
executable = run_espresso_grid.sh
request_cpus=8
request_memory = 10*1024
should_transfer_files = YES
when_to_transfer_output = ON_EXIT_OR_EVICT
transfer_input_files = bin.tar.gz, lib.tar.gz, pseudo.tar.gz, openmpi.tar.gz, h2o-64-grid.in, /usr/bin/rsh
transfer_output_files =h2o-64-grid.out
+RequiresWholeMachine=True
Requirements = CAN_RUN_WHOLE_MACHINE =?= TRUE
queue 

Note that we pull rsh from the submission machine.  OpenMPI does not actually use rsh to start the processes on a shared memory machine, but it does require that the RSH binary is available.

Acknowledgments

This was done with the tremendous help of Jun Wang.

This work is licensed under a Creative Commons Attribution 3.0 Unported License.

Using Bosco to submit to Amazon EC2

A homework assignment for my storage class required running ~30 hours of benchmarks of the btrfs and ext4 filesystems.  I thought this would be an excellent time to test Bosco's ability to submit to Amazon EC2 to parallelize the benchmarks.

Preparing Submission

In order to start instances on Amazon EC2, you first need to sign up.  Go to https://aws.amazon.com/ and sign up in the top right.  After you sign up, you will need the access and secret key.  These can be found in the 'Security Credentials' from the account drop down box.  They are in the 'Access Keys' tab under the 'Access Key ID' and 'Secret Access Key'.  Write those values in 2 files, you will need them when you submit EC2 instances.

Screenshot of Amazon Security credentials site

Next, you will need a script to run at startup of the Amazon instance.  When the instance starts up, a service named CloudInit also start on the instance.  It will interpret the user data file as a shell script which can setup and start any other services you would like.  My shell script is provided below.

Loading ....

This shell script will install python-boto (python bindings for S3 storage and ec2) and git onto the instance.  Next, it will download the filebenchrunner (Benchmark runner for the homework), and start it.  Most people will probably want to shut down the instance after you are done with processing, in that case you can just add to the bottom a 'poweroff'.

Running the Instance

Running an Amazon instance is as easy as running a Bosco job.  First, you must create a Bosco submit file.  Below is the one I used:

Some important things to note.  I specified ec2_spot_price, which is the amount I am willing to pay for my m1.medium instance to run per hour.  I said $0.04 an hour, which is pretty low, but reasonable for a medium instance.  You can find all of the current spot prices either in the AWS console, or on the EC2 website.  As you can see, the spot prices are much, much smaller than the on-demand price of an instance.  For example, for the m1.medium instance, which has 1.7 GB of ram and 1 core, the spot price currently is $0.013 per hour.  The on-demand price is $0.120 per hour.  That's a 90% discount on a m1.medium.  Of course, you should always read the downsides of using a spot instance, such as it can be terminated at any time, without warning, by Amazon.  For my benchmarks, I can always re-run benchmarks if my instance is terminated.  I needed to run 10 - 10 minute benchmarks, therefore after every benchmark, I uploaded the resulting data to S3 immediately so I wouldn't lose any work if the instance was terminated.

Also, I used the regular Amazon Linux AMI.  They are listed on the Amazon website.  I could have very well used a CentOS, Ubuntu, or any other linux image for my instance.  But, I prefer the official Linux AMI since it provides a very up to date OS which is very similar to the feel of a CentOS 6 instance.  For example, it uses yum for repository management, and RPM's to install.  And has versions (except for the kernel) similar to CentOS 6.

I also added a special command, periodic_remove, in order to terminate the instance if something went wrong inside the instance.  Sometimes yum can hang, or the instance may not start up properly.  In those cases, amazon will not notify you of the problem, and Bosco will not be able to determine there is an issue.  Since my benchmarks should not last longer than 100 minutes, I automatically remove the instance after 150 minutes (a little breathing room) of running.

You may submit the instance with the normal 'condor_submit' command.  The job will move to the Running state when the instance has begun running.

Once the instance has started, you may ssh into the instance by using the unique ssh key that Bosco generates for you.  It is specified in the submit file as ec2_keypair_file.  You also need the DNS name for the instance, which is available in the job's classad.

condor_q -run

The command will output the hostname of the EC2 host. You may connect to the EC2 instance with the command, replacing the XXXX with the job number, and hostname with the address you get from the above command:

$ ssh -i keyfile.XXXXX ec2-user@hostname

Summary

Pros of using Bosco to submit Amazon EC2 Jobs:

• Simple management of Amazon instance from your workstation.
• Specify spot price right inside of the job description.
• Ability to bootstrap the instance easily with user data scripts.
• Ability to use HTCondor policies in order to manage the instances, such as periodic remove statement above.

Cons:

• The EC2 universe is only available on Linux builds of Bosco.  You cannot manage EC2 instances on the Mac version of Bosco.
• Amazon EC2 has hundreds and hundreds of features, Bosco only allows you to use the simple submit EC2 instances and spot pricing.  You will not be able to use the vast majority when you are using Bosco to manage your instances.  But if all you need is to run some processing, Bosco is great!

This work is licensed under a Creative Commons Attribution 3.0 Unported License.

Rendering with Bosco

Example image rendered using HCC's distributed renderer

At the Nebraska Holland Computing Center, we take pride in eating our own dog food.  So I want to highlight one of our uses of Bosco to enable transparent usage of our clusters.

HCC recently has made a push for enabling the non-traditional users of clusters.  Sure it's easy to show how physics can benefit from a set of clusters, but what about the media arts?  In this case we are working with a professor whose students render short movies using Maya.

Maya can utilize another Autodesk product called Backburner, which enables rendering across multiple nodes.  The challenge was to allow Backburner to operate in a shared cluster environment.  The standard method to use Backburner expects nodes dedicated solely to running the Backburner Server daemons.  This Master-Worker fits the traditional HTC model well, therefore we felt that Bosco would be a great fit to enable submission to our clusters.

Architecture Diagram of the BOSCO enabled render (Credit: Adam Caprez)

In the architecture shown in the image above, we run a central service, hcc-render that monitors the Backburner queue in order to submit the Backburner servers to the Bosco queue.  Bosco then submits a glidein to Tusker, which in turn runs the Backburner server and renders the scene.

In this case, since we are using Backburner server for the actual processing, the data is handled internally by Backburner.  In practice this means that data is stored on the Tusker file system which is then mounted by the professors local machine that is in turn mounted by the clients.

This is a classic example of deeply integrating HTC into the user's application.
The user only needs to click the 'render' button from within Maya on their workstation, and we handle all the rest, automatically.  This is only made possible because Maya has a sane back end renderer that is designed to run on Linux.  This architecture may not work for all commercial applications.

Even though the user isn't directly using Bosco, a primary goal of Bosco, this is an excellent use of Bosco to enable HTC workflows.

Bosco 1.1.1 Release

Today I am pleased to announce that Bosco version 1.1.1 has been released. This is a patch fix for the 1.1 to address 2 issues that where affecting users.  The release is available on the Bosco Download Page.

Release notes for the 1.1.1 release are available on the OSG Twiki.

Bosco 1.1 was a major release with many new features. The 1.1 release notes are also available on the OSG Twiki.

On behalf of the Bosco Team,
Derek Weitzel
Dan Fraser
Marco Mambelli
Jaime Frey
Brooklin Gore
Miha Ahronovitz

Improving Gratia's Web Interface

Over the winter break, I worked on improving the interface that most users use for OSG accounting.  When I returned from break, I worked with Ashu to integrate the changes with some recent changes he had made. The new interface runs on gratiaweb-itb.  The source for the new web page is hosted on github.

The first thing users will notice is the newly designed interface:

New OSG Accounting Interface

The updated interface brings the style of the website inline with that of MyOSG and OIM (or, close).  The design stayed close the the original, but the menu on the right has changed significantly.  First is just the style of the menu.  But also we added a new category, Campus and Pilot View.

In the Campus and Pilot View, we have some new graphs that show usage by GlideinWMS, Campus users, XSEDE users, and in the future, Bosco users.

Lets run through a quick example.  In this example, lets assume I'm a VO manager and want to see where my VO is running, how many hours, and who is running.

1. Select the Pilot & Campus Accounting link.
2. Scroll to the bottom, to the Refine View.
3. Enter your VO name into the VO text box and hit enter.

This will pull up the custom page that shows usage for only your VO.  For example, if I look at the osg VO:

Usage by the OSG VO.

You can see from the graphs that the OSG VO has used ~80,000 CPU hours a day on the OSG.  Also, they are running at over 20 sites.  The sites at the bottom of the graph are listed in order of total hours (I am happy to see Nebraska resources as #3, #6, and #9).  

You can also see from the graph that usage at sites depends on the day.  Some days they get significant usage at the MWT2 (UChicago and IU), and some days they run a lot at Nebraska.

The new usage graphs are intended to help users, administrators, and VO managers view their usage.  I hope you find them as useful as we have in the past.

We hope that webpage is an improvement.  If there are any comments on further improvements, we are interested in your feedback.

Bosco 1.1 Release

In the last few months, I have outlined the new features of Bosco 1.1.

• Mutli-Cluster Support
• Multi-OS Support
• Single Port Usage
• SSH File Transfer

I am happy to announce that today we are releasing Bosco 1.1.  The official Release notes are available on the OSG Twiki.  It includes all of the features given above.  Try it out!