OSG Site Admin meeting: Day 2

Another beautiful (read: HOT) day in Lubbock.

Last night we ate at the National Ranching Heritage Center as part of social get together.  The food was great!  I had a great time talking other CMS collaborators as well.  I talked with some T3 admins from UC Riverside and Baylor, learning about what is like to run a T3 (and talking some Big Bang Theory).

I also talked with Horst from OU to find out that they are no longer running the CoLinux cluster on their desktop machines.  Rather, they are using VMWare to run Linux inside Windows on their desktops.  Something like 1600 cores.

The heritage center had a number of historical buildings from around the area.

(Credit: Horst Severini more)

The talks have been good.  Here's a quick picture from Doug's Globus Online talk.

OSG Site Admin meeting: Day 1

Hello from the hot, hot, hot Lubbock.

Had a good morning of broad plenary talks.  The indico schedule  gives links to the talks.

In the afternoon, I gave my 2 talks: GlideinWMS Frontend install and Campus Factory install.  We had one person install and get the campus factory up and working (though, only one was trying to install).  We had to add some custom pbs commands for their cluster in the pbs_local_submit_attributes.sh file.

Meanwhile, Jose was installing the glideinwms-vofrontend from RPMs.  He had a conflict with an igtf rpm of certificates.  Was solved by removing the igtf rpm, and allowing OSG to own the certificates.

Here's a picture from Marco's talk.  I'll upload more pictures later.

And from Greg's Talk:

And from my Hands-On:

Testing Release of OSG-Client RPM

It's finally here!  The first official testing release of the osg-client in RPM form.  We're asking anyone and everyone to go out there and start testing.  The official instructions are on the OSG Twiki.  We even made a Testing Client page if you want to run through some simple commands.

The instructions are simple, install EPEL, then install the VDT repo:

rpm -Uvh http://vdt.cs.wisc.edu/repos/3.0/el5/development/x86_64//vdt-release-3.0-2.noarch.rpm 

Install the osg-client (expect a lot of dependencies). My latest build downloaded 140MB of dependencies.

yum install --enablerepo=vdt-testing --nogpgcheck osg-client 

Then, go and do your normal workflows.  Create proxies, run globus-job-run's, start condor and run jobs.  Whatever, just test.  If you have any issues, email osg-software@opensciencegrid.org.  Or join me on the OSG jabber, osg@conference.indiana.edu.

All current bugs (I'm happy with how few there are) with the osg-client can be found on Jira.

Go forth and test...

Job Visualization

A year or so ago, Ian Stokes-Rees showed me a job visualization that he had put together to see how his workflows where going.  Gives a good overview per site.  Recently, we where investigating usage from a NEES user, and I adapted the aging script to visualize the workflow.  I think it turned out really well.

Note:  Be prepared to zoom in a lot.  Job ID's are printed at the beginning of execution.  Green lines are successful completions.  Red lines are Evicted or Job Disconnected jobs.

View on Google Docs

Very ugly source is on github: https://github.com/djw8605/condor_log_analyze

It's times like this I wish I had a tile wall again.

Note, I found chrome's pdf viewer faster than Preview.

HCC Walltime Effeciency

Rob's post on walltime efficiency was interesting.  I did the same with HCC and found that we had much better efficiency than I expected.  Especially with GlideinWMS's default behavior of sticking around for 20 minutes waiting for a job before exiting.

What's interesting is that we rarely run jobs at RENCI-Engagement, OSCER, or GPGRID.  It's difficult to interpret this.  Since we run so many workflows, it's not clear to me that this means much.

Need more cores, Scotty!

A researcher/grid user from UNL has come back!  Monitoring

100K+ Jobs.

Even though we have a lot of jobs idle, we are only able to get maybe 4.5k running jobs.  Doing some investigating, it looks like HCC is competing with GLOW and Engage for slots.

Special thanks to Purdue!

Barely a light load on the glidein machine.  This many jobs would have not worked on the old glidein machine, which was a Xen VM.

Of course with Condor, load and memory usage are more a function of # running jobs rather than jobs in queue.

Also, the new GlideinWMS gratia monitoring is picking up the usage:

Gratia DB Rates

In my recent discussions with NCSA on gratia, I found something that is probably obvious to other people, but I would still like to point out.

Status captured 6-8-11

According to the OSG PR Display, the OSG runs and accounts for 420,000 jobs daily.  That's 4.8 records a second added into MySQL JobUsageRecord table.  Also, for each job, summary data is updated in the summary table.  So that's at least 5 inserts/updates a second, but most likely many more.  All the while, we are able to interactively query the database (well, a replicated one) using pages such as the UNL graphs.

We also keep track of transfers, but they are summarized and aggregated at the probe before reaching the database.

So, in summary, kudos to the gratia operations team.

(NOTE: I may have forgotten some optimizations that we use.  But still, that's a lot of records)