Installing your own gratia webpage

In the OSG, we use a technology called Gratia for our accounting.  Every single job and data transfer on the OSG is accounted for in the database.  Of course, this is a lot of data, ~500,000 jobs and 1.2M transfers just today.  Therefore, we have a simple web interface to visualize this data.

Here's the quick instructions of how I setup my own version of it:

1. Install a newer version of python-cherrypy from rpmforge.  EPEL version is not new enough.
   

   rpm -Uvh http://rpms.arrfab.net/rpmforge/packages/python-cherrypy/python-cherrypy-3.1.2-1.el5.rf.noarch.rpm 

2. Install the OSG repos.
3. Install the Metrics RPM:
   

   yum install OSG-Measurements-Metrics-Web -y

4. Copy /etc/DBParam.xml.rpmnew to /etc/DBParam.xml
   

   cp /etc/DBParam.xml.rpmnew /etc/DBParam.xml 

5. Now, you can edit DBParam.xml file to point to your own gratia databases.  For example, at Nebraska, we have an instance that points to our own Nebraska gratia server.  This way we can see only local usage.  To use the OSG's, you will need to use the readonly account.  Replace all of the ******'s with 'reader'.  In VIM, you can do:
   

   :%s/\*\*\*\*\*\*/reader/g

6. The website relies on a set of static graphs that are updated every few hours.  They have to be saved and served by the systems http server.  So install the http server:
   

   yum -y install httpd

7. Make the directory for the static graphs to be saved into:
   

   mkdir -p /var/www/html/gratiastatic

8. Configure the static graph generator to put the images in this directory, and to generate the images from the gratia instance.  You will need to change both the Source and Dest.  The configuration is in /etc/osg_graphs.conf:
   
9. Change the static graphs location in the DBParam.xml:
   

   <staticfilehostname value="http://10.148.2.148/gratiastatic"> </staticfilehostname>

10. Start the services:
    

    service httpd start

    service GratiaWeb start

Then, you should have a functioning gratia web instance.

Running private instance of gratia web

In NY for CHEP

Hanging out in Times Square

I'm in New York this week for CHEP 2012.  I'll be writing about my experiences here.  Stay Tuned.

Developments at Nebraska

I thought I would do a quick post about recent developments at Nebraska.

Tusker & OSG

New Tusker Cluster

We recently received a new cluster, Tusker.  It is our newest in a line of clusters that prioritize memory per core, and cores per node over clock speed.  Therefore, the cluster is 104 nodes, 102 of which are 64 core, 256GB nodes.

The goal of this cluster is to enable higher throughput of local user jobs while enabling backfilling of grid jobs.  The current breakdown of local and grid jobs can be found on the hcc monitoring page.

A common complaint among our local users is interference between processes on the nodes.  To address this, we patched torque to add cgroups support for cpu isolation.  Memory isolation should come into production in the next few weeks.  This will affect grid jobs by locking down their usage to only a single core.

Nebraska's goal is to support all OSG VO's, and give them equal priority (albeit lower than local users).  All OSG VO's are welcome to run on Tusker.


Nebraska's Contribution to OSG Opportunistic Usage

Opportunistic usage by Site (source)

Nebraska resources have become the largest contributor to opportunistic resources.  Easily over 1/4 of opportunistic usage is happening at Nebraska.  We are #1 (Tusker), #2 (prairiefire), and after adding Firefly's different CE's, #7.  We are very proud of this contribution, and hope it continues.

Stay tuned, the next year should be exciting...

BOSCO + Campus Factory

Checklist while implementing CF + Bosco integration

For the last several months, the campus infrastructure team has worked on software that will help users create a larger, more inclusive campus grid.  The goal largely has been to make the software easier to install and expand.

Integrating the Campus Factory (which is already used on many campuses) with Bosco has been a key goal for this effort.  Last week, I finally integrated the two (Beta Install Doc).  This will have many benefits for the user over both the current Campus Factory and current Bosco.

Feature	Campus Factory	Bosco v0	Campus Factory + Bosco
Installation	Large installation/configuration instructions. Install Condor and campus factory on every cluster.	Install on a central submit node. Configuration is handled automatically.
Adding Resources	Install Condor and the campus factory on every cluster. Configure to link to other submit and campus factory clusters.	Run the command bosco_cluster -add Installation and configuration handled auto-magically.
File Transfer	Using Condor file transfer, can transfer input and output.	Manually with scp by the user. No stderr or stdout from job.	Using Condor file transfer, can transfer input and output.
Job State	Accurate user job state.	Delayed user job state. No exit codes from the user jobs.	Accurate user job state.

As you can see from the above table, the combined Campus Factory + Bosco takes the best from both technologies.

OSG AHM 2012

This years all hands meeting was a great success!  There where a few sessions that I really enjoyed.

AHM Pictures

Campus Caucus
There where many user engagement people there.  I believe that we reached a consensus that there isn't much an engagement community.

For example, Wisconsin has a great method for distributing and running MATLAB and R application on the OSG, but there has been no knowledge transfer to other engagement folks.  I know a few UNL users that have wanted to run MATLAB on our resources.  If we could move a HCC MATLAB workflow to the Grid, I believe that would be a great success.

I completely agree that there is no Engagement 'community'.  But I think that's true of most of the OSG.  Though, there have been recently many improvements.

• I think the centralized Jira has helped tremendously.  It's very easy to see what other people have been working on and even the general direction of progress.  Though this only works for OSG 'employees' and OSG projects.
• The OSG blogs have been successful for the technology group to explain what they are working on.  Though, I wish they had shorter and more often posts.

I hope that the blogs can be a way to spread the OSG Engagement activity.  It's also a great way to point to code and work that is being done.  Also, blog posts shouldn't be limited to things that the author is doing, but could point to what other people are doing.  For example, I knew nothing about the Rosetta people at Wisconsin until my poster was setup next to theirs and was able to have a conversation.  It would have been great to see some information that of what they where doing outside of the once a year meeting.


Science on the OSG

Talk

I thought this talk by Frank was great.  I felt he had the same feeling that we where all feeling, that Protein processing was becoming a very large user of the OSG.  We've seen this at HCC with both CPASS, CS-Rosetta, and Autodock.  

Walltime usage for non-HEP

Frank also pointed out a graph of usage.  At the end of the graph, there seems to be a plateau.  Possibly we are hitting opportunistic resource limits?

Here's an updated usage graph (source):

Walltime usage for non-HEP updated

 The thing to note is the explosive growth of GLOW VO.  Their usage has increased dramatically recently.

OSG in 2017

Talk

I really liked to see what people thought the OSG would look like in 2017.

Chander's prediction that people will come to us to use the OSG.  I believe this will take a critical mass of users.  I think we have a good product to sell, we just need publicity.  

Chander's comment on data is also important.  But I believe the problem with data isn't necessarily storage, but it's access to the data.  Take for example Dropbox.  For free, they offer very little storage.  The main advantage is that it's accessible from anywhere, laptop, desktop, iphone, web...  I think a uniform data access method can get us a lot further than distributed storage.

Alain's prediction that we will be using more community software.  This will take a large effort to be part of the distribution's community.  I foresee us contributing packages, patches, and effort to Fedora EPEL and possibly Ubuntu.  I think we are making great strides with the packaging, and would like to continue injecting us into the Fedora community.

Nebraska Campus Infrastructure

Talk

Of course, my talk is worth looking at.

This post ended up larger than I was hoping for.   Oh well.

Burning the LiveUSBs

In my last post, I talked about the OSG LiveUSBs.  Now that the conference is next week, I have started burning the USB with the image.

USB Key Piles

I burned 4 USBs at a time, using the script below.  Parted didn't work, never really found out why.  The symptoms where that the USB would not boot, but they where readable by Macs.  So I scripted fdisk.

Building an OSG-Client LiveUSB

Nebraska/OSG USB keys to be distributed at OSG-AHM 2012

Since we have started using RPM's for osg software, I've been interested if it where possible to make a LiveUSB of the client.  Due to the great documentation provided on the Scientific Linux LiveCD page, along with the CentOS LiveCD page, I've created a OSG Client LiveUSB that will be put on the keys.

Desktop of OSG Client Live CD

From the picture, notice links on the desktop to OSG User Docs, OSG LiveCD Docs, and How to get a certificate.

Live image creation
The live image creation was done using the livecd-tools package.  I used a Fedora 16 instance on the HCC private cloud to make SL6 images.  The kickstart file used can be found on github.

What's Installed
The goal of the LiveUSB is to give an easily deliverable demo of the OSG-Client, therefore only the OSG-Client and Condor are installed.  The LiveUSB has some persistant data storage area, but not much.

Tools are installed in order to install the live image, including the OSG-Client components, to the local hard drive.  Researchers can then easily have a node up and running.

Also, people that know how to run virtual machines on their computers can easily create a virtual machine with the OSG-Client from this USB.  Just boot from the USB, and click on the Install to Hard Drive icon on the desktop.

These keys will be distributed to attendees of the OSG All Hands meeting.

I am open for suggestions on what should be on the LiveUSB.  The image is not final yet.

UPDATE:
Link to Current ISO: OSG-SL6.2-x86_64-LiveUSB.iso