Wednesday, January 16, 2013

Rendering with Bosco

Example image rendered using HCC's distributed renderer

At the Nebraska Holland Computing Center, we take pride in eating our own dog food.  So I want to highlight one of our uses of Bosco to enable transparent usage of our clusters.

HCC recently has made a push for enabling the non-traditional users of clusters.  Sure it's easy to show how physics can benefit from a set of clusters, but what about the media arts?  In this case we are working with a professor whose students render short movies using Maya.

Maya can utilize another Autodesk product called Backburner, which enables rendering across multiple nodes.  The challenge was to allow Backburner to operate in a shared cluster environment.  The standard method to use Backburner expects nodes dedicated solely to running the Backburner Server daemons.  This Master-Worker fits the traditional HTC model well, therefore we felt that Bosco would be a great fit to enable submission to our clusters.

Architecture Diagram of the BOSCO enabled render (Credit: Adam Caprez)
In the architecture shown in the image above, we run a central service, hcc-render that monitors the Backburner queue in order to submit the Backburner servers to the Bosco queue.  Bosco then submits a glidein to Tusker, which in turn runs the Backburner server and renders the scene.

In this case, since we are using Backburner server for the actual processing, the data is handled internally by Backburner.  In practice this means that data is stored on the Tusker file system which is then mounted by the professors local machine that is in turn mounted by the clients.

This is a classic example of deeply integrating HTC into the user's application.
The user only needs to click the 'render' button from within Maya on their workstation, and we handle all the rest, automatically.  This is only made possible because Maya has a sane back end renderer that is designed to run on Linux.  This architecture may not work for all commercial applications.

Even though the user isn't directly using Bosco, a primary goal of Bosco, this is an excellent use of Bosco to enable HTC workflows.

Bosco Download

Monday, January 14, 2013

Bosco 1.1.1 Release

Today I am pleased to announce that Bosco version 1.1.1 has been released. This is a patch fix for the 1.1 to address 2 issues that where affecting users.  The release is available on the Bosco Download Page.

Release notes for the 1.1.1 release are available on the OSG Twiki.

Bosco 1.1 was a major release with many new features. The 1.1 release notes are also available on the OSG Twiki.

On behalf of the Bosco Team,
Derek Weitzel
Dan Fraser
Marco Mambelli
Jaime Frey
Brooklin Gore
Miha Ahronovitz
Bosco Download

Improving Gratia's Web Interface

Over the winter break, I worked on improving the interface that most users use for OSG accounting.  When I returned from break, I worked with Ashu to integrate the changes with some recent changes he had made. The new interface runs on gratiaweb-itb.  The source for the new web page is hosted on github.

The first thing users will notice is the newly designed interface:
New OSG Accounting Interface
The updated interface brings the style of the website inline with that of MyOSG and OIM (or, close).  The design stayed close the the original, but the menu on the right has changed significantly.  First is just the style of the menu.  But also we added a new category, Campus and Pilot View.

In the Campus and Pilot View, we have some new graphs that show usage by GlideinWMS, Campus users, XSEDE users, and in the future, Bosco users.

Lets run through a quick example.  In this example, lets assume I'm a VO manager and want to see where my VO is running, how many hours, and who is running.

  1. Select the Pilot & Campus Accounting link.
  2. Scroll to the bottom, to the Refine View.
  3. Enter your VO name into the VO text box and hit enter.
This will pull up the custom page that shows usage for only your VO.  For example, if I look at the osg VO:
Usage by the OSG VO.
You can see from the graphs that the OSG VO has used ~80,000 CPU hours a day on the OSG.  Also, they are running at over 20 sites.  The sites at the bottom of the graph are listed in order of total hours (I am happy to see Nebraska resources as #3, #6, and #9).  

You can also see from the graph that usage at sites depends on the day.  Some days they get significant usage at the MWT2 (UChicago and IU), and some days they run a lot at Nebraska.

The new usage graphs are intended to help users, administrators, and VO managers view their usage.  I hope you find them as useful as we have in the past.

We hope that webpage is an improvement.  If there are any comments on further improvements, we are interested in your feedback.

Tuesday, January 8, 2013

Bosco 1.1 Release

In the last few months, I have outlined the new features of Bosco 1.1.
I am happy to announce that today we are releasing Bosco 1.1.  The official Release notes are available on the OSG Twiki.  It includes all of the features given above.  Try it out!