Wednesday, March 12, 2014

A HCC Dashboard with OSG Accounting

After the 2013 SuperComputing Conference, we found ourselves with a extra monitor at HCC.  Therefore, I set about creating a dashboard which can show the current status of HCC.

Creating the Dashboard

I have an interest in data visualization, and follow many blogs that show off new methods.  On one occasion, I saw Dashing mentioned.

Dashing is a dashboard framework made by Shopify for their own use, and released as open source.  It is mostly written in Ruby and CoffeeScript (a higher level javascript, if you can imagine).  It has a concept of jobs which fetch data and forward the data to the framework.  The data is sent to clients viewing the dashboard, where it is parsed by the Coffeescript and modeled with a combination of data bindings from batman.js, CSS with SCSS, and plain old HTML.

I wrote several jobs to retrieve data from numerous sources.  Most of the information is from HCC's local instance of OSG's Gratia accounting system.  The HCC Dashboard uses our gratia system for:
  • How many CPU hours where consumed on our resources.
  • Current usage by User (http://hcc.unl.edu//gratia/)
The job to retrieve the top user's also communicates with HCC's user database to retrieve college and department information.  The storage meters use an external probe on the clusters to periodically report the used storage space of our filesystems.

Each box is an instances of a widget.  A widget is a combination of HTML, SCSS, and CoffeeScript that are used to parse and present the data.

Current Dashboard Design

Most of the information on the dashboard is in the form of monitoring.  The current number of cores used on our resources and the top users widgets use Gratia monitoring information.  The networking graph uses Ganglia.

We also include a "Hourly Price on Amazon EC2" widget.  This combines the computing, storage, and networking costs (extrapolated from current values), and displays an expected price per hour on Amazon.  The computing is easily the most expensive component.

Who Uses it?

HCC uses it to display the current status of our computing center.  It is useful to see if anything is working incorrectly.  For example, we where able to spot problems on one of our clusters when the number of running cores decreased significantly, which was caused by the scheduler draining off a significant portion of the cluster in order for a single user to run a toy job.

The top users is also interesting for HCC researchers when the come into the offices.  They are able to see their own usernames on the big display, prominently displayed.


Growing collection of visualizations

More Information

The source for the dashboard is available on Github.  Also, the live instance of the dashboard is available here.

5 comments:

  1. Murdock
    Martell


    Murdock Martell provides finance and accounting solutions to established and emerging growth companies. The firm's clients include startups and Fortune 500 companies and it provides services throughout the San Francisco Bay Area. Twenty plus year industry veterans Leslie Murdock and Claire Martell launched their partnership in 2007. The two worked together for eleven years at the Murdock & Associates, a firm founded by Leslie Murdock in 1989 that became one of Silicon Valley's premier finance and accounting consulting firms before its sale to a national firm in 2000.

    Contacts Us:
    Murdock Martell, Inc.
    Claire Martell, 408-892-7108 (Media)
    CMartell@MurdockMartell.com
    www.MurdockMartell.com

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Nice blog and Nice Presentation!! Your blog is an academic standard blog which helps thousands of students worldwide. I have felt pleasure to visit your blog and read your informative topics.

    Like you, I have also started a Business,Accounting and Finance Blog for academia. I hope my blog will also serve some of the
    visitors who want to know knowledgiate topics..Anyway, thanks for your blog and good luck!!

    ReplyDelete
  4. MSN & Associates is a finest Chartered Accountants firms for Accounting Service in Delhi with 100% Satisfaction. Compare-Price, Reviews, Photos and Maps. Contact us- 9818089610
    More@ http://www.eindiacharteredaccountants.com/

    ReplyDelete
  5. A really interesting and a well written blog. Thank you so much for all your insights. As we all know here, how important is it to track records and simplify and automate the running of a firm's practice. I have been using similar software, CCH iFirm, a Dashboard for Chartered Accountant firms. With its highly dynamic, integrated and cloud based solutions, our workflow is more organized, able to allocate resources as and when required.

    ReplyDelete