Querying an Elasticsearch Cluster for Gratia Records

For the last few days I have been working on email reports for GRACC, OSG's new prototype accounting system.  The source of the email reports are located on Github.

I have learned a significant amount about queries and aggregations for ElasticSearch.  For example, below is the query that counts the number of records for a date range.

The above query searches for queries in the date range specific, and counts the number of records.  It uses the Elasticsearch-dsl python library.  It does not return the actual records, just a number.  This is useful for generating raw counts and a delta for records processed over the last few days.

The other query I designed is to aggregate the number of records per probe.  This query is designed to help us understand differences in specific probe's reporting behavior.

This query is much more complicated than the simple count query above.  First, it creates a search selecting the "gracc-osg-*" indexes.  It also creates an aggregation "A" which will be used later to aggregate by the ProbeName field.

Next, we create a bucket called day_range which is of type range.  It aggregates in two ranges, the last 24 hours and the 24 hours previous to that.  Next, we attach our ProbeName aggregation "A" defined above.  In return we get an aggregation for each of the ranges, for each of the probes, how many records exist for that probe.

This nested aggregation is a powerful feature that will be used in the summarization of the records.

Part 1: Features of the Chrome App - Embedding

In my previous post, I introduced my OSG Chrome App.  In this post, I want to introduce one feature of the chrome app, Embedding profiles.

Part of my morning routine is checking multiple websites for the status of the resources at Nebraska.  For example, I check our dashboard page that I've written about before.  And I check the HCC GlideinWMS status page.

The HCC GlideinWMS status page shows only the monitoring information.  In order to see any accounting data, I have to navigate to the GratiaWeb site and filter for hcc usage.  Instead I want to view the accounting data on the same page as I am viewing GlideinWMS status.  And so I added the ability to Embed a set of graphs using the Chrome App.

Creating a HCC GlideinWMS Profile

First, I need to create a glideinwms profile for hcc.  I have 2 questions that I want answered, who is running, and where are they running.  In this case, I am playing the role of a VO Manager.  I want to see usage filtered by VO, specifically my VO, HCC.  The VO Manager role gives me 2 graphs by default, total usage per VO, and where the VO is running.  But it doesn't include what users are running in my VO.  Therefore, I want to add a new graph, the Glidein (since HCC uses GlideinWMS) per user graph, which is on gratiaweb.

To add a graph, I followed the documentation to add the Glidein per user graph.  I copied and pasted the graph URL into the box, and it added the graph to my list.

HCC Usage with new graph.  Not much usage...

Embedding the Profile

Now that I have the profile showing the data I want to see, I want to embed this data in my webpage.  I followed the documentation again, for sharing the graphs and embedding it.

I clicked the green share button at the top, and entered the name and description of the profile.  Then I submitted the profile for sharing.  It returned an embed link that I can use on the webpage.

Share profile showing embed URL

Once I have this embed URL, I can copy / paste that HTML into the HCC GlideinWMS page and it will show HCC data every time I visit.

HCC Usage embedded on our own website

And that's it.

Links

• Chrome Web Store
• OSG Usage App Documentation
• Source

OSG Usage Chrome App

Hi, I'm Derek Weitzel.  You may remember me from such side projects as:

• OSG Android App
• Redesign of Gratiaweb Interface
• Native Mac Installer for Bosco
• LiveUSB for the OSG Client
• Hadoop Visualization
• ...

On today's episode, I will introduce a Chrome Application designed for the OSG's accounting system.

OSG Usage Chrome Application

The OSG Usage Viewer is a packaged chrome app designed to display accounting graphs from Gratia, the OSG's accounting system.  It also features:

• Intuitive data filtering
• Sharing of sets of graphs

The app allows you to add graphs from the OSG's Gratia web interface, to manipulate the data filters, and share with others.  Full documentation of the app is available.  Download the app today!

Links

• Chrome Web Store
• OSG Usage App Documentation
• Source

Improving Gratia's Web Interface

Over the winter break, I worked on improving the interface that most users use for OSG accounting.  When I returned from break, I worked with Ashu to integrate the changes with some recent changes he had made. The new interface runs on gratiaweb-itb.  The source for the new web page is hosted on github.

The first thing users will notice is the newly designed interface:

New OSG Accounting Interface

The updated interface brings the style of the website inline with that of MyOSG and OIM (or, close).  The design stayed close the the original, but the menu on the right has changed significantly.  First is just the style of the menu.  But also we added a new category, Campus and Pilot View.

In the Campus and Pilot View, we have some new graphs that show usage by GlideinWMS, Campus users, XSEDE users, and in the future, Bosco users.

Lets run through a quick example.  In this example, lets assume I'm a VO manager and want to see where my VO is running, how many hours, and who is running.

1. Select the Pilot & Campus Accounting link.
2. Scroll to the bottom, to the Refine View.
3. Enter your VO name into the VO text box and hit enter.

This will pull up the custom page that shows usage for only your VO.  For example, if I look at the osg VO:

Usage by the OSG VO.

You can see from the graphs that the OSG VO has used ~80,000 CPU hours a day on the OSG.  Also, they are running at over 20 sites.  The sites at the bottom of the graph are listed in order of total hours (I am happy to see Nebraska resources as #3, #6, and #9).  

You can also see from the graph that usage at sites depends on the day.  Some days they get significant usage at the MWT2 (UChicago and IU), and some days they run a lot at Nebraska.

The new usage graphs are intended to help users, administrators, and VO managers view their usage.  I hope you find them as useful as we have in the past.

We hope that webpage is an improvement.  If there are any comments on further improvements, we are interested in your feedback.