Moving to a new Blogging Website

I wanted to write a quick note.  I am moving to a new blogging platform at https://djw8605.github.io/.  Please update any RSS or Atom feeds you may have.

Querying an Elasticsearch Cluster for Gratia Records

For the last few days I have been working on email reports for GRACC, OSG's new prototype accounting system.  The source of the email reports are located on Github.

I have learned a significant amount about queries and aggregations for ElasticSearch.  For example, below is the query that counts the number of records for a date range.

The above query searches for queries in the date range specific, and counts the number of records.  It uses the Elasticsearch-dsl python library.  It does not return the actual records, just a number.  This is useful for generating raw counts and a delta for records processed over the last few days.

The other query I designed is to aggregate the number of records per probe.  This query is designed to help us understand differences in specific probe's reporting behavior.

This query is much more complicated than the simple count query above.  First, it creates a search selecting the "gracc-osg-*" indexes.  It also creates an aggregation "A" which will be used later to aggregate by the ProbeName field.

Next, we create a bucket called day_range which is of type range.  It aggregates in two ranges, the last 24 hours and the 24 hours previous to that.  Next, we attach our ProbeName aggregation "A" defined above.  In return we get an aggregation for each of the ranges, for each of the probes, how many records exist for that probe.

This nested aggregation is a powerful feature that will be used in the summarization of the records.

Fedora copr: Slurm per job tmp directories!

A Slurm site on the OSG was having problems with /tmp filling up occasionally.  Our own clusters have a Slurm Spank plugin that creates a per job tmp directory.  This per job directory is useful for grid jobs which may be preempted without time to clean up after themselves.  The motivation for this plugin came from HTCondor, where each job has it's own job directory.

In order to share this plugin, I started a few Github repos (slurm-tmpdir and slurm-plugins-lua) for the plugin and dependencies.  Next, I used Fedora's Copr build system to build a repo that I could share with others, the Slurm-Plugins repo.

Copr Slurm-Plugins Project

I'm sure that I don't have an exhaustive list of plugins in my repo.  Actually, I only have one.  But I can always add more by simply adding a Github repo.

Copr was very easy to use to build my packages, and it works for all OS's that I care about.  I'm very happy with how Copr worked for sharing Nebraska's Slurm-Plugins.