The Journey Thus Far (Day 8)

As I mentioned in the previous post, the first part of this project involved finding a way to make the Grids "Logging and Bookkeeping" (L&B) data accessible to a Google Gadget. Initially it was suggested that I used a python CGI script to simply parse the tab delimited data files provided. This data could then be used to the build a JSON object, using the Google Visualization API, which could be returned in response to a GET request sent to the CGI. However, upon receiving a sample of the L&B data it immediately became clear that this would not be a sensible approach. This was because for just one weeks worth of data the uncompressed file size exceeded several hundred MB. As such using a single CGI script to parse, process and return the data would result in all 882MB (or more if additional weeks are to be considered) of data being read in response to each GET request. This would be very computationally inefficient and result in large delays for the end user (it is also likely that the GET request would timeout before any data was returned).

I therefore determined that the best approach would be to load the L&B data into a database, which could then be queried to in response to a GET request and the required JSON object built from the database's response. The architecture I had planned for the system looked something like this:

For the database RDBMS I am currently using a the excellent open-source mySQL, although the L&B data only forms a single table. I think, although I haven't quite reached the current state of the project, I'll end this post for now, as I feel an explanation of the basic structure/layout of Google Gadget is needed before going any further.

A Brief Introduction

Over the coming weeks I will be developing one or more iGoogle Gadget with the aim a making the monitoring data recorded by "the Grid" more accessible. Whilst there are currently a range of tools available for visualising the Grid monitoring data "it can still be quite difficult to find answers to many straightforward questions". I will be undertaking this project at the University of Birmingham over a 10 week period, which will hope result in the production of a genuinely useful grid monitoring tool.

For the benefit of anyone who happens to stumble upon this blog, who does not have a physics background, "the Grid" is large distributed network of computers predominantly used by particle physics researchers including those involved in the LHC experiments at CERN. The Grid has a hierarchical design consisting of several higher tier regional centers, each connected to many lower tier nodes, totaling "more than 140 computing centres in 34 countries". The "distributed grid" approach was selected due to the large computing requirement of the LHC's data analysis.

In this project I will specifically working with the data recorded by the grids "Logging and Bookkeeping" service inconjunction with the various "Workload Management Systems" (WMS). This data consists of 37 fields of information relating to each "job" submitted to the Grid. Initially I will be dealing with the issue of making this data available in such a form that it can easily be accessed and read by the the Google Visualization API. Once I have achieved this I should be able to move on to creating gadgets to display this data in the most useful and informative way.

So far I have been working on this project for 7 days. I intend to follow up this post with a summery of my progress thus far, as well as a regular series of progress reports.