The Journey Thus Far (Day 8)

Views

As I mentioned in the previous post, the first part of this project involved finding a way to make the Grids "Logging and Bookkeeping" (L&B) data accessible to a Google Gadget. Initially it was suggested that I used a python CGI script to simply parse the tab delimited data files provided. This data could then be used to the build a JSON object, using the Google Visualization API, which could be returned in response to a GET request sent to the CGI. However, upon receiving a sample of the L&B data it immediately became clear that this would not be a sensible approach. This was because for just one weeks worth of data the uncompressed file size exceeded several hundred MB. As such using a single CGI script to parse, process and return the data would result in all 882MB (or more if additional weeks are to be considered) of data being read in response to each GET request. This would be very computationally inefficient and result in large delays for the end user (it is also likely that the GET request would timeout before any data was returned).

I therefore determined that the best approach would be to load the L&B data into a database, which could then be queried to in response to a GET request and the required JSON object built from the database's response. The architecture I had planned for the system looked something like this:

For the database RDBMS I am currently using a the excellent open-source mySQL, although the L&B data only forms a single table. I think, although I haven't quite reached the current state of the project, I'll end this post for now, as I feel an explanation of the basic structure/layout of Google Gadget is needed before going any further.