Select Page

nzseis.phy.auckland.ac.nz

is a linux box (currently -31 October- running ubuntu 12.04 lts), configured as a fairly standard LAMP.

It has been the site of various experiments in capturing and managing data from the field, and retains artefacts of some of these explorations.

In particular, I built and installed the ringserver software provided by IRIS, which was to be used to capture data feeds from the stations we placed in schools. This aspect of the service was never developed, as the jAmaseis software (seismograph data logging/sharing created through IRIS) and the server appeared to be incompatible, and there were simply more urgent things to sort out.

NZseis system design, April 2014

NZseis system design, April 2014

The earliest version of the system design shows how it was intended to work.

services

As it is configured at the moment, nzseis.phy sits in the middle of a collection of seismic stations placed in various schools across the country (and also collects information from some overseas stations). The network as it stands is probably best visualized in our earthquakes and stations map –Last 50 earthquakes around New Zealand.

Key services provided by nzseis.phy are:

  • image retrieval/receipt
    • 5 minute cycle to pick up images placed on the ftp server (winfiles.fos), or uploaded through the pyjAmaseis service.
  • image collation
    • Imagery is stored in folders associated with each station, the most recent is available as latest.png, the older ones are renamed and 1 image per hour of the day retained.
  • data collection
    • A number of the machines synchronise data using owncloud -this practice was discontinued because of the load on the file system, and the fact that we were not doing anything with the data we retrieved, but a number of the earlier stations still share the data they collect.
    • One of the pyJamaseis services includes a sac upload interface -this allows a station to push traces it collects to the server for analysis on other software. This is not being used at the time of writing (31 Oct), but the mechanisms are running.
    • Every 10 minutes or so, earthquake data is collected from one of the published data feeds at GeoNet (last 50 earthquakes).
  • data transformation
  • supporting web information

Timers

a number of cron jobs constitute the cogs that keep the services flowing.

The  Science IT user

$ crontab -l
9,19,29,39,49,59 * * * * /var/scripts/getimages.sh >/var/scripts/getimages.out 2>&1
6,16,26,36,46,56 * * * * /var/scripts/pygetimages.sh >/var/scripts/pygetimages.out 2>&1
2,12,22,32,42,52 * * * * /var/scripts/downloadeqs.sh >/var/scripts/downloadeqs.out 2>&1

respectively:

  • load/collate latest station images from FTP or HTTP repositories
  • load/collate the images uploaded by pyjamaseis clients
  • download the earthquake data from GeoNet, create the image and kml files

The apache user

$ sudo crontab -u <apache user> -l
*/15  *  *  *  * php -f /var/www/owncloud/cron.php

Used by owncloud to keep the gears turning.

Root

$ sudo crontab -l
0 6 * * * /var/backup/backup.sh >/var/backup/backup.out 2>&1

A back up -essentially a snapshot of important parts of the file system, See FBS – a simple backup service.

The clients

The client machines also have timers that can affect the server

jAmaseis in its normal state, will upload a screen grab every 10 minutes. These go to winfiles via FTP upload.  nzseis.phy graps these via a specifically configured https link.

If something happens between nzseis.phy and winfiles, images may not be updated in the stations folders.

Where owncloud is still being used, the updates are continuous (while the client and server are active). These are direct communications via https. If we are synchronising a seismometer data folder, then we can see traffic of up to 250MB/day!

Where we have had to implement a workaround for FTP not working (the new networks in schools block things like ftp and ssh from within the school to the internet), we use https to push up image updates via the interfaces created for the pyJamaseis tool.

 

pyjAmaseis

py-jAmaseis is a data collect/sharing tool being developed as part of the whole Rū project -it is a python based version of the key tool sets provided by jAmaseis and its predecessors (Amaseis, etc). At the time of writing (31 Oct), this was still in development, though the server based infrastructure is functional, albeit a bit rough due to its rapid development.

File system spaghetti

Two key file systems:

/dev/sda1 on / type ext4 (rw,errors=remount-ro)
/dev/sdb1 on /media/nzseis type ext4 (rw)

and a bunch of links

lrwxrwxrwx  1 root       root         24 Aug 22 20:55 backup -> /media/nzseis/var/backup
lrwxrwxrwx  1 root       root         26 Aug 22 21:17 owncloud -> /media/nzseis/var/owncloud
lrwxrwxrwx  1 root       root         25 Sep  2 11:20 sacdata -> /media/nzseis/var/sacdata
lrwxrwxrwx  1 root       root         21 Aug 22 20:50 www -> /media/nzseis/var/www

The result of not being able to resize the first volume 🙁

Things that need manual care

The file system fills up 🙁

All of the images, data collection via owncloud, etc, result in an accrual of stuff. Every now and then, the stuff needs to be packaged and moved -I have been moving it back to winfiles via FTP (manually).

Ideally we reduce the number of images we save, but I simply have not gotten around to writing the tweaks to the image collation code run by the crons mentioned above -an exercise for the administrator who inherits this?

The culprit is owncloud. In spite of reducing the quota available to each, some stations keep large collections of previous versions. Manual processes for managing this involve logging on as each owncloud user and moving the data to the same place I store the images.

12.04 LTS

Needs to be updated to 14.04.1 (or better). Key parts of the current operating system are no longer supported by standard releases of updates. The plan was to snapshot this system, and attempt the upgrade on a clone before committing to doing it on the original. There is nothing in the system (that I am aware of) that will break as a result of the upgrade, but there will invariably be some necessary tweaks to configurations as a result of new defaults being put in place.

Given that I am not averse to compiling code from the sources used by the Ubuntu community, the more spectacular recent security issues have been dealt with.

Given the way I capture key configuration and content each day as part of a backup, it would also be possible to merge the configuration into a new host (which was part of a longer term plan for this service).

The clients

The clients pose an interesting problem -how to manage a bunch of computers at a distance across a variety of configuration states. This has been an ongoing interactive exercise, where the realities of IT in schools meets the aspirations of the researcher. Each issue tends to be brokered by Kasper van Wijk (the primary investigator), and requires support akin to developing teaching material in many cases -it’s fun! enjoy!

The original systems were running OS X 10.8, the current systems are running OS X 10.10. There have been updates to the java 6 runtime required by jAmaseis, changes to network configurations, new versions of jAmaseis, and shifts in the vision Kasper has for the clients as we have moved forward. Keep agile, allow for change…

 

 

Ka kite

Matiu Carr, October 2014

 

 

 

 

 

Skip to toolbar