Concatenating traces

'Istituto agronomico per l'oltremare, int., biblioteca, schedario 05' by I, Sailko. Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:Istituto_agronomico_per_l%27oltremare,_int.,_biblioteca,_schedario_05.JPG#/media/File:Istituto_agronomico_per_l%27oltremare,_int.,_biblioteca,_schedario_05.JPG

Without 'concatenating traces' our program will save each block separately. A block being 128 samples (our block length). This means the folder of all the samples can become 'messy', containing hundreds of small files, only with sequential sorting.

The solution to this mess is to combine the blocks into files containing a day's worth of blocks. Therefore we can quickly find a specific time within our folder, without saving too many files.

Programming

The majority of this revision is very similar to Jitter and sampling rate, therefore I will omit portions of the code, assuming you have them already.

We import all the same things as before, but now including 'os.path', as so:

import numpy
from obspy.core import read,Trace,Stream,UTCDateTime
import Queue
from threading import Thread
import os.path
import subprocess

The set-up of the ADC is the same, and we declare the same variables, but now we specify the folders where we store the data (separate for the samples and jitter now, as the names are just dates, not with anything to identify them as samples or jitter).

#directories for data
mseed_directory = 'mseed'
jitter_directory = 'jitter'

The reading function is identical to before. But the saving function is very different. We don't need to declare 'block_id', so get rid off the three files (incl. the comment if you have it), straight after the 'def'.

def save_data():
   while True:
      if queue.qsize()>=block_length:

The rest up to and including the stream and trace functions is identical. Although the saving process has changed a lot. We can get rid of the three lines, the two for writing and the last for incrementing block_id.

Instead, to start we declare a variable of the name of the file. This is the date of the 'starttime'. We also declare the name of a temporary file which will be used in the saving process. We must next check whether the file exists (using the 'os.path' library we imported), as if this not the first block of the day, it must be merged. This happens next, by writing the block as the temporary file, then concatenating them. This is done using the 'cat' command, which is used in python with the subprocess library. If you have never used 'cat' or 'rm' before, it is appending the temp_file to the File, then removing the temp_file afterwards. Otherwise this could be the first block of the day, so there could be no file to combine with, therefore we have the 'else' to write just that stream to file.

#write sample data
File = mseed_directory + str(sample_stream[0].stats.starttime.date) + '.mseed'
temp_file = mseed_directory + ".temp.tmp"
if os.path.isfile(File):
   #writes temp file, then merges it with the whole file, then removes file after
   sample_stream.write(temp_file,format='MSEED',encoding='INT16',reclen=512)
   subprocess.call("cat "+temp_file+" >> "+File,shell=True)
   subprocess.call(["rm",temp_file])
else:
   #if this is the first block of day
   sample_stream.write(File,format='MSEED',encoding='INT16',reclen=512)

We also do this for the jitter, it is identical apart from the directory, the data type, and the stream to save.

#write jitter data
File = jitter_directory + str(jitter_stream[0].stats.starttime.date) + '.mseed'
temp_file = jitter_directory + ".temp.tmp"
if os.path.isfile(File):
   #writes temp file, then merges it with the whole file, then removes file after
   jitter_stream.write(temp_file,format='MSEED',encoding='FLOAT32',reclen=512)
   subprocess.call("cat "+temp_file+" >> "+File,shell=True)
   subprocess.call(["rm",temp_file])
else:
   #if this is the first block of day
   jitter_stream.write(File,format='MSEED',encoding='FLOAT32',reclen=512)

Finally we start the thread and start reading the data like before.

worker_sample = Thread(target=save_data)
worker_sample.start()
read_data()

Next step... Apache and Ring Server

Now the program is complete! You can edit it slightly to your needs, such as commenting out saving the jitter, to save space, or changing the block_length.

It is very useful, although not essential, to be able to access the data through the Wi-Fi, which can be done with a server — see Apache and Ring Server.