preservation of seed flags in case of blocksize change

[mwalther]

Does obspy.mseed preserve the mseed flags (clipped, time lag, etc.) if several records are combined?

E.g. eight input data records with reclen 512 bytes should be recombined to one 4k record. One of the 512's has the clipped flag set. The resulting record should have the clipped flag set too.

Pseudo-code

>>> a=read("mseed-512")
>>> a[0].write("mseed-4k", reclen=4096, format="MSEED", encoding=...)

Unfortunately I have no sample data "at my fingertips"...

[barsch]

Short answer: No!

Long answer: No, we omit this information! AFAIR does libmseed not return such information - I know Lion wrote some stuff to preserve those quality information - but its all Python based and incredible slow. Maybe he states some more infos on that.

[krischer]

I am sorry. I must have overlooked this request.

As Robert already pointed out it is currently not supported and most likely never will be, because it would require a significant change of the current implementation.

The main reason is that all the additional information might change from record to record but obspy.mseed combines many of these records together to form contiguous data traces and only stores any additional information per trace. It is not feasible to store all records using Python because it would become very very slow and memory intensive. libmseed AFAIK allows access to the original header in binary form so it should be possible to extract the information using ctypes but that again would be very slow.

But it is actually fairly easy to write the data quality field by hand if the need arises (untested but should work in more or less this manner):

RECLEN = 4096
# Get a stream somehow, write is as MiniSEED and use a fixed record length.
stream.write(filename, format='MSEED', reclen=RECLEN)

# Now read the file again with numpy as unsigned 8 bit integers.
import numpy as np
mseed_file = np.fromfile(filename, dtype='uint8')

# Mofify byte 38 (double check!), the data quality flags, for all records.
# The SEED manual states that bit 1 of this field means 'Digitizer
# clipping detected'.
# Bit 1 of any integer set and all others unset means that it has the value
# 2 (00000010).
# Byteorder does not matter because it is just 1 byte.
mseed_file[38::RECLEN] = 2
# Write the file again.
mseed_file.tofile(filename)

If some other data quality flags also have to be set just change the 2 to whatever needed. The code should be, aside from the additional I/O required, pretty performant. Please remember that this sets the data quality flags for all records in the file. Reading the data quality flags works in a very similar way. I will actually update the getDataQualityFlagsCount method of libmseed to use the above method soon as I think it is much faster than the current implementation.

I do not think it makes sense to incorporate this code into obspy.mseed because it is a very specific request.

[mwalther]

I'd like to vote for an optional function for accessing these flags record-based (no matter of speed issues).

[megies]

Well, as Lion pointed out this would require most likely a lot of work and even more good tests and there are many things we want to go after already, so this probably gets a pretty low priority.

[mwalther]

I understand. But I like to add that everyone will face exactly this issue if one wants to build SEED volumes out of MSEED files (of course only if the record length differs and flags should be preserved).