Proposal: Introduction of Interfaces and Plug-ins

[barsch]

Hi all,

I'm really unhappy about our plug-in structure - currently we are not using all the power we could use by introducing interfaces.

A common pattern in Python is using zope.interfaces - interface classes essentially are only templates describe the behavior of an object by containing useful information about the object, e.g. a simple example:

from zope.interface import Interface, Attribute

class IWaveformReader(Interface):
    """
    General documentation ...
    """"
    def readWaveform(filename, param1, param2, ...):
        """
        General description about reading waveforms, params etc.
        """

    format = Attribute("""
        The name of the waveform format.
        """)

    allowed_byteorders = Attribute("""
        A list of allowed byte orders containing '<' and/or '>'.
        """)

class IWaveformWriter(Interface):
    """
    General documentation ...
    """"
    def saveWaveform(filename, param1, param2, ...):
        """
        General description about saving waveforms, params etc.
        """

    format = Attribute("""
        The name of the waveform format.
        """)

    allowed_byteorders = Attribute("""
        A list of allowed byte orders containing '<' and/or '>'.
        """)

class IEventReader(Interface):
    """
    Blah
    """
    ...

class IEventWriter(Interface):
    """
    Blah
    """
    ...

class IWaveformFetcherRESTWebService(Interface):
    """
    General interface for fetching waveforms via RESTful web services.
    """
    ...

...

Those interface will be defined on one single place and contain only description about the API no code itself. (Note the missing self within the class methods)

Now after declaring such interfaces it is possible to write modules using multiple interfaces in a single class, e.g. a waveform format named NEWFORMAT contains events and waveforms but we support only reading in a certain byteorder.

from zope.interface import implements

class NEWFORMATReader(object):
    implements(IWaveformReader, IEventReader)

    format = 'NEWFORMAT'
    allowed_byteorders = ['<']

    def readWaveform(filename, param1, param2, ...):
        ...
        if kwargs['byteorder'] not in self.allowed_byteorders:
            raise
        ...
        return stream

    def readEvents(filename, param1, param2, ...):
        ...
        if kwargs['byteorder'] not in self.allowed_byteorders:
            raise
        ...
        return events

Later on we want to extend this format with a event writer - ok so we just add the Interface IEventWriter and add the missing method for that.

class SomeNewFormatReader(object):
    implements(IWaveformReader, IEventReader, IEventWriter)

    ...

    def writeEvents(filename, param1, param2, ...):
        ...

Pros:

• Zope.interface allows easy discovery of the functionality of a package/module (without modifying the setup.py) by just searching over the applied interfaces.
• Plug-ins using interfaces can be validated against the given interfaces - e.g. check if all required methods and attributes are existing etc.
• Extending existing modules with new functionalities is pretty easy by implementing the interface and methods to the module - no need to modify the setup.py anymore
• unifying the plug-in structure for other stuff such as event reading/writing, fetching stuff from web services, header manipulation without interfering with the actual data etc.

Cons:

• However this a complete overhaul of our existing plug-in structure (core.py files within the waveform import/export modules) and would require some effort to get everything done
• Interface definition must be well-thought-out

I'd like to hear opinions about this proposal. Please feel free to comment and/or criticize.

[beyreuth]

I think it's a good idea. We should also think if we could use this architecture to plugin the signal functions to the stream objects, which currently is done by hand for each function individually (e.g. stream.filter).

[megies]

Well, not much to add.
Of course it's a good idea. I'm not sure at which point it would be the right time to go after it, though.. before taking a shot at the event structure? or after?
Honestly, the slight differences in all the client modules have been going around in my head a lot of late. I was thinking about adopting a structure in core (like the read() and write() we have for the waveform plugins right now) where you just specify which server type to use.
And with the signal processing routines I also agree there is room for improvement. And it should be pretty straight forward (e.g. feeding in a trace and making sure to get one out with same data.shape for instance).

However, we should agree on a clear priority for all the construction areas that seem to spring up like mushrooms currently. Better to do one big thing at a time..

[barsch]

to much to rewrite - also using zope.components/zope.interfaces is not faster than current implementation