Skip to content

Plugins

Blissdata plugins allow for creating new kind of streams. Plugins are quite flexible in the sense they add a layer of custom logic on top of conventional streams. We will cover some use cases in the next section, but plugins are not limited to these.

Common patterns#

All plugin streams provide the same API to the user, which is defined by the BaseStream class. However, their implementation generally falls into one of the following patterns:

Fallback#

A fallback stream is built on top of a conventional stream. It aims to catch any potential IndexNoMoreThereError that would occur when requesting indices that have expired on Redis. In that case, it will collect the missing points from a predefined place to complete the request transparently.

Example: Hdf5BackedStream is a fallback plugin stream. It is one of the built-in plugins packaged with blissdata. It collects data from a known HDF5 file in case of expired content in Redis. Obviously, it implies having some external service producing that file.

Reference#

Such stream only transmits references to data stored in another place. Reasons for not to store content directly inside a stream could be:

  • Not having enough space in Redis
  • Exposing an already existing storage
  • Using an higher bandwidth storage
  • Streaming dynamic objects (see ScanStream below)

Example: ScanStream is a reference plugin stream. It is a stream to transport scans, allowing to create scan sequences in a recursive way. Because scans are dynamic objects, only references are sent (scan keys) for them to be automatically loaded on receiver side by the plugin.

Status#

Status stream is quite similar to using references, except the last status is enough to locate all the data. Because there is no one-to-one relationship between data points and references, it may simply remove the overhead of sending tons of references on a high rate stream.

Example: LimaStream is a status based stream as each Redis message only tells for the highest index being available. No matter the detector is working on the kHz scale, we can only send a few updates per second, while the additional information to locate images on their storage is defined statically.

Plugins on publisher side#

On publisher side, using a plugin is done by choosing the appropriate stream class. As seen in the stream documentation, recipes are used to create streams. Each plugin provides a stream class that can create recipes.

from simple_plugin import SimpleStream

scan = ...
args = ...

# args are specific to each plugin, but it always returns a StreamRecipe
recipe = SimpleStream.recipe("foobar", *args)

# returns an instance of the plugin's stream, here a SimpleStream
stream = scan.create_stream(recipe)

# start the scan...

stream.send(data) # data should be of the type expected by this plugin

Plugins on receiver side#

On receiver side, Blissdata performs auto-discovery to find the corresponding plugin. It basically means you have nothing to do in particular, if the plugin is already installed.

In the event you are missing a plugin when loading a Scan, the faulty stream won't be loaded, but a MissingPluginStream will take its place. Any attempt to read that stream will raise an exception, suggesting you to install the missing plugin.

MissingPluginException: No plugin matching 'simple' entry point for stream 'foobar'

Plugins auto-discovery#

Blissdata plugins relies on Python entry points. You can find a precise explanation of what entry points exactly are in the setuptools documentation.

Each Python package can bring its own set of plugins by defining entry points. Here are for example the entry points of the blissdata's built-in plugins:

pyproject.toml

[project.entry-points.blissdata]
hdf5_fallback = "blissdata.streams.hdf5_fallback"
scan_sequence = "blissdata.streams.scan_sequence"
lima = "blissdata.streams.lima"
lima2 = "blissdata.streams.lima2"

The first line tells for the namespace, be sure to use blissdata for your entry points to be found: 

[project.entry-points.blissdata]

Then, each line associates a name to a module from which blissdata will import two things:

  • stream_cls: Stream class of the plugin.
  • view_cls: View class of the plugin.

For example, blissdata will load the lima plugin using something like this: 

from blissdata.streams.lima import stream_cls, view_cls

which point to that file in lima plugin:

blissdata/streams/lima/__init__.py

from .stream import LimaStream, LimaView  # noqa: F401

stream_cls = LimaStream
view_cls = LimaView

We now have seen how Blissdata can associate a name ("hdf5_fallback", "lima", ...) to some specific Stream and View classes. This name is stored in the stream's .plugin attribute for readers to know what plugin to use.

Create your own plugin#

Creating your own plugin implies not only to define an entry point, but to create associated Stream and View classes. These must be derived from BaseStream and BaseView respectively:

from blissdata.streams.base import BaseStream, BaseView, StreamRecipe

class MyPluginStream(BaseStream):
    def __init__(self, event_stream):
        NotImplemented

    @property
    def kind(self):
        NotImplemented

    @staticmethod
    def recipe(...) -> StreamRecipe:
        NotImplemented

    @property
    def plugin(self):
        NotImplemented

    def __len__(self):
        NotImplemented

    def __getitem__(self, key):
        NotImplemented

    @abstractmethod
    def _need_last_only(self, last_only):
        NotImplemented

    def _build_view_from_events(
        self, hl_index: int, events: EventRange, last_only: bool
    ):
        NotImplemented


class MyPluginView(BaseView):
    @property
    def index(self) -> int:
        NotImplemented

    def __len__(self) -> int:
        NotImplemented

    def get_data(self, start: Union[int, None] = None, stop: Union[int, None] = None):
        NotImplemented