Publishing

Data produced by BLISS sessions is published into Redis (RAM storage).

In Redis, data is stored for a limited period of time (1 day by default) and for a limited amount (1GB by default).

Experiment and data files structure¶

A complete experiment can be seen has a succession of measurements with different samples in different conditions.

In a Bliss session, the experiment can be seen as a tree, where the trunk is the session itself and where each measurement performed for a given sample and experimental conditions is a branch.

As an example, let’s consider:

two samples: ‘sample1‘ and ‘sample2‘
a session named ‘test_session‘
several measurements with the two samples
a measurement that consists in scanning one of the samples by moving a motor ‘roby‘ and measuring the value of a counter ‘diode‘ along the scan.

The sample scan is performed with the following command (for details, see: scan commands):

ascan(roby, 0, 9, 10, 0.1, diode)

To start the session:

bliss -s test_session

Format data saving path for the experiment:

While starting a session for the first time, a directory with the same name as the session is created and the scan data is stored in this directory.

As two different samples will be scanned, one sub-directory per sample will be created. To do that, the SCAN_SAVING object has to be used. The data saving path is customized by adding a new parameter ‘s_name‘ usable in the template of the PATH.

SCAN_SAVING.add('s_name', '')                # add a parameter named "s_name".
SCAN_SAVING.template = '{session}/{s_name}/' # modify data saving path template.
SCAN_SAVING.s_name = 'sample1'               # set value of parameter "s_name".

Perform a first measurement:

ascan(roby, 0, 9, 10, 0.1, diode)

Perform a second measurement:

ascan(roby, 0, 9, 10, 0.1, diode)

Change the data saving path for measurements on sample2:

SCAN_SAVING.s_name = 'sample2'

Perform a measurement:

ascan(roby, 0, 9, 10, 0.1, diode)

For this experiment the files structure inside the session main folder is described by the following tree:

Screenshot

The measurements data can be accessed by reading the content of the hdf5 files (see silx, pymca). One file usually contains multiple scans data.

Screenshot

Experiment and Redis data structure¶

In parallel of the in-file data storage describe above, Redis stores the data in RAM memory as soon as it is produced. Therefore retrieving data from Redis allows a fast and live access to the data.

Sessions accessing

bliss.data.node.sessions_list() gives access to the list of sessions having data published in Redis. It returns a list of DataNodeContainer objects.

DEMO [6]: import bliss
DEMO [7]: bliss.data.node.sessions_list()
 Out [7]: [<bliss.data.node.DataNodeContainer object at 0x7f9e83ebab00>,
           <bliss.data.node.DataNodeContainer object at 0x7f9e83ffb828>]

DEMO [8]: bliss.data.node.sessions_list()[0].name
 Out [8]: 'demo'

Redis stores the data as a flatten list of (key:value) pairs but hopefully the bliss.data.node module provides a simple interface to access the data in a structured way that reflects the session structure.

from blissdata.data.node import get_node
n = get_node("test_session")
for node in n.walk(wait=False):
        print(node.name, node.db_name, node)

test_session    test_session                                                      <bliss.data.node.DataNodeContainer object at 0x7ff7064f36d8>
sample1         test_session:mnt:c:tmp:sample1                                    <bliss.data.node.DataNodeContainer object at 0x7ff7063d3b38>
1_ascan         test_session:mnt:c:tmp:sample1:1_ascan                            <bliss.data.scan.Scan object at 0x7ff7063e7e10>
axis            test_session:mnt:c:tmp:sample1:1_ascan:axis                       <bliss.data.node.DataNodeContainer object at 0x7ff7063e7f60>
roby            test_session:mnt:c:tmp:sample1:1_ascan:axis:roby                  <bliss.data.channel.ChannelDataNode object at 0x7ff7062a4668>
timer           test_session:mnt:c:tmp:sample1:1_ascan:axis:timer                 <bliss.data.node.DataNodeContainer object at 0x7ff7063f5978>
elapsed_time    test_session:mnt:c:tmp:sample1:1_ascan:axis:timer:elapsed_time    <bliss.data.channel.ChannelDataNode object at 0x7ff70633e748>
diode           test_session:mnt:c:tmp:sample1:1_ascan:axis:timer:diode           <bliss.data.node.DataNodeContainer object at 0x7ff70633ef98>
diode           test_session:mnt:c:tmp:sample1:1_ascan:axis:timer:diode:diode     <bliss.data.channel.ChannelDataNode object at 0x7ff7063670f0>
2_ascan         test_session:mnt:c:tmp:sample1:2_ascan                            <bliss.data.scan.Scan object at 0x7ff7063e7438>
axis            test_session:mnt:c:tmp:sample1:2_ascan:axis                       <bliss.data.node.DataNodeContainer object at 0x7ff70633ee10>
roby            test_session:mnt:c:tmp:sample1:2_ascan:axis:roby                  <bliss.data.channel.ChannelDataNode object at 0x7ff7062edd30>
timer           test_session:mnt:c:tmp:sample1:2_ascan:axis:timer                 <bliss.data.node.DataNodeContainer object at 0x7ff7062edd68>
elapsed_time    test_session:mnt:c:tmp:sample1:2_ascan:axis:timer:elapsed_time    <bliss.data.channel.ChannelDataNode object at 0x7ff70633e470>
diode           test_session:mnt:c:tmp:sample1:2_ascan:axis:timer:diode           <bliss.data.node.DataNodeContainer object at 0x7ff70633efd0>
diode           test_session:mnt:c:tmp:sample1:2_ascan:axis:timer:diode:diode     <bliss.data.channel.ChannelDataNode object at 0x7ff7063e7908>
sample2         test_session:mnt:c:tmp:sample2                                    <bliss.data.node.DataNodeContainer object at 0x7ff7063f59b0>
1_ascan         test_session:mnt:c:tmp:sample2:1_ascan                            <bliss.data.scan.Scan object at 0x7ff7063e74e0>
axis            test_session:mnt:c:tmp:sample2:1_ascan:axis                       <bliss.data.node.DataNodeContainer object at 0x7ff7062edb00>
roby            test_session:mnt:c:tmp:sample2:1_ascan:axis:roby                  <bliss.data.channel.ChannelDataNode object at 0x7ff7063f5240>
timer           test_session:mnt:c:tmp:sample2:1_ascan:axis:timer                 <bliss.data.node.DataNodeContainer object at 0x7ff7063f5710>
elapsed_time    test_session:mnt:c:tmp:sample2:1_ascan:axis:timer:elapsed_time    <bliss.data.channel.ChannelDataNode object at 0x7ff70633ea20>
diode           test_session:mnt:c:tmp:sample2:1_ascan:axis:timer:diode           <bliss.data.node.DataNodeContainer object at 0x7ff70633e358>
diode           test_session:mnt:c:tmp:sample2:1_ascan:axis:timer:diode:diode     <bliss.data.channel.ChannelDataNode object at 0x7ff70633ea90>

With the function n = get_node("node_db_name") node_db_name is your entry point and n is the associated DataNodeContainer.

Note

Better use get_session_node() when you know that get_node() will return a session node.

With the function n.walk(wait=False) you can iterate over all the child nodes of the node n.

Among the child nodes, two other types of nodes can be ditinguished:

the ChannelDataNode
and the scan.

They both inherit from the DataNodeContainer class.

The experimental measures are associated to the ChannelDataNodes, and the scan object to the Scan nodes.

Screenshot

You can access any node using its full name (db_name):

cdn_roby = get_node("test_session:mnt:c:tmp:sample1:1_ascan:axis:roby")

Online data analysis¶

The classes inheriting from the DataNode class provide the best ways to monitor the events happening during the experiment and follow the data production.

The method walk(include_filter=None, wait=True) iterates over existing child nodes that match the include_filter argument. If wait is True (default), the function blocks until a new node appears. It returns the new node when it appears and then waits again for a new node.

The method walk_events(include_filter=None) walks through child nodes, just like walk function but waits for node events. It returns an Event object which contains event type, node object and data (if any).

session = get_node("test_session")
def f(include_filter=None):
    """wait for any new node in the session"""
    for node in session.walk(include_filter=include_filter):
        print(node.name,node.type)

def g(include_filter='channel'):
    """wait for a new event happening in any node of the
    type 'channel' (ChannelDataNode)
    """
    for event_type, node, event_data in session.walk_events(include_filter=include_filter):
        print(event_type, node.name, node.get(-1))

# spawn greenlets to avoid blocking
g1 = gevent.spawn(f)
g2 = gevent.spawn(g)

# start a scan
ascan(roby,0,9,10,0.01,diode)

# the monitoring prints pop out during the scan
# (produced by f and g running in greenlets)
10_ascan scan
axis None
roby channel
event.NEW_CHILD roby None
timer None
elapsed_time channel
event.NEW_CHILD elapsed_time None
diode None
diode channel
event.NEW_CHILD diode None
event.NEW_DATA elapsed_time 0.0
event.NEW_DATA roby 0.0
event.NEW_DATA diode 70.0
event.NEW_DATA elapsed_time 0.157334566116333
event.NEW_DATA roby 1.0
event.NEW_DATA diode -57.0
event.NEW_DATA elapsed_time 0.33751416206359863
event.NEW_DATA roby 2.0
event.NEW_DATA diode -61.0
...

# do not forget to kill the greenlet at the end
g1.kill()
g2.kill()

Note

In the example above node.get(-1) is used to retrieve the last data value produced on this node. For more details see method get(from_index, to_index=None) of the ChannelDataNode.

Lima Data View¶

2D images from Lima servers have a special lima node type, which corresponds to a LimaImageChannelDataNode class. When executing .get() on those nodes, the returned value is not directly the raw images, since it can be very voluminous, but a “view” on the data. It represents references to data already produced by the image channel at the moment the view object is instantiated.

g3 = gevent.spawn(g, "lima")

lima_simulator=config.get("lima_simulator")
ct(0.1, lima_simulator)
event.NEW_NODE image <bliss.data.lima.LimaImageChannelDataNode.LimaDataView object at 0x7f4d28931be0>
event.NEW_DATA image <bliss.data.lima.LimaImageChannelDataNode.LimaDataView object at 0x7f4d28931c18>

Function g() calls .get(-1) on the node object, thus returning the last image view. This works the same as with other nodes. It is perfectly valid to only get a view with partial data, for example: .get(0, 10) will return a view object for images from 0 to 10 only.

Getting raw image data¶

The .get_image() method can be called on LimaDataView objects to retrieve raw image data from references stored within a Lima data view object.

ct(0.1, lima_simulator)
Scan(number=1, name=ct, path=<no saving>)

SCANS[-1].node
<bliss.data.scan.Scan object at 0x7efda1dd0898>

# there is only one Lima node for this 'ct'
lima_node = next(SCANS[-1].node.walk(include_filter="lima", wait=False))

lima_data_view = lima_node.get(-1)

raw_image_data = lima_data_view.get_image(-1)

Note

.get_image(-1) means “get latest image from the view”. In the example above, there is only one image so it would be equivalent to .get_image(0).

Scanning & experiment sequences¶

In the context of an experiment or a scan procedure it exist a more convenient way to obtain the data produced by a scan. At the shell level, it exist a global variable SCANS that stores all the scan objects that have been launched in this shell session.

For example SCANS[-1] returns the last scan object and SCANS[-1].get_data() returns the data of that scan.

SCANS
deque([Scan(number=1, name=ascan, path=/mnt/c/tmp/sample2/test_session/data.h5),
       Scan(number=2, name=ascan, path=/mnt/c/tmp/sample2/test_session/data.h5)],
maxlen=20)

SCANS[-1]
Scan(number=11, name=ascan, path=/mnt/c/tmp/sample2/test_session/data.h5)

SCANS[-1].get_data()
{
'roby':         array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.]),
'elapsed_time': array([0., 0.15733, 0.33716, 0.49581, 0.65329,
                       0.80668, 0.94995, 1.12723, 1.28504, 1.43661 ]),
'diode':        array([ 70., -57., -61., -43., 89., 54., 23., -89., -87., -98.])
}

Note

Scans are stored per session in the .scans property of Session object. The global SCANS is only available from Bliss Shell and refers to current_session.scans.

Getting image data¶

Image data is not directly retrieved via the scan object .get_data() method, since it can be very voluminous. In case of an acquisition with images, the returned dictionary will contain a key with the image channel fullname, that gives access to a data view object.

The view object is used to get the raw image data from references. Indeed, the image data can be still available from the Lima server memory, if it is not the case it may try other data sources – at the last resort it tries to open the file (if image saving is activated).

The Lima data view object has a .get_image(image_index) method that returns the raw image data:

BLISS [1]: lima_simulator=config.get("lima_simulator")
BLISS [2]: ct(0.1, lima_simulator)
  Out [2]: Scan(number=14, name=ct, path=<no saving>)
BLISS [3]: SCANS[-1].get_data()['lima_simulator:image'].get_image(-1)
  Out [3]: array([[0, 0, 0, ..., 0, 0, 0],
                  [0, 0, 0, ..., 0, 0, 0],
                  [0, 0, 0, ..., 0, 0, 0],
                   ...,
                  [0, 0, 0, ..., 0, 0, 0],
                  [0, 0, 0, ..., 0, 0, 0],
                  [0, 0, 0, ..., 0, 0, 0]], dtype=uint32)