Blissdata Installation and Configuration Guide#

Installation#

Blissdata installation on itself is straightforward:

Install via PyPi

pip install blissdata

Install via Conda (from the ESRF-BCU channel)

conda install blissdata -c esrf-bcu

If you plan to connect to an existing instance, then you're done.

Otherwise, you have no database running, then you should follow next steps to start one.

Redis Setup#

Blissdata works with a Redis database which is installed separately. Additionally, ReJson and RediSearch extensions modules have to be present on the Redis server.

Redis can be installed by various means, thus we present two methods with Docker and Conda. These methods provide a redis.conf file to Redis, which will be covered later.

Using Docker#

The simplest way to run Redis with the required extensions is by using the Redis Stack distribution. This docker image includes Redis Server with several extensions, including the ones we need.

# start redis in a container while providing a config file
docker run --name myredis -d -p 6379:6379 \
-v /path/to/my/redis.conf:/redis-stack.conf \
redis/redis-stack:7.2.0-v5

# once created you scan start, stop and delete the container like this:
docker start myredis
docker stop myredis
docker rm myredis

Redis is now running on port 6379 with the configuration file provided in /path/to/my/redis.conf

Using Conda#

Alternatively, you can install Redis and its extensions via Conda:

# in a conda env you created
conda install "redis-server>=7" redisjson redisearch -c conda-forge -c esrf-bcu

# execute server
redis-server \
/path/to/my/redis.conf \
--loadmodule $CONDA_PREFIX/lib/librejson.so \
--loadmodule $CONDA_PREFIX/lib/redisearch.so

Same as with docker, Redis starts on port 6379 with the configuration file provided.

Alternatives#

Finally you can have a look at the official Redis installation guide if none of the previous methods are applicable for you.

Configuring Redis#

Blissdata requires some specific Redis configuration to manage memory usage and persistence. Redis is configured by providing a Redis configuration file, usually called redis.conf.

For Blissdata to work properly, the following values must be set:

redis.conf

maxmemory 4gb                    # Adjust to your desired memory limit
maxmemory-policy noeviction      # Memory eviction is managed by Blissdata’s memory tracker
save ""                          # Disable Redis backup files

Basically, it defines the maximum memory allocated and disables automatic memory eviction and backup files.

Memory Tracker#

IMPORTANT

Blissdata comes with a daemon process called memory tracker that monitors Redis memory usage and clears space when necessary.

IT MUST BE STARTED once Redis is running. This is what prevents Redis from overflowing.

To start the memory tracker, run:

memory_tracker --redis-url redis://localhost:6379 --init-db

Notice the --init-db option, it tells Redis server on what data it should keep an index (by using RediSearch extension), so blissdata can perform search queries. This has to be done only once on a running database. It can either be done from the Blisdata python API, or the memory tracker which is more convenient.

Once started, you will see logs indicating the memory limits and cleanup thresholds:

2023-10-31 15:23:16,158 INFO: INIT | Redis max. memory: 4GB
2023-10-31 15:23:16,158 INFO: INIT | Mem. usage to trigger cleanup: 80% (3.2GB)
2023-10-31 15:23:16,158 INFO: INIT | Protected history length: 180 seconds
2023-10-31 15:23:16,159 INFO: Memory usage: 1.602MB

The memory tracker accepts several parameters to adjust memory cleanup strategy, depending on your needs. You can have a look at it with --help:

usage: memory_tracker [-h] --redis-url REDIS_URL [--cleaning-threshold CLEANING_THRESHOLD]
                      [--protected-history PROTECTED_HISTORY] [--monitoring-period MONITORING_PERIOD]
                      [--inactive-scan-deletion INACTIVE_SCAN_DELETION]
                      [--closed-scan-deletion CLOSED_SCAN_DELETION]
                      [--cleaning-time-slice CLEANING_TIME_SLICE] [--init-db]

Redis memory cleaner for blissdata.

optional arguments:
  -h, --help            show this help message and exit
  --redis-url REDIS_URL
                        Redis server address
  --cleaning-threshold CLEANING_THRESHOLD
                        Percentage of memory usage to trigger a cleaning routine.
  --protected-history PROTECTED_HISTORY
                        Recent data protection in seconds. Cleaning routine can only erase data older
                        than this. Be careful, protecting too much of the history may prevent the
                        cleaning routine to release enough space.
  --monitoring-period MONITORING_PERIOD
                        Tracker updates period in seconds
  --inactive-scan-deletion INACTIVE_SCAN_DELETION
                        Time in seconds after which an inactive and non-terminated scan is completely
                        deleted (data streams may be trimmed earlier).
  --closed-scan-deletion CLOSED_SCAN_DELETION
                        Time in seconds after which a properly terminated scan is completely deleted
                        (data streams may be trimmed earlier).
  --cleaning-time-slice CLEANING_TIME_SLICE
                        Size of the history slice which is released by a cleaning routine. It is a
                        percentage of the total time covered by history (i.e. the oldest 20 percent of
                        all history).
  --init-db             Initialize a fresh redis database

That's it! Blissdata is now installed, configured, and ready to use.