Installing a new beamline computer with Ansible¶

Quickstart¶

Purpose

Ansible bcu_env playbook is used to install and configure a proper environment on a beamline computer.

It is NOT intended to update BLISS or other packages. This task will be performed using conda or mamba.

Checklist

Ensure that the target computer has a /opt/bliss/conda partition and a /users/blissadm/conda symbolic link to it.
Clone ansible repository: git clone https://gitlab.esrf.fr/bliss/ansible.git
Install ansible if it is not yet the case: conda install --file requirements.txt
Customize the inventory file(s): inventory/idXX
- this inventory file must now be the “simplified” one.
- this directory must be a clone of the repository: https://gitlab.esrf.fr/bliss/inventory
Run in main directory:

./bcu_ansible <hostname> [-v] [-vv] [-vvv]

Ports being used

beacon: 25000
redis: 25001
beacon web app: 9030
supervisor web app: 9001
multivisor rpc port: 9002
multivisor web app: 22000

Requirements

ansible version >= 2.13 is needed
python >= 3.6 is needed
jinja2 is needed

BCU-deploy

You can use the bcu-deploy computer with your own account or a conda environment on your own computer for that.
Accounts of bcu members should be already created on bcu-deploy.
You must have your private ssh key (corresponding to your public ssh key on gitlab) installed on your bcu-deploy account.

Principles¶

Ansible executes a playbook, which is a set of instructions for doing administrative tasks (installation, configuration etc.). The playbook is located on a source host where ansible is executed.

Tasks are ran on the target host you want to install. Only one host can be installed at once.

Target hosts are listed in inventory files (example: inventory/id99). In this file, specific configuration rules are detailed for each host depending on its usage.

For example, the installation of the lid991 target computer can be done with:

./bcu_ansible lid991

Note

Do not use an alias of the computer name.

Tasks are performed on the target as blissadm by default and root, if needed. This implies to be able to get these identities without password. That is accomplished using BCU members personal accounts on the target computers. The group soft is put in the sudoers file, bcu-members can become root without a password.

Usage¶

Requirements¶

Requirements on the “target” machine

conda partition: /opt/bliss/conda
python 2.7 installed
blcontrol service installed:
- /csadmin/common/install/scripts/Install.blcontrol.share executed
blissadm account correctly installed:
- /users/blissadm existing
- /users/blissadm/bin existing
You must be able to use your user name to login
You must have privilege to use sudo without password

Requirements on the “source” machine

These commands have to be run with your own user id:

Python >= 3.6
jinja2
sshpass program sudo apt-get install sshpass
libffi-dev sudo apt-get install libffi-dev

On bcu-deploy, these packages are already installed.

Install ansible¶

On bcu-deploy, ansible is already installed.

Install Ansible runtime with pip (this takes time)
- pip install ansible --user
- ansible-galaxy collection install community.general
- info: if you get errors about setuptools_ext, upgrade the setuptools package first with pip
  - pip install setuptools --upgrade --user
Make sure your PATH environment variable contains the local Python bin directory
- export PATH=~/.local/bin:$PATH

Note

ansible version >= 2.13 is needed
on bcu-deploy, version 2.17.1 is installed

Clone the ansible playbook¶

Clone the Ansible repository from gitlab
- git clone https://gitlab.esrf.fr/bliss/ansible.git
- note : the ansible files are distributed among everyone

In case of error

Check you have removed all proxy settings:

in environment variables
in the git configuration

Running the playbook¶

Before running the playbook ensure the beamline computer has a /opt/bliss/conda (installed by CS).
Once the inventory has been filled with the hosts it is time to run the playbook.
You must be in ansible/ directory to force ansible to use local ansible.cfg configuration file.
For now the bliss playbook restricts execution to one single host at the time to prevent accidental updates.
Running the playbook is as simple as:

./bcu_ansible <computer_name>

bcu_ansible script converts automatically the simple inventory file (inventory/idXX) into the ansible compatible one that wil be put in inventory_ansible/idXX.

If bliss is defined in your inventory file, BLISS will be installed:

from the BLISS conda package in bliss environment
from git repository in bliss_dev environment

Test installation¶

Log into the target machine to run BLISS
. blissenv enters bliss environment
- Use bliss to start BLISS once in bliss environment
. blissenv -d is the same, but for the development environment
When on the mainstation, the beacon server is started automatically with supervisor
- Check status with supervisor http://{beacon_host}:9001
- Or multivisor http://{beacon_host}:22000
- Browse beamline configuration http://{beacon_host}:9030
YAML configuration files go into ~blissadm/local/beamline_configuration/ on the mainstation

The `bcu_env` playbook¶

The bcu_env ansible playbook (bcu_env.yml) consists of a set of roles with some pre tasks and post tasks.

Roles¶

The playbook consists of complementary roles explained below. These roles are not exclusive:

base-station
bliss-station
control station
main-station
vacuum
lima-station
daiquiri
Tango Database
vacuum

A computer must be at least a base-station.

It may also be a bliss-station, a main-station or a lima-station.

A main-station must be a bliss-station.

Screenshot

Common tasks¶

debugage.yml: print info for debugging purposes.
pre_/post_ install_log.yml: create a report log file in target host in ~/local/ansible/ directory on the target machine
package_to_database.yml: create cron job to to fill a database with package installation history
register_in_database.yml: Register host/installation in mysql database

`debugage.yml`¶

Debugging

pre_/post_install_log.yml¶

pre_install_log.yml creates a report log file in target host in ~/local/ansible/ directory on the target machine.

post_install_log.yml finalizes the log file.

Example of a report log file:

#
# This is bcu_env Ansible installation report file
#

[installation]
date=2019-03-25
time=20:15:41
iso_time=2019-03-25T19:15:41Z

[roles]
base-station=true
bliss-station=true
control-station=true
main-station=true
lima-station=false
ser2net=false

[source_computer]
host=pcsht
user=guilloud
inventory_file= ..../PROJECTS/deploy/ansible/inventory_ansible/id16na
ansible_version=2.7.8

[target_computer]
host=lid16na3
ansible_nodename=lid16na3
install_user=blissadm

[installed]
conda_version=4.6.8
multivisor_version=666
black_version=666
bliss_conda_package_version=0.1.0
bliss_dev_sha=8e93f638

### END OF LOGFILE ###

bashfiles¶

.bash_profile and .bashrc file creation, if either file is absent

`package_to_database.yml`¶

package_to_database.yml creates a cron job to to fill a database with package installation history.

`register_in_database.yml`¶

register_in_database.yml registers host/installation in mysql database.

Base-station¶

A “base-station” will be installed with common services and configurations. It is needed by other roles:

bashrc¶

Adds the usual shell prompt to the .bashrc.

conda¶

conda.yml:

install miniconda3 (which create base environment)
- 2019-03-25: version 4.6.8
in base environment, install Conda packages:
- black (forces version 18.6b4)
- git latest version
- mamba, pip, pymysql
- conda init for blissadm shell
- deactivate base env at shell start
- bash completion for conda

multivisor¶

multivisor.yml installs multivisor-rpc to gather supervisor status among all the beamline’s computers.

create multivisor Conda environment (with python3.* installed) (as blissadm) if not present
install dependencies:
- conda dependencies: gevent, zope, pendulum, maya, louie, six, blinker, flask, supervisor
- pip dependencies: zerorpc
pip-install multivisor version 6.0.1 in multivisor environment (as blissadm)
remove supervisor debian package if installed.
configure supervisor:
- put start-up script (template: files/supervisor.j2) in initd/ to start supervisor at startup
- install supervisor default variable file
- create supervisor /etc/supervisor/conf.d/ directory
- install supervisor config file (template: files/supervisor.conf.j2) /etc/supervisor/supervisord.conf
- make supervisor start on boot
- create local supervisor directory: ~blissadm/local/daemon/config/supervisor.d/
- install supervisor start-up script templates in: ~blissadm/local/daemon/config/supervisor.d/templates/
- clean deprecated old supervisor configuration
- create links to supervisorctl and multivisor-cli
- start supervisor daemon

!!!note: Multivisor part of the playbook can be run alone using tag multivisor ex: ``

Bliss-station¶

The “bliss-station” role mainly performs the installation of the BLISS control system.

“bliss-station” role is needed by “main-station” role.

bliss.yml:
- create a Conda environment called bliss with python 3.7
- create activation / deactivation scripts
  - /opt/bliss/conda/miniconda/envs/bliss/etc/conda/activate.d/bliss.sh: defines TANGO_HOST and BEACON_HOST
  - /opt/bliss/conda/miniconda/envs/bliss/etc/conda/deactivate.d/bliss.sh: empty for now…
- Conda-install BLISS package
- create files:
  - /users/blissadm/local/blissenv.ini:
  - /users/blissadm/bin/blissenv: file to be sourced to get into bliss environment
bliss_dev.yml:
- create a Conda environment called bliss_dev
- clone BLISS repository in ~blissadm/local/bliss.git
  - set push URL to http://
  - Set credential helper timeout to 300 s to avoid to re-type password when pushing multiple times
  - set push.default to traditional value to avoid warning message
- create (de)activation scripts
- pip-install BLISS in bliss_dev Conda environment from ~blissadm/local/bliss.git
local_bliss_code.yml:
- check if local directory for beamline software (/users/blissadm/local/id99.git) exists
- check if remote gitlab repo (gitlab.esrf.fr/ID99/id99) exists
- create beamline specific code repository located in ~blissadm/local/id99.git
  - if remote gitlab repo exists: clone it into local directory
  - if remote gitlab repo does not exist: create a new repository with cookiecutter:
    - clone and instantiates gitlab.esrf.fr:bliss/cookiecutter.git template for ‘id99’

Control station¶

ser2net.yml (executed only if target host is present in ser2net section of the inventory file):
- create ser2net Conda environment
- Conda-install ser2net package in this environment
- install config file
- restart supervisor daemon (as root)
dkms.yml
espia_driver.yml
p201_driver.yml
rp_driver.yml
gpib_driver.yml

Main-station¶

“main-station” role performs installation of beamline configuration files and configures BEACON and MULTIVISOR master.

“main-station” role requires “bliss-station” role.

logbliss.yml:
- create bliss logs directory (/var/log/bliss)
configuration.yml:
- beamline configuration directory (~/local/beamline_configuration)
- init files:
  - ~/local/beamline_configuration/__init__.yml
  - ~/local/beamline_configuration/sessions/__init__.yml
  - ~/local/beamline_configuration/sessions/scripts/__init__.py
supervisor.yml:
- adds beacon startup file in ~/local/daemon/config/supervisor.d/
- configure multivisor ~/local/daemon/config/multivisor.conf
- restart supervisor
nginx.yml:
- install and enable nginx service
- nginx configuration for beacon homepage reverse proxy /etc/nginx/conf.d/beacon.conf

Lima-station¶

“lima” role performs, for each camera declared, the installation of relevant lima packages in a camera-specific Conda environment.

install_cam.yml : For each declared camera:
- Creation of a dedicated environment per camera
- Installation of lima-camera-<camera_name>-tango package and its dependencies

Note

Lima-Station role is also used to install the Lima BPM Web server with role: lima-bpm-web-server

Now that you know which types of hosts can be installed, it is time to find out how to add a new host to the ansible host inventory.

Tango Database¶

Warning

It is unlikely, you can still install a debian machine with this playbook.
But just to be sure: for Ubuntu only

Installation of a conda env named tango to install mysqld and the tango and taco service processes, together with their supervisor startup yml files.

tangodb role installs:

tango database
taco database
icepapcms

vacuum¶

This role deploys the vacuum tango devices at a beamline machine.

It creates:

The tango env to execute the devices (only if the env does not exists)
The supervisor file to launch this device servers (only if the file does’t exists)

After the installation the supervisor file still have to be adapted to the beamline.

daiquiri¶

This role deploys Daiquiri in a beamline machine.

Before the installation:

A user/account have to be requested for the DB daiquiri-db.esrf.fr for this beamline (ask the BLISS UI team)
A SSL certificate have to be requested for this machine (create a TID ticket)
- Now the TID use a certbot to deploy and update the certificates
- The certificate is usually requested for the alias, not the machine bm18.esrf.fr instead of lbm18ctrl.esrf.fr
- /etc/letsencypt/live/bm18.esrf.fr/
- The Ansible recipe is setup to use the alias by default, else app.yml have to be updated after the installation

After the installation:

bliss.yml has to be updated with the right BLISS session
pass.yml has to be updated to use the Daiquiri SQL password
app.yml may have to be updated to fix the path to the SSL certificate

Here is the installation directory structure:

/var/log/
     └── daiquiri/
/opt/bliss/conda/miniconda/envs/
        ├── daiquiri       # To run Daiquiri server
        └── daiquiri-npm   # To build the web application
/users/blissadm/local
        │
        ├── daemon/config/supervisor.d
        │                 └── daiquiri.conf
        ├── beamline_configuration
        │   └── daiquiri
        │       ├── config
        │       │   ├── app.yml        # MAY have to be updated to feed the right path the SSL certificat
        │       │   └── bliss.yml      # MANUAL POST INSTALL -> this file have to be updated
        │       ├── external
        │       │   └── utils.py
        │       ├── __init__.yml
        │       └── layout
        ├── daiquiri_private
        │   └── config
        │       ├── pass.yml           # MANUAL POST INSTALL -> daiquiri SQL DB password for this beamline DB user
        │       └── secrets.yml
        ├── id00.git/id00
        │            └── daiquiri
        │                └── implementors
        ├── daiquiri.git
        └── daiquiri-ui.git
            └── build                  # The directory is build with NPM

Inventory¶

Adding a beamline computer¶

To list what needs to be done on a beamline (or a branch), one should add an ansible inventory file, usually called after the beamline (or the branch) to be installed.

This file must be located in the ./inventory/ directory, which must be a clone of https://gitlab.esrf.fr/bliss/inventory repository (NO SUB-MODULE).

The inventory file consists of:

a list of computers names followed by the definition of their various roles.
a [vars] section to define environment variables:
- BEACON_HOST
- TANGO_HOST
- and user defined ones.

Base station

mandatory for all computers

Every host in the inventory is implicitely a base station. As soon as your host is in the inventory the basestation installation will be done.

Bliss station

requires to run a BLISS session

Just add the bliss role under the name of the computer.

Main station

The mainstation role is automatically active, if the target computer has the same name than the BEACON_HOST variable.

Lima station

On a lima station, a dedicated environment per camera type (named <camera_name>) type will be created. In this environment, the lima-camera-<camera_name>-tango and all its dependencies will be installed.

To add support for a camera on a lima station, a lima-role must be added:

lima-<camera_name>

where <camera_name> is the camera type you want to use (e.g : prosilica, basler, frelon, …).

The following example will create two environments named frelon and prosilica:

lima-frelon
lima-prosilica

List of cameras (11-2019) supported by ansible install (i.e a Conda package exists):

lima-andor
lima-andor3
lima-basler
lima-dexela
lima-eiger
lima-frelon
lima-imxpad
lima-marccd
lima-maxipix
lima-merlin
lima-pco
lima-pilatus
lima-pixirad
lima-pointgrey
lima-prosilica
lima-simulator
lima-slsdetector
lima-ueye
lima-ultra
lima-v4l2
lima-xh

lima-bpm-web-server role will:

Create a specific conda environment (named )
install the Lima BPM Web Server on corresponding computer.

ID99 Example¶

This example represents the fake ID99 beamline. The inventory file is called id99 (see the example file) and it includes the following hosts and features:

lid991 is a simple basestation.
lid992 is installed with bliss environment.
lid993 is a control station and drivers will be installed where the keywords are employed.
id99ctrl has roles mainstation and lima and then also basestation and bliss
lid99bas1 has lima role and support for basler will be installed.
lid99frelon1 has lima and bliss roles with support for frelon and basler

The corresponding inventory looks like this:

[lid991]

[lid992]
bliss

[lid993]
driver-espia
driver-gpib
driver-rocket_port
driver-p201
driver-ser2net

[lid99bas1]
lima-basler

[lid99frelon1]
driver-espia
bliss
lima-frelon
lima-andor3

[lid99ctrl]
mainstation
driver-gpib
driver-ser2net
bliss
lima-basler
lima-bpm-web-server


[vars]
BEACON_HOST=lid99ctrl:25000
TANGO_HOST=lid99ctrl:20000
BEAMLINENAME=ID99
TRUC=bidule:12345
MACHIN=chose

TRUC and MACHIN variables will be added in [vars] section of ~blissadm/local/blissenv.ini and exported in bliss and bliss_dev conda environments.

Specify local beamline repository group¶

Sometimes the beamline team has its local repository under a specific group in Gitlab.

With the addition of the REPO_GROUP variable in the beamline inventory, ansible known where to clone the repo from.

This variable is optional, if not set, it defaults to the current behavior (clone from IDXX/idXX.git)

Variable `REPO_GROUP`¶

defined in a beamline inventory (it’s explicit)
allows to clone the beamline local repo from a specific group or subgroup in gitlab
it’s optional (default to IDXX)

Example from inventory/bm32:

    [vars]
    # gitlab repo is at F-CRG/bm32
    REPO_GROUP=F-CRG
    CONFIG_SYMLINK=True
    TANGO_HOST=ld324:20000
    BEACON_HOST=crg5:25000

The clone will be using this repo https://gitlab.esrf.fr/F-CRG/bm32

Info

This can also be used to arrange beamlines in sub-groups (e.g. BCU-Vercors, BCU_cheval_blanc)

    REPO_GROUP=BCU-Vercors/IDxx

Beamline configuration symlink¶

Sometimes the beamline configuration is in the same repository as the local code.

With the addition of the CONFIG_SYMLINK boolean variable in the beamline inventory, ansible will not try to clone a specific repository for the beamline configuration.

Instead, a symbolic link will be made to the local code checkout:

    ~blissadm/local/beamline_configuration -> ~blissadm/local/IDxx.git/beamline_configuration

This variable is optional, if not set, it defaults to the current behavior (clone from IDXX/config.git)

Variable `CONFIG_SYMLINK`¶

defined in a beamline inventory (it’s explicit)
will link to the beamline_configuration inside the local repo, instead of cloning a separate repo.
it’s optional (default to False)

Example from inventory/bm32:

    [vars]
    CONFIG_SYMLINK=True
    REPO_GROUP=F-CRG
    TANGO_HOST=ld324:20000
    BEACON_HOST=crg5:25000

The beamline_configuration will be symlinked to ~blissadm/local/bm32.git/beamline_configuration

Database entries¶

phpmyadmin service

Our playbook contains yml entries which will register the installation in the database bcu_deployment on the trusted mysql.esrf.fr. It can be browsed using phpmyadmin service.

The table ansible_machine contains a single entry for each installed host, whereas the table ansible_install will contain one record for each installation.
Furthermore, a cronjob is created, running a python script, which enters the packages installed with conda into the table bcupack in a fashion, where each conda environment will have one entry for one and the same package.

Updating BLISS¶

BLISS is installed with a Conda package, so it must be updated with Conda tools:

in command line: conda update bliss or conda install bliss=X.Y.Z
or using mamba: mamba update bliss or mamba install bliss=X.Y.Z

If you prefer to follow the latest developments, use the development environment and update with git.

BLISS development environment¶

When the development environment is activated, any change in ~blissadm/local/bliss.git is reflected automatically when BLISS is (re)loaded. Indeed, in the case of the development environment, BLISS is installed with pip install -e ., which means the package is linked with the source.

If you need to switch to the development environment for operation, like just after a bug has been fixed but not merged to BLISS yet so you cannot redeploy a stable release of BLISS with the normal BLISS production environment, there is a .ini file ~blissadm/local/blissenv.ini: in the [global] section it is possible to set use_dev = True to make . blissenv activating the development environment instead of the production environment, so users will automatically use the right environment without needing further modifications of startup scripts.

The above changes will only take effect in future calls to . blissenv .

It is also possible to activate the bliss development environment only for the console you are without affecting the beamline. Just add a flag -d like this:

ldcm1:~ % . blissenv -d

setting BLISS environment
Using BLISS bliss_dev

(bliss_dev) ldcm1:~ %

Bug fixing in development environment¶

Scenario: users are using BLISS in production environment. A bug is identified, and has to be fixed.

The workflow is to fix the bug locally, and then switching users to the development environment, until the bug fix is merged in BLISS master:

identify which commit in the master BLISS branch is currently used within the production environment
- grep master@ ~blissadm/bin/blissenv
- look for the hash following master@, for example: 86a088
enter bliss dev. environment: . activate bliss_dev
execute git fetch in ~blissadm/local/bliss.git
stash your current changes: git stash
create a bug fix branch called my_bug_fix for example: git checkout -b my_bug_fix
reset branch to current production master: git reset --hard 86a088
fix the bug
commit the result: use git gui
do the merge request for BLISS project
- git push origin my_bug_fix
- click on the proposed link and follow instructions
switch production environment to bug fix branch
- edit ~blissadm/local/blissenv.ini: set use_dev to True

Updating BLISS in development environment¶

This is done like with any other git project:

git fetch: get changes from remote origin
git rebase origin/master: rebase local commits on top of remote master

Dev notes¶

Third-party installations¶

The installation of miniconda is done via a third-part Ansible role developed by Akihiro Uchida:

uchida.miniconda

The installation of nginx is done via a third-part Ansible role developed by NGINX INC:

nginxinc.nginx

Purge an existing installation¶

To purge an existing installation, use the cleaning playbook.

Installing a new beamline computer with Ansible¶

Quickstart¶

Principles¶

Usage¶

Requirements¶

Install ansible¶

Clone the ansible playbook¶

Running the playbook¶

Test installation¶

The bcu_env playbook¶

Roles¶

Common tasks¶

debugage.yml¶

pre_/post_install_log.yml¶

bashfiles¶

package_to_database.yml¶

register_in_database.yml¶

Base-station¶

bashrc¶

conda¶

multivisor¶

Bliss-station¶

Control station¶

Main-station¶

Lima-station¶

Tango Database¶

vacuum¶

daiquiri¶

Inventory¶

Adding a beamline computer¶

ID99 Example¶

Specify local beamline repository group¶

Variable REPO_GROUP¶

Beamline configuration symlink¶

Variable CONFIG_SYMLINK¶

Database entries¶

Updating BLISS¶

BLISS development environment¶

Bug fixing in development environment¶

Updating BLISS in development environment¶

Dev notes¶

Third-party installations¶

Purge an existing installation¶

The `bcu_env` playbook¶

`debugage.yml`¶

`package_to_database.yml`¶

`register_in_database.yml`¶

Variable `REPO_GROUP`¶

Variable `CONFIG_SYMLINK`¶