Add tools to aid installation and a Dockerfile

[hugobuddel]

The toolbox directory contains some scripts to install the METIS Pipeline on a fedora machine. The toplevel Dockerfile calls these scripts to create an image.

Everything compiles, and the tests pass, but it doesn't work.

Compile with:

docker build -t metispipeline .

Wait an eternity, then start with:

docker run -ti metispipeline

Then run esorex:

$ esorex --recipe-dir=/home/metis/metis_pipeline/ --recipes

     ***** ESO Recipe Execution Tool, version 3.13.6  *****

List of Available Recipes :

  metis                 : An example recipe.
  metis_calib           : An example recipe.

So far so good, everything installs. But then:

esorex --recipe-dir=/home/metis/metis_pipeline/ metis

     ***** ESO Recipe Execution Tool, version 3.13.6  *****

*** buffer overflow detected ***: terminated
Aborted (core dumped)

or

esorex --recipe-dir=/home/metis/metis_pipeline/  --help metis_calib

     ***** ESO Recipe Execution Tool, version 3.13.6  *****

*** buffer overflow detected ***: terminated
Aborted (core dumped)

What do you think @ivh ?

[ivh]

I havn't tried your docker setup yet, but my compiled svn-snapshot from Lars does not crash, I get the correct help-output.

[ivh]

Oh, and I simply did
./bootstrap && ./configure --prefix=$ESOREX_PLUGIN_DIR/../.. && make && make install

[hugobuddel]

You probably already have a working setup for the crires pipeline, so that should help.

I got a bit further by first installing iiinstrument. But that should not be necessary.

What OS and ESO libraries do you use?

[hugobuddel]

OK, I think I've figured it out.

When installing esorex from the official ESO RPM repositories, then it
has built-in support to run Python recipes. But that is not at all
compatible with installing pipelines from source, because then it
thinks that all the interactive elements of reflex workflows are
recipes because they are python files. E.g. esorex thinks that
reflex/interactive_window.py in the iiinstrument example is a recipe.
And subsequently the real recipes also don't work anymore.

So installing packages through make install, does not actually work
with the official ESO esorex rpm.

And ESO knows this, because for the RPMs they split each pipeline up
in different packages, that put files in different locations
(esopipe-iiinstrument-recipes vs esopipe-iiinstrument-wkf). But "make
install" (or install_pipeline) puts everything in one location, so
esorex gets confused and breaks. But installing esorex through
install_pipeline will /not/ enable Python support, so then everything
works.

What I find most frustrating is that apparently ESO has a perfect way
to build pipelines from source that works with the official RPMs, but
they don't share that with us. We now have to use install_pipeline
which does not work with the released RPMs. It just doesn't make sense
at all to me. (Not trying to dis your baby Lars! Without
install_pipeline we would be even more in pain.)

Simply deleting the interactive_window.py seems to resolve the issue.

And since we hopefully will not have to write reflex workflows, I'm
simply going to delete that file.

[hugobuddel]

OK, I won't delete the file yet, because then we break the entire reflex workflow.

[ivh]

Good detective work! 😅 Indeed this sounds sub-optimal.

I've never encountered this because I never bothered with the RPMs - using only debian-based linuxes here. The kit-install makes sure only the libs that come with it are linked (I think), so after that I simply set CPLDIR to the same dir as the base of the install script, which allows to compile pipelines with that set of libs. And I also use the esorex from the kit.

Environment sth like this:
export PIPES=$HOME/pipes
export CPLDIR=$PIPES/kitinstall/
export ESOREX_MSG_LEVEL=debug
export ESOREX_MEM_CHECK=TRUE
export ESOREX_SUPPRESS_PREFIX=TRUE
export ESOREX_PLUGIN_DIR=$PIPES/installed/lib/esopipes-plugins
export PATH="$CPLDIR/bin:${PATH}"

[hugobuddel]

I have resolved it! I didn't have the lib/esopipes-plugins part of the ESOREX_PLUGIN_DIR...

Nevertheless, esorex should not crash and burn just because it finds a Python file it does not understand. So created an issue on Jira: https://jira.eso.org/browse/PIPE-11107

Since this now kinda works, I'll also invite Jennifer and Chi-Hung to take a look.

[hugobuddel]

Thanks for your setup @ivh . Perhaps it would be good to also add those exports to the docker image. But for now it works I think.

I've not used the RPMs much either, because I usually use debian-based systems too. Switching over to Guix though (exactly to prevent this stuff). But I thought, lets do things in the official way.