bare metal installation fails due to missing dependencies

[crockercaria]

I assume that I can install KNIX via an ansible installation on localhost as described in https://knix.io/documentation.html
However I am unable to and I am not able to see the reason why or solution.
Also why is there a repository on port 5000.
Perhaps the documentation is missing some information.
thanks

I have Ubuntu 20.04
I have fetched knix from git in order to get the cfg files and the code
After following the steps in the documentation.html without problem when I run make all I get is

vagrant@ubuntu2004:~/ansible/knix/deploy/ansible$ make
cd ../../riak && make lib/jiffy.tgz && cd -
make[1]: Entering directory '/home/vagrant/ansible/knix/riak'
fatal: No names found, cannot describe anything.
..
fatal: No names found, cannot describe anything.
mkdir -p lib
echo "Building jiffy"
Building jiffy
Error response from daemon: no such image: riak_libs_build:: invalid reference format
invalid reference format
Error parsing reference: "localhost:5000/riak_libs_build:" is not a valid repository/tag: invalid reference format
Error response from daemon: no such image: riak_libs_build:: invalid reference format
Dockerfile is newer than the container
invalid argument "riak_libs_build:" for "-t, --tag" flag: invalid reference format

[iakkus]

Hi @crockercaria!

Thanks for trying out knix. How did you obtain the knix source? Did you clone the repo or just downloaded it as a zip?

The errors fatal: No names found, cannot describe anything. are due to the fact that git cannot get the tag and the commit id of your codebase.

[crockercaria]

I used a quick shallow git clone (--depth=1). I will do a full clone and try again.

Update:
After a full clone the the software starts a long make process that eventually fails.

TASK [install riak if not present] *************************************************************************************************************************************************************************
fatal: [ubuntu2004.localdomain]: FAILED! => {"changed": false, "msg": "Dependency is not satisfiable: libssl1.0.0 (>= 1.0.0)\n"}

Full log
riak.log.tmp.txt

[crockercaria]

Update:
After a full clone the the software starts a long make process that eventually fails.

TASK [install riak if not present] *************************************************************************************************************************************************************************
fatal: [ubuntu2004.localdomain]: FAILED! => {"changed": false, "msg": "Dependency is not satisfiable: libssl1.0.0 (>= 1.0.0)\n"}

Full log above

[iakkus]

Thanks!

I've also encountered a similar problem when I was trying to do an ansible installation with Ubuntu 20.04. The libssl library package in 20.04 is different and it seemed like it is not available for some reason.
Check out the following page: https://packages.ubuntu.com/bionic/amd64/libssl1.0.0/download
It looks like the focal version is missing: https://packages.ubuntu.com/focal/amd64/libssl1.0.0/download

If you manually install this library from the above link, you should be able to move forward.

Note that I obviously cannot give any guarantees whether this will cause any other errors with other stuff running on your computer (perhaps make this a VM dedicated for knix if you're testing things out?).

As a personal data point, I am running 20.04 for about a year now and I have not seen any issues with the manually installed library.

[crockercaria]

The ansible setup does not seem to work ! I have tried different branches (baster and develop) - same problems. I also tried some alterations to the Docker file (riak) - no luck either.

I have tried compiling the helm chart targets - conatiners etc and there is no problem here. (however I do not have kubernetes set up)
make -C ManagementService ; make -C DataLayerService ; make -C riak ; make -C GUI ; make -C TriggersFrontend
these are all ok.

[iakkus]

What are the error messages you see? What kind of alterations did you make to the Dockerfile?

Did you try my suggestion about the libssl on ubuntu 20.04?

[iakkus]

The ansible Makefile also creates log files for each component in the same folder. You can also check those out.

[crockercaria]

The suggestion about libssl on ubuntu 20.04 worked ok abd the riak images are built ok. I am in fact using a dedicated VM.
Currently the elastic search make is still building (+20hours..) python3, and java jobs are active and using CPU
The ./esconfig.sh script is running which seems like its one of the last things in the (elasticsearch) yaml file

Also the open ports are (except sshd)
vagrant@ubuntu2004:~/knix/deploy/ansible$ sudo lsof -nP -iTCP -sTCP:LISTEN
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd-r 539 systemd-resolve 13u IPv4 21755 0t0 TCP 127.0.0.53:53 (LISTEN)
dockerd 759 root 8u IPv6 23351 0t0 TCP *:2375 (LISTEN)
java 3207 elasticsearch 198u IPv6 48740 0t0 TCP 192.168.252.232:9300 (LISTEN)
java 3207 elasticsearch 212u IPv6 48761 0t0 TCP 192.168.252.232:9200 (LISTEN)

but
curl -X GET 'http://localhost:9200'
curl: (7) Failed to connect to localhost port 9200: Connection refused

maybe I should tun off ipv6 ?

[iakkus]

I assume that you run the curl command inside the VM?

The elasticsearch build running 20+ doesn't sound right. What are the specs of your VM?

[crockercaria]

I run curl inside the VM . i turned off ip6.
The host is Windows 10. vagrant with hyper-v provider. intel i7-4790 @3.6GHZ and the VM is 2 virtual cpus and 2GB RAM.
It's true that the VM is a bit slow but the make processes only seems to run mainly python3 jobs with bursts of java ....
I will turn off the other VM (even though it wasn't doing anything)
In the VM apt can download and install valgrind and dependencies (gdb/trace) a download of 34MB install of 178MB disk space in about 10s which is quite reasonable
I will wait a while longer !
What are the typical build times ?

[iakkus]

Honestly speaking, 2GB memory might not be enough, but let us know.

The build times depend on the spec you're building on: the more cpu and/or memory you have, the shorter.

[crockercaria]

Thanks for the help . I have upped the memory to 6GB as I see that each java process is consuming at 1.2GB and total memory use for the build is currently about at 4GB.

[iakkus]

Any progress?

[crockercaria]

I left it running remotely these days.
make has now just hung at the esconfig.sh script trying to do the
until curl ..:9200 line
So I guess some service has not started up that it was expecting to start up.
I have tried changing back and forth master to develop branch but the same thing happens.
I will consult the documentation at https://www.elastic.co/elasticsearch/ for any clue.

[crockercaria]

1. I followed the elasticsearch documentation for running elasticsearch with systemd
2. The frontend triigers did not make - I installed rust and then ran make in that directory
3. rerun the make in the deploy/ansible directory and ... "Installed KNIX MircoFunctions"
4. opening the webpage on a host browser gives me a spinner and then "Knix .. is currently down"

[iakkus]

Please check whether the management service is running by docker ps.

If not, then please go to the installation directory you specified in the settings file for ansible (by default, /opt/knix) and run start-all.sh in a terminal.

Then check again the management service and/or the browser.

[crockercaria]

Dear iakkus. Many thanks for your help.
The start-all.sh script was not a success ! The mfn servcies were not started - in docker or otherwise
Therefore using ubuntu 20.04 I have had to admit defeat.

I decided to have one last go and install a new VM - ubuntu18.04 LTS and assign all the memory to the VM and let Hyper-V dynamically manage the physical memory - it bumps along ok by assigning about 12GB of physical memory
The make process needs to be restarted several times (building in wrong order?) and the ansible playbook commands to be sometimes run seperartely (!)
e.g ansible-playbook -K -e "@settings.json" init_once.yaml

Other dependencies need to be installed seperately after checking the log files.

Anyway it finaly finsihed the make process and I can run the qr code example. So you can close this issue.

My recipe (ubuntu 18.04) is below

sudo apt-get update
sudo apt-get install -y lynx nodejs binutils build-essential manpages-dev python3 python3-dev python3-pip pkg-config
sudo apt-get install -y openssh-server openssh-client apt-transport-https ca-certificates curl gnupg lsb-release libssl-dev
sudo apt-get install -y default-jre default-jdk
sudo apt-get update

..setup ssh keys for ssh login without password
..install and check docker and add user to dockergroup

sudo apt update
sudo apt install -y software-properties-common
sudo apt-add-repository --yes --update ppa:ansible/ansible
sudo apt install -y ansible

sudo pip3 install --upgrade pip ##upgrade pip
sudo apt install cargo ##install rust

sudo pip3 install netaddr
sudo apt install python-netaddr ## !!

git clone knix ...

[iakkus]

Glad you could make it work and thanks for the steps you have taken.

We need to update the installation steps to include the missing dependencies for rust ansible. Will close this when that's done.

[wwt02]

Dear iakkus. A few days ago, I did the experiment and I met some problems. When I made in the directory of knix/deploy/ansible, the screen shows some errors. The content of screen shows as follows:
ansible-playbook -e "@settings.json" sandbox.yaml 2>&1 >sandbox.log.tmp
BECOME password:
make: *** [Makefile:42: sandbox.log] Error 2
The content of sandbox.log.tmp shows as follows:
TASK [build sandbox on host with proxy configuration] **************************
fatal: [ubuntu20]: FAILED! => {"changed": true, "cmd": ["docker", "-D", "-l", "debug", "build", "--network", "host", "--build-arg", "HTTP_PROXY=${HTT P_PROXY}", "--build-arg", "HTTPS_PROXY=${HTTPS_PROXY}", "--build-arg", "http_proxy=${HTTP_PROXY}", "--build-arg", "https_proxy=${HTTPS_PROXY}", "-t", "microfn/sandbox", "-f", "Dockerfile", "."], "delta": "0:00:00.448837", "end": "2021-10-16 06:30:51.925332", "msg": "non-zero return code", "rc": 10 0, "start": "2021-10-16 06:30:51.476495", "stderr": "The command '/bin/sh -c apt-get -y --no-install-recommends install build-essential' returned a n on-zero code: 100", "stderr_lines": ["The command '/bin/sh -c apt-get -y --no-install-recommends install build-essential' returned a non-zero code: 1 00"], "stdout": "Sending build context to Docker daemon 3.542MB\r\r\nStep 1/28 : FROM ubuntu:18.04\n ---> 5a214d77f5d7\nStep 2/28 : RUN apt-get upda te --fix-missing\n ---> Using cache\n ---> 8a57d5583a95\nStep 3/28 : RUN apt-get -y --no-install-recommends install build-essential\n ---> Running in c8831d15beea\nReading package lists...\nBuilding dependency tree...\nReading state information...\n\u001b[91mE\u001b[0m\u001b[91m: \u001b[0m\u001b[9 1mUnable to locate package build-essential\u001b[0m\u001b[91m\n\u001b[0m", "stdout_lines": ["Sending build context to Docker daemon 3.542MB", "", "S tep 1/28 : FROM ubuntu:18.04", " ---> 5a214d77f5d7", "Step 2/28 : RUN apt-get update --fix-missing", " ---> Using cache", " ---> 8a57d5583a95", "Step 3/28 : RUN apt-get -y --no-install-recommends install build-essential", " ---> Running in c8831d15beea", "Reading package lists...", "Building depen dency tree...", "Reading state information...", "\u001b[91mE\u001b[0m\u001b[91m: \u001b[0m\u001b[91mUnable to locate package build-essential\u001b[0m \u001b[91m", "\u001b[0m"]}
Although I tried many ways, I couldn’t solve this problem. I hope that I can get your help.

[iakkus]

Hello,

The package build-essential seems to be failing. It looks like it might have encountered a network error. Please try it again. If it still fails, then you can try:

• installing the missing package manually on the machine and retrying
• remove the sandbox image (i.e., microfn/sandbox and microfn/sandbox_java) and retrying
• add a dummy line at the beginning of the Dockerfile (i.e., Sandbox/Dockerfile) like RUN apt update and retrying
• manually build the sandbox on the target machine using the source

Hope this helps.

[MoralCode]

Was this issue resolved? I am also having issues using an ubuntu 20.04 target machine and seeing a Dependency is not satisfiable: libssl1.0.0 (>= 1.0.0) error via ansible.

To me it seems like libssl has a newer version on Ubuntu 20.04 and newer (sudo apt list --installed | grep ssl shows im on libssl1.1) Is there somewhere in the codebase I can update this dependency check to resolve this issue without encouraging people to downgrade critical security libraries?

Edit: upon looking a little deeper, this seems to be due to the fact that riak is built using an ubuntu 18.04 docker image (using libssl1.0.0, see

knix/riak/Dockerfile

Line 26 in 8996023

libssl1.0.0 \

), which causes an issue if the resulting package is installed on a 20.04 machine (see

knix/deploy/ansible/riak.yaml

Line 136 in 8996023

- name: install riak if not present

)

[MoralCode]

can confirm, setting up a new VM with ubuntu 18.04 seems to allow me to get past this issue