Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove all cross development settings #141

Merged
merged 1 commit into from
Jun 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 7 additions & 32 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
.PHONY: \
install \
clean \
build \
distribution-tarball \
test \
test-container \
publish \
development \
coverage \
coverage-html

Expand All @@ -22,16 +19,6 @@ ifeq ($(shell find . -name vendor), ./vendor)
BUILDFLAGS += -mod=vendor
endif

# -----------------------------------------------------------------------------

# Development constants
_PYTHON ?= python3
_PIP ?= pip3
_VENV ?= .venv3
_PRE_COMMIT ?= pre-commit
_CLIENT_ID ?= 00000000-0000-0000-0000-0000000000000
_SERVED_FILENAME ?= example_bash.yml

ifdef KEEP_TEST_CONTAINER
_CONTAINER_RM =
else
Expand All @@ -40,26 +27,21 @@ endif

# -----------------------------------------------------------------------------

all: clean build

install: .install .pre-commit

.install:
virtualenv --system-site-packages --python $(_PYTHON) $(_VENV); \
. $(_VENV)/bin/activate; \
$(_PIP) install --upgrade -r ./development/python/requirements.txt; \
touch $@
all: clean .pre-commit build

.pre-commit:
$(_PRE_COMMIT) install --install-hooks
pre-commit install --install-hooks
touch $@

clean:
rm -rf build
@rm -rf build/
@find . -name '.pre-commit' -exec rm -fr {} +
@find . -name 'coverage.out' -exec rm -rf {} +


build: $(GO_SOURCES)
mkdir -p build
go build $(BUILDFLAGS) -ldflags "$(LDFLAGS)" -o build/rhc-script-worker $^
CGO_ENABLED=0 go build $(BUILDFLAGS) -ldflags "$(LDFLAGS)" -o build/rhc-script-worker $^

distribution-tarball:
go mod vendor
Expand Down Expand Up @@ -90,10 +72,3 @@ coverage: test

coverage-html: test
go tool cover -html=coverage.out

publish:
. $(_VENV)/bin/activate; python development/python/mqtt_publish.py $(_CLIENT_ID) $(_SERVED_FILENAME)

development:
@podman-compose -f development/podman-compose.yml down
podman-compose -f development/podman-compose.yml up -d
115 changes: 10 additions & 105 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,113 +4,33 @@

# RHC Worker

Remote Host Configuration (rhc) worker for executing scripts on hosts
managed by Red Hat Insights. Interpreter used to execute the script is defined inside the supplied yaml file - served by insights.
Remote Host Configuration (rhc) worker for executing scripts on hosts managed
by Red Hat Insights. Interpreter used to execute the script is defined inside
the supplied yaml file - served by insights.

- [RHC Worker Script](#rhc-worker-script)
- [RHC Worker](#rhc-worker)
- [General workflow of the worker](#general-workflow-of-the-worker)
- [Getting started with local development](#getting-started-with-local-development)
- [Publish first message](#publish-first-message)
- [Worker playbooks](#worker-playbooks)
- [Custom playbook](#custom-playbook)
- [Convert2RHEL Playbook](#convert2rhel-playbook)
- [FAQ](#faq)
- [Are there special environment variables used by `rhc-worker-script`?](#are-there-special-environment-variables-used-by-rhc-worker-script)
- [Can I change behavior of `rhc-worker-script`?](#can-i-change-behavior-of-rhc-worker-script)
- [Can I change the location of `rhc-worker-script` config?](#can-i-change-the-location-of-rhc-worker-script-config)
- [Contact](#contact)
- [Package maintainers](#package-fmaintainers)
- [Package maintainers](#package-maintainers)

## General workflow of the worker

Everything starts when message is sent to rhcd. Worker then:

1. Picks up the message from rhcd
2. Downloads the worker playbook as temporary file (see [Worker playbooks](#worker-playbooks))
2. Downloads the worker playbook as temporary file (see [Worker playbooks](https://github.com/oamg/convert2rhel-insights-tasks/blob/main/playbooks/))
3. Verify the integrity of the playbook with `insights-client`
4. Executes the script
5. Reads stdout of the script
6. Sends the stdout wrapped in JSON back to rhcd

Then rhcd sends the message to upload service (with data from worker) in order to show the results in Insights UI - our setup for local development simulates the upload with minio storage.

## Getting started with local development

Almost everything that is needed for local development is placed in `development` folder.

Overview of what is needed:

- Script to be executed and data host serving our script
- Example is present inside the folder `development/nginx/data`
- **Set it up yourself** - see [Worker playbooks](#worker-playbooks) below
- System connected via rhc with running rhcd == the system on which the script will be executed
- **Set it up yourself** - for vagrant box see commands below

```bash
# Get a new centos-7 box
vagrant init eurolinux-vagrant/centos-7

# Install insights-client and rhc
...

# Connect via rhc
vagrant ssh -- -t 'rhc connect --server=$(RHSM_SERVER_URL) --username=$(RHSM_USERNAME) --password=$(RHSM_PASSWORD)'
# Run rhcd
vagrant ssh -- -t 'rhcd --log-level trace \
--socket-addr $(YGG_SOCKET_ADDR) \
--broker $(HOST_IP):1883 \
--topic-prefix yggdrasil \
--data-host $(HOST_IP):8000'
```

- MQTT broker for sending messages
- Set up as part of `make development` call
- Storage to simulate upload by the ingress service
- Set up as part of `make development` call

### Publish first message

1. Have system connected with rhc and running rhcd
- depends on you if you want to use vagrant or different approach
2. Start MQTT broker, data host for serving the script and minio storage
- You can take advantage of `make development` command to create neccessary containers, inspect `development/podman-compose.yml` for more details
3. Publish new message to broker
- [Optional] Change values in `development/python/mqtt_publish.py`
- `CLIENT_ID` - can be found in logs after running rhcd
- `SERVED_FILENAME` - one of the files inside `development/nginx/data`
- Call `make publish`
4. You should see logs in rhcd and file with stdout of your script uploaded to the minio storage
- Go to <http://localhost:9990/login> and use credentials from `.env` file

### Worker playbooks

There is an [example bash playbook](
https://github.com/oamg/rhc-worker-script/blob/main/development/nginx/data/example_bash.yaml)
available under `development/nginx/data`, with a minimal bash script to use
during the worker execution.

If there's a need to test any other playbook provided in this repository, one
must change what playbook will be used during the message consumption in the
[mqtt_publish.py](https://github.com/oamg/rhc-worker-script/blob/main/development/python/mqtt_publish.py#L22)
file with the name that corresponds the ones present in `development/nginx/data`. Currently, the ones available are:

1. [example_bash.yml](https://github.com/oamg/rhc-worker-script/blob/main/development/nginx/data/example_bash.yml)
2. [example_python.yml](https://github.com/oamg/rhc-worker-script/blob/main/development/nginx/data/example_python.yml)
3. [convert2rhel.yml](https://github.com/oamg/rhc-worker-script/blob/main/development/nginx/data/convert2rhel.yml)

#### Custom playbook

Create or update a yaml file inside the folder `development/nginx/data/*`.
Correct structure with exampe bash script can be seen below:

#### Convert2RHEL Playbook

A specialized [Convert2RHEL](https://github.com/oamg/convert2rhel) playbook can be found under the `development/nginx/data` as well. The playbook will take of the following functions:

1. Setup Convert2RHEL (Download certificates, repositories and etc...)
2. Set a couple of environment variables for the Convert2RHEl execution (Based on the `content_vars` defined in the playbook)
3. Run convert2rhel with default commands
4. A function to run any post-execution commands needed by the conversion (Currently empty.)
Then rhcd sends the message to upload service (with data from worker) in order
to show the results in Insights UI - our setup for local development simulates
the upload with minio storage.

## FAQ

Expand All @@ -122,22 +42,7 @@ Other than that there are no special variables, however if downloaded yaml file

### Can I change behavior of `rhc-worker-script`?

Yes, some values can be changed in the config file located at `/etc/rhc/workers/rhc-worker-script.yml`. After installing the `rhc-worker-script` package, a config file will be created with the default values required for the worker to start processing messages, **the config must have valid yaml format**, see all available fields below.

Example of full config (with default values):

```yaml
# rhc-worker-script configuration

# recipient directive to register with dispatcher
directive: "rhc-worker-script"

# whether to verify incoming yaml files
verify_yaml: true

# temporary directory in which the temporary files with executed bash scripts are created
temporary_worker_directory: "/var/lib/rhc-worker-script"
```
Yes, some values can be changed in the config file located at `/etc/rhc/workers/rhc-worker-script.yml`. After installing the `rhc-worker-script` package, a config file will be created with the default values required for the worker to start processing messages, **the config must have valid yaml format**, see all available fields in the [rhc-worker-script.yml](https://github.com/oamg/rhc-worker-script/blob/main/rhc-worker-script.yml) in the root of the repository.

### Can I change the location of `rhc-worker-script` config?

Expand Down
5 changes: 0 additions & 5 deletions development/.env

This file was deleted.

2 changes: 0 additions & 2 deletions development/mosquitto/mosquitto.conf

This file was deleted.

Loading