Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Schedules not launching without notice in the Interface / API #15878

Open
5 of 11 tasks
mqufflc opened this issue Mar 10, 2025 · 1 comment
Open
5 of 11 tasks

Schedules not launching without notice in the Interface / API #15878

mqufflc opened this issue Mar 10, 2025 · 1 comment
Labels
community needs_triage type:bug

Comments

@mqufflc
Copy link

mqufflc commented Mar 10, 2025

Please confirm the following

  • I agree to follow this project's code of conduct.
  • I have checked the current issues for duplicates.
  • I understand that AWX is open source software provided for free and that I might not receive a timely response.
  • I am NOT reporting a (potential) security vulnerability. (These should be emailed to [email protected] instead.)

Bug Summary

On our AWX instance, which is running for 4 years now, we noticed that some schedules are not launching from time to time everyday for a month now. It seems to be a minority as a lot of schedules are still respected.

We haven't found anything to explain it, and there are no error whatsoever in the web interface / API. It is like no schedules were created in the first place.
We have found a python stacktrace in the awx task logs every time the problem seems to occur (see below).

On one of the workflow scheduled job, one of our user updated the schedule to launch it after noticing the schedule missed the time frame. With the new time frame, the schedule executed correctly.

AWX version

24.6.1

Select the relevant components

  • UI
  • UI (tech preview)
  • API
  • Docs
  • Collection
  • CLI
  • Other

Installation method

kubernetes

Modifications

yes

Ansible version

2.15.12

Operating system

centOS Stream 9

Web browser

No response

Steps to reproduce

Create a Schedule for a Workflow Job Template.

Expected results

The Workflow Job is executed respecting the schedule.

Actual results

The Workflow Job is not created nor any error that could help us solve the issue.
The following stacktrace can be found in awx-task pods:

ERROR 2025-03-10T04:00:15.541271139Z [resource.labels.containerName: awx-task] Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/dispatch/worker/task.py", line 103, in perform_work result = self.run_callable(body)
ERROR 2025-03-10T04:00:15.541292019Z [resource.labels.containerName: awx-task] ^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2025-03-10T04:00:15.541297069Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/dispatch/worker/task.py", line 78, in run_callable
ERROR 2025-03-10T04:00:15.541302689Z [resource.labels.containerName: awx-task] return _call(*args, **kwargs)
ERROR 2025-03-10T04:00:15.541308089Z [resource.labels.containerName: awx-task] ^^^^^^^^^^^^^^^^^^^^^^
ERROR 2025-03-10T04:00:15.541322939Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/system.py", line 745, in awx_periodic_scheduler
ERROR 2025-03-10T04:00:15.541329749Z [resource.labels.containerName: awx-task] schedule.update_computed_fields() # To update next_run timestamp.
ERROR 2025-03-10T04:00:15.541334729Z [resource.labels.containerName: awx-task] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2025-03-10T04:00:15.541339849Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/schedules.py", line 301, in update_computed_fields
ERROR 2025-03-10T04:00:15.541345319Z [resource.labels.containerName: awx-task] self.unified_job_template.update_computed_fields()
ERROR 2025-03-10T04:00:15.541350699Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/unified_jobs.py", line 292, in update_computed_fields
ERROR 2025-03-10T04:00:15.541355859Z [resource.labels.containerName: awx-task] self.save(update_fields=['next_schedule', 'next_job_run'])
ERROR 2025-03-10T04:00:15.541361089Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/jobs.py", line 356, in save
ERROR 2025-03-10T04:00:15.541366169Z [resource.labels.containerName: awx-task] return super(JobTemplate, self).save(*args, **kwargs)
ERROR 2025-03-10T04:00:15.541371239Z [resource.labels.containerName: awx-task] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2025-03-10T04:00:15.541376749Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/unified_jobs.py", line 305, in save
ERROR 2025-03-10T04:00:15.541382259Z [resource.labels.containerName: awx-task] super(UnifiedJobTemplate, self).save(*args, **kwargs)
ERROR 2025-03-10T04:00:15.541387489Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/polymorphic/models.py", line 87, in save
ERROR 2025-03-10T04:00:15.541392509Z [resource.labels.containerName: awx-task] return super().save(*args, **kwargs)
ERROR 2025-03-10T04:00:15.541413019Z [resource.labels.containerName: awx-task] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2025-03-10T04:00:15.541418779Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/base.py", line 344, in save
ERROR 2025-03-10T04:00:15.541424159Z [resource.labels.containerName: awx-task] super(PrimordialModel, self).save(*args, **kwargs)
ERROR 2025-03-10T04:00:15.541429999Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/base.py", line 192, in save
ERROR 2025-03-10T04:00:15.541437239Z [resource.labels.containerName: awx-task] super(CreatedModifiedModel, self).save(*args, **kwargs)
ERROR 2025-03-10T04:00:15.541442669Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/mixins.py", line 606, in save
ERROR 2025-03-10T04:00:15.541448369Z [resource.labels.containerName: awx-task] super().save(*args, **kwargs)
ERROR 2025-03-10T04:00:15.541453699Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/base.py", line 814, in save
ERROR 2025-03-10T04:00:15.541459659Z [resource.labels.containerName: awx-task] self.save_base(
ERROR 2025-03-10T04:00:15.541465039Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/base.py", line 892, in save_base
ERROR 2025-03-10T04:00:15.541470349Z [resource.labels.containerName: awx-task] post_save.send(
ERROR 2025-03-10T04:00:15.541475839Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
ERROR 2025-03-10T04:00:15.541481239Z [resource.labels.containerName: awx-task] return [
ERROR 2025-03-10T04:00:15.541486259Z [resource.labels.containerName: awx-task] ^
ERROR 2025-03-10T04:00:15.541491889Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
ERROR 2025-03-10T04:00:15.541497179Z [resource.labels.containerName: awx-task] (receiver, receiver(signal=self, sender=sender, **named))
ERROR 2025-03-10T04:00:15.541502029Z [resource.labels.containerName: awx-task] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2025-03-10T04:00:15.541506629Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/fields.py", line 362, in _post_save
ERROR 2025-03-10T04:00:15.541511579Z [resource.labels.containerName: awx-task] update_role_parentage_for_instance(latest_instance)
ERROR 2025-03-10T04:00:15.541517179Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/fields.py", line 226, in update_role_parentage_for_instance
ERROR 2025-03-10T04:00:15.541522829Z [resource.labels.containerName: awx-task] cur_role.parents.remove(*list(removals))
ERROR 2025-03-10T04:00:15.541528239Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/fields/related_descriptors.py", line 1164, in remove
ERROR 2025-03-10T04:00:15.541533579Z [resource.labels.containerName: awx-task] self._remove_items(self.source_field_name, self.target_field_name, *objs)
ERROR 2025-03-10T04:00:15.541538759Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/fields/related_descriptors.py", line 1496, in _remove_items
ERROR 2025-03-10T04:00:15.541544519Z [resource.labels.containerName: awx-task] signals.m2m_changed.send(
ERROR 2025-03-10T04:00:15.541549729Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/dispatch/dispatcher.py", line 176, in send
ERROR 2025-03-10T04:00:15.541554909Z [resource.labels.containerName: awx-task] return [
ERROR 2025-03-10T04:00:15.541559919Z [resource.labels.containerName: awx-task] ^
ERROR 2025-03-10T04:00:15.541565149Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/dispatch/dispatcher.py", line 177, in <listcomp>
ERROR 2025-03-10T04:00:15.541570939Z [resource.labels.containerName: awx-task] (receiver, receiver(signal=self, sender=sender, **named))
ERROR 2025-03-10T04:00:15.541575889Z [resource.labels.containerName: awx-task] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2025-03-10T04:00:15.541581249Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/rbac.py", line 694, in sync_parents_to_new_rbac
ERROR 2025-03-10T04:00:15.541602079Z [resource.labels.containerName: awx-task] parent_role = Role.objects.get(id=role_id)
ERROR 2025-03-10T04:00:15.541608079Z [resource.labels.containerName: awx-task] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2025-03-10T04:00:15.541613389Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
ERROR 2025-03-10T04:00:15.541618809Z [resource.labels.containerName: awx-task] return getattr(self.get_queryset(), name)(*args, **kwargs)
ERROR 2025-03-10T04:00:15.541635319Z [resource.labels.containerName: awx-task] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2025-03-10T04:00:15.541641359Z [resource.labels.containerName: awx-task] File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/django/db/models/query.py", line 637, in get
ERROR 2025-03-10T04:00:15.541646429Z [resource.labels.containerName: awx-task] raise self.model.DoesNotExist(
ERROR 2025-03-10T04:00:15.541651839Z [resource.labels.containerName: awx-task] awx.main.models.rbac.Role.DoesNotExist: Role matching query does not exist.

Additional information

We only modified the execution environment provided to add some packages and certificates from our internal Information System.
No modification to the awx-task or awx-web images.

We use a managed GCP Cloud SQL database for AWX database.
AWX is deployed to GKE using awx-operator v2.19.1
@fosterseth
Copy link
Member

Have you tried to see if this problem still occurs on awx-operator:devel and awx:devel? as v2.19.1 is dated at this point

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
community needs_triage type:bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fosterseth @mqufflc