Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Redash Workers Becomes Inactive Automatically #5885

Closed
shubhwip opened this issue Jan 24, 2023 · 0 comments · Fixed by #5886
Closed

Redash Workers Becomes Inactive Automatically #5885

shubhwip opened this issue Jan 24, 2023 · 0 comments · Fixed by #5886

Comments

@shubhwip
Copy link
Contributor

shubhwip commented Jan 24, 2023 •
edited
Loading

Issue Summary

Everyday we see redash workers become inactive/stale and they are unable to process queries.
This has mostly happened for adhoc workers and sometimes for other workers too.
We have to restart the workers everyday and then everything works fine again. This is very big pain.

Steps to Reproduce

No clear steps are there to reproduce
But one scenario could be if when workers are about to start and connect to redis instance if they are unable to connect then worker health check fails and they never come into alive state.

This is a bug because it requires manual restart always.

Technical details:

  • Redash Version: redash/redash:10.1.0.b50633
  • Browser/OS: GKE
  • How did you install Redash: Helm Charts

Further Investigation

This has been reported earlier as well with different title and problems
getredash/contrib-helm-chart#124
getredash/contrib-helm-chart#72
#5801

We should add a liveness probe to make restarts automatic when supervisord workers are dead.
@shubhwip shubhwip mentioned this issue Jan 24, 2023
Merged
@shubhwip shubhwip mentioned this issue Jan 24, 2023
Merged
10 tasks
shubhwip added a commit to shubhwip/redash that referenced this issue Mar 27, 2023
@shubhwip
Add liveness check script for workers
5a86aff 
closes getredash#5885
arikfr added a commit that referenced this issue Mar 27, 2023
@shubhwip @arikfr
Add liveness check for workers (#5886)
5b9fd40 
* Add liveness check script for workers

closes #5885

* delete extra script

* Rename worker_healthcheck -> workers_healthcheck

---------

Co-authored-by: Arik Fraimovich <[email protected]>
AkaashK pushed a commit to tharzeez/redash that referenced this issue Jul 18, 2023
@shubhwip @arikfr
Add liveness check for workers (getredash#5886)
ca45020 
* Add liveness check script for workers

closes getredash#5885

* delete extra script

* Rename worker_healthcheck -> workers_healthcheck

---------

Co-authored-by: Arik Fraimovich <[email protected]>
harveyrendell pushed a commit to pushpay/redash that referenced this issue Jan 8, 2025
@shubhwip @arikfr @harveyrendell
Add liveness check for workers (getredash#5886)
6cd3dcb 
* Add liveness check script for workers

closes getredash#5885

* delete extra script

* Rename worker_healthcheck -> workers_healthcheck

---------

Co-authored-by: Arik Fraimovich <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add liveness check for workers shubhwip/redash
1 participant
@shubhwip