Enhance separate ingress solution

[thirdeyenick]

Summary

In the current documentation there are 2 options how to configure kubernetes ingresses to access Argo CD. We went with the second option and are using 2 separate ingresses together with nginx-ingress. However, we sometimes experienced issues when we used the Argo CD UI in parallel with the Argo CD cli app. When we tried to use the cli we sometimes got the following error:

argocd login <cli ingress host> --sso
FATA[0000] rpc error: code = Internal desc = transport: received the unexpected content-type "text/html"

Issuing the same command again afterwards worked flawless. In the nginx ingress logs we saw the following error messages:

2020/07/03 12:42:23 [error] 941#941: *12825628 no connection data found for keepalive http2 connection while sending request to upstream, client: <IP>, server: <cli ingress host>, request: "POST /cluster.SettingsService/Get HTTP/2.0", upstream: "grpc://172.16.2.9:8080", host: "<cli ingress host>:443"

We then created an issue for nginx-ingress , which sadly never got any attention. There is also this other issue which says that nginx does not support this (mixing http1 and http2 to the same target port).

For us the issue remained and it still seems to be the case that nginx-ingress can not handle HTTP1 and HTTP2 connections which need to be proxied to the same pod on the same port.

We are now working around the issue by using the annotation nginx.ingress.kubernetes.io/proxy-http-version: "1.0" on the normal web UI ingress. This disables keep-alive connections from nginx-ingress to the argocd server pod and our cli commands now work all the time.

Nevertheless, this is just a workaround and we would like to use HTTP/1.1 as the upstream connection protocol again.

Would it be possible to provide an option to have a HTTP1 and a HTTP2 port exposed by argocd-server? Or is there any other solution known for this issue?

Motivation

Please see above

[balusarakesh]

@thirdeyenick
FYI: we are still seeing the rpc error: code = Internal desc = transport: received the unexpected content-type "text/html" errors even after setting nginx.ingress.kubernetes.io/proxy-http-version: "1.0", is there anything else we can do fix those errors?

[josecsotomorales]

I'm also getting this issue periodically

[lblazewski]

Also an issue here, when trying to use https://docs.konghq.com/kubernetes-ingress-controller/latest/ which would be configured accordingly to the 2 services setup. Kong is able to proxy grpc but the keepalive connections which are stored in the pool as ip|port (where ip is argocd pod ip and port is 8080) are being reused which cause intermittent failures both on the http and grcp calls.

[josecsotomorales]

FYI folks going to provide my ingresses annotations that apparently fixes those intermittent 502 unexpected content-type "text/html" issues:

HTTP:

  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
    nginx.ingress.kubernetes.io/proxy-http-version: "1.0"

GRPC:

  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    nginx.ingress.kubernetes.io/configuration-snippet: |
      grpc_next_upstream error timeout http_502 non_idempotent;
      grpc_next_upstream_tries 3;

Hope it helps.

[Whisper40]

Hello @alexmt, we are encountering this problem too, the problem is periodic, sometimes it's a succession of KO calls. We use Nginx 1.10.1 and ArgoCD 2.11.7.
We get the following error on our side:

time=“2024-09-05T09:38:31Z” level=fatal msg=“rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \”text/plain; charset=utf-8\"”

[Angelin01]

Using Kubernetes Ingress Nginx, adding the nginx.ingress.kubernetes.io/service-upstream: "true" annotation to the web ingress seems to mitigate the issue. It seems that anything that makes the Ingress Controller not share connections between HTTP 1.1 and HTTP 2.0 should be enough.

It seems that, for ArgoCD to avoid this issue entirely, they should allow exposing the gRPC service in a separate port from the web service.

[Whisper40]

Hello @crenshaw-dev , do you think this problem will be fixed ?