Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bazel server caches the google default creds and won't reload them even when they're expired #23368

Open
novas0x2a opened this issue Aug 20, 2024 · 4 comments
Assignees
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug

Comments

@novas0x2a
Copy link

novas0x2a commented Aug 20, 2024

Description of the bug:

It seems like the bazel server won't refresh the creds generated from --google_default_credentials even when they're invalid. This seems to apply to both the normal expiration process and the artificial expiration process (early revocation) that I've documented below in the repro (it's just easier to see it in a reasonable amount of time using revocation instead of waiting for expiration)

Which category does this issue belong to?

Core, Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

# make sure the server's already down
bazel shutdown

# generate auth
gcloud auth application-default login

# make bazel load the creds
bazel build --google_default_credentials --remote_cache=MYHOST ...

# expire the auth
gcloud auth application-default revoke

# wait for the auth to actually expire on the remote
# as per docs:
# https://cloud.google.com/apigee/docs/api-platform/security/oauth/validating-and-invalidating-access-tokens
#     Warning: OAuth tokens are cached for three minutes (180 seconds); therefore, a revoked 
#     token may still succeed for up to three minutes, until its cache limit expires.)
sleep 300

# rerun the build again (without shutting the server down):
# This should _correctly_ show warnings from the dead token
bazel clean
bazel build --google_default_credentials --remote_cache=MYHOST ...

# log in again
gcloud auth application-default login

# rerun the build (without shutting the server down):
# This will try to use the invalid token (generating the same warnings), and won't 
# reload the token from disk.
bazel clean
bazel build --google_default_credentials --remote_cache=MYHOST ...

# shut the server down
bazel shutdown

# rerun the build again
# This will load the valid token and work again
bazel clean
bazel build --google_default_credentials --remote_cache=MYHOST ...

Which operating system are you running Bazel on?

linux (ubuntu 22.04)

What is the output of bazel info release?

release 7.3.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

n/a

What's the output of git remote get-url origin; git rev-parse HEAD ?

A private repo, sorry.

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

This seems to be similar (or maybe the same?) as these issues:

Any other information, logs, or outputs that you want to share?

The generated warnings are:

WARNING: Remote Cache: 401 Unauthorized
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AuthenticationRequired</Code><Message>Authentication required.</Message></Error>
com.google.devtools.build.lib.remote.http.HttpException: 401 Unauthorized
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AuthenticationRequired</Code><Message>Authentication required.</Message></Error>
	at com.google.devtools.build.lib.remote.http.HttpDownloadHandler.channelRead0(HttpDownloadHandler.java:134)
	at com.google.devtools.build.lib.remote.http.HttpDownloadHandler.channelRead0(HttpDownloadHandler.java:43)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: com.google.auth.oauth2.GoogleAuthException: com.google.api.client.http.HttpResponseException: 400 Bad Request
POST https://oauth2.googleapis.com/token
{
  "error": "invalid_grant",
  "error_description": "Token has been expired or revoked."
}
		at com.google.auth.oauth2.GoogleAuthException.createWithTokenEndpointResponseException(GoogleAuthException.java:127)
		at com.google.auth.oauth2.GoogleAuthException.createWithTokenEndpointResponseException(GoogleAuthException.java:143)
		at com.google.auth.oauth2.UserCredentials.doRefreshAccessToken(UserCredentials.java:279)
		at com.google.auth.oauth2.UserCredentials.refreshAccessToken(UserCredentials.java:191)
		at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:257)
		at com.google.auth.oauth2.OAuth2Credentials$1.call(OAuth2Credentials.java:254)
		at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
		at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
		at com.google.auth.oauth2.OAuth2Credentials$AsyncRefreshResult.executeIfNew(OAuth2Credentials.java:580)
		at com.google.auth.oauth2.OAuth2Credentials.refresh(OAuth2Credentials.java:180)
		at com.google.devtools.build.lib.authandtls.credentialhelper.CredentialHelperCredentials.refresh(CredentialHelperCredentials.java:131)
		at com.google.devtools.build.lib.remote.http.HttpCacheClient.refreshCredentials(HttpCacheClient.java:841)
		at com.google.devtools.build.lib.remote.http.HttpCacheClient.lambda$get$6(HttpCacheClient.java:549)
		at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
		at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557)
		at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
		at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
		at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
		at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:110)
		at io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:89)
		at com.google.devtools.build.lib.remote.http.HttpDownloadHandler.failAndReset(HttpDownloadHandler.java:219)
		at com.google.devtools.build.lib.remote.http.HttpDownloadHandler.channelRead0(HttpDownloadHandler.java:135)
		... 42 more
	Caused by: com.google.api.client.http.HttpResponseException: 400 Bad Request
POST https://oauth2.googleapis.com/token
{
  "error": "invalid_grant",
  "error_description": "Token has been expired or revoked."
}
		at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
		at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)
		at com.google.auth.oauth2.UserCredentials.doRefreshAccessToken(UserCredentials.java:277)
		... 61 more
WARNING: Remote Cache: 401 Unauthorized
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AuthenticationRequired</Code><Message>Authentication required.</Message></Error>
com.google.devtools.build.lib.remote.http.HttpException: 401 Unauthorized
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AuthenticationRequired</Code><Message>Authentication required.</Message></Error>
	at com.google.devtools.build.lib.remote.http.HttpDownloadHandler.channelRead0(HttpDownloadHandler.java:134)
	at com.google.devtools.build.lib.remote.http.HttpDownloadHandler.channelRead0(HttpDownloadHandler.java:43)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)
@github-actions github-actions bot added team-Core Skyframe, bazel query, BEP, options parsing, bazelrc team-Remote-Exec Issues and PRs for the Execution (Remote) team labels Aug 20, 2024
@tjgq tjgq removed the team-Core Skyframe, bazel query, BEP, options parsing, bazelrc label Aug 22, 2024
@coeuvre coeuvre self-assigned this Aug 27, 2024
@novas0x2a
Copy link
Author

novas0x2a commented Sep 4, 2024

This also applies to creds loaded from --credential_helper, though that protocol does have an expires field now, which gives a bit more flexibility. I'm not sure if it makes sense to leave a known-broken token cached, even if the expires field says it isn't expired yet, but maybe?

@joeleba joeleba added P2 We'll consider working on this in future. (Assignee optional) and removed untriaged labels Sep 10, 2024
@acecilia
Copy link

acecilia commented Nov 21, 2024

that protocol does have an expires field #21429, which gives a bit more flexibility

I could not find a way to obtain the expiration date for a google cloud token, so I was not able to provide expires field via the credentials helper. So not sure the expires field in the credentials helper helps in this case.

EDIT: is possible to obtain the expiration time from inside the credential helper with something like:

readonly TOKEN="$(gcloud auth print-access-token)"

# Get expiration time in RFC3339 format
readonly EXPIRY_UNIX_EPOCH="$(curl --silent "https://oauth2.googleapis.com/tokeninfo?access_token=${TOKEN}" | grep -o '"exp": *"[^"]*"' | awk -F': ' '{print $2}' | tr -d '"')"
readonly EXPIRY_RFC3339="$(date -u -r ${EXPIRY_UNIX_EPOCH} +"%Y-%m-%dT%H:%M:%SZ")"

# For context about the format of this json, see:
# * https://github.com/EngFlow/credential-helper-spec/blob/7df9bef60ef05636fd93114a17a7b2ea08143af6/schemas/get-credentials-response.schema.json
# * https://github.com/EngFlow/credential-helper-spec/blob/7df9bef60ef05636fd93114a17a7b2ea08143af6/spec.md
# * https://github.com/bazelbuild/bazel/blob/222d47f3bd57370f9a462ebdd86dfe1510795cd2/src/main/java/com/google/devtools/build/lib/authandtls/credentialhelper/GetCredentialsResponse.java
cat << EOF
{
  "headers": {
    "Authorization": ["Bearer ${TOKEN}"]
  },
  "expires": "${EXPIRY_RFC3339}"
}
EOF

@acecilia
Copy link

acecilia commented Nov 21, 2024

Also it looks like, if using --credential_helper, when a credential is cached with:

common --credential_helper_cache_duration=60m

Changing to:

common --credential_helper_cache_duration=0

Does not have effect: the credential stays cached. Will only take effect after running bazel clean or bazel shutdown

@tjgq
Copy link
Contributor

tjgq commented Nov 21, 2024

(Filed a separate issue to keep the discussion here focused on --google_default_credentials.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Remote-Exec Issues and PRs for the Execution (Remote) team type: bug
Projects
None yet
Development

No branches or pull requests

8 participants