Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Authentication is temporarily unavailable when creating Azure keyvault backed secret scope #338

Open
JCZuurmond opened this issue Oct 12, 2020 · 41 comments
Labels
platform issue is related to platform and not CLI

Comments

@JCZuurmond
Copy link

We get the following message when creating a secret scope with an Azure keyvault back end in the CI.

 { 'error_code': 'TEMPORARILY_UNAVAILABLE',
  'message': 'Authentication is temporarily unavailable. Please try again '
             'later.'}

Locally it works, the aad-token is set in the CI script, I have tried it at different times. Anyone can help with what to do with this message?

@michalmlaka
Copy link

Hi,

we observed a same issue. Its connected to aad-token authorization. I am able to connect and list secret scopes but not to create a new one.

We tried to call Databricks API directly but it is same there (so it is not problem of CLI). We are not able to deploy ntbs or create secret scope with AAD token.

@JCZuurmond
Copy link
Author

@michalmlaka : Thanks for adding that it is an API thing. We'll see if they fix it there.

@aravish
Copy link

aravish commented Nov 3, 2020

This currently does not work with AAD tokens acquired with Service principals

“Service Principals cannot be used with the AKV secret integration. This is because they do not support the on-behalf-of flow. We are requesting this of the AAD team and working on documenting it. You need to use a user's AAD token.”

In the interim, use below approach to create secret scope using user's AAD token

Obtain token using user creds

from adal import AuthenticationContext

authority_host_url = "https://login.microsoftonline.com/"
# the Application ID of  AzureDatabricks
azure_databricks_resource_id = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"

# Required user input
user_parameters = {
   "tenant" : "fa5cf446-0f25-4992-9e15-8723c0a0b170",
   "client_id" : "5662111a-38a8-403c-9d9a-f9b690985796",
   "username" : "user_upn",
   "password" : "xxxxxxxxxxxxxxxxx"
}

# configure AuthenticationContext
# authority URL and tenant ID are used
authority_url = authority_host_url + user_parameters['tenant']
context = AuthenticationContext(authority_url)

# API call to get the token
token_response = context.acquire_token_with_username_password(
  azure_databricks_resource_id,
  user_parameters['username'],
  user_parameters['password'],
  user_parameters['client_id']
)

access_token = token_response['accessToken']
refresh_token = token_response['refreshToken']
access_token

REST API request to create secret scope

POST https://adb-6187942982036570.10.azuredatabricks.net/api/2.0/secrets/scopes/create HTTP/1.1
Authorization: Bearer <token>

# For Keyvault backed scope
{
    "scope":  "restapiaad-keyvault-scope",
    "initial_manage_principal":  "users",
    "backend_azure_keyvault":  {
                                   "dns_name":  "https://aravish369vault.vault.azure.net/",
                                   "resource_id":  "/subscriptions/f34f9848-8b93-4cfb-9ced-0c4548a74cce/resourceGroups/aravishexternalrsgrp/providers/Microsoft.KeyVault/vaults/aravish369vault"
                               },
    "scope_backend_type":  "AZURE_KEYVAULT"
}

@gramhagen
Copy link

I am able to authenticate with AAD tokens to create a databricks personal access token, then authenticate again with the api using the databricks token to create secret scopes. I'm not sure if that works with keyvault backends, but I was running into the same issue just creating databricks backend scopes

@aravish
Copy link

aravish commented Nov 11, 2020

@gramhagen,
Ultimately the AAD token should be used to create the secret scopes which should work as expected and not run into issues.

@michalmlaka
Copy link

Unfortunately does not working. Run into these issues:

https://forums.databricks.com/questions/50626/azure-databricks-api-20-cant-create-azure-keyvault.html

@hannwang1
Copy link

@aravish , Is there ETA to fix the issue that unable to create Azure Key Vault scope using Service Principal token? Using user AAD token is not a good solution for automation that is running on service principal. We cannot store personal password in automation.

We are automating the Azure databricks configuration, including Azure resource creation, databricks cluster creation, Azure key Vault secret scope creation with CI/CD. This issue is blocking us to automate the process.

@sushi1998
Copy link
Contributor

sushi1998 commented Dec 22, 2020

I believe the PMs were talking to msft about enabling AKV creation with service principal tokens. @annashres might be the best person to talk to about this?

@hannwang1
Copy link

I believe the PMs were talking to msft about enabling AKV creation with service principal tokens. @AnnaShrestinan might be the best person to talk to about this?

Thanks @sushi1998 for the info. How to contact AnnaShrestinan?

@michalmlaka
Copy link

We tried it once again. It looks better now. (error msg.).

{"error_code":"CUSTOMER_UNAUTHORIZED","message":"Unable to grant read/list permission to Databricks service principal to KeyVault 'https://keyvaultname.vault.azure.net/': key not found: https://graph.windows.net/"}(base)

I assigned "Directory.Read.All" permission for AAD Graph API (https://graph.windows.net/). But it is not enough. What else do I need?

@JCZuurmond
Copy link
Author

JCZuurmond commented Feb 2, 2021

I think we are waiting to hear from @AnnaShrestinan like @sushi1998 mentioned. To the best of my knowledge it does not have to do with setting the right permissions for your SP.

@KevSpally-collab
Copy link

Any update on this, i am running into the same issue?

@JCZuurmond
Copy link
Author

Not that I am aware off. What I know: databricks people are talking to Azure people. It most likely takes a while: months not weeks

@emerrf
Copy link

emerrf commented Mar 26, 2021

+1 same issue here. Please fix it. Creating a Secret Scope using a Service Principal with AAD Token is still not working.

@dkataria1990
Copy link

Same issue here, please fix as in my company the only way we can do things in higher envs is with SPN so it is not possible to config any other way!

@annashres
Copy link

We are working with Azure to enable the use of Service Principals for creating an AKV-backed secret scope. @hpsin from Azure AD will have updates on the timeline.

@mcalnd70
Copy link

@annashres @hpsin Has there been any updates on this with timelines etc.? The alternative way is just too much rework for CI/CD processes right now, especially as I am using a regular SP for everything else in the Databricks CLI

Thanks

@eosho
Copy link

eosho commented May 26, 2021

Any updates on this? I am also having issues trying to set up key vault backed secret scope.

@ivo-o
Copy link

ivo-o commented Jun 8, 2021

+1 only manual step in a huge deployment currently. Any updates on this? @hpsin

@hpsin
Copy link

hpsin commented Jun 8, 2021

We don't have timelines to announce right now on when SP impersonation is going to be supported. We're investigating how we build this - it requires the creation of a new class of scopes (delegation scopes for service principals) and updating the authorization logic of every API. We understand the importance of this feature for automation flows, and building it securely is our number one concern.

@m1nkeh
Copy link

m1nkeh commented Jun 8, 2021

We don't have timelines to announce right now on when SP impersonation is going to be supported. We're investigating how we build this - it requires the creation of a new class of scopes (delegation scopes for service principals) and updating the authorization logic of every API. We understand the importance of this feature for automation flows, and building it securely is our number one concern.

thanks for the update 👌

@atul-ram
Copy link

atul-ram commented Nov 9, 2021

is this still in progress... Any updates !

@ghost
Copy link

ghost commented Nov 29, 2021

@hpsin why not just use a managed identity like azure datafactory ?? this will probably also solve some other problems related to cloud auth, currently when creating scope on UI, how the refresh token is passed to clusters ?? is the user AAD Token stored somewhere permanently ??, what happens if the user token is revoked ??

@mallecespedes
Copy link

mallecespedes commented Mar 1, 2022

Any updates on this? @hpsin @annashres

@hpsin
Copy link

hpsin commented Mar 3, 2022

The creation of a new class of scope is almost complete, and the team is hoping to test soon. @psignoret and @annashres should be able to provide updates from here as I've transitioned away from Azure AD.

@jaokeeffe
Copy link

jaokeeffe commented Apr 5, 2022

Just checking in to see how this request is tracking? Lots of users are eagerly awaiting this capability!

@nfx nfx added feature New enhancement or feature request platform issue is related to platform and not CLI and removed feature New enhancement or feature request labels May 4, 2022
@lordlinus
Copy link

+1 looking for update on this one

@anocendi
Copy link

anocendi commented Jun 8, 2022

+1 @annashres and @psignoret, could you give the community an update regarding this issue? We are all eagerly waiting for this feature's arrival!

@dabruehl
Copy link

+1 I run into same issue. Any updates after 1.5 years?

@mwillebrands
Copy link

@annashres @hpsin any update?

@ralphke
Copy link

ralphke commented Jul 22, 2022

@hpsin @annashres shouldn't the Azure AAD OAuth 2.0 on-behalf-of-flow improvement help to implement this in DBX?
https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-on-behalf-of-flow

@annashres
Copy link

Hi @ralphke and @mwillebrands, @hpsin and I have both switched roles. Please refer to @nfx for updates from the product team.

@nfx
Copy link
Contributor

nfx commented Jul 22, 2022

@ralphke in scope of databricks/terraform-provider-databricks#1490 we’ll know if it’s already possible or not.

@ghost
Copy link

ghost commented Jul 24, 2022

Unsubscribing from ticket.

@crystalgeek
Copy link

@nfx - Are there any updates on this - do we have a timeline?

@CaishenNefri
Copy link

Hey, Do we have roadmap for this issue?

@avinashpancham
Copy link

Hi @nfx, sorry for the spam, but do you have new information on this issue since the 22nd of July? Issue #1490 is still open, so I am hoping that would help us out.

@pietern
Copy link
Contributor

pietern commented Sep 30, 2022

We don't have new information to share on this topic.

Per the documentation at https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-on-behalf-of-flow:

The OBO flow only works for user principals at this time. A service principal cannot request an app-only token, send it to an API, and have that API exchange that for another token that represents that original service principal. Additionally, the OBO flow is focused on acting on another party's behalf, known as a delegated scenario - this means that it uses only delegated scopes, and not application roles, for reasoning about permissions. Roles remain attached to the principal (the user) in the flow, never the application operating on the users behalf.

This is the flow we need to create an AKV backed secret scope on behalf of an SP.

@jjgriff93
Copy link

Do we have an update on this? It's hard to believe that this still isn't supported.

@cabral1888
Copy link

Any update on this?

@psignoret
Copy link

@pietern @jjgriff93 @cabral1888 The statement in the article @pietern cites is true in the general case, but Azure Databricks has an exception and is able to make an OBO request on behalf of a service principal ("service principal on-behalf-of", or "SP OBO") to Azure Resource Manager. The Databricks team are working on making the necessary updates to make use of this when creating Azure Key Vault backed secret scopes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
platform issue is related to platform and not CLI
Projects
None yet
Development

No branches or pull requests