Authentication is temporarily unavailable when creating Azure keyvault backed secret scope

[JCZuurmond]

We get the following message when creating a secret scope with an Azure keyvault back end in the CI.

 { 'error_code': 'TEMPORARILY_UNAVAILABLE',
  'message': 'Authentication is temporarily unavailable. Please try again '
             'later.'}

Locally it works, the aad-token is set in the CI script, I have tried it at different times. Anyone can help with what to do with this message?

[michalmlaka]

Hi,

we observed a same issue. Its connected to aad-token authorization. I am able to connect and list secret scopes but not to create a new one.

We tried to call Databricks API directly but it is same there (so it is not problem of CLI). We are not able to deploy ntbs or create secret scope with AAD token.

[JCZuurmond]

@michalmlaka : Thanks for adding that it is an API thing. We'll see if they fix it there.

[aravish]

This currently does not work with AAD tokens acquired with Service principals

“Service Principals cannot be used with the AKV secret integration. This is because they do not support the on-behalf-of flow. We are requesting this of the AAD team and working on documenting it. You need to use a user's AAD token.”

In the interim, use below approach to create secret scope using user's AAD token

Obtain token using user creds

from adal import AuthenticationContext

authority_host_url = "https://login.microsoftonline.com/"
# the Application ID of  AzureDatabricks
azure_databricks_resource_id = "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d"

# Required user input
user_parameters = {
   "tenant" : "fa5cf446-0f25-4992-9e15-8723c0a0b170",
   "client_id" : "5662111a-38a8-403c-9d9a-f9b690985796",
   "username" : "user_upn",
   "password" : "xxxxxxxxxxxxxxxxx"
}

# configure AuthenticationContext
# authority URL and tenant ID are used
authority_url = authority_host_url + user_parameters['tenant']
context = AuthenticationContext(authority_url)

# API call to get the token
token_response = context.acquire_token_with_username_password(
  azure_databricks_resource_id,
  user_parameters['username'],
  user_parameters['password'],
  user_parameters['client_id']
)

access_token = token_response['accessToken']
refresh_token = token_response['refreshToken']
access_token

REST API request to create secret scope

POST https://adb-6187942982036570.10.azuredatabricks.net/api/2.0/secrets/scopes/create HTTP/1.1
Authorization: Bearer <token>

# For Keyvault backed scope
{
    "scope":  "restapiaad-keyvault-scope",
    "initial_manage_principal":  "users",
    "backend_azure_keyvault":  {
                                   "dns_name":  "https://aravish369vault.vault.azure.net/",
                                   "resource_id":  "/subscriptions/f34f9848-8b93-4cfb-9ced-0c4548a74cce/resourceGroups/aravishexternalrsgrp/providers/Microsoft.KeyVault/vaults/aravish369vault"
                               },
    "scope_backend_type":  "AZURE_KEYVAULT"
}

[gramhagen]

I am able to authenticate with AAD tokens to create a databricks personal access token, then authenticate again with the api using the databricks token to create secret scopes. I'm not sure if that works with keyvault backends, but I was running into the same issue just creating databricks backend scopes

[aravish]

@gramhagen,
Ultimately the AAD token should be used to create the secret scopes which should work as expected and not run into issues.

[michalmlaka]

Unfortunately does not working. Run into these issues:

https://forums.databricks.com/questions/50626/azure-databricks-api-20-cant-create-azure-keyvault.html

[hannwang1]

@aravish , Is there ETA to fix the issue that unable to create Azure Key Vault scope using Service Principal token? Using user AAD token is not a good solution for automation that is running on service principal. We cannot store personal password in automation.

We are automating the Azure databricks configuration, including Azure resource creation, databricks cluster creation, Azure key Vault secret scope creation with CI/CD. This issue is blocking us to automate the process.

[sushi1998]

I believe the PMs were talking to msft about enabling AKV creation with service principal tokens. @annashres might be the best person to talk to about this?

[hannwang1]

I believe the PMs were talking to msft about enabling AKV creation with service principal tokens. @AnnaShrestinan might be the best person to talk to about this?

Thanks @sushi1998 for the info. How to contact AnnaShrestinan?

[michalmlaka]

We tried it once again. It looks better now. (error msg.).

{"error_code":"CUSTOMER_UNAUTHORIZED","message":"Unable to grant read/list permission to Databricks service principal to KeyVault 'https://keyvaultname.vault.azure.net/': key not found: https://graph.windows.net/"}(base)

I assigned "Directory.Read.All" permission for AAD Graph API (https://graph.windows.net/). But it is not enough. What else do I need?

[JCZuurmond]

I think we are waiting to hear from @AnnaShrestinan like @sushi1998 mentioned. To the best of my knowledge it does not have to do with setting the right permissions for your SP.

[KevSpally-collab]

Any update on this, i am running into the same issue?

[JCZuurmond]

Not that I am aware off. What I know: databricks people are talking to Azure people. It most likely takes a while: months not weeks

[emerrf]

+1 same issue here. Please fix it. Creating a Secret Scope using a Service Principal with AAD Token is still not working.

[dkataria1990]

Same issue here, please fix as in my company the only way we can do things in higher envs is with SPN so it is not possible to config any other way!

[annashres]

We are working with Azure to enable the use of Service Principals for creating an AKV-backed secret scope. @hpsin from Azure AD will have updates on the timeline.

[mcalnd70]

@annashres @hpsin Has there been any updates on this with timelines etc.? The alternative way is just too much rework for CI/CD processes right now, especially as I am using a regular SP for everything else in the Databricks CLI

Thanks

[eosho]

Any updates on this? I am also having issues trying to set up key vault backed secret scope.

[ivo-o]

+1 only manual step in a huge deployment currently. Any updates on this? @hpsin

[hpsin]

We don't have timelines to announce right now on when SP impersonation is going to be supported. We're investigating how we build this - it requires the creation of a new class of scopes (delegation scopes for service principals) and updating the authorization logic of every API. We understand the importance of this feature for automation flows, and building it securely is our number one concern.

[m1nkeh]

We don't have timelines to announce right now on when SP impersonation is going to be supported. We're investigating how we build this - it requires the creation of a new class of scopes (delegation scopes for service principals) and updating the authorization logic of every API. We understand the importance of this feature for automation flows, and building it securely is our number one concern.

thanks for the update 👌

[atul-ram]

is this still in progress... Any updates !

[ghost]

@hpsin why not just use a managed identity like azure datafactory ?? this will probably also solve some other problems related to cloud auth, currently when creating scope on UI, how the refresh token is passed to clusters ?? is the user AAD Token stored somewhere permanently ??, what happens if the user token is revoked ??

[mallecespedes]

Any updates on this? @hpsin @annashres

[hpsin]

The creation of a new class of scope is almost complete, and the team is hoping to test soon. @psignoret and @annashres should be able to provide updates from here as I've transitioned away from Azure AD.

[jaokeeffe]

Just checking in to see how this request is tracking? Lots of users are eagerly awaiting this capability!

[lordlinus]

+1 looking for update on this one

[anocendi]

+1 @annashres and @psignoret, could you give the community an update regarding this issue? We are all eagerly waiting for this feature's arrival!

[dabruehl]

+1 I run into same issue. Any updates after 1.5 years?

[mwillebrands]

@annashres @hpsin any update?

[ralphke]

@hpsin @annashres shouldn't the Azure AAD OAuth 2.0 on-behalf-of-flow improvement help to implement this in DBX?
https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-on-behalf-of-flow

[annashres]

Hi @ralphke and @mwillebrands, @hpsin and I have both switched roles. Please refer to @nfx for updates from the product team.

[nfx]

@ralphke in scope of databricks/terraform-provider-databricks#1490 we’ll know if it’s already possible or not.

[ghost]

Unsubscribing from ticket.

[crystalgeek]

@nfx - Are there any updates on this - do we have a timeline?

[CaishenNefri]

Hey, Do we have roadmap for this issue?

[avinashpancham]

Hi @nfx, sorry for the spam, but do you have new information on this issue since the 22nd of July? Issue #1490 is still open, so I am hoping that would help us out.

[pietern]

We don't have new information to share on this topic.

Per the documentation at https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth2-on-behalf-of-flow:

The OBO flow only works for user principals at this time. A service principal cannot request an app-only token, send it to an API, and have that API exchange that for another token that represents that original service principal. Additionally, the OBO flow is focused on acting on another party's behalf, known as a delegated scenario - this means that it uses only delegated scopes, and not application roles, for reasoning about permissions. Roles remain attached to the principal (the user) in the flow, never the application operating on the users behalf.

This is the flow we need to create an AKV backed secret scope on behalf of an SP.

[jjgriff93]

Do we have an update on this? It's hard to believe that this still isn't supported.

[cabral1888]

Any update on this?

[psignoret]

@pietern @jjgriff93 @cabral1888 The statement in the article @pietern cites is true in the general case, but Azure Databricks has an exception and is able to make an OBO request on behalf of a service principal ("service principal on-behalf-of", or "SP OBO") to Azure Resource Manager. The Databricks team are working on making the necessary updates to make use of this when creating Azure Key Vault backed secret scopes.