Databricks - Access via Service Principal

Problem: to be able to use Azure Service Principal to access Databricks via JDBC or call it’s API.

Solution is below.

1. Create a Service Principal

I did it manually, but that can probably be scripted (outside of this scope).

2. Add Service Principal to Databricks

Even if you are using SCIM to provision users/groups to Databricks from Azure AD, it won’t work with Service Principals, and those need to be added to Databricks manually.

Unfortunately there is no UI to do that, but you can use SCIM API (Service Principal). Specifically, the Add call:

POST {{apiRoot}}preview/scim/v2/ServicePrincipals
Authorization: Bearer {{adminPat}}
Content-Type: application/scim+json

{
  "schemas":[
    "urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal"
  ],
  "applicationId":"{{clientId}}",
  "displayName":"My Cool SP",
  "groups":[
    {
       "value":"123456"
    }
  ],
  "entitlements":[
    {
       "value":"allow-cluster-create"
    }
  ]
}

where adminPat is your admin user’s PAT - you need it once to add SP. Once that is done, you can use Databricks UI to add SP to a group - it does actually work as the picture implies:

but after adding you’ll see it’s categorised as a User, which is probably a bug in databricks itself.

Once you have SP, it becomes a valid user-like object (hence wrongly classified as a “User”?) so you can add it to a group, assign permissions and so on.

3. Use Service Principal

You can now use BI Tools, API etc. using this Service Principal. This is how you use clientId/clientSecret of the SP to authenticate to

Azure AD

Like described here:

POST https://login.microsoftonline.com/{{tenantId}}/oauth2/token
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials
&client_id={{clientId}}
&resource=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d
&client_secret={{secret}}

which returns something like:

HTTP/1.1 200 OK
...

{
  "token_type": "Bearer",
  "expires_in": "3599",
  "ext_expires_in": "3599",
  "expires_on": "...",
  "not_before": "...",
  "resource": "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d",
  "access_token": "..."
}

Remember the access_token from the response, you need to to authenticate all the further calls.

Service Principal Personal Access Token (PAT)

Service Principal can also issue personal access tokens for itself, for instance:

###
# Create PAT as SP
POST {{apiRoot}}token/create
Authorization: Bearer {{token}}

{
  "lifetime_seconds": 31622400,
  "comment": "test pat"
}

which returns the token as a part of HTTP response. You can then use this PAT in the usual way.

Em, excuse me! Have Android 📱 and use Databricks? You might be interested in my totally free (and ad-free) Pocket Bricks . You can get it from Google Play too: