At Ackee, we use the Hashicorp Vault as the main secret storage. Usually, when we provision customer’s infrastructure with Hashicorp Terraform, we save credentials for GKE clusters and Cloud SQL instances in Vault so we can securely access them any time from Terraform while they are securely saved and not present in Terraform’s state file. Also, our applications fetch secrets utilizing Vault Injector.

Vault authentication

We try to gain as much as possible from the fact that we use Google Workspace as a single source of truth about who is doing what in our beautiful company. For our team, the team which is responsible for granting (and also removing of course) access to all services that our employees are using, Google account’s OAuth is the perfect way to decrease overhead of authorization granting. Offboarding is not even in our hands but is handled by the People & Culture department, so we don’t have to care about removing any user accounts. Vault was, unfortunately, an exception in this, we were using userpass auth method to authenticate for a long time and (de-)provisioning users in Terraform.

No more userpass, here comes OIDC

Previously we used a model where we had provisioned users with Terraform’s vault_generic_endpoint resource. It looked something like this:

resource "vault_generic_endpoint" "tomas.hejatko" {
  path    = "auth/userpass/users/tomas.hejatko"
  ignore_absent_fields = true
  write_fields = ["policies"]

  data_json = jsonencode({
   "policies" = [
        "projects/internal/project_one",
        "projects/internal/project_two",
        "projects/userpolicy/tomas.hejatko"
   ]
  })
}

Every user had their own ACL policy enabling them to change their own attributes (mainly password), looking somewhat like this:

path "auth/userpass/users/tomas.hejatko" {
  capabilities = ["list", "read", "update", "delete", "create"]
}

Then users were granted access to projects (their secrets, actually) by associating userpass object with project ACL policies, these looked like this:

# Project rule managed by terraform
path "secret/data/projects/internal/project_one/*" {
  capabilities = ["list", "read"]
}

# Frontend rule allowing access to list CloudFlare managed sites
path "secret/data/devops/cloudflare/sites/*" {
  capabilities = ["list"]
}

# Frontend rule allowing access to CF tokens managed by terraform
path "secret/data/devops/cloudflare/sites/domain.tld" {
  capabilities = ["read"]
}

Secrets were created in corresponding paths and users were granted access to projects they were working on by associating ACL policy.

Then we sit together and agreed that we have enough courage to stop using userpass auth method and authorize the user by Google Account via the OIDC auth method.

We’ve started with https://www.vaultproject.io/docs/auth/jwt/oidc_providers#google, followed instructions and came up with these commands:

OIDC auth method config:

vault write auth/oidc/config -<<EOF
{
 "oidc_discovery_url": "https://accounts.google.com",
 "oidc_client_id": "xxx.apps.googleusercontent.com",
 "oidc_client_secret": "xxx",
 "default_role": "oauth_default_role",
 "provider_config": {
     "provider": "gsuite",
     "gsuite_service_account": "/vault/sa.json",
     "gsuite_admin_impersonate": "tomas.hejatko@ackee.cz",
     "fetch_groups": false,
     "fetch_user_info": true
 }
}
EOF

We didn’t want to define this in Terraform, there are secrets needed to run Vault, so we don’t want to create cycle dependency. If we stored these secrets in Vault, then we would need running Vault to run Terraform to set up Vault.

OIDC default role

vault write auth/oidc/role/oauth_default_role \               
allowed_redirect_uris="https://vault-dev.ackee.cz/ui/vault/auth/oidc/oid
c/callback,http://localhost:8250/oidc/callback" \
      bound_audiences="xxx.googleusercontent.com" \
      user_claim="email" \
      oidc_scopes="openid email" \
      token_no_default_policy="true"

There are few points in this command I would like to point out:

oidc_scopes="openid email" – this configuration parameter requests the user’s email as an additional possible identifier

user_claim="email" – We use the user’s email as their identifier. This allows us to create vault_identity_entity resources bound to the user’s email, not their UID (which is a 21-digit number that users don’t know and it is not a user-friendly identifier). Please note that this practice goes AGAINST OIDC best practice, because the user’s email is not a unique identifier. In Ackee, we have an in-house employee management system, which creates Google accounts for us programmatically, so even if we delete a user from Google Workspace, we only “soft-delete” him in our internal system. This ensures that email is a de-facto unique identifier for us (our system does not allow us to create user with the same email, thus Google Workspace account is never done with an email that was previously used in our organization), but you should be aware of it and make sure it is enough for you.

token_no_default_policy="true" – If somebody outside of our organization (without @ackee.cz email) logs into Vault with OIDC, we have no way to prevent the authentication from succeeding. Thus we disable the default policy for accounts that we “don’t know”, effectively disabling them to do “anything after auth”. For more information about this please see this Github issue, which helped us understand not only why we need this parameter.

OIDC Users provisioning in Terraform

Now we have the OIDC auth method set up and we can provision users. To be more specific, we pre-create Identity entities which are identified by users’ emails and associate them with policies allowing them to see projects’ secrets.

First, define a pre-created user identity:

resource "vault_identity_entity" "oidc_user_identity" {
  name = "tomas.hejatko@ackee.cz"
  policies = [
    "projects/internal/project_one",
    "projects/internal/project_two",
    "projects/userpolicy/tomas.hejatko"
  ]
}

Now create an alias that will bind our pre-created identity with the OIDC auth backend:


data "vault_auth_backend" "oidc" {
  path = "oidc"
}

resource "vault_identity_entity_alias" "oidc_user_identity_binding" {
  name        = "tomas.hejatko@ackee.cz"
  mount_accessor = data.vault_auth_backend.oidc.accessor
  canonical_id   = vault_identity_entity.oidc_user_identity.id
}

What we got

We’ve set up a Vault instance that uses the OIDC auth backend, which authorizes against Google OAuth 2.0. We use Terraform to pre-create users in Vault by defining Vault identity entities identified by email from our Google Workspace. We create Vault identity aliases to bind identity entities to OIDC entities. We are aware that using a user’s email is not good practice because email is not a unique identifier, but it simplifies workflow for us a lot. By binding ACL policies to an identity entity object, we can reuse policies we were binding to userpass resources and we can avoid small policy resources which were granting users permission to edit their user password. 

Are you interested in working together? We wanna know more. Let’s discuss it in person!

Get in touch >